Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

Date

Authors

Philip Wilsdon (Unlicensed) John Simmons (Deactivated)

Status

Resolved

Summary

  • The sync jobs and ETLS failed

  • Running of the correct ETL’s, gmc-sync-prod and intrepid-reval-etl-all-prod jobs fixed the issue.

  • We decided not to improve monitoring to the existing sync jobs as new revalidation is soon going to be live

  • Do need to add monitoring to the new GMC sync job and if we build any new ETLs or transformation services for new revalidation when we do this

Impact

Non-technical summary

Reval was showing information that had not been updated from GMC, on further investigation it turned out that the jobs that run overnight to get the data from GMC and then add it into TIS had not run successfully. Once they had been fixed and rerun, Reval showed the correct information.

Timeline

-09:48 AM

- 10:25 AM

Created ticket and incident page https://hee-tis.atlassian.net/browse/TISNEW-5728

2020-11-18 Reval Legacy/Old GMC Sync

- Between 10:25 and 11:21

Ran the jobs intrepid-reval-etl and intrepid-reval-etl-all-prod

Fixed the refresh of data but difference between the under notice values between TIS legacy/existing reval and GMC Connect

12:06

Ran the correct ETL’s (gmc-sync-prod and intrepid-reval-etl-all-prod) via Jenkins

12:25

Problem is assumed to have been fixed

Root Causes

  • The Jenkins scheduled jobs had been amended to run on a different server when we had the prod outage on Friday 13th November 2020. This change had been overlooked and not rolled back. Therefore the jobs could not run afterwards as there was a conflict in the inventory (as shown in the Jenkins output for each job, and the fact that each job ran for less than 1 second).

Trigger

  • A user reported in Teams Support Channel that their connections had not been working correctly

Resolution

  • Running of the correct ETL’s, gmc-sync-prod and intrepid-reval-etl-all-prod jobs fixed the issue.

Detection

  • A user reported in Teams Support Channel

Actions

  • For new reval - we need add monitoring so we know if the sync job to get the data from GMC and if we build any ETL/Transformation service are run successfully or fail - we could use this ticket https://hee-tis.atlassian.net/browse/TISNEW-3264

  • Decision taken not to address the monitoring in the current reval application as the new one is pretty close to being live (December 2020)

Lessons Learned (Good and Bad)

  • Still limited knowledge within the existing teams about how the existing module works (which is why the rebuild is taking place)

  • Current monitoring requires more investment to get it to work more reliably - problems with set to fail on the first occurrence and alerting would need to be written into the individual apps rather than checking of logs

  • Jobs need to be started via Jenkins

  • check the jenkins jobs for 1. what they do, and 2. what the logs for that run said.

  • No labels

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213