2020-11-18 Reval Legacy/Old GMC Sync

Date	18 Nov 2020
Authors	Philip Wilsdon (Unlicensed) John Simmons (Deactivated)
Status	Resolved
Summary	The sync jobs and ETLS failed Running of the correct ETL’s, `gmc-sync-prod` and `intrepid-reval-etl-all-prod` jobs fixed the issue. We decided not to improve monitoring to the existing sync jobs as new revalidation is soon going to be live Do need to add monitoring to the new GMC sync job and if we build any new ETLs or transformation services for new revalidation when we do this
Impact

Non-technical summary

Reval was showing information that had not been updated from GMC, on further investigation it turned out that the jobs that run overnight to get the data from GMC and then add it into TIS had not run successfully. Once they had been fixed and rerun, Reval showed the correct information.

Timeline

18 Nov 2020 -09:48 AM
18 Nov 2020 - 10:25 AM	Created ticket and incident page https://hee-tis.atlassian.net/browse/TISNEW-5728 2020-11-18 Reval Legacy/Old GMC Sync
18 Nov 2020 - Between 10:25 and 11:21	Ran the jobs `intrepid-reval-etl` and `intrepid-reval-etl-all-prod` Fixed the refresh of data but difference between the under notice values between TIS legacy/existing reval and GMC Connect
18 Nov 2020 12:06	Ran the correct ETL’s (`gmc-sync-prod` and `intrepid-reval-etl-all-prod`) via Jenkins
18 Nov 2020 12:25	Problem is assumed to have been fixed

Root Causes

The Jenkins scheduled jobs had been amended to run on a different server when we had the prod outage on Friday 13th November 2020. This change had been overlooked and not rolled back. Therefore the jobs could not run afterwards as there was a conflict in the inventory (as shown in the Jenkins output for each job, and the fact that each job ran for less than 1 second).

Trigger

A user reported in Teams Support Channel that their connections had not been working correctly

Resolution

Running of the correct ETL’s, gmc-sync-prod and intrepid-reval-etl-all-prod jobs fixed the issue.

Detection

A user reported in Teams Support Channel

Actions

For new reval - we need add monitoring so we know if the sync job to get the data from GMC and if we build any ETL/Transformation service are run successfully or fail - we could use this ticket https://hee-tis.atlassian.net/browse/TISNEW-3264
Decision taken not to address the monitoring in the current reval application as the new one is pretty close to being live (December 2020)

Lessons Learned (Good and Bad)

Still limited knowledge within the existing teams about how the existing module works (which is why the rebuild is taking place)
Current monitoring requires more investment to get it to work more reliably - problems with set to fail on the first occurrence and alerting would need to be written into the individual apps rather than checking of logs
Jobs need to be started via Jenkins
check the jenkins jobs for 1. what they do, and 2. what the logs for that run said.