...

• 16 Mar 2021 - 14:30 - TCS service deployed to production

• 16 Mar 2021 - 14:42 - Error in ESR APP generation notified in Slack

• 16 Mar 2021 - 14:44 - Issue picked up by devs

• 16 Mar 2021 - 16:16 - Fix deployed for EsrAppRecordGeneratorService

• 16 Mar 2021 - 16:32- Fix deployed for EsrNotificationGeneratorService

• 16 Mar 2021 - 17:39 - Fix deployed for EsrInboundDataWriterService

• 16 Mar 2021 - 18:48 - Fix deployed for TIS-EsrReconciliationService

• 17 Mar 2021 - 00:09 - TIS overnight sync jobs failed

• 17 Mar 2021 - 02:30 - Issue picked up by devs

• 17 Mar 2021 - 03:55 - Fix deployed to stage for TIS-SYNC - decision made not to merge to prod at this time due to auto-start logic before 05:00

• 17 Mar 2021 - 05:20 - TIS-SYNC fix deployed to production environment and sync jobs trigger - NIMDTA jobs didn’t fully trigger due to permissions(?)

• 17 Mar 2021 - 10:45 - NIMDTA jobs re-ran and completed successfully

• 19 Mar 2021 - 18:20 - Retriggered applicant generation based on trainees reported in Sentry errors

Root Cause(s)

• TCS deployed with changes to the PermitToWork field, changing it from an enumeration to a string.

• The (de)serialization of the RightToWork object in ESR and TIS-SYNC projects began to fail.

• Outdated tcs-client and tcs-persistence used so those project still tried to treat PermitToWork as an enumeration.

• Updating dependencies in those projects was missed as a search for usages of PermitToWorkType did not find them.

• Those projects do not directly use PermitToWork, but do (de)serialize the Person object which has it nested.

...