Issue of large numbers of ‘failed’ onboarding emails reported to TSS team 7 Nov 2024.

5 Whys (or other analysis of Root Cause)

Onboarding notification emails were not sent because no trainee email address was available in TSS.
No trainee email address was available in TSS because it had not been sync’d from TIS.
Data had not been sync’d from TIS because no full-data request had been made from TSS, and the records in question had not been updated since the profile was created, which would have triggered a normal sync.
No full-data request had been made from TSS because the TSS sync service could not deserialize the messages instructing it to make the request.
The TSS sync service could not deserialize the messages because of a new message attribute was instructing it to deserialize to a non-available class.
The new message attribute was included as part of a major version update to the library component that handles messaging.
The breaking change was not detected by automated tests within individual service components, because it only impacts messaging between components, for which we have no automated tests.
The subsequent failures in production were not detected because of a lack of error detection and reporting for the specific message queue and the sync component.

...

Resolution

...

All times GMT unless otherwise indicated.

...