...

• Sentry error received in #sentry-esr that the ESR-NOTIFICATIONGENERATOR service could not access trust codes.

• User notification that bulk upload was displaying error messages.

...

Resolution

• TIS-REFERENCE restarted on prod blue to restore functionality.

• Affected services re-deployed (ESR-NOTIFICATIONGENERATOR, TIS-CONCERNS, TIS-PROFILE and TIS-SYNC).

• TIS-REFERENCE stopped again on prod blue.

...

• 04 Mar 2022 16:00 - TIS-REFERENCE stopped on all environments

• 04 Mar 2022 16:03 - ESR notification generator exception picked up by Sentry and sent as slack notification.

• 04 Mar 2022 16:16 - TIS-REFERENCE restarted on prod blue.

• 04 Mar 2022 16:19 - Redeployments underway for ESR-NOTIFICATIONGENERATOR, TIS-CONCERNS, TIS-PROFILE and TIS-SYNC

• 04 Mar 2022 18:15 - Last of the deployments completed

• 07 Mar 2022 12:30 - TIS-REFERENCE stopped on prod blue

• 08 Mar 2022 14:41 - Redeployed TIS-GENERIC-UPLOAD following a report that bulk upload was not working

...

Root Cause(s)

• The load balancer config was updated to point to the new ECS instance of TIS-REFERENCE

• The load balancer changes were not deployed to ESR-NOTIFICATIONGENERATOR, TIS-CONCERNS, TIS-GENERIC-UPLOAD, TIS-PROFILE and TIS-SYNC

• Stopping the TIS-REFERENCE service on blue/green then meant that the service could not be found instead of using the ECS instance.

...

• Ensure services aren’t missed when deploying changes

• Decommissioning/migrating is a bit more challenging: could we validate no active use as we decommission other services?