Non-technical Description

The connection between TIS Self-Service and TIS was broken intermittently 23-24 July 2023. As a result, the COJ signed in TIS Self-Service during that period were not loaded into TIS, which meant that several programme memberships did not have the correct COJ status in Admins UI and Tableau.

Trigger

• RabbitMQ instability in the time period mentioned.

Detection

• Missing CoJ reported 10:30AM 4 Aug 2023 on the self-service-support Slack channel (https://hee-nhs-tis.slack.com/archives/G0135LS4JVA/p1691141400259379 ), and another example reported shortly thereafter.

Resolution

• Audit of CoJ’s signed during the period 23 - 24th July 2023 uncovered 10 that had not been processed correctly and received by TIS. These were manually resent and their correct processing verified.

Timeline

All times in BST unless indicated

• 23 Jul 2023: ~23:50 - RabbitMQ instability begins within weekly maintenance window (10PM - Midnight UTC / 11PM - 1AM BST).

  

• 24 Jul 2023: 02:10 - Exceptions in services

• 24 Jul 2023: ~16:15 - RabbitMQ instability ends. Some services (MySQL CDC, ESR Reconciliation) which would have added message load were paused while messages in the broker were consumed and/or cleared out.

• 04 Aug 2023: 10:30 - Missing CoJ in TIS reported.

• 07 Aug 2023: 16:24 - Audit of CoJ messages completed and missing CoJ resent to TIS.

Root Cause(s)

• RabbitMQ became unstable due to resource limits being reached (memory, due to excessive messages being held and not processed).
  

  

• TIS Self-Service code to submit messages to RabbitMQ did not check for successful processing.

• CoJs were successfully saved within TIS Self-Service, but not received by TIS due to messaging failure.

• The lack of alerting of the failures, or of the resulting data discrepancy, meant that we relied on user reports to become aware of the issue.

Action Items

Lessons Learned

• We need to check and handle errors from ‘infrastructure’ more carefully

• The lack of alerting on failures and/or automated data consistency checks meant we were not aware of the problem until notified by users, which is poor.