Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Current »

Date

Authors

Joseph (Pepe) Kelly, Marcello Fabbri (Unlicensed)

Status

Done.

Summary

Some exported placements show an unknown ESR status (?) on TIS instead of correctly displaying their exported status (✔)

Impact

Inaccurate information regarding some placement’s ESR status

Non-technical Description

The TIS-ESR interface exports data to ESR daily. All the Applicants were sent to ESR as expected. However the interface failed to successfully communicate the completed export of some Placements to TIS, due to TIS’s momentary unavailability.

As a result, these Placement’s status on TIS remained unclear. They appeared in lists of placements with a question mark on the frontend (?) instead of displaying the correct exported status (marked with a tick ✔).

The process has been updated so that when updates fail because of connection problems, there are reasonable attempts to re-attempt the communication to TIS.

Trigger

  • TCS momentarily unavailable (updates sent via REST calls not processed)

Detection

  • Great number of messages being dumped in the Dead Letter Queue (triggering a monitor alert)

Resolution

  • When Placements are exported to ESR, the Data Export service sends a message via RabbitMQ queue to the Inbound Data Writer service.

Timeline

  • : 15:33 BST - Slack notification regarding a high volume of messages in the Dead Letter Queue

  • : 15:58 BST - Poor interaction between the InboundDataWriterService and TCS identified as the culprit for messages being discarded when info hadn’t successfully been recorded into TIS

  • : Fix put in place to increase resilience of the InboundDataWriterService when interacting with TCS

  • : Affected data amended in order to display accurate ESR export status

Root Cause(s)

  • TCS didn’t have the data to show the tick to say the placement was updated.

  • The Inbound Data Writer service updates to TCS failed, which is responsible for updating the PlacementEsrEvent table where this data’s stored.

  • The message was not requeued (therefore re-processing was not attempted), and the updates where not applied.

  • TCS was momentarily unavailable* right when the Inbound Data Writer service sent the REST call and didn’t accommodate that call. It didn’t update anything.

  • The Inbound Data Writer service, treated TCS’s unavailability like a problem with the message which aren’t requeued.

  • *why was TCS unavaliable?

Action Items

Action Items

Owner

Status

Fix current Placements whose status is currently inaccurate

Edward Barclay

https://hee-tis.atlassian.net/browse/TIS21-1650

Make the Inbound Data Writer service more resilient so it requeues the messages when TCS doesn’t respond

Marcello Fabbri (Unlicensed)

https://hee-tis.atlassian.net/browse/TIS21-1651

Check elsewhere in the ESR interface for places where requeuing would be appropriate

Marcello Fabbri (Unlicensed)

https://hee-tis.atlassian.net/browse/TIS21-1746

Lessons Learned

  • Consider more carefully when it’s appropriate to requeue a message (re-attempt processing it) and when it’s ok not to requeue a message.

  • No labels

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213