Versions Compared

Old Version 5

changes.mady.by.user Marcello Fabbri

Saved on

compared with

New Version 6

changes.mady.by.user Marcello Fabbri

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Authors

Joseph (Pepe) Kelly, Marcello Fabbri (Unlicensed)

Status

LiveDefect done. Investigating mitigations for the future

Summary

Some exported placements show an unknown ESR status (?) on TIS instead of correctly displaying their exported status (✔)

Impact

Inaccurate information regarding some placement’s ESR status

...

  • .When Placements are exported to ESR, the Data Export service sends a message via RabbitMQ queue to the Inbound Data Writer service.

...

Timeline

  • : : 15:33 BST - Slack notification regarding a high volume of messages in the Dead Letter Queue

  • :

  • :

  • :

  • : 15:58 BST - Poor interaction between the InboundDataWriterService and TCS identified as the culprit for messages being discarded when info hadn’t successfully been recorded into TIS

  • : Fix put in place to increase resilience of the InboundDataWriterService when interacting with TCS

  • : Affected data amended in order to display accurate ESR export status (TBC)

Root Cause(s)

  • .When Placements are exported to ESR, the Data Export service sends a message via RabbitMQ queue to the Inbound Data Writer service.

  • The Inbound Data Writer service normally sends the updates to TCS via REST call, which is responsible for updating the PlacementEsrEvent table where this data’s stored.

  • TCS was momentarily unavailable right when the Inbound Data Writer service sent the REST call and didn’t accommodate that call. It didn’t update anything.

  • The Inbound Data Writer service, receiving a specific error in response to TCS’s unavailability, had a clause in place aimed at not requeuing the message in such case.

  • The message was not requeued (therefore re-processing was not attempted), and the updates where not applied.

...

Action Items

Owner

Status

Fix current Placements whose status is currently inaccurate

Edward Barclay

ongoing

Make the Inbound Data Writer service more resilient so it requeues the messages when TCS doesn’t respond

Marcello Fabbri (Unlicensed)

done

Check elsewhere in the ESR interface for places where requeuing would be appropriate

Marcello Fabbri (Unlicensed)

ongoing

...

Lessons Learned

  • Consider more carefully when it’s appropriate to requeue a message (re-attempt processing it) and when it’s ok not to requeue a message.