Non-technical Description

• .The TIS-ESR interface exports data to ESR daily. The interface failed to successfully communicate the completed export of some Placements to TIS, due to TIS’s momentary unavailability. As a result, these Placement’s status on TIS remained unclear (marked with a question mark on the frontend (?) instead of displaying the correct exported status (marked with a tick ✔).

Trigger

• . TCS momentarily unavailable (updates sent via REST calls not processed)

Detection

• .

Resolution

• .When Placements are exported to ESR, the Data Export service sends a message via RabbitMQ queue to the Inbound Data Writer service.

Timeline

• 08 Jun 2021:

• 08 Jun 2021:

• 08 Jun 2021:

• 08 Jun 2021:

• 08 Jun 2021:

• 08 Jun 2021:

Root Cause(s)

• .When Placements are exported to ESR, the Data Export service sends a message via RabbitMQ queue to the Inbound Data Writer service.

• The Inbound Data Writer service normally sends the updates to TCS via REST call, which is responsible for updating the PlacementEsrEvent table where this data’s stored.

• TCS was momentarily unavailable right when the Inbound Data Writer service sent the REST call and didn’t accommodate that call. It didn’t update anything.

• The Inbound Data Writer service, receiving a specific error in response to TCS’s unavailability, had a clause in place aimed at not requeuing the message in such case.

• The message was not requeued (therefore re-processing was not attempted), and the updates where not applied.

Action Items

Lessons Learned

• Consider more carefully when it’s appropriate to requeue a message (re-attempt processing it) and when it’s ok not to requeue a message.