Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Action Items

Owner

Status

Why so many placementSpecialties are not found in TISSS sync, triggering another call to TIS sync? Are these for deleted records, if they are in turn not found by TIS sync, or is there a bug / other issue here?

[Found that: “spot checked some of the PlacementSpecialty messages in the DLQ, and found that the their placementIds are for Placements that have been recently deleted (found them in the ESR Audit queue tis.placement.deleted If a Placement gets deleted, as far as I know, the PlacementSpecialty gets deleted as well]

Marcello Fabbri (Unlicensed)

Done

Investigate adjusting the cron record-resending job for TISSS sync: either stop it running during the nightly sync job timeframe, run it every 10min instead of every 1min, or determine how to disable it while another job is running

TIS21-2107

Rerun the sync jobs one by one, confirm success at each step

Joseph (Pepe) Kelly

Done

Write tickets to handle record deletions (currently marked with a //TODO)

Contact AWS to determine if there was an issue with SQS

Feedback was that there were no particular issues with SQS around the time of the incident, but an open issue on the aws-java-sdk was referenced (https://github.com/aws/aws-sdk-java/issues/2269 ).

Reuben Roberts

Done (Case ID 8876575931)

Add ticket to investigate better error handling (e.g. backoff and retry) on SQS receiveMessage as per Github issue referenced above.

Reuben Roberts

Done (https://hee-tis.atlassian.net/browse/TIS21-2110 )

...

Lessons Learned

  • Group review for RCA and identifying alternative scenarios is very useful!