2022-02-18 Person Search List failed to refresh

Owned by Andy Nash (Unlicensed)
Last updated: 18 Feb 2022 by Reuben Roberts
1 min read

Date

Feb 18, 2022

Authors

@Joseph (Pepe) Kelly @John Simmons (Deactivated) @Yafang Deng @Reuben Roberts @Jayanta Saha @Edward Barclay

Status

Resolved

Summary

Person search sync job failed

Impact

Up to 1,000 person records (out of 290,000) that weren’t findable on the Person search page from 01.41 - 03.24

Non-technical Description

We run a number of sync jobs overnight. This one failed (see ‘Impact', above) - another process was taking place that prevented it from successfully running.

We re-ran the job shortly afterwards and it completed successfully.

We investigated what tripped it up and will work to mitigate a recurrence.

Trigger

  • garbage collection activity taking longer than expected and eating into the sync job schedule.

Detection

Resolution

  • re-running the job as soon as it was noticed.

Timeline

  • 2022-02-18|01:41: “Sync [Person sync job] failed with exception…” message in Slack monitoring-prod channel.

  • 2022-02-18|03:13: Team member restarted the job when they notice the issue.

  • 2022-02-18|03:24: Rerun job completed successfully.

Root Cause(s)

Action Items

Action Items

Owner

Action Items

Owner

Refactor the sync job to be robust enough to retry on error - a spike ticket to look at the options?
- task executor like we do elsewhere in TIS?
- REST client retry?
- Spring component for retrying method calls (configurable) - example in Reval (thanks Uzair)?

@Reuben Roberts https://hee-tis.atlassian.net/browse/TIS21-2698

Lessons Learned

  • Its good to retry when you fail!

  • Even highly available systems have issues.

  • Task-based components could do with a bit more defensive development (around retries, consider things other than the ‘happy path’).

  • Our monitoring works nicely (for anyone who’s an insomniac).

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213