Date | |
Authors | Joseph (Pepe) Kelly John Simmons (Deactivated) Yafang Deng Reuben Roberts Jayanta Saha Edward Barclay |
Status | Resolved |
Summary | Person search sync job failed |
Impact |
Non-technical Description
We run a number of sync jobs overnight. This one failed - another process was taking place that prevented it from successfully running.
We re-ran the job shortly afterwards and it completed successfully.
We investigated what tripped it up and will work to mitigate a recurrence.
Trigger
garbage collection activity taking longer than expected and eating into the sync job schedule.
Detection
monitoring-prod Slack alert.
Resolution
re-running the job as soon as it was noticed.
Timeline
2022-02-18|01:41: “Sync [Person sync job] failed with exception…” message in Slack monitoring-prod channel.
2022-02-18|03:13: Team member restarted the job when they notice the issue.
2022-02-18|03:24: Rerun job completed successfully.
Root Cause(s)
see TIS21-2697.
Action Items
Action Items | Owner |
|---|---|
Lessons Learned
.
0 Comments