Non-technical Description

We run a number of sync jobs overnight. This one failed - another process was taking place that prevented it from successfully running.

We re-ran the job shortly afterwards and it completed successfully.

We investigated what tripped it up and will work to mitigate a recurrence.

Trigger

• garbage collection activity taking longer than expected and eating into the sync job schedule.

Detection

• monitoring-prod Slack alert.

Resolution

• re-running the job as soon as it was noticed.

Timeline

• 2022-02-18|01:41: “Sync [Person sync job] failed with exception…” message in Slack monitoring-prod channel.

• 2022-02-18|03:13: Team member restarted the job when they notice the issue.

• 2022-02-18|03:24: Rerun job completed successfully.

Root Cause(s)

• see TIS21-2697.

Action Items

Lessons Learned

• .