Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Date

Authors

Joseph (Pepe) Kelly Reuben Roberts

Status

In Progress

Summary

Person Search Sync Failed

Impact

Person search page was not showing some data between 07:35 and 08:00

Non-technical Description

The overnight sync procedure for TIS was unable to run. This meant only some trainees were being shown on the person search page.

We increased the resources for the affected component

Trigger

Detection

  • The issue was detected when errors were noted in the ‘Sync [Person sync job] started’ Slack notification, and the ‘Sync [Person sync job] finished’ notification failed to appear.

Resolution

  • Created a new production elasticsearch cluster to use based on the terraform description (instance_type = t3.medium.elasticsearch up from instance_type = t3.small.elasticsearch)

  • Manually triggered the Person sync job to rebuild the Person elasticsearch index.

Timeline

  • 01:46 - Sync job failed

  • 08:31 - Sync job failed when re-run

Root Cause(s)

  • The nightly sync job failed.

Action Items

Action Items

Owner

n/a

Lessons Learned

  • Man

  • No labels

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213