Date	12 Feb 2021
Authors	Joseph (Pepe) Kelly Reuben Roberts
Status	Resolved
Summary	Person Search Sync Failed
Impact	Person search page was not showing some data between 07:35 and 08:00

Non-technical Description

The overnight sync procedure for TIS was unable to run. This meant only some trainees were being shown on the person search page.

Trigger

The issue was detected when errors were noted in the ‘Sync [Person sync job] started’ Slack notification, and the ‘Sync [Person sync job] finished’ notification failed to appear.

Created a new production elasticsearch cluster to use based on the terraform description (instance_type = t3.medium.elasticsearch instead of the incorrectly set instance_type = t3.small.elasticsearch)
Manually triggered the Person sync job to rebuild the Person elasticsearch index.

12 Feb 2021 - 06:20 - Noted Person sync job errors on STAGE and PROD
12 Feb 2021 - 06:29 - Quickest fix (simply re-running the job) observed not to resolve the issue on STAGE
12 Feb 2021 - 07:35 - Question raised by user on Teams
12 Feb 2021 - 07:40-07:50 - Rebuilt the elasticsearch cluster infrastructure as noted in ‘Resolution’ above.
12 Feb 2021 - 07:51-07:58 - Manually re-ran the Sync job; trainees becoming visible during this time.
12 Feb 2021 - 08:00 - Confirmed issue resolved with users

The Person elasticsearch cluster had been incorrectly reconfigured on 11 Jan 2021 to use t3.small.elasticsearch instances. This caused the sync job to fail.

Action Items	Owner
n/a

Manually running the terraform script without going through the normal pull-request and approval process for the TIS-OPS project exposes the infrastructure to a greater risk of being accidentally misconfigured.