2020-07-07 Person Training Body Job failed.

Date

Jul 7, 2020

Authors

@Joseph (Pepe) Kelly

Status

 

Summary

One of the overnight updates failed (TrrainingBody Job)

Impact

Potential for some Trust users to be missing access to some trainees

 

Root Cause(s)

  • Unable to get connection to MySQL

Trigger

  • Dropped/Unhealthy connection

Resolution

  • Logged into the sync webpage

  • Ran PersonPlacementTrainingBodyTrustJob and PersonElasticSearchJob to associate trust with people on placements there and then make it available via the search page.

Detection

  • Slack monitoring (07:30)

 

Timeline

  • 01:36 - Connections to the database fail/closed

  • 07:30 - Slack notification

  • 09:30 - Restarted sync services to force new connections and reran jobs