2019-06-21 Person Trust Jobs failed

Owned by Joseph (Pepe) Kelly
Last updated: 25 Jun 2019
Date

 

AuthorsJoseph (Pepe) Kelly
StatusComplete
Summary


ImpactNone found.

Root Cause(s)

Person Elastic Search scheduled job during the night at 1:30am failed to synchronize on different cluster's nodes.

Trigger

The TIS-SYNC application failed with an OutOfMemoryError.

Resolution

Re-ran jobs as on Sync Service.

Detection

Alerting in Slack.

Action Items

Action ItemTypeOwnerIssue
Restart the jobsRestore serviceJoseph (Pepe) Kelly

TISNEW-3099 - Getting issue details... STATUS

Increased JVM Max Heap Size parameterPrevent recurrence

John Simmons (Deactivated) and Joseph (Pepe) Kelly



Timeline

  • 0:29 am TIS-SYNC job failed
  • 7.30 am Slack alerting of failed job in #monitoring
  • 8.58 am Re-ran jobs
  • 1:30 pm change to parameter implemented to reduce the likelihood of a re-occurrence.

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213