| Date |
|
| Authors | Graham O'Regan (Unlicensed) |
| Status | Complete |
| Summary | Elasticsearch indexing appeared to fail because it exceeded the 30 second timeout. However, the snapshot was created so the job couldn't complete if run again. |
| Impact | None |
| Table of Contents |
|---|
Root Cause
The timeout for the task was the default 30secs but taking the snapshot and copying it to Blob Storage took longer.
Trigger
The nightly ES index snapshotting job on Jenkins
Resolution
The timeout was set to 20mins to allow more time for the snapshot to complete before raising an error.
Detection
The Jenkins job alerts to the #dev Slack channel.
Action Items
| Action Item | Type | Owner | Issue |
|---|---|---|---|
| Increased timeout to 20mins | prevent | Graham O'Regan (Unlicensed) |
Timeline
- Nightly job failed and alerted #dev in Slack
- An attempt to run the snapshot manually with curl also failed with a different exception because the snapshot did exist.
Supporting Information
Link to initial failed task.
https://build-hee.transformcloud.net/jenkins/job/elasticsearch-snapshot-prod/25/