Date	06 Jan 2017
Authors	Graham O'Regan (Unlicensed)
Status	Complete
Summary	Elasticsearch indexing appeared to fail because it exceeded the 30 second timeout. However, the snapshot was created so the job couldn't complete if run again.
Impact	None

Root Cause

The timeout for the task was the default 30secs but taking the snapshot and copying it to Blob Storage took longer.

Trigger

The nightly ES index snapshotting job on Jenkins

Resolution

The timeout was set to 20mins to allow more time for the snapshot to complete before raising an error.

Detection

The Jenkins job alerts to the #dev Slack channel. Graham O'Regan (Unlicensed) checked to make sure that the container was running on the prod server and ran the commands manually with curl to see what the underlying problem was.

Action Items

Action Item	Type	Owner	Issue
Increased timeout to 20mins	prevent	Graham O'Regan (Unlicensed)

Timeline

Nightly job failed and alerted #dev in Slack
An attempt to run the snapshot manually with curl also failed with a different exception because the snapshot did exist.

Supporting Information

Link to initial failed task.

https://build-hee.transformcloud.net/jenkins/job/elasticsearch-snapshot-prod/25/

TIS21 Confluence Space

2017-01-06 ES Snapshotting error

Analytics