Date	22 Dec 2016
Authors	Grante Marshall (Unlicensed) Graham O'Regan (Unlicensed)
Status	Complete
Summary	Elasticsearch snapshots being deleted in Production
Impact	didn't affect service

Root Cause

The configuration settings for Curator on production were set to only retain a single snapshot. When the snapshot process failed Curator removed the only remaining snapshot in Azure Blobstorage.

Trigger

The nightly snapshotting jobs from Jenkins failed because existing snapshots already existed. Curator then ran and deleted the only remaining snapshot.

Resolution

The number of snapshots was increased to 5, one per day.

Detection

There are Jenkins jobs that run the Elasticsearch snapshots which failed and the failures were reported to the #dev channel in Slack.

Action Items

Action Item	Type	Owner	Issue
Increase the number of snapshots retained	prevent	Grante Marshall (Unlicensed)	TISDEV-1459 - Getting issue details... STATUS

Timeline

11pm Jenkins job ran
Jenkins sent notification to Slack
19 Dec 2016 N (Unlicensed) changed the Curator settings to retain 5 days of snapshots.

Supporting Information

elasticsearch-snapshot-prod#5

Browser not supported