Date |
|
Authors | |
Status | Documenting |
Summary | |
Impact | No end user impact. The database that holds the information for communicating with ESR was less resilient for a couple of hours and the integration was paused for several hours. |
Non-technical Description
The database has 3 copies for resilience against failures. Attempting to create a backup used all disk space available to one of these copies, one “node” in the mongo replicaset. This caused the node to fail. The synchronisation between it and the other nodes meant that it could not automatically recover when the disk space was released again.
The data for the node was removed to allow a “full sync/initialisation” to take place. This took several hours and the integration services were switched off during this period to prevent cascading issues. When the services resumed, the pending events from TIS, e.g. updates to personal details, were processed.
Trigger
Disk space full
Detection
TIS team. The backup process was being monitored.
Resolution
Reinitialising the data directory for the failed node and allowing a full synchronisation to take place.
Restarting the machine the database was on following the synchronisation
Timeline
: BST - Backup run
: BST -
: BST -
Root Cause(s)
Backup written to
Action Items
Action Items | Owner | |
|---|---|---|
Lessons Learned
Just never output to the same device that data is written on.
Add Comment