...
Non-technical Description
The system we use to send messages between our services ran out of memory. This led to a delay in updates across a number of TIS services, in particular Reval.
...
Trigger
RabbitMQ ran out of memory
...
Detection
Inspection of lastUpdatedDate column by developer
...
Resolution
Restarted Neo4j Consumers and managed syncing of queues until resources were cleared
...
Timeline
21:23 Recommendations service reports failure to connect to rabbitmq
09:41 Issue identified on revalidation by developer
~10:00 Restarted TCS - recreated queues
10:42 Revalidation doctors list restored, users notified
5 Whys (or other analysis of Root Cause)
Why was the doctors list not updated? Because the overnight sync failed to run
Why did the overnight sync fail to run? because the message to start the sync job was “stuck”
Why was the message to start the sync job stuck? because there was not enough available memory in the rabbitmq cluster and no new messages were being consumed
Why was there not enough memory in the rabbitmq cluster? because the neo-audit queue does not get processed quickly enough (1,000,000+ messages in the broker at all times) -other reasons?
Why does the neo-audit queue not get processed quickly enough? because it is running on old infrastructure and only has a couple of very slow instances consuming it
...
Action Items
Action Items | Owner | Comments |
|---|---|---|
Complete this ticket | ||
...
Lessons Learned
Tech improvement tickets are important