Date |
| ||||||||
Authors | |||||||||
Status | DocumentingDone | ||||||||
Summary |
| ||||||||
Impact | Doctors were not updated and reval was not available for some regions for the whole day. The updates happened sequentially, as below. |
...
Why didn’t the GMC Overnight Sync Job Start on time? - Because RabbitMQ ran out of memory
Why did RabbitMQ run out of memory? - Because of millions of messages in the following queues:
esr.queue.audit.neo,tis.queue.cdc.created,reval.queue.gmcsync.connection(is this the real reason?)Why were there millions of messages in these queues? - Why so many? Is this normal? Why was it taking so long to process them?
Why
WhyNeoAudit containers have a habit of dropping connection to RabbitMQ, CDC rabbit router was unable to consume/publish messages from the queue
Why do the containers drop connection to RabbitMQ? - This requires an investigation, it’s a not uncommon issue where RabbitMQ and ECS drop connections without an obvious cause
Causes
RabbitMQ ran out of memory
esr.queue.audit.neoandtis.queue.cdc.createdhad millions of messages (recorded at about 10:30 am) (Active queues)reval.queue.gmcsync.connectionhad 1.6 million messages (Obsolete unused queue)
...
Action Items | Owner | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Unbind and delete the queue | DONE | |||||||||
Investigate why NeoAudit lost connection to RabbitMQ (Ticket) | Ticket Created:
| |||||||||
Set up alerts for RabbitMQ low memory (Ticket) | Ticket Created:
| |||||||||
(Nice to have) Broader RabbitMQ health check alerting |
...
Lessons Learned
Don’t leave obsolete queues lying around - especially when they’re still being published to
There’s no such thing as too much alerting