...
Why ESR files were not processed in the morning of 26th/Jun? - Because the RabbitMQ was not accepting any incoming traffic and was not able to process transactions.
Why was RabbitMQ not able to accept incoming traffic and process transactions? - Because there was not enough memory.
Why there was not enough memory in RabbitMQ? - Because it was occupied by
esr.queue.audit.neoqueue. There were more than 4 million messages in it and it ate up the resource.Why there were more than 4 million messages in
esr.queue.audit.neoqueue? - Currently, we don’t knowThe message consumer could not read messages because the database was unavailable.
...
Action Items
Action Items | Owner | Comments | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Investigate why there are so many message in Monitor the queue for a period of time to find out if there’re still rapid increasing on the number of messages. |
| |||||||||
Investigate why we still have incoming messages in | This has led to missing another issue which appears to have increased since the start of June | |||||||||
Following above, test whether instance size or green/blue impacts the number of error messages produced? | ||||||||||
Clean up queues which aren’t needed right now, even if likely to be added in the near future? |
...
Lessons Learned
Learn more knowledge about ESR.