22 Oct 2023: 23:56 Restarting Consumer logs appear from recommendation service
22 Oct 2023: 23:56 Suspended all listeners and will no longer accept client connections logs appear from AWS RabbitMQ broker
23 Oct 2023: 00:01memory resource limit alarm set on node and Publishers will be blocked until this alarm clears logs appear from AWS RabbitMQ broker
23 Oct 2023: 00:05 Start message has been sent to start gmc sync log appears in recommendation service, but this message is not delivered until much later
23 Oct 2023: 09:52 GMC Client Service receives a message to start the doctor sync, and the sync begins.

Root Cause(s)

5 Whys

Why didn’t the GMC Overnight Sync Job Start on time? - Because RabbitMQ ran out of memory
Why did RabbitMQ run out of memory? -
Why
Why
Why

Action Items	Owner
Unbind and delete the queue `reval.queue.gmcsync.connection` (It’s not currently used by any application)

...