Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Documentation

Date

Authors

Joseph (Pepe) Kelly Yafang Deng Jayanta Saha

Status

In Progress

Summary

Impact

Table of Contents

Non-technical Description

Trigger

...

Detection

There were 2274 messages in the dlq until we got the notification in #monitoring-prod Slack channel and shovelled them.

...

Resolution

...

Timeline

BST unless otherwise stated

  • 14:34 noticed the alert: “RabbitMQ PROD too many messages in the DLQ” in #monitoring-prod channel, then shovelled them to another queue “esr.dlq.2023.08.10“

  • 10:32 had a huddle and looked into the messages in queue “esr.dlq.2023.08.10“

Root Cause(s)

...

...

Action Items

Action Items

Owner

...