Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The system we use to send messages between our services ran out of memory. This led to a delay in updates across a number of TIS serviesservices, in particular Reval.

...

Trigger

RabbitMQ ran out of memory

...

Detection

Inspection of lastUpdatedDate column by developer

...

Resolution

Restarted Neo4j Consumers and managed syncing of queues until resources were cleared

...

Timeline

  • 21:23 Recommendations service reports failure to connect to rabbitmq

  • -

  • Restarted TCS - recreated queues -

  • -

5 Whys (or other analysis of Root Cause)

...

Action Items

Owner

Comments

Complete this ticket

Cai Willis

...

Lessons Learned