Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Date

Authors

Cai Willis

Status

TIS21-6217 - Getting issue details... STATUS

Summary

RabbitMQ ran out of memory, affecting TIS services (particularly Reval)

Impact

Any data updated on GMC day before has not/will not reflect on our system

Non-technical Description

The system we use to send messages between our services ran out of memory. This led to a delay in updates across a number of TIS services, in particular Reval.


Trigger

RabbitMQ ran out of memory


Detection

Inspection of lastUpdatedDate column by developer


Resolution

Restarted Neo4j Consumers and managed syncing of queues until resources were cleared


Timeline

  • 21:23 Recommendations service reports failure to connect to rabbitmq

  • -

  • Restarted TCS - recreated queues -

  • -

5 Whys (or other analysis of Root Cause)


Action Items

Action Items

Owner

Comments

Complete this ticket

Cai Willis


Lessons Learned

  • No labels

0 Comments

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account.