Date |
|
Authors | |
Status | TIS21-6217 - Getting issue details... STATUS |
Summary | RabbitMQ ran out of memory, affecting TIS services (particularly Reval) |
Impact | Any data updated on GMC day before has not/will not reflect on our system |
Non-technical Description
The system we use to send messages between our services ran out of memory. This led to a delay in updates across a number of TIS services, in particular Reval.
Trigger
RabbitMQ ran out of memory
Detection
Inspection of lastUpdatedDate column by developer
Resolution
Restarted Neo4j Consumers and managed syncing of queues until resources were cleared
Timeline
21:23 Recommendations service reports failure to connect to rabbitmq
-
Restarted TCS - recreated queues -
-
5 Whys (or other analysis of Root Cause)
Action Items
Action Items | Owner | Comments |
|---|---|---|
Complete this ticket | ||
Add Comment