...

The push of TIS data to NDW was not run on Apr 11.

...

Trigger

• Nightly data push from TIS to NDW

...

Detection

• Email from NDW team in the morning of 11/04/2023

• In #monitoring-ndw channel on Slack, no notifications found for ndw-etl-prod task:

  Image RemovedImage Added

...

Resolution

• Contacted GMC support and technical contact at the GMC

• Resolved by GMC

• Manually ran the ndw-etl-prod on ECS.

• The start and finish of the task was notified in Slack.

...

Timeline

BST unless otherwise stated

...

• We expect the ndw-etl-prod job to be triggered by the AWS eventbridge rule every day at 2am UTC.

• From the metrics, the everntbridge rule was triggered on Apr 11, but there’re no logs found on Cloudwatch. And from the ECSStoppedTasksEvent, we can also find the ndw-etl-prod task was not started.

• The CloudTrail event history shows the reason of failure: "Capacity is unavailable at this time. Please try again later or in a different availability zone"

...

Action Items

...