Versions Compared

Old Version 5

changes.mady.by.user Yafang Deng

Saved on

compared with

New Version 6

changes.mady.by.user James Harris

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • 04:03 - Since then, no more notifications had been received for the overnight ndw etl jobs.

  • 12:07 - James redirected the email that push of TIS data into NDW failed

  • 12:40 - Yafang found no logs found on ndw-etl-prod in the midnight

  • 13:02 - Andy D found the event bridge triggered the task and the service considered itself healthy, but no record for the task on prod

  • 14:26 - Yafang & Jay triggered the task ndw-etl-prod manually

  • 14:56 - The task has been run and exited on etl-prod successfully

  • 15:11 - James let Guy know and NDW started their jobs.

  • Apr 12, 2023 13:45 - Guy confirmed there were no subsequent issues.

...

Root Cause(s)

  • We expect the ndw-etl-prod job to be triggered by the AWS eventbridge rule every day at 2am UTC.

  • From the metrics, the everntbridge rule was triggered on Apr 11, but there’re no logs found on Cloudwatch. And from the ECSStoppedTasksEvent, we can also find the ndw-etl-prod task was not started.

  • The CloudTrail event history shows the reason of failure: "Capacity is unavailable at this time. Please try again later or in a different availability zone"

...