Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Route all other traffic through Green
  • Change ESR ETL to allow a parameter to be passed which provides a specific date for the notification load so that they can be picked up retrospectively if necessary (rather than just using the current system date)

  • Kick off the notification load routine on Blue

  • Wait for process to finish before re-enabling Blue for other traffic

Detection / Timeline

  • 0200 0600 7th May: ESR ETL started.
  • 0600 7th May: ESR ETL found to have failed (at some point)
  • 0900 7th May: ESR ETL (notification-daily-load) Started again
  • 1039 7th May: ESR ETL found to have failed (at some point)
  • 1300 7th May: ESR ETL (notification-daily-load) Started again
  • 1500 7th May: All jobs jobs disabled from 8pm that evening to allow notification-daily-load opportunity to run
  • 2000 7th May: ESR ETL (notification-daily-load) Started again (to give largest window before other processes start)
  • 0400 8th May: ESR ETL failed (TCS GC overhead)
  • 0900 8th May: Hard memory limit removed from TCS
  • 1400 9th May: ESR-ETL code change to allow parameter to be passed to specify the date of the notifications (instead of the default of LocalDate.now())
  • 1500 9th May: ESR ETL started
  • 1730 10th May: ESR ETL completed

...