...

• The SMS costs exceeded the configured limits

  • The limit was not set appropriately

    • Data based limits not yet set, still using guesswork

  • The alert for reaching 90% of the limit was not seen

    • Noisy #monitoring channel due to tis-log-size alerts?

      • Placement sync log spam

      • Prod → Stage DB sync log spam

      • Alarm not configured appropriately - e.g. using more datapoints would help avoid the yoyoing of the alert status which causes slack spam

  • The daily cloudwatch CloudWatch alarm reminders were not checked/actioned

    • ???Only 3 alarms are visible in Slack notice (and an ‘and X more…’ message), and these tend to be the usual tis-trainee-sync DLQ errors which are often ignored

    • The SMS alarm would only be seen by visiting the CloudWatch page directly

...

Action Items

...