2022-12-13: Hack day – Monitoring and Alerting

We decided to hold hack days on a more regular 6-week cycle. For this hack day we decided to hold a remote event, looking at a tech improvement challenge, one Discovery, two-team approach, and to follow the GDS Agile Delivery framework which worked well last Hack day.

General benefits of a hack-day:

practice collaboration outside the normal group of people you collaborate with
test your specialist skills, or, conversely, test your non-specialist skills
challenge a group of people that may not always work together to plan a time-bound task
maintain focus on one task for a whole day

Table of contents

Topic selection process

We came to organising this hack day quite late. As a result we decided to look at our backlog and select something that was an investigation-type ticket that would involve the whole team.

Andy Dingley , John Simmons (Deactivated) and Cai Willis assisted Andy Nash (Unlicensed) in organising this one.

We decided on investigating the longer term approach to Monitoring and alerting because it needs doing and has both an obvious technical element, and a more business-focused element of the wording of the alerts and onward comms.

Why did we chose Monitoring and alerting as a Tech improvement Hack day?

We currently have monitoring and alerting offered by a range of tools / tool combinations that have evolved over time.

We have recently been looking at streamlining our costs where possible; make it easier for everyone in the team to help out with anything it’s possible for them to help out with, and reduced our security risk overall.

We also identified that current alerting is not pitched at non-techies. And even within those that are techie, the alerts are sometimes impenetrable. So there is a need to improve the alerting in terms of impact on user, and indication of how to resolve the problem especially.

Team members

The TIS Team broke into two teams:

One team looked specifically at an AWS Native approach to monitoring and logging

The other team looked specifically as an AWS Manager approach.

Subject matter experts

During discovery, we identified 4 main user groups:

TIS Team ourselves (Problem Manager especially);
Users of products affected by any issues (in all cases, it was felt that onward comms should be handled by Local Office administrators - as they have existing relationships with the end users);
TIS leads (for the Products they work on); and
Data Leads (where data is affected - notably stale data).

AWS Native (Cloudwatch, EventBridge, Lambdas):

AdeO
Andy D
Ash
Cai
Doris
Ed
James
Kav
Rob
Steve
Ade B

AWS Managed (Prometheus, Grafana, etc):

Ade A
Anita
Catherine
Jay
John S
JohnO
Naz
Pepe
Reuben
Saedhia
Stan

Structure of the day

1 hour: discovery (who are the users? what are their problems (might be different problems for different users)? what are the risks and assumptions we need to shine a light on before committing to potentially wasted development effort?)
1 hour: alpha experiments (testing assumptions and risks and narrowing down a range of possible solutions, to the preferred one to develop in beta)
1 hour: lunch
3 hours (incl. breaks): beta development (incrementing a solution - developing a bit, reconvening, developing a bit more, reconvening. Avoiding long periods of silo/solo working)
1/2 hour: presentations back to each other

Hack day Retro
Hack day Retros are best carried out at the end of a Hack day, while everything is still fresh in everyone’s minds. Because of time and teleconferencing constraints, this was unfortunately not possible. Many participants were sick / on leave in the following few days.