Date |
|
Authors | |
Status | Done |
Summary | TSS was sending verification emails that failed the DMARC (spam) checks, and hence were not delivered |
Impact | There was a 6-day period during which users were unable to sign-up (they would not be able to verify their email address) |
Non-technical Description
When users sign up for TSS they are prompted to confirm their email address via a code sent in a verification email. This email is sent from the no-reply@tis-selfservice.nhs.uk mailbox.
When a mail server receives an email, it checks whether it is likely to be spam, and if so, it rejects it or places it on hold. Domain-based Message Authentication and Conformance (DMARC) is a standard email authentication method. DMARC helps mail administrators prevent hackers and other attackers from spoofing their organization and domain. Spoofing is a type of attack in which the From address of an email message is forged. A spoofed message appears to be from the impersonated organization or domain.
The emails sent from TSS did not include a DomainKeys Identified Mail (DKIM) header, as they were using a verified email address but the domain itself was not verified. As such, the emails were failing DMARC.
.png?version=1&modificationDate=1669196648771&cacheVersion=1&api=v2)
The issue was triggered by an apparently insignificant change in the AWS SES region for sending emails, from Ireland (eu-west-1) to London (eu-west-2).
Trigger
Users signed up and never received their verification email.
Detection
Users sent a number of support requests
Resolution
Manual addition of the correct domain to AWS SES.
Timeline
GMT unless otherwise stated
~10:40 - Manual change to SES region from eu-west-1 to eu-west-2, breaking verification emails
11:20 - Support request posted on #self-services-support slack channel
11:39 - DKIM set-up for tis-selfservice.nhs.uk, restoring functionality
Root Cause(s)
Emails from no-reply@tis-selfservice.nhs.uk were failing DMARC and not being delivered to users
The AWS SES email address was verified in both eu-west-1 and eu-west-2, but the domain (tis-selfservice.nhs.uk) was misspelt in eu-west-2 ('tis-selservice.nhs.uk’)
When the AWS region for sending Cognito account verification emails was changed from eu-west-1 to eu-west-2, the missing (i.e. unverified) domain in eu-west-2 caused the emails to lack a DKIM header
Without a DKIM header, the emails would fail DMARC and be rejected by the receiving mailservers.
Action Items
Action Items | Owner |
|---|---|
Lessons Learned
Where manual changes to AWS configurations have been made, do not assume that there is consistency between regions (or anywhere else).
Add Comment