...
14:45 Sync process accidentally triggered
14:45 Impact on production environment noticed by developer
16:41 Failure in last stage of process noticed
16:50 Cause of failure identified
16:57 Failure rectified, process resumed
20:33 Process completed, system restored
23:18 Recommendations Bug Discovered
07:15 Users notified on Teams of Recommendation issues
08:00am Fix to Recommendation issues deployed successfully, issue resolved ~ 08:00 Users notified of resolution
Root Cause(s)
Human Error
“preprod” and “prod” are not visually distinct enough?
...