Root Cause

Hicom disabled access to the Consolidated DR

Trigger

Modifications to their DR configurations.

Resolution

Hicom reenabled our account and we reran the ETLs to catch up.

Detection

Our ingest pipeline failed.

Action Items

None

Timeline

8:15 Graham O'Regan (Unlicensed) sent message on Slack to notify that the ingest pipeline had failed. Checking the logs, it showed that our account didn't have access to the database on the DR server.

8:28 Graham O'Regan (Unlicensed) emailed Hicom to let them know that the account wasn't working

9:00 Hicom replied to acknowledge that there had been a problem

9:41 Hicom emailed to say that the issue had been resolved.

9:42 Russell Steel (Unlicensed) confirmed that access had been restored.

Supporting Information

Slack Transcript

graham [8:15 AM]
@srochani the pipeline failed, error is down to credentials, looking at them now

[8:15]
nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Cannot open database \"Intrepid.DataRepository.HEE\" requested by the login

graham [8:20 AM]
@russ the TRANSFORM account details aren’t working, can you login or change them?

[8:22]
worryingly, the reval etl has also failed but didn’t report an error

russ [8:22 AM]
I'm logged in, but there is an issue with Intrepid.DataRepository.HEE as I can't browse it, and the table names I'm getting are vwHEE_ prefix

[8:23]
There are only HELASE object in the DR this morning, all with vwHEE_;

TABLE_SCHEMA TABLE_NAME
HELASE vwCourseAttendance
HELASE vwDates
HELASE vwDataQuality
HELASE vwTags
HELASE vwHEE_AccreditationEvidence
HELASE vwHEE_AccreditationPathwayAssignment
HELASE vwHEE_Assessment
HELASE vwHEE_CourseAttendanceContact
HELASE vwHEE_CourseAttendanceStaff
HELASE vwHEE_Curriculum
HELASE vwHEE_CurriculumGrade
HELASE vwHEE_CurriculumMembership
HELASE vwHEE_EmployerReport
HELASE vwHEE_Entitlement
HELASE vwHEE_FormRPartA
HELASE vwHEE_FormRPartB
HELASE vwHEE_FormRPartBEvent
HELASE vwHEE_GPVisit
HELASE vwHEE_GPVisitHost
HELASE vwHEE_GPVisitRecommendation
HELASE vwHEE_GPVisitRecommendationAction
HELASE vwHEE_GPVisitSite
HELASE vwHEE_GPVisitVisitor
HELASE vwHEE_Grade
HELASE vwHEE_Leave
HELASE vwHEE_LeaveExpense
HELASE vwHEE_NTN
HELASE vwHEE_Person
HELASE vwHEE_PersonAssociatedSite
HELASE vwHEE_PersonAssociatedSpecialty
HELASE vwHEE_PersonAssociatedTrust
HELASE vwHEE_PersonExam
HELASE vwHEE_PersonQualification
HELASE vwHEE_Placement
HELASE vwHEE_PlacementSupervisor
HELASE vwHEE_Post
HELASE vwHEE_PostFunding
HELASE vwHEE_PostFundingComponent
HELASE vwHEE_PostOtherGrade
HELASE vwHEE_PostOtherSite
HELASE vwHEE_PostOtherSpecialty
HELASE vwHEE_PostSubSpecialty
HELASE vwHEE_Programme
HELASE vwHEE_ProgrammeCurriculum
HELASE vwHEE_ProgrammeMembership
HELASE vwHEE_ProgrammeNTN
HELASE vwHEE_ProgrammePost
HELASE vwHEE_ProgrammeRotationMembership
HELASE vwHEE_ProgrammeRotationPost
HELASE vwHEE_ProgrammeSchool
HELASE vwHEE_ProgrammeSupervisor
HELASE vwHEE_QuestionnaireAnswer
HELASE vwHEE_QuestionnaireRelease
HELASE vwHEE_RevalidationEpisode
HELASE vwHEE_SecurityDataProfileMembership
HELASE vwHEE_SecurityRoleMembership
HELASE vwHEE_Site
HELASE vwHEE_SiteType
HELASE vwHEE_Specialty
HELASE vwHEE_StaffPostOtherSpecialty
HELASE vwHEE_TeachingSession
HELASE vwHEE_Trust

graham [8:23 AM]
k, i can’t get in at all

[8:25]
if you disconnect, can you get back on?

[8:26]
it seems to be permissions, i can get a connection to the server but not to the database

russ [8:26 AM]
I can, the issue is only this schema can be accessed by the TRANSFORM user;

TABLE_CATALOG table_schema (No column name)
Intrepid.DataRepository HELASE 62

Looks like Hicom has some major DR issues last night.

graham [8:28 AM]
k, sent them an email

[8:28]
going to head over to hee now, i’m at transform at the moment

graham [9:04 AM]
@russ have you seen that email from yvonne?

[9:04]
can you send her the query that you ran and the output?

[9:06]
they have enabled the system account but i still can’t see the views in .HEE