2021-03-01 Legacy Reval: unable to submit revalidation

Date

01 Mar 2021

Authors

Joseph (Pepe) Kelly Adewale Adekoya

Status

Open

Summary

• GMC Sync failed overnight

• Reval users informed us they are not able to revalidate trainees

• Checked logs - access forbidden to the GMC

• Contacted GMC - they made some changes on Sunday 28 Feb 2021 (moving their services behind cloudflare) which caused the issue

Impact

• Reval data out of date

• Users unable to revalidate and manage connections

01 Mar 2021 08:08

01 Mar 2021 08:30

Created ticket and incident page

2021-03-01 Legacy Reval: unable to submit revalidation

01 Mar 2021 09:15

Rerun failed with the same error… Forbidden (HTTP 403)

01 Mar 2021 09:33

Pinged users to let them know we have contacted GMC to check the issue

01 Mar 2021 09:34 AM

Emailed GMC

01 Mar 2021 10:43

Replied to GMC to confirm its the production/LIVE Envrionment and not our new Reval module

01 Mar 2021 11:47

GMC have made some changes over the weekend, they are looking into it

01 Mar 2021 16:10

Email to chase GMC

01 Mar 2021 16:31

01 Mar 2021 - 16:53

Confirming with the GMC what IPs we are hitting

01 Mar 2021 17:50 PM

Email back from GMC

02 Mar 2021 10:55

We found that with the new reval module, no issues, we are able to get data

However, the issue is with legacy/existing.

Need to check if the IPs/Servers that legacy reval runs on are still whitelisted

02 Mar 2021 11:01 AM

02 Mar 2021 11:46

Cheking if authentication error

02 Mar 2021 16:10

We confused the GMC

03 Mar 2021 - 11:17

Call to investigate further - more fault analysis done,

• review errors and look at gmc-sync repo to see where the problem might be

Some Findings

• 403 in place in Prod as it is an authentication issue in our side.

• We are able to curl the GMC end points which shows that the credentials are correct but the Java code is not able to get them due to some restrictions (?) in our Prod2.

• We know 99 error in stage as the IPs are not white listed by GMC for our stage env3.

• We want to trace logging in Java code so that we can see the credentials are fetched correctly
  
  

03 Mar 2021 - 14:00

• Call to review draft PR created to check authentication issues - what are GMC sending us - what response

• 1st of the month and timings - nothing has changed for 6 months so why a change now - version updates?

03 Mar 2021 -14:11

• tried to update java base - didnt do anything

03 Mar 2021 -14:41

• Updated PR waiting for review

03 Mar 2021 - 15:17

• PR reviewed

• Ran gmc-sync-prod and it failed as expected

• Investigating now with the extra logging

03 Mar 2021 - 15:30

• its getting the correct username and password but still failing

• Before it gets to the gmc return code it throws exception

• The app is failing in the SOAP api call

03 Mar 2021 - 16:33

Email to GMC

03 Mar 2021 - 16:38

So just to sum up our thought process:

• We can CURL Prod (Authentication is fine)

• There have been no significant recent changes to the codebase

• The Java app appears to be building the request correctly

• The Java app appears to be using the correct credentials

• We are still receiving 403 from GMC endpoint

Conclusion: most likely cause is a permissions error (Authorization issue) internally on GMC side

03 Mar 2021 - 16:42

03 Mar 2021 - 16:50

Updated users on teams with our conclusion

03 Mar 2021 - 17:06

Clarification of IPs

03 Mar 2021 -05:47