2021-03-01 Legacy Reval: unable to submit revalidation - User Agent/Cloudflare GMC Issue

Date

Mar 1, 2021

Authors

@Philip Wilsdon (Unlicensed)

Status

Completed

Summary

Open

• GMC Sync failed overnight due to the GMC adding Cloudflare into their infrastructure setup on the Feb 28, 2021

• As part of this implementation, they blocked our requests based on rules they had setup - blocking based on the user agent.

• Rule disabled until new reval module is live

Impact

• Users unable to revalidate and manage connections from Monday morning Mar 1, 2021 to the morning of Mar 4, 2021 (however, they could still use GMC connect)

Mar 1, 2021 08:08

Mar 1, 2021 08:30

Created ticket and incident page

2021-03-01 Legacy Reval: unable to submit revalidation - User Agent/Cloudflare GMC Issue

Mar 1, 2021 09:15

Rerun failed with the same error… Forbidden (HTTP 403)

Mar 1, 2021 09:33

Pinged users to let them know we have contacted GMC to check the issue

Mar 1, 2021 09:34

Emailed GMC

Mar 1, 2021 10:43

Replied to GMC to confirm its the Production/Live environment and not our new Reval module

Mar 1, 2021 11:47

GMC have made some changes over the weekend, they are looking into it

Mar 1, 2021 16:10

Email to chase GMC

Mar 1, 2021 16:31

Mar 1, 2021 16:53

Confirming with the GMC what IPs we are hitting

Mar 1, 2021 17:50

Email back from GMC

Mar 2, 2021 10:55

We found that with the new reval module, no issues, we are able to get data

However, the issue is with legacy/existing.

Need to check if the IPs/Servers that legacy reval runs on are still whitelisted

Mar 2, 2021 11:01

Mar 2, 2021 11:46

Checking if authentication error

Mar 2, 2021 16:10

We confused the GMC

Mar 3, 2021 11:17

Call to investigate further - more fault analysis done,

• review errors and look at gmc-sync repo to see where the problem might be

Some Findings

• 403 in place in Prod as it is an authentication issue in our side.

• We are able to curl the GMC end points which shows that the credentials are correct but the Java code is not able to get them due to some restrictions (?) in our Prod2.

• We know 99 error in stage as the IPs are not white listed by GMC for our stage env3.

• We want to trace logging in Java code so that we can see the credentials are fetched correctly

Mar 3, 2021 14:00

• Call to review draft PR created to check authentication issues - what are GMC sending us - what response

• 1st of the month and timings - nothing has changed for 6 months so why a change now - version updates?

Mar 3, 2021 14:11

• tried to update java base - didn't do anything

Mar 3, 2021 14:41

• Updated PR waiting for review

Mar 3, 2021 15:17

• PR reviewed

• Ran gmc-sync-prod and it failed as expected

• Investigating now with the extra logging

Mar 3, 2021 15:30

• its getting the correct username and password but still failing

• Before it gets to the gmc return code it throws exception

• The app is failing in the SOAP API call

Mar 3, 2021 16:33

Email to GMC

Mar 3, 2021 16:38

So just to sum up our thought process:

• We can CURL Prod (Authentication is fine)

• There have been no significant recent changes to the codebase

• The Java app appears to be building the request correctly

• The Java app appears to be using the correct credentials

• We are still receiving 403 from GMC endpoint

Conclusion: most likely cause is a permissions error (Authorization issue) internally on GMC side

Mar 3, 2021 16:42

Mar 3, 2021 16:50

Updated users on teams with our conclusion

Mar 3, 2021 17:06

Clarification of IPs

Mar 3, 2021 05:47

Mar 3, 2021 22:02

Email from the GMC - they have been able to re-create the 403 error

Mar 3, 2021 22:48 to 23:19

• Cloudflare (now being used by the GMC) - Cloudflare Browser Integrity Check seems to block Java 8 user agents by default.

• GMC-Sync with Java 9 and Java 11 didn't work (no surprise there)

• Changed back to Java 8 and it's working now

Mar 4, 2021 00:05

• 00:05 cron run of the GMC-Sync ran successfully, so after the intrepid ETL runs at 01:00 Reval should be working again

Mar 4, 2021 08:02

• Slim-buster image (presumably with a later update of Java 8) - we don’t think the GMC modified any the rules in Cloudflare.

• Using a recent maintained docker base image (as long as it works with the GMC config) - that would be using the slim-buster image.

Mar 4, 2021 08:08

• User reports fixed

Mar 4, 2021 09:59

Trying to get some clarification from the GMC relating to the user agent

Mar 4, 2021 11:26

GMC disabled the security feature relating to the user agent so thats why it worked

We need to update the user agent

Mar 4, 2021 13:05

Mar 8, 2021 10:06

Chasing GMC - No reply regarding User agent / is there filtering back on?

Mar 8, 2021 14:18

Reply from GMC and some thoughts

• What the LTS status of those two Java 8 versions

• Assuming they're in LTS, are the GMC willing and able to tweak their Cloudflare config to allow in future?

• New reval is Java 11, So just keep the filtering disabled until we switch over is an option?

Mar 8, 2021 15:37

• Java 8 has LTS until 2030 so GMC are blocking/filtering something that is supported

• We don’t want to spend effort on the existing module

• New module in Java 11 won’t have an issue so requested GMC just leave the filter then enabled for now turned off

Mar 8, 2021 16:33

• GMC ok to disable the Cloudflare filter until new reval developed