...

The database got very busy and wasn’t available to process changes from TIS. A notification for a new post was not sent out to trusts. We also did not update other notifications. Other “updates” failed but these did not contain any changes relevant to Notifications or Applicants to be sent to ESR. We have replayed a message manually to generate the necessary notification for the creation of post NWN/REM21/006/HT/001.

...

Trigger

Under reviewCluster became unhealthy. Contributing factors under review.

...

Detection

• Application Error Alerting to slack

...

• Slack Message triggered by exceptions in all ESR services.

• The ESR services were unable to connect to the database.

• Hearbeat Heartbeat checks between replicaSet nodes failed.

• The VM was non-responsive.

• The following MongoDB (mongo1) log samples give an indication of the instability of the cluster at that time:
  {"log":"2021-08-17T14:52:50.374+0000 I REPL_HB [replexec-67] Error in heartbeat (requestId: 3436593) to mongo3:27013, response status: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit\n","stream":"stdout","time":"2021-08-17T14:52:50.582068394Z"}
{"log":"2021-08-17T14:52:50.598+0000 I REPL [replexec-67] Member mongo3:27013 is now in state RS_DOWN\n","stream":"stdout","time":"2021-08-17T14:52:50.598267401Z"}
  

  {"log":"2021-08-17T14:53:14.795+0000 I REPL_HB [replexec-67] Error in heartbeat (requestId: 3436596) to mongo3:27013, response status: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit\n","stream":"stdout","time":"2021-08-17T14:53:16.436295842Z"}
  

  {"log":"2021-08-17T14:53:17.723+0000 I REPL_HB [replexec-68] Error in heartbeat (requestId: 3436597) to mongo2:27012, response status: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit\n","stream":"stdout","time":"2021-08-17T14:53:19.63301921Z"}
  

  {"log":"2021-08-17T14:53:19.615+0000 I REPL [replexec-67] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)\n","stream":"stdout","time":"2021-08-17T14:53:19.681137738Z"}
  

  {"log":"2021-08-17T14:54:36.354+0000 I REPL [replexec-68] VoteRequester(term 101 dry run) received a no vote from mongo2:27012 with reason "candidate's term (101) is lower than mine (102)"; response message: { term: 102, voteGranted: false, reason: "candidate's term (101) is lower than mine (102)", ok: 1.0, operationTime: Timestamp(1629211887, 1), $clusterTime: { clusterTime: Timestamp(1629211887, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } }\n","stream":"stdout","time":"2021-08-17T14:54:37.533307206Z"}
{"log":"2021-08-17T14:54:36.354+0000 I REPL [replexec-68] not running for primary, we have been superseded already during dry run. original term: 101, current term: 102\n","stream":"stdout","time":"2021-08-17T14:54:37.533313897Z"}
  

  {"log":"2021-08-17T14:55:02.747+0000 I CONNPOOL [RS] Ending connection to host mongo2:27012 due to bad connection status; 1 connections to that host remain open\n","stream":"stdout","time":"2021-08-17T14:55:09.266545288Z"}

  {"log":"2021-08-17T14:56:03.032+0000 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to mongo2:27012: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit\n","stream":"stdout","time":"2021-08-17T14:56:06.838424722Z"}
{"log":"2021-08-17T14:56:03.100+0000 I ASIO [RS] Failed to connect to mongo2:27012 - HostUnreachable: Error connecting to mongo2:27012 :: caused by :: Could not find address for mongo2:27012: SocketException: Host not found (non-authoritative), try again later\n","stream":"stdout","time":"2021-08-17T14:56:06.838430607Z"}

...

Action Items

...