Versions Compared

Old Version 5

changes.mady.by.user Reuben Roberts

Saved on

compared with

New Version 6

changes.mady.by.user Reuben Roberts

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • 14:03 - Mongo logs on prod-mongodb-replicaset (10.170.0.151) report mongo replica failure, or connectivity failure between them.

  • 14:10 - HttpServerErrorException$InternalServerError reported by ESR-NotificationGenerator in Slack sentry-esr channel.

  • 14:17 - Mongo recovery and re-establishment of sync between replicas.

  • 14:17 - Mongo3 instability / 'busy-ness' continues as per e.g.
    2022-05-05T15:37:26.017+0000 I COMMAND [conn209] command esrinbounddatawriter.person_mapping command: find { find: "person_mapping", filter: { assignmentId: 21695773, positionId: 5090293 }, limit: 2, $db: "esrinbounddatawriter", $clusterTime: { clusterTime: Timestamp(1651765045, 24), signature: { hash: BinData(0, 7CEAE9D1B5F05579EDB760935C42FFA0BCAAC9B7), keyId: 7050087017069150210 } }, lsid: { id: UUID("a314d8e2-52be-46ad-938a-91ff1b21ae
    ef") } } planSummary: COLLSCAN keysExamined:0 docsExamined:257493 cursorExhausted:1 numYields:2011 nreturned:1 reslen:411 locks:{ Global: { acquireCount: { r: 2012 } }, Database: { acquireCount: { r: 2012 } }, Collection: { acquireCount: { r: 2012 } } } storage:{} protocol:op_msg 115ms
    2022-05-05T15:37:26.394+0000 I TXN [conn236] transaction parameters:{ lsid: { id: UUID("0d220ebd-f2ce-4bd8-bcce-414c08a4716b"), uid: BinData(0, 3911A6AADDA0C6EB92636A3942D0CDDA85E6EB84D7936794A99B8EE3518382CD) }, txnNumber: 61, autocommit: false, readConcern: { level: "snapshot" } }, readTimestamp:Timestamp(1651765045, 25), keysExamined:0 docsExamined:318159 terminationCause:committed timeActiveMicros:120840 timeInactiveMicros:8250 nu
    mYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } storage:{} 129ms

  • 18:47 - Mongo3 stabilises? The logging goes back to looking ‘normal’.

  • 15:10 - 19:43 Failures reported by ESR-AuditService, ESR-NotificationGenerator, ESR-Reconciliation, ESR-AppRecordGenerator and ESR-InboundDataWriter in Slack sentry-esr channel.

...