Versions Compared

Old Version 5

changes.mady.by.user Joseph (Pepe) Kelly

Saved on

compared with

New Version Current

changes.mady.by.user catherine.odukale

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Documenting

Date

Authors

Joseph (Pepe) Kelly

Status

Done

Summary

Jira Legacy
serverSystem JIRA
serverId4c843cd5-e5a9-329d-ae88-66091fcfe3c7
keyTIS21-5681
We received loads of sentry errors which showed that recommendation changes over the weekend wouldn’t have been reflected in the reval search.

Impact

No user impact determined.

...

Action Items

Action Items

Owner

Comments

Investigating why the position in capped collection was deleted (see errors on weekend):

  • When were the last changes to recommendation records before Sat 3rd Feb?

    The previous change was on Friday evening

  • How is the Document DB change stream configured? Is it a 7 day window?
    The change stream retention period is the default 3 hours but is only removed once the log is full with an AWS configured 51,200MB. It is possible that the data represents the first time this threshold was crossed without additional writes since the cluster was last modified. Increasing the retention period will impact storage and performance.

Joseph (Pepe) Kelly

Done

Increase the change stream retention period.

Jira Legacy
serverSystem JIRA
serverId4c843cd5-e5a9-329d-ae88-66091fcfe3c7
keyTIS21-5713

Make the resume token check work if there isn’t one…

  1. Upgrade and

    1. Use a time based check

    2. Change the Lambda to use Change Stream events directly

    3. look for how to directly integrate the change stream with SQS or other asynchronous messaging service

  2. Check & modify DocumentDB resume token to “upsert” a new value.

Not right now. We can review this if the database is upgraded as expectedwe expect to do soon.

...

Lessons Learned