04/07/2022 Post deployment review / release washup

Open

GHA unavailable

Not much we could do (apart from migrate to Jenkins :trollface:)

Recommendations starting

Why? Not much of an idea

• Tried comparing already:

  • Recommendation vs. Integration (no diff obvious)

• Checks to do:

  • Healthcheck grace period, both are 0 grace period

  • Service startup time

  • ECS Service Events

  • Logs; ECS tasks appeared to start then

• It is reproducable/consistent across envs

ES Rebuild failed

What's the value in understanding the cause/root cause:

• General understanding of ES & SpringBoot clients

• We didn't get this same exact failure mode but we did have issues in stage too

• Make it more reproducable:

  • Copying data from Prod to Stage

Master Index Logs

ES HTTP500 exception:

• Modifying a mapping requires reindex to a new index

• Hypothesis: The mapping was being added by another task so the task with the job wasn't able to put the mapping

The recommendation mapping wasn't created as needed

It looked like the mapping didn't come through

Can try locally to improve how we maintain an index mapping

A queue to Connections not being processed

Need to address this; code-rot (neglected) unused service
It's another reason we'd need to rebuild the index

It might be worth looking at if/how this gets built up without requiring recommendations to be rebuilt

Sync start

Message seen on a queue, not being consumed.

• Look into what might have happened

Slow copying to Recommendations

Would have taken too long to page through all the masterdoc idx
Recommendation index is the masterdoc index on prod
Sharding doesn't make a difference
Connections relies on transformations/filtering

We will either not need the recommendation index or should apply the business logic for Recommendations

Recommendations;

• can carry on for now, defer

• make preprod & prod look the same

• Look at alias?

  • updates (messages, HTTP PUT/POST/etc.) during deployment need to be handled considering blue / green deployment setup

Duplicated data on recommendation list (which we don't have searching Mongo)

Issue would have existed any

If e.g. name are different, we should use: "probably"