GHA unavailable
Not much we could do (apart from migrate to Jenkins :trollface:)
Recommendations starting
Why? Not much of an idea
Tried comparing already:
Recommendation vs. Integration (no diff obvious)
Checks to do:
Healthcheck grace period, both are 0 grace period
Service startup time
ECS Service Events
Logs; ECS tasks appeared to start then
It is reproducable/consistent across envs
ES Rebuild failed
What's the value in understanding the cause/root cause:
General understanding of ES & SpringBoot clients
We didn't get this same exact failure mode but we did have issues in stage too
Make it more reproducable:
Copying data from Prod to Stage
Master Index Logs
ES HTTP500 exception:
Modifying a mapping requires reindex to a new index
Hypothesis: The mapping was being added by another task so the task with the job wasn't able to put the mapping
The recommendation mapping wasn't created as needed
It looked like the mapping didn't come through
Can try locally to improve how we maintain an index mapping
A queue to Connections not being processed
Need to address this; code-rot (neglected) unused service
It's another reason we'd need to rebuild the index
It might be worth looking at if/how this gets built up without requiring recommendations to be rebuilt
Sync start
Message seen on a queue, not being consumed.
Look into what might have happened
Slow copying to Recommendations
Would have taken too long to page through all the masterdoc idx
Recommendation index is the masterdoc index on prod
Sharding doesn't make a difference
Connections relies on transformations/filtering
We will either not need the recommendation index or should apply the business logic for Recommendations
Recommendations;
can carry on for now, defer
make preprod & prod look the same
Look at alias?
updates (messages, HTTP PUT/POST/etc.) during deployment need to be handled considering blue / green deployment setup
Duplicated data on recommendation list (which we don't have searching Mongo)
Issue would have existed any
If e.g. name are different, we should use: "probably"
Add Comment