...
- Check which files (if any) weren't sent to ESR (comparing export ETLs and FTP Sync job in #esr_operations channel in slack)
Moved missing file to the 'outbound' folder in Azure for today (22nd) ready to be processed at 18:00.
- Validate the file was processed by ESR the next day.
- Add application level retries to cope with temporary connectivity issues.
Detection / Timeline
- 2019-05-21 1700: Ansible message to #esr_operations channel reporting failure.
- 2019-05-21 1745 (approx.): Message from Ansible picked up and job run manually via jenkins.
- 2019-05-21 1800: FTP Sync runs and picks up all but 1 file.
- 2019-05-22 1130: Investigation started. Found that there was 1 file placed in Azure after the FTP sync ran and the last file uploaded was processed by ESR.
- 2019-05-22 1458: Copied file from yesterday's outbound folder (2019-05-21) to today's outbound folder (2019-05-21).
- 2019-05-23 0807: Checked that file was processed by ESR. Email in #esr_emails confirms file contents
...
- A transient connection problem caused the job to fail.
- The jobs are very close to each other. Monitoring wasnThe restarted job didn't picked up complete before the next step in the chain ran.
Where we got lucky
Supporting information
ESR log:
2019-05-21 16:00:39.199 ERROR 1 — main o.s.boot.SpringApplication : Application startup failed
java.lang.IllegalStateException: Failed to execute ApplicationRunner
...
at com.transformuk.hee.tis.esr.Application.main(Application.java:49) 1.0.27
...
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://tcs:8093/tcs/api/placements/filter": {
"timestamp" : "2019-05-21T16:00:39.185+0000",
"status" : 401,
"error" : "Unauthorized",
"message" : "Unauthorized",
"path" : "/tcs/api/placements/filter"
}; nested exception is java.io.IOException: {
"timestamp" : "2019-05-21T16:00:39.185+0000",
"status" : 401,
"error" : "Unauthorized",
"message" : "Unauthorized",
"path" : "/tcs/api/placements/filter"
}
TCS and Profile logs don't show
...