If you see this, panic (a little and then do something):
...
Decode the logfile
sudo mysqlbinlog --base64-output=decode -vv /var/log/mysql/{mysql-bin_file_from_error} > /tmp/someTempFile.sql
.Search for the position by the number in the error. From that point in the file you can find the next position by searching forward for
end_log_pos
. You may need to search past a few; finding a suitable point such as the end of the transaction.Update the
binlog_position
field inmaxwell
.positions
MySQL table to move where the CDC will try to resumestart point for CDC. I initially tried the first but this was also a problem. It may be that anything in the same transaction will also be an issue.
Misc thoughts:
It’s not appropriate for to skip transactions in production without taking lots of mitigating actions (i.e. what to do about the transactions that haven’t been forwarded).
The recurring issue on STAGE may be caused by the weekly prod->stage synchronisation but the cdc logs (STAGE) showed the following error.