It may only be down the weekly prod->stage synchronisation but the cdc logs (STAGE) showed the following error:
... 2020-06-16 11:14:39 ERROR Error on bin log position Position[BinlogPosition[mysql-bin.003382:68853804], lastHeartbeat=1592034510892] 2020-06-16 11:14:39 INFO Binlog disconnected. 2020-06-16 11:14:44 WARN Timed out waiting for heartbeat 1592306079319 2020-06-16 11:14:44 INFO Stopping 4 tasks ...
Non-production resolution:
To find (and skip over) a problem statement:
Decode the logfile
sudo mysqlbinlog --base64-output=decode-rows /var/log/mysql/{mysql-bin_file_from_error} > /tmp/someTempFile.sql
(binlogs are only readable by mysql user & group).Search for the position by the number in the error. From that point in the file you can find the next position by searching forward for
end_log_pos
.Update the
binlog_position
field inmaxwell
.positions
to move where the cdc will try to resume. I initially tried the first but this was also a problem. It may be that anything in the same transaction will also be an issue.
Misc thoughts:
It’s not appropriate for production without taking lots of mitigating actions (i.e. what to do about the transactions that haven’t been forwarded).
Add Comment