Forgot to add that after checking db02/03 nodes, the alter table commands have not been executed there
Hi Alexey,
I can't see a cnf in MDEV-4136, can you?
We had one in MDEV-4179, with all wsrep* options disabled, did you mean this one?
Right, this one. Basically, this is the my.cnf with all wsrep_* options un-disabled 
Also, we've tried to reproduce this problem in our test environment w/o success using same configs (the only difference is memory size/threads number because test machines are smaller than production).
We also found these issues that look related:
http://www.perconaforum.com/index.php?t=msg&goto=9378&
https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1112363
https://groups.google.com/forum/?fromgroups=#!topic/percona-discussion/bOEclT7K7ow
https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1049105
Okay, thanks, I see – many questions, no answers... Lets see if we can figure it out.
Does it happen on any table at all?
Does it only happen on ALTER TABLE? what about other DDL – CREATE TABLE, DROP TABLE, ALTER DATABASE,...?
What happens if you execute a similar statement on db02 or db03, does it work there? Does it get replicated to db01?
When you get such an error in your node log, there is also a GRA*log file created in the datadir of this node. The files are tiny and basically just contain the offending event. Could you please attach one of those, for the certainty – the latest one, and quote the corresponding error message from the log?
Thanks.
1) Yes, we tried a few tables and all have the same issue. Of course, all tables were InnoDB
2) We didn't try other DDLs unfortunately
3) Yes, the same issue regardless of the origin of the DDL. Exactly the same log entries.
4) Unfortunately, we had to rollback to the plain Master-Slave setup last night thus I don't have the error log anymore. And as I said, we can't repro it on our test environment
Sorry, not a lot of data unfortunately. I was planning to build the latest 5.5.29, test it in dev environment and may be try another production rollout next week if everything goes well. I'll definitely keep an eye on the GRA*log files this time.
We found it. The issue is 100% reproducible with
binlog_checksum=1
master_verify_checksum=1
slave_sql_verify_checksum=1
And things work as expected if
#binlog_checksum=1
#master_verify_checksum=1
#slave_sql_verify_checksum=1
clarification: binlog_checksum=1 is enough
Thank you, Aleksey.
The problem is still reproducible on current maria-5.5-galera (revno 3386).
Problem happens because the chosen checksum algorithm is not communicated to receiving nodes, currently there is no method for it.
Note that, Galera replication has checksums already, these binlog checksums should not be used
>> these binlog checksums should not be used
Then the server should disable/ignore them automatically (with a proper warning in the error log). We cannot expect every user to know all subtle limitations and fight with the issues caused by them.
A temporary fix has been pushed, which sends a format description event before query event, the FD carries current binlog checksum setting. This fix enables checksums only for DDL statements, and binlog_checksum option should still not be used.
Fix pushed in: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3389
I disagree that checksums for binlog should not be used. In a mixed setup with Galera cluster streaming data with normal replication (e.g. for delayed replication) these checksums in binlog are useful.
Aleksey, that's a valid use case, indeed. And binlog checksumming should not hurt with Galera replication in general. It is just that this issue was acknowledged so close to the release deadline, that there was no time for a proper fix with full binlog checksum support. The plan is to continue with this development for future releases.
Hi Seppo. Thanks for your reply. It is perfectly fine to have temporarily hack for the release I was just concerned that the issue was marked as "fixed" in the same time so it was not clear if there are plans to implement the "right" fix.