--binlog_row_metadata=full stores columns names and other metadata into the binary log.
However this is not used by replication threads, which can cause data loss on multi-master if one adds or drop fields with ALTER TABLE when binlog_format=row or binlog_format=statement is used.
ALTER TABLE will work if columns are added last to the table. However if one deletes columns or add a column in the middle of the table will cause all future DML to update wrong fields.
The fix is that if --binlog_row_metadata=full is used, we should not use field positions but instead use field names for insert and update when applying row events.
mariadb-binlog should also be update to show fieldnames instead of field positions if field names are available.
I will add a simple test case that one can use to verify the issue.
We should probably also add an option --slave_type_conversions=IGNORE_MISSING_COLUMNS to allow one to drop columns with ALTER TABLE
monty Enums are also broken. That is, if the slave modified its table's enums to have different ordering than the master, the enums are picked based on index, rather than value. Modifying your skr.test to instead test enums shows this:
--source include/have_innodb.inc
--source include/have_binlog_format_row.inc
--source include/master-slave.inc
CREATE TABLE t1 (a INT PRIMARY KEY, en ENUM('a','b','c')) ENGINE = innodb;
INSERT INTO t1 VALUES(100,'a'), (200,'b');
--sync_slave_with_master
--connection slave
alter table t1 modify column en enum ('a','d', 'b', 'c');
connection master;
INSERT INTO t1 VALUES(300, 'b');
--sync_slave_with_master
select * from t1;
connection master;
update t1 set en='b' where A=100;
--sync_slave_with_master
select * from t1;
connection master;
--source include/rpl_end.inc
The slave's result (where rows with a=100 and a=300 should both actually have value b, but show d because of the re-arrangement:
a en
100 d
200 b
300 d
Note too that binlog_row_metadata=FULL will also print the expectations for the ENUMs, so we can use this to reference the desired enum value.
#250313 14:49:07 server id 1 end_log_pos 944 CRC32 0x2167fc1b Table_map: `test`.`t1` mapped to number 33
Brandon Nesterenko
added a comment - - edited monty Enums are also broken. That is, if the slave modified its table's enums to have different ordering than the master, the enums are picked based on index, rather than value. Modifying your skr.test to instead test enums shows this:
--source include/have_innodb.inc
--source include/have_binlog_format_row.inc
--source include/master-slave.inc
CREATE TABLE t1 (a INT PRIMARY KEY, en ENUM('a','b','c')) ENGINE = innodb;
INSERT INTO t1 VALUES(100,'a'), (200,'b');
--sync_slave_with_master
--connection slave
alter table t1 modify column en enum ('a','d', 'b', 'c');
connection master;
INSERT INTO t1 VALUES(300, 'b');
--sync_slave_with_master
select * from t1;
connection master;
update t1 set en='b' where A=100;
--sync_slave_with_master
select * from t1;
connection master;
--source include/rpl_end.inc
The slave's result (where rows with a=100 and a=300 should both actually have value b , but show d because of the re-arrangement:
a en
100 d
200 b
300 d
Note too that binlog_row_metadata=FULL will also print the expectations for the ENUMs, so we can use this to reference the desired enum value.
#250313 14:49:07 server id 1 end_log_pos 944 CRC32 0x2167fc1b Table_map: `test`.`t1` mapped to number 33
# Columns(`a` INT NOT NULL,
# `en` ENUM('a','b','c') CHARSET latin1 COLLATE latin1_swedish_ci)
# Primary Key(a)
I discussed this with Monty, and here is the summary of our chat:
The use case is limited to multi-master, with a distributed dataset where different masters update the same table but work on distinct ranges of data. So if one node runs an ALTER TABLE which re-arranges the order of the columns to be inconsistent with its master, then when that node replicates insert/update row events that target its altered table from its master, there will indeed be data loss.
This should be fixed in 10.6, as we have many users/customers that run this topology running in 10.6, and the fix should be relatively easy with low risk. That is, to populate the fields of table->record[0] during unpacking using column names, rather than column indices. The patch applied in 10.6 should be relatively small; and any complex edge cases (e.g. enums) should just report as errors in 10.6, with fixes to be handled in newer versions. The patch in 10.6 can leverage ideas from the ONLINE ALTER work, and then in some 11.4+, the code can instead leverage the actual ONLINE ALTER code.
To the risk, this work will break functionality if a slave renames columns from its master. Currently, this use-case works as long as the column index is in the same location. With this patch, we will not be able to find the column on the slave-side table to write to. Monty says this is not a practical case though, and that users don't/wouldn't actually do this.
Brandon Nesterenko
added a comment - serg , elenst :
I discussed this with Monty, and here is the summary of our chat:
The use case is limited to multi-master, with a distributed dataset where different masters update the same table but work on distinct ranges of data. So if one node runs an ALTER TABLE which re-arranges the order of the columns to be inconsistent with its master, then when that node replicates insert/update row events that target its altered table from its master, there will indeed be data loss.
This should be fixed in 10.6, as we have many users/customers that run this topology running in 10.6, and the fix should be relatively easy with low risk. That is, to populate the fields of table->record [0] during unpacking using column names, rather than column indices. The patch applied in 10.6 should be relatively small; and any complex edge cases (e.g. enums) should just report as errors in 10.6, with fixes to be handled in newer versions. The patch in 10.6 can leverage ideas from the ONLINE ALTER work, and then in some 11.4+, the code can instead leverage the actual ONLINE ALTER code.
To the risk, this work will break functionality if a slave renames columns from its master. Currently, this use-case works as long as the column index is in the same location. With this patch, we will not be able to find the column on the slave-side table to write to. Monty says this is not a practical case though, and that users don't/wouldn't actually do this.
People
Brandon Nesterenko
Michael Widenius
Votes:
0Vote for this issue
Watchers:
6Start watching this issue
Dates
Created:
Updated:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":834.7999999523163,"ttfb":304.5,"pageVisibility":"visible","entityId":133316,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"cda5fea1-0e47-4ce5-8b2b-659541a8c24c","navigationType":0,"readyForUser":899.2999999523163,"redirectCount":0,"resourceLoadedEnd":1282.2999999523163,"resourceLoadedStart":310.09999990463257,"resourceTiming":[{"duration":20.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":310.09999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":310.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":330.59999990463257,"responseStart":0,"secureConnectionStart":0},{"duration":21.300000190734863,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":310.39999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":310.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":331.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":68.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":310.59999990463257,"connectEnd":310.59999990463257,"connectStart":310.59999990463257,"domainLookupEnd":310.59999990463257,"domainLookupStart":310.59999990463257,"fetchStart":310.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":336.2999999523163,"responseEnd":379.09999990463257,"responseStart":349.5,"secureConnectionStart":310.59999990463257},{"duration":84.20000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":310.7999999523163,"connectEnd":310.7999999523163,"connectStart":310.7999999523163,"domainLookupEnd":310.7999999523163,"domainLookupStart":310.7999999523163,"fetchStart":310.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":344.7999999523163,"responseEnd":395,"responseStart":377,"secureConnectionStart":310.7999999523163},{"duration":54.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":310.89999985694885,"connectEnd":351.7999999523163,"connectStart":351.7999999523163,"domainLookupEnd":351.7999999523163,"domainLookupStart":351.7999999523163,"fetchStart":310.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":352.2000000476837,"responseEnd":365.39999985694885,"responseStart":363.7999999523163,"secureConnectionStart":351.7999999523163},{"duration":57.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":311.09999990463257,"connectEnd":311.09999990463257,"connectStart":311.09999990463257,"domainLookupEnd":311.09999990463257,"domainLookupStart":311.09999990463257,"fetchStart":311.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":353.2999999523163,"responseEnd":368.5,"responseStart":365.89999985694885,"secureConnectionStart":311.09999990463257},{"duration":59.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":311.2999999523163,"connectEnd":311.2999999523163,"connectStart":311.2999999523163,"domainLookupEnd":311.2999999523163,"domainLookupStart":311.2999999523163,"fetchStart":311.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":356,"responseEnd":370.39999985694885,"responseStart":368.7999999523163,"secureConnectionStart":311.2999999523163},{"duration":42.60000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":311.39999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":311.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":354,"responseStart":0,"secureConnectionStart":0},{"duration":61,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":311.59999990463257,"connectEnd":311.59999990463257,"connectStart":311.59999990463257,"domainLookupEnd":311.59999990463257,"domainLookupStart":311.59999990463257,"fetchStart":311.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":358.5,"responseEnd":372.59999990463257,"responseStart":370.7000000476837,"secureConnectionStart":311.59999990463257},{"duration":45.09999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":311.7999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":311.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":356.89999985694885,"responseStart":0,"secureConnectionStart":0},{"duration":64.60000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":311.89999985694885,"connectEnd":311.89999985694885,"connectStart":311.89999985694885,"domainLookupEnd":311.89999985694885,"domainLookupStart":311.89999985694885,"fetchStart":311.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":360.2999999523163,"responseEnd":376.5,"responseStart":372.89999985694885,"secureConnectionStart":311.89999985694885},{"duration":363.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":322.2000000476837,"connectEnd":322.2000000476837,"connectStart":322.2000000476837,"domainLookupEnd":322.2000000476837,"domainLookupStart":322.2000000476837,"fetchStart":322.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":550.8999998569489,"responseEnd":685.5,"responseStart":679.8999998569489,"secureConnectionStart":322.2000000476837},{"duration":213.80000019073486,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":590.8999998569489,"connectEnd":590.8999998569489,"connectStart":590.8999998569489,"domainLookupEnd":590.8999998569489,"domainLookupStart":590.8999998569489,"fetchStart":590.8999998569489,"redirectEnd":0,"redirectStart":0,"requestStart":771.0999999046326,"responseEnd":804.7000000476837,"responseStart":803.7999999523163,"secureConnectionStart":590.8999998569489},{"duration":124.79999995231628,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":810.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":810.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":935.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":169.89999985694885,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":833.5,"connectEnd":833.5,"connectStart":833.5,"domainLookupEnd":833.5,"domainLookupStart":833.5,"fetchStart":833.5,"redirectEnd":0,"redirectStart":0,"requestStart":969.5999999046326,"responseEnd":1003.3999998569489,"responseStart":1002,"secureConnectionStart":833.5},{"duration":349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":933.2999999523163,"connectEnd":933.2999999523163,"connectStart":933.2999999523163,"domainLookupEnd":933.2999999523163,"domainLookupStart":933.2999999523163,"fetchStart":933.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":1266.7999999523163,"responseEnd":1282.2999999523163,"responseStart":1279.5999999046326,"secureConnectionStart":933.2999999523163}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":156,"responseStart":305,"responseEnd":390,"domLoading":308,"domInteractive":1311,"domContentLoadedEventStart":1311,"domContentLoadedEventEnd":1347,"domComplete":1598,"loadEventStart":1598,"loadEventEnd":1599,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1283.7000000476837},{"name":"bigPipe.sidebar-id.end","time":1284.5},{"name":"bigPipe.activity-panel-pipe-id.start","time":1284.7000000476837},{"name":"bigPipe.activity-panel-pipe-id.end","time":1286.7999999523163},{"name":"activityTabFullyLoaded","time":1361.5999999046326}],"measures":[],"correlationId":"da77ee267aa85a","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":91,"dbReadsTimeInMs":12,"dbConnsTimeInMs":20,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
monty Enums are also broken. That is, if the slave modified its table's enums to have different ordering than the master, the enums are picked based on index, rather than value. Modifying your skr.test to instead test enums shows this:
--source include/have_innodb.inc
--source include/have_binlog_format_row.inc
--source include/master-slave.inc
CREATE TABLE t1 (a INT PRIMARY KEY, en ENUM('a','b','c')) ENGINE = innodb;
INSERT INTO t1 VALUES(100,'a'), (200,'b');
--sync_slave_with_master
--connection slave
alter table t1 modify column en enum ('a','d', 'b', 'c');
connection master;
INSERT INTO t1 VALUES(300, 'b');
--sync_slave_with_master
select * from t1;
connection master;
update t1 set en='b' where A=100;
--sync_slave_with_master
select * from t1;
connection master;
--source include/rpl_end.inc
The slave's result (where rows with a=100 and a=300 should both actually have value b, but show d because of the re-arrangement:
a en
100 d
200 b
300 d
Note too that binlog_row_metadata=FULL will also print the expectations for the ENUMs, so we can use this to reference the desired enum value.
#250313 14:49:07 server id 1 end_log_pos 944 CRC32 0x2167fc1b Table_map: `test`.`t1` mapped to number 33
# Columns(`a` INT NOT NULL,
# `en` ENUM('a','b','c') CHARSET latin1 COLLATE latin1_swedish_ci)
# Primary Key(a)