[MDEV-19708] RBR replication loses data silently ignoring important column attributes Created: 2019-06-07  Updated: 2023-12-22

Status: Stalled
Project: MariaDB Server
Component/s: Replication
Fix Version/s: 11.5

Type: Task Priority: Major
Reporter: Alexander Barkov Assignee: Andrei Elkin
Resolution: Unresolved Votes: 2
Labels: upstream-fixed

Issue Links:
PartOf
includes MDEV-20477 Merge binlog extended metadata suppor... Closed
Relates
relates to MDEV-5671 Add Table_metadata event Open
Epic Link: Data type cleanups

 Description   

RBR replication does not store enough column attributes, which leads to silent data loss.

These important attributes are known not to be stored:

  • The UNSIGNED flag for INT-alike columns
  • The character set for STRING columns

It should be fixed to store all important attributes in the binary log.
Note, we'll need to store the precise data type name for pluggable data types soon (MDEV-4912). It would be nice to fix this problem at once.

Detailed description:

The UNSIGNED flag

In the following test I create a table with an INT UNSIGNED column on the master, but on the slave I intentionally remove the UNSIGNED flag, to make fields not fully compatible.

I create a test file AAA-INT.test with the following content:

--source include/master-slave.inc
 
connection master;
CREATE TABLE t1 (a INT UNSIGNED);
sync_slave_with_master;
connection slave;
 
SET GLOBAL slave_type_conversions='';
 
ALTER TABLE t1 MODIFY a INT;
connection master;
 
INSERT INTO t1 VALUES (0);
INSERT INTO t1 VALUES (0xFFFFFFFF);
 
sync_slave_with_master;
connection slave;
SELECT * FROM t1;
SET GLOBAL slave_type_conversions=DEFAULT;
 
connection master;
DROP TABLE t1;
sync_slave_with_master;
 
--source include/rpl_end.inc

Now if I run "./mtr AAA-INT,row", the test works without errors and displays the following output:

..
connection slave;
SELECT * FROM t1;
a
0
-1
..

Notice:

  • The slave did not refuse such conversion, ignoring the fact that ALL_LOSSY is not enabled in slave_type_conversions.
  • It silently re-interpreted a huge unsigned value to -1.

Note, if I run "./mtr AAA-INT,stmt", the test returns on an error as expected:

Last_Error	Error 'Out of range value for column 'a' at row 1' on query. Default database: 'test'. Query: 'INSERT INTO t1 VALUES (0xFFFFFFFF)'

The character set

The same problem happens with string fields having different character sets.
I create a test file AAA-STR.test with the following content:

--source include/master-slave.inc
 
connection master;
SET NAMES utf8;
CREATE TABLE t1 (a CHAR(10) CHARACTER SET latin1);
sync_slave_with_master;
 
connection slave;
SET NAMES utf8;
SET GLOBAL slave_type_conversions='';
ALTER TABLE t1 MODIFY a CHAR(10) CHARACTER SET latin2;
connection master;
INSERT INTO t1 VALUES (_latin1 0xC0);
SELECT a FROM t1;
 
sync_slave_with_master;
connection slave;
SELECT * FROM t1;
SET GLOBAL slave_type_conversions=DEFAULT;
 
connection master;
DROP TABLE t1;
sync_slave_with_master;
 
--source include/rpl_end.inc

Now I run "./mtr AAA-STR,row", the test works without errors and displays the following output:

...
connection master;
INSERT INTO t1 VALUES (_latin1 0xC0);
SELECT a FROM t1;
a
À
connection slave;
connection slave;
SELECT * FROM t1;
a
Ŕ
...

Notice:

  • The slave did not refuse such conversion, ignoring the fact that ALL_LOSSY is not enabled in slave_type_conversions.
  • It silently re-interpreted U+00C0 LATIN CAPITAL LETTER A WITH GRAVE to U+0154 LATIN CAPITAL LETTER R WITH ACUTE

Note, if I run "./mtr AAA-STR,stmt", the test returns on an error as expected:

Last_Error	Error 'Incorrect string value: '\xC0' for column `test`.`t1`.`a` at row 1' on query. Default database: 'test'. Query: 'INSERT INTO t1 VALUES (_latin1 0xC0)'



 Comments   
Comment by Alexander Barkov [ 2019-06-07 ]

Worklog 4618 fixed this problem in MySQL-8.0.1

Related changes:

commit c019294ca8fece7af3bca6e6190e4e1efafa22af
Author: Libing Song <libing.song@oracle.com>
Date:   Tue Feb 28 09:56:53 2017 +0800
 
    WL#4618 RBR: extended table metadata in the binary log

commit 80c33082d0d033ba35e9ddcd5b848f50e59a1aa4
Author: Jon Olav Hauglid <jon.hauglid@oracle.com>
Date:   Mon Mar 6 11:36:05 2017 +0100
 
    WL#4618 RBR: extended table metadata in the binary log
    
    Post-push fix: Fix Clang -Wtautological-constant-out-of-range-compare
    build warning.

commit 0cdb2d9c5882d3c1c698dfe4b31510c27deacd45
Author: Maria Couceiro <maria.couceiro@oracle.com>
Date:   Thu May 18 15:35:21 2017 +0100
 
    BUG#26020990 --PRINT-TABLE-METADATA DOES NOT DISPLAY ALL METADATA IN TABLE_MAP_LOG EVENT

Comment by Sachin Setiya (Inactive) [ 2019-06-22 ]

Also this commit

commit e5b96670ed2aa075bab2eb33c6e5d675100979dd
Author: Sven Sandberg <sven.sandberg@oracle.com>
Date:   Fri Oct 12 10:58:35 2018 +0200
 
    BUG#28706307: CHARACTER SET OF ENUM DATA TYPE IS NOT AVAILABLE AS PART OF OPTIONAL METADATA
    BUG#28774144: MYSQLBINLOG --PRINT-TABLE-METADATA SHOW WRONG INFO ABOUT DEFAULT CHARSET
    
    Problems:

Comment by Sachin Setiya (Inactive) [ 2019-06-25 ]

http://lists.askmonty.org/pipermail/commits/2019-June/013877.html

Comment by Sachin Setiya (Inactive) [ 2019-08-14 ]

bb-10.5-19708

Comment by Alexander Barkov [ 2019-09-04 ]

The patch
https://github.com/MariaDB/server/commit/05061cd4ff0dbc7738c9e360b1b52472476cd224
looks OK to push for me.
Thanks for addressing review suggestions!

Generated at Thu Feb 08 08:53:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.