[MDEV-23586] Mariabackup: GTID saved for replication in 10.4.14 is wrong - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Fixed
Affects Version/s: 10.4.14
Fix Version/s: 10.4.16, 10.5.7
Component/s: mariabackup, Server
Labels:
None

Description

It appears that the fix for https://jira.mariadb.org/browse/MDEV-21953 has introduced a serious regression and now any backup taken with MariaDB server 10.4.14 and above saves the wrong GTID.

To reproduce, take a mariabackup from a 10.4.14 instance, then notice how the reported position does refer to a GTID which is not proper. In the example below, the PIT corresponds to position 25035029, but the reported GTID 0-1-18691 is wrong and if used, yields several duplicate keys errors. In fact the correct GTID for that position would be 0-1-18699 as reported by BINLOG_GTID_POS() function.

This wrong GTID is saved to the xtrabackup_slave_info and since the position is no longer saved there (WHY???) the resulting backup does not carry a valid PIT information anymore.
This is hitting a customer of ours and makes all backups inconsistent due to wrong PIT info, hence the critical priority.

Thanks to aponzo who found and verified this issue.

Workaround: downgrade server to 10.4.13

[00] 2020-08-26 12:25:52 Backup created in directory '/root/backup/andre2/xtrabackup_backupfiles/'

[00] 2020-08-26 12:25:52 MySQL binlog position: filename 'm2-bin.000007', position '25035029', GTID of the last change '0-1-18691'

[00] 2020-08-26 12:25:52 MySQL slave binlog position: master host '192.168.56.101', gtid_slave_pos 0-1-18691

[00] 2020-08-26 12:25:52 Streaming backup-my.cnf

[00] 2020-08-26 12:25:52         ...done

[00] 2020-08-26 12:25:52 Streaming xtrabackup_info

[00] 2020-08-26 12:25:52         ...done

[00] 2020-08-26 12:25:52 Redo log (from LSN 29523906 to 31884588) was copied.

[00] 2020-08-26 12:25:52 completed OK!

prepare

2020-08-26 12:26:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=30269374

2020-08-26 12:26:53 0 [Note] InnoDB: Starting final batch to recover 286 pages from redo log.

2020-08-26 12:26:53 0 [Note] InnoDB: Last binlog file './m2-bin.000007', position 25035029

[00] 2020-08-26 12:26:54 Last binlog file ./m2-bin.000007, position 25035029

[00] 2020-08-26 12:26:54 completed OK!

MariaDB [test]> SELECT BINLOG_GTID_POS("m2-bin.000007",25035029 );

+--------------------------------------------+

| BINLOG_GTID_POS("m2-bin.000007",25035029 ) |

+--------------------------------------------+

| 0-1-18699                                  |

+--------------------------------------------+

1 row in set (0.133 sec)

Attachments

Issue Links

causes

MDEV-25334 FTWRL/Backup blocks DDL on temporary tables with binlog enabled, assertion fails in Diagnostics_area::set_error_status

Closed

relates to

MDEV-21611 mariabackup should update binlog position in InnoDB during prepare

Open

MDEV-21953 deadlock between BACKUP STAGE BLOCK_COMMIT and parallel replication

Closed

Activity

Ascending order - Click to sort in descending order

Rick Pizzi (Inactive) created issue - 2020-08-26 12:14

Rick Pizzi (Inactive) made changes - 2020-08-26 12:14

Field	Original Value	New Value
Link		This issue relates to ~~MDEV-21953~~ [ ~~MDEV-21953~~ ]

Rick Pizzi (Inactive) made changes - 2020-08-26 12:14

Summary

GTID saved for replication in 10.4.14 is wrong

Mariabackup: GTID saved for replication in 10.4.14 is wrong

Julien Fritsch made changes - 2020-08-26 12:15

Assignee

Michael Widenius [ monty ]

Julien Fritsch made changes - 2020-08-26 12:16

Fix Version/s

10.4 [ 22408 ]

Geoff Montee (Inactive) made changes - 2020-08-26 17:25

Link

This issue relates to MDEV-21611 [ MDEV-21611 ]

Rick Pizzi (Inactive) made changes - 2020-08-27 10:36

Description

It appears that the fix for https://jira.mariadb.org/browse/MDEV-21953 has introduced a serious regression and now any backup taken with MariaDB server 10.4.14 and above saves the wrong GTID.

To reproduce, take a mariabackup from a 10.4.14 instance, then notice how the reported position does refer to a GTID which is not proper. In the example below, the PIT corresponds to position 25035029, but the reported GTID 0-1-18691 is wrong and if used, yields several duplicate keys errors. In fact the correct GTID for that position would be 0-1-18699 as reported by BINLOG_GTID_POS() function.

This wrong GTID is saved to the xtrabackup_binlog_info and since the position is no longer saved there (WHY???) *the resulting backup does not carry a valid PIT information anymore*.
This is hitting a customer of ours and makes all backups inconsistent due to wrong PIT info, hence the critical priority.

Thanks to [~aponzo] who found and verified this issue.

Workaround: downgrade server to 10.4.13

{code}
[00] 2020-08-26 12:25:52 Backup created in directory '/root/backup/andre2/xtrabackup_backupfiles/'
[00] 2020-08-26 12:25:52 MySQL binlog position: filename 'm2-bin.000007', position '25035029', GTID of the last change '0-1-18691'
[00] 2020-08-26 12:25:52 MySQL slave binlog position: master host '192.168.56.101', gtid_slave_pos 0-1-18691
[00] 2020-08-26 12:25:52 Streaming backup-my.cnf
[00] 2020-08-26 12:25:52 ...done
[00] 2020-08-26 12:25:52 Streaming xtrabackup_info
[00] 2020-08-26 12:25:52 ...done
[00] 2020-08-26 12:25:52 Redo log (from LSN 29523906 to 31884588) was copied.
[00] 2020-08-26 12:25:52 completed OK!
prepare
2020-08-26 12:26:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=30269374
2020-08-26 12:26:53 0 [Note] InnoDB: Starting final batch to recover 286 pages from redo log.
2020-08-26 12:26:53 0 [Note] InnoDB: Last binlog file './m2-bin.000007', position 25035029
[00] 2020-08-26 12:26:54 Last binlog file ./m2-bin.000007, position 25035029
[00] 2020-08-26 12:26:54 completed OK!
MariaDB [test]> SELECT BINLOG_GTID_POS("m2-bin.000007",25035029 );
+--------------------------------------------+
| BINLOG_GTID_POS("m2-bin.000007",25035029 ) |
+--------------------------------------------+
| 0-1-18699 |
+--------------------------------------------+
1 row in set (0.133 sec)
{code}

It appears that the fix for https://jira.mariadb.org/browse/MDEV-21953 has introduced a serious regression and now any backup taken with MariaDB server 10.4.14 and above saves the wrong GTID.

To reproduce, take a mariabackup from a 10.4.14 instance, then notice how the reported position does refer to a GTID which is not proper. In the example below, the PIT corresponds to position 25035029, but the reported GTID 0-1-18691 is wrong and if used, yields several duplicate keys errors. In fact the correct GTID for that position would be 0-1-18699 as reported by BINLOG_GTID_POS() function.

This wrong GTID is saved to the xtrabackup_slave_info and since the position is no longer saved there (WHY???) *the resulting backup does not carry a valid PIT information anymore*.
This is hitting a customer of ours and makes all backups inconsistent due to wrong PIT info, hence the critical priority.

Thanks to [~aponzo] who found and verified this issue.

Workaround: downgrade server to 10.4.13

{code}
[00] 2020-08-26 12:25:52 Backup created in directory '/root/backup/andre2/xtrabackup_backupfiles/'
[00] 2020-08-26 12:25:52 MySQL binlog position: filename 'm2-bin.000007', position '25035029', GTID of the last change '0-1-18691'
[00] 2020-08-26 12:25:52 MySQL slave binlog position: master host '192.168.56.101', gtid_slave_pos 0-1-18691
[00] 2020-08-26 12:25:52 Streaming backup-my.cnf
[00] 2020-08-26 12:25:52 ...done
[00] 2020-08-26 12:25:52 Streaming xtrabackup_info
[00] 2020-08-26 12:25:52 ...done
[00] 2020-08-26 12:25:52 Redo log (from LSN 29523906 to 31884588) was copied.
[00] 2020-08-26 12:25:52 completed OK!
prepare
2020-08-26 12:26:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=30269374
2020-08-26 12:26:53 0 [Note] InnoDB: Starting final batch to recover 286 pages from redo log.
2020-08-26 12:26:53 0 [Note] InnoDB: Last binlog file './m2-bin.000007', position 25035029
[00] 2020-08-26 12:26:54 Last binlog file ./m2-bin.000007, position 25035029
[00] 2020-08-26 12:26:54 completed OK!
MariaDB [test]> SELECT BINLOG_GTID_POS("m2-bin.000007",25035029 );
+--------------------------------------------+
| BINLOG_GTID_POS("m2-bin.000007",25035029 ) |
+--------------------------------------------+
| 0-1-18699 |
+--------------------------------------------+
1 row in set (0.133 sec)
{code}

Julien Fritsch made changes - 2020-08-27 13:10

Labels

need_feedback

Julien Fritsch made changes - 2020-08-28 08:37

Assignee

Michael Widenius [ monty ]

Vladislav Lesin [ vlad.lesin ]

Julien Fritsch made changes - 2020-08-28 08:37

Labels

need_feedback

Vladislav Lesin made changes - 2020-09-04 06:08

Assignee

Vladislav Lesin [ vlad.lesin ]

Michael Widenius [ monty ]

Michael Widenius made changes - 2020-09-07 07:23

Status

Open [ 1 ]

In Progress [ 3 ]

Michael Widenius made changes - 2020-09-07 07:40

Assignee	Michael Widenius [ monty ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Vladislav Lesin made changes - 2020-09-08 17:54

Link

This issue relates to MDEV-21611 [ MDEV-21611 ]

Sergei Golubchik made changes - 2020-09-21 12:54

Priority

Critical [ 2 ]

Blocker [ 1 ]

Sergei Golubchik made changes - 2020-09-21 12:54

Fix Version/s

10.5 [ 23123 ]

Michael Widenius made changes - 2020-09-25 14:22

issue.field.resolutiondate

2020-09-25 14:22:50.0

2020-09-25 14:22:50.189

Michael Widenius made changes - 2020-09-25 14:22

Fix Version/s		10.4.15 [ 24507 ]
Fix Version/s		10.5.6 [ 24508 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Assignee	Sergei Golubchik [ serg ]	Michael Widenius [ monty ]
Resolution		Fixed [ 1 ]
Status	In Review [ 10002 ]	Closed [ 6 ]

Ralf Gebhardt made changes - 2020-10-06 18:35

Fix Version/s

10.5.7 [ 25019 ]

Ralf Gebhardt made changes - 2020-10-06 18:37

Fix Version/s

10.5.6 [ 24508 ]

Ralf Gebhardt made changes - 2020-10-06 18:40

Fix Version/s

10.4.16 [ 25020 ]

Ralf Gebhardt made changes - 2020-10-06 18:42

Fix Version/s

10.4.15 [ 24507 ]

Elena Stepanova made changes - 2021-04-03 17:13

Link

This issue causes ~~MDEV-25334~~ [ ~~MDEV-25334~~ ]

Marko Mäkelä made changes - 2021-08-27 10:14

Link

This issue relates to MDEV-21611 [ MDEV-21611 ]

Sergei Golubchik made changes - 2021-12-06 21:52

Workflow

MariaDB v3 [ 112953 ]

MariaDB v4 [ 158300 ]

Jira Automation (IT) made changes - 2024-07-04 02:52

Zendesk Related Tickets

120782 129317

People

Assignee:: Michael Widenius

Reporter:: Rick Pizzi (Inactive)

Votes:: 4 Vote for this issue

Watchers:: 14 Start watching this issue

Dates

Created:: 2020-08-26 12:14

Updated:: 2024-07-07 21:57

Resolved:: 2020-09-25 14:22

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration