[MDEV-30157] Galera SST doesn't properly handle undo* files from innodb Created: 2022-12-05  Updated: 2023-04-04  Resolved: 2023-01-11

Status: Closed
Project: MariaDB Server
Component/s: Galera, Galera SST
Affects Version/s: 10.3.37, 10.4.27, 10.5.18, 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2, 10.11.0
Fix Version/s: 10.11.2, 10.3.38, 10.4.28, 10.5.19, 10.6.12, 10.7.8, 10.8.7, 10.9.5, 10.10.3

Type: Bug Priority: Blocker
Reporter: Julius Goryavsky Assignee: Julius Goryavsky
Resolution: Fixed Votes: 1
Labels: None

Issue Links:
Blocks
blocks MDEV-29986 Set innodb_undo_tablespaces=3 by default Closed
Duplicate
duplicates MDEV-28669 Cannot join second node to new cluste... Closed
Issue split
split to MDEV-30383 improvements to mtr to properly proce... Stalled

 Description   

Galera SST doesn't handle undo* files from innodb correctly - as a result, SST may terminate incorrectly when --innodb-undo-tablespaces is set to 3 or more (and possibly other settings related to undo tablespaces):

Dec  1 09:52:47 debian-buster-amd64 mariadbd[3996]: 2022-12-01  9:52:47 0 [Note] InnoDB: Completed initialization of buffer pool
 
Dec  1 09:52:47 debian-buster-amd64 mariadbd[3996]: 2022-12-01  9:52:47 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
 
Dec  1 09:52:47 debian-buster-amd64 mariadbd[3996]: 2022-12-01  9:52:47 0 [ERROR] InnoDB: Expected to open innodb_undo_tablespaces=3 but was able to find only 0
 
Dec  1 09:52:47 debian-buster-amd64 mariadbd[3996]: 2022-12-01  9:52:47 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
 
Dec  1 09:52:47 debian-buster-amd64 mariadbd[3996]: 2022-12-01  9:52:47 0 [Note] InnoDB: Starting shutdown...
 
Dec  1 09:52:47 debian-buster-amd64 mariadbd[3996]: 2022-12-01  9:52:47 0 [ERROR] Plugin 'InnoDB' init function returned error.



 Comments   
Comment by Marko Mäkelä [ 2022-12-12 ]

A fairly large number of tests is affected by this, as illustrated by this work-around.

Comment by Marko Mäkelä [ 2022-12-12 ]

I noticed one more related test failure:

CURRENT_TEST: galera_3nodes.galera_garbd_backup
 
…
 
2022-12-12  9:23:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2022-12-12  9:23:53 2 [ERROR] InnoDB: Transaction id 0x1e associated with recordCOMPACT RECORD(info_bits=32, 7 fields): {[36]a847d38c-79fe-11ed-900d-3a9f991f9ce5(0x61383437643338632D373966652D313165642D393030642D336139663939316639636535),[6]      (0x00000000001E),[7]    1  (0x0A0000003101D4),[8]        (0x8000000000000002),[8]        (0x8000000000000002),[4]    (0x80000004),[4]    (0x8002D17F)} in index `PRIMARY` of table `mysql`.`wsrep_cluster` is greater than the global counter 21! The table is corrupted.
 
2022-12-12  9:23:53 2 [ERROR] InnoDB: Transaction id 0x1e associated with recordCOMPACT RECORD(info_bits=32, 7 fields): {[36]a847d38c-79fe-11ed-900d-3a9f991f9ce5(0x61383437643338632D373966652D313165642D393030642D336139663939316639636535),[6]      (0x00000000001E),[7]    1  (0x0A0000003101D4),[8]        (0x8000000000000002),[8]        (0x8000000000000002),[4]    (0x80000004),[4]    (0x8002D17F)} in index `PRIMARY` of table `mysql`.`wsrep_cluster` is greater than the global counter 21! The table is corrupted.
 
mariadbd: /home/buildbot/buildbot/build/storage/innobase/btr/btr0cur.cc:3233: dberr_t btr_cur_upd_rec_sys(buf_block_t*, rec_t*, dict_index_t*, const rec_offs*, const trx_t*, roll_ptr_t, mtr_t*): Assertion `roll_ptr == 0 || lock_check_trx_id_sanity(trx_read_trx_id(rec + offset), rec, index, offsets)' failed.

Comment by Jan Lindström (Inactive) [ 2022-12-21 ]

ok to push.

Please make 10.4-10.6 versions in one branch (all of these in same).

Comment by Julius Goryavsky [ 2023-01-11 ]

Fixed, https://github.com/MariaDB/server/commit/b84f3fa769228c4ffd367bc8f2426e6e912325a5 (and https://github.com/MariaDB/server/commit/e4a4aad7cf5a8cdb3ae4d97527f489cc590146a1)

Generated at Thu Feb 08 10:14:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.