[MDEV-30967] sst mariabackup broken on rocksdb - cannot GET_LOCK Created: 2023-03-30  Updated: 2023-06-06  Resolved: 2023-06-05

Status: Closed
Project: MariaDB Server
Component/s: Galera, mariabackup
Affects Version/s: 10.4.28
Fix Version/s: 10.4.30, 10.5.21, 10.6.14, 10.9.7, 10.10.5, 10.11.4

Type: Bug Priority: Critical
Reporter: Daniel Black Assignee: Sergei Golubchik
Resolution: Duplicate Votes: 0
Labels: regression, regression-10.4

Issue Links:
Duplicate
duplicates MDEV-31325 Can not run db patches using flyway w... Closed
Problem/Incident
is caused by MDEV-30473 Do not allow GET_LOCK() / RELEASE_LOC... Closed
Relates
relates to MDEV-13122 Backup myrocks with mariabackup Closed

 Description   

GET_LOCK / RELEASE_LOCK now errors on Galera due to non-support. RocksDB uses this for checkpoint locking, an alternative locking mechanism is needed.

10.5.a6780df49b443b172124e7e881ed0bea54d75907, url=https://buildbot.mariadb.org/#/builders/408/builds/8250 daemon log file

 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [00] 2023-03-29 03:26:34 Skipping db: ./#rocksdb
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [00] 2023-03-29 03:26:34 Finished backing up non-InnoDB tables and files
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [01] 2023-03-29 03:26:34 Streaming ./aria_log_control to <STDOUT>
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [01] 2023-03-29 03:26:34         ...done
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [01] 2023-03-29 03:26:34 Streaming ./aria_log.00000001 to <STDOUT>
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [01] 2023-03-29 03:26:34         ...done
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [00] 2023-03-29 03:26:34 Obtaining rocksdb checkpoint lock.
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -innobackupex-backup: [00] FATAL ERROR: 2023-03-29 03:26:34 failed to execute query SELECT GET_LOCK('mariabackup_rocksdb_checkpoint',3600): This version of MariaDB doesn't yet support 'GET_LOCK in cluster (WSREP_ON=ON)'
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-donor: mariadb-backup finished with error: 1. Check syslog or '/var/lib/node1/mariabackup.backup.log' for details
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-donor: Cleanup after exit with status: 22
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-donor: Removing /var/lib/node1/xtrabackup_galera_info file due to signal
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-donor: Cleaning up temporary directories
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-joiner: xtrabackup_checkpoints missing, failed mariadb-backup/SST on donor
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-joiner: Cleanup after exit with status: 2
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-joiner: Removing /var/lib/node2/.sst/xtrabackup_galera_info file due to signal
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-joiner: Removing the sst_in_progress file
 
Mar 29 03:26:34 bb-hz-bbw5-debian-10 -wsrep-sst-joiner: Cleaning up temporary directories
 
 



 Comments   
Comment by Jan Lindström [ 2023-06-01 ]

Why FLUSH TABLES WITH READ LOCK is not enough in this case ?

Comment by Daniel Black [ 2023-06-06 ]

the GET_LOCK is protecting access to the rocksdb_create_checkpoint and the directory snapshot it creates from another mariadb-backup process.

Some mutual file advisory locks on file representing xtrabackup_target_dir would be just as protective.

Generated at Thu Feb 08 10:20:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.