[MDEV-31546] Galera mariabackup SST ignores mysqld_safe --chroot directive and loses track of the datadir, causing it to abort Created: 2023-06-26  Updated: 2023-07-24

Status: Confirmed
Project: MariaDB Server
Component/s: Galera SST
Affects Version/s: 10.6.8
Fix Version/s: 10.6

Type: Bug Priority: Critical
Reporter: Juan Assignee: Julius Goryavsky
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Centos 7.7


 Description   

When running the server from within a chroot jail everything works as expected and SSTs complete successfully.

However, when starting a chrooted MariaDB server from outside and using mysqld_safe with the --chroot directive, mysqld_safe requires a de-referenced configuration file.

For example, if the root of the jail is at /var/mdbj, and the config file is at /var/mdbj/etc/my.cnf, then mysqld_safe needs to be started with an absolute path for the configuration file. Then wsrep_sst_mariabackup appears to run outside the jail, failing to find the data directory, and aborts:

~# mysqld_safe --defaults-file=/var/mdbj/etc/my.cnf --chroot=/var/mdbj &
 
230626 09:00:48 mysqld_safe Logging to '/var/lib/mysql/cluster.err'.
 
230626 09:00:48 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
 
~# tail /var/mdbj/var/lib/mysql/cluster.err
 
 
 
2023-06-26  9:00:49 2 [Note] WSREP: Server status change connected -> joiner
 
2023-06-26  9:00:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
2023-06-26  9:00:49 0 [Note] WSREP: Joiner monitor thread started to monitor
 
2023-06-26  9:00:49 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.0.0.45' --datadir '/var/lib/mysql/' --defaults-file '/var/mdbj/etc/my.cnf' --parent '2906' --binlog 'cluster-bin' --binlog-index 'cluster-bin.index' --mysqld-args --defaults-file=/var/mdbj/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/lib64/galera-4/libgalera_smm.so --chroot=/var/mdbj --log-error=/var/lib/mysql/cluster.err --pid-file=cluster.pid --wsrep_start_position=22632b72-13f4-11ee-9de8-afc8fbe29119:10,44-45-2'
 
WSREP_SST: [INFO] mariabackup SST started on joiner (20230626 09:00:49.542)
 
Could not open required defaults file: /var/mdbj/etc/my.cnf
 
Fatal error in defaults handling. Program aborted

#... these last 2 lines repeats several times, and then:

WSREP_SST: [ERROR] Streaming with mbstream, but mbstream not found in path (20230626 09:00:49.659)
 
2023-06-26  9:00:49 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_mariabackup --role 'joiner' --address '10.0.0.45' --datadir '/var/lib/mysql/' --defaults-file '/var/mdbj/etc/my.cnf' --parent '2906' --binlog 'cluster-bin' --binlog-index 'cluster-bin.index' --mysqld-args --defaults-file=/var/mdbj/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/lib64/galera-4/libgalera_smm.so --chroot=/var/mdbj --log-error=/var/lib/mysql/cluster.err --pid-file=cluster.pid --wsrep_start_position=22632b72-13f4-11ee-9de8-afc8fbe29119:10,44-45-2
 
        Read: '(null)'
 
2023-06-26  9:00:49 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.0.0.45' --datadir '/var/lib/mysql/' --defaults-file '/var/mdbj/etc/my.cnf' --parent '2906' --binlog 'cluster-bin' --binlog-index 'cluster-bin.index' --mysqld-args --defaults-file=/var/mdbj/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/lib64/galera-4/libgalera_smm.so --chroot=/var/mdbj --log-error=/var/lib/mysql/cluster.err --pid-file=cluster.pid --wsrep_start_position=22632b72-13f4-11ee-9de8-afc8fbe29119:10,44-45-2: 42 (No message of desired type)
 
2023-06-26  9:00:49 2 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
 
2023-06-26  9:00:49 2 [ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.

It's worth noting that when using rsync, the SST fails the first time, but if run a second time it works...

 Comments   
Comment by Juan [ 2023-07-19 ]

Hi julien.fritsch,

> Juan why setting this customer bug to minor? If so, it will never be fixed.

I set it as minor because it's only a problem starting the chrooted server one specific way so there's a trivial workaround. While it would be correct to either remove the switch or fix it, I didn't feel entirely comfortable calling it "major". If you feel that it needs to be set to Major, please do so, or let me know & I will.

> Also, have you been able to reproduce this?

Yes: the description goes into some detail about how and what the findings were.

Generated at Thu Feb 08 10:24:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.