Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31546

Galera mariabackup SST ignores mysqld_safe --chroot directive and loses track of the datadir, causing it to abort

    XMLWordPrintable

Details

    • Bug
    • Status: Confirmed (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.6.8
    • 10.6
    • Galera SST
    • None
    • Centos 7.7

    Description

      When running the server from within a chroot jail everything works as expected and SSTs complete successfully.

      However, when starting a chrooted MariaDB server from outside and using mysqld_safe with the --chroot directive, mysqld_safe requires a de-referenced configuration file.

      For example, if the root of the jail is at /var/mdbj, and the config file is at /var/mdbj/etc/my.cnf, then mysqld_safe needs to be started with an absolute path for the configuration file. Then wsrep_sst_mariabackup appears to run outside the jail, failing to find the data directory, and aborts:

      ~# mysqld_safe --defaults-file=/var/mdbj/etc/my.cnf --chroot=/var/mdbj &
      230626 09:00:48 mysqld_safe Logging to '/var/lib/mysql/cluster.err'.
      230626 09:00:48 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
      ~# tail /var/mdbj/var/lib/mysql/cluster.err
       
      2023-06-26  9:00:49 2 [Note] WSREP: Server status change connected -> joiner
      2023-06-26  9:00:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-06-26  9:00:49 0 [Note] WSREP: Joiner monitor thread started to monitor
      2023-06-26  9:00:49 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.0.0.45' --datadir '/var/lib/mysql/' --defaults-file '/var/mdbj/etc/my.cnf' --parent '2906' --binlog 'cluster-bin' --binlog-index 'cluster-bin.index' --mysqld-args --defaults-file=/var/mdbj/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/lib64/galera-4/libgalera_smm.so --chroot=/var/mdbj --log-error=/var/lib/mysql/cluster.err --pid-file=cluster.pid --wsrep_start_position=22632b72-13f4-11ee-9de8-afc8fbe29119:10,44-45-2'
      WSREP_SST: [INFO] mariabackup SST started on joiner (20230626 09:00:49.542)
      Could not open required defaults file: /var/mdbj/etc/my.cnf
      Fatal error in defaults handling. Program aborted
      

      #... these last 2 lines repeats several times, and then:

      WSREP_SST: [ERROR] Streaming with mbstream, but mbstream not found in path (20230626 09:00:49.659)
      2023-06-26  9:00:49 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_mariabackup --role 'joiner' --address '10.0.0.45' --datadir '/var/lib/mysql/' --defaults-file '/var/mdbj/etc/my.cnf' --parent '2906' --binlog 'cluster-bin' --binlog-index 'cluster-bin.index' --mysqld-args --defaults-file=/var/mdbj/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/lib64/galera-4/libgalera_smm.so --chroot=/var/mdbj --log-error=/var/lib/mysql/cluster.err --pid-file=cluster.pid --wsrep_start_position=22632b72-13f4-11ee-9de8-afc8fbe29119:10,44-45-2
              Read: '(null)'
      2023-06-26  9:00:49 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.0.0.45' --datadir '/var/lib/mysql/' --defaults-file '/var/mdbj/etc/my.cnf' --parent '2906' --binlog 'cluster-bin' --binlog-index 'cluster-bin.index' --mysqld-args --defaults-file=/var/mdbj/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/lib64/galera-4/libgalera_smm.so --chroot=/var/mdbj --log-error=/var/lib/mysql/cluster.err --pid-file=cluster.pid --wsrep_start_position=22632b72-13f4-11ee-9de8-afc8fbe29119:10,44-45-2: 42 (No message of desired type)
      2023-06-26  9:00:49 2 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
      2023-06-26  9:00:49 2 [ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.
      

      It's worth noting that when using rsync, the SST fails the first time, but if run a second time it works...

      Attachments

        Activity

          People

            sysprg Julius Goryavsky
            juan.vera Juan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.