Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15466

SST not working after upgrade from 10.2.11

Details

    Description

      gcache.size was sufficient enough not to require a SST on the previous upgrades so the problem may be older.

      2 node + garb setup running since 10.2.9

      Yesterday after a blackout, tried restarting the cluster. 2nd node was damaged bad, so I wiped out the entire datadir in order to do a full SST

      initial config was using rsync. tried to do it with mariabackup and xtrabackup-v2

      both servers have identical config, except for node names
      firewall and selinux both off

      Attachments

        1. donor-mariabackup-innobackup.backup.log
          7 kB
          Cătălin Nicolescu
        2. donor-mariabackup-mysql-error.log
          18 kB
          Cătălin Nicolescu
        3. donor-rsync-mysql-error.log
          24 kB
          Cătălin Nicolescu
        4. donor-xtrabackup-mysql-error.log
          14 kB
          Cătălin Nicolescu
        5. joiner-mariabackup-mysql-error.log
          10 kB
          Cătălin Nicolescu
        6. joiner-rsync-mysql-error.log
          18 kB
          Cătălin Nicolescu
        7. joiner-xtrabackup-mysql-error.log
          8 kB
          Cătălin Nicolescu
        8. server.cnf
          3 kB
          Cătălin Nicolescu

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15607 mysqld crashed few after node is being joined with sst

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15436 If log_bin and log_bin_index is different SST with rsync fails.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15453 IST failed during upgrade of version and SST failed too from 10.2.12 to 10.2.13

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            jplindst Jan Lindström (Inactive) added a comment -

            Hi, thank you for your bug report and detailed logs. Joiner is killed during SST, at least in mariabackup case
            WSREP_SST: [INFO] Waiting for SST streaming to complete! (20180305 10:52:16.898)
            2018-03-05 10:52:18 140327973275392 [Note] WSREP: (80ad3090, 'tcp://0.0.0.0:4567') turning message relay requesting off
            WSREP_SST: [ERROR] Removing /var/lib/mysql//.sst/xtrabackup_galera_info file due to signal (20180305 10:53:44.684)
            could it be due to systemd?

            jplindst Jan Lindström (Inactive) added a comment - Hi, thank you for your bug report and detailed logs. Joiner is killed during SST, at least in mariabackup case WSREP_SST: [INFO] Waiting for SST streaming to complete! (20180305 10:52:16.898) 2018-03-05 10:52:18 140327973275392 [Note] WSREP: (80ad3090, 'tcp://0.0.0.0:4567') turning message relay requesting off WSREP_SST: [ERROR] Removing /var/lib/mysql//.sst/xtrabackup_galera_info file due to signal (20180305 10:53:44.684) could it be due to systemd?
            catalin.nicolescu Cătălin Nicolescu added a comment -

            Yes.
            Doing systemctl edit mariadb.service and overriding with

            [Service]
            		 
            TimeoutStartSec=0

            did the trick for version 10.2.15 but I think that this was the issue all along

            catalin.nicolescu Cătălin Nicolescu added a comment - Yes. Doing systemctl edit mariadb.service and overriding with [Service] TimeoutStartSec= 0 did the trick for version 10.2.15 but I think that this was the issue all along

            People

              jplindst Jan Lindström (Inactive)
              catalin.nicolescu Cătălin Nicolescu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.