Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10480

Unable to start Galera cluster

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 10.1.16
    • Fix Version/s: 10.1.17
    • Component/s: Galera
    • Labels:
    • Environment:
      Ubuntu 16.04

      Description

      One of cluster nodes crashed and was unable to join the group with the following message:

      2016-08-01 19:50:19 140521508780288 [Note] WSREP: Read nil XID from storage engines, skipping position init
      2016-08-01 19:50:19 140521508780288 [Note] WSREP: wsrep_load(): loading provider library 'none'
      2016-08-01 19:50:19 140521508780288 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: The InnoDB memory heap is disabled
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Memory barrier is not used
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Compressed tables use zlib 1.2.8
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Using Linux native AIO
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Using SSE crc32 instructions
      2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Initializing buffer pool, size = 54.0G
      2016-08-01 19:50:24 140521508780288 [Note] InnoDB: Completed initialization of buffer pool
      2016-08-01 19:50:26 140521508780288 [Note] InnoDB: Highest supported file format is Barracuda.
      2016-08-01 19:50:27 140521508780288 [Note] InnoDB: 128 rollback segment(s) are active.
      2016-08-01 19:50:27 140521508780288 [Note] InnoDB: Waiting for purge to start
      2016-08-01 19:50:28 140521508780288 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.30-76.3 started; log sequence number 5155418059026
      2016-08-01 19:50:28 140457849882368 [Note] InnoDB: Dumping buffer pool(s) not yet started
      2016-08-01 19:50:28 140521508780288 [Note] Plugin 'FEEDBACK' is disabled.
      2016-08-01 19:50:32 140111175645440 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: The InnoDB memory heap is disabled
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Memory barrier is not used
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Compressed tables use zlib 1.2.8
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Using Linux native AIO
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Using SSE crc32 instructions
      2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Initializing buffer pool, size = 54.0G
      2016-08-01 19:50:37 140111175645440 [Note] InnoDB: Completed initialization of buffer pool
      2016-08-01 19:50:39 140111175645440 [Note] InnoDB: Highest supported file format is Barracuda.
      2016-08-01 19:50:40 140111175645440 [Note] InnoDB: 128 rollback segment(s) are active.
      2016-08-01 19:50:40 140111175645440 [Note] InnoDB: Waiting for purge to start
      2016-08-01 19:50:40 140111175645440 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.30-76.3 started; log sequence number 5155418059036
      2016-08-01 19:50:40 140111175645440 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
      2016-08-01 19:50:40 140111175645440 [Note] Plugin 'FEEDBACK' is disabled.
      2016-08-01 19:50:40 140111175645440 [Note] Recovering after a crash using /var/lib/mysql/mysql-bin
      2016-08-01 19:50:40 140111175645440 [ERROR] Binlog file '/var/lib/mysql/mysql-bin.000025' not found in binlog index, needed for recovery. Aborting.
      2016-08-01 19:50:40 140111175645440 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete (or rename) binary log and start mysqld with --tc-heuristic-recover={commit$
      2016-08-01 19:50:40 140111175645440 [ERROR] Can't init tc log
      2016-08-01 19:50:40 140111175645440 [ERROR] Aborting
      

      Removing /var/lib/mysql for a full resync did not do a trick. I also noticed that db2 (the donor) had mysql-bin logs numbered 0025-0038, however during the SST only the last log file has been transferred.

      Next step was to reinitiate the cluster, however a working node db2 did not start up again after a restart:

      Aug  1 21:18:24 db2 systemd[1]: Starting MariaDB database server...
      Aug  1 21:18:32 db2 mysqld[17595]: WSREP: Failed to recover position: '2016-08-01 21:18:25 139645985028352 [Note] /usr/sbin/mysqld (mysqld 10.1.16-MariaDB-1~xenial) starting as process 17759 ...'
      Aug  1 21:18:32 db2 systemd[1]: mariadb.service: Control process exited, code=exited status=1
      Aug  1 21:18:32 db2 systemd[1]: Failed to start MariaDB database server.
      Aug  1 21:18:32 db2 systemd[1]: mariadb.service: Unit entered failed state.
      Aug  1 21:18:32 db2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
      

      Same message is shown if I bootstrap db1, even with backups taken 2 months ago (using xtrabackup).

      When I disable wsrep_on=off, the database starts up correctly.

      MariaDB 10.1.16
      xtrabackup 2.3.5
      galera-3 25.3.15

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              nirbhay_c Nirbhay Choubey (Inactive)
              Reporter:
              mtxd Artur Čuvašov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: