Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Duplicate
-
10.1.16
-
Ubuntu 16.04
Description
One of cluster nodes crashed and was unable to join the group with the following message:
2016-08-01 19:50:19 140521508780288 [Note] WSREP: Read nil XID from storage engines, skipping position init
|
2016-08-01 19:50:19 140521508780288 [Note] WSREP: wsrep_load(): loading provider library 'none'
|
2016-08-01 19:50:19 140521508780288 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: The InnoDB memory heap is disabled
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Memory barrier is not used
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Compressed tables use zlib 1.2.8
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Using Linux native AIO
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Using SSE crc32 instructions
|
2016-08-01 19:50:19 140521508780288 [Note] InnoDB: Initializing buffer pool, size = 54.0G
|
2016-08-01 19:50:24 140521508780288 [Note] InnoDB: Completed initialization of buffer pool
|
2016-08-01 19:50:26 140521508780288 [Note] InnoDB: Highest supported file format is Barracuda.
|
2016-08-01 19:50:27 140521508780288 [Note] InnoDB: 128 rollback segment(s) are active.
|
2016-08-01 19:50:27 140521508780288 [Note] InnoDB: Waiting for purge to start
|
2016-08-01 19:50:28 140521508780288 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.30-76.3 started; log sequence number 5155418059026
|
2016-08-01 19:50:28 140457849882368 [Note] InnoDB: Dumping buffer pool(s) not yet started
|
2016-08-01 19:50:28 140521508780288 [Note] Plugin 'FEEDBACK' is disabled.
|
2016-08-01 19:50:32 140111175645440 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: The InnoDB memory heap is disabled
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Memory barrier is not used
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Compressed tables use zlib 1.2.8
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Using Linux native AIO
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Using SSE crc32 instructions
|
2016-08-01 19:50:32 140111175645440 [Note] InnoDB: Initializing buffer pool, size = 54.0G
|
2016-08-01 19:50:37 140111175645440 [Note] InnoDB: Completed initialization of buffer pool
|
2016-08-01 19:50:39 140111175645440 [Note] InnoDB: Highest supported file format is Barracuda.
|
2016-08-01 19:50:40 140111175645440 [Note] InnoDB: 128 rollback segment(s) are active.
|
2016-08-01 19:50:40 140111175645440 [Note] InnoDB: Waiting for purge to start
|
2016-08-01 19:50:40 140111175645440 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.30-76.3 started; log sequence number 5155418059036
|
2016-08-01 19:50:40 140111175645440 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
|
2016-08-01 19:50:40 140111175645440 [Note] Plugin 'FEEDBACK' is disabled.
|
2016-08-01 19:50:40 140111175645440 [Note] Recovering after a crash using /var/lib/mysql/mysql-bin
|
2016-08-01 19:50:40 140111175645440 [ERROR] Binlog file '/var/lib/mysql/mysql-bin.000025' not found in binlog index, needed for recovery. Aborting.
|
2016-08-01 19:50:40 140111175645440 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete (or rename) binary log and start mysqld with --tc-heuristic-recover={commit$
|
2016-08-01 19:50:40 140111175645440 [ERROR] Can't init tc log
|
2016-08-01 19:50:40 140111175645440 [ERROR] Aborting
|
Removing /var/lib/mysql for a full resync did not do a trick. I also noticed that db2 (the donor) had mysql-bin logs numbered 0025-0038, however during the SST only the last log file has been transferred.
Next step was to reinitiate the cluster, however a working node db2 did not start up again after a restart:
Aug 1 21:18:24 db2 systemd[1]: Starting MariaDB database server...
|
Aug 1 21:18:32 db2 mysqld[17595]: WSREP: Failed to recover position: '2016-08-01 21:18:25 139645985028352 [Note] /usr/sbin/mysqld (mysqld 10.1.16-MariaDB-1~xenial) starting as process 17759 ...'
|
Aug 1 21:18:32 db2 systemd[1]: mariadb.service: Control process exited, code=exited status=1
|
Aug 1 21:18:32 db2 systemd[1]: Failed to start MariaDB database server.
|
Aug 1 21:18:32 db2 systemd[1]: mariadb.service: Unit entered failed state.
|
Aug 1 21:18:32 db2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
|
Same message is shown if I bootstrap db1, even with backups taken 2 months ago (using xtrabackup).
When I disable wsrep_on=off, the database starts up correctly.
MariaDB 10.1.16
xtrabackup 2.3.5
galera-3 25.3.15
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-