[MDEV-15067] Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Cannot Reproduce
Affects Version/s: 10.2.12, 10.3.6
Fix Version/s: N/A
Component/s: Backup, Galera SST
Labels:
- need_feedback
Environment:
3 Nodes CentOS 7.4

Description

Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info

expected : Node is rejoining the cluster after recovering with mariabackup with option --galera-info

1.created mariadb galera cluster composed from 3 Master Nodes
it's used wsrep_sst method mariabackup ;
all Nodes are wsrep ready, connected and in Synced state
2. Perform mariabackup with option --galera-info on 1 Node
in the user guide is recommended to use this option when backing up a node in Galera Cluster.
3. stop mysql service on the backuped Node
4. remove all content of datadir
5. issue mariabackup --prepare step
6. restore data with mariabackup --copy-back --galera-info
7. start mysql service – Node fail to rejoin the cluster
note: attached logs from rejoining _node

2. Perform mariabackup with option --galera-info on 1 Node

[root@t4w3 mysql]# mariabackup -u root  --backup  --galera-info --target-dir /var/mariadb/backupg2

180125 13:05:18 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set

Using server version 10.2.12-MariaDB

mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)

mariabackup: uses posix_fadvise().

mariabackup: cd to /var/lib/mysql/

mariabackup: open files limit requested 0, set to 1024

mariabackup: using the following InnoDB configuration:

mariabackup:   innodb_data_home_dir = .

mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend

mariabackup:   innodb_log_group_home_dir = ./

2018-01-25 13:05:18 140272286804096 [Note] InnoDB: Number of pools: 1

180125 13:05:18 >> log scanned up to (1041390203)

mariabackup: Generating a list of tablespaces

2018-01-25 13:05:18 140272286804096 [Warning] InnoDB: Allocated tablespace ID 3 for mysql/gtid_slave_pos, old maximum was 0

180125 13:05:18 [01] Copying ./ibdata1 to /var/mariadb/backupg2/ibdata1

180125 13:05:19 [01]        ...done

180125 13:05:19 >> log scanned up to (1041390203)

180125 13:05:19 [01] Copying ./mysql/gtid_slave_pos.ibd to /var/mariadb/backupg2/mysql/gtid_slave_pos.ibd

180125 13:05:19 [01]        ...done

180125 13:05:19 [01] Copying ./mysql/innodb_index_stats.ibd to /var/mariadb/backupg2/mysql/innodb_index_stats.ibd

180125 13:05:19 [01]        ...done

180125 13:05:19 [01] Copying ./mysql/innodb_table_stats.ibd to /var/mariadb/backupg2/mysql/innodb_table_stats.ibd

180125 13:05:19 [01]        ...done

180125 13:05:19 [01] Copying ./boxoffice_mdb/dim_showtime.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_showtime.ibd

180125 13:05:21 [01]        ...done

180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_theater_mdc.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_theater_mdc.ibd

180125 13:05:22 [01]        ...done

180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_theater.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_theater.ibd

180125 13:05:22 [01]        ...done

180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_ticket_type.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_ticket_type.ibd

180125 13:05:22 [01]        ...done

180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_title.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_title.ibd

180125 13:05:22 [01]        ...done

180125 13:05:22 [01] Copying ./boxoffice_mdb/fact_tickets.ibd to /var/mariadb/backupg2/boxoffice_mdb/fact_tickets.ibd

180125 13:05:27 [01]        ...done

180125 13:05:29 Executing FLUSH NO_WRITE_TO_BINLOG TABLES...

180125 13:05:29 Executing FLUSH TABLES WITH READ LOCK...

180125 13:05:29 Starting to backup non-InnoDB tables and files

180125 13:05:29 [01] Copying ./mysql/roles_mapping.MYI to /var/mariadb/backupg2/mysql/roles_mapping.MYI

180125 13:05:29 [01]        ...done

180125 13:05:29 [01] Copying ./mysql/column_stats.MYD to /var/mariadb/backupg2/mysql/column_stats.MYD

180125 13:05:29 [01]        ...done

180125 13:05:29 [01] Copying ./mysql/table_stats.MYI to /var/mariadb/backupg2/mysql/table_stats.MYI

180125 13:05:29 [01]        ...done

180125 13:05:29 [01] Copying ./mysql/column_stats.MYI to /var/mariadb/backupg2/mysql/column_stats.MYI

180125 13:05:29 [01]        ...done

180125 13:05:29 [01] Copying ./mysql/table_stats.frm to /var/mariadb/backupg2/mysql/table_stats.frm

180125 13:05:29 [01]        ...done

180125 13:05:29 [01] Copying ./mysql/column_stats.frm to /var/mariadb/backupg2/mysql/column_stats.frm

180125 13:05:29 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/tables_priv.MYD to /var/mariadb/backupg2/mysql/tables_priv.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/columns_priv.MYD to /var/mariadb/backupg2/mysql/columns_priv.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/tables_priv.MYI to /var/mariadb/backupg2/mysql/tables_priv.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/columns_priv.MYI to /var/mariadb/backupg2/mysql/columns_priv.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/tables_priv.frm to /var/mariadb/backupg2/mysql/tables_priv.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/columns_priv.frm to /var/mariadb/backupg2/mysql/columns_priv.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/db.MYD to /var/mariadb/backupg2/mysql/db.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/db.MYI to /var/mariadb/backupg2/mysql/db.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/db.frm to /var/mariadb/backupg2/mysql/db.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/event.MYD to /var/mariadb/backupg2/mysql/event.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/event.MYI to /var/mariadb/backupg2/mysql/event.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/event.frm to /var/mariadb/backupg2/mysql/event.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/func.MYD to /var/mariadb/backupg2/mysql/func.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/func.MYI to /var/mariadb/backupg2/mysql/func.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/func.frm to /var/mariadb/backupg2/mysql/func.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone.MYD to /var/mariadb/backupg2/mysql/time_zone.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/general_log.CSM to /var/mariadb/backupg2/mysql/general_log.CSM

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone.MYI to /var/mariadb/backupg2/mysql/time_zone.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/general_log.CSV to /var/mariadb/backupg2/mysql/general_log.CSV

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone.frm to /var/mariadb/backupg2/mysql/time_zone.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/general_log.frm to /var/mariadb/backupg2/mysql/general_log.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.MYD to /var/mariadb/backupg2/mysql/time_zone_leap_second.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/gtid_slave_pos.frm to /var/mariadb/backupg2/mysql/gtid_slave_pos.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.MYI to /var/mariadb/backupg2/mysql/time_zone_leap_second.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.frm to /var/mariadb/backupg2/mysql/time_zone_leap_second.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_category.MYD to /var/mariadb/backupg2/mysql/help_category.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_name.MYD to /var/mariadb/backupg2/mysql/time_zone_name.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_category.MYI to /var/mariadb/backupg2/mysql/help_category.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_name.MYI to /var/mariadb/backupg2/mysql/time_zone_name.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_category.frm to /var/mariadb/backupg2/mysql/help_category.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_name.frm to /var/mariadb/backupg2/mysql/time_zone_name.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_keyword.MYD to /var/mariadb/backupg2/mysql/help_keyword.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_transition.MYD to /var/mariadb/backupg2/mysql/time_zone_transition.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_keyword.MYI to /var/mariadb/backupg2/mysql/help_keyword.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_transition.MYI to /var/mariadb/backupg2/mysql/time_zone_transition.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_keyword.frm to /var/mariadb/backupg2/mysql/help_keyword.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/time_zone_transition.frm to /var/mariadb/backupg2/mysql/time_zone_transition.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_relation.MYD to /var/mariadb/backupg2/mysql/help_relation.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/user.MYI to /var/mariadb/backupg2/mysql/user.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_relation.MYI to /var/mariadb/backupg2/mysql/help_relation.MYI

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/user.MYD to /var/mariadb/backupg2/mysql/user.MYD

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/help_relation.frm to /var/mariadb/backupg2/mysql/help_relation.frm

180125 13:05:30 [01]        ...done

180125 13:05:30 [01] Copying ./mysql/user.frm to /var/mariadb/backupg2/mysql/user.frm

180125 13:05:30 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/help_topic.MYD to /var/mariadb/backupg2/mysql/help_topic.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/help_topic.MYI to /var/mariadb/backupg2/mysql/help_topic.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/help_topic.frm to /var/mariadb/backupg2/mysql/help_topic.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/host.MYD to /var/mariadb/backupg2/mysql/host.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/host.MYI to /var/mariadb/backupg2/mysql/host.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/host.frm to /var/mariadb/backupg2/mysql/host.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/index_stats.MYD to /var/mariadb/backupg2/mysql/index_stats.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/index_stats.MYI to /var/mariadb/backupg2/mysql/index_stats.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/index_stats.frm to /var/mariadb/backupg2/mysql/index_stats.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.MYD to /var/mariadb/backupg2/mysql/time_zone_transition_type.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/innodb_index_stats.frm to /var/mariadb/backupg2/mysql/innodb_index_stats.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/proxies_priv.MYI to /var/mariadb/backupg2/mysql/proxies_priv.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/proxies_priv.frm to /var/mariadb/backupg2/mysql/proxies_priv.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/innodb_table_stats.frm to /var/mariadb/backupg2/mysql/innodb_table_stats.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/roles_mapping.MYD to /var/mariadb/backupg2/mysql/roles_mapping.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/plugin.MYD to /var/mariadb/backupg2/mysql/plugin.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/plugin.MYI to /var/mariadb/backupg2/mysql/plugin.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/plugin.frm to /var/mariadb/backupg2/mysql/plugin.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/proc.MYD to /var/mariadb/backupg2/mysql/proc.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/proc.MYI to /var/mariadb/backupg2/mysql/proc.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/proc.frm to /var/mariadb/backupg2/mysql/proc.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/procs_priv.MYD to /var/mariadb/backupg2/mysql/procs_priv.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/procs_priv.MYI to /var/mariadb/backupg2/mysql/procs_priv.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/procs_priv.frm to /var/mariadb/backupg2/mysql/procs_priv.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/proxies_priv.MYD to /var/mariadb/backupg2/mysql/proxies_priv.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/roles_mapping.frm to /var/mariadb/backupg2/mysql/roles_mapping.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/servers.MYD to /var/mariadb/backupg2/mysql/servers.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/servers.MYI to /var/mariadb/backupg2/mysql/servers.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/servers.frm to /var/mariadb/backupg2/mysql/servers.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/slow_log.CSM to /var/mariadb/backupg2/mysql/slow_log.CSM

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/slow_log.CSV to /var/mariadb/backupg2/mysql/slow_log.CSV

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/slow_log.frm to /var/mariadb/backupg2/mysql/slow_log.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/table_stats.MYD to /var/mariadb/backupg2/mysql/table_stats.MYD

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.MYI to /var/mariadb/backupg2/mysql/time_zone_transition_type.MYI

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.frm to /var/mariadb/backupg2/mysql/time_zone_transition_type.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./boxoffice_mdb/db.opt to /var/mariadb/backupg2/boxoffice_mdb/db.opt

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./boxoffice_mdb/dim_showtime.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_showtime.frm

180125 13:05:31 [01]        ...done

180125 13:05:31 [01] Copying ./boxoffice_mdb/dim_theater_mdc.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_theater_mdc.frm

180125 13:05:31 [01]        ...done

180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_theater.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_theater.frm

180125 13:05:32 [01]        ...done

180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_ticket_type.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_ticket_type.frm

180125 13:05:32 [01]        ...done

180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_title.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_title.frm

180125 13:05:32 [01]        ...done

180125 13:05:32 [01] Copying ./boxoffice_mdb/fact_tickets.frm to /var/mariadb/backupg2/boxoffice_mdb/fact_tickets.frm

180125 13:05:32 [01]        ...done

180125 13:05:32 [01] Copying ./performance_schema/db.opt to /var/mariadb/backupg2/performance_schema/db.opt

180125 13:05:32 [01]        ...done

180125 13:05:32 [01] Copying ./test/db.opt to /var/mariadb/backupg2/test/db.opt

180125 13:05:32 [01]        ...done

180125 13:05:32 Finished backing up non-InnoDB tables and files

180125 13:05:32 [01] Copying aria_log.00000001 to /var/mariadb/backupg2/aria_log.00000001

180125 13:05:32 [01]        ...done

180125 13:05:32 [01] Copying aria_log_control to /var/mariadb/backupg2/aria_log_control

180125 13:05:32 [01]        ...done

180125 13:05:32 [00] Writing xtrabackup_galera_info

180125 13:05:32 [00]        ...done

180125 13:05:32 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...

mariabackup: The latest check point (for incremental): '1041390194'

mariabackup: Stopping log copying thread.

180125 13:05:32 >> log scanned up to (1041390203)

180125 13:05:32 Executing UNLOCK TABLES

180125 13:05:32 All tables unlocked

180125 13:05:32 [00] Copying ib_buffer_pool to /var/mariadb/backupg2/ib_buffer_pool

180125 13:05:32 [00]        ...done

180125 13:05:32 Backup created in directory '/var/mariadb/backupg2/'

180125 13:05:32 [00] Writing backup-my.cnf

180125 13:05:32 [00]        ...done

180125 13:05:32 [00] Writing xtrabackup_info

180125 13:05:32 [00]        ...done

mariabackup: Redo log (from LSN 1041390194 to 1041390203) was copied.

180125 13:05:32 completed OK!

[root@t4w3 mysql]# ls  /var/mariadb/backupg2

aria_log.00000001  backup-my.cnf  ib_buffer_pool  ib_logfile0  performance_schema  xtrabackup_checkpoints  xtrabackup_info

aria_log_control   boxoffice_mdb  ibdata1         mysql        test                xtrabackup_galera_info

[root@t4w3 mysql]#

3. stop mysql service on the backuped Node

[root@t4w3 mysql]# service mysql stop

Stopping mysql (via systemctl):                            [  OK  ]

4. remove all content of datadir

[root@t4w3 mysql]# rm -rf /var/lib/mysql/*

5. issue mariabackup --prepare step

[root@t4w3 mysql]#

[root@t4w3 mysql]# mariabackup -u prepare  --backup  --galera-info --target-dir /var/mariadb/backupg2

180125 13:07:48 Connecting to MySQL server host: localhost, user: prepare, password: not set, port: not set, socket: not set

Failed to connect to MySQL server: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory").

[root@t4w3 mysql]# mariabackup -u  root --prepare    --galera-info --target-dir /var/mariadb/backupg2

mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)

mariabackup: cd to /var/mariadb/backupg2/

mariabackup: This target seems to be not prepared yet.

mariabackup: using the following InnoDB configuration for recovery:

mariabackup:   innodb_data_home_dir = .

mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend

mariabackup:   innodb_log_group_home_dir = .

mariabackup: Starting InnoDB instance for recovery.

mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Uses event mutexes

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Compressed tables use zlib 1.2.7

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Number of pools: 1

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Using generic crc32 instructions

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Completed initialization of buffer pool

2018-01-25 13:08:08 140279319353088 [Note] InnoDB: page_cleaner coordinator priority: -20

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Highest supported file format is Barracuda.

2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1041390194

180125 13:08:09 completed OK!

[root@t4w3 mysql]#

6. restore data with mariabackup --copy-back --galera-info

[root@t4w3 mysql]# mariabackup -u  root --copy-back    --galera-info --target-dir /var/mariadb/backupg2

mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)

180125 13:08:26 [01] Copying ibdata1 to /var/lib/mysql/ibdata1

180125 13:08:27 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/gtid_slave_pos.ibd to /var/lib/mysql/mysql/gtid_slave_pos.ibd

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/innodb_index_stats.ibd to /var/lib/mysql/mysql/innodb_index_stats.ibd

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/innodb_table_stats.ibd to /var/lib/mysql/mysql/innodb_table_stats.ibd

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/roles_mapping.MYI to /var/lib/mysql/mysql/roles_mapping.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/column_stats.MYD to /var/lib/mysql/mysql/column_stats.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/table_stats.MYI to /var/lib/mysql/mysql/table_stats.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/column_stats.MYI to /var/lib/mysql/mysql/column_stats.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/table_stats.frm to /var/lib/mysql/mysql/table_stats.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/column_stats.frm to /var/lib/mysql/mysql/column_stats.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/tables_priv.MYD to /var/lib/mysql/mysql/tables_priv.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/columns_priv.MYD to /var/lib/mysql/mysql/columns_priv.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/tables_priv.MYI to /var/lib/mysql/mysql/tables_priv.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/columns_priv.MYI to /var/lib/mysql/mysql/columns_priv.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/tables_priv.frm to /var/lib/mysql/mysql/tables_priv.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/columns_priv.frm to /var/lib/mysql/mysql/columns_priv.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/db.MYD to /var/lib/mysql/mysql/db.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/db.MYI to /var/lib/mysql/mysql/db.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/db.frm to /var/lib/mysql/mysql/db.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/event.MYD to /var/lib/mysql/mysql/event.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/event.MYI to /var/lib/mysql/mysql/event.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/event.frm to /var/lib/mysql/mysql/event.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/func.MYD to /var/lib/mysql/mysql/func.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/func.MYI to /var/lib/mysql/mysql/func.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/func.frm to /var/lib/mysql/mysql/func.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/time_zone.MYD to /var/lib/mysql/mysql/time_zone.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/general_log.CSM to /var/lib/mysql/mysql/general_log.CSM

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/time_zone.MYI to /var/lib/mysql/mysql/time_zone.MYI

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/general_log.CSV to /var/lib/mysql/mysql/general_log.CSV

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/time_zone.frm to /var/lib/mysql/mysql/time_zone.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/general_log.frm to /var/lib/mysql/mysql/general_log.frm

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/time_zone_leap_second.MYD to /var/lib/mysql/mysql/time_zone_leap_second.MYD

180125 13:08:28 [01]        ...done

180125 13:08:28 [01] Copying ./mysql/gtid_slave_pos.frm to /var/lib/mysql/mysql/gtid_slave_pos.frm

180125 13:08:28 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_leap_second.MYI to /var/lib/mysql/mysql/time_zone_leap_second.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_leap_second.frm to /var/lib/mysql/mysql/time_zone_leap_second.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_category.MYD to /var/lib/mysql/mysql/help_category.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_name.MYD to /var/lib/mysql/mysql/time_zone_name.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_category.MYI to /var/lib/mysql/mysql/help_category.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_name.MYI to /var/lib/mysql/mysql/time_zone_name.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_category.frm to /var/lib/mysql/mysql/help_category.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_name.frm to /var/lib/mysql/mysql/time_zone_name.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_keyword.MYD to /var/lib/mysql/mysql/help_keyword.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_transition.MYD to /var/lib/mysql/mysql/time_zone_transition.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_keyword.MYI to /var/lib/mysql/mysql/help_keyword.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_transition.MYI to /var/lib/mysql/mysql/time_zone_transition.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_keyword.frm to /var/lib/mysql/mysql/help_keyword.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/time_zone_transition.frm to /var/lib/mysql/mysql/time_zone_transition.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_relation.MYD to /var/lib/mysql/mysql/help_relation.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/user.MYI to /var/lib/mysql/mysql/user.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_relation.MYI to /var/lib/mysql/mysql/help_relation.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/user.MYD to /var/lib/mysql/mysql/user.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_relation.frm to /var/lib/mysql/mysql/help_relation.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/user.frm to /var/lib/mysql/mysql/user.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_topic.MYD to /var/lib/mysql/mysql/help_topic.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_topic.MYI to /var/lib/mysql/mysql/help_topic.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/help_topic.frm to /var/lib/mysql/mysql/help_topic.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/host.MYD to /var/lib/mysql/mysql/host.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/host.MYI to /var/lib/mysql/mysql/host.MYI

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/host.frm to /var/lib/mysql/mysql/host.frm

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/index_stats.MYD to /var/lib/mysql/mysql/index_stats.MYD

180125 13:08:29 [01]        ...done

180125 13:08:29 [01] Copying ./mysql/index_stats.MYI to /var/lib/mysql/mysql/index_stats.MYI

180125 13:08:29 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/index_stats.frm to /var/lib/mysql/mysql/index_stats.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.MYD to /var/lib/mysql/mysql/time_zone_transition_type.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/innodb_index_stats.frm to /var/lib/mysql/mysql/innodb_index_stats.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/proxies_priv.MYI to /var/lib/mysql/mysql/proxies_priv.MYI

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/proxies_priv.frm to /var/lib/mysql/mysql/proxies_priv.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/innodb_table_stats.frm to /var/lib/mysql/mysql/innodb_table_stats.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/roles_mapping.MYD to /var/lib/mysql/mysql/roles_mapping.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/plugin.MYD to /var/lib/mysql/mysql/plugin.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/plugin.MYI to /var/lib/mysql/mysql/plugin.MYI

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/plugin.frm to /var/lib/mysql/mysql/plugin.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/proc.MYD to /var/lib/mysql/mysql/proc.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/proc.MYI to /var/lib/mysql/mysql/proc.MYI

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/proc.frm to /var/lib/mysql/mysql/proc.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/procs_priv.MYD to /var/lib/mysql/mysql/procs_priv.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/procs_priv.MYI to /var/lib/mysql/mysql/procs_priv.MYI

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/procs_priv.frm to /var/lib/mysql/mysql/procs_priv.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/proxies_priv.MYD to /var/lib/mysql/mysql/proxies_priv.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/roles_mapping.frm to /var/lib/mysql/mysql/roles_mapping.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/servers.MYD to /var/lib/mysql/mysql/servers.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/servers.MYI to /var/lib/mysql/mysql/servers.MYI

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/servers.frm to /var/lib/mysql/mysql/servers.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/slow_log.CSM to /var/lib/mysql/mysql/slow_log.CSM

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/slow_log.CSV to /var/lib/mysql/mysql/slow_log.CSV

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/slow_log.frm to /var/lib/mysql/mysql/slow_log.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/table_stats.MYD to /var/lib/mysql/mysql/table_stats.MYD

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.MYI to /var/lib/mysql/mysql/time_zone_transition_type.MYI

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.frm to /var/lib/mysql/mysql/time_zone_transition_type.frm

180125 13:08:30 [01]        ...done

180125 13:08:30 [01] Copying ./boxoffice_mdb/dim_showtime.ibd to /var/lib/mysql/boxoffice_mdb/dim_showtime.ibd

180125 13:08:31 [01]        ...done

180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_theater_mdc.ibd to /var/lib/mysql/boxoffice_mdb/dim_theater_mdc.ibd

180125 13:08:32 [01]        ...done

180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_theater.ibd to /var/lib/mysql/boxoffice_mdb/dim_theater.ibd

180125 13:08:32 [01]        ...done

180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_ticket_type.ibd to /var/lib/mysql/boxoffice_mdb/dim_ticket_type.ibd

180125 13:08:32 [01]        ...done

180125 13:08:33 [01] Copying ./boxoffice_mdb/dim_title.ibd to /var/lib/mysql/boxoffice_mdb/dim_title.ibd

180125 13:08:33 [01]        ...done

180125 13:08:33 [01] Copying ./boxoffice_mdb/fact_tickets.ibd to /var/lib/mysql/boxoffice_mdb/fact_tickets.ibd

180125 13:08:37 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/db.opt to /var/lib/mysql/boxoffice_mdb/db.opt

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_showtime.frm to /var/lib/mysql/boxoffice_mdb/dim_showtime.frm

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_theater_mdc.frm to /var/lib/mysql/boxoffice_mdb/dim_theater_mdc.frm

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_theater.frm to /var/lib/mysql/boxoffice_mdb/dim_theater.frm

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_ticket_type.frm to /var/lib/mysql/boxoffice_mdb/dim_ticket_type.frm

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_title.frm to /var/lib/mysql/boxoffice_mdb/dim_title.frm

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./boxoffice_mdb/fact_tickets.frm to /var/lib/mysql/boxoffice_mdb/fact_tickets.frm

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./performance_schema/db.opt to /var/lib/mysql/performance_schema/db.opt

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./test/db.opt to /var/lib/mysql/test/db.opt

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./aria_log.00000001 to /var/lib/mysql/aria_log.00000001

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./aria_log_control to /var/lib/mysql/aria_log_control

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./xtrabackup_galera_info to /var/lib/mysql/xtrabackup_galera_info

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./ib_buffer_pool to /var/lib/mysql/ib_buffer_pool

180125 13:08:39 [01]        ...done

180125 13:08:39 [01] Copying ./xtrabackup_info to /var/lib/mysql/xtrabackup_info

180125 13:08:39 [01]        ...done

180125 13:08:39 completed OK!

7. start mysql service – Node fail to rejoin the cluster

[root@t4w3 mysql]# service mysql start

Starting mysql (via systemctl):  Job for mariadb.service failed because a fatal signal was delivered to the control process. See "systemctl status mariadb.service" and "journalctl -xe" for details.

                                                           [FAILED]

mysql is restarting endlessly but Node fails to rejoin cluster

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

galera_info.sh
2018-09-21 07:54
2 kB
Seppo Jaakola
innobackup.move.log
2018-06-18 07:46
0.3 kB
Seppo Jaakola
innobackup.prepare.log
2018-06-18 07:46
2 kB
Seppo Jaakola
logs_from_rejoining _node.txt
2018-01-25 12:30
26 kB
Zdravelina Sokolovska

Issue Links

relates to

MDEV-15921 mariabackup hangs at FLUSH TABLES WITH READ LOCK when Node is wsrep_desync

Closed

MDEV-15933 Cannot resume Node SYNCED state when wsrep_desync is done after FTWRL

Closed

MDEV-16062 Galera Node is rejoined with SST after backup - restore with mariabackup

Closed

Activity

People

Assignee:: Seppo Jaakola

Reporter:: Zdravelina Sokolovska (Inactive)

Votes:: 2 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2018-01-25 12:45

Updated:: 2024-07-08 00:22

Resolved:: 2020-06-19 19:50

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.