[MDEV-15067] Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info Created: 2018-01-25  Updated: 2020-06-19  Resolved: 2020-06-19

Status: Closed
Project: MariaDB Server
Component/s: Backup, Galera SST
Affects Version/s: 10.2.12, 10.3.6
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Zdravelina Sokolovska (Inactive) Assignee: Seppo Jaakola
Resolution: Cannot Reproduce Votes: 2
Labels: need_feedback
Environment:

3 Nodes CentOS 7.4

Attachments: File galera_info.sh     File innobackup.move.log     File innobackup.prepare.log     Text File logs_from_rejoining _node.txt    
Issue Links:
Relates
relates to MDEV-15921 mariabackup hangs at FLUSH TABLES WI... Closed
relates to MDEV-15933 Cannot resume Node SYNCED state when ... Closed
relates to MDEV-16062 Galera Node is rejoined with SST afte... Closed

 Description   

Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info

expected : Node is rejoining the cluster after recovering with mariabackup with option --galera-info

1.created mariadb galera cluster composed from 3 Master Nodes
it's used wsrep_sst method mariabackup ;
all Nodes are wsrep ready, connected and in Synced state
2. Perform mariabackup with option --galera-info on 1 Node
in the user guide is recommended to use this option when backing up a node in Galera Cluster.
3. stop mysql service on the backuped Node
4. remove all content of datadir
5. issue mariabackup --prepare step
6. restore data with mariabackup --copy-back --galera-info
7. start mysql service – Node fail to rejoin the cluster
note: attached logs from rejoining _node

2. Perform mariabackup with option --galera-info on 1 Node

[root@t4w3 mysql]# mariabackup -u root  --backup  --galera-info --target-dir /var/mariadb/backupg2
 
180125 13:05:18 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set
 
Using server version 10.2.12-MariaDB
 
mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)
 
mariabackup: uses posix_fadvise().
 
mariabackup: cd to /var/lib/mysql/
 
mariabackup: open files limit requested 0, set to 1024
 
mariabackup: using the following InnoDB configuration:
 
mariabackup:   innodb_data_home_dir = .
 
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
 
mariabackup:   innodb_log_group_home_dir = ./
 
2018-01-25 13:05:18 140272286804096 [Note] InnoDB: Number of pools: 1
 
180125 13:05:18 >> log scanned up to (1041390203)
 
mariabackup: Generating a list of tablespaces
 
2018-01-25 13:05:18 140272286804096 [Warning] InnoDB: Allocated tablespace ID 3 for mysql/gtid_slave_pos, old maximum was 0
 
180125 13:05:18 [01] Copying ./ibdata1 to /var/mariadb/backupg2/ibdata1
 
180125 13:05:19 [01]        ...done
 
180125 13:05:19 >> log scanned up to (1041390203)
 
180125 13:05:19 [01] Copying ./mysql/gtid_slave_pos.ibd to /var/mariadb/backupg2/mysql/gtid_slave_pos.ibd
 
180125 13:05:19 [01]        ...done
 
180125 13:05:19 [01] Copying ./mysql/innodb_index_stats.ibd to /var/mariadb/backupg2/mysql/innodb_index_stats.ibd
 
180125 13:05:19 [01]        ...done
 
180125 13:05:19 [01] Copying ./mysql/innodb_table_stats.ibd to /var/mariadb/backupg2/mysql/innodb_table_stats.ibd
 
180125 13:05:19 [01]        ...done
 
180125 13:05:19 [01] Copying ./boxoffice_mdb/dim_showtime.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_showtime.ibd
 
180125 13:05:21 [01]        ...done
 
180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_theater_mdc.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_theater_mdc.ibd
 
180125 13:05:22 [01]        ...done
 
180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_theater.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_theater.ibd
 
180125 13:05:22 [01]        ...done
 
180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_ticket_type.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_ticket_type.ibd
 
180125 13:05:22 [01]        ...done
 
180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_title.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_title.ibd
 
180125 13:05:22 [01]        ...done
 
180125 13:05:22 [01] Copying ./boxoffice_mdb/fact_tickets.ibd to /var/mariadb/backupg2/boxoffice_mdb/fact_tickets.ibd
 
180125 13:05:27 [01]        ...done
 
180125 13:05:29 Executing FLUSH NO_WRITE_TO_BINLOG TABLES...
 
180125 13:05:29 Executing FLUSH TABLES WITH READ LOCK...
 
180125 13:05:29 Starting to backup non-InnoDB tables and files
 
180125 13:05:29 [01] Copying ./mysql/roles_mapping.MYI to /var/mariadb/backupg2/mysql/roles_mapping.MYI
 
180125 13:05:29 [01]        ...done
 
180125 13:05:29 [01] Copying ./mysql/column_stats.MYD to /var/mariadb/backupg2/mysql/column_stats.MYD
 
180125 13:05:29 [01]        ...done
 
180125 13:05:29 [01] Copying ./mysql/table_stats.MYI to /var/mariadb/backupg2/mysql/table_stats.MYI
 
180125 13:05:29 [01]        ...done
 
180125 13:05:29 [01] Copying ./mysql/column_stats.MYI to /var/mariadb/backupg2/mysql/column_stats.MYI
 
180125 13:05:29 [01]        ...done
 
180125 13:05:29 [01] Copying ./mysql/table_stats.frm to /var/mariadb/backupg2/mysql/table_stats.frm
 
180125 13:05:29 [01]        ...done
 
180125 13:05:29 [01] Copying ./mysql/column_stats.frm to /var/mariadb/backupg2/mysql/column_stats.frm
 
180125 13:05:29 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/tables_priv.MYD to /var/mariadb/backupg2/mysql/tables_priv.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/columns_priv.MYD to /var/mariadb/backupg2/mysql/columns_priv.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/tables_priv.MYI to /var/mariadb/backupg2/mysql/tables_priv.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/columns_priv.MYI to /var/mariadb/backupg2/mysql/columns_priv.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/tables_priv.frm to /var/mariadb/backupg2/mysql/tables_priv.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/columns_priv.frm to /var/mariadb/backupg2/mysql/columns_priv.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/db.MYD to /var/mariadb/backupg2/mysql/db.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/db.MYI to /var/mariadb/backupg2/mysql/db.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/db.frm to /var/mariadb/backupg2/mysql/db.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/event.MYD to /var/mariadb/backupg2/mysql/event.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/event.MYI to /var/mariadb/backupg2/mysql/event.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/event.frm to /var/mariadb/backupg2/mysql/event.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/func.MYD to /var/mariadb/backupg2/mysql/func.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/func.MYI to /var/mariadb/backupg2/mysql/func.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/func.frm to /var/mariadb/backupg2/mysql/func.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone.MYD to /var/mariadb/backupg2/mysql/time_zone.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/general_log.CSM to /var/mariadb/backupg2/mysql/general_log.CSM
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone.MYI to /var/mariadb/backupg2/mysql/time_zone.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/general_log.CSV to /var/mariadb/backupg2/mysql/general_log.CSV
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone.frm to /var/mariadb/backupg2/mysql/time_zone.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/general_log.frm to /var/mariadb/backupg2/mysql/general_log.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.MYD to /var/mariadb/backupg2/mysql/time_zone_leap_second.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/gtid_slave_pos.frm to /var/mariadb/backupg2/mysql/gtid_slave_pos.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.MYI to /var/mariadb/backupg2/mysql/time_zone_leap_second.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.frm to /var/mariadb/backupg2/mysql/time_zone_leap_second.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_category.MYD to /var/mariadb/backupg2/mysql/help_category.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_name.MYD to /var/mariadb/backupg2/mysql/time_zone_name.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_category.MYI to /var/mariadb/backupg2/mysql/help_category.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_name.MYI to /var/mariadb/backupg2/mysql/time_zone_name.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_category.frm to /var/mariadb/backupg2/mysql/help_category.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_name.frm to /var/mariadb/backupg2/mysql/time_zone_name.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_keyword.MYD to /var/mariadb/backupg2/mysql/help_keyword.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_transition.MYD to /var/mariadb/backupg2/mysql/time_zone_transition.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_keyword.MYI to /var/mariadb/backupg2/mysql/help_keyword.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_transition.MYI to /var/mariadb/backupg2/mysql/time_zone_transition.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_keyword.frm to /var/mariadb/backupg2/mysql/help_keyword.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/time_zone_transition.frm to /var/mariadb/backupg2/mysql/time_zone_transition.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_relation.MYD to /var/mariadb/backupg2/mysql/help_relation.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/user.MYI to /var/mariadb/backupg2/mysql/user.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_relation.MYI to /var/mariadb/backupg2/mysql/help_relation.MYI
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/user.MYD to /var/mariadb/backupg2/mysql/user.MYD
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/help_relation.frm to /var/mariadb/backupg2/mysql/help_relation.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:30 [01] Copying ./mysql/user.frm to /var/mariadb/backupg2/mysql/user.frm
 
180125 13:05:30 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/help_topic.MYD to /var/mariadb/backupg2/mysql/help_topic.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/help_topic.MYI to /var/mariadb/backupg2/mysql/help_topic.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/help_topic.frm to /var/mariadb/backupg2/mysql/help_topic.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/host.MYD to /var/mariadb/backupg2/mysql/host.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/host.MYI to /var/mariadb/backupg2/mysql/host.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/host.frm to /var/mariadb/backupg2/mysql/host.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/index_stats.MYD to /var/mariadb/backupg2/mysql/index_stats.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/index_stats.MYI to /var/mariadb/backupg2/mysql/index_stats.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/index_stats.frm to /var/mariadb/backupg2/mysql/index_stats.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.MYD to /var/mariadb/backupg2/mysql/time_zone_transition_type.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/innodb_index_stats.frm to /var/mariadb/backupg2/mysql/innodb_index_stats.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/proxies_priv.MYI to /var/mariadb/backupg2/mysql/proxies_priv.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/proxies_priv.frm to /var/mariadb/backupg2/mysql/proxies_priv.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/innodb_table_stats.frm to /var/mariadb/backupg2/mysql/innodb_table_stats.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/roles_mapping.MYD to /var/mariadb/backupg2/mysql/roles_mapping.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/plugin.MYD to /var/mariadb/backupg2/mysql/plugin.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/plugin.MYI to /var/mariadb/backupg2/mysql/plugin.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/plugin.frm to /var/mariadb/backupg2/mysql/plugin.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/proc.MYD to /var/mariadb/backupg2/mysql/proc.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/proc.MYI to /var/mariadb/backupg2/mysql/proc.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/proc.frm to /var/mariadb/backupg2/mysql/proc.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/procs_priv.MYD to /var/mariadb/backupg2/mysql/procs_priv.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/procs_priv.MYI to /var/mariadb/backupg2/mysql/procs_priv.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/procs_priv.frm to /var/mariadb/backupg2/mysql/procs_priv.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/proxies_priv.MYD to /var/mariadb/backupg2/mysql/proxies_priv.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/roles_mapping.frm to /var/mariadb/backupg2/mysql/roles_mapping.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/servers.MYD to /var/mariadb/backupg2/mysql/servers.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/servers.MYI to /var/mariadb/backupg2/mysql/servers.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/servers.frm to /var/mariadb/backupg2/mysql/servers.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/slow_log.CSM to /var/mariadb/backupg2/mysql/slow_log.CSM
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/slow_log.CSV to /var/mariadb/backupg2/mysql/slow_log.CSV
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/slow_log.frm to /var/mariadb/backupg2/mysql/slow_log.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/table_stats.MYD to /var/mariadb/backupg2/mysql/table_stats.MYD
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.MYI to /var/mariadb/backupg2/mysql/time_zone_transition_type.MYI
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.frm to /var/mariadb/backupg2/mysql/time_zone_transition_type.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./boxoffice_mdb/db.opt to /var/mariadb/backupg2/boxoffice_mdb/db.opt
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./boxoffice_mdb/dim_showtime.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_showtime.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:31 [01] Copying ./boxoffice_mdb/dim_theater_mdc.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_theater_mdc.frm
 
180125 13:05:31 [01]        ...done
 
180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_theater.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_theater.frm
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_ticket_type.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_ticket_type.frm
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_title.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_title.frm
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [01] Copying ./boxoffice_mdb/fact_tickets.frm to /var/mariadb/backupg2/boxoffice_mdb/fact_tickets.frm
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [01] Copying ./performance_schema/db.opt to /var/mariadb/backupg2/performance_schema/db.opt
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [01] Copying ./test/db.opt to /var/mariadb/backupg2/test/db.opt
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 Finished backing up non-InnoDB tables and files
 
180125 13:05:32 [01] Copying aria_log.00000001 to /var/mariadb/backupg2/aria_log.00000001
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [01] Copying aria_log_control to /var/mariadb/backupg2/aria_log_control
 
180125 13:05:32 [01]        ...done
 
180125 13:05:32 [00] Writing xtrabackup_galera_info
 
180125 13:05:32 [00]        ...done
 
180125 13:05:32 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
 
mariabackup: The latest check point (for incremental): '1041390194'
 
mariabackup: Stopping log copying thread.
 
 
 
180125 13:05:32 >> log scanned up to (1041390203)
 
180125 13:05:32 Executing UNLOCK TABLES
 
180125 13:05:32 All tables unlocked
 
180125 13:05:32 [00] Copying ib_buffer_pool to /var/mariadb/backupg2/ib_buffer_pool
 
180125 13:05:32 [00]        ...done
 
180125 13:05:32 Backup created in directory '/var/mariadb/backupg2/'
 
180125 13:05:32 [00] Writing backup-my.cnf
 
180125 13:05:32 [00]        ...done
 
180125 13:05:32 [00] Writing xtrabackup_info
 
180125 13:05:32 [00]        ...done
 
mariabackup: Redo log (from LSN 1041390194 to 1041390203) was copied.
 
180125 13:05:32 completed OK!


[root@t4w3 mysql]# ls  /var/mariadb/backupg2
 
aria_log.00000001  backup-my.cnf  ib_buffer_pool  ib_logfile0  performance_schema  xtrabackup_checkpoints  xtrabackup_info
 
aria_log_control   boxoffice_mdb  ibdata1         mysql        test                xtrabackup_galera_info
 
[root@t4w3 mysql]#

3. stop mysql service on the backuped Node

[root@t4w3 mysql]# service mysql stop
 
Stopping mysql (via systemctl):                            [  OK  ]

4. remove all content of datadir

[root@t4w3 mysql]# rm -rf /var/lib/mysql/*

5. issue mariabackup --prepare step 

[root@t4w3 mysql]#
 
[root@t4w3 mysql]# mariabackup -u prepare  --backup  --galera-info --target-dir /var/mariadb/backupg2
 
180125 13:07:48 Connecting to MySQL server host: localhost, user: prepare, password: not set, port: not set, socket: not set
 
Failed to connect to MySQL server: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory").
 
[root@t4w3 mysql]# mariabackup -u  root --prepare    --galera-info --target-dir /var/mariadb/backupg2
 
mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)
 
mariabackup: cd to /var/mariadb/backupg2/
 
mariabackup: This target seems to be not prepared yet.
 
mariabackup: using the following InnoDB configuration for recovery:
 
mariabackup:   innodb_data_home_dir = .
 
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
 
mariabackup:   innodb_log_group_home_dir = .
 
mariabackup: Starting InnoDB instance for recovery.
 
mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Uses event mutexes
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Number of pools: 1
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Using generic crc32 instructions
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Completed initialization of buffer pool
 
2018-01-25 13:08:08 140279319353088 [Note] InnoDB: page_cleaner coordinator priority: -20
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Highest supported file format is Barracuda.
 
2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1041390194
 
180125 13:08:09 completed OK!
 
[root@t4w3 mysql]#

6. restore data with mariabackup --copy-back --galera-info

[root@t4w3 mysql]# mariabackup -u  root --copy-back    --galera-info --target-dir /var/mariadb/backupg2
 
mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)
 
180125 13:08:26 [01] Copying ibdata1 to /var/lib/mysql/ibdata1
 
180125 13:08:27 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/gtid_slave_pos.ibd to /var/lib/mysql/mysql/gtid_slave_pos.ibd
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/innodb_index_stats.ibd to /var/lib/mysql/mysql/innodb_index_stats.ibd
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/innodb_table_stats.ibd to /var/lib/mysql/mysql/innodb_table_stats.ibd
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/roles_mapping.MYI to /var/lib/mysql/mysql/roles_mapping.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/column_stats.MYD to /var/lib/mysql/mysql/column_stats.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/table_stats.MYI to /var/lib/mysql/mysql/table_stats.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/column_stats.MYI to /var/lib/mysql/mysql/column_stats.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/table_stats.frm to /var/lib/mysql/mysql/table_stats.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/column_stats.frm to /var/lib/mysql/mysql/column_stats.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/tables_priv.MYD to /var/lib/mysql/mysql/tables_priv.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/columns_priv.MYD to /var/lib/mysql/mysql/columns_priv.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/tables_priv.MYI to /var/lib/mysql/mysql/tables_priv.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/columns_priv.MYI to /var/lib/mysql/mysql/columns_priv.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/tables_priv.frm to /var/lib/mysql/mysql/tables_priv.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/columns_priv.frm to /var/lib/mysql/mysql/columns_priv.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/db.MYD to /var/lib/mysql/mysql/db.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/db.MYI to /var/lib/mysql/mysql/db.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/db.frm to /var/lib/mysql/mysql/db.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/event.MYD to /var/lib/mysql/mysql/event.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/event.MYI to /var/lib/mysql/mysql/event.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/event.frm to /var/lib/mysql/mysql/event.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/func.MYD to /var/lib/mysql/mysql/func.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/func.MYI to /var/lib/mysql/mysql/func.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/func.frm to /var/lib/mysql/mysql/func.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/time_zone.MYD to /var/lib/mysql/mysql/time_zone.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/general_log.CSM to /var/lib/mysql/mysql/general_log.CSM
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/time_zone.MYI to /var/lib/mysql/mysql/time_zone.MYI
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/general_log.CSV to /var/lib/mysql/mysql/general_log.CSV
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/time_zone.frm to /var/lib/mysql/mysql/time_zone.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/general_log.frm to /var/lib/mysql/mysql/general_log.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/time_zone_leap_second.MYD to /var/lib/mysql/mysql/time_zone_leap_second.MYD
 
180125 13:08:28 [01]        ...done
 
180125 13:08:28 [01] Copying ./mysql/gtid_slave_pos.frm to /var/lib/mysql/mysql/gtid_slave_pos.frm
 
180125 13:08:28 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_leap_second.MYI to /var/lib/mysql/mysql/time_zone_leap_second.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_leap_second.frm to /var/lib/mysql/mysql/time_zone_leap_second.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_category.MYD to /var/lib/mysql/mysql/help_category.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_name.MYD to /var/lib/mysql/mysql/time_zone_name.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_category.MYI to /var/lib/mysql/mysql/help_category.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_name.MYI to /var/lib/mysql/mysql/time_zone_name.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_category.frm to /var/lib/mysql/mysql/help_category.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_name.frm to /var/lib/mysql/mysql/time_zone_name.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_keyword.MYD to /var/lib/mysql/mysql/help_keyword.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_transition.MYD to /var/lib/mysql/mysql/time_zone_transition.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_keyword.MYI to /var/lib/mysql/mysql/help_keyword.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_transition.MYI to /var/lib/mysql/mysql/time_zone_transition.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_keyword.frm to /var/lib/mysql/mysql/help_keyword.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/time_zone_transition.frm to /var/lib/mysql/mysql/time_zone_transition.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_relation.MYD to /var/lib/mysql/mysql/help_relation.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/user.MYI to /var/lib/mysql/mysql/user.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_relation.MYI to /var/lib/mysql/mysql/help_relation.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/user.MYD to /var/lib/mysql/mysql/user.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_relation.frm to /var/lib/mysql/mysql/help_relation.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/user.frm to /var/lib/mysql/mysql/user.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_topic.MYD to /var/lib/mysql/mysql/help_topic.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_topic.MYI to /var/lib/mysql/mysql/help_topic.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/help_topic.frm to /var/lib/mysql/mysql/help_topic.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/host.MYD to /var/lib/mysql/mysql/host.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/host.MYI to /var/lib/mysql/mysql/host.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/host.frm to /var/lib/mysql/mysql/host.frm
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/index_stats.MYD to /var/lib/mysql/mysql/index_stats.MYD
 
180125 13:08:29 [01]        ...done
 
180125 13:08:29 [01] Copying ./mysql/index_stats.MYI to /var/lib/mysql/mysql/index_stats.MYI
 
180125 13:08:29 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/index_stats.frm to /var/lib/mysql/mysql/index_stats.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.MYD to /var/lib/mysql/mysql/time_zone_transition_type.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/innodb_index_stats.frm to /var/lib/mysql/mysql/innodb_index_stats.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/proxies_priv.MYI to /var/lib/mysql/mysql/proxies_priv.MYI
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/proxies_priv.frm to /var/lib/mysql/mysql/proxies_priv.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/innodb_table_stats.frm to /var/lib/mysql/mysql/innodb_table_stats.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/roles_mapping.MYD to /var/lib/mysql/mysql/roles_mapping.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/plugin.MYD to /var/lib/mysql/mysql/plugin.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/plugin.MYI to /var/lib/mysql/mysql/plugin.MYI
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/plugin.frm to /var/lib/mysql/mysql/plugin.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/proc.MYD to /var/lib/mysql/mysql/proc.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/proc.MYI to /var/lib/mysql/mysql/proc.MYI
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/proc.frm to /var/lib/mysql/mysql/proc.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/procs_priv.MYD to /var/lib/mysql/mysql/procs_priv.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/procs_priv.MYI to /var/lib/mysql/mysql/procs_priv.MYI
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/procs_priv.frm to /var/lib/mysql/mysql/procs_priv.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/proxies_priv.MYD to /var/lib/mysql/mysql/proxies_priv.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/roles_mapping.frm to /var/lib/mysql/mysql/roles_mapping.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/servers.MYD to /var/lib/mysql/mysql/servers.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/servers.MYI to /var/lib/mysql/mysql/servers.MYI
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/servers.frm to /var/lib/mysql/mysql/servers.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/slow_log.CSM to /var/lib/mysql/mysql/slow_log.CSM
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/slow_log.CSV to /var/lib/mysql/mysql/slow_log.CSV
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/slow_log.frm to /var/lib/mysql/mysql/slow_log.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/table_stats.MYD to /var/lib/mysql/mysql/table_stats.MYD
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.MYI to /var/lib/mysql/mysql/time_zone_transition_type.MYI
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.frm to /var/lib/mysql/mysql/time_zone_transition_type.frm
 
180125 13:08:30 [01]        ...done
 
180125 13:08:30 [01] Copying ./boxoffice_mdb/dim_showtime.ibd to /var/lib/mysql/boxoffice_mdb/dim_showtime.ibd
 
180125 13:08:31 [01]        ...done
 
180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_theater_mdc.ibd to /var/lib/mysql/boxoffice_mdb/dim_theater_mdc.ibd
 
180125 13:08:32 [01]        ...done
 
180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_theater.ibd to /var/lib/mysql/boxoffice_mdb/dim_theater.ibd
 
180125 13:08:32 [01]        ...done
 
180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_ticket_type.ibd to /var/lib/mysql/boxoffice_mdb/dim_ticket_type.ibd
 
180125 13:08:32 [01]        ...done
 
180125 13:08:33 [01] Copying ./boxoffice_mdb/dim_title.ibd to /var/lib/mysql/boxoffice_mdb/dim_title.ibd
 
180125 13:08:33 [01]        ...done
 
180125 13:08:33 [01] Copying ./boxoffice_mdb/fact_tickets.ibd to /var/lib/mysql/boxoffice_mdb/fact_tickets.ibd
 
180125 13:08:37 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/db.opt to /var/lib/mysql/boxoffice_mdb/db.opt
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_showtime.frm to /var/lib/mysql/boxoffice_mdb/dim_showtime.frm
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_theater_mdc.frm to /var/lib/mysql/boxoffice_mdb/dim_theater_mdc.frm
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_theater.frm to /var/lib/mysql/boxoffice_mdb/dim_theater.frm
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_ticket_type.frm to /var/lib/mysql/boxoffice_mdb/dim_ticket_type.frm
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_title.frm to /var/lib/mysql/boxoffice_mdb/dim_title.frm
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./boxoffice_mdb/fact_tickets.frm to /var/lib/mysql/boxoffice_mdb/fact_tickets.frm
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./performance_schema/db.opt to /var/lib/mysql/performance_schema/db.opt
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./test/db.opt to /var/lib/mysql/test/db.opt
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./aria_log.00000001 to /var/lib/mysql/aria_log.00000001
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./aria_log_control to /var/lib/mysql/aria_log_control
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./xtrabackup_galera_info to /var/lib/mysql/xtrabackup_galera_info
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./ib_buffer_pool to /var/lib/mysql/ib_buffer_pool
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 [01] Copying ./xtrabackup_info to /var/lib/mysql/xtrabackup_info
 
180125 13:08:39 [01]        ...done
 
180125 13:08:39 completed OK!

7. start mysql service – Node fail to rejoin the cluster 

[root@t4w3 mysql]# service mysql start
 
Starting mysql (via systemctl):  Job for mariadb.service failed because a fatal signal was delivered to the control process. See "systemctl status mariadb.service" and "journalctl -xe" for details.
 
                                                           [FAILED]

mysql is restarting endlessly but Node fails to rejoin cluster



 Comments   
Comment by Zdravelina Sokolovska (Inactive) [ 2018-05-03 ]

Working scenarios

_Get mariadb full maraibackup _
garbd --address gcomm://192.168.104.191:4567,192.168.104.195:4567,192.168.104.196:4567?gmcast.listen_addr=tcp://0.0.0.0:4444 --group cluster1

Get mariadb backup/
Node B.
MariaDB [(none)]> set global wsrep_desync=1;
Query OK, 0 rows affected (0.001 sec)
2018-05-02 15:15:23 0 [Note] WSREP: Member 2.0 (t4w6) desyncs itself from group
2018-05-02 15:15:23 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 137764)

MariaDB [(none)]> flush tables with read lock;
Query OK, 0 rows affected (0.001 sec)
2018-05-02 15:17:00 11 [Note] WSREP: Provider paused at b4c974d2-49fe-11e8-b950-9b4c947b49f6:137764 (8)

mariabackup --backup -u root -p1 --galera-info --no-lock --target-dir=/var/mariadb/backup/B1

MariaDB [(none)]> unlock tables ;
Query OK, 0 rows affected (0.000 sec)

MariaDB [(none)]> set global wsrep_desync=0;
Query OK, 0 rows affected, 1 warning (0.000 sec)

Case A :
New Node has to initiate galera cluster using the backuped data
apply the backpacked data on Remote Node

  1. rsync -avrP /var/mariadb/backup/B1 root@t4w1:/var/mariadb/backup/
    root@t4w3's password:
    issue mariabackup prepare and mariabackup copy-back on the remote Node
    Ensure the correct permissions and ownership of the restored data in data-dir

initiate galera cluster from
[root@t4w1 ~]# galera_new_cluster

Case B
New Node has to join an already established cluster
apply the backpacked data on Remote Node

  1. rsync -avrP /var/mariadb/backup/B1 root@t4w1:/var/mariadb/backup/
    root@t4w3's password:

Node D:

mariabackup --prepare -u root -p1 --galera-info --target-dir=/var/mariadb/backup/B1

mariabackup --copy-back -u root -p1 --galera-info --target-dir=/var/mariadb/backup/B1
mariabackup based on MariaDB server 10.3.6-MariaDB Linux (x86_64)
180502 15:53:22 [01] Copying ibdata1 to /var/lib/mysql/ibdata1
~~~~~~~

reconstruct the grastate.dat file
copy grastate.dat file to the remote Node to give a chance to IST in case of Node has to be joined to an already established cluster
scp /var/lib/mysql/grastate.dat root@t4w1:/var/lib/mysql/
Or reconstruct the grastate.dat file from the /var/lib/mysql/xtrabackup_galera_info file
~~~~~~~

Ensure the correct permissions and ownership of the restored data in data-dir
Set correct permissions of the files in mysql dir
for i in $(ls -l /var/lib/mysql/ | grep -v ^d | awk '

{print $9}') ; do chmod g+w /var/lib/mysql/$i ; done

Set correct permissions of the files in mysql data dirs
for i in $(ls -l /var/lib/mysql/ | grep ^d | awk '{print $9}

'); do chmod g+w /var/lib/mysql/${i}/* ; done

Set correct ownership of all files and data dirs in mysql directory
chown -R mysql:mysql /var/lib/mysql/

Start mariadb service
service mysql start --wsrep_cluster_address=gcomm://xxx.xxx.xxx.xxx

Case C
+The same Node should rejoin cluster after recovering with already backuped data +

Comment by Jan Lindström (Inactive) [ 2018-05-07 ]

Root cause seems to be:

WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20180125 13:34:47.474)
 
WSREP_SST: [ERROR] Cleanup after exit with status:2 (20180125 13:34:47.477)
 
2018-01-25 13:34:47 140511583127296 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '192.168.104.193' --datadir '/var/lib/mysql/'   --parent '1995'  '' : 2 (No such file or directory)
 
2018-01-25 13:34:47 140511583127296 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
 
2018-01-25 13:34:47 140511928027264 [ERROR] WSREP: SST failed: 2 (No such file or directory)
 
2018-01-25 13:34:47 140511928027264 [ERROR] Aborting

Comment by Zdravelina Sokolovska (Inactive) [ 2018-05-15 ]

After setting 'wsrep_desync=1' and executing 'flush tables with read lock' on the Galera Backuping Node,
start maribackup with the option --no-lock to avoid the current MDEV-15921 behavior
where maribackup is not performed, hanging at "Executing FLUSH TABLES WITH READ LOCK... "

Comment by Seppo Jaakola [ 2018-06-13 ]

The first part of this scenario of taking mariabackup, removing the data directory and loading back the backup should not relate to the actual issue.
When node tries to join back to cluster, it happens by SST, which removes the data directory and starts to load new mariabackup from donor node.

Or was the expectation, that joining would happen by IST in this case?

To troubleshoot the SST error, more logs will be needed: MariaDB error log from donor node, and log files created by mariabackup execution both in donor and joiner nodes, are essential to analyze here.

Comment by Chris Calender (Inactive) [ 2018-06-13 ]

seppo No, it is not expected that IST will be used. The problem is that SST fails when the backup was taken by mariabackup using the --galera-info option. When the backup was taken without the --galera-info option, it works as expected.

Many of the logs/outputs seem to be above, and the issue should be reproducible if you just create the backup with --galera-info and then force SST to use that backup (at least that is the way I understand it). At least I see the mariabackup logs from both the donor and joiner. And the MariaDB error log from the failing node is above (the logs_from_rejoining _node.txt file).

If you need anything further, winstone, can you please provide the logs, as hopefully you have these logs?

Comment by Seppo Jaakola [ 2018-06-18 ]

I was able to reproduce this issue with mtr test in MariaDB 10.2.12.
Attached joiner side mariabackup log files, and there innobackup.move.log shows the actual problem:

--innobackupex based on MariaDB server 10.2.12-MariaDB Linux (x86_64) 
Error: datadir must be specified.

However, datadir is configured both in donor and joiner side my.cnf

I run the same test with current 10.2 tree HEAD version (10.2.16), and SST works fine there. So I assume mariabackup or SST scripts have been fixed to deal better with datadir forwarding, between these versions.

I will still try if 10.3 HEAD has this problem or not

Comment by Seppo Jaakola [ 2018-06-18 ]

Test with 10.3 HEAD version (10.3.8) is passing as well

Comment by Julien Fritsch [ 2018-06-26 ]

seppo what is the status on this one ? Any reason why it's stalled since 1 week?

Comment by Seppo Jaakola [ 2018-09-21 ]

For some reason I'm not able to reproduce this issue anymore.
I have, originally, created a bash script for reproducing the issue, and it made the issue surface with 10.2.12 and 10.3.6. Also, the test script showed no issues with 10.2 and 10.3 HEAD versions.
But, now that I try the test, in different development environment though, I cannot see failures with any MariaDB versions. So far I have tried with 10.2.11, 10.2.12, 10.2.13 and 10.3.6 - all those with Debug and RelWithDebugInfo source builds for 20 times loop.

I will attach the test script, for the interested reader.

winstone what was in my.cnf when having this issue? Could you try, if the test script can show the issue in your environment?

Comment by Zdravelina Sokolovska (Inactive) [ 2018-10-11 ]

seppo, there was noting set in my.cnf file and nothing specific in server.cnf file, just the min required configuration to run galera cluster.
In the first comment. I actually have added + working scenarios+
I suppose that one of the main and possible reasons of failure is in grastate.dat file. Mariabackup is creating xtrabackup_galera_info file, but grastate.dat file is not restored and user has to prepare it manually.

Generated at Thu Feb 08 08:18:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.