Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15067

Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Cannot Reproduce
    • 10.2.12, 10.3.6
    • N/A
    • Backup, Galera SST
    • 3 Nodes CentOS 7.4

    Description

      Node fails to rejoin mdb cluster after recovering with mariabackup with option --galera-info

      expected : Node is rejoining the cluster after recovering with mariabackup with option --galera-info

      1.created mariadb galera cluster composed from 3 Master Nodes
      it's used wsrep_sst method mariabackup ;
      all Nodes are wsrep ready, connected and in Synced state
      2. Perform mariabackup with option --galera-info on 1 Node
      in the user guide is recommended to use this option when backing up a node in Galera Cluster.
      3. stop mysql service on the backuped Node
      4. remove all content of datadir
      5. issue mariabackup --prepare step
      6. restore data with mariabackup --copy-back --galera-info
      7. start mysql service – Node fail to rejoin the cluster
      note: attached logs from rejoining _node

      2. Perform mariabackup with option --galera-info on 1 Node

      [root@t4w3 mysql]# mariabackup -u root  --backup  --galera-info --target-dir /var/mariadb/backupg2
      180125 13:05:18 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set
      Using server version 10.2.12-MariaDB
      mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)
      mariabackup: uses posix_fadvise().
      mariabackup: cd to /var/lib/mysql/
      mariabackup: open files limit requested 0, set to 1024
      mariabackup: using the following InnoDB configuration:
      mariabackup:   innodb_data_home_dir = .
      mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      mariabackup:   innodb_log_group_home_dir = ./
      2018-01-25 13:05:18 140272286804096 [Note] InnoDB: Number of pools: 1
      180125 13:05:18 >> log scanned up to (1041390203)
      mariabackup: Generating a list of tablespaces
      2018-01-25 13:05:18 140272286804096 [Warning] InnoDB: Allocated tablespace ID 3 for mysql/gtid_slave_pos, old maximum was 0
      180125 13:05:18 [01] Copying ./ibdata1 to /var/mariadb/backupg2/ibdata1
      180125 13:05:19 [01]        ...done
      180125 13:05:19 >> log scanned up to (1041390203)
      180125 13:05:19 [01] Copying ./mysql/gtid_slave_pos.ibd to /var/mariadb/backupg2/mysql/gtid_slave_pos.ibd
      180125 13:05:19 [01]        ...done
      180125 13:05:19 [01] Copying ./mysql/innodb_index_stats.ibd to /var/mariadb/backupg2/mysql/innodb_index_stats.ibd
      180125 13:05:19 [01]        ...done
      180125 13:05:19 [01] Copying ./mysql/innodb_table_stats.ibd to /var/mariadb/backupg2/mysql/innodb_table_stats.ibd
      180125 13:05:19 [01]        ...done
      180125 13:05:19 [01] Copying ./boxoffice_mdb/dim_showtime.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_showtime.ibd
      180125 13:05:21 [01]        ...done
      180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_theater_mdc.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_theater_mdc.ibd
      180125 13:05:22 [01]        ...done
      180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_theater.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_theater.ibd
      180125 13:05:22 [01]        ...done
      180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_ticket_type.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_ticket_type.ibd
      180125 13:05:22 [01]        ...done
      180125 13:05:22 [01] Copying ./boxoffice_mdb/dim_title.ibd to /var/mariadb/backupg2/boxoffice_mdb/dim_title.ibd
      180125 13:05:22 [01]        ...done
      180125 13:05:22 [01] Copying ./boxoffice_mdb/fact_tickets.ibd to /var/mariadb/backupg2/boxoffice_mdb/fact_tickets.ibd
      180125 13:05:27 [01]        ...done
      180125 13:05:29 Executing FLUSH NO_WRITE_TO_BINLOG TABLES...
      180125 13:05:29 Executing FLUSH TABLES WITH READ LOCK...
      180125 13:05:29 Starting to backup non-InnoDB tables and files
      180125 13:05:29 [01] Copying ./mysql/roles_mapping.MYI to /var/mariadb/backupg2/mysql/roles_mapping.MYI
      180125 13:05:29 [01]        ...done
      180125 13:05:29 [01] Copying ./mysql/column_stats.MYD to /var/mariadb/backupg2/mysql/column_stats.MYD
      180125 13:05:29 [01]        ...done
      180125 13:05:29 [01] Copying ./mysql/table_stats.MYI to /var/mariadb/backupg2/mysql/table_stats.MYI
      180125 13:05:29 [01]        ...done
      180125 13:05:29 [01] Copying ./mysql/column_stats.MYI to /var/mariadb/backupg2/mysql/column_stats.MYI
      180125 13:05:29 [01]        ...done
      180125 13:05:29 [01] Copying ./mysql/table_stats.frm to /var/mariadb/backupg2/mysql/table_stats.frm
      180125 13:05:29 [01]        ...done
      180125 13:05:29 [01] Copying ./mysql/column_stats.frm to /var/mariadb/backupg2/mysql/column_stats.frm
      180125 13:05:29 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/tables_priv.MYD to /var/mariadb/backupg2/mysql/tables_priv.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/columns_priv.MYD to /var/mariadb/backupg2/mysql/columns_priv.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/tables_priv.MYI to /var/mariadb/backupg2/mysql/tables_priv.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/columns_priv.MYI to /var/mariadb/backupg2/mysql/columns_priv.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/tables_priv.frm to /var/mariadb/backupg2/mysql/tables_priv.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/columns_priv.frm to /var/mariadb/backupg2/mysql/columns_priv.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/db.MYD to /var/mariadb/backupg2/mysql/db.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/db.MYI to /var/mariadb/backupg2/mysql/db.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/db.frm to /var/mariadb/backupg2/mysql/db.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/event.MYD to /var/mariadb/backupg2/mysql/event.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/event.MYI to /var/mariadb/backupg2/mysql/event.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/event.frm to /var/mariadb/backupg2/mysql/event.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/func.MYD to /var/mariadb/backupg2/mysql/func.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/func.MYI to /var/mariadb/backupg2/mysql/func.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/func.frm to /var/mariadb/backupg2/mysql/func.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone.MYD to /var/mariadb/backupg2/mysql/time_zone.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/general_log.CSM to /var/mariadb/backupg2/mysql/general_log.CSM
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone.MYI to /var/mariadb/backupg2/mysql/time_zone.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/general_log.CSV to /var/mariadb/backupg2/mysql/general_log.CSV
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone.frm to /var/mariadb/backupg2/mysql/time_zone.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/general_log.frm to /var/mariadb/backupg2/mysql/general_log.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.MYD to /var/mariadb/backupg2/mysql/time_zone_leap_second.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/gtid_slave_pos.frm to /var/mariadb/backupg2/mysql/gtid_slave_pos.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.MYI to /var/mariadb/backupg2/mysql/time_zone_leap_second.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_leap_second.frm to /var/mariadb/backupg2/mysql/time_zone_leap_second.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_category.MYD to /var/mariadb/backupg2/mysql/help_category.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_name.MYD to /var/mariadb/backupg2/mysql/time_zone_name.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_category.MYI to /var/mariadb/backupg2/mysql/help_category.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_name.MYI to /var/mariadb/backupg2/mysql/time_zone_name.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_category.frm to /var/mariadb/backupg2/mysql/help_category.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_name.frm to /var/mariadb/backupg2/mysql/time_zone_name.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_keyword.MYD to /var/mariadb/backupg2/mysql/help_keyword.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_transition.MYD to /var/mariadb/backupg2/mysql/time_zone_transition.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_keyword.MYI to /var/mariadb/backupg2/mysql/help_keyword.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_transition.MYI to /var/mariadb/backupg2/mysql/time_zone_transition.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_keyword.frm to /var/mariadb/backupg2/mysql/help_keyword.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/time_zone_transition.frm to /var/mariadb/backupg2/mysql/time_zone_transition.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_relation.MYD to /var/mariadb/backupg2/mysql/help_relation.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/user.MYI to /var/mariadb/backupg2/mysql/user.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_relation.MYI to /var/mariadb/backupg2/mysql/help_relation.MYI
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/user.MYD to /var/mariadb/backupg2/mysql/user.MYD
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/help_relation.frm to /var/mariadb/backupg2/mysql/help_relation.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:30 [01] Copying ./mysql/user.frm to /var/mariadb/backupg2/mysql/user.frm
      180125 13:05:30 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/help_topic.MYD to /var/mariadb/backupg2/mysql/help_topic.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/help_topic.MYI to /var/mariadb/backupg2/mysql/help_topic.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/help_topic.frm to /var/mariadb/backupg2/mysql/help_topic.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/host.MYD to /var/mariadb/backupg2/mysql/host.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/host.MYI to /var/mariadb/backupg2/mysql/host.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/host.frm to /var/mariadb/backupg2/mysql/host.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/index_stats.MYD to /var/mariadb/backupg2/mysql/index_stats.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/index_stats.MYI to /var/mariadb/backupg2/mysql/index_stats.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/index_stats.frm to /var/mariadb/backupg2/mysql/index_stats.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.MYD to /var/mariadb/backupg2/mysql/time_zone_transition_type.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/innodb_index_stats.frm to /var/mariadb/backupg2/mysql/innodb_index_stats.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/proxies_priv.MYI to /var/mariadb/backupg2/mysql/proxies_priv.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/proxies_priv.frm to /var/mariadb/backupg2/mysql/proxies_priv.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/innodb_table_stats.frm to /var/mariadb/backupg2/mysql/innodb_table_stats.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/roles_mapping.MYD to /var/mariadb/backupg2/mysql/roles_mapping.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/plugin.MYD to /var/mariadb/backupg2/mysql/plugin.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/plugin.MYI to /var/mariadb/backupg2/mysql/plugin.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/plugin.frm to /var/mariadb/backupg2/mysql/plugin.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/proc.MYD to /var/mariadb/backupg2/mysql/proc.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/proc.MYI to /var/mariadb/backupg2/mysql/proc.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/proc.frm to /var/mariadb/backupg2/mysql/proc.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/procs_priv.MYD to /var/mariadb/backupg2/mysql/procs_priv.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/procs_priv.MYI to /var/mariadb/backupg2/mysql/procs_priv.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/procs_priv.frm to /var/mariadb/backupg2/mysql/procs_priv.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/proxies_priv.MYD to /var/mariadb/backupg2/mysql/proxies_priv.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/roles_mapping.frm to /var/mariadb/backupg2/mysql/roles_mapping.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/servers.MYD to /var/mariadb/backupg2/mysql/servers.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/servers.MYI to /var/mariadb/backupg2/mysql/servers.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/servers.frm to /var/mariadb/backupg2/mysql/servers.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/slow_log.CSM to /var/mariadb/backupg2/mysql/slow_log.CSM
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/slow_log.CSV to /var/mariadb/backupg2/mysql/slow_log.CSV
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/slow_log.frm to /var/mariadb/backupg2/mysql/slow_log.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/table_stats.MYD to /var/mariadb/backupg2/mysql/table_stats.MYD
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.MYI to /var/mariadb/backupg2/mysql/time_zone_transition_type.MYI
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./mysql/time_zone_transition_type.frm to /var/mariadb/backupg2/mysql/time_zone_transition_type.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./boxoffice_mdb/db.opt to /var/mariadb/backupg2/boxoffice_mdb/db.opt
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./boxoffice_mdb/dim_showtime.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_showtime.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:31 [01] Copying ./boxoffice_mdb/dim_theater_mdc.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_theater_mdc.frm
      180125 13:05:31 [01]        ...done
      180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_theater.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_theater.frm
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_ticket_type.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_ticket_type.frm
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [01] Copying ./boxoffice_mdb/dim_title.frm to /var/mariadb/backupg2/boxoffice_mdb/dim_title.frm
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [01] Copying ./boxoffice_mdb/fact_tickets.frm to /var/mariadb/backupg2/boxoffice_mdb/fact_tickets.frm
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [01] Copying ./performance_schema/db.opt to /var/mariadb/backupg2/performance_schema/db.opt
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [01] Copying ./test/db.opt to /var/mariadb/backupg2/test/db.opt
      180125 13:05:32 [01]        ...done
      180125 13:05:32 Finished backing up non-InnoDB tables and files
      180125 13:05:32 [01] Copying aria_log.00000001 to /var/mariadb/backupg2/aria_log.00000001
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [01] Copying aria_log_control to /var/mariadb/backupg2/aria_log_control
      180125 13:05:32 [01]        ...done
      180125 13:05:32 [00] Writing xtrabackup_galera_info
      180125 13:05:32 [00]        ...done
      180125 13:05:32 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
      mariabackup: The latest check point (for incremental): '1041390194'
      mariabackup: Stopping log copying thread.
       
      180125 13:05:32 >> log scanned up to (1041390203)
      180125 13:05:32 Executing UNLOCK TABLES
      180125 13:05:32 All tables unlocked
      180125 13:05:32 [00] Copying ib_buffer_pool to /var/mariadb/backupg2/ib_buffer_pool
      180125 13:05:32 [00]        ...done
      180125 13:05:32 Backup created in directory '/var/mariadb/backupg2/'
      180125 13:05:32 [00] Writing backup-my.cnf
      180125 13:05:32 [00]        ...done
      180125 13:05:32 [00] Writing xtrabackup_info
      180125 13:05:32 [00]        ...done
      mariabackup: Redo log (from LSN 1041390194 to 1041390203) was copied.
      180125 13:05:32 completed OK!
      

      [root@t4w3 mysql]# ls  /var/mariadb/backupg2
      aria_log.00000001  backup-my.cnf  ib_buffer_pool  ib_logfile0  performance_schema  xtrabackup_checkpoints  xtrabackup_info
      aria_log_control   boxoffice_mdb  ibdata1         mysql        test                xtrabackup_galera_info
      [root@t4w3 mysql]#
      

      3. stop mysql service on the backuped Node

      [root@t4w3 mysql]# service mysql stop
      Stopping mysql (via systemctl):                            [  OK  ]
      

      4. remove all content of datadir

      [root@t4w3 mysql]# rm -rf /var/lib/mysql/*
      

      5. issue mariabackup --prepare step

      [root@t4w3 mysql]#
      [root@t4w3 mysql]# mariabackup -u prepare  --backup  --galera-info --target-dir /var/mariadb/backupg2
      180125 13:07:48 Connecting to MySQL server host: localhost, user: prepare, password: not set, port: not set, socket: not set
      Failed to connect to MySQL server: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory").
      [root@t4w3 mysql]# mariabackup -u  root --prepare    --galera-info --target-dir /var/mariadb/backupg2
      mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)
      mariabackup: cd to /var/mariadb/backupg2/
      mariabackup: This target seems to be not prepared yet.
      mariabackup: using the following InnoDB configuration for recovery:
      mariabackup:   innodb_data_home_dir = .
      mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      mariabackup:   innodb_log_group_home_dir = .
      mariabackup: Starting InnoDB instance for recovery.
      mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Uses event mutexes
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Number of pools: 1
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Using generic crc32 instructions
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Completed initialization of buffer pool
      2018-01-25 13:08:08 140279319353088 [Note] InnoDB: page_cleaner coordinator priority: -20
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Highest supported file format is Barracuda.
      2018-01-25 13:08:08 140279674140800 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1041390194
      180125 13:08:09 completed OK!
      [root@t4w3 mysql]#
      

      6. restore data with mariabackup --copy-back --galera-info

      [root@t4w3 mysql]# mariabackup -u  root --copy-back    --galera-info --target-dir /var/mariadb/backupg2
      mariabackup based on MariaDB server 10.2.12-MariaDB Linux (x86_64)
      180125 13:08:26 [01] Copying ibdata1 to /var/lib/mysql/ibdata1
      180125 13:08:27 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/gtid_slave_pos.ibd to /var/lib/mysql/mysql/gtid_slave_pos.ibd
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/innodb_index_stats.ibd to /var/lib/mysql/mysql/innodb_index_stats.ibd
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/innodb_table_stats.ibd to /var/lib/mysql/mysql/innodb_table_stats.ibd
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/roles_mapping.MYI to /var/lib/mysql/mysql/roles_mapping.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/column_stats.MYD to /var/lib/mysql/mysql/column_stats.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/table_stats.MYI to /var/lib/mysql/mysql/table_stats.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/column_stats.MYI to /var/lib/mysql/mysql/column_stats.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/table_stats.frm to /var/lib/mysql/mysql/table_stats.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/column_stats.frm to /var/lib/mysql/mysql/column_stats.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/tables_priv.MYD to /var/lib/mysql/mysql/tables_priv.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/columns_priv.MYD to /var/lib/mysql/mysql/columns_priv.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/tables_priv.MYI to /var/lib/mysql/mysql/tables_priv.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/columns_priv.MYI to /var/lib/mysql/mysql/columns_priv.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/tables_priv.frm to /var/lib/mysql/mysql/tables_priv.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/columns_priv.frm to /var/lib/mysql/mysql/columns_priv.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/db.MYD to /var/lib/mysql/mysql/db.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/db.MYI to /var/lib/mysql/mysql/db.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/db.frm to /var/lib/mysql/mysql/db.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/event.MYD to /var/lib/mysql/mysql/event.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/event.MYI to /var/lib/mysql/mysql/event.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/event.frm to /var/lib/mysql/mysql/event.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/func.MYD to /var/lib/mysql/mysql/func.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/func.MYI to /var/lib/mysql/mysql/func.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/func.frm to /var/lib/mysql/mysql/func.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/time_zone.MYD to /var/lib/mysql/mysql/time_zone.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/general_log.CSM to /var/lib/mysql/mysql/general_log.CSM
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/time_zone.MYI to /var/lib/mysql/mysql/time_zone.MYI
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/general_log.CSV to /var/lib/mysql/mysql/general_log.CSV
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/time_zone.frm to /var/lib/mysql/mysql/time_zone.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/general_log.frm to /var/lib/mysql/mysql/general_log.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/time_zone_leap_second.MYD to /var/lib/mysql/mysql/time_zone_leap_second.MYD
      180125 13:08:28 [01]        ...done
      180125 13:08:28 [01] Copying ./mysql/gtid_slave_pos.frm to /var/lib/mysql/mysql/gtid_slave_pos.frm
      180125 13:08:28 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_leap_second.MYI to /var/lib/mysql/mysql/time_zone_leap_second.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_leap_second.frm to /var/lib/mysql/mysql/time_zone_leap_second.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_category.MYD to /var/lib/mysql/mysql/help_category.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_name.MYD to /var/lib/mysql/mysql/time_zone_name.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_category.MYI to /var/lib/mysql/mysql/help_category.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_name.MYI to /var/lib/mysql/mysql/time_zone_name.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_category.frm to /var/lib/mysql/mysql/help_category.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_name.frm to /var/lib/mysql/mysql/time_zone_name.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_keyword.MYD to /var/lib/mysql/mysql/help_keyword.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_transition.MYD to /var/lib/mysql/mysql/time_zone_transition.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_keyword.MYI to /var/lib/mysql/mysql/help_keyword.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_transition.MYI to /var/lib/mysql/mysql/time_zone_transition.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_keyword.frm to /var/lib/mysql/mysql/help_keyword.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/time_zone_transition.frm to /var/lib/mysql/mysql/time_zone_transition.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_relation.MYD to /var/lib/mysql/mysql/help_relation.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/user.MYI to /var/lib/mysql/mysql/user.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_relation.MYI to /var/lib/mysql/mysql/help_relation.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/user.MYD to /var/lib/mysql/mysql/user.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_relation.frm to /var/lib/mysql/mysql/help_relation.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/user.frm to /var/lib/mysql/mysql/user.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_topic.MYD to /var/lib/mysql/mysql/help_topic.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_topic.MYI to /var/lib/mysql/mysql/help_topic.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/help_topic.frm to /var/lib/mysql/mysql/help_topic.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/host.MYD to /var/lib/mysql/mysql/host.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/host.MYI to /var/lib/mysql/mysql/host.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/host.frm to /var/lib/mysql/mysql/host.frm
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/index_stats.MYD to /var/lib/mysql/mysql/index_stats.MYD
      180125 13:08:29 [01]        ...done
      180125 13:08:29 [01] Copying ./mysql/index_stats.MYI to /var/lib/mysql/mysql/index_stats.MYI
      180125 13:08:29 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/index_stats.frm to /var/lib/mysql/mysql/index_stats.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.MYD to /var/lib/mysql/mysql/time_zone_transition_type.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/innodb_index_stats.frm to /var/lib/mysql/mysql/innodb_index_stats.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/proxies_priv.MYI to /var/lib/mysql/mysql/proxies_priv.MYI
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/proxies_priv.frm to /var/lib/mysql/mysql/proxies_priv.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/innodb_table_stats.frm to /var/lib/mysql/mysql/innodb_table_stats.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/roles_mapping.MYD to /var/lib/mysql/mysql/roles_mapping.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/plugin.MYD to /var/lib/mysql/mysql/plugin.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/plugin.MYI to /var/lib/mysql/mysql/plugin.MYI
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/plugin.frm to /var/lib/mysql/mysql/plugin.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/proc.MYD to /var/lib/mysql/mysql/proc.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/proc.MYI to /var/lib/mysql/mysql/proc.MYI
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/proc.frm to /var/lib/mysql/mysql/proc.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/procs_priv.MYD to /var/lib/mysql/mysql/procs_priv.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/procs_priv.MYI to /var/lib/mysql/mysql/procs_priv.MYI
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/procs_priv.frm to /var/lib/mysql/mysql/procs_priv.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/proxies_priv.MYD to /var/lib/mysql/mysql/proxies_priv.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/roles_mapping.frm to /var/lib/mysql/mysql/roles_mapping.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/servers.MYD to /var/lib/mysql/mysql/servers.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/servers.MYI to /var/lib/mysql/mysql/servers.MYI
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/servers.frm to /var/lib/mysql/mysql/servers.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/slow_log.CSM to /var/lib/mysql/mysql/slow_log.CSM
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/slow_log.CSV to /var/lib/mysql/mysql/slow_log.CSV
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/slow_log.frm to /var/lib/mysql/mysql/slow_log.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/table_stats.MYD to /var/lib/mysql/mysql/table_stats.MYD
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.MYI to /var/lib/mysql/mysql/time_zone_transition_type.MYI
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./mysql/time_zone_transition_type.frm to /var/lib/mysql/mysql/time_zone_transition_type.frm
      180125 13:08:30 [01]        ...done
      180125 13:08:30 [01] Copying ./boxoffice_mdb/dim_showtime.ibd to /var/lib/mysql/boxoffice_mdb/dim_showtime.ibd
      180125 13:08:31 [01]        ...done
      180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_theater_mdc.ibd to /var/lib/mysql/boxoffice_mdb/dim_theater_mdc.ibd
      180125 13:08:32 [01]        ...done
      180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_theater.ibd to /var/lib/mysql/boxoffice_mdb/dim_theater.ibd
      180125 13:08:32 [01]        ...done
      180125 13:08:32 [01] Copying ./boxoffice_mdb/dim_ticket_type.ibd to /var/lib/mysql/boxoffice_mdb/dim_ticket_type.ibd
      180125 13:08:32 [01]        ...done
      180125 13:08:33 [01] Copying ./boxoffice_mdb/dim_title.ibd to /var/lib/mysql/boxoffice_mdb/dim_title.ibd
      180125 13:08:33 [01]        ...done
      180125 13:08:33 [01] Copying ./boxoffice_mdb/fact_tickets.ibd to /var/lib/mysql/boxoffice_mdb/fact_tickets.ibd
      180125 13:08:37 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/db.opt to /var/lib/mysql/boxoffice_mdb/db.opt
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_showtime.frm to /var/lib/mysql/boxoffice_mdb/dim_showtime.frm
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_theater_mdc.frm to /var/lib/mysql/boxoffice_mdb/dim_theater_mdc.frm
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_theater.frm to /var/lib/mysql/boxoffice_mdb/dim_theater.frm
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_ticket_type.frm to /var/lib/mysql/boxoffice_mdb/dim_ticket_type.frm
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/dim_title.frm to /var/lib/mysql/boxoffice_mdb/dim_title.frm
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./boxoffice_mdb/fact_tickets.frm to /var/lib/mysql/boxoffice_mdb/fact_tickets.frm
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./performance_schema/db.opt to /var/lib/mysql/performance_schema/db.opt
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./test/db.opt to /var/lib/mysql/test/db.opt
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./aria_log.00000001 to /var/lib/mysql/aria_log.00000001
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./aria_log_control to /var/lib/mysql/aria_log_control
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./xtrabackup_galera_info to /var/lib/mysql/xtrabackup_galera_info
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./ib_buffer_pool to /var/lib/mysql/ib_buffer_pool
      180125 13:08:39 [01]        ...done
      180125 13:08:39 [01] Copying ./xtrabackup_info to /var/lib/mysql/xtrabackup_info
      180125 13:08:39 [01]        ...done
      180125 13:08:39 completed OK!
      

      7. start mysql service – Node fail to rejoin the cluster

      [root@t4w3 mysql]# service mysql start
      Starting mysql (via systemctl):  Job for mariadb.service failed because a fatal signal was delivered to the control process. See "systemctl status mariadb.service" and "journalctl -xe" for details.
                                                                 [FAILED]
      

      mysql is restarting endlessly but Node fails to rejoin cluster

      Attachments

        1. galera_info.sh
          2 kB
          Seppo Jaakola
        2. innobackup.move.log
          0.3 kB
          Seppo Jaakola
        3. innobackup.prepare.log
          2 kB
          Seppo Jaakola
        4. logs_from_rejoining _node.txt
          26 kB
          Zdravelina Sokolovska

        Issue Links

          Activity

            People

              seppo Seppo Jaakola
              winstone Zdravelina Sokolovska (Inactive)
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.