Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34411

Replica failed to join to the cluster during initial start

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.6.17
    • None
    • Galera
    • None

    Description

      Trying to bootstrap the cluster for first time.

      • mariadb-server-0 started succesffully
      • mariadb-server-1 is not able to join to the cluster for the first time

      mariadb-server-0

      2024-06-15T12:59:01.406770045+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Non-primary view
      2024-06-15T12:59:01.408830704+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Server status change connected -> connected
      2024-06-15T12:59:01.410193325+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2024-06-15T12:59:01.411723458+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 48 [Warning] Aborted connection 48 to db: 'unconnected' user: 'sst' host: '127.0.0.1' (Got an error reading communication packets)
      2024-06-15T12:59:01.413151194+00:00 info 2024-06-15 12:59:00,651 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] mariadb-backup finished with error: 1. Check syslog or '/var/lib/mysql/mariabackup.backup.log' for details (20240615 12:59:00.646)
      2024-06-15T12:59:01.414541433+00:00 info 2024-06-15 12:59:00,651 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] Cleanup after exit with status: 22 (20240615 12:59:00.650)
      

      mariadb-server-0 mariabackup.backup.log

      mysql@mariadb-server-0:/var/lib/mysql$ cat mariabackup.backup.log
      [00] 2024-06-15 12:58:59 Connecting to MariaDB server host: localhost, user: sst, password: set, port: 3306, socket: /run/mysqld/mysqld.sock
      [00] 2024-06-15 12:58:59 Using server version 10.6.17-MariaDB-1:10.6.17+maria~ubu2004
      [00] 2024-06-15 12:58:59 Warning: missing required privilege REPLICA MONITOR on *.*
      /usr//bin/mariadb-backup based on MariaDB server 10.6.17-MariaDB debian-linux-gnu (x86_64)
      [00] 2024-06-15 12:58:59 uses posix_fadvise().
      [00] 2024-06-15 12:58:59 cd to /var/lib/mysql/
      [00] 2024-06-15 12:58:59 open files limit requested 10240, set to 131072
      [00] 2024-06-15 12:58:59 mariabackup: using the following InnoDB configuration:
      [00] 2024-06-15 12:58:59 innodb_data_home_dir = 
      [00] 2024-06-15 12:58:59 innodb_data_file_path = ibdata1:12M:autoextend
      [00] 2024-06-15 12:58:59 innodb_log_group_home_dir = ./
      [00] 2024-06-15 12:58:59 InnoDB: Using Linux native AIO
      2024-06-15 12:58:59 0 [Note] InnoDB: Number of pools: 1
      [00] 2024-06-15 12:58:59 mariabackup: Generating a list of tablespaces
      [01] 2024-06-15 12:58:59 Streaming ./mysql/transaction_registry.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ./mysql/innodb_index_stats.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ./mysql/innodb_table_stats.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ./mysql/wsrep_cluster_members.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ./mysql/wsrep_streaming_log.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ./mysql/gtid_slave_pos.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ./mysql/wsrep_cluster.ibd
      [01] 2024-06-15 12:58:59         ...done
      [01] 2024-06-15 12:58:59 Streaming ibdata1
      [01] 2024-06-15 12:58:59         ...done
      [00] 2024-06-15 12:59:00 Acquiring BACKUP LOCKS...
      [00] FATAL ERROR: 2024-06-15 12:59:00 failed to execute query BACKUP STAGE START: WSREP has not yet prepared node for application use
      

      On server-1

      2024-06-15T12:59:01.050091202+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Flow-control interval: [16, 16]
      2024-06-15T12:59:01.051900576+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Received NON-PRIMARY.
      2024-06-15T12:59:01.053355940+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 6)
      2024-06-15T12:59:01.054749792+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2024-06-15T12:59:01.056107698+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Flow-control interval: [16, 16]
      2024-06-15T12:59:01.057603754+00:00 info 2024-06-15 12:59:00,735 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Received NON-PRIMARY.
      2024-06-15T12:59:01.059075749+00:00 info 2024-06-15 12:59:00,735 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed mariadb-backup/SST on donor (20240615 12:59:00.731)
      2024-06-15T12:59:01.060497307+00:00 info 2024-06-15 12:59:00,735 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] Cleanup after exit with status: 2 (20240615 12:59:00.734)
      2024-06-15T12:59:01.061992988+00:00 info 2024-06-15 12:59:00,740 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] Removing /var/lib/mysql/.sst/xtrabackup_galera_info file due to signal (20240615 12:59:00.739)
      2024-06-15T12:59:01.063539107+00:00 info 2024-06-15 12:59:00,744 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [INFO] Removing the sst_in_progress file (20240615 12:59:00.743)
      2024-06-15T12:59:01.065389961+00:00 info 2024-06-15 12:59:00,750 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [INFO] Cleaning up temporary directories (20240615 12:59:00.749)
      2024-06-15T12:59:01.066928663+00:00 info 2024-06-15 12:59:00,758 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '192.168.205.39' --datadir '/var/lib/mysql/' --parent 99 --progress 0: 2 (No such file or directory)
      2024-06-15T12:59:01.068267723+00:00 info 2024-06-15 12:59:00,758 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
      2024-06-15T12:59:01.069610038+00:00 info 2024-06-15 12:59:00,758 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 3 [Note] WSREP: SST received
      2024-06-15T12:59:01.071137872+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 3 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
      2024-06-15T12:59:01.072504404+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 3 [Note] WSREP: SST succeeded for position 00000000-0000-0000-0000-000000000000:-1
      2024-06-15T12:59:01.073821377+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Joiner monitor thread ended with total time 12 sec
      2024-06-15T12:59:01.075120533+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [ERROR] WSREP: Application received wrong state: 
      2024-06-15T12:59:01.076501110+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 	Received: 00000000-0000-0000-0000-000000000000
      2024-06-15T12:59:01.078031985+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 	Required: c5c97f6a-2b16-11ef-b05a-07e4f539bef2
      2024-06-15T12:59:01.079578371+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
      2024-06-15T12:59:01.081064790+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: ReplicatorSMM::abort()
      2024-06-15T12:59:01.082711267+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Closing send monitor...
      2024-06-15T12:59:01.084292935+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Closed send monitor.
      2024-06-15T12:59:01.086097560+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: gcomm: terminating thread
      2024-06-15T12:59:01.087896491+00:00 info 2024-06-15 12:59:00,760 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: gcomm: joining thread
      2024-06-15T12:59:02.071568965+00:00 info 2024-06-15 12:59:00,760 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: gcomm: closing backend
      2024-06-15T12:59:02.073451971+00:00 info 2024-06-15 12:59:01,024 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: declaring 017faa65-aee6 at tcp://192.168.149.80:4567 stable
      2024-06-15T12:59:02.075021660+00:00 info 2024-06-15 12:59:01,024 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: declaring d897d71a-ab2a at tcp://192.168.66.19:4567 stable
      2024-06-15T12:59:02.076654827+00:00 info 2024-06-15 12:59:01,024 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: re-bootstrapping prim from partitioned components
      2024-06-15T12:59:02.078155203+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Warning] WSREP: user message in state LEAVING
      2024-06-15T12:59:02.079536952+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Warning] WSREP: 017fa8a1-9e88 sending install message failed: Transport endpoint is not connected
      2024-06-15T12:59:02.080933221+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: PC protocol downgrade 1 -> 0
      2024-06-15T12:59:02.082528380+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: view((empty))
      2024-06-15T12:59:02.084032665+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: gcomm: closed
      2024-06-15T12:59:02.085788745+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: New SELF-LEAVE.
      2024-06-15T12:59:02.087647884+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: Flow-control interval: [0, 0]
      2024-06-15T12:59:02.089433039+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      2024-06-15T12:59:02.091063587+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 6)
      2024-06-15T12:59:02.092713335+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: RECV thread exiting 0: Success
      2024-06-15T12:59:02.094382032+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: recv_thread() joined.
      2024-06-15T12:59:02.096174663+00:00 info 2024-06-15 12:59:01,028 - OpenStack-Helm Mariadb - INFO - 
      2024-06-15T12:59:02.097720260+00:00 info 2024-06-15 12:59:01,028 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: Closing replication queue.
      2024-06-15T12:59:02.099262412+00:00 info 2024-06-15 12:59:01,029 - OpenStack-Helm Mariadb - INFO - 
      

      All logs are available in archive.

      It looks like we trying to start server-1 and join to the cluster when server-0 is not yet fully initialized.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jumpojoy Vasyl Saienko
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.