Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.6.17
-
None
-
None
Description
Trying to bootstrap the cluster for first time.
- mariadb-server-0 started succesffully
- mariadb-server-1 is not able to join to the cluster for the first time
mariadb-server-0
2024-06-15T12:59:01.406770045+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Non-primary view |
2024-06-15T12:59:01.408830704+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Server status change connected -> connected |
2024-06-15T12:59:01.410193325+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
2024-06-15T12:59:01.411723458+00:00 info 2024-06-15 12:59:00,648 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 48 [Warning] Aborted connection 48 to db: 'unconnected' user: 'sst' host: '127.0.0.1' (Got an error reading communication packets) |
2024-06-15T12:59:01.413151194+00:00 info 2024-06-15 12:59:00,651 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] mariadb-backup finished with error: 1. Check syslog or '/var/lib/mysql/mariabackup.backup.log' for details (20240615 12:59:00.646) |
2024-06-15T12:59:01.414541433+00:00 info 2024-06-15 12:59:00,651 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] Cleanup after exit with status: 22 (20240615 12:59:00.650) |
mariadb-server-0 mariabackup.backup.log
mysql@mariadb-server-0:/var/lib/mysql$ cat mariabackup.backup.log |
[00] 2024-06-15 12:58:59 Connecting to MariaDB server host: localhost, user: sst, password: set, port: 3306, socket: /run/mysqld/mysqld.sock |
[00] 2024-06-15 12:58:59 Using server version 10.6.17-MariaDB-1:10.6.17+maria~ubu2004 |
[00] 2024-06-15 12:58:59 Warning: missing required privilege REPLICA MONITOR on *.* |
/usr//bin/mariadb-backup based on MariaDB server 10.6.17-MariaDB debian-linux-gnu (x86_64) |
[00] 2024-06-15 12:58:59 uses posix_fadvise(). |
[00] 2024-06-15 12:58:59 cd to /var/lib/mysql/ |
[00] 2024-06-15 12:58:59 open files limit requested 10240, set to 131072 |
[00] 2024-06-15 12:58:59 mariabackup: using the following InnoDB configuration: |
[00] 2024-06-15 12:58:59 innodb_data_home_dir = |
[00] 2024-06-15 12:58:59 innodb_data_file_path = ibdata1:12M:autoextend |
[00] 2024-06-15 12:58:59 innodb_log_group_home_dir = ./ |
[00] 2024-06-15 12:58:59 InnoDB: Using Linux native AIO |
2024-06-15 12:58:59 0 [Note] InnoDB: Number of pools: 1 |
[00] 2024-06-15 12:58:59 mariabackup: Generating a list of tablespaces |
[01] 2024-06-15 12:58:59 Streaming ./mysql/transaction_registry.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ./mysql/innodb_index_stats.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ./mysql/innodb_table_stats.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ./mysql/wsrep_cluster_members.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ./mysql/wsrep_streaming_log.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ./mysql/gtid_slave_pos.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ./mysql/wsrep_cluster.ibd |
[01] 2024-06-15 12:58:59 ...done |
[01] 2024-06-15 12:58:59 Streaming ibdata1 |
[01] 2024-06-15 12:58:59 ...done |
[00] 2024-06-15 12:59:00 Acquiring BACKUP LOCKS... |
[00] FATAL ERROR: 2024-06-15 12:59:00 failed to execute query BACKUP STAGE START: WSREP has not yet prepared node for application use |
On server-1
2024-06-15T12:59:01.050091202+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Flow-control interval: [16, 16] |
2024-06-15T12:59:01.051900576+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Received NON-PRIMARY. |
2024-06-15T12:59:01.053355940+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 6) |
2024-06-15T12:59:01.054749792+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2024-06-15T12:59:01.056107698+00:00 info 2024-06-15 12:59:00,734 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Flow-control interval: [16, 16] |
2024-06-15T12:59:01.057603754+00:00 info 2024-06-15 12:59:00,735 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Received NON-PRIMARY. |
2024-06-15T12:59:01.059075749+00:00 info 2024-06-15 12:59:00,735 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed mariadb-backup/SST on donor (20240615 12:59:00.731) |
2024-06-15T12:59:01.060497307+00:00 info 2024-06-15 12:59:00,735 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] Cleanup after exit with status: 2 (20240615 12:59:00.734) |
2024-06-15T12:59:01.061992988+00:00 info 2024-06-15 12:59:00,740 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [ERROR] Removing /var/lib/mysql/.sst/xtrabackup_galera_info file due to signal (20240615 12:59:00.739) |
2024-06-15T12:59:01.063539107+00:00 info 2024-06-15 12:59:00,744 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [INFO] Removing the sst_in_progress file (20240615 12:59:00.743) |
2024-06-15T12:59:01.065389961+00:00 info 2024-06-15 12:59:00,750 - OpenStack-Helm Mariadb - INFO - WSREP_SST: [INFO] Cleaning up temporary directories (20240615 12:59:00.749) |
2024-06-15T12:59:01.066928663+00:00 info 2024-06-15 12:59:00,758 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '192.168.205.39' --datadir '/var/lib/mysql/' --parent 99 --progress 0: 2 (No such file or directory) |
2024-06-15T12:59:01.068267723+00:00 info 2024-06-15 12:59:00,758 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. |
2024-06-15T12:59:01.069610038+00:00 info 2024-06-15 12:59:00,758 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 3 [Note] WSREP: SST received |
2024-06-15T12:59:01.071137872+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 3 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1 |
2024-06-15T12:59:01.072504404+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 3 [Note] WSREP: SST succeeded for position 00000000-0000-0000-0000-000000000000:-1 |
2024-06-15T12:59:01.073821377+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 0 [Note] WSREP: Joiner monitor thread ended with total time 12 sec |
2024-06-15T12:59:01.075120533+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [ERROR] WSREP: Application received wrong state: |
2024-06-15T12:59:01.076501110+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - Received: 00000000-0000-0000-0000-000000000000 |
2024-06-15T12:59:01.078031985+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - Required: c5c97f6a-2b16-11ef-b05a-07e4f539bef2 |
2024-06-15T12:59:01.079578371+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required. |
2024-06-15T12:59:01.081064790+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: ReplicatorSMM::abort() |
2024-06-15T12:59:01.082711267+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Closing send monitor... |
2024-06-15T12:59:01.084292935+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: Closed send monitor. |
2024-06-15T12:59:01.086097560+00:00 info 2024-06-15 12:59:00,759 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: gcomm: terminating thread |
2024-06-15T12:59:01.087896491+00:00 info 2024-06-15 12:59:00,760 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: gcomm: joining thread |
2024-06-15T12:59:02.071568965+00:00 info 2024-06-15 12:59:00,760 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:00 2 [Note] WSREP: gcomm: closing backend |
2024-06-15T12:59:02.073451971+00:00 info 2024-06-15 12:59:01,024 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: declaring 017faa65-aee6 at tcp://192.168.149.80:4567 stable |
2024-06-15T12:59:02.075021660+00:00 info 2024-06-15 12:59:01,024 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: declaring d897d71a-ab2a at tcp://192.168.66.19:4567 stable |
2024-06-15T12:59:02.076654827+00:00 info 2024-06-15 12:59:01,024 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: re-bootstrapping prim from partitioned components |
2024-06-15T12:59:02.078155203+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Warning] WSREP: user message in state LEAVING |
2024-06-15T12:59:02.079536952+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Warning] WSREP: 017fa8a1-9e88 sending install message failed: Transport endpoint is not connected |
2024-06-15T12:59:02.080933221+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2024-06-15T12:59:02.082528380+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: view((empty)) |
2024-06-15T12:59:02.084032665+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: gcomm: closed |
2024-06-15T12:59:02.085788745+00:00 info 2024-06-15 12:59:01,025 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: New SELF-LEAVE. |
2024-06-15T12:59:02.087647884+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: Flow-control interval: [0, 0] |
2024-06-15T12:59:02.089433039+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. |
2024-06-15T12:59:02.091063587+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 6) |
2024-06-15T12:59:02.092713335+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 0 [Note] WSREP: RECV thread exiting 0: Success |
2024-06-15T12:59:02.094382032+00:00 info 2024-06-15 12:59:01,026 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: recv_thread() joined. |
2024-06-15T12:59:02.096174663+00:00 info 2024-06-15 12:59:01,028 - OpenStack-Helm Mariadb - INFO - |
2024-06-15T12:59:02.097720260+00:00 info 2024-06-15 12:59:01,028 - OpenStack-Helm Mariadb - INFO - 2024-06-15 12:59:01 2 [Note] WSREP: Closing replication queue. |
2024-06-15T12:59:02.099262412+00:00 info 2024-06-15 12:59:01,029 - OpenStack-Helm Mariadb - INFO - |
All logs are available in archive.
It looks like we trying to start server-1 and join to the cluster when server-0 is not yet fully initialized.