Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.11.9
-
None
-
Ubuntu 22.04.5
Description
Hi team,
we have a 5 node Galera Cluster 26.4.19 with MariaDB Server 10.11.9.
After a test update of MariaDB on one node (maria-ham-3) I switched back to the versions above. I cleared data directory and wanted to sync and join (what worked fine in the past) with the node:
{{
- rm -rf /GALERANFS_HAM_DATA/data/mariadb/* /GALERANFS_HAM_DATA/InnoDB/ibdata1 /GALERANFS_HAM_LOG/ib_log/ib_logfile0
- systemctl start mariadb
}}
This leads to a transfer starting but the later failing:
{{
Sep 27 09:20:52 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:52 0 [Note] WSREP: REPORTING SST PROGRESS: ' { "from": 1, "to": 3, "total": 19922598377, "done": 19922598377, "indefinite": -1 }'
{ "from": 1, "to": 3, "total": 19935177778, "done": 19935177778, "indefinite": -1 }
Sep 27 09:20:58 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:58 0 [Note] WSREP: REPORTING SST PROGRESS: ''
{ "from": 1, "to": 3, "total": 19947753124, "done": 19947753124, "indefinite": -1 }
Sep 27 09:21:04 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:04 0 [Note] WSREP: REPORTING SST PROGRESS: ''
{ Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: a591b16e-9113,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Warning] WSREP: 0.0 (maria-muc-1): State transfer to 3.0 (maria-ham-3) failed: -22 (Invalid argument)
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [ERROR] WSREP: ./gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1217: Will never receive state. Need to abort.
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: terminating thread
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: joining thread
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closing backend
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view(view_id(NON_PRIM,0abc8cb4-9ae0,609) membjoined
{ Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }left
{ Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }partitioned
{ Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 0abc8cb4-9ae0,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 29958514-86e6,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 7150c82e-8f59,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: dfc13b8b-8838,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: })
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: PC protocol downgrade 1 -> 0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view((empty))
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closed
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: /usr/sbin/mariadbd: Terminated.
Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 27 09:21:08 maria-ham-3 systemd[1]: Failed to start MariaDB 10.11.9 database server.
}}
I configured the wsrep_sst_donor to one node to ease the debugging.
The logs of both sides are attached.
Filesystems are not full.
I cannot find out what stopped the transfer. Can you give me any hint of what to do for further debugging?
Thanks in advance
Lars