Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.11, 11.4, 11.8
-
Can result in hang or crash
-
If user has provided unknown option when server was started it could get stuck on shutdown because SST/IST newer interrupted.
-
Q4/2026 Replic. Development
Description
galera.galera_restart_on_unknown_option fails with Test case timeout after 900 seconds on the 10.11, 11.4 and 11.8 branches: after the test starts mariadbd with --galera-unknown-option, the process hangs in unireg_abort() during the wsrep provider disconnect and never exits.
Build Run Info
| Branch | Builder name | Date | Commit | Variant |
|---|---|---|---|---|
| 10.11 | s390x-ubuntu-2204 | 2026-10-31T04:17:56Z | 488f55471cba07b9e5bf889151ea1795efe8000d | none |
| 11.4 | s390x-ubuntu-2204 | 2026-10-31T04:03:28Z | 8e739ec336137cdc835a9d3f726f288d8756b20b | none |
| 11.8 | s390x-ubuntu-2204 | 2026-10-29T23:21:51Z | fdac341f27405def4fe20c8d6b09d45fba00ea08 | none |
Failure text
Latest run on 10.11 (excerpt: the full text is about 115 KB, the omitted part is gdb output of the other processes and threads; worker number, timestamps, pids and paths differ between runs):
galera.galera_restart_on_unknown_option w10 [ fail ] timeout after 900 seconds
|
Test ended at 2026-10-31 03:37:01
|
|
|
Test case timeout after 900 seconds
|
|
|
== /home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/10/log/galera_restart_on_unknown_option.log ==
|
FOUND 1 /unknown option '--galera-unknown-option'/ in galera_restart_on_unknown_option.err
|
FOUND 1 /State transfer interrupted, shutting down gracefully/ in galera_restart_on_unknown_option.err
|
NOT FOUND /Assertion .* failed/ in galera_restart_on_unknown_option.err
|
Starting server ...
|
SELECT * FROM t1;
|
f1 f2
|
1 a
|
2 b
|
3 c
|
Shutting down server ...
|
connection node_1;
|
UPDATE t1 SET f2 = 'd' WHERE f1 > 1;
|
UPDATE t1 SET f2 = 'd' WHERE f1 > 2;
|
SELECT * FROM t1;
|
f1 f2
|
1 a
|
2 d
|
3 d
|
connection node_2;
|
Starting server with an unknown option ...
|
|
|
== /home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/10/tmp/analyze-timeout-mysqld.2.err ==
|
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/tmp/10/mys' (2)
|
|
|
== /home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/10/tmp/analyze-timeout-mysqld.1.err ==
|
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/tmp/10/mys' (111)
|
|
|
|
|
- found 'core.268694', moving it to '/home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/10/log/galera.galera_restart_on_unknown_option'
|
- found 'core.286043', moving it to '/home/buildbot/s390x-ubuntu-2204/build/mysql-test/var/10/log/galera.galera_restart_on_unknown_option'
|
- found 'core.286043' (0/2)
|
|
|
< snip >
|
|
|
Core generated by '/home/buildbot/s390x-ubuntu-2204/build/sql/mariadbd'
|
Output from gdb follows. The first stack trace is from the failing thread.
|
The following stack traces are from all threads (so the failing one is
|
duplicated).
|
--------------------------
|
|
|
warning: Can't open file /dev/shm/var_auto_ifAV/10/mysqld.2/data/galera.cache during file-backed mapping note processing
|
[New LWP 286045]
|
[New LWP 286068]
|
[New LWP 286440]
|
[New LWP 286449]
|
[New LWP 286439]
|
[New LWP 288362]
|
[New LWP 286448]
|
[New LWP 286067]
|
[New LWP 287004]
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
|
Core was generated by `/home/buildbot/s390x-ubuntu-2204/build/sql/mariadbd --defaults-group-suffix=.2'.
|
Program terminated with signal SIGABRT, Aborted.
|
#0 0x000003ff9b548ce2 in kill () from /lib/s390x-linux-gnu/libc.so.6
|
[Current thread is 1 (Thread 0x3ff9c567820 (LWP 286045))]
|
#0 0x000003ff9b548ce2 in kill () from /lib/s390x-linux-gnu/libc.so.6
|
#1 0x000002aa2f9fc194 in handle_fatal_signal (sig=<optimized out>) at /home/buildbot/s390x-ubuntu-2204/build/sql/signal_handler.cc:312
|
#2 <signal handler called>
|
#3 0x000003ff9b592b36 in ?? () from /lib/s390x-linux-gnu/libc.so.6
|
#4 0x000003ff9b59574a in pthread_cond_wait () from /lib/s390x-linux-gnu/libc.so.6
|
#5 0x000003ff966442e6 in gu_cond_wait_SYS (mutex=<optimized out>, cond=0x2aa60d36f28) at ./galerautils/src/gu_threads.h:251
|
#6 gu::Lock::wait (this=<synthetic pointer>, this=<synthetic pointer>, cond=...) at ./galerautils/src/gu_lock.hpp:46
|
#7 galera::ReplicatorSMM::wait_for_CLOSED (lock=..., this=<optimized out>) at ./galera/src/replicator_smm.cpp:289
|
#8 galera::ReplicatorSMM::close (this=0x2aa60d36d50) at ./galera/src/replicator_smm.cpp:379
|
#9 0x000003ff966248c4 in galera_disconnect (gh=<optimized out>) at ./galera/src/wsrep_provider.cpp:238
|
#10 0x000002aa2ffb29fe in wsrep::wsrep_provider_v26::disconnect (this=<optimized out>) at /home/buildbot/s390x-ubuntu-2204/build/wsrep-lib/src/wsrep_provider_v26.cpp:866
|
#11 0x000002aa2f6801e6 in unireg_abort (exit_code=<optimized out>) at /home/buildbot/s390x-ubuntu-2204/build/sql/wsrep_server_state.h:41
|
#12 0x000002aa2f6814fc in init_server_components () at /home/buildbot/s390x-ubuntu-2204/build/sql/mysqld.cc:5499
|
#13 0x000002aa2f686f3e in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/s390x-ubuntu-2204/build/sql/mysqld.cc:6021
|
#14 0x000003ff9b52a712 in ?? () from /lib/s390x-linux-gnu/libc.so.6
|
#15 0x000003ff9b52a7f0 in __libc_start_main () from /lib/s390x-linux-gnu/libc.so.6
|
#16 0x000002aa2f67aa00 in _start ()
|
Analysis
The test was rewritten by the fix of MDEV-41553 (commit 744b2ca9988f, https://github.com/MariaDB/server/pull/5898, 2026-10-19), which describes the former try: line of the expect file as silently ignored; the test file and the result file are identical on 10.11, 11.4, 11.8, 12.3, 13.1 and main. The test now starts node 2 directly with --galera-unknown-option twice, once after removing grastate.dat (SST) and once after updates on node 1 (IST), and waits in --exec for exit code 134. In all 56 timeouts the core of the hung mariadbd (node 2) shows the main thread in unireg_abort() waiting in galera::ReplicatorSMM::wait_for_CLOSED (replicator_smm.cpp:289) under wsrep::wsrep_provider_v26::disconnect, an IST receiver waiting in galera::ist::Receiver::run (ist.cpp:355) and the SST joiner thread waiting in wsrep::server_state::sst_received (server_state.cpp:734); the latest run also shows the Galera receive thread waiting in galera::ReplicatorSMM::request_state_transfer (replicator_str.cpp:938). This is the same stack as MDEV-41926, reported for galera.galera_mdev_31517. Between 2026-09-15 and 2026-11-02 the test was recorded failing 209 times; the 56 timeouts (24 on release branches: 10.11 11, 11.4 10, 11.8 3) came first on 2026-10-19 (a PR head) and on a release branch on 2026-10-20 (10.11, four commits above the fix commit), none of the 131 failures recorded from 2026-09-15 to 2026-10-18 is a timeout, and the first and the latest timing-out build of each of the three branches contain the fix commit. On the release branches 12 of the 24 logs end at the first start with the unknown option (the node needs an SST) and 12 at the second (the node needs an IST). Two builders report it, s390x-ubuntu-2204 (17 release rows) and aarch64-debian-12 (7), on two CPU architectures, which points at the test and the server and not at one builder. 12.3, 13.1 and main contain the rewritten test now and record no timeout of it yet.
Created via the mtr-failure-create-bug skill.
Attachments
Issue Links
- is part of
-
MDEV-30172 Galera test case cleanup
-
- Stalled
-
- relates to
-
-
- Closed
-
-
-
- In Progress
-
