galera_3nodes.galera_join_with_cc_B fails on wsrep_debug_sync_waiters status check

The test galera_3nodes.galera_join_with_cc_B failed in a Jenkins test-package run.

First attempt failed at 2026-11-09 02:48:24 with a result diff:

 SET SESSION wsrep_on = 0;

 SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_debug_sync_waiters';

 VARIABLE_NAME	VARIABLE_VALUE

-WSREP_DEBUG_SYNC_WAITERS	process_primary_configuration

+WSREP_DEBUG_SYNC_WAITERS	

 SET GLOBAL wsrep_provider_options = 'signal=process_primary_configuration';

 SET GLOBAL wsrep_provider_options = 'dbug=';

 connection node_1;

The test expects the debug sync wait point process_primary_configuration to still show up as an active waiter in wsrep_debug_sync_waiters at that point. The value came back empty instead.

Root cause: a spurious wakeup bug in galera's debug sync helper, in the codership-galera repository, not a replication defect.

gu_debug_sync_wait() in galerautils/src/gu_debug_sync.cpp registers the calling thread in a waiter map, then calls lock.wait(cond) exactly once. gu::Lock::wait() in galerautils/src/gu_lock.hpp wraps a single, unlooped call to pthread_cond_wait(), which POSIX allows to return without any matching signal (a spurious wakeup). Nothing here re-checks a predicate in a loop, so a spurious wakeup is treated the same as a real gu_debug_sync_signal() call. Every other condition variable wait in galerautils loops on the real predicate before returning (gu_to.c, gu_fifo.c); gu_debug_sync_wait() is the one place that does not.

process_prim_conf_change() in galera/src/replicator_smm.cpp calls GU_DBUG_SYNC_WAIT("process_primary_configuration") once per primary configuration change. The test arms this point once and expects to see it block on two separate configuration changes in a row. If a spurious wakeup fires on the second block before the test runs its own SELECT ... wsrep_debug_sync_waiters, the thread resumes and erases its own waiter entry on its own, with no real signal ever sent. The SELECT then finds the map empty, which is the reported diff. An earlier wait_condition.inc poll does not show a timeout, because it genuinely saw the waiter registered before the spurious wakeup released it.

Fix: branch 4.x-MDEV-42139 in codership-galera adds a per-waiter signalled flag and re-checks it in a loop around the wait call, matching the pattern already used in gu_to.c and gu_fifo.c. A new regression test (galerautils/tests/gu_debug_sync_test.cpp) reproduces the defect deterministically: it blocks a thread in gu_debug_sync_wait(), injects a simulated spurious wakeup through a test-only hook, and asserts the waiter stays registered until a real gu_debug_sync_signal() call. The test fails against the pre-fix code and passes with the fix, confirmed over repeated runs.

The retry's worker did not start, so the retry outcome is not known from the Jenkins log:

worker[05] > Restart  - not started

worker[05] > Restart  - not started

worker[05] > Restart  - not started

The same batch logged this warning on several galera_3nodes tests, not only this one:

2026-11-09 2:47:50 0 [Warning] WSREP: Rejecting JOIN message from 0.0 (node1): new State Transfer required.

2026-11-09 2:48:23 0 [Warning] WSREP: Rejecting JOIN message from 0.0 (node1): new State Transfer required.

Tests affected: galera_join_with_cc_A, galera_join_with_cc_B, MDEV-36360, galera_certification_double_failure, galera-features#119, galera_certification_ccc, galera_garbd. This warning looks like pre-existing noise for this suite, not specific to this failure.

Created via the jira-create skill.