galera_duplicate_primary_value hangs until debug-sync timeout (300s) due to single applier thread

Analysis by Claude:

Summary

galera_3nodes.galera_duplicate_primary_value (a big_test) passes but takes
the full --debug-sync-timeout (300s in a normal run) every time it runs. The
test needs two write sets applied concurrently by two appliers on node 1, but
it never sets wsrep-slave-threads, so it defaults to 1. With a single applier
the second write set can never be applied while the first applier is parked at a
debug sync point, the WAIT_FOR wsrep_after_kill_reached_2 times out, and the
test then finishes "green" without ever having exercised the concurrent
brute-force (BF) scenario the test (MDEV-31658) was written to cover.

Affected test

• Suite/test: galera_3nodes.galera_duplicate_primary_value
• Type: big_test, requires debug + debug_sync
• Origin: added by MDEV-31658 ("Deadlock found when trying to get lock during applying")
• Observed: ~300238 ms in a --big-test galera run (vs a few hundred ms expected)

Scenario the test intends to create

Per MDEV-31658, the bug needed *two non-conflicting high-priority BF
transactions holding S-locks on different records at the same time* on one node.
The test sets that up on node 1:

• node_1c: open transaction, insert into t1 values (2,2) – holds a lock on PK=2
• node_1d: open transaction, insert into t1 values (3,3) – holds a lock on PK=3
• node_2: insert into t1 values (2,6) – replicated write set, BF-aborts node_1c on node 1
• node_3: insert into t1 values (3,9) – replicated write set, BF-aborts node_1d on node 1

The two replicated inserts touch different primary keys, so on node 1 they are
independent write sets that must be applied concurrently — that is the state
the fix in lock_rec_has_to_wait_wsrep had to handle.

Root cause

The BF-abort path lock_wait_wsrep_kill contains a debug sync point that, when
armed, makes the applier signal and then block:

now SIGNAL wsrep_after_kill_reached  WAIT_FOR wsrep_after_kill_continue

The test drives it like this (node node_1a):

SET GLOBAL DEBUG_DBUG='+d,wsrep_after_kill';      -- arm #1

-- node_2 insert -> applier-A BF-aborts node_1c, parks at WAIT_FOR wsrep_after_kill_continue

SET DEBUG_SYNC='now WAIT_FOR wsrep_after_kill_reached';   -- OK, applier-A reached it

SET GLOBAL DEBUG_DBUG='+d,wsrep_after_kill_2';    -- arm #2

-- node_3 insert -> expects applier-B to BF-abort node_1d and park at wsrep_after_kill_2

SET DEBUG_SYNC='now WAIT_FOR wsrep_after_kill_reached_2';  -- <-- HANGS

SET DEBUG_SYNC='now SIGNAL wsrep_after_kill_continue';     -- only reached after the WAIT_FOR above

With wsrep-slave-threads at its default of 1 there is only one applier.
Applier-A is parked at wsrep_after_kill_continue (still inside the lock wait,
write set not yet applied). The node_3 write set therefore sits in the apply
queue with no free thread, so wsrep_after_kill_reached_2 is never signalled.
The signal that would release applier-A (wsrep_after_kill_continue) is the
next statement after the WAIT_FOR wsrep_after_kill_reached_2, so the two
sides can never make progress:

node_1a : WAIT_FOR wsrep_after_kill_reached_2   (blocks)

            |  needs node_3 applied

            v

applier   : parked at wsrep_after_kill_continue (only 1 thread)

            ^

            |  released only by the SIGNAL that comes *after* the WAIT_FOR

The deadlock is broken only by --debug-sync-timeout (default 300s in MTR).
A WAIT_FOR that times out does not fail the test — it logs a warning and
continues — so after 300s the test disarms the sync points, signals
wsrep_after_kill_continue, the single applier serially applies both write
sets, both COMMIT}}s get the expected {{ER_LOCK_DEADLOCK (1213), and the test
reports pass. The concurrent-BF state was never created, so the test is also a
false negative: it would not catch a regression of MDEV-31658.

This is deterministic, not flaky — it stalls on every run; it is just rarely
noticed because the test is big_test and still passes.

Confirmation

Reproduced on a 13.1.0 debug build with the galera provider, forcing a short
sync timeout so the stall is obvious. The test time tracks the sync timeout
exactly:

So the 300238 ms seen in the production run = the default 300s sync timeout plus
test overhead.

Tentative fix

Give node 1 a second applier so the two independent write sets apply
concurrently, which is exactly the state the test is meant to create. Add to
mysql-test/suite/galera_3nodes/t/galera_duplicate_primary_value.cnf under
[mysqld.1]:

# The test parks one applier at the wsrep_after_kill sync point while a

# second, independent write set must be applied concurrently to reach

# wsrep_after_kill_2. That requires at least two appliers on node 1; with

# the default of 1 the second write set can never be applied and the test

# stalls until --debug-sync-timeout (300s) before passing vacuously.

wsrep-slave-threads=2

Only node 1 applies the two conflicting write sets, so the setting is only
needed there. With it, the test runs in ~60 ms and now genuinely exercises the
two-concurrent-BF path. Verified above (9/9 fast passes, including 3 runs driven
purely by the .cnf change with no command-line override).

Follow-up worth considering

The sync points wsrep_after_kill / wsrep_after_kill_2 are armed/disarmed
with SET GLOBAL DEBUG_DBUG, which is racy against background applier threads
picking up the keyword. The applier-count fix removes the deterministic stall;
if intermittent misses are seen later, switching the global arming to a more
deterministic per-statement handshake (or asserting that the test never relies
on the --debug-sync-timeout firing) would harden it further.