Jan Lindström added a comment - 2023-09-28 06:00 marko I could do a PR containing exactly that change. I can't write MariaDB server repositories directly.

Marko Mäkelä added a comment - 2023-09-28 08:07 janlindstrom , I thought that you could apply 10.4-recv_no_log_write.patch (which I just created for you) into a local build and run some stress tests. I do not think that it is feasible to try to reproduce this in a CI environment. I don’t think that this can be analyzed without having a good enough execution trace, such as rr record . First it could be worthwhile to come up with a stress test that increases the failure probability, and then employ rr record . The patch actually triggered one test failure, unrelated to Galera SST: 10.4 4e2594768d658194220ae5678a686ed680aa3217 RelWithDebInfo with patch mariabackup.innodb_xa_rollback 'innodb' w55 [ fail ] … 2023-09-28 11:02:46 0 [Note] InnoDB: 1 transactions in prepared state after recovery 2023-09-28 11:02:46 0x7fcfb1bdb800 InnoDB: Assertion failure in file /mariadb/10.4/storage/innobase/mtr/mtr0mtr.cc line 429 InnoDB: Failing assertion: !m_modifications || !recv_no_log_write This is most likely because I forgot to remove the ut_d() around assignments to recv_no_log_write in extra/mariabackup/xtrabackup.cc .

Jan Lindström added a comment - 2023-09-29 09:54 Same assertion was triggered on CI but same test does not crash locally or running full suite using --rr and stack trace is useless. Thread 1 (Thread 0x7fa3617fa700 (LWP 16225)): #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57 #1 0x0000564798628441 in my_write_core (sig=6) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/mysys/stacktrace.c:386 #2 0x0000564797d44da0 in handle_fatal_signal (sig=6) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/signal_handler.cc:372 #3 <signal handler called> #4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #5 0x00007fa3c2be27f1 in __GI_abort () at abort.c:79 #6 0x00005647983bde1a in ut_dbg_assertion_failed (expr=0x5647989e306b "!recv_no_log_write", file=0x5647989e2c58 "/home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/log/log0log.cc", line=1484) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/ut/ut0dbg.cc:60 #7 0x00005647981f8d9d in log_checkpoint (sync=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/log/log0log.cc:1484 #8 0x00005647981f90ef in log_make_checkpoint () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/log/log0log.cc:1551 #9 0x0000564798121a4a in sst_disable_innodb_writes () at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:1892 #10 0x0000564798121aea in innodb_disable_internal_writes (disable=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:1909 #11 0x0000564797d49be2 in plugin_disable_internal_writes (plugin=0x7fa34c183f20, disable=0x7fa3617f990c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/handler.cc:886 #12 0x0000564797a20594 in plugin_foreach_with_mask (thd=0x0, func=0x564797d49b9e <plugin_disable_internal_writes(THD*, plugin_ref, void*)>, type=1, state_mask=8, arg=0x7fa3617f990c) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/sql_plugin.cc:2521 #13 0x0000564797d49c18 in ha_disable_internal_writes (disable=true) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/handler.cc:893 #14 0x00005647980ed588 in sst_flush_tables (thd=0x7fa338000d50) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/wsrep_sst.cc:1640 #15 0x00005647980edbff in sst_donor_thread (a=0x7fa3bc32ca20) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/sql/wsrep_sst.cc:1738 #16 0x00005647980bd889 in pfs_spawn_thread (arg=0x7fa3a00b4590) at /home/jenkins/workspace/mtr-galera-4.x-mariadb-10.4/mysql-wsrep/storage/perfschema/pfs.cc:1869 #17 0x00007fa3c3add6db in start_thread (arg=0x7fa3617fa700) at pthread_create.c:463 #18 0x00007fa3c2cc361f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Marko Mäkelä added a comment - 2023-09-29 10:19 The assertion failure catches the fact that some log records had been written before the previous log_make_checkpoint() completed. An execution trace would reveal where that offending write happened. Were you able to reproduce this on 10.5 or later major versions? The page flushing and checkpoint logic is much simpler there, because everything is handled by the single buf_flush_page_cleaner thread.

Jan Lindström added a comment - 2023-10-04 13:39 marko After several rounds and history on Jenkings I can say on some confidence this problem does not reproduce 10.5 or later major versions.