Walter Doekes added a comment - 2021-06-03 11:26 Okay, so we notice this too: 2021-06-02 20:58:51 0 [Note] WSREP: Signalling provider to continue. 2021-06-02 20:58:51 0 [Note] WSREP: SST received: 13af80aa-836a-11eb-8c23-32f57cb601d4:54087553 2021-06-02 20:58:51 2 [Note] WSREP: Receiving IST: 1683 writesets, seqnos 54087553-54089236 2021-06-02 20:58:51 0 [Note] WSREP: Receiving IST... 0.0% ( 0/1683 events) complete. 2021-06-02 20:58:51 0 [Note] Reading of all Master_info entries succeeded 2021-06-02 20:58:51 0 [Note] Added new Master_info '' to hash table 2021-06-02 20:58:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.29-MariaDB-1:10.3.29+maria~bionic-log' socket: '/run/mysqld/MY_DEPLOYMENT-mysqld.sock' port: 3306 mariadb.org binary distribution 2021-06-02 20:58:51 0 [ERROR] WSREP: Trx 54088471 tries to abort slave trx 54088527. This could be caused by: 1) unsupported configuration options combination, please check documentation. 2) a bug in the code. 3) a database corruption. Node consistency compromized, need to abort. Restart the node to resync with cluster. It went into a crashloop, with the same: Trx 54088471 tries to abort slave trx 54088527. First core dump looks like: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007f30f06fa9fb in __GI_abort () at abort.c:100 #2 0x000055d85b55d914 in wsrep_abort_slave_trx (bf_seqno=<optimized out>, victim_seqno=<optimized out>) at ./storage/innobase/handler/ha_innodb.cc:18893 #3 0x000055d85b56283c in bg_wsrep_kill_trx (void_arg=0x7f1088023830) at ./storage/innobase/handler/ha_innodb.cc:19058 #4 0x000055d85b1e5a83 in handle_manager (arg=arg@entry=0x0) at ./sql/sql_manager.cc:112 #5 0x000055d85b8ea67a in pfs_spawn_thread (arg=0x55d8c61e5228) at ./storage/perfschema/pfs.cc:1869 #6 0x00007f30f0ab26db in start_thread (arg=0x7f1153399700) at pthread_create.c:463 case QUERY_IDLE: { WSREP_DEBUG("kill IDLE for " TRX_ID_FMT, victim_trx->id);   if (wsrep_thd_exec_mode(thd) == REPL_RECV) { WSREP_DEBUG("kill BF IDLE, seqno: %lld", (long long)wsrep_thd_trx_seqno(thd)); wsrep_abort_slave_trx(arg->bf_seqno, // <-- CRASH wsrep_thd_trx_seqno(thd)); goto ret_unlock; } /* This will lock thd from proceeding after net_read() */ wsrep_thd_set_conflict_state(thd, ABORTING); (gdb) print *arg $2 = {thd_id = 67, trx_id = 7493842400, bf_seqno = 54088471, signal = 1}   (gdb) print *(unsigned*)((void*)thd + 0x58e0) $4 = 54088527   (gdb) print thd->wsrep_trx_meta $22 = {gtid = {uuid = {data = "\023\257\200\252\203j\021\353\214#2\365|\266", <incomplete sequence \324>}, seqno = 54088527}, depends_on = 54088417}   (gdb) print thd->proc_info $23 = 0x7f0fb0006570 "Applied write set 54088527"   (gdb) info locals rcode = <optimized out> arg = 0x7f1088023830 thd = 0x7f0fb0000c08 victim_trx = 0x7f242978d1f8 awake = false (gdb) up #4 0x000055d85b1e5a83 in handle_manager (arg=arg@entry=0x0) at ./sql/sql_manager.cc:112   (gdb) info locals next = 0x0 cb = 0x7f1088023868 error = 0 abstime = {tv_sec = 94387747591648, tv_nsec = 139848169358893} reset_flush_time = true Went back to 10.3.25 because of MDEV-25190 (among others). Let me know if I can get you more debug info from the dump.

Jan Horstmann added a comment - 2021-06-04 12:26 @wdoekes would you consider it to be save to go back from 10.3.29 to 10.3.28 or lower? Did you do a full SST per node and which method do you use?

Walter Doekes added a comment - 2021-06-04 13:55 If you're not affected by MDEV-25190 , then I guess 10.3.28 is fine. We've had a bad streak after 10.3.25, so we're reverting back to that. I'm not aware what ops did when reverting. SSTs generally start automatically here after certain failures, more often than appears necessary. So if there was an SST, I'm not sure if it was intentional either. The crash loop started on the second node, apparently. ("Rebooted, started maria on the 2nd node. Crashloop. Downgrading made it happy again.")

Patrick Schlirf added a comment - 2021-06-25 09:24 Hey guys, are there any updates? We downgraded after the first three times this happened and freeze the patch cycle of a big number of database servers to prevent to hit this bug again. Kind regards, Patrick

Jan Lindström (Inactive) added a comment - 2021-07-26 06:56 - edited This issue will be fixed when (if) MDEV-25114 is reviewed and applied.

Jan Lindström (Inactive) added a comment - 2021-09-20 09:17 I need more information. Can you run with --wsrep-debug=1 and provide error logs ? Naturally, steps how to reproduce would be most useful.

Seppo Jaakola added a comment - 2021-10-20 15:08 This issue seems to cause applier-applier conflicts, configuring for only one applier (wsrep_slave_threads=1) should be immediate help for this. I suspect this is a duplicate of MDEV-24978 , as the table definition has foreign key constraint over multi-byte character set

Seppo Jaakola added a comment - 2021-10-20 15:09 probably a duplicate

Daniel Black added a comment - 2021-10-29 05:20 With 47ba5523046094db33e68c92a182491a629bbd56 reverting MDEV-24978 , is this still closed?

Jan Lindström (Inactive) added a comment - 2021-10-29 08:34 Yes, MDEV-24978 was fixed on d5a15f04f4ab0738b0a5f993b208bcfaad522fd4