Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24660

MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed in MYSQL_BIN_LOG::cleanup on SHUTDOWN

Details

    Description

      I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with rr it was not immediately self evident how to replay this bug. However, running continue; multiple times in rr the bug still reproduces with this rr trace on the rr server:

      cd /data/611902/369/rr/mysqld-0
      rr replay "${PWD}"
      (rr) continue;  -> leads to SIGUSR1 in thread 1
      (rr) continue;  -> leads to SIGUSR1 in thread 24
      (rr) continue;  -> leads to SIGUSR1 in thread 25
      (rr) continue;  -> leads to SIGABRT as described in this ticket
      

      Leads to (rr based trace):

      Thread 1 received signal SIGABRT, Aborted.
      [Switching to Thread 3673578.3673578]
      __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
      (rr) bt
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x00001481826b9859 in __GI_abort () at abort.c:79
      #2  0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
          assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, 
          function=<optimized out>) at assert.c:92
      #3  0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", 
          file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, 
          function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
      #4  0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
          at /test/10.6_dbg/sql/log.cc:3449
      #5  0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
      #6  0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
          at /test/10.6_dbg/sql/mysqld.cc:5728
      #7  0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
      

      Leads to (mysqld based trace):

      10.6.0 9a08fcbf60567992971262ececee8d8429c20756

      mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
      

      10.6.0 9a08fcbf60567992971262ececee8d8429c20756

      Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x0000000070000002 in ?? ()
      [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
      (gdb) bt
      #0  0x0000000070000002 in ?? ()
      #1  0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
      #2  0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
      #3  0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
      #4  syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
      #5  syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
      #6  0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
      #7  0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
      #8  0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
      #9  0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
      #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
      #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
      #12 <signal handler called>
      #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #14 0x00001481826b9859 in __GI_abort () at abort.c:79
      #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
      #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
      #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
      #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
      #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
      #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
      

      From the error log

      10.6.0 9a08fcbf60567992971262ececee8d8429c20756

      END OF INNODB MONITOR OUTPUT
      ============================
      2021-01-22  1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
      2021-01-22  1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
      2021-01-22  1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
      2021-01-22  1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
      2021-01-22  1:11:33 23 [Note] Slave I/O thread killed while connecting to master
      2021-01-22  1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
      2021-01-22  1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
      2021-01-22  1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
      2021-01-22  1:11:33 0 [Note] InnoDB: SYNC words: 0
      2021-01-22  1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
      2021-01-22  1:11:33 0 [Note] InnoDB: to purge 57 transactions
      mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
      210122  1:11:33 [ERROR] mysqld got signal 6 ;
      ...
      Server version: 10.6.0-MariaDB-debug-log
      key_buffer_size=16777216
      read_buffer_size=131072
      max_used_connections=11
      max_threads=153
      thread_count=0
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
      mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
      sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
      ??:0(gsignal)[0x1481826da18b]
      ??:0(abort)[0x1481826b9859]
      /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
      ??:0(__assert_fail)[0x1481826caf36]
      /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
      sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
      sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
      sql/main.cc:26(main)[0x5627705a0b36]
      ??:0(__libc_start_main)[0x1481826bb0b3]
      /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /dev/shm/611902/369/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units
      Max cpu time              unlimited            unlimited            seconds
      Max file size             unlimited            unlimited            bytes
      Max data size             unlimited            unlimited            bytes
      Max stack size            unlimited            unlimited            bytes
      Max core file size        unlimited            unlimited            bytes
      Max resident set          unlimited            unlimited            bytes
      Max processes             unlimited            unlimited            processes
      Max open files            1048576              1048576              files
      Max locked memory         unlimited            unlimited            bytes
      Max address space         unlimited            unlimited            bytes
      Max file locks            unlimited            unlimited            locks
      Max pending signals       unlimited            unlimited            signals
      Max msgqueue size         unlimited            unlimited            bytes
      Max nice priority         0                    0
      Max realtime priority     0                    0
      Max realtime timeout      unlimited            unlimited            us
      

      mysqld options that were used in this run:

      --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
      

      A core dump is also available:

      cd /data/611902/369
      gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
      ...
      Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x0000000070000002 in ?? ()
      [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
      (gdb) f 15
      #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
          assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, 
          function=<optimized out>) at assert.c:92
      (gdb) 
      

      Other versions may or may not be affected. Optimized builds may or may not be affected.
      It could be that this issue is only possible in multi-threaded client setups. May be related to MDEV-742.

      Attachments

        Issue Links

          Activity

            Roel Roel Van de Paar created issue -
            Roel Roel Van de Paar made changes -
            Field Original Value New Value
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to nail down this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb)
            {nofomat}
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb)
            {nofomat}
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb)
            {noformat}
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb)
            {noformat}
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Roel Roel Van de Paar added a comment - - edited

            Is the issue perhaps simple to understand or deduce? For example; server shutdown -> InnoDB attempted further work -> where such work was unexpected and unhandled by binlog code, or similar?

                  /*
                    There should be no pending XIDs at shutdown, and only one entry (for
                    the active binlog file) in the list.
                  */
                  DBUG_ASSERT(b->xid_count == 0);
            

            Roel Roel Van de Paar added a comment - - edited Is the issue perhaps simple to understand or deduce? For example; server shutdown -> InnoDB attempted further work -> where such work was unexpected and unhandled by binlog code, or similar? /* There should be no pending XIDs at shutdown, and only one entry (for the active binlog file) in the list. */ DBUG_ASSERT(b->xid_count == 0);
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Other versions may or may not be affected. Optimized builds may or may not be affected.
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Other versions may or may not be affected. Optimized builds may or may not be affected.
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Other versions may or may not be affected. Optimized builds may or may not be affected.
            It could be that this issue is only possible in multi-threaded client setups.
            Roel Roel Van de Paar made changes -
            Description I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Other versions may or may not be affected. Optimized builds may or may not be affected.
            It could be that this issue is only possible in multi-threaded client setups.
            I have seen this bug many times in the runs, but have never been able to generate a testcase for it. Even with {{rr}} it was not immediately self evident how to replay this bug. However, running {{continue;}} multiple times in {{rr}} the bug still reproduces with this {{rr}} trace on the {{rr}} server:
            {noformat}
            cd /data/611902/369/rr/mysqld-0
            rr replay "${PWD}"
            (rr) continue; -> leads to SIGUSR1 in thread 1
            (rr) continue; -> leads to SIGUSR1 in thread 24
            (rr) continue; -> leads to SIGUSR1 in thread 25
            (rr) continue; -> leads to SIGABRT as described in this ticket
            {noformat}

            Leads to (rr based trace):

            {noformat}
            Thread 1 received signal SIGABRT, Aborted.
            [Switching to Thread 3673578.3673578]
            __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (rr) bt
            #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1 0x00001481826b9859 in __GI_abort () at abort.c:79
            #2 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            #3 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0",
                file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449,
                function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
                at /test/10.6_dbg/sql/log.cc:3449
            #5 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #6 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>)
                at /test/10.6_dbg/sql/mysqld.cc:5728
            #7 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            Leads to (mysqld based trace):

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            {noformat}

            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) bt
            #0 0x0000000070000002 in ?? ()
            #1 0x00001481824513b6 in _raw_syscall () at /build/rr-S0CLEN/rr-5.3.0/src/preload/raw_syscall.S:120
            #2 0x000014818244d2ff in traced_raw_syscall (call=call@entry=0x681fffa0) at ./src/preload/syscallbuf.c:229
            #3 0x000014818244e978 in sys_fcntl (call=<optimized out>) at ./src/preload/syscallbuf.c:1291
            #4 syscall_hook_internal (call=0x681fffa0) at ./src/preload/syscallbuf.c:2855
            #5 syscall_hook (call=0x681fffa0) at ./src/preload/syscallbuf.c:2987
            #6 0x000014818244d1da in _syscall_hook_trampoline () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
            #7 0x000014818244d20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
            #8 0x000014818244d263 in _syscall_hook_trampoline_89_c2_f7_da () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:463
            #9 0x000040d01c1a7f0c in __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
            #10 0x00005627711fdab1 in my_write_core (sig=sig@entry=6) at /test/10.6_dbg/mysys/stacktrace.c:424
            #11 0x0000562770986856 in handle_fatal_signal (sig=6) at /test/10.6_dbg/sql/signal_handler.cc:330
            #12 <signal handler called>
            #13 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #14 0x00001481826b9859 in __GI_abort () at abort.c:79
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449, function=<optimized out>) at assert.c:92
            #16 0x00001481826caf36 in __GI___assert_fail (assertion=assertion@entry=0x562771519387 "b->xid_count == 0", file=file@entry=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=line@entry=3449, function=function@entry=0x562771519399 "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #17 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>) at /test/10.6_dbg/sql/log.cc:3449
            #18 0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            #19 0x00005627705b008f in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/mysqld.cc:5728
            #20 0x00005627705a0b36 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.6_dbg/sql/main.cc:25
            {noformat}

            From the error log
            {noformat:title=10.6.0 9a08fcbf60567992971262ececee8d8429c20756}
            END OF INNODB MONITOR OUTPUT
            ============================
            2021-01-22 1:11:33 0 [Note] /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): *Normal shutdown*
            2021-01-22 1:11:33 0 [Note] Event Scheduler: Purging the queue. 0 events
            2021-01-22 1:11:33 22 [Note] Error reading relay log event: slave SQL thread was killed
            2021-01-22 1:11:33 22 [Note] Slave SQL thread exiting, replication stopped in log 'FIRST' at position 0
            2021-01-22 1:11:33 23 [Note] Slave I/O thread killed while connecting to master
            2021-01-22 1:11:33 23 [Note] Slave I/O thread exiting, read up to log 'FIRST', position 4
            2021-01-22 1:11:33 23 [Note] cannot connect to master to kill slave io_thread's connection
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS SYNC for table `test`.`t-26`, deleted count: 0 size: 0 bytes
            2021-01-22 1:11:33 0 [Note] InnoDB: SYNC words: 0
            2021-01-22 1:11:33 0 [Note] InnoDB: FTS optimize thread exiting.
            2021-01-22 1:11:33 0 [Note] InnoDB: to purge 57 transactions
            mysqld: /test/10.6_dbg/sql/log.cc:3449: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            210122 1:11:33 [ERROR] mysqld got signal 6 ;
            ...
            Server version: 10.6.0-MariaDB-debug-log
            key_buffer_size=16777216
            read_buffer_size=131072
            max_used_connections=11
            max_threads=153
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 36561 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x49000
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(my_print_stacktrace+0x2f)[0x5627711fdcc2]
            mysys/stacktrace.c:212(my_print_stacktrace)[0x562770986599]
            sigaction.c:0(__restore_rt)[0x40d01c1ab3c0]
            ??:0(gsignal)[0x1481826da18b]
            ??:0(abort)[0x1481826b9859]
            /lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x1481826b9729]
            ??:0(__assert_fail)[0x1481826caf36]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x221)[0x562770aedc77]
            sql/log.cc:3450(MYSQL_BIN_LOG::cleanup())[0x5627705a2428]
            sql/mysqld.cc:1964(clean_up(bool))[0x5627705b008f]
            sql/main.cc:26(main)[0x5627705a0b36]
            ??:0(__libc_start_main)[0x1481826bb0b3]
            /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld(_start+0x2e)[0x5627705a0a6e]
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /dev/shm/611902/369/data
            Resource Limits:
            Limit Soft Limit Hard Limit Units
            Max cpu time unlimited unlimited seconds
            Max file size unlimited unlimited bytes
            Max data size unlimited unlimited bytes
            Max stack size unlimited unlimited bytes
            Max core file size unlimited unlimited bytes
            Max resident set unlimited unlimited bytes
            Max processes unlimited unlimited processes
            Max open files 1048576 1048576 files
            Max locked memory unlimited unlimited bytes
            Max address space unlimited unlimited bytes
            Max file locks unlimited unlimited locks
            Max pending signals unlimited unlimited signals
            Max msgqueue size unlimited unlimited bytes
            Max nice priority 0 0
            Max realtime priority 0 0
            Max realtime timeout unlimited unlimited us
            {noformat}
            mysqld options that were used in this run:
            {noformat}
            --no-defaults --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            {noformat}
            A core dump is also available:
            {noformat}
            cd /data/611902/369
            gdb /test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld ./data/core
            ...
            Core was generated by `/test/MD160121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_a'.
            Program terminated with signal SIGABRT, Aborted.
            #0 0x0000000070000002 in ?? ()
            [Current thread is 1 (Thread 0x148182886800 (LWP 3673578))]
            (gdb) f 15
            #15 0x00001481826b9729 in __assert_fail_base (fmt=0x14818284f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
                assertion=0x562771519387 "b->xid_count == 0", file=0x562771518c44 "/test/10.6_dbg/sql/log.cc", line=3449,
                function=<optimized out>) at assert.c:92
            (gdb)
            {noformat}
            Other versions may or may not be affected. Optimized builds may or may not be affected.
            It could be that this issue is only possible in multi-threaded client setups. May be related to MDEV-742.
            Roel Roel Van de Paar made changes -
            Assignee Andrei Elkin [ elkin ]
            Elkin Andrei Elkin added a comment -

            The assert may relate to MDEV-24302, MDEV-24526.
            Roel: It'd be helpful to learn about the load specifics. Could you please provide that somehow?

            Elkin Andrei Elkin added a comment - The assert may relate to MDEV-24302 , MDEV-24526 . Roel : It'd be helpful to learn about the load specifics. Could you please provide that somehow?
            Roel Roel Van de Paar added a comment - - edited

            Here is what I know so far;
            1) The issue can be reproduced using a single thread to the server
            2) None of the error logs I checked (24) have a crashing Query noted for this bug
            3) The issue is hard to reproduce
            4) The issue also exists on 10.5.6 build 24/9/20
            5) We have an `rr` trace which reproduces the issue. We also have multiple core dumps. Both can be analyzed further
            I can help you or one of your team members to get setup with access to rr trace (like gdb) and core dump.
            Please let me know if you have any other questions.

            Roel Roel Van de Paar added a comment - - edited Here is what I know so far; 1) The issue can be reproduced using a single thread to the server 2) None of the error logs I checked (24) have a crashing Query noted for this bug 3) The issue is hard to reproduce 4) The issue also exists on 10.5.6 build 24/9/20 5) We have an `rr` trace which reproduces the issue. We also have multiple core dumps. Both can be analyzed further I can help you or one of your team members to get setup with access to rr trace (like gdb) and core dump. Please let me know if you have any other questions.
            Roel Roel Van de Paar made changes -
            Affects Version/s 10.5 [ 23123 ]
            Roel Roel Van de Paar made changes -
            Fix Version/s 10.5 [ 23123 ]
            Roel Roel Van de Paar added a comment - - edited

            I have a very rough testcase with only 21 lines. I am confident that this testcases could reproduce the said issue at one point in time, against 10.5.6, build 24/9/20, even though I cannot get it to reproduce even a single time now, even against the same build. The testcase may have many bits with are unnecessary/superfluous. For sure, not all of the mysqld options are required; likely less then 2-5 are required, but it not clear which ones. Note that we have a SHUTDOWN just before the creation of an InnoDB table, which indeed seems to correspond much with what we know of this bug so far. Feel free to play around with the testcase. Please ignore ";#...." messages after queries; whilst they are relevant as to the original/first occurrence of the bug, the reduction may have rendered them incorrect and current output can look different due to a different buildup sequence. The only reason I left them is that I did not want to make any changes to a known-to-work (at least at one point) testcase. 100k-500k repeats, replay via pquery or other C-connector based clients may help. Single SQL thread is sufficient. The reduction was done using the pquery client. Also interesting is the binlog_size change, though it is not established that this is necessary for this particular bug. Sequential replay should be sufficient, though random order replay may be required. Hopefully this will shed some light. Elkin fyi.

            # mysqld options used: --log-bin --sql_mode=ONLY_FULL_GROUP_BY --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=256M --innodb_use_native_aio=0 --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_lock_schedule_algorithm=fcfs --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800
            DROP DATABASE test;
            CREATE DATABASE test;
            USE test;
            CREATE TABLE t (a CHAR(65));#ERROR: 1100 - Table 't' was not locked with LOCK TABLES
            create table t1 (c1 char(10), c2 char(10)) engine=RocksDB;#ERROR: 1100 - Table 't1' was not locked with LOCK TABLES
            CREATE TEMPORARY TABLE tmp_myisam_394 (a VARCHAR(256)) ENGINE=MyISAM;#NOERROR
            create table t2 (c1 int);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES
            insert into t2 values (12772+0.33);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES
            create temporary table t1 (a int key) engine=MEMORY;#NOERROR
            insert into t2 values (37591);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES
            drop database if exists db_drop3;#ERROR: 1192 - Can't execute the given command because you have active locked tables or an active transaction
            DROP PROCEDURE IF EXISTS p1;#ERROR: 1192 - Can't execute the given command because you have active locked tables or an active transaction
            insert into t2 values (3871);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES
            DROP TABLES t1, t2;#ERROR: 1051 - Unknown table 'test.t2'
            INSERT INTO t1  VALUES(-8388608,0),(0,0),(8388607,16777215),('-8388608','0'),('8388607','16777215'),(-8388608.0,0.0),(8388607.0,16777215.0);#NOERROR
            SET @@global.max_binlog_size = 4096;#NOERROR
            insert into t1 values (6719,'6719');#ERROR: 1136 - Column count doesn't match value count at row 1
            CREATE TEMPORARY TABLE t1 (c1 INT, INDEX(c1)) ENGINE=MRG_RocksDB UNION=(t1,t2);#ERROR: 1050 - Tabela 't1' ve\0107 postoji
            shutdown;#NOERROR
            CREATE TABLE t1 (id MEDIUMINT NOT NULL AUTO_INCREMENT PRIMARY KEY) ENGINE=innodb;#ERROR: 2006 - MySQL server has gone away
            

            A cleaned up version for readability, but this testcase was never proven to produce the bug even once, whereas the one above was;

            # mysqld options required for replay: --log-bin --sql_mode=ONLY_FULL_GROUP_BY --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=256M --innodb_use_native_aio=0 --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_lock_schedule_algorithm=fcfs --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800
            DROP DATABASE test;
            CREATE DATABASE test;
            USE test;
            CREATE TABLE t (a CHAR(65));
            CREATE TABLE t1 (c1 CHAR(10), c2 CHAR(10)) ENGINE=RocksDB;
            CREATE TEMPORARY TABLE tmp_myisam_394 (a VARCHAR(256)) ENGINE=MyISAM;
            CREATE TABLE t2 (c1 INT);
            INSERT INTO t2 VALUES (12772+0.33);
            CREATE TEMPORARY TABLE t1 (a INT KEY) ENGINE=MEMORY;
            INSERT INTO t2 VALUES (37591);
            DROP DATABASE IF EXISTS db_drop3;
            DROP PROCEDURE IF EXISTS p1;
            INSERT INTO t2 VALUES (3871);
            DROP TABLES t1, t2;
            INSERT INTO t1 VALUES (-8388608,0), (0,0), (8388607,16777215), ('-8388608','0'), ('8388607','16777215'), (-8388608.0,0.0), (8388607.0,16777215.0);
            SET GLOBAL max_binlog_size = 4096;
            INSERT INTO t1 VALUES (6719,'6719');
            CREATE TEMPORARY TABLE t1 (c1 INT, INDEX (c1)) ENGINE=mrg_rocksdb UNION= (t1,t2);
            SHUTDOWN;
            CREATE TABLE t1 (id MEDIUMINT NOT NULL AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB;
            

            Roel Roel Van de Paar added a comment - - edited I have a very rough testcase with only 21 lines. I am confident that this testcases could reproduce the said issue at one point in time, against 10.5.6, build 24/9/20, even though I cannot get it to reproduce even a single time now, even against the same build. The testcase may have many bits with are unnecessary/superfluous. For sure, not all of the mysqld options are required; likely less then 2-5 are required, but it not clear which ones. Note that we have a SHUTDOWN just before the creation of an InnoDB table, which indeed seems to correspond much with what we know of this bug so far. Feel free to play around with the testcase. Please ignore ";#...." messages after queries; whilst they are relevant as to the original/first occurrence of the bug, the reduction may have rendered them incorrect and current output can look different due to a different buildup sequence. The only reason I left them is that I did not want to make any changes to a known-to-work (at least at one point) testcase. 100k-500k repeats, replay via pquery or other C-connector based clients may help. Single SQL thread is sufficient. The reduction was done using the pquery client. Also interesting is the binlog_size change, though it is not established that this is necessary for this particular bug. Sequential replay should be sufficient, though random order replay may be required. Hopefully this will shed some light. Elkin fyi. # mysqld options used: --log-bin --sql_mode=ONLY_FULL_GROUP_BY --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=256M --innodb_use_native_aio=0 --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_lock_schedule_algorithm=fcfs --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 DROP DATABASE test; CREATE DATABASE test; USE test; CREATE TABLE t (a CHAR(65));#ERROR: 1100 - Table 't' was not locked with LOCK TABLES create table t1 (c1 char(10), c2 char(10)) engine=RocksDB;#ERROR: 1100 - Table 't1' was not locked with LOCK TABLES CREATE TEMPORARY TABLE tmp_myisam_394 (a VARCHAR(256)) ENGINE=MyISAM;#NOERROR create table t2 (c1 int);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES insert into t2 values (12772+0.33);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES create temporary table t1 (a int key) engine=MEMORY;#NOERROR insert into t2 values (37591);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES drop database if exists db_drop3;#ERROR: 1192 - Can't execute the given command because you have active locked tables or an active transaction DROP PROCEDURE IF EXISTS p1;#ERROR: 1192 - Can't execute the given command because you have active locked tables or an active transaction insert into t2 values (3871);#ERROR: 1100 - Table 't2' was not locked with LOCK TABLES DROP TABLES t1, t2;#ERROR: 1051 - Unknown table 'test.t2' INSERT INTO t1 VALUES(-8388608,0),(0,0),(8388607,16777215),('-8388608','0'),('8388607','16777215'),(-8388608.0,0.0),(8388607.0,16777215.0);#NOERROR SET @@global.max_binlog_size = 4096;#NOERROR insert into t1 values (6719,'6719');#ERROR: 1136 - Column count doesn't match value count at row 1 CREATE TEMPORARY TABLE t1 (c1 INT, INDEX(c1)) ENGINE=MRG_RocksDB UNION=(t1,t2);#ERROR: 1050 - Tabela 't1' ve\0107 postoji shutdown;#NOERROR CREATE TABLE t1 (id MEDIUMINT NOT NULL AUTO_INCREMENT PRIMARY KEY) ENGINE=innodb;#ERROR: 2006 - MySQL server has gone away A cleaned up version for readability, but this testcase was never proven to produce the bug even once, whereas the one above was; # mysqld options required for replay: --log-bin --sql_mode=ONLY_FULL_GROUP_BY --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=256M --innodb_use_native_aio=0 --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_lock_schedule_algorithm=fcfs --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 DROP DATABASE test; CREATE DATABASE test; USE test; CREATE TABLE t (a CHAR(65)); CREATE TABLE t1 (c1 CHAR(10), c2 CHAR(10)) ENGINE=RocksDB; CREATE TEMPORARY TABLE tmp_myisam_394 (a VARCHAR(256)) ENGINE=MyISAM; CREATE TABLE t2 (c1 INT); INSERT INTO t2 VALUES (12772+0.33); CREATE TEMPORARY TABLE t1 (a INT KEY) ENGINE=MEMORY; INSERT INTO t2 VALUES (37591); DROP DATABASE IF EXISTS db_drop3; DROP PROCEDURE IF EXISTS p1; INSERT INTO t2 VALUES (3871); DROP TABLES t1, t2; INSERT INTO t1 VALUES (-8388608,0), (0,0), (8388607,16777215), ('-8388608','0'), ('8388607','16777215'), (-8388608.0,0.0), (8388607.0,16777215.0); SET GLOBAL max_binlog_size = 4096; INSERT INTO t1 VALUES (6719,'6719'); CREATE TEMPORARY TABLE t1 (c1 INT, INDEX (c1)) ENGINE=mrg_rocksdb UNION= (t1,t2); SHUTDOWN; CREATE TABLE t1 (id MEDIUMINT NOT NULL AUTO_INCREMENT PRIMARY KEY) ENGINE=InnoDB;
            Roel Roel Van de Paar made changes -
            Labels rr-profile affects-tests rr-profile
            Roel Roel Van de Paar added a comment - - edited

            Another testcase which I was able to reduce to 28 lines. Non-cleaned up version which was sure to produce the issue at least once:

            # mysqld options required for replay: --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            CREATE DATABASE transforms;
            DROP DATABASE test;
            CREATE DATABASE test;
            USE test;
            SET @@session.tx_isolation= 'READ-COMMITTED';#ERROR: 1231 - Variable 'tx_isolation' can't be set to the value of 'READ-COMMITTED'
            CREATE TABLE t1 (id int PRIMARY KEY, b char(3), INDEX(b));#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            CREATE TABLE ti (a TINYINT UNSIGNED, b BIGINT UNSIGNED, c BINARY(6) NOT NULL, d VARBINARY(99) NOT NULL, e VARBINARY(54) NOT NULL, f VARBINARY(77) NOT NULL, g LONGBLOB NOT NULL, h BLOB NOT NULL, id BIGINT NOT NULL, KEY(b), KEY(e), PRIMARY KEY(id)) ENGINE=RocksDB;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            CREATE TEMPORARY TABLE t1 (a INT, b INT);#ERROR: 1050 - Tabela 't1' već postoji
            set global max_binlog_size=100;#ERROR: 1231 - Variable 'max_binlog_size' can't be set to the value of '100'
            INSERT INTO ti VALUES (161,11484719563904639465,'9n9dSoKcYLJzJPeyPEmmRiqNSrYlV0EjP','t4u2XPQ4KW6BJLaMayzsGESjAy8vrEU4Ja','wtQbaDgPzbqJivpDBQLjBxDXptqExVW8rMS5r2TNrqgVILShXgY1pT837cJt','SyN2O98mFGxkxvvF8MT8LzsPG8V5GmGIdD1qIYsIZKHi039gpejoWdoZ6Wk1JvBOwrwNxwER1Um3GhiJKgMEoEN6cyhuvD79wEgp8SBuG','d','cXE',6);#ERROR: 1264 - Out of range value for column 'b' at row 1
            CREATE TABLE test_repair_table ( val integer not null ) ENGINE = CSV;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            RENAME TABLE t1 TO tt1tt1;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            INSERT INTO ti VALUES (569784834,17686,'lT79eTvuYN0CBnamoxazOjEtuEZ8R2p','eHbtzcUVbWl7NUfruAg40Vi2WxKQ4mfhjYMoY2uLy8QuXDgNeNppunOBQX00EE7BecTDJklwodwWp1tc8TvLHhdyyP9wmwOFGlZzgyp2wS6uBjEiW0NrBo01tUO91EL1v0xwvGOBtXzx4Ed2nBnoSZplnHsa8Ikfm2Kohuv62iX8c0wJSr3qeSEKD9jvHH8QW5RDIytCRnd','EZ3EXKSQ48vGowMUpQSVzz723HG','ZLuDwRZb7v17gTSa3yvEVJiWYPwmOI8ZM4Selb7f6zagtMyc5N8NQ4N489uqRyHCz76dOeva00z7WTCkDvcPPmQBtB9ibZu4GJRdcQKvtOe59tLTOTIBav8clCtzLvKGbqPkJl847Aj3UpbguYOwjO3uS7gz7Tm8UtsP3Gr5pnOu7TdvwGY7lmDrRVZM0amM45FsCQe5PRN','c','S',4);#ERROR: 1264 - Out of range value for column 'a' at row 1
            CREATE DATABASE RocksDB_test;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            CREATE FUNCTION bug48766 () RETURNS ENUM('а','б','в','г') CHARACTER SET ucs2 RETURN 0;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            DROP PROCEDURE IF EXISTS spexecute11;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            CREATE USER test_2@localhost;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            INSERT INTO ti VALUES (1894749719158306246,-2519559641696011780,'Ab75','hxHCQ8ESQs1Y9pOxc1DRm6bIeyR5scaCBSoIP0PzTtr7v6NFVvhdO9MbVUyT9l7gSdf8maFEhwF8YBD5PQ1TMn4PLxucQ4l7pPJiFgV7ByQT1k3ICNU36sBCqzsMu','6K98fqdjH2HNGWSaFGugTiFunW0txxTHoFRbDEIkXy3','17G3zliWOzQXA4JXM','0','B',3);#ERROR: 1264 - Out of range value for column 'a' at row 1
            CREATE TABLE TokuDB (a INT) ENGINE = TokuDB;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            DROP TABLE IF EXISTS `t2`;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            CREATE TABLE t10(c1 INT, c2 char(20)) ENGINE = InnoDB;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            CREATE TABLE IF NOT EXISTS `龗龗龗`(`丄丄丄` char(1)) DEFAULT CHARSET = utf8 engine=RocksDB;#ERROR: 1300 - Invalid big5 character string: '\xE9\xBE\x97\xE9\xBE\x97\xE9\xBE\x97'
            grant select on RocksDBtest.* to RocksDBtest_1@localhost;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            drop table if exists `T9a`;#ERROR: 1300 - Invalid big5 character string: '\xEF\xBC\xB4\xEF\xBC\x99a'
            replace INTO t1  values ('cc', '-cc');#ERROR: 1136 - Broj kolona ne odgovara broju vrednosti u slogu 1
            DROP PROCEDURE IF EXISTS p1;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the  ACTIVE state
            SELECT SLEEP(3);
            

            A cleaned up version for readability, but this testcase was never proven to produce the bug even once, whereas the one above was;

            # mysqld options required for replay: --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M
            CREATE DATABASE transforms;
            DROP DATABASE test;
            CREATE DATABASE test;
            USE test;
            SET SESSION tx_isolation= 'READ-COMMITTED';
            CREATE TABLE t1 (id INT PRIMARY KEY, b CHAR(3), INDEX (b));
            CREATE TABLE ti (a TINYINT UNSIGNED, b BIGINT UNSIGNED, c BINARY (6) NOT NULL, d VARBINARY (99) NOT NULL, e VARBINARY (54) NOT NULL, f VARBINARY (77) NOT NULL, g LONGBLOB NOT NULL, h BLOB NOT NULL, id BIGINT NOT NULL, KEY(b), KEY(e), PRIMARY KEY(id)) ENGINE=RocksDB;
            CREATE TEMPORARY TABLE t1 (a INT, b INT);
            SET GLOBAL max_binlog_size=100;
            INSERT INTO ti VALUES(161,11484719563904639465,'9n9dSoKcYLJzJPeyPEmmRiqNSrYlV0EjP','t4u2XPQ4KW6BJLaMayzsGESjAy8vrEU4Ja','wtQbaDgPzbqJivpDBQLjBxDXptqExVW8rMS5r2TNrqgVILShXgY1pT837cJt','SyN2O98mFGxkxvvF8MT8LzsPG8V5GmGIdD1qIYsIZKHi039gpejoWdoZ6Wk1JvBOwrwNxwER1Um3GhiJKgMEoEN6cyhuvD79wEgp8SBuG','d','cXE',6);
            CREATE TABLE test_repair_table (val INT NOT NULL) ENGINE=CSV;
            RENAME TABLE t1 TO tt1tt1;
            INSERT INTO ti VALUES(569784834,17686,'lT79eTvuYN0CBnamoXAzOjEtuEZ8R2p','eHbtzcUVbWl7NUfruAg40Vi2WxKQ4mfhjYMoY2uLy8QuXDgNeNppunOBQX00EE7BecTDJklwodwWp1tc8TvLHhdyyP9wmwOFGlZzgyp2wS6uBjEiW0NrBo01tUO91EL1v0xwvGOBtXzx4Ed2nBnoSZplnHsa8Ikfm2Kohuv62iX8c0wJSr3qeSEKD9jvHH8QW5RDIytCRnd','EZ3EXKSQ48vGowMUpQSVzz723HG','ZLuDwRZb7v17gTSa3yvEVJiWYPwmOI8ZM4Selb7f6zagtMyc5N8NQ4N489uqRyHCz76dOeva00z7WTCkDvcPPmQBtB9ibZu4GJRdcQKvtOe59tLTOTIBav8clCtzLvKGbqPkJl847Aj3UpbguYOwjO3uS7gz7Tm8UtsP3Gr5pnOu7TdvwGY7lmDrRVZM0amM45FsCQe5PRN','c','S',4);
            CREATE DATABASE rocksdb_test;
            CREATE FUNCTION bug48766() RETURNS ENUM ('а','б','в','г') CHARACTER SET ucs2 RETURN 0;
            DROP PROCEDURE IF EXISTS spexecute11;
            CREATE USER test_2@localhost;
            INSERT INTO ti VALUES(1894749719158306246,-2519559641696011780,'Ab75','hxHCQ8ESQs1Y9pOxc1DRm6bIeyR5scaCBSoIP0PzTtr7v6NFVvhdO9MbVUyT9l7gSdf8maFEhwF8YBD5PQ1TMn4PLxucQ4l7pPJiFgV7ByQT1k3ICNU36sBCqzsMu','6K98fqdjH2HNGWSaFGugTiFunW0txxTHoFRbDEIkXy3','17G3zliWOzQXA4JXM','0','B',3);
            CREATE TABLE TokuDB (a INT) ENGINE=TokuDB;
            DROP TABLE IF EXISTS t2;
            CREATE TABLE t10 (c1 INT, c2 CHAR(20)) ENGINE=InnoDB;
            CREATE TABLE IF NOT EXISTS 龗龗龗 (丄丄丄 CHAR(1)) DEFAULT CHARSET=utf8 ENGINE=RocksDB;
            GRANT SELECT ON RocksDBtest.* TO rocksdbtest_1@localhost;
            DROP TABLE IF EXISTS T9a;
            REPLACE INTO t1 VALUES('cc', '-cc');
            DROP PROCEDURE IF EXISTS p1;
            SELECT SLEEP (3);
            

            Roel Roel Van de Paar added a comment - - edited Another testcase which I was able to reduce to 28 lines. Non-cleaned up version which was sure to produce the issue at least once: # mysqld options required for replay: --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M CREATE DATABASE transforms; DROP DATABASE test; CREATE DATABASE test; USE test; SET @@session.tx_isolation= 'READ-COMMITTED';#ERROR: 1231 - Variable 'tx_isolation' can't be set to the value of 'READ-COMMITTED' CREATE TABLE t1 (id int PRIMARY KEY, b char(3), INDEX(b));#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state CREATE TABLE ti (a TINYINT UNSIGNED, b BIGINT UNSIGNED, c BINARY(6) NOT NULL, d VARBINARY(99) NOT NULL, e VARBINARY(54) NOT NULL, f VARBINARY(77) NOT NULL, g LONGBLOB NOT NULL, h BLOB NOT NULL, id BIGINT NOT NULL, KEY(b), KEY(e), PRIMARY KEY(id)) ENGINE=RocksDB;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state CREATE TEMPORARY TABLE t1 (a INT, b INT);#ERROR: 1050 - Tabela 't1' već postoji set global max_binlog_size=100;#ERROR: 1231 - Variable 'max_binlog_size' can't be set to the value of '100' INSERT INTO ti VALUES (161,11484719563904639465,'9n9dSoKcYLJzJPeyPEmmRiqNSrYlV0EjP','t4u2XPQ4KW6BJLaMayzsGESjAy8vrEU4Ja','wtQbaDgPzbqJivpDBQLjBxDXptqExVW8rMS5r2TNrqgVILShXgY1pT837cJt','SyN2O98mFGxkxvvF8MT8LzsPG8V5GmGIdD1qIYsIZKHi039gpejoWdoZ6Wk1JvBOwrwNxwER1Um3GhiJKgMEoEN6cyhuvD79wEgp8SBuG','d','cXE',6);#ERROR: 1264 - Out of range value for column 'b' at row 1 CREATE TABLE test_repair_table ( val integer not null ) ENGINE = CSV;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state RENAME TABLE t1 TO tt1tt1;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state INSERT INTO ti VALUES (569784834,17686,'lT79eTvuYN0CBnamoxazOjEtuEZ8R2p','eHbtzcUVbWl7NUfruAg40Vi2WxKQ4mfhjYMoY2uLy8QuXDgNeNppunOBQX00EE7BecTDJklwodwWp1tc8TvLHhdyyP9wmwOFGlZzgyp2wS6uBjEiW0NrBo01tUO91EL1v0xwvGOBtXzx4Ed2nBnoSZplnHsa8Ikfm2Kohuv62iX8c0wJSr3qeSEKD9jvHH8QW5RDIytCRnd','EZ3EXKSQ48vGowMUpQSVzz723HG','ZLuDwRZb7v17gTSa3yvEVJiWYPwmOI8ZM4Selb7f6zagtMyc5N8NQ4N489uqRyHCz76dOeva00z7WTCkDvcPPmQBtB9ibZu4GJRdcQKvtOe59tLTOTIBav8clCtzLvKGbqPkJl847Aj3UpbguYOwjO3uS7gz7Tm8UtsP3Gr5pnOu7TdvwGY7lmDrRVZM0amM45FsCQe5PRN','c','S',4);#ERROR: 1264 - Out of range value for column 'a' at row 1 CREATE DATABASE RocksDB_test;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state CREATE FUNCTION bug48766 () RETURNS ENUM('а','б','в','г') CHARACTER SET ucs2 RETURN 0;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state DROP PROCEDURE IF EXISTS spexecute11;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state CREATE USER test_2@localhost;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state INSERT INTO ti VALUES (1894749719158306246,-2519559641696011780,'Ab75','hxHCQ8ESQs1Y9pOxc1DRm6bIeyR5scaCBSoIP0PzTtr7v6NFVvhdO9MbVUyT9l7gSdf8maFEhwF8YBD5PQ1TMn4PLxucQ4l7pPJiFgV7ByQT1k3ICNU36sBCqzsMu','6K98fqdjH2HNGWSaFGugTiFunW0txxTHoFRbDEIkXy3','17G3zliWOzQXA4JXM','0','B',3);#ERROR: 1264 - Out of range value for column 'a' at row 1 CREATE TABLE TokuDB (a INT) ENGINE = TokuDB;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state DROP TABLE IF EXISTS `t2`;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state CREATE TABLE t10(c1 INT, c2 char(20)) ENGINE = InnoDB;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state CREATE TABLE IF NOT EXISTS `龗龗龗`(`丄丄丄` char(1)) DEFAULT CHARSET = utf8 engine=RocksDB;#ERROR: 1300 - Invalid big5 character string: '\xE9\xBE\x97\xE9\xBE\x97\xE9\xBE\x97' grant select on RocksDBtest.* to RocksDBtest_1@localhost;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state drop table if exists `T9a`;#ERROR: 1300 - Invalid big5 character string: '\xEF\xBC\xB4\xEF\xBC\x99a' replace INTO t1 values ('cc', '-cc');#ERROR: 1136 - Broj kolona ne odgovara broju vrednosti u slogu 1 DROP PROCEDURE IF EXISTS p1;#ERROR: 1399 - XAER_RMFAIL: The command cannot be executed when global transaction is in the ACTIVE state SELECT SLEEP(3); A cleaned up version for readability, but this testcase was never proven to produce the bug even once, whereas the one above was; # mysqld options required for replay: --max_allowed_packet=33554432 --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_stats_persistent=off --loose-idle_write_transaction_timeout=0 --loose-idle_transaction_timeout=0 --loose-idle_readonly_transaction_timeout=0 --connect_timeout=60 --interactive_timeout=28800 --slave_net_timeout=60 --net_read_timeout=30 --net_write_timeout=60 --loose-table_lock_wait_timeout=50 --wait_timeout=28800 --lock-wait-timeout=86400 --innodb-lock-wait-timeout=50 --log_output=FILE --log-bin --log_bin_trust_function_creators=1 --loose-max-statement-time=30 --loose-debug_assert_on_not_freed_memory=0 --innodb-buffer-pool-size=300M CREATE DATABASE transforms; DROP DATABASE test; CREATE DATABASE test; USE test; SET SESSION tx_isolation= 'READ-COMMITTED'; CREATE TABLE t1 (id INT PRIMARY KEY, b CHAR(3), INDEX (b)); CREATE TABLE ti (a TINYINT UNSIGNED, b BIGINT UNSIGNED, c BINARY (6) NOT NULL, d VARBINARY (99) NOT NULL, e VARBINARY (54) NOT NULL, f VARBINARY (77) NOT NULL, g LONGBLOB NOT NULL, h BLOB NOT NULL, id BIGINT NOT NULL, KEY(b), KEY(e), PRIMARY KEY(id)) ENGINE=RocksDB; CREATE TEMPORARY TABLE t1 (a INT, b INT); SET GLOBAL max_binlog_size=100; INSERT INTO ti VALUES(161,11484719563904639465,'9n9dSoKcYLJzJPeyPEmmRiqNSrYlV0EjP','t4u2XPQ4KW6BJLaMayzsGESjAy8vrEU4Ja','wtQbaDgPzbqJivpDBQLjBxDXptqExVW8rMS5r2TNrqgVILShXgY1pT837cJt','SyN2O98mFGxkxvvF8MT8LzsPG8V5GmGIdD1qIYsIZKHi039gpejoWdoZ6Wk1JvBOwrwNxwER1Um3GhiJKgMEoEN6cyhuvD79wEgp8SBuG','d','cXE',6); CREATE TABLE test_repair_table (val INT NOT NULL) ENGINE=CSV; RENAME TABLE t1 TO tt1tt1; INSERT INTO ti VALUES(569784834,17686,'lT79eTvuYN0CBnamoXAzOjEtuEZ8R2p','eHbtzcUVbWl7NUfruAg40Vi2WxKQ4mfhjYMoY2uLy8QuXDgNeNppunOBQX00EE7BecTDJklwodwWp1tc8TvLHhdyyP9wmwOFGlZzgyp2wS6uBjEiW0NrBo01tUO91EL1v0xwvGOBtXzx4Ed2nBnoSZplnHsa8Ikfm2Kohuv62iX8c0wJSr3qeSEKD9jvHH8QW5RDIytCRnd','EZ3EXKSQ48vGowMUpQSVzz723HG','ZLuDwRZb7v17gTSa3yvEVJiWYPwmOI8ZM4Selb7f6zagtMyc5N8NQ4N489uqRyHCz76dOeva00z7WTCkDvcPPmQBtB9ibZu4GJRdcQKvtOe59tLTOTIBav8clCtzLvKGbqPkJl847Aj3UpbguYOwjO3uS7gz7Tm8UtsP3Gr5pnOu7TdvwGY7lmDrRVZM0amM45FsCQe5PRN','c','S',4); CREATE DATABASE rocksdb_test; CREATE FUNCTION bug48766() RETURNS ENUM ('а','б','в','г') CHARACTER SET ucs2 RETURN 0; DROP PROCEDURE IF EXISTS spexecute11; CREATE USER test_2@localhost; INSERT INTO ti VALUES(1894749719158306246,-2519559641696011780,'Ab75','hxHCQ8ESQs1Y9pOxc1DRm6bIeyR5scaCBSoIP0PzTtr7v6NFVvhdO9MbVUyT9l7gSdf8maFEhwF8YBD5PQ1TMn4PLxucQ4l7pPJiFgV7ByQT1k3ICNU36sBCqzsMu','6K98fqdjH2HNGWSaFGugTiFunW0txxTHoFRbDEIkXy3','17G3zliWOzQXA4JXM','0','B',3); CREATE TABLE TokuDB (a INT) ENGINE=TokuDB; DROP TABLE IF EXISTS t2; CREATE TABLE t10 (c1 INT, c2 CHAR(20)) ENGINE=InnoDB; CREATE TABLE IF NOT EXISTS 龗龗龗 (丄丄丄 CHAR(1)) DEFAULT CHARSET=utf8 ENGINE=RocksDB; GRANT SELECT ON RocksDBtest.* TO rocksdbtest_1@localhost; DROP TABLE IF EXISTS T9a; REPLACE INTO t1 VALUES('cc', '-cc'); DROP PROCEDURE IF EXISTS p1; SELECT SLEEP (3);

            We now have 8 rr traces + cores for this bug.

            Roel Roel Van de Paar added a comment - We now have 8 rr traces + cores for this bug.
            Elkin Andrei Elkin added a comment -

            Roel, howdy. I am refreshing my understanding of the case and apparently I need to explore the rr traces. Before to jump to that, I could not understand refinements of how-to-reproduce that you left in the last comments.
            The description shows a stack with MYSQL_BIN_LOG::cleanup which is caused by mysql> shutdown;. I don't see the latter in the above https://jira.mariadb.org/browse/MDEV-24660?focusedCommentId=179811&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-179811 .

            Could you please explain?

            Cheers, Andrei.

            Elkin Andrei Elkin added a comment - Roel , howdy. I am refreshing my understanding of the case and apparently I need to explore the rr traces. Before to jump to that, I could not understand refinements of how-to-reproduce that you left in the last comments. The description shows a stack with MYSQL_BIN_LOG::cleanup which is caused by mysql> shutdown; . I don't see the latter in the above https://jira.mariadb.org/browse/MDEV-24660?focusedCommentId=179811&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-179811 . Could you please explain? Cheers, Andrei.
            Elkin Andrei Elkin added a comment - - edited

            Roel,
            First, still to the 2nd of https://jira.mariadb.org/browse/MDEV-24660?focusedCommentId=179811&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-179811 paste,
            it just can not cause the stack of the description, which runs:

            #4  0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup (this=0x562771cd43c0 <mysql_bin_log>)
             
                at /test/10.6_dbg/sql/log.cc:3449
             
            #5  0x00005627705a2428 in clean_up (print_message=print_message@entry=true) at /test/10.6_dbg/sql/mysqld.cc:1962
            
            

            So I feel it must've meant something else than the description stack happened to you..

            Secondly, I have not gotten yet the rr server access. I remember Marko e-mailed that, I should find.
            But before I do, I'd suggest we wait for MDEV-24302 fixes be tested to try again with MDEV-24302 fixes pushed to 10.5 to reproduce the description issue.
            There's a good chance that issue was related.
            Could you please arrange that for one last time?

            Elkin Andrei Elkin added a comment - - edited Roel , First, still to the 2nd of https://jira.mariadb.org/browse/MDEV-24660?focusedCommentId=179811&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-179811 paste, it just can not cause the stack of the description, which runs: # 4 0x0000562770aedc77 in MYSQL_BIN_LOG::cleanup ( this = 0x562771cd43c0 <mysql_bin_log>)   at /test/ 10 .6_dbg/sql/log.cc: 3449   # 5 0x00005627705a2428 in clean_up (print_message=print_message @entry = true ) at /test/ 10 .6_dbg/sql/mysqld.cc: 1962 So I feel it must've meant something else than the description stack happened to you.. Secondly, I have not gotten yet the rr server access. I remember Marko e-mailed that, I should find. But before I do, I'd suggest we wait for MDEV-24302 fixes be tested to try again with MDEV-24302 fixes pushed to 10.5 to reproduce the description issue. There's a good chance that issue was related. Could you please arrange that for one last time?
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Roel Van de Paar [ roel ]
            Roel Roel Van de Paar made changes -
            Labels affects-tests rr-profile affects-tests need_feedback rr-profile
            Roel Roel Van de Paar made changes -
            Comment [ Added need_feedback to remind myself to check this regularly against runs. I have disabled the filter. ]
            Roel Roel Van de Paar made changes -
            Labels affects-tests need_feedback rr-profile affects-tests rr-profile
            Roel Roel Van de Paar added a comment - - edited

            OK, after a lot of work on the testcase of another occurrence I have finally got more to report on this one.

            Firstly, the only option which seems to be required is --log-bin.

            Secondly, this testcase can reproduce the issue using reducer as a base run tool + pquery (in other words; a C/C++ based client connection seems to be required and potentially this may not replay from the CLI):

            # mysqld options required for replay: --log-bin 
            CREATE DATABASE transforms;
            DROP DATABASE test;
            CREATE DATABASE test;
            USE test;
            SET sql_mode='';
            CREATE TEMPORARY TABLE `���`(`���` char(1)) DEFAULT CHARSET=sjis engine=InnoDB;#NOERROR
            delete from mysql.user where user='user1' or user='user2';#NOERROR
            CREATE TABLE t1(c1 char(1)) DEFAULT CHARSET=ujis ENGINE=InnoDB;#ERROR: 1050 - Table 't1' already exists
            INSERT INTO t1 VALUES(17509);#NOERROR
            DELETE FROM t1;#NOERROR
            create table t2 (a int auto_increment,b int,PRIMARY KEY (a)) ENGINE=InnoDB;#NOERROR
            INSERT INTO t1 VALUES (-123.456e0);#NOERROR
            LOAD DATA INFILE '../../tmp/proc.txt' INTO TABLE InnoDB.proc;#ERROR: 1100 - Table''was not locked with LOCK TABLES
            CREATE TABLE `アアア`(`カカカ` char(1)) DEFAULT CHARSET=utf8 engine=InnoDB;#ERROR: 1100 - Table 'アアア' was not locked with LOCK TABLES
            INSERT INTO t1 VALUES(0xADBF);#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction
            CREATE TABLE ti (a SMALLINT UNSIGNED,b BIGINT UNSIGNED ,c CHAR(32) ,d VARCHAR(31) ,e VARCHAR(31),f VARBINARY(58),g TINYBLOB,h BLOB ,id BIGINT ,KEY(b),KEY(e),PRIMARY KEY(id)) ENGINE=InnoDB;#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction
            insert into t1 values (-1),(-2),(-3);#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction
            INSERT INTO ti VALUES (4023595403684610118,7836,'7swVX','xWYjZYSkX0P','JJgE','so18BIYlJMt8r05JqWP3Q7e5i8xMnZyp','1','7',11);#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction
            SET @@global.max_binlog_size=4095;#NOERROR
            shutdown;#NOERROR
            

            The testcase is highly sporadic and requires continuous startup/shutdowns of mysqld.

            Thirdly, any attempts at removing the # comments (which are no longer correct; they are from the original run) have failed thus far. As such, there is a suspicion that "Exact length" is required, and I note the max_binlog_size=4095, present in other ways in the other testcases also. This may be an 'off by one' issue or similar.

            Given that there are rr traces available (and additional ones can be produced at will), there would seem to be sufficient information for this bug to be analyzed further by Elkin.

            Roel Roel Van de Paar added a comment - - edited OK, after a lot of work on the testcase of another occurrence I have finally got more to report on this one. Firstly, the only option which seems to be required is --log-bin . Secondly, this testcase can reproduce the issue using reducer as a base run tool + pquery (in other words; a C/C++ based client connection seems to be required and potentially this may not replay from the CLI): # mysqld options required for replay: --log-bin CREATE DATABASE transforms; DROP DATABASE test; CREATE DATABASE test; USE test; SET sql_mode=''; CREATE TEMPORARY TABLE `���`(`���` char(1)) DEFAULT CHARSET=sjis engine=InnoDB;#NOERROR delete from mysql.user where user='user1' or user='user2';#NOERROR CREATE TABLE t1(c1 char(1)) DEFAULT CHARSET=ujis ENGINE=InnoDB;#ERROR: 1050 - Table 't1' already exists INSERT INTO t1 VALUES(17509);#NOERROR DELETE FROM t1;#NOERROR create table t2 (a int auto_increment,b int,PRIMARY KEY (a)) ENGINE=InnoDB;#NOERROR INSERT INTO t1 VALUES (-123.456e0);#NOERROR LOAD DATA INFILE '../../tmp/proc.txt' INTO TABLE InnoDB.proc;#ERROR: 1100 - Table''was not locked with LOCK TABLES CREATE TABLE `アアア`(`カカカ` char(1)) DEFAULT CHARSET=utf8 engine=InnoDB;#ERROR: 1100 - Table 'アアア' was not locked with LOCK TABLES INSERT INTO t1 VALUES(0xADBF);#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction CREATE TABLE ti (a SMALLINT UNSIGNED,b BIGINT UNSIGNED ,c CHAR(32) ,d VARCHAR(31) ,e VARCHAR(31),f VARBINARY(58),g TINYBLOB,h BLOB ,id BIGINT ,KEY(b),KEY(e),PRIMARY KEY(id)) ENGINE=InnoDB;#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction insert into t1 values (-1),(-2),(-3);#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction INSERT INTO ti VALUES (4023595403684610118,7836,'7swVX','xWYjZYSkX0P','JJgE','so18BIYlJMt8r05JqWP3Q7e5i8xMnZyp','1','7',11);#ERROR: 1792 - Cannot execute statement in a READ ONLY transaction SET @@global.max_binlog_size=4095;#NOERROR shutdown;#NOERROR The testcase is highly sporadic and requires continuous startup/shutdowns of mysqld. Thirdly, any attempts at removing the # comments (which are no longer correct; they are from the original run) have failed thus far. As such, there is a suspicion that "Exact length" is required, and I note the max_binlog_size=4095 , present in other ways in the other testcases also. This may be an 'off by one' issue or similar. Given that there are rr traces available (and additional ones can be produced at will), there would seem to be sufficient information for this bug to be analyzed further by Elkin .
            Roel Roel Van de Paar made changes -
            Assignee Roel Van de Paar [ roel ] Andrei Elkin [ elkin ]
            Roel Roel Van de Paar made changes -
            Attachment MDEV-24660.sql [ 58242 ]
            Roel Roel Van de Paar added a comment - - edited

            Also uploaded that last testcase as MDEV-24660.sql to ensure the special chars are captured correctly.

            Roel Roel Van de Paar added a comment - - edited Also uploaded that last testcase as MDEV-24660 .sql to ensure the special chars are captured correctly.
            Roel Roel Van de Paar made changes -
            Attachment MDEV-24660.sql [ 58242 ]
            Roel Roel Van de Paar made changes -
            Attachment MDEV-24660.sql [ 58243 ]
            Roel Roel Van de Paar made changes -
            Fix Version/s 10.6 [ 24028 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 118423 ] MariaDB v4 [ 142520 ]
            Elkin Andrei Elkin made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Roel Roel Van de Paar made changes -
            Affects Version/s 10.10 [ 27530 ]
            Roel Roel Van de Paar made changes -

            Also seen in 10.10, regularly.

            mysqld: /test/10.10_dbg/sql/log.cc:3557: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            

            Core was generated by `/test/MD310522-mariadb-10.10.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_'.
            Program terminated with signal SIGABRT, Aborted.
            #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            [Current thread is 1 (Thread 0x150584635800 (LWP 4172928))]
            (gdb) bt
            #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
            #1  0x000015058480e859 in __GI_abort () at abort.c:79
            #2  0x000015058480e729 in __assert_fail_base (fmt=0x1505849a4588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5585ea58250c "b->xid_count == 0", file=0x5585ea581fe4 "/test/10.10_dbg/sql/log.cc", line=3557, function=<optimized out>) at assert.c:92
            #3  0x000015058481ffd6 in __GI___assert_fail (assertion=assertion@entry=0x5585ea58250c "b->xid_count == 0", file=file@entry=0x5585ea581fe4 "/test/10.10_dbg/sql/log.cc", line=line@entry=3557, function=function@entry=0x5585ea58251e "void MYSQL_BIN_LOG::cleanup()") at assert.c:101
            #4  0x00005585e9c2955e in MYSQL_BIN_LOG::cleanup (this=0x5585eaceb660 <mysql_bin_log>) at /test/10.10_dbg/sql/log.cc:3557
            #5  0x00005585e96ab259 in clean_up (print_message=print_message@entry=true) at /test/10.10_dbg/sql/mysqld.cc:1968
            #6  0x00005585e96b766c in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.10_dbg/sql/mysqld.cc:5937
            #7  0x00005585e96aab66 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.10_dbg/sql/main.cc:34
            

            Roel Roel Van de Paar added a comment - Also seen in 10.10, regularly. mysqld: /test/10.10_dbg/sql/log.cc:3557: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed. Core was generated by `/test/MD310522-mariadb-10.10.0-linux-x86_64-dbg/bin/mysqld --no-defaults --max_'. Program terminated with signal SIGABRT, Aborted. #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 [Current thread is 1 (Thread 0x150584635800 (LWP 4172928))] (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x000015058480e859 in __GI_abort () at abort.c:79 #2 0x000015058480e729 in __assert_fail_base (fmt=0x1505849a4588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5585ea58250c "b->xid_count == 0", file=0x5585ea581fe4 "/test/10.10_dbg/sql/log.cc", line=3557, function=<optimized out>) at assert.c:92 #3 0x000015058481ffd6 in __GI___assert_fail (assertion=assertion@entry=0x5585ea58250c "b->xid_count == 0", file=file@entry=0x5585ea581fe4 "/test/10.10_dbg/sql/log.cc", line=line@entry=3557, function=function@entry=0x5585ea58251e "void MYSQL_BIN_LOG::cleanup()") at assert.c:101 #4 0x00005585e9c2955e in MYSQL_BIN_LOG::cleanup (this=0x5585eaceb660 <mysql_bin_log>) at /test/10.10_dbg/sql/log.cc:3557 #5 0x00005585e96ab259 in clean_up (print_message=print_message@entry=true) at /test/10.10_dbg/sql/mysqld.cc:1968 #6 0x00005585e96b766c in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.10_dbg/sql/mysqld.cc:5937 #7 0x00005585e96aab66 in main (argc=<optimized out>, argv=<optimized out>) at /test/10.10_dbg/sql/main.cc:34

            The issue is always seen at shudown only.

            10.10.0 081a284712bb661349e2e3802077b12211cede3e (Debug)

            2022-06-12 18:46:57 0 [Note] /test/MD310522-mariadb-10.10.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
            2022-06-12 18:46:57 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 23
            2022-06-12 18:46:57 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply
            2022-06-12 18:46:57 0 [Note] Event Scheduler: Stopped
            2022-06-12 18:46:57 18 [Note] Stopping ack receiver thread
            2022-06-12 18:46:57 0 [Note] InnoDB: FTS optimize thread exiting.
            mysqld: /test/10.10_dbg/sql/log.cc:3557: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            

            Roel Roel Van de Paar added a comment - The issue is always seen at shudown only. 10.10.0 081a284712bb661349e2e3802077b12211cede3e (Debug) 2022-06-12 18:46:57 0 [Note] /test/MD310522-mariadb-10.10.0-linux-x86_64-dbg/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown 2022-06-12 18:46:57 0 [Note] Event Scheduler: Killing the scheduler thread, thread id 23 2022-06-12 18:46:57 0 [Note] Event Scheduler: Waiting for the scheduler thread to reply 2022-06-12 18:46:57 0 [Note] Event Scheduler: Stopped 2022-06-12 18:46:57 18 [Note] Stopping ack receiver thread 2022-06-12 18:46:57 0 [Note] InnoDB: FTS optimize thread exiting. mysqld: /test/10.10_dbg/sql/log.cc:3557: void MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed.
            Roel Roel Van de Paar made changes -
            Summary MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed in MYSQL_BIN_LOG::cleanup MYSQL_BIN_LOG::cleanup(): Assertion `b->xid_count == 0' failed in MYSQL_BIN_LOG::cleanup on SHUTDOWN

            New RR traces of the shutdown hang issues (using RESET MASTER) produced on latest 10.6.9 source and handed to Elkin.
            With many thanks to @marko for his ever-helpful assistance with rr and related matters.

            Roel Roel Van de Paar added a comment - New RR traces of the shutdown hang issues (using RESET MASTER) produced on latest 10.6.9 source and handed to Elkin . With many thanks to @marko for his ever-helpful assistance with rr and related matters.
            Elkin Andrei Elkin added a comment -

            Roel, as a summary of my analysis of rr traces available, the /data/934191/1/rr/mysqld-0, 9a08fcbf60567 trace deals with a stale version. We need the latest 10.5 to be checked as it contains bb-10.5-MDEV-24302 two commits. It's also fine if it's easier to work on 10.10 that your latest comments refer to.
            The 2nd /data/320105/1/rr/mysqld-0 trace does not represent a hanging server. The final state prior to the server kill does not show any RESET-MASTER nor SHUTDOWN.

            Elkin Andrei Elkin added a comment - Roel , as a summary of my analysis of rr traces available, the /data/934191/1/rr/mysqld-0, 9a08fcbf60567 trace deals with a stale version. We need the latest 10.5 to be checked as it contains bb-10.5- MDEV-24302 two commits. It's also fine if it's easier to work on 10.10 that your latest comments refer to. The 2nd /data/320105/1/rr/mysqld-0 trace does not represent a hanging server. The final state prior to the server kill does not show any RESET-MASTER nor SHUTDOWN.

            Elkin Great, thank you for the analysis. Ack on 1st trace, and for 2nd I am surprised as msqladmin shutdown was definitely hanging for 90 seconds before SIGKILL was sent to mysqladmin shutdown (whilst mysqld kept running) and then SIGABRT was sent to mysqld which finalized/rounded up the rr trace. It may be that the server was in some other state than RESET MASTER or SHUTDOWN, however it would seem that something was definitely hanging. I will next generate additional traces also attempting 10.10 as discussed. I may also see if somehow I can reproduce the assert abort rather than the RESET MASTER hang.

            Roel Roel Van de Paar added a comment - Elkin Great, thank you for the analysis. Ack on 1st trace, and for 2nd I am surprised as msqladmin shutdown was definitely hanging for 90 seconds before SIGKILL was sent to mysqladmin shutdown (whilst mysqld kept running) and then SIGABRT was sent to mysqld which finalized/rounded up the rr trace. It may be that the server was in some other state than RESET MASTER or SHUTDOWN , however it would seem that something was definitely hanging. I will next generate additional traces also attempting 10.10 as discussed. I may also see if somehow I can reproduce the assert abort rather than the RESET MASTER hang.
            Roel Roel Van de Paar added a comment - - edited

            Status update: reproduced the assert b->xid_count == 0 under rr in 10.10, handed to Andrei, who found the cause, and is creating a patch.

            Roel Roel Van de Paar added a comment - - edited Status update: reproduced the assert b->xid_count == 0 under rr in 10.10, handed to Andrei, who found the cause, and is creating a patch.
            Roel Roel Van de Paar made changes -
            Labels affects-tests rr-profile affects-tests rr-profile-analyzed
            Roel Roel Van de Paar made changes -
            Fix Version/s 10.7 [ 24805 ]
            Fix Version/s 10.8 [ 26121 ]
            Roel Roel Van de Paar added a comment - - edited

            The patch created by Elkin resolves both the b->xid_count == 0 assert, as well as the RESET MASTER hang on shutdown.

            Roel Roel Van de Paar added a comment - - edited The patch created by Elkin resolves both the b->xid_count == 0 assert, as well as the RESET MASTER hang on shutdown.
            Elkin Andrei Elkin added a comment -

            Salve, Brandon.

            Could you please check the commit at
            5bf4dee3694..bca750f81fb HEAD -> bb-10.5-andrei

            Cheers,

            Andrei

            Elkin Andrei Elkin added a comment - Salve, Brandon. Could you please check the commit at 5bf4dee3694..bca750f81fb HEAD -> bb-10.5-andrei Cheers, Andrei
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Brandon Nesterenko [ JIRAUSER48702 ]
            Status In Progress [ 3 ] In Review [ 10002 ]

            Looks good to me! Thanks, Elkin!

            bnestere Brandon Nesterenko added a comment - Looks good to me! Thanks, Elkin !
            bnestere Brandon Nesterenko made changes -
            Assignee Brandon Nesterenko [ JIRAUSER48702 ] Andrei Elkin [ elkin ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            Elkin Andrei Elkin added a comment - - edited

            For merging to 10.6 notice bb-10.6-andrei-MDEV-24660.

            Elkin Andrei Elkin added a comment - - edited For merging to 10.6 notice bb-10.6-andrei-MDEV-24660 .
            Elkin Andrei Elkin made changes -
            Fix Version/s 10.5.18 [ 28421 ]
            Fix Version/s 10.6.10 [ 28407 ]
            Fix Version/s 10.7.6 [ 28408 ]
            Fix Version/s 10.8.5 [ 28308 ]
            Fix Version/s 10.9.3 [ 28409 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.7 [ 24805 ]
            Fix Version/s 10.8 [ 26121 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 198889

            People

              Elkin Andrei Elkin
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.