Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26951

gcol.innodb_virtual_basic fails with "purge non-delete-marked" error

Details

    Description

      gcol.innodb_virtual_basic 'innodb'       w4 [ fail ]  Found warnings/errors in server log file!
              Test ended at 2021-10-29 06:39:04
      line
      2021-10-29  6:39:04 134229689757888 [ERROR] InnoDB: tried to purge non-delete-marked record in index `idx` of table `test`.`t`: tuple: TUPLE (info_bits=0, 3 fields): {[100]7474747474747474747474747474747474747474747474747474747474747474747474747474747474747474747474747474(0x37343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734),[20]bbbbbbbbbbbbbbbbbbbb(0x6262626262626262626262626262626262626262),[4]    (0x80000001)}, record: RECORD(info_bits=0, 3 fields): {[100]7474747474747474747474747474747474747474747474747474747474747474747474747474747474747474747474747474(0x37343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734373437343734),[20]bbbbbbbbbbbbbbbbbbbb(0x6262626262626262626262626262626262626262),[4]    (0x80000001)}
      ^ Found warnings in /dev/shm/var/4/log/mysqld.1.err
      

      The fail is not deterministic, and has a very low trigger rate.

      The second, very much likely connected failure on 10.4. Note the free(0x80) call:

      CURRENT_TEST: gcol.innodb_virtual_basic
      mysqltest: At line 65: query 'DROP INDEX idx1 ON t' failed: 2013: Lost connection to MySQL server during query
       
      The result from queries just before the failure was:
      < snip >
      7
      33
      SELECT m FROM t;
      m
      5
      11
      30
      SELECT p FROM t;
      p
      ummuooo
      uuuuu
      XXXAAA
      SELECT * FROM t;
      a	b	c	h	j	m	n	p
      11	22	33	AAA	8	30	XXX	XXXAAA
      1	2	3	uuu	9	11	uu	uuuuu
      3	4	7	uooo	1	5	umm	ummuooo
      update t set a = 13 where a =11;
      delete from t where a =13;
      DROP INDEX idx1 ON t;
       
      More results from queries before failure can be found in /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysql-test/var/9/log/innodb_virtual_basic.log
       
       
      Server [mysqld.1 - pid: 2425096, winpid: 2425096, exit: 256] failed during test run
      Server log from this test:
      ----------SERVER LOG START-----------
      $ /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld --defaults-group-suffix=.1 --defaults-file=/home/buildbot/maria-slave/fulltest-debug-big-only/build/mysql-test/var/9/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --loose-enable-partition --core-file --loose-debug-sync-timeout=300
      2021-10-26 17:58:32 0 [Note] /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld (mysqld 10.4.22-MariaDB-debug-log) starting as process 2425097 ...
      2021-10-26 17:58:32 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32190)
      2021-10-26 17:58:32 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
      2021-10-26 17:58:32 0 [Note] Plugin 'SEQUENCE' is disabled.
      2021-10-26 17:58:32 0 [Note] InnoDB: Using Linux native AIO
      2021-10-26 17:58:32 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
      2021-10-26 17:58:32 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2021-10-26 17:58:32 0 [Note] InnoDB: Uses event mutexes
      2021-10-26 17:58:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2021-10-26 17:58:32 0 [Note] InnoDB: Number of pools: 1
      2021-10-26 17:58:32 0 [Note] InnoDB: Using SSE2 crc32 instructions
      2021-10-26 17:58:32 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
      2021-10-26 17:58:32 0 [Note] InnoDB: Completed initialization of buffer pool
      2021-10-26 17:58:32 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      2021-10-26 17:58:32 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
      2021-10-26 17:58:32 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2021-10-26 17:58:32 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2021-10-26 17:58:32 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      2021-10-26 17:58:32 0 [Note] InnoDB: Waiting for purge to start
      2021-10-26 17:58:32 0 [Note] InnoDB: 10.4.22 started; log sequence number 68844; transaction id 33
      2021-10-26 17:58:32 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_YC5v/9/mysqld.1/data/ib_buffer_pool
      2021-10-26 17:58:32 0 [Note] InnoDB: Buffer pool(s) load completed at 211026 17:58:32
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_CMP' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'FEEDBACK' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'user_variables' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
      2021-10-26 17:58:32 0 [Note] Plugin 'unix_socket' is disabled.
      2021-10-26 17:58:32 0 [Warning] /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld: unknown option '--loose-pam-debug'
      2021-10-26 17:58:32 0 [Warning] /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld: unknown option '--loose-aria'
      2021-10-26 17:58:32 0 [Note] Server socket created on IP: '127.0.0.1'.
      2021-10-26 17:58:32 0 [Note] Reading of all Master_info entries succeeded
      2021-10-26 17:58:32 0 [Note] Added new Master_info '' to hash table
      2021-10-26 17:58:32 0 [Note] /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld: ready for connections.
      Version: '10.4.22-MariaDB-debug-log'  socket: '/home/buildbot/maria-slave/fulltest-debug-big-only/build/mysql-test/var/tmp/9/mysqld.1.sock'  port: 16140  Source distribution
      211026 17:58:32 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.4.22-MariaDB-debug-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=1
      max_threads=153
      thread_count=6
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63649 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f742c000d90
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f747e257dc8 thread_stack 0x49000
      mysys/stacktrace.c:175(my_print_stacktrace)[0x563236b27f7d]
      sql/signal_handler.cc:222(handle_fatal_signal)[0x56323658e45d]
      sigaction.c:0(__restore_rt)[0x7f74897373c0]
      /lib/x86_64-linux-gnu/libc.so.6(cfree+0x20)[0x7f7489279870]
      include/dict0dict.ic:1240(dict_sys_t::remove(dict_table_t*, bool, bool))[0x56323695679e]
      handler/handler0alter.cc:10267(ha_innobase::commit_inplace_alter_table(TABLE*, Alter_inplace_info*, bool))[0x5632367998a7]
      sql/sql_table.cc:7873(mysql_inplace_alter_table(THD*, TABLE_LIST*, TABLE*, TABLE*, Alter_inplace_info*, MDL_request*, Alter_table_ctx*) [clone .isra.0])[0x5632363e6689]
      sql/sql_table.cc:10256(mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool))[0x5632363f420d]
      sql/sql_parse.cc:5676(mysql_execute_command(THD*))[0x56323634482c]
      sql/sql_prepare.cc:5014(Prepared_statement::execute(String*, bool))[0x563236361fc7]
      sql/sql_prepare.cc:4485(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*))[0x56323636219f]
      sql/sql_prepare.cc:3465(mysql_stmt_execute_common(THD*, unsigned long, unsigned char*, unsigned char*, unsigned long, bool, bool))[0x563236363525]
      sql/sql_prepare.cc:3253(mysqld_stmt_execute(THD*, char*, unsigned int))[0x56323636365c]
      sql/sql_parse.cc:1783(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x563236350074]
      sql/sql_parse.cc:1375(do_command(THD*))[0x563236351839]
      sql/sql_connect.cc:1420(do_handle_one_connection(CONNECT*))[0x5632364508d4]
      sql/sql_connect.cc:1318(handle_one_connection)[0x5632364509f6]
      perfschema/pfs.cc:1872(pfs_spawn_thread)[0x563236adb041]
      nptl/pthread_create.c:478(start_thread)[0x7f748972b609]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f74892fe293]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f742c0c7418): DROP INDEX idx1 ON t
       
      Connection ID (thread ID): 9
      Status: NOT_KILLED
       
      --------------------------
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      Core was generated by `/home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld --defaults-'.
      Program terminated with signal SIGSEGV, Segmentation fault.
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
      56	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
      [Current thread is 1 (Thread 0x7f747e258700 (LWP 2425127))]
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
      #1  0x0000563236b27d4e in my_write_core (sig=sig@entry=11) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/stacktrace.c:386
      #2  0x000056323658e478 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/signal_handler.cc:356
      #3  <signal handler called>
      #4  __GI___libc_free (mem=0x80) at malloc.c:3102
      #5  0x000056323695679e in dict_free_vc_templ (vc_templ=0x7f745000d560) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/dict0dict.ic:1237
      #6  dict_sys_t::remove (this=<optimized out>, table=table@entry=0x7f742c199790, lru=lru@entry=false, keep=keep@entry=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/dict/dict0dict.cc:1778
      #7  0x00005632367998a7 in innobase_reload_table (table_name=..., table=0x7f742c199790, thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:10266
      #8  ha_innobase::commit_inplace_alter_table (this=<optimized out>, altered_table=<optimized out>, ha_alter_info=<optimized out>, commit=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:11500
      #9  0x00005632363e6689 in mysql_inplace_alter_table (thd=0x7f742c000d90, table_list=0x7f742c0c7510, table=0x7f742c079720, altered_table=0x7f747e2533a0, ha_alter_info=0x7f747e253300, alter_ctx=0x7f747e254e30, target_mdl_request=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:7873
      #10 0x00005632363f420d in mysql_alter_table (thd=thd@entry=0x7f742c000d90, new_db=new_db@entry=0x7f742c0c7528, new_name=new_name@entry=0x7f742c0c7538, create_info=create_info@entry=0x7f747e255a70, table_list=<optimized out>, table_list@entry=0x7f742c0c7510, alter_info=alter_info@entry=0x7f747e2559b0, order_num=0, order=0x0, ignore=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:10253
      #11 0x000056323634482c in mysql_execute_command (thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_parse.cc:4240
      #12 0x0000563236361fc7 in Prepared_statement::execute (this=0x7f742c1962c0, expanded_query=<optimized out>, open_cursor=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:5014
      #13 0x000056323636219f in Prepared_statement::execute_loop (this=0x7f742c1962c0, expanded_query=0x7f747e257300, open_cursor=false, packet=<optimized out>, packet_end=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:4483
      #14 0x0000563236363525 in mysql_stmt_execute_common (thd=0x7f742c000d90, stmt_id=<optimized out>, packet=0x7f742c00abfa "", packet_end=0x7f742c00abfa "", cursor_flags=0, bulk_op=<optimized out>, read_types=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:3465
      #15 0x000056323636365c in mysqld_stmt_execute (thd=thd@entry=0x7f742c000d90, packet_arg=packet_arg@entry=0x7f742c00abf1 "$", packet_length=packet_length@entry=9) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:3250
      #16 0x0000563236350074 in dispatch_command (command=COM_STMT_EXECUTE, thd=0x7f742c000d90, packet=0x7f742c00abf1 "$", packet_length=9, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_parse.cc:1781
      #17 0x0000563236351839 in do_command (thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_parse.cc:1373
      #18 0x00005632364508d4 in do_handle_one_connection (connect=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_connect.cc:1420
      #19 0x00005632364509f6 in handle_one_connection (arg=arg@entry=0x5632396ed090) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_connect.cc:1316
      #20 0x0000563236adb041 in pfs_spawn_thread (arg=0x5632396dfb30) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #21 0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #22 0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 27 (Thread 0x7f7466ffd700 (LWP 2425115)):
      #0  0x00007f74892e189b in sched_yield () at ../sysdeps/unix/syscall-template.S:78
      #1  0x000056323680af79 in os_thread_yield () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0thread.cc:212
      #2  0x00005632368bbe15 in trx_purge_wait_for_workers_to_complete () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/trx/trx0purge.cc:1240
      #3  trx_purge (n_purge_threads=<optimized out>, truncate=<optimized out>, slot=slot@entry=0x56323746f5c8 <srv_sys+584>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/trx/trx0purge.cc:1299
      #4  0x000056323689db35 in srv_do_purge (slot=0x56323746f5c8 <srv_sys+584>, n_total_purged=<synthetic pointer>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2613
      #5  srv_purge_coordinator_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2754
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 26 (Thread 0x7f744f7fe700 (LWP 2425121)):
      #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x563239325ab8) at ../sysdeps/nptl/futex-internal.h:183
      #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x563239325a68, cond=0x563239325a90) at pthread_cond_wait.c:508
      #2  __pthread_cond_wait (cond=cond@entry=0x563239325a90, mutex=mutex@entry=0x563239325a68) at pthread_cond_wait.c:638
      #3  0x000056323680aa69 in os_event::wait (this=0x563239325a50) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  os_event::wait_low (this=0x563239325a50, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:325
      #5  0x000056323692b6ff in buf_resize_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/buf/buf0buf.cc:3142
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 25 (Thread 0x7f7464ff9700 (LWP 2425119)):
      #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5632393259b8) at ../sysdeps/nptl/futex-internal.h:183
      #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x563239325968, cond=0x563239325990) at pthread_cond_wait.c:508
      #2  __pthread_cond_wait (cond=cond@entry=0x563239325990, mutex=mutex@entry=0x563239325968) at pthread_cond_wait.c:638
      #3  0x000056323680aa69 in os_event::wait (this=0x563239325950) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  os_event::wait_low (this=0x563239325950, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:325
      #5  0x00005632369321df in buf_dump_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/buf/buf0dump.cc:832
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 24 (Thread 0x7f74657fa700 (LWP 2425118)):
      #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5632393257b8) at ../sysdeps/nptl/futex-internal.h:183
      #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x563239325768, cond=0x563239325790) at pthread_cond_wait.c:508
      #2  __pthread_cond_wait (cond=cond@entry=0x563239325790, mutex=mutex@entry=0x563239325768) at pthread_cond_wait.c:638
      #3  0x000056323680aa69 in os_event::wait (this=0x563239325750) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  os_event::wait_low (this=0x563239325750, reset_sig_count=101) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:325
      #5  0x000056323689d90b in srv_resume_thread (slot=0x56323746f7b8 <srv_sys+1080>, sig_count=0, wait=<optimized out>, timeout_usec=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:904
      #6  0x000056323689e479 in srv_worker_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2518
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 23 (Thread 0x7f7465ffb700 (LWP 2425117)):
      #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x563239325838) at ../sysdeps/nptl/futex-internal.h:183
      #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5632393257e8, cond=0x563239325810) at pthread_cond_wait.c:508
      #2  __pthread_cond_wait (cond=cond@entry=0x563239325810, mutex=mutex@entry=0x5632393257e8) at pthread_cond_wait.c:638
      #3  0x000056323680aa69 in os_event::wait (this=0x5632393257d0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  os_event::wait_low (this=0x5632393257d0, reset_sig_count=111) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:325
      #5  0x000056323689d90b in srv_resume_thread (slot=0x56323746f8b0 <srv_sys+1328>, sig_count=0, wait=<optimized out>, timeout_usec=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:904
      #6  0x000056323689e479 in srv_worker_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2518
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 22 (Thread 0x7f74667fc700 (LWP 2425116)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f74667fafa0, clockid=<optimized out>, expected=0, futex_word=0x7f7450000f88) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f74667fafa0, clockid=<optimized out>, mutex=0x7f7450000ed8, cond=0x7f7450000f60) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=cond@entry=0x7f7450000f60, mutex=mutex@entry=0x7f7450000ed8, abstime=abstime@entry=0x7f74667fafa0) at pthread_cond_wait.c:656
      #3  0x0000563236b2d064 in safe_cond_timedwait (cond=cond@entry=0x7f7450000f60, mp=mp@entry=0x7f7450000eb0, abstime=abstime@entry=0x7f74667fafa0, file=file@entry=0x563236bb2a80 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h", line=line@entry=1211) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_mutex.c:546
      #4  0x0000563236459146 in inline_mysql_cond_timedwait (src_file=0x563236c03130 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mdl.cc", src_line=1176, abstime=0x7f74667fafa0, mutex=0x7f7450000eb0, that=0x7f7450000f60) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h:1211
      #5  MDL_wait::timed_wait (this=0x7f7450000eb0, owner=0x7f7450000e60, abs_timeout=0x7f74667fafa0, set_status_on_timeout=<optimized out>, wait_state_name=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mdl.cc:1176
      #6  0x000056323645af2d in MDL_context::acquire_lock (this=this@entry=0x7f7450000eb0, mdl_request=mdl_request@entry=0x7f745000b3a8, lock_wait_timeout=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mdl.h:480
      #7  0x00005632362cfe8f in open_table_get_mdl_lock (thd=0x7f7450000d90, ot_ctx=0x7f74667fb850, mdl_request=0x7f745000b3a8, flags=<optimized out>, mdl_ticket=0x7f74667fb288) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_base.h:555
      #8  0x00005632362d4c54 in open_table (thd=thd@entry=0x7f7450000d90, table_list=table_list@entry=0x7f745000af58, ot_ctx=ot_ctx@entry=0x7f74667fb850) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_base.cc:1915
      #9  0x00005632362f344a in open_purge_table (thd=thd@entry=0x7f7450000d90, db=<optimized out>, db@entry=0x7f74667fb910 "test", dblen=4, tb=<optimized out>, tb@entry=0x7f74667fbab0 "t", tblen=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_class.cc:4813
      #10 0x0000563236764e1c in innodb_acquire_mdl (table=0x7f742c199790, thd=0x7f7450000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/ha_innodb.cc:20565
      #11 innodb_find_table_for_vc (thd=0x7f7450000d90, table=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/ha_innodb.cc:20639
      #12 0x00005632367798f8 in innobase_init_vc_templ (table=0x7f742c199790) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/my_pthread.h:374
      #13 0x0000563236877441 in row_purge_parse_undo_rec (thr=0x5632395a1df0, updated_extern=0x7f74667fbd07, undo_rec=0x5632395a2560 "", node=0x5632395a1fc0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/row/row0purge.cc:1113
      #14 row_purge (thr=0x5632395a1df0, undo_rec=0x5632395a2560 "", node=0x5632395a1fc0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/row/row0purge.cc:1261
      #15 row_purge_step (thr=thr@entry=0x5632395a1df0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/row/row0purge.cc:1342
      #16 0x0000563236828388 in que_thr_step (thr=0x5632395a1df0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/que/que0que.cc:1038
      #17 que_run_threads_low (thr=0x5632395a1df0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/que/que0que.cc:1100
      #18 que_run_threads (thr=thr@entry=0x5632395a1df0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/que/que0que.cc:1140
      #19 0x000056323689e43c in srv_task_execute (slot=0x56323746f6c0 <srv_sys+832>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2465
      #20 srv_worker_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2520
      #21 0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #22 0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 21 (Thread 0x7f7476ffd700 (LWP 2425107)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f7476ffcbe0, clockid=<optimized out>, expected=0, futex_word=0x563239325a3c) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f7476ffcbe0, clockid=<optimized out>, mutex=0x5632393259e8, cond=0x563239325a10) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=0x563239325a10, mutex=0x5632393259e8, abstime=abstime@entry=0x7f7476ffcbe0) at pthread_cond_wait.c:656
      #3  0x000056323680a9c5 in os_event::timed_wait (this=<optimized out>, abstime=0x7f7476ffcbe0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  0x000056323680ab11 in os_event::wait_time_low (this=0x5632393259d0, time_in_usec=<optimized out>, reset_sig_count=2) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:385
      #5  0x000056323618d0e3 in pc_sleep_if_needed (cur_time=9516500915, sig_count=2, next_loop_time=9516501909) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/buf/buf0flu.cc:2637
      #6  buf_flush_page_cleaner_coordinator () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/buf/buf0flu.cc:3096
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 20 (Thread 0x7f7467fff700 (LWP 2425113)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f7467ffebf0, clockid=<optimized out>, expected=0, futex_word=0x5632394beb78) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f7467ffebf0, clockid=<optimized out>, mutex=0x5632394beb28, cond=0x5632394beb50) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=0x5632394beb50, mutex=0x5632394beb28, abstime=abstime@entry=0x7f7467ffebf0) at pthread_cond_wait.c:656
      #3  0x000056323680a9c5 in os_event::timed_wait (this=<optimized out>, abstime=0x7f7467ffebf0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  0x000056323680ab11 in os_event::wait_time_low (this=0x5632394beb10, time_in_usec=<optimized out>, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:385
      #5  0x00005632368e1f3c in ib_wqueue_timedwait (wq=wq@entry=0x5632394be960, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/ut/ut0wqueue.cc:158
      #6  0x00005632369b259c in fts_optimize_thread (arg=0x5632394be960) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fts/fts0opt.cc:2832
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 19 (Thread 0x7f74677fe700 (LWP 2425114)):
      #0  0x00007f74892bc3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7f74677fdda0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
      #1  0x00007f74892c2047 in __GI___nanosleep (requested_time=requested_time@entry=0x7f74677fdda0, remaining=remaining@entry=0x0) at nanosleep.c:27
      #2  0x000056323680afc0 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0thread.cc:231
      #3  0x000056323689e7e7 in srv_master_sleep () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2335
      #4  srv_master_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:2375
      #5  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #6  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 18 (Thread 0x7f7477fff700 (LWP 2425105)):
      #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
      #1  0x00007f7489c3727b in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #2  0x0000563236806e09 in LinuxAIOHandler::collect (this=0x7f7477ffebe0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1823
      #3  0x0000563236807314 in LinuxAIOHandler::poll (this=0x7f7477ffebe0, m1=0x7f7477ffec80, m2=0x7f7477ffec88, request=0x7f7477ffec90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1984
      #4  0x0000563236808354 in os_aio_linux_handler (request=0x7f7477ffec90, m2=0x7f7477ffec88, m1=0x7f7477ffec80, global_segment=4) at /usr/include/c++/9/bits/stl_vector.h:915
      #5  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f7477ffec80, m2=m2@entry=0x7f7477ffec88, request=request@entry=0x7f7477ffec90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:5366
      #6  0x000056323697d965 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fil/fil0fil.cc:4315
      #7  0x00005632368a2538 in io_handler_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0start.cc:324
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 17 (Thread 0x7f74755f5700 (LWP 2425110)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f74755f4c60, clockid=<optimized out>, expected=0, futex_word=0x5632393258b8) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f74755f4c60, clockid=<optimized out>, mutex=0x563239325868, cond=0x563239325890) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=0x563239325890, mutex=0x563239325868, abstime=abstime@entry=0x7f74755f4c60) at pthread_cond_wait.c:656
      #3  0x000056323680a9c5 in os_event::timed_wait (this=<optimized out>, abstime=0x7f74755f4c60) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  0x000056323680ab11 in os_event::wait_time_low (this=0x563239325850, time_in_usec=<optimized out>, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:385
      #5  0x000056323689bc4b in srv_error_monitor_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:1848
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 16 (Thread 0x7f74765f7700 (LWP 2425112)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f74765f6df0, clockid=<optimized out>, expected=0, futex_word=0x56323958b708) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f74765f6df0, clockid=<optimized out>, mutex=0x56323958b6b8, cond=0x56323958b6e0) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=0x56323958b6e0, mutex=0x56323958b6b8, abstime=abstime@entry=0x7f74765f6df0) at pthread_cond_wait.c:656
      #3  0x000056323680a9c5 in os_event::timed_wait (this=<optimized out>, abstime=0x7f74765f6df0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  0x000056323680ab11 in os_event::wait_time_low (this=0x56323958b6a0, time_in_usec=<optimized out>, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:385
      #5  0x000056323697519c in dict_stats_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/dict/dict0stats_bg.cc:490
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 15 (Thread 0x7f7475df6700 (LWP 2425109)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f7475df5a50, clockid=<optimized out>, expected=0, futex_word=0x563239559f98) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f7475df5a50, clockid=<optimized out>, mutex=0x563239559f48, cond=0x563239559f70) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=0x563239559f70, mutex=0x563239559f48, abstime=abstime@entry=0x7f7475df5a50) at pthread_cond_wait.c:656
      #3  0x000056323680a9c5 in os_event::timed_wait (this=<optimized out>, abstime=0x7f7475df5a50) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  0x000056323680ab11 in os_event::wait_time_low (this=0x563239559f30, time_in_usec=<optimized out>, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:385
      #5  0x00005632367df8c5 in lock_wait_timeout_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/lock/lock0wait.cc:527
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 14 (Thread 0x7f747e058700 (LWP 2425101)):
      #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
      #1  0x00007f7489c3727b in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #2  0x0000563236806e09 in LinuxAIOHandler::collect (this=0x7f747e057be0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1823
      #3  0x0000563236807314 in LinuxAIOHandler::poll (this=0x7f747e057be0, m1=0x7f747e057c80, m2=0x7f747e057c88, request=0x7f747e057c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1984
      #4  0x0000563236808354 in os_aio_linux_handler (request=0x7f747e057c90, m2=0x7f747e057c88, m1=0x7f747e057c80, global_segment=0) at /usr/include/c++/9/bits/stl_vector.h:915
      #5  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f747e057c80, m2=m2@entry=0x7f747e057c88, request=request@entry=0x7f747e057c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:5366
      #6  0x000056323697d965 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fil/fil0fil.cc:4315
      #7  0x00005632368a2538 in io_handler_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0start.cc:324
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 13 (Thread 0x7f7474df4700 (LWP 2425111)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f7474df3ba0, clockid=<optimized out>, expected=0, futex_word=0x563239325938) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f7474df3ba0, clockid=<optimized out>, mutex=0x5632393258e8, cond=0x563239325910) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=0x563239325910, mutex=0x5632393258e8, abstime=abstime@entry=0x7f7474df3ba0) at pthread_cond_wait.c:656
      #3  0x000056323680a9c5 in os_event::timed_wait (this=<optimized out>, abstime=0x7f7474df3ba0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/sync0types.h:463
      #4  0x000056323680ab11 in os_event::wait_time_low (this=0x5632393258d0, time_in_usec=<optimized out>, reset_sig_count=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0event.cc:385
      #5  0x00005632368a11e0 in srv_monitor_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0srv.cc:1684
      #6  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #7  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 12 (Thread 0x7f744effd700 (LWP 2425122)):
      #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f744effcd48) at ../sysdeps/nptl/futex-internal.h:183
      #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f744effcd88, cond=0x7f744effcd20) at pthread_cond_wait.c:508
      #2  __pthread_cond_wait (cond=cond@entry=0x7f744effcd20, mutex=mutex@entry=0x7f744effcd88) at pthread_cond_wait.c:638
      #3  0x0000563236b2cdd3 in safe_cond_wait (cond=cond@entry=0x7f744effcd20, mp=mp@entry=0x7f744effcd60, file=file@entry=0x563236bb2a80 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h", line=line@entry=1185) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_mutex.c:492
      #4  0x0000563236765d75 in inline_mysql_cond_wait (src_file=0x563236d887e0 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/ha_innodb.cc", src_line=311, mutex=0x7f744effcd60, that=0x7f744effcd20) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h:1185
      #5  thd_destructor_proxy () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/ha_innodb.cc:311
      #6  0x0000563236adb041 in pfs_spawn_thread (arg=0x5632394d8490) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 11 (Thread 0x7f74777fe700 (LWP 2425106)):
      #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
      #1  0x00007f7489c3727b in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #2  0x0000563236806e09 in LinuxAIOHandler::collect (this=0x7f74777fdbe0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1823
      #3  0x0000563236807314 in LinuxAIOHandler::poll (this=0x7f74777fdbe0, m1=0x7f74777fdc80, m2=0x7f74777fdc88, request=0x7f74777fdc90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1984
      #4  0x0000563236808354 in os_aio_linux_handler (request=0x7f74777fdc90, m2=0x7f74777fdc88, m1=0x7f74777fdc80, global_segment=5) at /usr/include/c++/9/bits/stl_vector.h:915
      #5  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f74777fdc80, m2=m2@entry=0x7f74777fdc88, request=request@entry=0x7f74777fdc90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:5366
      #6  0x000056323697d965 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fil/fil0fil.cc:4315
      #7  0x00005632368a2538 in io_handler_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0start.cc:324
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 10 (Thread 0x7f747c855700 (LWP 2425104)):
      #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
      #1  0x00007f7489c3727b in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #2  0x0000563236806e09 in LinuxAIOHandler::collect (this=0x7f747c854be0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1823
      #3  0x0000563236807314 in LinuxAIOHandler::poll (this=0x7f747c854be0, m1=0x7f747c854c80, m2=0x7f747c854c88, request=0x7f747c854c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1984
      #4  0x0000563236808354 in os_aio_linux_handler (request=0x7f747c854c90, m2=0x7f747c854c88, m1=0x7f747c854c80, global_segment=3) at /usr/include/c++/9/bits/stl_vector.h:915
      #5  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f747c854c80, m2=m2@entry=0x7f747c854c88, request=request@entry=0x7f747c854c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:5366
      #6  0x000056323697d965 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fil/fil0fil.cc:4315
      #7  0x00005632368a2538 in io_handler_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0start.cc:324
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 9 (Thread 0x7f7484098700 (LWP 2425123)):
      #0  0x00007f7489223322 in __GI___sigtimedwait (set=set@entry=0x7f7484097d00, info=info@entry=0x7f7484097d80, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
      #1  0x00007f748922338b in __GI___sigwaitinfo (set=set@entry=0x7f7484097d00, info=info@entry=0x7f7484097d80) at ../sysdeps/unix/sysv/linux/sigwaitinfo.c:25
      #2  0x000056323626d724 in my_sigwait (code=<synthetic pointer>, sig=0x7f7484097cf8, set=0x7f7484097d00) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/my_pthread.h:196
      #3  signal_hand (arg=arg@entry=0x0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld.cc:3219
      #4  0x0000563236adb041 in pfs_spawn_thread (arg=0x56323962cd70) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #5  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #6  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 8 (Thread 0x7f74890607c0 (LWP 2425097)):
      #0  0x00007f74892f1aff in __GI___poll (fds=fds@entry=0x7ffeebb95330, nfds=nfds@entry=2, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
      #1  0x00007f748930eaa2 in __poll_chk (fds=fds@entry=0x7ffeebb95330, nfds=nfds@entry=2, timeout=timeout@entry=-1, fdslen=fdslen@entry=24) at poll_chk.c:27
      #2  0x0000563236277141 in poll (__timeout=-1, __nfds=<optimized out>, __fds=0x7ffeebb95330) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
      #3  handle_connections_sockets () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld.cc:6463
      #4  0x0000563236277f1b in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld.cc:5899
      #5  0x00007f74892030b3 in __libc_start_main (main=0x56323624d4a0 <main(int, char**)>, argc=24, argv=0x7ffeebb95788, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffeebb95778) at ../csu/libc-start.c:308
      #6  0x000056323626ad4e in _start () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/mysqld.cc:4654
       
      Thread 7 (Thread 0x7f748404d700 (LWP 2425124)):
      #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5632374f9ba8 <COND_manager+40>) at ../sysdeps/nptl/futex-internal.h:183
      #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5632374f9be8 <LOCK_manager+40>, cond=0x5632374f9b80 <COND_manager>) at pthread_cond_wait.c:508
      #2  __pthread_cond_wait (cond=cond@entry=0x5632374f9b80 <COND_manager>, mutex=mutex@entry=0x5632374f9be8 <LOCK_manager+40>) at pthread_cond_wait.c:638
      #3  0x0000563236b2cdd3 in safe_cond_wait (cond=cond@entry=0x5632374f9b80 <COND_manager>, mp=mp@entry=0x5632374f9bc0 <LOCK_manager>, file=file@entry=0x563236bb2a80 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h", line=line@entry=1174) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_mutex.c:492
      #4  0x000056323633849f in inline_mysql_cond_wait (that=0x5632374f9b80 <COND_manager>, mutex=0x5632374f9bc0 <LOCK_manager>, src_file=0x563236bd92e0 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_manager.cc", src_line=102) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h:1174
      #5  handle_manager (arg=arg@entry=0x0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_manager.cc:102
      #6  0x0000563236adb041 in pfs_spawn_thread (arg=0x56323963ca50) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 6 (Thread 0x7f7484c6e700 (LWP 2425098)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f7484c6ddd0, clockid=<optimized out>, expected=0, futex_word=0x563237d8f508 <COND_timer+40>) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f7484c6ddd0, clockid=<optimized out>, mutex=0x563237d8f548 <LOCK_timer+40>, cond=0x563237d8f4e0 <COND_timer>) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=cond@entry=0x563237d8f4e0 <COND_timer>, mutex=mutex@entry=0x563237d8f548 <LOCK_timer+40>, abstime=abstime@entry=0x7f7484c6ddd0) at pthread_cond_wait.c:656
      #3  0x0000563236b2d064 in safe_cond_timedwait (cond=cond@entry=0x563237d8f4e0 <COND_timer>, mp=mp@entry=0x563237d8f520 <LOCK_timer>, abstime=abstime@entry=0x7f7484c6ddd0, file=file@entry=0x563236bb2a80 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h", line=line@entry=1211) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_mutex.c:546
      #4  0x0000563236b2d725 in inline_mysql_cond_timedwait (that=0x563237d8f4e0 <COND_timer>, mutex=0x563237d8f520 <LOCK_timer>, src_file=0x563236e7e800 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_timer.c", src_line=292, abstime=0x7f7484c6ddd0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h:1211
      #5  timer_handler (arg=arg@entry=0x0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_timer.c:292
      #6  0x0000563236adb041 in pfs_spawn_thread (arg=0x56323921e9d0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #7  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #8  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 5 (Thread 0x7f747f85a700 (LWP 2425099)):
      #0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f747f859d60, clockid=<optimized out>, expected=0, futex_word=0x563237d7d828 <COND_checkpoint+40>) at ../sysdeps/nptl/futex-internal.h:320
      #1  __pthread_cond_wait_common (abstime=0x7f747f859d60, clockid=<optimized out>, mutex=0x563237d7d868 <LOCK_checkpoint+40>, cond=0x563237d7d800 <COND_checkpoint>) at pthread_cond_wait.c:520
      #2  __pthread_cond_timedwait (cond=cond@entry=0x563237d7d800 <COND_checkpoint>, mutex=mutex@entry=0x563237d7d868 <LOCK_checkpoint+40>, abstime=abstime@entry=0x7f747f859d60) at pthread_cond_wait.c:656
      #3  0x0000563236b2d064 in safe_cond_timedwait (cond=cond@entry=0x563237d7d800 <COND_checkpoint>, mp=mp@entry=0x563237d7d840 <LOCK_checkpoint>, abstime=abstime@entry=0x7f747f859d60, file=file@entry=0x563236bb2a80 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h", line=line@entry=1211) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/thr_mutex.c:546
      #4  0x0000563236a1c9f5 in inline_mysql_cond_timedwait (src_file=0x563236e3fce8 "/home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/maria/ma_servicethread.c", src_line=115, abstime=0x7f747f859d60, mutex=0x563237d7d840 <LOCK_checkpoint>, that=0x563237d7d800 <COND_checkpoint>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/include/mysql/psi/mysql_thread.h:1211
      #5  my_service_thread_sleep (control=control@entry=0x5632374749c0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/maria/ma_servicethread.c:115
      #6  0x0000563236a14cd2 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/maria/ma_checkpoint.c:707
      #7  0x0000563236adb041 in pfs_spawn_thread (arg=0x5632392dce60) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 4 (Thread 0x7f747d056700 (LWP 2425103)):
      #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
      #1  0x00007f7489c3727b in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #2  0x0000563236806e09 in LinuxAIOHandler::collect (this=0x7f747d055be0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1823
      #3  0x0000563236807314 in LinuxAIOHandler::poll (this=0x7f747d055be0, m1=0x7f747d055c80, m2=0x7f747d055c88, request=0x7f747d055c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1984
      #4  0x0000563236808354 in os_aio_linux_handler (request=0x7f747d055c90, m2=0x7f747d055c88, m1=0x7f747d055c80, global_segment=2) at /usr/include/c++/9/bits/stl_vector.h:915
      #5  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f747d055c80, m2=m2@entry=0x7f747d055c88, request=request@entry=0x7f747d055c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:5366
      #6  0x000056323697d965 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fil/fil0fil.cc:4315
      #7  0x00005632368a2538 in io_handler_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0start.cc:324
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 3 (Thread 0x7f744ffff700 (LWP 2425120)):
      #0  0x00007f74892bc3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7f744fffe760, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
      #1  0x00007f74892c2047 in __GI___nanosleep (requested_time=requested_time@entry=0x7f744fffe760, remaining=remaining@entry=0x0) at nanosleep.c:27
      #2  0x000056323680afc0 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0thread.cc:231
      #3  0x000056323691be88 in btr_defragment_thread () at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/btr/btr0defragment.cc:717
      #4  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #5  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 2 (Thread 0x7f747d857700 (LWP 2425102)):
      #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
      #1  0x00007f7489c3727b in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
      #2  0x0000563236806e09 in LinuxAIOHandler::collect (this=0x7f747d856be0) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1823
      #3  0x0000563236807314 in LinuxAIOHandler::poll (this=0x7f747d856be0, m1=0x7f747d856c80, m2=0x7f747d856c88, request=0x7f747d856c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:1984
      #4  0x0000563236808354 in os_aio_linux_handler (request=0x7f747d856c90, m2=0x7f747d856c88, m1=0x7f747d856c80, global_segment=1) at /usr/include/c++/9/bits/stl_vector.h:915
      #5  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f747d856c80, m2=m2@entry=0x7f747d856c88, request=request@entry=0x7f747d856c90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/os/os0file.cc:5366
      #6  0x000056323697d965 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/fil/fil0fil.cc:4315
      #7  0x00005632368a2538 in io_handler_thread (arg=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/srv/srv0start.cc:324
      #8  0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #9  0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
       
      Thread 1 (Thread 0x7f747e258700 (LWP 2425127)):
      #0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
      #1  0x0000563236b27d4e in my_write_core (sig=sig@entry=11) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/mysys/stacktrace.c:386
      #2  0x000056323658e478 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/signal_handler.cc:356
      #3  <signal handler called>
      #4  __GI___libc_free (mem=0x80) at malloc.c:3102
      #5  0x000056323695679e in dict_free_vc_templ (vc_templ=0x7f745000d560) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/dict0dict.ic:1237
      #6  dict_sys_t::remove (this=<optimized out>, table=table@entry=0x7f742c199790, lru=lru@entry=false, keep=keep@entry=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/dict/dict0dict.cc:1778
      #7  0x00005632367998a7 in innobase_reload_table (table_name=..., table=0x7f742c199790, thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:10266
      #8  ha_innobase::commit_inplace_alter_table (this=<optimized out>, altered_table=<optimized out>, ha_alter_info=<optimized out>, commit=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:11500
      #9  0x00005632363e6689 in mysql_inplace_alter_table (thd=0x7f742c000d90, table_list=0x7f742c0c7510, table=0x7f742c079720, altered_table=0x7f747e2533a0, ha_alter_info=0x7f747e253300, alter_ctx=0x7f747e254e30, target_mdl_request=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:7873
      #10 0x00005632363f420d in mysql_alter_table (thd=thd@entry=0x7f742c000d90, new_db=new_db@entry=0x7f742c0c7528, new_name=new_name@entry=0x7f742c0c7538, create_info=create_info@entry=0x7f747e255a70, table_list=<optimized out>, table_list@entry=0x7f742c0c7510, alter_info=alter_info@entry=0x7f747e2559b0, order_num=0, order=0x0, ignore=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:10253
      #11 0x000056323634482c in mysql_execute_command (thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_parse.cc:4240
      #12 0x0000563236361fc7 in Prepared_statement::execute (this=0x7f742c1962c0, expanded_query=<optimized out>, open_cursor=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:5014
      #13 0x000056323636219f in Prepared_statement::execute_loop (this=0x7f742c1962c0, expanded_query=0x7f747e257300, open_cursor=false, packet=<optimized out>, packet_end=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:4483
      #14 0x0000563236363525 in mysql_stmt_execute_common (thd=0x7f742c000d90, stmt_id=<optimized out>, packet=0x7f742c00abfa "", packet_end=0x7f742c00abfa "", cursor_flags=0, bulk_op=<optimized out>, read_types=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:3465
      #15 0x000056323636365c in mysqld_stmt_execute (thd=thd@entry=0x7f742c000d90, packet_arg=packet_arg@entry=0x7f742c00abf1 "$", packet_length=packet_length@entry=9) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_prepare.cc:3250
      #16 0x0000563236350074 in dispatch_command (command=COM_STMT_EXECUTE, thd=0x7f742c000d90, packet=0x7f742c00abf1 "$", packet_length=9, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_parse.cc:1781
      #17 0x0000563236351839 in do_command (thd=0x7f742c000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_parse.cc:1373
      #18 0x00005632364508d4 in do_handle_one_connection (connect=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_connect.cc:1420
      #19 0x00005632364509f6 in handle_one_connection (arg=arg@entry=0x5632396ed090) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_connect.cc:1316
      #20 0x0000563236adb041 in pfs_spawn_thread (arg=0x5632396dfb30) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/perfschema/pfs.cc:1869
      #21 0x00007f748972b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #22 0x00007f74892fe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      gcol.innodb_virtual_basic was disabled for long due to being marked as big-test, so it was much likely skipped before.

      Attachments

        Issue Links

          Activity

            We got SIGSEGV on 10.4 as well. Probably an issue with missing or incorrect MDL protection? Note: MDEV-25506 in 10.6 introduced dict_table_t::mdl_name that might fix this, by making a user-visible table name available also during DDL operations.

            10.4 b53ee760ff17e3d9ed95b75e5ea2de124ae69384

            #4  __GI___libc_free (mem=0x80) at malloc.c:3102
            #5  0x000055f31e1c4f1e in dict_free_vc_templ (vc_templ=0x7f8f50013f70) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/dict0dict.ic:1237
            #6  dict_sys_t::remove (this=<optimized out>, table=table@entry=0x7f8f28083870, lru=lru@entry=false, keep=keep@entry=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/dict/dict0dict.cc:1778
            #7  0x000055f31e0080d7 in innobase_reload_table (table_name=..., table=0x7f8f28083870, thd=0x7f8f28000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:10292
            #8  ha_innobase::commit_inplace_alter_table (this=<optimized out>, altered_table=<optimized out>, ha_alter_info=<optimized out>, commit=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:11526
            #9  0x000055f31db7abf8 in mysql_inplace_alter_table (thd=0x7f8f28000d90, table_list=0x7f8f28014890, table=0x7f8f28073470, altered_table=0x7f8f80bd6610, ha_alter_info=0x7f8f80bd6570, alter_ctx=0x7f8f80bd80a0, target_mdl_request=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:7895
            

            marko Marko Mäkelä added a comment - We got SIGSEGV on 10.4 as well. Probably an issue with missing or incorrect MDL protection? Note: MDEV-25506 in 10.6 introduced dict_table_t::mdl_name that might fix this, by making a user-visible table name available also during DDL operations. 10.4 b53ee760ff17e3d9ed95b75e5ea2de124ae69384 #4 __GI___libc_free (mem=0x80) at malloc.c:3102 #5 0x000055f31e1c4f1e in dict_free_vc_templ (vc_templ=0x7f8f50013f70) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/include/dict0dict.ic:1237 #6 dict_sys_t::remove (this=<optimized out>, table=table@entry=0x7f8f28083870, lru=lru@entry=false, keep=keep@entry=false) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/dict/dict0dict.cc:1778 #7 0x000055f31e0080d7 in innobase_reload_table (table_name=..., table=0x7f8f28083870, thd=0x7f8f28000d90) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:10292 #8 ha_innobase::commit_inplace_alter_table (this=<optimized out>, altered_table=<optimized out>, ha_alter_info=<optimized out>, commit=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/storage/innobase/handler/handler0alter.cc:11526 #9 0x000055f31db7abf8 in mysql_inplace_alter_table (thd=0x7f8f28000d90, table_list=0x7f8f28014890, table=0x7f8f28073470, altered_table=0x7f8f80bd6610, ha_alter_info=0x7f8f80bd6570, alter_ctx=0x7f8f80bd80a0, target_mdl_request=<optimized out>) at /home/buildbot/maria-slave/fulltest-debug-big-only/build/sql/sql_table.cc:7895

            While I was trying to reproduce the issue, I've caught ut_ad(mysql_table) in innobase_init_vc_templ.

            What I did is, I've added usleep(random() % 200); in the beginning of innodb_acquire_mdl
            and run the following snip from innodb_virtual_basic.test:

            --let $i= 1
            --while ($i) {
            --echo ========================== $i =========================================
            --inc $i
            CREATE TABLE t (a INT, b INT, c INT GENERATED ALWAYS AS(a+b), h VARCHAR(10), j INT, m INT  GENERATED ALWAYS AS(b + j), n VARCHAR(10), p VARCHAR(20) GENERATED ALWAYS AS(CONCAT(n, h)), INDEX idx1(c), INDEX idx2 (m), INDEX idx3(p));
            INSERT INTO t VALUES(11, 22, DEFAULT, "AAA", 8, DEFAULT, "XXX", DEFAULT);
            INSERT INTO t VALUES(1, 2, DEFAULT, "uuu", 9, DEFAULT, "uu", DEFAULT);
            INSERT INTO t VALUES(3, 4, DEFAULT, "uooo", 1, DEFAULT, "umm", DEFAULT);
            SELECT c FROM t;
            SELECT m FROM t;
            SELECT p FROM t;
            SELECT * FROM t;
            update t set a = 13 where a =11;
            delete from t where a =13;
            DROP INDEX idx1 ON t;
            DROP INDEX idx2 ON t;
            DROP TABLE t;
            --}
            

            This happens when the server is in progress of mysql_inplace_alter_table, around mysql_rename_table

            (gdb) bt
            #0  0x0000000070000002 in syscall_traced ()
            #1  0x00007f18cae9c905 in _raw_syscall () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/raw_syscall.S:120
            #2  0x00007f18cae9a677 in traced_raw_syscall (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:278
            #3  syscall_hook_internal (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:3420
            #4  syscall_hook (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:3454
            #5  syscall_hook (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:3438
            #6  0x00007f18cae97270 in _syscall_hook_trampoline () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscall_hook.S:313
            #7  0x00007f18cae972cf in __morestack () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscall_hook.S:458
            #8  0x00007f18cae972eb in _syscall_hook_trampoline_48_3d_00_f0_ff_ff () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscall_hook.S:477
            #9  0x00007f18ca4499f1 in rename () from target:/usr/lib/libc.so.6
            #10 0x0000000001afec75 in my_rename (from=0x7f18bc0a63b0 "./test/#sql-5afec_9.frm", to=0x7f18bc0a61b0 "./test/t.frm", MyFlags=0) at /home/nik/mariadb/mysys/my_rename.c:89
            #11 0x000000000112d699 in inline_mysql_file_rename (key=12, src_file=0x69794e "/home/nik/mariadb/sql/table.cc", src_line=4521, from=0x7f18bc0a63b0 "./test/#sql-5afec_9.frm", to=0x7f18bc0a61b0 "./test/t.frm", flags=0) at /home/nik/mariadb/include/mysql/psi/mysql_file.h:1362
            #12 0x000000000112d5d8 in rename_file_ext (from=0x7f18bc0a6e60 "./test/#sql-5afec_9", to=0x7f18bc0a6c50 "./test/t", ext=0x1f0cc90 <reg_ext> ".frm") at /home/nik/mariadb/sql/table.cc:4521
            #13 0x00000000010d8573 in mysql_rename_table (base=0x30bcaf8, old_db=0x7f18bc0a9ca8, old_name=0x7f18bc0a9cd8, new_db=0x7f18bc0a9c78, new_name=0x7f18bc0a9c98, flags=17) at /home/nik/mariadb/sql/sql_table.cc:5656
            #14 0x00000000010e7e9a in mysql_inplace_alter_table (thd=0x7f18b4018d18, table_list=0x7f18b4024198, table=0x0, altered_table=0x7f18bc0a7e38, ha_alter_info=0x7f18bc0a9190, target_mdl_request=0x7f18bc0a9ab8, alter_ctx=0x7f18bc0a9c68) at /home/nik/mariadb/sql/sql_table.cc:7928
            #15 0x00000000010e106d in mysql_alter_table (thd=0x7f18b4018d18, new_db=0x7f18b40241b0, new_name=0x7f18b40241c0, create_info=0x7f18bc0ac140, table_list=0x7f18b4024198, alter_info=0x7f18bc0ac088, order_num=0, order=0x0, ignore=false) at /home/nik/mariadb/sql/sql_table.cc:10302
            #16 0x0000000000fb8aa8 in mysql_execute_command (thd=0x7f18b4018d18) at /home/nik/mariadb/sql/sql_parse.cc:4258
            #17 0x0000000000faedb4 in mysql_parse (thd=0x7f18b4018d18, rawbuf=0x7f18b40240a0 "DROP INDEX idx2 ON t", length=20, parser_state=0x7f18bc0ad450, is_com_multi=false, is_next_command=false) at /home/nik/mariadb/sql/sql_parse.cc:8013
            #18 0x0000000000fab51f in dispatch_command (command=COM_QUERY, thd=0x7f18b4018d18, packet=0x7f18b400d3a9 "DROP INDEX idx2 ON t", packet_length=20, is_com_multi=false, is_next_command=false) at /home/nik/mariadb/sql/sql_parse.cc:1873
            #19 0x0000000000fafd63 in do_command (thd=0x7f18b4018d18) at /home/nik/mariadb/sql/sql_parse.cc:1378
            #20 0x00000000011a2456 in do_handle_one_connection (connect=0x3472a28) at /home/nik/mariadb/sql/sql_connect.cc:1420
            #21 0x00000000011a214a in handle_one_connection (arg=0x3472a28) at /home/nik/mariadb/sql/sql_connect.cc:1316
            #22 0x00000000014bc645 in pfs_spawn_thread (arg=0x3493a58) at /home/nik/mariadb/storage/perfschema/pfs.cc:1869
            #23 0x00007f18ca4785c2 in start_thread () from target:/usr/lib/libc.so.6
            #24 0x00007f18ca4fd584 in clone () from target:/usr/lib/libc.so.6
            

            I believe the real reason is this call:

            close_all_tables_for_name(thd, table->s,
                 alter_ctx->is_table_renamed() ?
                 HA_EXTRA_PREPARE_FOR_RENAME :
                 HA_EXTRA_NOT_USED,
                 NULL);
            

            right before mysql_rename_table call

            nikitamalyavin Nikita Malyavin added a comment - While I was trying to reproduce the issue, I've caught ut_ad(mysql_table) in innobase_init_vc_templ . What I did is, I've added usleep(random() % 200); in the beginning of innodb_acquire_mdl and run the following snip from innodb_virtual_basic.test : --let $i= 1 --while ($i) { --echo ========================== $i ========================================= --inc $i CREATE TABLE t (a INT , b INT , c INT GENERATED ALWAYS AS (a+b), h VARCHAR (10), j INT , m INT GENERATED ALWAYS AS (b + j), n VARCHAR (10), p VARCHAR (20) GENERATED ALWAYS AS (CONCAT(n, h)), INDEX idx1(c), INDEX idx2 (m), INDEX idx3(p)); INSERT INTO t VALUES (11, 22, DEFAULT , "AAA" , 8, DEFAULT , "XXX" , DEFAULT ); INSERT INTO t VALUES (1, 2, DEFAULT , "uuu" , 9, DEFAULT , "uu" , DEFAULT ); INSERT INTO t VALUES (3, 4, DEFAULT , "uooo" , 1, DEFAULT , "umm" , DEFAULT ); SELECT c FROM t; SELECT m FROM t; SELECT p FROM t; SELECT * FROM t; update t set a = 13 where a =11; delete from t where a =13; DROP INDEX idx1 ON t; DROP INDEX idx2 ON t; DROP TABLE t; --} This happens when the server is in progress of mysql_inplace_alter_table , around mysql_rename_table (gdb) bt #0 0x0000000070000002 in syscall_traced () #1 0x00007f18cae9c905 in _raw_syscall () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/raw_syscall.S:120 #2 0x00007f18cae9a677 in traced_raw_syscall (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:278 #3 syscall_hook_internal (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:3420 #4 syscall_hook (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:3454 #5 syscall_hook (call=0x7f1893df3fa0) at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscallbuf.c:3438 #6 0x00007f18cae97270 in _syscall_hook_trampoline () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscall_hook.S:313 #7 0x00007f18cae972cf in __morestack () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscall_hook.S:458 #8 0x00007f18cae972eb in _syscall_hook_trampoline_48_3d_00_f0_ff_ff () at /home/nik/.cache/yay/rr/src/rr-5.5.0/src/preload/syscall_hook.S:477 #9 0x00007f18ca4499f1 in rename () from target:/usr/lib/libc.so.6 #10 0x0000000001afec75 in my_rename (from=0x7f18bc0a63b0 "./test/#sql-5afec_9.frm", to=0x7f18bc0a61b0 "./test/t.frm", MyFlags=0) at /home/nik/mariadb/mysys/my_rename.c:89 #11 0x000000000112d699 in inline_mysql_file_rename (key=12, src_file=0x69794e "/home/nik/mariadb/sql/table.cc", src_line=4521, from=0x7f18bc0a63b0 "./test/#sql-5afec_9.frm", to=0x7f18bc0a61b0 "./test/t.frm", flags=0) at /home/nik/mariadb/include/mysql/psi/mysql_file.h:1362 #12 0x000000000112d5d8 in rename_file_ext (from=0x7f18bc0a6e60 "./test/#sql-5afec_9", to=0x7f18bc0a6c50 "./test/t", ext=0x1f0cc90 <reg_ext> ".frm") at /home/nik/mariadb/sql/table.cc:4521 #13 0x00000000010d8573 in mysql_rename_table (base=0x30bcaf8, old_db=0x7f18bc0a9ca8, old_name=0x7f18bc0a9cd8, new_db=0x7f18bc0a9c78, new_name=0x7f18bc0a9c98, flags=17) at /home/nik/mariadb/sql/sql_table.cc:5656 #14 0x00000000010e7e9a in mysql_inplace_alter_table (thd=0x7f18b4018d18, table_list=0x7f18b4024198, table=0x0, altered_table=0x7f18bc0a7e38, ha_alter_info=0x7f18bc0a9190, target_mdl_request=0x7f18bc0a9ab8, alter_ctx=0x7f18bc0a9c68) at /home/nik/mariadb/sql/sql_table.cc:7928 #15 0x00000000010e106d in mysql_alter_table (thd=0x7f18b4018d18, new_db=0x7f18b40241b0, new_name=0x7f18b40241c0, create_info=0x7f18bc0ac140, table_list=0x7f18b4024198, alter_info=0x7f18bc0ac088, order_num=0, order=0x0, ignore=false) at /home/nik/mariadb/sql/sql_table.cc:10302 #16 0x0000000000fb8aa8 in mysql_execute_command (thd=0x7f18b4018d18) at /home/nik/mariadb/sql/sql_parse.cc:4258 #17 0x0000000000faedb4 in mysql_parse (thd=0x7f18b4018d18, rawbuf=0x7f18b40240a0 "DROP INDEX idx2 ON t", length=20, parser_state=0x7f18bc0ad450, is_com_multi=false, is_next_command=false) at /home/nik/mariadb/sql/sql_parse.cc:8013 #18 0x0000000000fab51f in dispatch_command (command=COM_QUERY, thd=0x7f18b4018d18, packet=0x7f18b400d3a9 "DROP INDEX idx2 ON t", packet_length=20, is_com_multi=false, is_next_command=false) at /home/nik/mariadb/sql/sql_parse.cc:1873 #19 0x0000000000fafd63 in do_command (thd=0x7f18b4018d18) at /home/nik/mariadb/sql/sql_parse.cc:1378 #20 0x00000000011a2456 in do_handle_one_connection (connect=0x3472a28) at /home/nik/mariadb/sql/sql_connect.cc:1420 #21 0x00000000011a214a in handle_one_connection (arg=0x3472a28) at /home/nik/mariadb/sql/sql_connect.cc:1316 #22 0x00000000014bc645 in pfs_spawn_thread (arg=0x3493a58) at /home/nik/mariadb/storage/perfschema/pfs.cc:1869 #23 0x00007f18ca4785c2 in start_thread () from target:/usr/lib/libc.so.6 #24 0x00007f18ca4fd584 in clone () from target:/usr/lib/libc.so.6 I believe the real reason is this call: close_all_tables_for_name(thd, table->s, alter_ctx->is_table_renamed() ? HA_EXTRA_PREPARE_FOR_RENAME : HA_EXTRA_NOT_USED, NULL); right before mysql_rename_table call

            marko please review commits a6efb30e...f66b5064,
            branch bb-10.4-nikita

            I will also have to merge it down to 10.3, but let's first start from review.

            nikitamalyavin Nikita Malyavin added a comment - marko please review commits a6efb30e...f66b5064, branch bb-10.4-nikita I will also have to merge it down to 10.3, but let's first start from review.

            I think that thiru can do a first review.

            I understood that this does not affect 10.5 or later versions, thanks to MDEV-16678. All test failures that I could find for 10.5 or 10.6 were for something unrelated in a MemorySanitizer build, which were since then fixed.

            The final commit message should say that this is based on MDEV-16678. I would like to see a test run based on the latest 10.3 and 10.4 branches before the final approval.

            marko Marko Mäkelä added a comment - I think that thiru can do a first review. I understood that this does not affect 10.5 or later versions, thanks to MDEV-16678 . All test failures that I could find for 10.5 or 10.6 were for something unrelated in a MemorySanitizer build, which were since then fixed. The final commit message should say that this is based on MDEV-16678 . I would like to see a test run based on the latest 10.3 and 10.4 branches before the final approval.

            nikitamalyavin, what is the status of this? If this is only targeting 10.3 and 10.4, why do the fix versions include 10.5 and 10.6?

            If this is not changing anything in 10.5 or 10.6, then I suppose that the the following, which I just got in a local AddressSanitizer build, would not be fixed by this? The issue is repeatable for my build, so I reran it with ASAN_OPTIONS=fast_unwind_on_malloc=0 to get a useful stack trace for the allocation:

            10.6 19de8b7013536d61afcec2c0a4f2b4ccc5bf6b0d

            CURRENT_TEST: gcol.innodb_virtual_basic
            mysqltest: At line 1209: query 'replace into t(c) values ('')' failed: <Unknown> (2013): Lost connection to server during query
            …
            ==728994==ERROR: AddressSanitizer: use-after-poison on address 0x61a000352327 at pc 0x5616539f2780 bp 0x7fbcd6763450 sp 0x7fbcd6763448
            READ of size 1 at 0x61a000352327 thread T22
                #0 0x5616539f277f in page_cur_insert_rec_low(page_cur_t const*, dict_index_t*, unsigned char const*, unsigned short*, mtr_t*) /mariadb/10.6/storage/innobase/page/page0cur.cc:1664
                #1 0x561653c525e5 in page_cur_tuple_insert /mariadb/10.6/storage/innobase/include/page0cur.inl:199
                #2 0x561653c53532 in btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned short**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, unsigned long, que_thr_t*, mtr_t*) /mariadb/10.6/storage/innobase/btr/btr0cur.cc:3500
                #3 0x561653aa4a57 in row_ins_sec_index_entry_low(unsigned long, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*) /mariadb/10.6/storage/innobase/row/row0ins.cc:3033
                #4 0x561653aa5023 in row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool) /mariadb/10.6/storage/innobase/row/row0ins.cc:3231
                #5 0x561653aa75ba in row_ins_index_entry /mariadb/10.6/storage/innobase/row/row0ins.cc:3279
                #6 0x561653aa78b7 in row_ins_index_entry_step /mariadb/10.6/storage/innobase/row/row0ins.cc:3445
                #7 0x561653aa7e11 in row_ins /mariadb/10.6/storage/innobase/row/row0ins.cc:3591
                #8 0x561653aa8909 in row_ins_step(que_thr_t*) /mariadb/10.6/storage/innobase/row/row0ins.cc:3731
                #9 0x561653ae06ab in row_insert_for_mysql(unsigned char const*, row_prebuilt_t*, ins_mode_t) /mariadb/10.6/storage/innobase/row/row0mysql.cc:1308
                #10 0x56165382f9db in ha_innobase::write_row(unsigned char const*) /mariadb/10.6/storage/innobase/handler/ha_innodb.cc:7935
            …
            0x61a001924927 is located 167 bytes inside of 1176-byte region [0x61a001924880,0x61a001924d18)
            allocated by thread T11 here:
                #0 0x7fdd31cb89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
                #1 0x55a565b80492 in ut_allocator<unsigned char, true>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) /mariadb/10.6/storage/innobase/include/ut0new.h:375
                #2 0x55a565ce994d in mem_heap_create_block_func(mem_block_info_t*, unsigned long, char const*, unsigned int, unsigned long) /mariadb/10.6/storage/innobase/mem/mem0mem.cc:277
                #3 0x55a565dcb6f5 in mem_heap_create_func /mariadb/10.6/storage/innobase/include/mem0mem.inl:377
                #4 0x55a565dd3f6b in row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool) /mariadb/10.6/storage/innobase/row/row0ins.cc:3212
                #5 0x55a565dd65ba in row_ins_index_entry /mariadb/10.6/storage/innobase/row/row0ins.cc:3279
                #6 0x55a565dd68b7 in row_ins_index_entry_step /mariadb/10.6/storage/innobase/row/row0ins.cc:3445
                #7 0x55a565dd6e11 in row_ins /mariadb/10.6/storage/innobase/row/row0ins.cc:3591
            

            With ./mtr --rr you should even get a stack trace of the poisoning. It worked for me. It looks like a buffer overflow:

            #1  0x000055d4b14c4201 in mem_heap_alloc (heap=<optimized out>, n=15, 
                n@entry=7) at /mariadb/10.6/storage/innobase/include/mem0mem.inl:208
            #2  0x000055d4b14cd460 in page_cur_tuple_insert (
                cursor=cursor@entry=0x7f8427da1fa8, tuple=tuple@entry=0x61b0008c6320, 
                index=index@entry=0x616000a7ef20, offsets=offsets@entry=0x7f8427da1f20, 
                heap=heap@entry=0x7f8427da1f80, n_ext=0, mtr=0x7f8427da2360)
            

            We allocated 7 bytes (which the ASAN instrumentation expands to 15), and the read will be reported for byte offset 7 (1 byte after the allocated buffer).

            Also, could this ticket be duplicating MDEV-16950 in any way?

            marko Marko Mäkelä added a comment - nikitamalyavin , what is the status of this? If this is only targeting 10.3 and 10.4, why do the fix versions include 10.5 and 10.6? If this is not changing anything in 10.5 or 10.6, then I suppose that the the following, which I just got in a local AddressSanitizer build, would not be fixed by this? The issue is repeatable for my build, so I reran it with ASAN_OPTIONS=fast_unwind_on_malloc=0 to get a useful stack trace for the allocation: 10.6 19de8b7013536d61afcec2c0a4f2b4ccc5bf6b0d CURRENT_TEST: gcol.innodb_virtual_basic mysqltest: At line 1209: query 'replace into t(c) values ('')' failed: <Unknown> (2013): Lost connection to server during query … ==728994==ERROR: AddressSanitizer: use-after-poison on address 0x61a000352327 at pc 0x5616539f2780 bp 0x7fbcd6763450 sp 0x7fbcd6763448 READ of size 1 at 0x61a000352327 thread T22 #0 0x5616539f277f in page_cur_insert_rec_low(page_cur_t const*, dict_index_t*, unsigned char const*, unsigned short*, mtr_t*) /mariadb/10.6/storage/innobase/page/page0cur.cc:1664 #1 0x561653c525e5 in page_cur_tuple_insert /mariadb/10.6/storage/innobase/include/page0cur.inl:199 #2 0x561653c53532 in btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned short**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, unsigned long, que_thr_t*, mtr_t*) /mariadb/10.6/storage/innobase/btr/btr0cur.cc:3500 #3 0x561653aa4a57 in row_ins_sec_index_entry_low(unsigned long, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*) /mariadb/10.6/storage/innobase/row/row0ins.cc:3033 #4 0x561653aa5023 in row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool) /mariadb/10.6/storage/innobase/row/row0ins.cc:3231 #5 0x561653aa75ba in row_ins_index_entry /mariadb/10.6/storage/innobase/row/row0ins.cc:3279 #6 0x561653aa78b7 in row_ins_index_entry_step /mariadb/10.6/storage/innobase/row/row0ins.cc:3445 #7 0x561653aa7e11 in row_ins /mariadb/10.6/storage/innobase/row/row0ins.cc:3591 #8 0x561653aa8909 in row_ins_step(que_thr_t*) /mariadb/10.6/storage/innobase/row/row0ins.cc:3731 #9 0x561653ae06ab in row_insert_for_mysql(unsigned char const*, row_prebuilt_t*, ins_mode_t) /mariadb/10.6/storage/innobase/row/row0mysql.cc:1308 #10 0x56165382f9db in ha_innobase::write_row(unsigned char const*) /mariadb/10.6/storage/innobase/handler/ha_innodb.cc:7935 … 0x61a001924927 is located 167 bytes inside of 1176-byte region [0x61a001924880,0x61a001924d18) allocated by thread T11 here: #0 0x7fdd31cb89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 #1 0x55a565b80492 in ut_allocator<unsigned char, true>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) /mariadb/10.6/storage/innobase/include/ut0new.h:375 #2 0x55a565ce994d in mem_heap_create_block_func(mem_block_info_t*, unsigned long, char const*, unsigned int, unsigned long) /mariadb/10.6/storage/innobase/mem/mem0mem.cc:277 #3 0x55a565dcb6f5 in mem_heap_create_func /mariadb/10.6/storage/innobase/include/mem0mem.inl:377 #4 0x55a565dd3f6b in row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool) /mariadb/10.6/storage/innobase/row/row0ins.cc:3212 #5 0x55a565dd65ba in row_ins_index_entry /mariadb/10.6/storage/innobase/row/row0ins.cc:3279 #6 0x55a565dd68b7 in row_ins_index_entry_step /mariadb/10.6/storage/innobase/row/row0ins.cc:3445 #7 0x55a565dd6e11 in row_ins /mariadb/10.6/storage/innobase/row/row0ins.cc:3591 With ./mtr --rr you should even get a stack trace of the poisoning. It worked for me. It looks like a buffer overflow: #1 0x000055d4b14c4201 in mem_heap_alloc (heap=<optimized out>, n=15, n@entry=7) at /mariadb/10.6/storage/innobase/include/mem0mem.inl:208 #2 0x000055d4b14cd460 in page_cur_tuple_insert ( cursor=cursor@entry=0x7f8427da1fa8, tuple=tuple@entry=0x61b0008c6320, index=index@entry=0x616000a7ef20, offsets=offsets@entry=0x7f8427da1f20, heap=heap@entry=0x7f8427da1f80, n_ext=0, mtr=0x7f8427da2360) We allocated 7 bytes (which the ASAN instrumentation expands to 15), and the read will be reported for byte offset 7 (1 byte after the allocated buffer). Also, could this ticket be duplicating MDEV-16950 in any way?

            There are also MSAN failures on 10.6 with similar stack traces. It would make sense that the out-of-bounds location may contain an uninitialized value.

            10.6 19de8b7013536d61afcec2c0a4f2b4ccc5bf6b0d

            CURRENT_TEST: gcol.innodb_virtual_basic
            mysqltest: At line 1209: query 'replace into t(c) values ('')' failed: <Unknown> (2013): Lost connection to server during query
            …
            ==173428==WARNING: MemorySanitizer: use-of-uninitialized-value
                #0 0x5578f65f2c58 in page_cur_insert_rec_low(page_cur_t const*, dict_index_t*, unsigned char const*, unsigned short*, mtr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/page/page0cur.cc:1664:5
                #1 0x5578f6b86aac in page_cur_tuple_insert(page_cur_t*, dtuple_t const*, dict_index_t*, unsigned short**, mem_block_info_t**, unsigned long, mtr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/include/page0cur.inl:199:9
                #2 0x5578f6b7b8cf in btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned short**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, unsigned long, que_thr_t*, mtr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/btr/btr0cur.cc:3500:10
                #3 0x5578f67cbc70 in row_ins_sec_index_entry_low(unsigned long, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/row/row0ins.cc:3033:10
            

            marko Marko Mäkelä added a comment - There are also MSAN failures on 10.6 with similar stack traces. It would make sense that the out-of-bounds location may contain an uninitialized value. 10.6 19de8b7013536d61afcec2c0a4f2b4ccc5bf6b0d CURRENT_TEST: gcol.innodb_virtual_basic mysqltest: At line 1209: query 'replace into t(c) values ('')' failed: <Unknown> (2013): Lost connection to server during query … ==173428==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x5578f65f2c58 in page_cur_insert_rec_low(page_cur_t const*, dict_index_t*, unsigned char const*, unsigned short*, mtr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/page/page0cur.cc:1664:5 #1 0x5578f6b86aac in page_cur_tuple_insert(page_cur_t*, dtuple_t const*, dict_index_t*, unsigned short**, mem_block_info_t**, unsigned long, mtr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/include/page0cur.inl:199:9 #2 0x5578f6b7b8cf in btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned short**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, unsigned long, que_thr_t*, mtr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/btr/btr0cur.cc:3500:10 #3 0x5578f67cbc70 in row_ins_sec_index_entry_low(unsigned long, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*) /buildbot/amd64-ubuntu-2004-msan/build/storage/innobase/row/row0ins.cc:3033:10

            Sorry, I realized that the recent failure is due to my MDEV-29438. I will fix it in MDEV-29471.

            marko Marko Mäkelä added a comment - Sorry, I realized that the recent failure is due to my MDEV-29438 . I will fix it in MDEV-29471 .

            People

              nikitamalyavin Nikita Malyavin
              nikitamalyavin Nikita Malyavin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.