[MDEV-24750] Various corruptions caused by Aria subsystem asking system call to overwrite memory that it does not own (InnoDB stacks) Created: 2021-02-01  Updated: 2021-02-03  Resolved: 2021-02-03

Status: Closed
Project: MariaDB Server
Component/s: N/A
Affects Version/s: 10.6
Fix Version/s: 10.5.9

Type: Bug Priority: Critical
Reporter: Roel Van de Paar Assignee: Michael Widenius
Resolution: Fixed Votes: 0
Labels: affects-tests, corruption, hang

Attachments: File 1.sql     File 2.sql     File 3.sql    
Issue Links:
Duplicate
duplicates MDEV-24749 Various corruptions caused by Aria su... Closed

 Description   

This bug report is based on testcases very similar to the one in MDEV-24749, and it may be a duplicate. However, the results (crashes etc.) seen here are much more InnoDB oriented. It seems to me that the issues in Aria are affecting InnoDB. I am attaching a few different versions of this testcase as I keep getting different outcomes/results. It seems SOURCE is required at the CLI to reproduce these bugs. The testcase is also sporadic (though not much). Here are some of the stacks I have seen, all with some variation of the same testcase:

Seen with 1.sql and using 10.6 build from 26/1:
1) Assertion `table->magic_n == 76333786' failed in dict_table_get_first_index on debug:

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
mysqld: /data/builds/10.6_dbg/storage/innobase/include/dict0dict.ic:211: dict_index_t* dict_table_get_first_index(const dict_table_t*): Assertion `table->magic_n == 76333786' failed.

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
Core was generated by `/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
 
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
[Current thread is 1 (Thread 0x146aa80e4700 (LWP 4180868))]
 
(gdb) bt
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x000056027b3a1210 in my_write_core (sig=sig@entry=6) at /data/builds/10.6_dbg/mysys/stacktrace.c:424
 
#2  0x000056027ab362d0 in handle_fatal_signal (sig=6) at /data/builds/10.6_dbg/sql/signal_handler.cc:330
 
#3  <signal handler called>
 
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#5  0x0000146aa9347859 in __GI_abort () at abort.c:79
 
#6  0x0000146aa9347729 in __assert_fail_base (fmt=0x146aa94dd588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56027b748d26 "table->magic_n == 76333786", file=0x56027b74bd20 "/data/builds/10.6_dbg/storage/innobase/include/dict0dict.ic", line=211, function=<optimized out>) at assert.c:92
 
#7  0x0000146aa9358f36 in __GI___assert_fail (assertion=assertion@entry=0x56027b748d26 "table->magic_n == 76333786", file=file@entry=0x56027b74bd20 "/data/builds/10.6_dbg/storage/innobase/include/dict0dict.ic", line=line@entry=211, function=function@entry=0x56027b74ced0 "dict_index_t* dict_table_get_first_index(const dict_table_t*)") at assert.c:101
 
#8  0x000056027b0f05bf in dict_table_get_first_index (table=0x146a6400ace8) at /data/builds/10.6_dbg/storage/innobase/include/dict0dict.ic:211
 
#9  0x000056027b0ff68b in row_search_mvcc (buf=buf@entry=0x146a640236d8 "\376\002\255\345\060\061-01-01 10:10:10.999993", mode=<optimized out>, mode@entry=PAGE_CUR_UNSUPP, prebuilt=0x146a640253c8, match_mode=match_mode@entry=0, direction=direction@entry=1) at /data/builds/10.6_dbg/storage/innobase/row/row0sel.cc:4594
 
#10 0x000056027af328ac in ha_innobase::general_fetch (this=this@entry=0x146a64023b40, buf=buf@entry=0x146a640236d8 "\376\002\255\345\060\061-01-01 10:10:10.999993", direction=direction@entry=1, match_mode=match_mode@entry=0) at /data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc:8804
 
#11 0x000056027af423bb in ha_innobase::rnd_next (this=0x146a64023b40, buf=0x146a640236d8 "\376\002\255\345\060\061-01-01 10:10:10.999993") at /data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc:9008
 
#12 0x000056027ab3df27 in handler::ha_rnd_next (this=0x146a64023b40, buf=0x146a640236d8 "\376\002\255\345\060\061-01-01 10:10:10.999993") at /data/builds/10.6_dbg/sql/handler.cc:3066
 
#13 0x000056027ad11b6d in rr_sequential (info=0x146a6407fff0) at /data/builds/10.6_dbg/sql/records.h:82
 
#14 0x000056027a8bb04b in READ_RECORD::read_record (this=0x146a6407fff0) at /data/builds/10.6_dbg/sql/records.h:81
 
#15 sub_select (join=0x146a64014af8, join_tab=0x146a6407ff28, end_of_records=<optimized out>) at /data/builds/10.6_dbg/sql/sql_select.cc:20621
 
#16 0x000056027a8f3a22 in do_select (procedure=0x0, join=0x146a64014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:20149
 
#17 JOIN::exec_inner (this=this@entry=0x146a64014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4476
 
#18 0x000056027a8f3e92 in JOIN::exec (this=this@entry=0x146a64014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4256
 
#19 0x000056027a8f20f2 in mysql_select (thd=thd@entry=0x146a64000db8, tables=tables@entry=0x146a640127c0, fields=@0x146aa80e2d20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x56027be442e0 <end_of_list>, last = 0x146aa80e2d20, elements = 0}, <No data fields>}, conds=conds@entry=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2200096997504, result=0x146a64014a20, unit=0x146a64004f80, select_lex=0x146a64005780) at /data/builds/10.6_dbg/sql/sql_select.cc:4672
 
#20 0x000056027a969d93 in mysql_multi_update (thd=thd@entry=0x146a64000db8, table_list=0x146a640127c0, fields=fields@entry=0x146a640058d0, values=values@entry=0x146a64005e40, conds=0x0, options=0, handle_duplicates=DUP_ERROR, ignore=false, unit=0x146a64004f80, select_lex=0x146a64005780, result=0x146aa80e2f60) at /data/builds/10.6_dbg/sql/sql_update.cc:1950
 
#21 0x000056027a873366 in mysql_execute_command (thd=thd@entry=0x146a64000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:4372
 
#22 0x000056027a85e15e in mysql_parse (thd=thd@entry=0x146a64000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x146aa80e33d0) at /data/builds/10.6_dbg/sql/sql_parse.cc:7901
 
#23 0x000056027a86c24f in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x146a64000db8, packet=packet@entry=0x146a6401aac9 "UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))", packet_length=packet_length@entry=42) at /data/builds/10.6_dbg/sql/sql_class.h:1294
 
#24 0x000056027a86f581 in do_command (thd=0x146a64000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:1365
 
#25 0x000056027a9cb079 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x56027cf5d658, put_in_cache=put_in_cache@entry=true) at /data/builds/10.6_dbg/sql/sql_connect.cc:1410
 
#26 0x000056027a9cb77d in handle_one_connection (arg=arg@entry=0x56027cf5d658) at /data/builds/10.6_dbg/sql/sql_connect.cc:1312
 
#27 0x000056027ae7e43f in pfs_spawn_thread (arg=0x56027ce42ba8) at /data/builds/10.6_dbg/storage/perfschema/pfs.cc:2201
 
#28 0x0000146aa9855609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#29 0x0000146aa9444293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

2) SIGSEGV in plugin_lock on debug:

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
Core was generated by `/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
 
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
[Current thread is 1 (Thread 0x151148105700 (LWP 18478))]
 
(gdb) bt
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x000055acd8588210 in my_write_core (sig=sig@entry=11) at /data/builds/10.6_dbg/mysys/stacktrace.c:424
 
#2  0x000055acd7d1d2d0 in handle_fatal_signal (sig=11) at /data/builds/10.6_dbg/sql/signal_handler.cc:330
 
#3  <signal handler called>
 
#4  0x000055acd7a639a0 in plugin_lock (thd=thd@entry=0x0, ptr=0x151104008e08) at /data/builds/10.6_dbg/sql/sql_plugin.cc:1044
 
#5  0x000055acd7aad35e in create_internal_tmp_table_from_heap (thd=0x151104000db8, table=table@entry=0x151104084810, start_recinfo=<optimized out>, recinfo=<optimized out>, error=error@entry=135, ignore_last_dupp_key_error=ignore_last_dupp_key_error@entry=true, is_duplicate=0x0) at /data/builds/10.6_dbg/sql/sql_select.cc:19863
 
#6  0x000055acd7b4fa76 in multi_update::send_data (this=0x151104014a20, not_used_values=<optimized out>) at /data/builds/10.6_dbg/sql/sql_update.cc:2641
 
#7  0x000055acd7abcba8 in select_result_sink::send_data_with_check (sent=<optimized out>, u=<optimized out>, items=@0x151148103d20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x55acd902b2e0 <end_of_list>, last = 0x151148103d20, elements = 0}, <No data fields>}, this=<optimized out>) at /data/builds/10.6_dbg/sql/sql_class.h:5376
 
#8  end_send (join=0x151104014af8, join_tab=0x1511040815c8, end_of_records=<optimized out>) at /data/builds/10.6_dbg/sql/sql_select.cc:21802
 
#9  0x000055acd7a8b87e in evaluate_join_record (join=join@entry=0x151104014af8, join_tab=join_tab@entry=0x151104081218, error=error@entry=0) at /data/builds/10.6_dbg/sql/sql_select.cc:20825
 
#10 0x000055acd7aa2017 in sub_select (join=0x151104014af8, join_tab=0x151104081218, end_of_records=<optimized out>) at /data/builds/10.6_dbg/sql/sql_select.cc:20641
 
#11 0x000055acd7adaa22 in do_select (procedure=0x0, join=0x151104014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:20149
 
#12 JOIN::exec_inner (this=this@entry=0x151104014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4476
 
#13 0x000055acd7adae92 in JOIN::exec (this=this@entry=0x151104014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4256
 
#14 0x000055acd7ad90f2 in mysql_select (thd=thd@entry=0x151104000db8, tables=tables@entry=0x1511040127c0, fields=@0x151148103d20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x55acd902b2e0 <end_of_list>, last = 0x151148103d20, elements = 0}, <No data fields>}, conds=conds@entry=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2200096997504, result=0x151104014a20, unit=0x151104004f80, select_lex=0x151104005780) at /data/builds/10.6_dbg/sql/sql_select.cc:4672
 
#15 0x000055acd7b50d93 in mysql_multi_update (thd=thd@entry=0x151104000db8, table_list=0x1511040127c0, fields=fields@entry=0x1511040058d0, values=values@entry=0x151104005e40, conds=0x0, options=0, handle_duplicates=DUP_ERROR, ignore=false, unit=0x151104004f80, select_lex=0x151104005780, result=0x151148103f60) at /data/builds/10.6_dbg/sql/sql_update.cc:1950
 
#16 0x000055acd7a5a366 in mysql_execute_command (thd=thd@entry=0x151104000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:4372
 
#17 0x000055acd7a4515e in mysql_parse (thd=thd@entry=0x151104000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x1511481043d0) at /data/builds/10.6_dbg/sql/sql_parse.cc:7901
 
#18 0x000055acd7a5324f in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x151104000db8, packet=packet@entry=0x15110401aac9 "UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))", packet_length=packet_length@entry=42) at /data/builds/10.6_dbg/sql/sql_class.h:1294
 
#19 0x000055acd7a56581 in do_command (thd=0x151104000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:1365
 
#20 0x000055acd7bb2079 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55acdb872aa8, put_in_cache=put_in_cache@entry=true) at /data/builds/10.6_dbg/sql/sql_connect.cc:1410
 
#21 0x000055acd7bb277d in handle_one_connection (arg=arg@entry=0x55acdb872aa8) at /data/builds/10.6_dbg/sql/sql_connect.cc:1312
 
#22 0x000055acd806543f in pfs_spawn_thread (arg=0x55acdb7a7158) at /data/builds/10.6_dbg/storage/perfschema/pfs.cc:2201
 
#23 0x000015114afd0609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#24 0x000015114abbf293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

3) In optimized, Double free or corruption (out) then crash without stack and without core, on executing the testcase a few times and interrupting somewhere after a number of executions. This is already described in MDEV-24749.

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Optimized)

 
2021-02-01 12:22:11 0 [Note] /test/MD260121-mariadb-10.6.0-linux-x86_64-opt/bin/mysqld: ready for connections.
 
Version: '10.6.0-MariaDB'  socket: '/test/MD260121-mariadb-10.6.0-linux-x86_64-opt/socket.sock'  port: 16083  MariaDB Server
 
double free or corruption (out)
 
210201 12:24:00 [ERROR] mysqld got signal 6 ;

4) A hang in optimized after executing the testcase two times and then shutting down. This is different from MDEV-24749 as that hang happens during SQL execution. mysqladmin and the client just hang whereas the error log already shows a crash, again without stack in the error log, and without core.

Seen with 2.sql and using 10.6 build from 26/1:
1) SIGSEGV in dict_index_t::is_corrupted on 10.6 optimized (crashing at line 79 of 2.sql)
Notes: No additional information in error log. Issue seems highly reproducible, again using SOURCE 2.sql in CLI.

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Optimized)

 
Core was generated by `/test/MD260121-mariadb-10.6.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
 
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
[Current thread is 1 (Thread 0x15394c3df700 (LWP 606410))]
 
(gdb) bt
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x000055e63085c05f in my_write_core (sig=sig@entry=11) at /data/builds/10.6_opt/mysys/stacktrace.c:424
 
#2  0x000055e6302d0730 in handle_fatal_signal (sig=11) at /data/builds/10.6_opt/sql/signal_handler.cc:330
 
#3  <signal handler called>
 
#4  0x000055e6306c81c0 in dict_index_t::is_corrupted (this=0x15391001e250) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
 
#5  row_search_mvcc (buf=buf@entry=0x153910018198 "\376\002\255\345\060\061-01-01 10:10:10.999993", mode=mode@entry=PAGE_CUR_UNSUPP, prebuilt=0x15391001fcd0, match_mode=match_mode@entry=0, direction=direction@entry=1) at /data/builds/10.6_opt/storage/innobase/row/row0sel.cc:4301
 
#6  0x000055e630600ed8 in ha_innobase::general_fetch (match_mode=0, direction=1, buf=0x153910018198 "\376\002\255\345\060\061-01-01 10:10:10.999993", this=0x15391001f4a0) at /data/builds/10.6_opt/storage/innobase/handler/ha_innodb.cc:8804
 
#7  ha_innobase::rnd_next (this=0x15391001f4a0, buf=0x153910018198 "\376\002\255\345\060\061-01-01 10:10:10.999993") at /data/builds/10.6_opt/storage/innobase/handler/ha_innodb.cc:9008
 
#8  0x000055e6302d6c27 in handler::ha_rnd_next (this=0x15391001f4a0, buf=0x153910018198 "\376\002\255\345\060\061-01-01 10:10:10.999993") at /data/builds/10.6_opt/sql/handler.cc:3066
 
#9  0x000055e63042cab6 in rr_sequential (info=0x15391005a740) at /data/builds/10.6_opt/sql/records.h:82
 
#10 0x000055e6300fc66d in READ_RECORD::read_record (this=0x15391005a740) at /data/builds/10.6_opt/sql/records.h:81
 
#11 sub_select (end_of_records=false, join_tab=0x15391005a678, join=0x153910012818) at /data/builds/10.6_opt/sql/sql_select.cc:20621
 
#12 sub_select (join=0x153910012818, join_tab=0x15391005a678, end_of_records=false) at /data/builds/10.6_opt/sql/sql_select.cc:20531
 
#13 0x000055e63012aae2 in do_select (procedure=<optimized out>, join=0x153910012818) at /data/builds/10.6_opt/sql/sql_select.cc:20149
 
#14 JOIN::exec_inner (this=0x153910012818) at /data/builds/10.6_opt/sql/sql_select.cc:4476
 
#15 0x000055e63012ad78 in JOIN::exec (this=this@entry=0x153910012818) at /data/builds/10.6_opt/sql/sql_select.cc:4256
 
#16 0x000055e630128df8 in mysql_select (thd=thd@entry=0x153910000c58, tables=tables@entry=0x1539100104e0, fields=@0x15394c3ddde0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x55e6311bcf70 <end_of_list>, last = 0x15394c3ddde0, elements = 0}, <No data fields>}, conds=conds@entry=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2200096997504, result=0x153910012740, unit=0x153910004c60, select_lex=0x153910005460) at /data/builds/10.6_opt/sql/sql_select.cc:4672
 
#17 0x000055e63018214a in mysql_multi_update (thd=thd@entry=0x153910000c58, table_list=0x1539100104e0, fields=fields@entry=0x1539100055b0, values=values@entry=0x153910005b20, conds=0x0, options=0, handle_duplicates=DUP_ERROR, ignore=false, unit=0x153910004c60, select_lex=0x153910005460, result=0x15394c3ddfe0) at /data/builds/10.6_opt/sql/sql_update.cc:1950
 
#18 0x000055e6300c668c in mysql_execute_command (thd=0x153910000c58) at /data/builds/10.6_opt/sql/sql_parse.cc:4372
 
#19 0x000055e6300b3336 in mysql_parse (thd=0x153910000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/builds/10.6_opt/sql/sql_parse.cc:7901
 
#20 0x000055e6300bec18 in dispatch_command (command=COM_QUERY, thd=0x153910000c58, packet=0x153910008049 "UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))", packet_length=42) at /data/builds/10.6_opt/sql/sql_class.h:1294
 
#21 0x000055e6300c1016 in do_command (thd=0x153910000c58) at /data/builds/10.6_opt/sql/sql_parse.cc:1365
 
#22 0x000055e6301c60a1 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55e631fe19c8, put_in_cache=put_in_cache@entry=true) at /data/builds/10.6_opt/sql/sql_connect.cc:1410
 
#23 0x000055e6301c651d in handle_one_connection (arg=arg@entry=0x55e631fe19c8) at /data/builds/10.6_opt/sql/sql_connect.cc:1312
 
#24 0x000055e63054f2c9 in pfs_spawn_thread (arg=0x55e631f85ee8) at /data/builds/10.6_opt/storage/perfschema/pfs.cc:2201
 
#25 0x0000153962774609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#26 0x0000153962363293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

2) Failing assertion: strchr(table->name.m_name, '/') != NULL failed in dict_stats_update on 10.6 debug (crashing again at line 80 of 2.sql) which seems to be a secondary crash after the main one

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
2021-02-01 12:47:18 0 [Note] /test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld: ready for connections.
 
Version: '10.6.0-MariaDB-debug'  socket: '/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.sock'  port: 10503  MariaDB Server
 
Error: Freeing overrun buffer 0x15542c027050 at mysys/safemalloc.c:194, mysys/my_malloc.c:210, maria/ma_sort.c:719, maria/ma_check.c:4551, maria/ha_maria.cc:1657, maria/ha_maria.cc:2024, maria/ha_maria.cc:2263, sql/handler.cc:4654
 
Allocated at maria/ma_sort.c:631, maria/ma_check.c:4551, maria/ha_maria.cc:1657, maria/ha_maria.cc:2024, maria/ha_maria.cc:2263, sql/handler.cc:4654, sql/sql_select.cc:19840, sql/sql_update.cc:2641
 
Error: Freeing overrun buffer 0x15542c00bdc0 at mysys/safemalloc.c:194, mysys/my_malloc.c:210, maria/ma_sort.c:719, maria/ma_check.c:4551, maria/ha_maria.cc:1657, maria/ha_maria.cc:2024, maria/ha_maria.cc:2263, sql/handler.cc:4654
 
Allocated at maria/ma_sort.c:631, maria/ma_check.c:4551, maria/ha_maria.cc:1657, maria/ha_maria.cc:2024, maria/ha_maria.cc:2263, sql/handler.cc:4654, sql/sql_select.cc:19840, sql/sql_update.cc:2641
 
double free or corruption (out)
 
210201 12:47:24 [ERROR] mysqld got signal 6 ;
 
...
 
Server version: 10.6.0-MariaDB-debug
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=2
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467973 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x15542c000db8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x15545c907d38 thread_stack 0x49000
 
mysys/stacktrace.c:212(my_print_stacktrace)[0x5589e786e421]
 
sql/signal_handler.cc:208(handle_fatal_signal)[0x5589e7003013]
 
2021-02-01 12:47:33 0x15544e7fb700  InnoDB: Assertion failure in file /data/builds/10.6_dbg/storage/innobase/dict/dict0stats.cc line 3213
 
InnoDB: Failing assertion: strchr(table->name.m_name, '/') != NULL
 
InnoDB: We intentionally generate a memory trap.

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
Core was generated by `/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
[Current thread is 1 (Thread 0x15544e7fb700 (LWP 1135994))]
 
(gdb) bt
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#1  0x00001554727b4859 in __GI_abort () at abort.c:79
 
#2  0x00005589e764df5c in ut_dbg_assertion_failed (expr=expr@entry=0x5589e7ca08e8 "strchr(table->name.m_name, '/') != NULL", file=file@entry=0x5589e7c9f118 "/data/builds/10.6_dbg/storage/innobase/dict/dict0stats.cc", line=line@entry=3213) at /data/builds/10.6_dbg/storage/innobase/ut/ut0dbg.cc:60
 
#3  0x00005589e774f31b in dict_stats_update (table=table@entry=0x15542c020db8, stats_upd_option=stats_upd_option@entry=DICT_STATS_RECALC_PERSISTENT) at /data/builds/10.6_dbg/storage/innobase/dict/dict0stats.cc:3213
 
#4  0x00005589e7751dfd in dict_stats_process_entry_from_recalc_pool () at /data/builds/10.6_dbg/storage/innobase/dict/dict0stats_bg.cc:374
 
#5  dict_stats_func () at /data/builds/10.6_dbg/storage/innobase/dict/dict0stats_bg.cc:408
 
#6  0x00005589e77f9ece in tpool::thread_pool_generic::timer_generic::run (this=0x5589e9365240) at /data/builds/10.6_dbg/tpool/tpool_generic.cc:309
 
#7  tpool::thread_pool_generic::timer_generic::execute (arg=0x5589e9365240) at /data/builds/10.6_dbg/tpool/tpool_generic.cc:329
 
#8  0x00005589e77fae39 in tpool::task::execute (this=0x5589e9365280) at /data/builds/10.6_dbg/tpool/task.cc:52
 
#9  0x00005589e77f99e9 in tpool::thread_pool_generic::worker_main (this=0x5589e8fe17f0, thread_var=0x5589e8ff1290) at /data/builds/10.6_dbg/tpool/tpool_generic.cc:546
 
#10 0x00005589e77f9d20 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/9/bits/invoke.h:89
 
#11 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/9/bits/invoke.h:95
 
#12 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/9/thread:244
 
#13 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/9/thread:251
 
#14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/9/thread:195
 
#15 0x0000155472ba6d84 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 
#16 0x0000155472cc2609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#17 0x00001554728b1293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

3) Assertion `trx == thd_to_trx(m_user_thd)' failed in ha_innobase::general_fetch on 10.6 debug (crashing again at line 79 of 2.sql)

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
mysqld: /data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc:8791: int ha_innobase::general_fetch(uchar*, uint, uint): Assertion `trx == thd_to_trx(m_user_thd)' failed.

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
Core was generated by `/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
 
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
[Current thread is 1 (Thread 0x146d241f9700 (LWP 811041))]
 
(gdb) bt
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x0000560feea6c210 in my_write_core (sig=sig@entry=6) at /data/builds/10.6_dbg/mysys/stacktrace.c:424
 
#2  0x0000560fee2012d0 in handle_fatal_signal (sig=6) at /data/builds/10.6_dbg/sql/signal_handler.cc:330
 
#3  <signal handler called>
 
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#5  0x0000146d38e63859 in __GI_abort () at abort.c:79
 
#6  0x0000146d38e63729 in __assert_fail_base (fmt=0x146d38ff9588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x560feee149da "trx == thd_to_trx(m_user_thd)", file=0x560feee16dc8 "/data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc", line=8791, function=<optimized out>) at assert.c:92
 
#7  0x0000146d38e74f36 in __GI___assert_fail (assertion=assertion@entry=0x560feee149da "trx == thd_to_trx(m_user_thd)", file=file@entry=0x560feee16dc8 "/data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc", line=line@entry=8791, function=function@entry=0x560feee1ad40 "int ha_innobase::general_fetch(uchar*, uint, uint)") at assert.c:101
 
#8  0x0000560fee5fd8fd in ha_innobase::general_fetch (this=this@entry=0x146cf4025f70, buf=buf@entry=0x146cf4025b08 "\376\002\255\345\060\061-01-01 10:10:10.999993", direction=direction@entry=1, match_mode=match_mode@entry=0) at /data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc:8791
 
#9  0x0000560fee60d3bb in ha_innobase::rnd_next (this=0x146cf4025f70, buf=0x146cf4025b08 "\376\002\255\345\060\061-01-01 10:10:10.999993") at /data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc:9008
 
#10 0x0000560fee208f27 in handler::ha_rnd_next (this=0x146cf4025f70, buf=0x146cf4025b08 "\376\002\255\345\060\061-01-01 10:10:10.999993") at /data/builds/10.6_dbg/sql/handler.cc:3066
 
#11 0x0000560fee3dcb6d in rr_sequential (info=0x146cf4095710) at /data/builds/10.6_dbg/sql/records.h:82
 
#12 0x0000560fedf8604b in READ_RECORD::read_record (this=0x146cf4095710) at /data/builds/10.6_dbg/sql/records.h:81
 
#13 sub_select (join=0x146cf4014af8, join_tab=0x146cf4095648, end_of_records=<optimized out>) at /data/builds/10.6_dbg/sql/sql_select.cc:20621
 
#14 0x0000560fedfbea22 in do_select (procedure=0x0, join=0x146cf4014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:20149
 
#15 JOIN::exec_inner (this=this@entry=0x146cf4014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4476
 
#16 0x0000560fedfbee92 in JOIN::exec (this=this@entry=0x146cf4014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4256
 
#17 0x0000560fedfbd0f2 in mysql_select (thd=thd@entry=0x146cf4000db8, tables=tables@entry=0x146cf40127c0, fields=@0x146d241f7d20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x560fef50f2e0 <end_of_list>, last = 0x146d241f7d20, elements = 0}, <No data fields>}, conds=conds@entry=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2200096997504, result=0x146cf4014a20, unit=0x146cf4004f80, select_lex=0x146cf4005780) at /data/builds/10.6_dbg/sql/sql_select.cc:4672
 
#18 0x0000560fee034d93 in mysql_multi_update (thd=thd@entry=0x146cf4000db8, table_list=0x146cf40127c0, fields=fields@entry=0x146cf40058d0, values=values@entry=0x146cf4005e40, conds=0x0, options=0, handle_duplicates=DUP_ERROR, ignore=false, unit=0x146cf4004f80, select_lex=0x146cf4005780, result=0x146d241f7f60) at /data/builds/10.6_dbg/sql/sql_update.cc:1950
 
#19 0x0000560fedf3e366 in mysql_execute_command (thd=thd@entry=0x146cf4000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:4372
 
#20 0x0000560fedf2915e in mysql_parse (thd=thd@entry=0x146cf4000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x146d241f83d0) at /data/builds/10.6_dbg/sql/sql_parse.cc:7901
 
#21 0x0000560fedf3724f in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x146cf4000db8, packet=packet@entry=0x146cf401aac9 "UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))", packet_length=packet_length@entry=42) at /data/builds/10.6_dbg/sql/sql_class.h:1294
 
#22 0x0000560fedf3a581 in do_command (thd=0x146cf4000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:1365
 
#23 0x0000560fee096079 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x560ff0d9ca68, put_in_cache=put_in_cache@entry=true) at /data/builds/10.6_dbg/sql/sql_connect.cc:1410
 
#24 0x0000560fee09677d in handle_one_connection (arg=arg@entry=0x560ff0d9ca68) at /data/builds/10.6_dbg/sql/sql_connect.cc:1312
 
#25 0x0000560fee54943f in pfs_spawn_thread (arg=0x560ff0cd1438) at /data/builds/10.6_dbg/storage/perfschema/pfs.cc:2201
 
#26 0x0000146d39371609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#27 0x0000146d38f60293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

3) A hang, similar to previously described and also mentioned in MDEV-24749, this time while executing SQL. CLI + mysqladmin unusable (hang also on attempt), mysqld process still live while sig6 crash in error log already (the only thing that helps to kill mysqld here is kill -9 PID), no stack, no core, but Freeing overrun buffer message:

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
2021-02-01 12:39:21 0 [Note] /test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld: ready for connections.
 
Version: '10.6.0-MariaDB-debug'  socket: '/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.sock'  port: 10503  MariaDB Server
 
Error: Freeing overrun buffer 0x1530400286d0 at 0x55c0c2b99cb8, mysys/safemalloc.c:194, mysys/my_malloc.c:210, maria/ma_sort.c:719, maria/ma_check.c:4551, maria/ha_maria.cc:1657, maria/ha_maria.cc:2024, maria/ha_maria.cc:2263
 
Allocated at sql/handler.cc:4654, maria/ma_sort.c:631, maria/ma_check.c:4551, maria/ha_maria.cc:1657, maria/ha_maria.cc:2024, maria/ha_maria.cc:2263, sql/handler.cc:4654, sql/sql_select.cc:19840
 
realloc(): invalid old size
 
210201 12:39:26 [ERROR] mysqld got signal 6 ;



 Comments   
Comment by Roel Van de Paar [ 2021-02-01 ]

I was able to reduce the 3.sql (attached) to a more or less standard InnoDB bug testcase and report. Seems this particular sub-issue is present in 10.3. I do not expect all of the above issues and offshoots to be in 10.3 based on what I have seen in MDEV-24749 thus far.

DROP DATABASE test;
 
CREATE DATABASE test;
 
USE test;
 
SET SESSION aria_repair_threads=CAST(-1 AS UNSIGNED INT);
 
SET SESSION aria_sort_buffer_size=CAST(-1 AS UNSIGNED INT);
 
SET SESSION tmp_table_size=65535;
 
CREATE TABLE t1 (a BIT(7));
 
INSERT INTO t1 VALUES('C'), ('c');
 
ALTER TABLE t1 modify a VARCHAR(255);
 
XA BEGIN 'a';
 
INSERT INTO t1 VALUES('2001-01-01 00:00:01.000000');
 
INSERT INTO t1 VALUES('a');
 
INSERT INTO t1 VALUES(1), (3);
 
INSERT INTO t1 VALUES(0xACD4);
 
INSERT INTO t1 VALUES(0xABA8);
 
INSERT INTO t1 VALUES(1);
 
INSERT INTO t1 VALUES(0xF48F8080);
 
INSERT INTO t1 SELECT * FROM t1;
 
INSERT INTO t1 VALUES(0xA9A2);
 
INSERT t1 VALUES(30), (1230), ("1230"), ("12:30"), ("12:30:35"), ("1 12:30:31.32");
 
INSERT INTO t1 VALUES("19991101000000"), ("19990102030405"), ("19990630232922"), ("19990601000000");
 
INSERT INTO t1 VALUES('2004-01-01'), ('2004-02-29');
 
INSERT INTO t1 SELECT 1 FROM t1;
 
INSERT INTO t1 VALUES('2001-01-01 10:10:10.999993');
 
INSERT INTO t1 VALUES(0xADE5);
 
INSERT INTO t1 VALUES('');
 
INSERT INTO t1 SELECT * FROM t1;
 
INSERT INTO t1 VALUES('a');
 
INSERT INTO t1 VALUES('Z');
 
INSERT INTO t1 VALUES(12704);
 
INSERT INTO t1 VALUES('0.1');
 
INSERT INTO t1 VALUES('698aaaaaaaaaaaaaaaaaaaaaaaaaa');
 
INSERT INTO t1 VALUES(0xA9AA);
 
INSERT INTO t1 VALUES(unhex (hex (132)));
 
INSERT INTO t1 VALUES(1), (2), (1), (2), (1), (2), (3);
 
INSERT IGNORE INTO t1 VALUES(@inserted_value);
 
INSERT INTO t1 VALUES(15416);
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
INSERT INTO t1 VALUES('C'), ('c');
 
INSERT INTO t1 VALUES(1550);
 
INSERT INTO t1 VALUES('2001-01-01 00:00:01.000000');
 
INSERT INTO t1 VALUES('a');
 
INSERT INTO t1 VALUES(1), (3);
 
INSERT INTO t1 VALUES(0xACD4);
 
INSERT INTO t1 VALUES(0xABA8);
 
INSERT INTO t1 VALUES(1);
 
INSERT INTO t1 VALUES(0xF48F8080);
 
INSERT INTO t1 SELECT * FROM t1;
 
INSERT INTO t1 VALUES(0xA9A2);
 
INSERT t1 VALUES(30), (1230), ("1230"), ("12:30"), ("12:30:35"), ("1 12:30:31.32");
 
INSERT INTO t1 VALUES("19991101000000"), ("19990102030405"), ("19990630232922"), ("19990601000000");
 
INSERT INTO t1 VALUES('2004-01-01'), ('2004-02-29');
 
INSERT INTO t1 VALUES('2001-01-01 10:10:10.999993');
 
INSERT INTO t1 VALUES(0xADE5);
 
INSERT INTO t1 VALUES('');
 
INSERT INTO t1 SELECT * FROM t1;
 
INSERT INTO t1 VALUES('a');
 
INSERT INTO t1 VALUES('Z');
 
INSERT INTO t1 VALUES(12704);
 
INSERT INTO t1 VALUES('0.1');
 
INSERT INTO t1 VALUES('698aaaaaaaaaaaaaaaaaaaaaaaaaa');
 
INSERT INTO t1 VALUES(0xA9AA);
 
INSERT INTO t1 VALUES(unhex (hex (132)));
 
INSERT INTO t1 VALUES(1), (2), (1), (2), (1), (2), (3);
 
INSERT IGNORE INTO t1 VALUES(@inserted_value);
 
INSERT INTO t1 VALUES(15416);
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
 
XA END 'a';
 
USE test;

Leads to:

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
Core was generated by `/test/MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
 
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
[Current thread is 1 (Thread 0x14b518100700 (LWP 2167060))]
 
(gdb) bt
 
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
 
#1  0x0000556be7529210 in my_write_core (sig=sig@entry=11) at /data/builds/10.6_dbg/mysys/stacktrace.c:424
 
#2  0x0000556be6cbe2d0 in handle_fatal_signal (sig=11) at /data/builds/10.6_dbg/sql/signal_handler.cc:330
 
#3  <signal handler called>
 
#4  std::_Rb_tree<dict_table_t*, std::pair<dict_table_t* const, trx_mod_table_time_t>, std::_Select1st<std::pair<dict_table_t* const, trx_mod_table_time_t> >, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::_M_get_insert_unique_pos (__k=@0x14b4d400b6e8: 0x14b4d408dc28, this=0x14b5193bc320) at /usr/include/c++/9/bits/stl_function.h:433
 
#5  std::_Rb_tree<dict_table_t*, std::pair<dict_table_t* const, trx_mod_table_time_t>, std::_Select1st<std::pair<dict_table_t* const, trx_mod_table_time_t> >, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::_M_emplace_unique<dict_table_t*&, unsigned long&> (this=this@entry=0x14b5193bc320) at /usr/include/c++/9/bits/stl_tree.h:2413
 
#6  0x0000556be72d94c5 in std::map<dict_table_t*, trx_mod_table_time_t, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::emplace<dict_table_t*&, unsigned long&> (this=0x14b5193bc320) at /usr/include/c++/9/bits/stl_map.h:574
 
#7  trx_undo_report_row_operation (thr=thr@entry=0x14b4d40c5d18, index=index@entry=0x14b4d408e788, clust_entry=clust_entry@entry=0x0, update=update@entry=0x14b4d40b83f8, cmpl_info=cmpl_info@entry=1, rec=rec@entry=0x14b4f86e9157 "", offsets=0x14b4d402e860, roll_ptr=0x14b5180fd8d8) at /data/builds/10.6_dbg/storage/innobase/trx/trx0rec.cc:1998
 
#8  0x0000556be7336ea9 in btr_cur_upd_lock_and_undo (flags=flags@entry=10, cursor=cursor@entry=0x14b4d40a6648, offsets=0x14b4d402e860, update=update@entry=0x14b4d40b83f8, cmpl_info=cmpl_info@entry=1, thr=thr@entry=0x14b4d40c5d18, mtr=0x14b5180fe0a0, roll_ptr=0x14b5180fd8d8) at /data/builds/10.6_dbg/storage/innobase/btr/btr0cur.cc:3863
 
#9  0x0000556be7344fb3 in btr_cur_pessimistic_update (flags=flags@entry=10, cursor=cursor@entry=0x14b4d40a6648, offsets=offsets@entry=0x14b5180fd988, offsets_heap=offsets_heap@entry=0x14b5180fda48, entry_heap=<optimized out>, big_rec=big_rec@entry=0x14b5180fd998, update=0x14b4d40b83f8, cmpl_info=1, thr=0x14b4d40c5d18, trx_id=65, mtr=0x14b5180fe0a0) at /data/builds/10.6_dbg/storage/innobase/btr/btr0cur.cc:5020
 
#10 0x0000556be729c89f in row_upd_clust_rec (flags=flags@entry=0, node=node@entry=0x14b4d40b82d0, index=index@entry=0x14b4d408e788, offsets=<optimized out>, offsets@entry=0x14b5180fda60, offsets_heap=offsets_heap@entry=0x14b5180fda48, thr=thr@entry=0x14b4d40c5d18, mtr=0x14b5180fe0a0) at /data/builds/10.6_dbg/storage/innobase/row/row0upd.cc:2600
 
#11 0x0000556be729e6f1 in row_upd_clust_step (node=node@entry=0x14b4d40b82d0, thr=thr@entry=0x14b4d40c5d18) at /data/builds/10.6_dbg/storage/innobase/row/row0upd.cc:2888
 
#12 0x0000556be72a0ceb in row_upd (thr=0x14b4d40c5d18, node=0x14b4d40b82d0) at /data/builds/10.6_dbg/storage/innobase/row/row0upd.cc:2992
 
#13 row_upd_step (thr=thr@entry=0x14b4d40c5d18) at /data/builds/10.6_dbg/storage/innobase/row/row0upd.cc:3136
 
#14 0x0000556be7241ac1 in row_update_for_mysql (prebuilt=0x14b4d40b77a8) at /data/builds/10.6_dbg/storage/innobase/row/row0mysql.cc:1854
 
#15 0x0000556be70c6257 in ha_innobase::update_row (this=0x14b4d40b58c0, old_row=0x14b4d40abe90 "\376\001C\345\064\061\066", new_row=0x14b4d40abd88 "\376\002\255\345\064\061\066") at /data/builds/10.6_dbg/storage/innobase/handler/ha_innodb.cc:8130
 
#16 0x0000556be6cd2c37 in handler::ha_update_row (this=0x14b4d40b58c0, old_data=0x14b4d40abe90 "\376\001C\345\064\061\066", new_data=0x14b4d40abd88 "\376\002\255\345\064\061\066") at /data/builds/10.6_dbg/sql/handler.cc:7204
 
#17 0x0000556be6af2946 in multi_update::do_updates (this=this@entry=0x14b4d4014a20) at /data/builds/10.6_dbg/sql/sql_update.cc:2877
 
#18 0x0000556be6af3634 in multi_update::send_eof (this=0x14b4d4014a20) at /data/builds/10.6_dbg/sql/sql_class.h:2501
 
#19 0x0000556be6a7bb99 in do_select (procedure=<optimized out>, join=0x14b4d4014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:20204
 
#20 JOIN::exec_inner (this=this@entry=0x14b4d4014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4476
 
#21 0x0000556be6a7be92 in JOIN::exec (this=this@entry=0x14b4d4014af8) at /data/builds/10.6_dbg/sql/sql_select.cc:4256
 
#22 0x0000556be6a7a0f2 in mysql_select (thd=thd@entry=0x14b4d4000db8, tables=tables@entry=0x14b4d40127c0, fields=@0x14b5180fed20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x556be7fcc2e0 <end_of_list>, last = 0x14b5180fed20, elements = 0}, <No data fields>}, conds=conds@entry=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2200096997504, result=0x14b4d4014a20, unit=0x14b4d4004f80, select_lex=0x14b4d4005780) at /data/builds/10.6_dbg/sql/sql_select.cc:4672
 
#23 0x0000556be6af1d93 in mysql_multi_update (thd=thd@entry=0x14b4d4000db8, table_list=0x14b4d40127c0, fields=fields@entry=0x14b4d40058d0, values=values@entry=0x14b4d4005e40, conds=0x0, options=0, handle_duplicates=DUP_ERROR, ignore=false, unit=0x14b4d4004f80, select_lex=0x14b4d4005780, result=0x14b5180fef60) at /data/builds/10.6_dbg/sql/sql_update.cc:1950
 
#24 0x0000556be69fb366 in mysql_execute_command (thd=thd@entry=0x14b4d4000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:4372
 
#25 0x0000556be69e615e in mysql_parse (thd=thd@entry=0x14b4d4000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14b5180ff3d0) at /data/builds/10.6_dbg/sql/sql_parse.cc:7901
 
#26 0x0000556be69f424f in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14b4d4000db8, packet=packet@entry=0x14b4d401aac9 "UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))", packet_length=packet_length@entry=42) at /data/builds/10.6_dbg/sql/sql_class.h:1294
 
#27 0x0000556be69f7581 in do_command (thd=0x14b4d4000db8) at /data/builds/10.6_dbg/sql/sql_parse.cc:1365
 
#28 0x0000556be6b53079 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x556bea08a668, put_in_cache=put_in_cache@entry=true) at /data/builds/10.6_dbg/sql/sql_connect.cc:1410
 
#29 0x0000556be6b5377d in handle_one_connection (arg=arg@entry=0x556bea08a668) at /data/builds/10.6_dbg/sql/sql_connect.cc:1312
 
#30 0x0000556be700643f in pfs_spawn_thread (arg=0x556be9f6fed8) at /data/builds/10.6_dbg/storage/perfschema/pfs.cc:2201
 
#31 0x000014b51c1cc609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#32 0x000014b51bdbb293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.3.28 (dbg), 10.3.28 (opt), 10.4.18 (dbg), 10.4.18 (opt), 10.5.9 (dbg), 10.5.9 (opt), 10.6.0 (dbg), 10.6.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.37 (dbg), 10.2.37 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.33 (dbg), 5.7.33 (opt), 8.0.23 (dbg), 8.0.23 (opt)

This testcase again leads to different stacks on different releases. Here's a uniqueID's overview, can also provide full stacks if needed.

SIGSEGV|lf_pinbox_real_free|lf_pinbox_put_pins|MDL_context::destroy|THD::~THD
 
SIGSEGV|row_sel_store_mysql_rec|row_search_mvcc|ha_innobase::general_fetch|handler::ha_rnd_next
 
SIGSEGV|std::_Rb_tree<dict_table_t*, std::pair<dict_table_t* const, trx_mod_table_time_t>, std::_Select1st<std::pair<dict_table_t* const, trx_mod_table_time_t> >, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::_M_get_insert_unique_pos|std::_Rb_tree<dict_table_t*, std::pair<dict_table_t* const, trx_mod_table_time_t>, std::_Select1st<std::pair<dict_table_t* const, trx_mod_table_time_t> >, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::_M_emplace_unique<dict_table_t*&, unsigned long&>|std::map<dict_table_t*, trx_mod_table_time_t, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::emplace<dict_table_t*&, unsigned long&>|trx_undo_report_row_operation
 
SIGSEGV|std::less<dict_table_t*>::operatorstl_function.h|std::_Rb_tree<dict_table_t*, std::pair<dict_table_t* const, trx_mod_table_time_t>, std::_Select1st<std::pair<dict_table_t* const, trx_mod_table_time_t> >, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::_M_get_insert_unique_pos|std::_Rb_tree<dict_table_t*, std::pair<dict_table_t* const, trx_mod_table_time_t>, std::_Select1st<std::pair<dict_table_t* const, trx_mod_table_time_t> >, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::_M_insert_unique<std::pair<dict_table_t* const, trx_mod_table_time_t> >|std::map<dict_table_t*, trx_mod_table_time_t, std::less<dict_table_t*>, ut_allocator<std::pair<dict_table_t* const, trx_mod_table_time_t>, true> >::insert
 
mutex->__data.__owner == 0|SIGABRT|__GI___pthread_mutex_lock|inline_mysql_mutex_lock|maria_close|closefrm
 
open_tables == __null|SIGABRT|THD::cleanup|THD::free_connection|THD::~THD|THD::~THD
 
strchr(path, '/') != __null|SIGABRT|fil_op_write_log|fil_name_write|fil_name_write|fil_names_write

Comment by Roel Van de Paar [ 2021-02-01 ]

I ran the testcase from the last comment (not 3.sql but the in-comment reduced version) through UBSAN/ASAN. I got an direct stderr/stdout output on exiting from the client (with previous mysqld termination) besides a report to error log. Here is the console output as observed:

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
... testcase executing in cli...
 
Query OK, 1 row affected (0.000 sec)
 
 
 
ERROR 2013 (HY000) at line 84 in file: 'in.sql': Lost connection to MySQL server during query
 
ERROR 2006 (HY000) at line 85 in file: 'in.sql': MySQL server has gone away
 
No connection. Trying to reconnect...
 
ERROR 2002 (HY000) at line 85 in file: 'in.sql': Can't connect to local MySQL server through socket '/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.soc' (111)
 
ERROR at line 85 in file: 'in.sql': Can't connect to the server
 
 
 
No connection. Trying to reconnect...
 
ERROR 2002 (HY000) at line 86 in file: 'in.sql': Can't connect to local MySQL server through socket '/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.soc' (111)
 
ERROR at line 86 in file: 'in.sql': Can't connect to the server
 
 
 
No connection. Trying to reconnect...
 
ERROR 2002 (HY000) at line 87 in file: 'in.sql': Can't connect to local MySQL server through socket '/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.soc' (111)
 
ERROR at line 87 in file: 'in.sql': Can't connect to the server
 
 
 
10.6.0>exit
 
Bye
 
 
 
=================================================================
 
==3120955==ERROR: LeakSanitizer: detected memory leaks
 
 
 
Direct leak of 128 byte(s) in 1 object(s) allocated from:
 
    #0 0x55812618df46 in __interceptor_calloc (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x4b4f46)
 
    #1 0x558126208efa in mysql_init (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x52fefa)
 
    #2 0x5581261d94fd  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5004fd)
 
    #3 0x5581261d9c34  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500c34)
 
    #4 0x5581261d9ef7  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500ef7)
 
    #5 0x5581261e376f  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x50a76f)
 
    #6 0x5581261ec0c8  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5130c8)
 
    #7 0x5581261ef8af  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5168af)
 
    #8 0x5581261e9b6c  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x510b6c)
 
    #9 0x5581261eee83 in main (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x515e83)
 
    #10 0x1527d69e00b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
 
 
Direct leak of 128 byte(s) in 1 object(s) allocated from:
 
    #0 0x55812618df46 in __interceptor_calloc (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x4b4f46)
 
    #1 0x558126208efa in mysql_init (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x52fefa)
 
    #2 0x5581261d94fd  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5004fd)
 
    #3 0x5581261d9c34  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500c34)
 
    #4 0x5581261d9ef7  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500ef7)
 
    #5 0x5581261d9f9e in mysql_real_query_for_lazy(char const*, unsigned long) (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500f9e)
 
    #6 0x5581261e3697  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x50a697)
 
    #7 0x5581261ec0c8  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5130c8)
 
    #8 0x5581261ef8af  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5168af)
 
    #9 0x5581261e9b6c  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x510b6c)
 
    #10 0x5581261eee83 in main (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x515e83)
 
    #11 0x1527d69e00b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
 
 
Direct leak of 8 byte(s) in 1 object(s) allocated from:
 
    #0 0x55812618df46 in __interceptor_calloc (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x4b4f46)
 
    #1 0x558126208ea0 in mysql_init (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x52fea0)
 
    #2 0x5581261d94fd  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5004fd)
 
    #3 0x5581261d9c34  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500c34)
 
    #4 0x5581261d9ef7  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500ef7)
 
    #5 0x5581261e376f  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x50a76f)
 
    #6 0x5581261ec0c8  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5130c8)
 
    #7 0x5581261ef8af  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5168af)
 
    #8 0x5581261e9b6c  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x510b6c)
 
    #9 0x5581261eee83 in main (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x515e83)
 
    #10 0x1527d69e00b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
 
 
Direct leak of 8 byte(s) in 1 object(s) allocated from:
 
    #0 0x55812618df46 in __interceptor_calloc (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x4b4f46)
 
    #1 0x558126208ea0 in mysql_init (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x52fea0)
 
    #2 0x5581261d94fd  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5004fd)
 
    #3 0x5581261d9c34  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500c34)
 
    #4 0x5581261d9ef7  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500ef7)
 
    #5 0x5581261d9f9e in mysql_real_query_for_lazy(char const*, unsigned long) (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x500f9e)
 
    #6 0x5581261e3697  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x50a697)
 
    #7 0x5581261ec0c8  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5130c8)
 
    #8 0x5581261ef8af  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x5168af)
 
    #9 0x5581261e9b6c  (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x510b6c)
 
    #10 0x5581261eee83 in main (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadb+0x515e83)
 
    #11 0x1527d69e00b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
 
 
SUMMARY: AddressSanitizer: 272 byte(s) leaked in 4 allocation(s).
 
v./cl: line 4: 3120955 Aborted                 (core dumped) /test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysql -A -uroot -S/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.sock --force --prompt="$(/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --version | grep -o 'Ver [\.0-9]\+' | sed 's|[^\.0-9]*||')>" --binary-mode test

From the error log:

10.6.0 3f871b339429441ad907ecf7dfabdc414797e664 (Debug)

 
Version: '10.6.0-MariaDB-debug'  socket: '/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/socket.sock'  port: 13592  MariaDB Server
 
=================================================================
 
==3119705==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60d0000000cc at pc 0x5638c98a175d bp 0x1501a00d2930 sp 0x1501a00d20d8
 
WRITE of size 15 at 0x60d0000000cc thread T13
 
    #0 0x5638c98a175c in __interceptor_pread64.part.0 (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadbd+0x7b5d75c)
 
    #1 0x5638ce7ddf3b in my_pread /data/builds/10.6_dbg_san/mysys/my_pread.c:66
 
    #2 0x5638ce77ba9a in inline_mysql_file_pread /data/builds/10.6_dbg_san/include/mysql/psi/mysql_file.h:1206
 
    #3 0x5638ce77ba9a in my_b_pread /data/builds/10.6_dbg_san/mysys/mf_iocache2.c:198
 
    #4 0x5638ccf9b5ca in read_to_buffer_varlen /data/builds/10.6_dbg_san/storage/maria/ma_sort.c:955
 
    #5 0x5638ccf9c286 in merge_buffers /data/builds/10.6_dbg_san/storage/maria/ma_sort.c:1036
 
    #6 0x5638ccf9e05d in merge_index /data/builds/10.6_dbg_san/storage/maria/ma_sort.c:1146
 
    #7 0x5638ccfa5ad4 in _ma_thr_write_keys /data/builds/10.6_dbg_san/storage/maria/ma_sort.c:664
 
    #8 0x5638ccf78585 in maria_repair_parallel /data/builds/10.6_dbg_san/storage/maria/ma_check.c:4551
 
    #9 0x5638ccd71363 in ha_maria::repair(THD*, st_handler_check_param*, bool) /data/builds/10.6_dbg_san/storage/maria/ha_maria.cc:1657
 
    #10 0x5638ccd748f2 in ha_maria::enable_indexes(unsigned int) /data/builds/10.6_dbg_san/storage/maria/ha_maria.cc:2024
 
    #11 0x5638ccd75c3f in ha_maria::end_bulk_insert() /data/builds/10.6_dbg_san/storage/maria/ha_maria.cc:2262
 
    #12 0x5638cb7ff1a2 in handler::ha_end_bulk_insert() /data/builds/10.6_dbg_san/sql/handler.cc:4654
 
    #13 0x5638ca453ce3 in create_internal_tmp_table_from_heap(THD*, TABLE*, st_maria_columndef*, st_maria_columndef**, int, bool, bool*) /data/builds/10.6_dbg_san/sql/sql_select.cc:19840
 
    #14 0x5638ca94c682 in multi_update::send_data(List<Item>&) /data/builds/10.6_dbg_san/sql/sql_update.cc:2641
 
    #15 0x5638ca4bda3e in select_result_sink::send_data_with_check(List<Item>&, st_select_lex_unit*, unsigned long long) /data/builds/10.6_dbg_san/sql/sql_class.h:5376
 
    #16 0x5638ca4bda3e in end_send /data/builds/10.6_dbg_san/sql/sql_select.cc:21802
 
    #17 0x5638ca363f11 in evaluate_join_record /data/builds/10.6_dbg_san/sql/sql_select.cc:20825
 
    #18 0x5638ca3f3827 in sub_select(JOIN*, st_join_table*, bool) /data/builds/10.6_dbg_san/sql/sql_select.cc:20641
 
    #19 0x5638ca5b60ef in do_select /data/builds/10.6_dbg_san/sql/sql_select.cc:20149
 
    #20 0x5638ca5b60ef in JOIN::exec_inner() /data/builds/10.6_dbg_san/sql/sql_select.cc:4476
 
    #21 0x5638ca5b796c in JOIN::exec() /data/builds/10.6_dbg_san/sql/sql_select.cc:4256
 
    #22 0x5638ca5a8a97 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /data/builds/10.6_dbg_san/sql/sql_select.cc:4672
 
    #23 0x5638ca957657 in mysql_multi_update(THD*, TABLE_LIST*, List<Item>*, List<Item>*, Item*, unsigned long long, enum_duplicates, bool, st_select_lex_unit*, st_select_lex*, multi_update**) /data/builds/10.6_dbg_san/sql/sql_update.cc:1950
 
    #24 0x5638ca223ae9 in mysql_execute_command(THD*) /data/builds/10.6_dbg_san/sql/sql_parse.cc:4372
 
    #25 0x5638ca1802ea in mysql_parse(THD*, char*, unsigned int, Parser_state*) /data/builds/10.6_dbg_san/sql/sql_parse.cc:7901
 
    #26 0x5638ca1ef012 in dispatch_command(enum_server_command, THD*, char*, unsigned int) /data/builds/10.6_dbg_san/sql/sql_parse.cc:1833
 
    #27 0x5638ca2045e4 in do_command(THD*) /data/builds/10.6_dbg_san/sql/sql_parse.cc:1365
 
    #28 0x5638cabec5bc in do_handle_one_connection(CONNECT*, bool) /data/builds/10.6_dbg_san/sql/sql_connect.cc:1410
 
    #29 0x5638cabef83f in handle_one_connection /data/builds/10.6_dbg_san/sql/sql_connect.cc:1312
 
    #30 0x5638cd0f0631 in pfs_spawn_thread /data/builds/10.6_dbg_san/storage/perfschema/pfs.cc:2201
 
    #31 0x1501c2dac608 in start_thread /build/glibc-ZN95T4/glibc-2.31/nptl/pthread_create.c:477
 
    #32 0x1501c1f00292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292)
 
0x60d0000000cc is located 0 bytes to the right of 140-byte region [0x60d000000040,0x60d0000000cc)
 
allocated by thread T13 here:
 
    #0 0x5638c99565f8 in __interceptor_malloc (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadbd+0x7c125f8)
 
    #1 0x5638ce81cec8 in sf_malloc /data/builds/10.6_dbg_san/mysys/safemalloc.c:118
 
    #2 0x5638ce7da56b in my_malloc /data/builds/10.6_dbg_san/mysys/my_malloc.c:88
 
    #3 0x5638ccfa56cd in _ma_thr_write_keys /data/builds/10.6_dbg_san/storage/maria/ma_sort.c:631
 
    #4 0x5638ccf78585 in maria_repair_parallel /data/builds/10.6_dbg_san/storage/maria/ma_check.c:4551
 
    #5 0x5638ccd71363 in ha_maria::repair(THD*, st_handler_check_param*, bool) /data/builds/10.6_dbg_san/storage/maria/ha_maria.cc:1657
 
    #6 0x5638ccd748f2 in ha_maria::enable_indexes(unsigned int) /data/builds/10.6_dbg_san/storage/maria/ha_maria.cc:2024
 
    #7 0x5638ccd75c3f in ha_maria::end_bulk_insert() /data/builds/10.6_dbg_san/storage/maria/ha_maria.cc:2262
 
    #8 0x5638cb7ff1a2 in handler::ha_end_bulk_insert() /data/builds/10.6_dbg_san/sql/handler.cc:4654
 
    #9 0x5638ca453ce3 in create_internal_tmp_table_from_heap(THD*, TABLE*, st_maria_columndef*, st_maria_columndef**, int, bool, bool*) /data/builds/10.6_dbg_san/sql/sql_select.cc:19840
 
    #10 0x5638ca94c682 in multi_update::send_data(List<Item>&) /data/builds/10.6_dbg_san/sql/sql_update.cc:2641
 
    #11 0x5638ca4bda3e in select_result_sink::send_data_with_check(List<Item>&, st_select_lex_unit*, unsigned long long) /data/builds/10.6_dbg_san/sql/sql_class.h:5376
 
    #12 0x5638ca4bda3e in end_send /data/builds/10.6_dbg_san/sql/sql_select.cc:21802
 
    #13 0x5638ca363f11 in evaluate_join_record /data/builds/10.6_dbg_san/sql/sql_select.cc:20825
 
    #14 0x5638ca3f3827 in sub_select(JOIN*, st_join_table*, bool) /data/builds/10.6_dbg_san/sql/sql_select.cc:20641
 
    #15 0x5638ca5b60ef in do_select /data/builds/10.6_dbg_san/sql/sql_select.cc:20149
 
    #16 0x5638ca5b60ef in JOIN::exec_inner() /data/builds/10.6_dbg_san/sql/sql_select.cc:4476
 
    #17 0x5638ca5b796c in JOIN::exec() /data/builds/10.6_dbg_san/sql/sql_select.cc:4256
 
    #18 0x5638ca5a8a97 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /data/builds/10.6_dbg_san/sql/sql_select.cc:4672
 
    #19 0x5638ca957657 in mysql_multi_update(THD*, TABLE_LIST*, List<Item>*, List<Item>*, Item*, unsigned long long, enum_duplicates, bool, st_select_lex_unit*, st_select_lex*, multi_update**) /data/builds/10.6_dbg_san/sql/sql_update.cc:1950
 
    #20 0x5638ca223ae9 in mysql_execute_command(THD*) /data/builds/10.6_dbg_san/sql/sql_parse.cc:4372
 
    #21 0x5638ca1802ea in mysql_parse(THD*, char*, unsigned int, Parser_state*) /data/builds/10.6_dbg_san/sql/sql_parse.cc:7901
 
    #22 0x5638ca1ef012 in dispatch_command(enum_server_command, THD*, char*, unsigned int) /data/builds/10.6_dbg_san/sql/sql_parse.cc:1833
 
    #23 0x5638ca2045e4 in do_command(THD*) /data/builds/10.6_dbg_san/sql/sql_parse.cc:1365
 
    #24 0x5638cabec5bc in do_handle_one_connection(CONNECT*, bool) /data/builds/10.6_dbg_san/sql/sql_connect.cc:1410
 
    #25 0x5638cabef83f in handle_one_connection /data/builds/10.6_dbg_san/sql/sql_connect.cc:1312
 
    #26 0x5638cd0f0631 in pfs_spawn_thread /data/builds/10.6_dbg_san/storage/perfschema/pfs.cc:2201
 
    #27 0x1501c2dac608 in start_thread /build/glibc-ZN95T4/glibc-2.31/nptl/pthread_create.c:477
 
Thread T13 created by T0 here:
 
    #0 0x5638c9883265 in pthread_create (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadbd+0x7b3f265)
 
    #1 0x5638cd101149 in my_thread_create /data/builds/10.6_dbg_san/storage/perfschema/my_thread.h:38
 
    #2 0x5638cd101149 in pfs_spawn_thread_v1 /data/builds/10.6_dbg_san/storage/perfschema/pfs.cc:2252
 
    #3 0x5638c99b1305 in inline_mysql_thread_create /data/builds/10.6_dbg_san/include/mysql/psi/mysql_thread.h:1323
 
    #4 0x5638c99b1305 in create_thread_to_handle_connection(CONNECT*) /data/builds/10.6_dbg_san/sql/mysqld.cc:5806
 
    #5 0x5638c99c4d9f in create_new_thread(CONNECT*) /data/builds/10.6_dbg_san/sql/mysqld.cc:5865
 
    #6 0x5638c99c53d4 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /data/builds/10.6_dbg_san/sql/mysqld.cc:5930
 
    #7 0x5638c99c6e58 in handle_connections_sockets() /data/builds/10.6_dbg_san/sql/mysqld.cc:6057
 
    #8 0x5638c99ca96a in mysqld_main(int, char**) /data/builds/10.6_dbg_san/sql/mysqld.cc:5701
 
    #9 0x5638c9997baa in main /data/builds/10.6_dbg_san/sql/main.cc:25
 
    #10 0x1501c1e050b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
 
 
 
SUMMARY: AddressSanitizer: heap-buffer-overflow (/test/UBASAN_MD260121-mariadb-10.6.0-linux-x86_64-dbg/bin/mariadbd+0x7b5d75c) in __interceptor_pread64.part.0
 
Shadow bytes around the buggy address:
 
  0x0c1a7fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
  0x0c1a7fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
  0x0c1a7fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
  0x0c1a7fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 
  0x0c1a7fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
 
=>0x0c1a7fff8010: 00 00 00 00 00 00 00 00 00[04]fa fa fa fa fa fa
 
  0x0c1a7fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c1a7fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c1a7fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c1a7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c1a7fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07
 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
  Shadow gap:              cc
 
==3119705==ABORTING
 
210201 19:23:02 [ERROR] mysqld got signal 6 ;
 
...
 
Query (0x62b0000a12a8): UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))

Comment by Marko Mäkelä [ 2021-02-01 ]

I think that InnoDB is only a victim of corruption that was inflicted by Aria. Here is a quick analysis of a trace:

ssh rr
 
echo 'continue
 
print server_version_source_revision
 
reverse-continue
 
frame 4
 
watch -l *cursor->old_rec
 
reverse-continue
 
set height 0
 
backtrace
 
quit'|_RR_TRACE_DIR=/test/MD230121-10.6-dbg-MDEV-24750/rr rr replay

The output ends as follows:

10.6 9118fd360a3da0bba521caf2a35c424968235ac4

 
(rr) Hardware watchpoint 1: -location *cursor->old_rec
 
(rr) Continuing.
 
 
 
Thread 2 hit Hardware watchpoint 1: -location *cursor->old_rec
 
 
 
Old value = 2 '\002'
 
New value = 0 '\000'
 
0x00005631120de4dd in local_memcpy (n=15, source=0x153edf00d5e6, dest=0x2f3d54016048)
 
    at ./src/preload/syscallbuf.c:989
 
989	./src/preload/syscallbuf.c: No such file or directory.
 
(rr) (rr) #0  0x00005631120de4dd in local_memcpy (n=15, source=0x153edf00d5e6, dest=0x2f3d54016048)
 
    at ./src/preload/syscallbuf.c:989
 
#1  copy_output_buffer (buf2=0x153edf00d5e6, buf=0x2f3d54016048, ptr=0x153edf00d5f5, ret_size=15)
 
    at ./src/preload/syscallbuf.c:989
 
#2  sys_pread64 (call=0x153edf30bfa0) at ./src/preload/syscallbuf.c:2089
 
#3  syscall_hook_internal (call=0x153edf30bfa0) at ./src/preload/syscallbuf.c:2891
 
#4  syscall_hook (call=0x153edf30bfa0) at ./src/preload/syscallbuf.c:2987
 
#5  0x00005631120dc1da in _syscall_hook_trampoline ()
 
    at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:282
 
#6  0x00005631120dc20a in __morestack () at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:417
 
#7  0x00005631120dc225 in _syscall_hook_trampoline_48_3d_00_f0_ff_ff ()
 
    at /build/rr-S0CLEN/rr-5.3.0/src/preload/syscall_hook.S:428
 
#8  0x0000457e72501c15 in __libc_pread64 (offset=<optimized out>, count=15, buf=0x2f3d54016048, fd=51)
 
    at ../sysdeps/unix/sysv/linux/pread64.c:29
 
#9  __libc_pread64 (fd=fd@entry=51, buf=buf@entry=0x2f3d54016048, count=count@entry=15, 
    offset=offset@entry=1634) at ../sysdeps/unix/sysv/linux/pread64.c:27
 
#10 0x0000563110d5b2d9 in pread64 (__offset=1634, __nbytes=15, __buf=0x2f3d54016048, __fd=51)
 
    at /usr/include/x86_64-linux-gnu/bits/unistd.h:99
 
#11 my_pread (Filedes=Filedes@entry=51, Buffer=Buffer@entry=0x2f3d54016048 "\001\006", 
    Count=Count@entry=15, offset=offset@entry=1634, MyFlags=MyFlags@entry=532)
 
    at /test/10.6c_dbg/mysys/my_pread.c:66
 
#12 0x0000563110d41c85 in inline_mysql_file_pread (flags=532, offset=1634, count=15, 
    buffer=0x2f3d54016048 "\001\006", file=51, src_line=198, 
    src_file=0x5631111a8648 "/test/10.6c_dbg/mysys/mf_iocache2.c")
 
    at /test/10.6c_dbg/include/mysql/psi/mysql_file.h:1206
 
#13 my_b_pread (info=info@entry=0x2f3d540cedf0, Buffer=Buffer@entry=0x2f3d54016048 "\001\006", 
    Count=15, pos=1634) at /test/10.6c_dbg/mysys/mf_iocache2.c:198
 
#14 0x00005631107fecbe in read_to_buffer_varlen (fromfile=0x2f3d540cedf0, buffpek=0x4c9268001de8, 
    sort_length=16) at /test/10.6c_dbg/storage/maria/ma_sort.c:955
 
#15 0x00005631107fef18 in merge_buffers (info=info@entry=0x2f3d540cec78, 
    keys=keys@entry=1152921504606846975, from_file=from_file@entry=0x2f3d540cedf0, 
    to_file=to_file@entry=0x0, sort_keys=sort_keys@entry=0x2f3d54015a48, 
    lastbuff=lastbuff@entry=0x4c9268001de8, Fb=0x4c9268001de8, Tb=0x4c9268001e18)
 
    at /test/10.6c_dbg/storage/maria/ma_sort.c:1036
 
#16 0x00005631107ff73d in merge_index (info=info@entry=0x2f3d540cec78, 
    keys=keys@entry=1152921504606846975, sort_keys=sort_keys@entry=0x2f3d54015a48, 
    buffpek=0x4c9268001de8, maxbuffer=1, tempfile=tempfile@entry=0x2f3d540cedf0)
 
    at /test/10.6c_dbg/storage/maria/ma_sort.c:1146
 
#17 0x00005631108013ff in _ma_thr_write_keys (sort_param=sort_param@entry=0x2f3d540cec78)
 
    at /test/10.6c_dbg/storage/maria/ma_sort.c:664
 
#18 0x00005631107f67ee in maria_repair_parallel (param=param@entry=0x2f3d540fb270, info=0x2f3d540c8908, 
    name=name@entry=0x56311355db90 "/test/MD230121-10.6-dbg-MDEV-24750/data/#sql-temptable-a9bc8-3-17", 
    rep_quick=1 '\001') at /test/10.6c_dbg/storage/maria/ma_check.c:4551
 
#19 0x000056311077a01c in ha_maria::repair (this=this@entry=0x2f3d540d8f40, thd=thd@entry=
 
    0x2f3d54000db8, param=param@entry=0x2f3d540fb270, do_optimize=do_optimize@entry=false)
 
    at /test/10.6c_dbg/storage/maria/ha_maria.cc:1657
 
#20 0x000056311077ab66 in ha_maria::enable_indexes (this=this@entry=0x2f3d540d8f40, mode=mode@entry=2)
 
    at /test/10.6c_dbg/storage/maria/ha_maria.cc:2024
 
#21 0x000056311077ae47 in ha_maria::end_bulk_insert (this=0x2f3d540d8f40)
 
    at /test/10.6c_dbg/storage/maria/ha_maria.cc:2262
 
#22 0x00005631104f82ba in handler::ha_end_bulk_insert (this=0x2f3d540d8f40)
 
    at /test/10.6c_dbg/sql/handler.cc:4647
 
#23 0x000056311027c194 in create_internal_tmp_table_from_heap (thd=0x2f3d54000db8, 
    table=table@entry=0x2f3d540c4370, start_recinfo=<optimized out>, recinfo=<optimized out>, 
    error=error@entry=135, ignore_last_dupp_key_error=ignore_last_dupp_key_error@entry=true, 
    is_duplicate=0x0) at /test/10.6c_dbg/sql/sql_select.cc:19839
 
#24 0x000056311031d971 in multi_update::send_data (this=0x2f3d54018468, not_used_values=<optimized out>)
 
    at /test/10.6c_dbg/sql/sql_update.cc:2631
 
#25 0x000056311028bf31 in select_result_sink::send_data_with_check (sent=<optimized out>, 
    u=<optimized out>, items=
 
        @0x56311355ed20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x5631117fb0e0 <end_of_list>, last = 0x56311355ed20, elements = 0}, <No data fields>}, this=<optimized out>)
 
    at /test/10.6c_dbg/sql/sql_class.h:5375
 
#26 end_send (join=0x2f3d54018540, join_tab=0x2f3d540d14b8, end_of_records=<optimized out>)
 
    at /test/10.6c_dbg/sql/sql_select.cc:21798
 
#27 0x000056311025b68e in evaluate_join_record (join=join@entry=0x2f3d54018540, 
    join_tab=join_tab@entry=0x2f3d540d1108, error=error@entry=0)
 
    at /test/10.6c_dbg/sql/sql_select.cc:20821
 
#28 0x00005631102712e7 in sub_select (join=0x2f3d54018540, join_tab=0x2f3d540d1108, 
    end_of_records=<optimized out>) at /test/10.6c_dbg/sql/sql_select.cc:20637
 
#29 0x00005631102a9d4e in do_select (procedure=0x0, join=0x2f3d54018540)
 
    at /test/10.6c_dbg/sql/sql_select.cc:20145
 
#30 JOIN::exec_inner (this=this@entry=0x2f3d54018540) at /test/10.6c_dbg/sql/sql_select.cc:4472
 
#31 0x00005631102aa1be in JOIN::exec (this=this@entry=0x2f3d54018540)
 
    at /test/10.6c_dbg/sql/sql_select.cc:4252
 
#32 0x00005631102a841e in mysql_select (thd=thd@entry=0x2f3d54000db8, 
    tables=tables@entry=0x2f3d54016208, fields=
 
        @0x56311355ed20: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x5631117fb0e0 <end_of_list>, last = 0x56311355ed20, elements = 0}, <No data fields>}, conds=conds@entry=0x0, og_num=0, 
    order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2200096997504, 
    result=0x2f3d54018468, unit=0x2f3d54004f80, select_lex=0x2f3d54005780)
 
    at /test/10.6c_dbg/sql/sql_select.cc:4668
 
#33 0x000056311031ecc3 in mysql_multi_update (thd=thd@entry=0x2f3d54000db8, table_list=0x2f3d54016208, 
    fields=fields@entry=0x2f3d540058d0, values=values@entry=0x2f3d54005e40, conds=0x0, options=0, 
    handle_duplicates=DUP_ERROR, ignore=false, unit=0x2f3d54004f80, select_lex=0x2f3d54005780, 
    result=0x56311355ef60) at /test/10.6c_dbg/sql/sql_update.cc:1940
 
#34 0x000056311022a00c in mysql_execute_command (thd=thd@entry=0x2f3d54000db8)
 
    at /test/10.6c_dbg/sql/sql_parse.cc:4363
 
#35 0x0000563110214e6e in mysql_parse (thd=thd@entry=0x2f3d54000db8, rawbuf=<optimized out>, 
    length=<optimized out>, parser_state=parser_state@entry=0x56311355f3d0)
 
    at /test/10.6c_dbg/sql/sql_parse.cc:7881
 
#36 0x0000563110222f0d in dispatch_command (command=command@entry=COM_QUERY, 
    thd=thd@entry=0x2f3d54000db8, 
    packet=packet@entry=0x2f3d54008d39 "UPDATE t1 SET a=( (SELECT MAX(a) FROM t1))", 
    packet_length=packet_length@entry=42) at /test/10.6c_dbg/sql/sql_class.h:1293
 
#37 0x0000563110226236 in do_command (thd=0x2f3d54000db8) at /test/10.6c_dbg/sql/sql_parse.cc:1348
 
#38 0x00005631103805eb in do_handle_one_connection (connect=<optimized out>, 
    connect@entry=0x153ef00c5b78, put_in_cache=put_in_cache@entry=true)
 
    at /test/10.6c_dbg/sql/sql_connect.cc:1410
 
#39 0x0000563110380cef in handle_one_connection (arg=arg@entry=0x153ef00c5b78)
 
    at /test/10.6c_dbg/sql/sql_connect.cc:1312
 
#40 0x0000563110834f27 in pfs_spawn_thread (arg=0x37501662e6c8)
 
    at /test/10.6c_dbg/storage/perfschema/pfs.cc:2201
 
#41 0x0000457e724f6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#42 0x0000563112238293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This stack trace is proof that Aria is overwriting memory that belongs to InnoDB.

I can imagine that various things can be corrupted when a subsystem (Aria) asks a system call to overwrite memory that it does not own.

Comment by Roel Van de Paar [ 2021-02-01 ]

Closed as duplicate of MDEV-24749

Comment by Michael Widenius [ 2021-02-01 ]

Open as I want to ensure that the test case is added to MTR

Comment by Michael Widenius [ 2021-02-01 ]

The test case has nothing to do with real world scenarios related to memory overwrites
The reason things fails is that aria_sort_buffer_size is set to MAX_ULONGLONG -1 and my_malloc cannot handle that but instead returns an allocated buffer of a few bytes, which causes the problems; valgrind found the issue at once.

I have now fixed that so that one cannot allocate aria_sort_buffer to more than half of MAX_ULONGLONG. In addition I changed the code to not allocate much more than the file size, which will reduce memory usage for users that sets the sort buffer too high.

By the way, I was able to shrink the test case to:

SET SESSION aria_repair_threads=128;
SET SESSION aria_sort_buffer_size=CAST(-1 AS UNSIGNED INT);

SET SESSION tmp_table_size=65535;
CREATE TABLE t1 (a VARCHAR(255));
insert into t1 (a) select seq from seq_1_to_1000;
UPDATE t1 SET a=( (SELECT MAX(a) FROM t1));
DROP TABLE t1;

Comment by Michael Widenius [ 2021-02-03 ]

Fix pushed only to 10.5 as this bug can only happen if one sets totally unrealistic values for aria_sort_buffer_size.

The bug was that my_malloc didn't handle things gracefully if one called it with a value bigger than MAX_ULONGLONG-7. In this case it returned a memory block of 8 bytes, but the caller expected it to be bigger.

Fixed by returning "out of memory" errors for "unreasonable big blocks"

Generated at Thu Feb 08 09:32:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.