[MDEV-24560] SIGSEGV in st_join_table::cleanup + server and client hang + cross-mysqld-interaction + double free or corruption (!prev) Created: 2021-01-11  Updated: 2022-04-20  Resolved: 2022-04-07

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.2.44, 10.3.35, 10.4.25, 10.5.16, 10.6.8, 10.7.4, 10.8.3

Type: Bug Priority: Critical
Reporter: Roel Van de Paar Assignee: Oleg Smirnov
Resolution: Fixed Votes: 0
Labels: affects-tests, cross-mysqld-interaction, hang, memory_corruption

Issue Links:
Relates
relates to MDEV-24262 Server crashes in st_join_table::clea... Closed
relates to MDEV-28077 "Wrong create options" with "big_tabl... Closed
relates to MDEV-28354 SIGSEGV's in free_root and st_join_ta... Closed

 Description   

Similar to MDEV-24262, but different testcase

CREATE TABLE t(a VARCHAR(16383) CHARACTER SET UTF32, KEY k(a)) ENGINE=InnoDB;
SET SESSION sql_buffer_result=ON;
SET SESSION big_tables=ON;
SELECT DISTINCT COUNT(DISTINCT a) FROM t;

Leads to:

10.6.0 9118fd360a3da0bba521caf2a35c424968235ac4 (Debug)

Core was generated by `/test/MD010121-mariadb-10.6.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
    at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
[Current thread is 1 (Thread 0x1524b0e9f700 (LWP 2400942))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x0000561bf4e500d7 in my_write_core (sig=sig@entry=11) at /test/10.6_dbg/mysys/stacktrace.c:424
#2  0x0000561bf45e4ab1 in handle_fatal_signal (sig=11) at /test/10.6_dbg/sql/signal_handler.cc:330
#3  <signal handler called>
#4  0x0000561bf43777a9 in st_join_table::cleanup (this=this@entry=0x1524780155b8) at /test/10.6_dbg/sql/sql_select.cc:13444
#5  0x0000561bf4395951 in JOIN::cleanup (this=this@entry=0x152478013f70, full=full@entry=true) at /test/10.6_dbg/sql/sql_select.cc:13882
#6  0x0000561bf4395dfb in JOIN::destroy (this=0x152478013f70) at /test/10.6_dbg/sql/sql_select.cc:4501
#7  0x0000561bf440fd3b in st_select_lex::cleanup (this=this@entry=0x152478012778) at /test/10.6_dbg/sql/sql_union.cc:2746
#8  0x0000561bf43a0b40 in mysql_select (thd=thd@entry=0x152478000db8, tables=0x152478012ef8, fields=@0x1524780128c8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x152478012e98, last = 0x152478012e98, elements = 1}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147879681, result=0x152478013f48, unit=0x152478004f80, select_lex=0x152478012778) at /test/10.6_dbg/sql/sql_select.cc:4687
#9  0x0000561bf43a0cd0 in handle_select (thd=thd@entry=0x152478000db8, lex=lex@entry=0x152478004eb8, result=result@entry=0x152478013f48, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.6_dbg/sql/sql_select.cc:417
#10 0x0000561bf431319d in execute_sqlcom_select (thd=thd@entry=0x152478000db8, all_tables=0x152478012ef8) at /test/10.6_dbg/sql/sql_parse.cc:6116
#11 0x0000561bf431fc7c in mysql_execute_command (thd=thd@entry=0x152478000db8) at /test/10.6_dbg/sql/sql_parse.cc:3820
#12 0x0000561bf430c072 in mysql_parse (thd=thd@entry=0x152478000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x1524b0e9e3d0) at /test/10.6_dbg/sql/sql_parse.cc:7881
#13 0x0000561bf431a1ec in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x152478000db8, packet=packet@entry=0x152478008d39 "SELECT DISTINCT COUNT(DISTINCT a) FROM t", packet_length=packet_length@entry=40) at /test/10.6_dbg/sql/sql_class.h:1293
#14 0x0000561bf431d52d in do_command (thd=0x152478000db8) at /test/10.6_dbg/sql/sql_parse.cc:1348
#15 0x0000561bf44797fc in do_handle_one_connection (connect=<optimized out>, connect@entry=0x561bf7a2e6f8, put_in_cache=put_in_cache@entry=true) at /test/10.6_dbg/sql/sql_connect.cc:1410
#16 0x0000561bf4479f03 in handle_one_connection (arg=arg@entry=0x561bf7a2e6f8) at /test/10.6_dbg/sql/sql_connect.cc:1312
#17 0x0000561bf492f88f in pfs_spawn_thread (arg=0x561bf797a898) at /test/10.6_dbg/storage/perfschema/pfs.cc:2201
#18 0x00001524c622d609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#19 0x00001524c5e1c293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.6.0 9118fd360a3da0bba521caf2a35c424968235ac4 (Optimized)

2021-01-11 15:33:44 0 [Note] /test/MD010121-mariadb-10.6.0-linux-x86_64-opt/bin/mysqld: ready for connections.
Version: '10.6.0-MariaDB'  socket: '/test/MD010121-mariadb-10.6.0-linux-x86_64-opt/socket.sock'  port: 18336  MariaDB Server
double free or corruption (!prev)
210111 15:33:53 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.6.0-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467868 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x14799c000c58
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x147a08f2bd58 thread_stack 0x49000

Bug confirmed present in:
MariaDB: 10.2.37 (dbg), 10.2.37 (opt), 10.3.28 (dbg), 10.3.28 (opt), 10.4.18 (dbg), 10.4.18 (opt), 10.5.9 (dbg), 10.5.9 (opt), 10.6.0 (dbg), 10.6.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.50 (dbg), 5.6.50 (opt), 5.7.32 (dbg), 5.7.32 (opt), 8.0.22 (dbg), 8.0.22 (opt)

Optimized builds 10.2-10.6 will hang for both server and client (i.e. client will hang whilst trying to connect) even though the server is already crashed (with double free or corruption) as per the error log:



 Comments   
Comment by Roel Van de Paar [ 2021-06-28 ]

SET SESSION sql_buffer_result=1;
CREATE TABLE t (c INT) ENGINE=InnoDB;
SELECT GROUP_CONCAT(c ORDER BY 2) FROM t;

Leads to:

10.6.3 367c75c0990c7bd4bc4fb479a622a3acf2ee3769 (Debug)

Core was generated by `/test/MD220621-mariadb-10.6.3-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000558381f9b07f in st_join_table::cleanup (
    this=this@entry=0x155118017308) at /test/10.6_dbg/sql/sql_select.cc:13753
[Current thread is 1 (Thread 0x1551680cb700 (LWP 4115321))]
(gdb) bt
#0  0x0000558381f9b07f in st_join_table::cleanup (this=this@entry=0x155118017308) at /test/10.6_dbg/sql/sql_select.cc:13753
#1  0x0000558381fb88a5 in JOIN::cleanup (this=this@entry=0x155118015c58, full=full@entry=true) at /test/10.6_dbg/sql/sql_select.cc:14194
#2  0x0000558381fb8d1b in JOIN::destroy (this=0x155118015c58) at /test/10.6_dbg/sql/sql_select.cc:4755
#3  0x00005583820345cf in st_select_lex::cleanup (this=this@entry=0x155118014060) at /test/10.6_dbg/sql/sql_union.cc:2760
#4  0x0000558381fc3659 in mysql_select (thd=thd@entry=0x155118000db8, tables=0x155118014bf8, fields=@0x155118014300: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x155118014b90, last = 0x155118014b90, elements = 1}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147879680, result=0x155118015c30, unit=0x155118005130, select_lex=0x155118014060) at /test/10.6_dbg/sql/sql_select.cc:5001
#5  0x0000558381fc37d7 in handle_select (thd=thd@entry=0x155118000db8, lex=lex@entry=0x155118005068, result=result@entry=0x155118015c30, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.6_dbg/sql/sql_select.cc:544
#6  0x0000558381f37548 in execute_sqlcom_select (thd=thd@entry=0x155118000db8, all_tables=0x155118014bf8) at /test/10.6_dbg/sql/sql_parse.cc:6254
#7  0x0000558381f443e2 in mysql_execute_command (thd=thd@entry=0x155118000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.6_dbg/sql/sql_parse.cc:3949
#8  0x0000558381f30977 in mysql_parse (thd=thd@entry=0x155118000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x1551680ca400) at /test/10.6_dbg/sql/sql_parse.cc:8028
#9  0x0000558381f3f4e2 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x155118000db8, packet=packet@entry=0x15511800b769 "SELECT GROUP_CONCAT(c ORDER BY 2) FROM t", packet_length=packet_length@entry=40, blocking=blocking@entry=true) at /test/10.6_dbg/sql/sql_class.h:1340
#10 0x0000558381f428fa in do_command (thd=0x155118000db8, blocking=blocking@entry=true) at /test/10.6_dbg/sql/sql_parse.cc:1406
#11 0x00005583820a6a56 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x558385f9c458, put_in_cache=put_in_cache@entry=true) at /test/10.6_dbg/sql/sql_connect.cc:1410
#12 0x00005583820a705b in handle_one_connection (arg=arg@entry=0x558385f9c458) at /test/10.6_dbg/sql/sql_connect.cc:1312
#13 0x0000558382555c28 in pfs_spawn_thread (arg=0x558385e84af8) at /test/10.6_dbg/storage/perfschema/pfs.cc:2201
#14 0x000015516b399609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#15 0x000015516af87293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.6.3 367c75c0990c7bd4bc4fb479a622a3acf2ee3769 (Optimized)

Core was generated by `/test/MD220621-mariadb-10.6.3-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x14c1ac517700 (LWP 1899521))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x000014c1c23bd859 in __GI_abort () at abort.c:79
#2  0x000014c1c24283ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x14c1c2552285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x000014c1c243047c in malloc_printerr (str=str@entry=0x14c1c2554690 "double free or corruption (!prev)") at malloc.c:5347
#4  0x000014c1c243212c in _int_free (av=0x14c1c2583b80 <main_arena>, p=0x14c1700484b0, have_lock=<optimized out>) at malloc.c:4317
#5  0x0000562ffb0fe0a5 in free_root (root=root@entry=0x14c1ac515880, MyFlags=MyFlags@entry=0) at /test/10.6_opt/mysys/my_alloc.c:410
#6  0x0000562ffa9b5080 in free_tmp_table (thd=0x14c170000c58, entry=0x14c170047170) at /test/10.6_opt/sql/sql_select.cc:20303
#7  0x0000562ffa9ce2bf in JOIN::cleanup (this=this@entry=0x14c170012ba8, full=full@entry=true) at /test/10.6_opt/sql/sql_select.cc:14207
#8  0x0000562ffa9ce5da in JOIN::destroy (this=0x14c170012ba8) at /test/10.6_opt/sql/sql_select.cc:4755
#9  0x0000562ffaa27e55 in st_select_lex::cleanup (this=this@entry=0x14c170010948) at /test/10.6_opt/sql/sql_union.cc:2760
#10 0x0000562ffa9d551a in mysql_select (thd=0x14c170000c58, tables=0x14c170011890, fields=<optimized out>, conds=0x0, og_num=2, order=0x14c170012a40, group=0x14c1700120a8, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x14c170012b80, unit=0x14c170004e10, select_lex=0x14c170010948) at /test/10.6_opt/sql/sql_select.cc:5001
#11 0x0000562ffa9d5c07 in handle_select (thd=thd@entry=0x14c170000c58, lex=lex@entry=0x14c170004d48, result=result@entry=0x14c170012b80, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.6_opt/sql/sql_select.cc:544
#12 0x0000562ffa9665b1 in execute_sqlcom_select (thd=0x14c170000c58, all_tables=0x14c170011890) at /test/10.6_opt/sql/sql_parse.cc:6254
#13 0x0000562ffa974316 in mysql_execute_command (thd=0x14c170000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.6_opt/sql/sql_parse.cc:3949
#14 0x0000562ffa9615b6 in mysql_parse (thd=0x14c170000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.6_opt/sql/sql_parse.cc:8028
#15 0x0000562ffa96d4b5 in dispatch_command (command=COM_QUERY, thd=0x14c170000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.6_opt/sql/sql_class.h:1340
#16 0x0000562ffa96f3d7 in do_command (thd=0x14c170000c58, blocking=blocking@entry=true) at /test/10.6_opt/sql/sql_parse.cc:1406
#17 0x0000562ffaa7d557 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.6_opt/sql/sql_connect.cc:1410
#18 0x0000562ffaa7d8bd in handle_one_connection (arg=arg@entry=0x562ffd3856d8) at /test/10.6_opt/sql/sql_connect.cc:1312
#19 0x0000562ffae07c38 in pfs_spawn_thread (arg=0x562ffd3038c8) at /test/10.6_opt/storage/perfschema/pfs.cc:2201
#20 0x000014c1c28cc609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#21 0x000014c1c24ba293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.2.39 (dbg), 10.2.39 (opt), 10.3.30 (dbg), 10.3.30 (opt), 10.4.20 (dbg), 10.4.20 (opt), 10.5.11 (dbg), 10.5.11 (opt), 10.6.3 (dbg), 10.6.3 (opt)

Bug (or feature/syntax) confirmed not present in:
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.34 (dbg), 5.7.34 (opt), 8.0.24 (dbg), 8.0.24 (opt)

Optimized builds produce a double free or corruption:

10.6.3 367c75c0990c7bd4bc4fb479a622a3acf2ee3769 (Optimized)

2021-06-28 16:16:24 0 [Note] /test/MD220621-mariadb-10.6.3-linux-x86_64-opt/bin/mysqld: ready for connections.
Version: '10.6.3-MariaDB'  socket: '/test/MD220621-mariadb-10.6.3-linux-x86_64-opt/socket.sock'  port: 39600  MariaDB Server
2021-06-28 16:16:24 0 [Note] InnoDB: Buffer pool(s) load completed at 210628 16:16:24
double free or corruption (!prev)

Comment by Roel Van de Paar [ 2021-06-28 ]

The last testcase is almost identical to MDEV-24262.

Hangs only seem to happen when multiple mysqld's are started at once, i.e. cross-mysqld contamination.

Comment by Roel Van de Paar [ 2021-08-20 ]

Another memory stack seen:

# Excute via C based client
CREATE TABLE t (grp INT,c CHAR);
SET sql_buffer_result=1;
SELECT grp,GROUP_CONCAT(c ORDER BY 2) FROM t GROUP BY grp;

Result not guaranteed.

10.7.0 52505bf20de0ce77a5c0b0a74af021051987bb0d (Optimized)

Core was generated by `/test/MD160821-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --max_a'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x1507f0ecc700 (LWP 176128))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00001507f3a9b859 in __GI_abort () at abort.c:79
#2  0x00001507f3b063ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x1507f3c30285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00001507f3b0e47c in malloc_printerr (str=str@entry=0x1507f3c32690 "double free or corruption (!prev)") at malloc.c:5347
#4  0x00001507f3b1012c in _int_free (av=0x1507f3c61b80 <main_arena>, p=0x150788af3390, have_lock=<optimized out>) at malloc.c:4317
#5  0x0000564d21c33545 in root_free (root=0x1507f0eca890, size=<optimized out>, ptr=<optimized out>) at /test/10.7_opt/mysys/my_alloc.c:78
#6  free_root (root=root@entry=0x1507f0eca890, MyFlags=MyFlags@entry=0) at /test/10.7_opt/mysys/my_alloc.c:495
#7  0x0000564d215202e0 in free_tmp_table (thd=0x150788000c58, entry=0x15078828f000) at /test/10.7_opt/sql/sql_select.cc:20368
#8  0x0000564d2153956f in JOIN::cleanup (this=this@entry=0x150789b18778, full=full@entry=true) at /test/10.7_opt/sql/sql_select.cc:14249
#9  0x0000564d215398da in JOIN::destroy (this=0x150789b18778) at /test/10.7_opt/sql/sql_select.cc:4764
#10 0x0000564d2159310d in st_select_lex::cleanup (this=this@entry=0x150789b16918) at /test/10.7_opt/sql/sql_union.cc:2788
#11 0x0000564d2154083a in mysql_select (thd=0x150788000c58, tables=0x150789b17618, fields=<optimized out>, conds=0x0, og_num=1, order=0x0, group=0x150789b17e38, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x150789b18750, unit=0x150788004e00, select_lex=0x150789b16918) at /test/10.7_opt/sql/sql_select.cc:5010
#12 0x0000564d21540f27 in handle_select (thd=thd@entry=0x150788000c58, lex=lex@entry=0x150788004d38, result=result@entry=0x150789b18750, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.7_opt/sql/sql_select.cc:545
#13 0x0000564d214c4321 in execute_sqlcom_select (thd=0x150788000c58, all_tables=0x150789b17618) at /test/10.7_opt/sql/sql_parse.cc:6256
#14 0x0000564d214d21a8 in mysql_execute_command (thd=0x150788000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:3946
#15 0x0000564d214bf336 in mysql_parse (thd=0x150788000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:8030
#16 0x0000564d214cb225 in dispatch_command (command=COM_QUERY, thd=0x150788000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.7_opt/sql/sql_class.h:1357
#17 0x0000564d214cd147 in do_command (thd=0x150788000c58, blocking=blocking@entry=true) at /test/10.7_opt/sql/sql_parse.cc:1404
#18 0x0000564d215e8967 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.7_opt/sql/sql_connect.cc:1418
#19 0x0000564d215e8cad in handle_one_connection (arg=arg@entry=0x564d25231958) at /test/10.7_opt/sql/sql_connect.cc:1312
#20 0x0000564d2193b648 in pfs_spawn_thread (arg=0x564d24dd21d8) at /test/10.7_opt/storage/perfschema/pfs.cc:2201
#21 0x00001507f3faa609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#22 0x00001507f3b98293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

There is also a reworked version of this testcase:

# Must be executed at the command line
SET sql_buffer_result=1;
CREATE TABLE t (c1 INT,c2 INT);
SELECT c1,GROUP_CONCAT(c2 ORDER BY 2) FROM t GROUP BY c1;

Which will hang 10.2 optimized (both the server and client hang) and crash all optimized and debug builds from 10.2 to 10.7 (except the hang on 10.2 optimized)

Comment by Roel Van de Paar [ 2021-11-13 ]

This bug further seriously affects testing as it is not possible to filter out a generic error like "double free or corruption (!prev)" seen in the error log (as then other double free/corruptions may be missed), leading to the need to (needlessly) review test run trials which ran into this. Please fix this soon.

Comment by Roel Van de Paar [ 2022-03-01 ]

CREATE TABLE t (c INT);
SET @@sql_buffer_result=ON;
SELECT GROUP_CONCAT(c ORDER BY 0) FROM t;

10.9.0 4a2a9c02cd6611ef36bbb735c2b483dbc83580d4 (Optimized)

Core was generated by `/test/MD260222-mariadb-10.9.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[Current thread is 1 (Thread 0x148a341c0700 (LWP 2422407))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000148a4963e859 in __GI_abort () at abort.c:79
#2  0x0000148a496a93ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x148a497d3285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x0000148a496b147c in malloc_printerr (str=str@entry=0x148a497d5690 "double free or corruption (!prev)") at malloc.c:5347
#4  0x0000148a496b312c in _int_free (av=0x148a49804b80 <main_arena>, p=0x1489f404b890, have_lock=<optimized out>) at malloc.c:4317
#5  0x000055fff7b081c5 in root_free (root=0x148a341be8a0, size=<optimized out>, ptr=<optimized out>) at /test/10.9_opt/mysys/my_alloc.c:78
#6  free_root (root=root@entry=0x148a341be8a0, MyFlags=MyFlags@entry=0) at /test/10.9_opt/mysys/my_alloc.c:495
#7  0x000055fff73bb1e9 in free_tmp_table (thd=0x1489f4000c58, entry=0x1489f404a7b0) at /test/10.9_opt/sql/sql_select.cc:20406
#8  0x000055fff73d493f in JOIN::cleanup (this=this@entry=0x1489f4012598, full=full@entry=true) at /test/10.9_opt/sql/sql_select.cc:14280
#9  0x000055fff73d4caa in JOIN::destroy (this=0x1489f4012598) at /test/10.9_opt/sql/sql_select.cc:4778
#10 0x000055fff742fccd in st_select_lex::cleanup (this=this@entry=0x1489f4010970) at /test/10.9_opt/sql/sql_union.cc:2788
#11 0x000055fff73dbf6a in mysql_select (thd=0x1489f4000c58, tables=0x1489f4011538, fields=@0x1489f4010c10: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x1489f40114a0, last = 0x1489f40114a0, elements = 1}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x1489f4012570, unit=0x1489f4004ea8, select_lex=0x1489f4010970) at /test/10.9_opt/sql/sql_select.cc:5026
#12 0x000055fff73dc647 in handle_select (thd=thd@entry=0x1489f4000c58, lex=lex@entry=0x1489f4004dd0, result=result@entry=0x1489f4012570, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.9_opt/sql/sql_select.cc:543
#13 0x000055fff735d0a1 in execute_sqlcom_select (thd=0x1489f4000c58, all_tables=0x1489f4011538) at /test/10.9_opt/sql/sql_parse.cc:6252
#14 0x000055fff736b3ce in mysql_execute_command (thd=0x1489f4000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.9_opt/sql/sql_parse.cc:3943
#15 0x000055fff7357e26 in mysql_parse (thd=0x1489f4000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.9_opt/sql/sql_parse.cc:8027
#16 0x000055fff7363fd5 in dispatch_command (command=COM_QUERY, thd=0x1489f4000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.9_opt/sql/sql_class.h:1362
#17 0x000055fff73661c7 in do_command (thd=0x1489f4000c58, blocking=blocking@entry=true) at /test/10.9_opt/sql/sql_parse.cc:1402
#18 0x000055fff7485b47 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.9_opt/sql/sql_connect.cc:1418
#19 0x000055fff7485e8d in handle_one_connection (arg=arg@entry=0x55fff9665528) at /test/10.9_opt/sql/sql_connect.cc:1312
#20 0x000055fff77ff631 in pfs_spawn_thread (arg=0x55fff9665598) at /test/10.9_opt/storage/perfschema/pfs.cc:2201
#21 0x0000148a49b4d609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#22 0x0000148a4973b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Oleg Smirnov [ 2022-03-14 ]

Fixing this bug has revealed another problem:

CREATE TABLE t(a VARCHAR(16383) CHARACTER SET UTF32, KEY k(a)) ENGINE=InnoDB;
SET SESSION sql_buffer_result=ON;
SET SESSION big_tables=ON;
SELECT DISTINCT COUNT(DISTINCT a) FROM t;
>>ERROR 1030 (HY000): Got error 140 "Wrong create options" from storage engine Aria

Is it worth filing another Jira ticket?

Comment by Roel Van de Paar [ 2022-03-15 ]

oleg.smirnov Definitely! I wanted to create it for you, but the testcase still results in the crash on my latest 10.9 build. Please go ahead and create it. (Now MDEV-28077) Thanks

Comment by Sergei Petrunia [ 2022-03-25 ]

Review input: https://lists.launchpad.net/maria-developers/msg13115.html

Comment by Oleg Smirnov [ 2022-04-02 ]

Review comments are fixed, waiting for a new review cycle.

Comment by Sergei Petrunia [ 2022-04-04 ]

oleg.smirnov, the new change is ok to push. Please be sure to:

  • commit the first and the second patch as one.
  • Apply them to the right version (10.2)
Comment by Oleg Smirnov [ 2022-04-05 ]

Pushed to preview-10.2-MDEV-24560MDEV-24262MDEV-28077. When merging this branch into later versions of MariaDB Server some tests will start to fail: select.test, select_jcl6.test, select_pkeycache.test will start to produce such warnings on SET SESSION big_tables=ON:
Warnings:
Warning 1287 '@@big_tables' is deprecated and will be removed in a future release

Comment by Oleg Smirnov [ 2022-04-07 ]

Pushed to 10.2:

commit 53b580a91c12e9272623fc45496631be65313dd8
Author: Oleg Smirnov <olernov@gmail.com>
Date: Thu Mar 24 14:57:23 2022 +0700

MDEV-28077 'Wrong create options' error with 'big_tables' enabled

The cause of the bug is overflow of uint16 KEY_PART_INFO::length and/or
uint16 KEY_PART_INFO::store_length. The solution is to increase the size
of those variables to the 'uint' type (which is 32-bit long)

commit 85192553ae2c3cb5fb26ace4cd85377525ac7845
Author: Oleg Smirnov <olernov@gmail.com>
Date: Fri Mar 11 21:18:34 2022 +0700

MDEV-24560 SIGSEGV in st_join_table::cleanup

If JOIN::create_postjoin_aggr_table encounters errors during execution
then free_tmp_table() is then called twice for JOIN_TAB::aggr.
The solution is to initialize JOIN_TAB::aggr only on successful completion
of JOIN::create_postjoin_aggr_table

Comment by Roel Van de Paar [ 2022-04-08 ]

oleg.smirnov In one of the test runs which was still reducing, I found the following testcase (CLI):

SET big_tables=ON;
CREATE TABLE t (a VARCHAR(16383) CHARACTER SET UTF32,KEY k1 (a (768))) ENGINE=InnoDB;
SET SESSION sql_buffer_result=1;
DELETE FROM mysql.user WHERE USER=0;
EXPLAIN SELECT COUNT(DISTINCT a) FROM t;

Which produces the following uniqueID's:

SIGSEGV|_int_free|free_root|free_tmp_table|JOIN::cleanup
SIGSEGV|_int_free|root_free|free_root|free_tmp_table
SIGSEGV|st_join_table::cleanup|JOIN::cleanup|JOIN::destroy|st_select_lex::cleanup

With the first two of those being new ones.
Here are two example traces for those uniqueID's:

10.5.16 73fee39ea62037780c59161507e89dd76c10b7a3 (Optimized)

Core was generated by `/test/MD160322-mariadb-10.5.16-linux-x86_64-opt/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  _int_free (av=0x14aa57165b80 <main_arena>, p=0x14a9cc0b3650, 
    have_lock=<optimized out>) at malloc.c:4316
[Current thread is 1 (Thread 0x14aa543fa700 (LWP 2286515))]
(gdb) bt
#0  _int_free (av=0x14aa57165b80 <main_arena>, p=0x14a9cc0b3650, have_lock=<optimized out>) at malloc.c:4316
#1  0x000056297d628b45 in free_root (root=root@entry=0x14aa543f8d60, MyFlags=MyFlags@entry=0) at /test/10.5_opt/mysys/my_alloc.c:410
#2  0x000056297ce84b60 in free_tmp_table (thd=0x14a9cc000c58, entry=0x14a9cc0435e0) at /test/10.5_opt/sql/sql_select.cc:20211
#3  0x000056297ce9ea1f in JOIN::cleanup (this=this@entry=0x14a9cc012410, full=full@entry=true) at /test/10.5_opt/sql/sql_select.cc:14065
#4  0x000056297ce9ed3a in JOIN::destroy (this=0x14a9cc012410) at /test/10.5_opt/sql/sql_select.cc:4567
#5  0x000056297cef7abd in st_select_lex::cleanup (this=this@entry=0x14a9cc0104a8) at /test/10.5_opt/sql/sql_union.cc:2790
#6  0x000056297cef7cf0 in st_select_lex_unit::cleanup (this=0x14a9cc004c40) at /test/10.5_opt/sql/sql_union.cc:2596
#7  st_select_lex_unit::cleanup (this=this@entry=0x14a9cc004c40) at /test/10.5_opt/sql/sql_union.cc:2557
#8  0x000056297ce3e57c in mysql_execute_command (thd=0x14a9cc000c58) at /test/10.5_opt/sql/sql_parse.cc:6085
#9  0x000056297ce2ddb3 in mysql_parse (thd=0x14a9cc000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_parse.cc:8100
#10 0x000056297ce3abcd in dispatch_command (command=COM_QUERY, thd=0x14a9cc000c58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_class.h:1290
#11 0x000056297ce3d3a2 in do_command (thd=0x14a9cc000c58) at /test/10.5_opt/sql/sql_parse.cc:1370
#12 0x000056297cf44f31 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x562980469788, put_in_cache=put_in_cache@entry=true) at /test/10.5_opt/sql/sql_connect.cc:1418
#13 0x000056297cf453ad in handle_one_connection (arg=arg@entry=0x562980469788) at /test/10.5_opt/sql/sql_connect.cc:1312
#14 0x000056297d2da4f2 in pfs_spawn_thread (arg=0x5629803ea438) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
#15 0x000014aa574ac609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#16 0x000014aa57098163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.9.0 5be92887c2caacb45af87b1131db952ce627e83a (Optimized)

Core was generated by `/test/MD160322-mariadb-10.9.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  _int_free (av=0x150515628b80 <main_arena>, p=0x1504600b1f80, 
    have_lock=<optimized out>) at malloc.c:4316
[Current thread is 1 (Thread 0x1504f41a6700 (LWP 2454660))]
(gdb) bt
#0  _int_free (av=0x150515628b80 <main_arena>, p=0x1504600b1f80, have_lock=<optimized out>) at malloc.c:4316
#1  0x000055f19aad94f5 in root_free (root=0x1504f41a4de0, size=<optimized out>, ptr=<optimized out>) at /test/10.9_opt/mysys/my_alloc.c:78
#2  free_root (root=root@entry=0x1504f41a4de0, MyFlags=MyFlags@entry=0) at /test/10.9_opt/mysys/my_alloc.c:495
#3  0x000055f19a39f589 in free_tmp_table (thd=0x150460000c58, entry=0x15046003f560) at /test/10.9_opt/sql/sql_select.cc:20406
#4  0x000055f19a3b8cdf in JOIN::cleanup (this=this@entry=0x150460012708, full=full@entry=true) at /test/10.9_opt/sql/sql_select.cc:14280
#5  0x000055f19a3b904a in JOIN::destroy (this=0x150460012708) at /test/10.9_opt/sql/sql_select.cc:4778
#6  0x000055f19a41401d in st_select_lex::cleanup (this=this@entry=0x150460010968) at /test/10.9_opt/sql/sql_union.cc:2788
#7  0x000055f19a414258 in st_select_lex_unit::cleanup (this=0x150460004ea8) at /test/10.9_opt/sql/sql_union.cc:2594
#8  st_select_lex_unit::cleanup (this=this@entry=0x150460004ea8) at /test/10.9_opt/sql/sql_union.cc:2555
#9  0x000055f19a34b564 in mysql_execute_command (thd=0x150460000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.9_opt/sql/sql_parse.cc:6017
#10 0x000055f19a33c1c6 in mysql_parse (thd=0x150460000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.9_opt/sql/sql_parse.cc:8027
#11 0x000055f19a348375 in dispatch_command (command=COM_QUERY, thd=0x150460000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.9_opt/sql/sql_class.h:1362
#12 0x000055f19a34a567 in do_command (thd=0x150460000c58, blocking=blocking@entry=true) at /test/10.9_opt/sql/sql_parse.cc:1402
#13 0x000055f19a469e97 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.9_opt/sql/sql_connect.cc:1418
#14 0x000055f19a46a1dd in handle_one_connection (arg=arg@entry=0x55f19c89fc38) at /test/10.9_opt/sql/sql_connect.cc:1312
#15 0x000055f19a7e38d1 in pfs_spawn_thread (arg=0x55f19c8579c8) at /test/10.9_opt/storage/perfschema/pfs.cc:2201
#16 0x000015051596f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#17 0x000015051555b163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Additionally, 10.2.44 (opt) hangs when running the testcase, and any CLI connect will hang also. The issue is readily reproducible. GDB break-in coredump can be provided if needed, but I assume it's easy to reproduce on your site for the same (if the patch does not fix it)

Assuming you have a build ready, could you please check that this testcase is also resolved with your patch? If not, I can create a new bug.

Comment by Oleg Smirnov [ 2022-04-11 ]

Confirmed: trunk 10.2 (which includes the patch) crashes on your test case. Please file a new bug.

Comment by Roel Van de Paar [ 2022-04-20 ]

Thank you. Filed MDEV-28354 SIGSEGV's in free_root and st_join_table::cleanup

Generated at Thu Feb 08 09:30:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.