[MDEV-15065] Item_subselect::cleanup double free or corruption Created: 2018-01-25  Updated: 2018-07-20  Resolved: 2018-07-20

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.2.12, 10.2
Fix Version/s: 10.2.15, 10.3.7

Type: Bug Priority: Major
Reporter: Nickolay Ihalainen Assignee: Unassigned
Resolution: Fixed Votes: 1
Labels: crash
Environment:

docker mariadb:latest


Attachments: File error.sql     File mdev15065.test    
Issue Links:
Relates
relates to MDEV-15755 Query crashing MariaDB in cleanup_aft... Closed
relates to MDEV-15928 MariaDB SELECT query ends with lost c... Closed

 Description   

docker run -it --name ma10 -e MYSQL_ALLOW_EMPTY_PASSWORD=1 mariadb:latest
docker cp error.sql ma10:/root/
run commands inside docker exec -it ma10 bash

create database test;
CREATE TABLE `t` (
  `c1` bigint(20) NOT NULL AUTO_INCREMENT,
  `c2` bigint(20) DEFAULT NULL,
  `c3` varchar(256) NOT NULL,
  `c4` bigint(20) DEFAULT NULL,
  `c5` datetime DEFAULT NULL,
  PRIMARY KEY (`c1`)
) ENGINE=InnoDB
 
insert t values(1,1,'aaa',NULL,NOW()),(2,1,'aaa',NULL,NOW()),(3,1,'aaa',NULL,NOW()),(4,1,'aaa',NULL,NOW()),(5,1,'aaa',NULL,NOW()),(6,1,'aaa',NULL,NOW()),(7,1,'aaa',NULL,NOW()),(8,1,'aaa',NULL,NOW()),(9,1,'aaa',NULL,NOW()),(10,1,'aaa',NULL,NOW());
insert into t (c2,c3,c5) select t1.c2,t1.c3,t1.c5 from t t1, t t2, t t3, t t4, t t5;
update t set c2=c1, c4=c1;

mysql test < /root/error.sql
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query

On server error log (in docker run console):

Version: '10.2.12-MariaDB'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
*** Error in `bin/mysqld': double free or corruption (!prev): 0x00007f28a19e0aa0 ***
180125 11:15:43 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.12-MariaDB
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=1
max_threads=102
thread_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759915 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f28a00009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f2920164ec8 thread_stack 0x49000
bin/mysqld(my_print_stacktrace+0x2e)[0xde610e]
bin/mysqld(handle_fatal_signal+0x471)[0x7de601]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f2928852890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f292767c067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f292767d448]
/lib/x86_64-linux-gnu/libc.so.6(+0x731b4)[0x7f29276ba1b4]
/lib/x86_64-linux-gnu/libc.so.6(+0x7898e)[0x7f29276bf98e]
/lib/x86_64-linux-gnu/libc.so.6(+0x79696)[0x7f29276c0696]
bin/mysqld(_ZN14Item_subselect7cleanupEv+0x7c)[0x8c09bc]
bin/mysqld(_ZN3THD19cleanup_after_queryEv+0x140)[0x5ae170]
bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x21d)[0x5f40ed]
bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x26f0)[0x5f7190]
bin/mysqld(_Z10do_commandP3THD+0x15d)[0x5f84cd]
bin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x235)[0x6dc865]
bin/mysqld(handle_one_connection+0x3f)[0x6dca2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f292884b064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f292772f62d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f28a00aa0e0): is an invalid pointer
Connection ID (thread ID): 8
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
 
We think the query pointer is invalid, but we will try to print it anyway. 
Query: SELECT
    *
FROM
    (



 Comments   
Comment by Nickolay Ihalainen [ 2018-01-25 ]

Decoded stack trace from gdb:

#0  0x00007ffff69f8067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff69f9448 in __GI_abort () at abort.c:89
#2  0x00007ffff6a361b4 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff6b2b210 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff6a3b98e in malloc_printerr (action=1, str=0x7ffff6b2b360 "double free or corruption (out)", ptr=<optimized out>) at malloc.c:4996
#4  0x00007ffff6a3c696 in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5  0x00000000008c09bc in Item_subselect::cleanup (this=0x7fff74a02ce0) at /home/buildbot/buildbot/build/sql/item_subselect.cc:158
#6  0x00000000005ae170 in delete_self (this=<optimized out>) at /home/buildbot/buildbot/build/sql/item.h:1814
#7  free_items (this=0x7fff740009c0) at /home/buildbot/buildbot/build/sql/sql_class.cc:3548
#8  THD::cleanup_after_query (this=0x7fff740009a8) at /home/buildbot/buildbot/build/sql/sql_class.cc:2171
#9  0x00000000005f40ed in mysql_parse (thd=0x7fff740009a8, rawbuf=0x7fff740f5ae0 "SELECT\n    *\nFROM\n    (\n        SELECT\n", ' ' <repeats 12 times>, "contact.contact_id,\n", ' ' <repeats 12 times>, "contact.contact_type,\n", ' ' <repeats 12 times>, "COALESCE(common_map.folder_id, personal_map.folder_id) folder_id,\n", ' ' <repeats 12 times>, "conta"..., length=603166, parser_state=0x7ffff00de780, is_com_multi=<optimized out>, is_next_command=false) at /home/buildbot/buildbot/build/sql/sql_parse.cc:7921
#10 0x00000000005f7190 in dispatch_command (command=COM_QUERY, thd=0x7fff740009a8, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1805
#11 0x00000000005f84cd in do_command (thd=0x7fff740009a8) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1360
#12 0x00000000006dc865 in do_handle_one_connection (connect=0x1) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1335
#13 0x00000000006dca2f in handle_one_connection (arg=0x2de3618) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1241
#14 0x00007ffff7bc7064 in start_thread (arg=0x7ffff00df700) at pthread_create.c:309
#15 0x00007ffff6aab62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Comment by Alice Sherepa [ 2018-01-26 ]

Thanks, Nickolay!
Reproducible on MariaDB 10.2,10.3
little bit simplified testcase here: mdev15065.test

set @a='50004,50005,50006,50007,50008,50009.......'; # very long, uploaded in file mdev15065.test
 
CREATE TABLE `t` (
  `c1` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
  `c2` int,
  `c3` varchar(10) NOT NULL,
  `c4` int,
  `c5` datetime);
 
insert t values(1,1,'aaa',NULL,NOW()),(2,1,'aaa',NULL,NOW()),(3,1,'aaa',NULL,NOW()),(4,1,'aaa',NULL,NOW()),(5,1,'aaa',NULL,NOW()),(6,1,'aaa',NULL,NOW()),(7,1,'aaa',NULL,NOW()),(8,1,'aaa',NULL,NOW()),(9,1,'aaa',NULL,NOW()),(10,1,'aaa',NULL,NOW());
insert into t (c2,c3,c5) select t1.c2,t1.c3,t1.c5 from t t1, t t2, t t3, t t4, t t5;
update t set c2=c1, c4=c1;
  
SELECT  * FROM
  (SELECT t.c1,t.c5 
  FROM t LEFT JOIN t t1 ON t.c4 = t1.c2
  WHERE(t.c1 IN (
        SELECT CAST(SUBSTRING_INDEX(GROUP_CONCAT(c1 ORDER BY c5, c1), ',', 1) AS SIGNED)
        FROM t
        WHERE (FIND_IN_SET(c1,@a) AND c3 <> '') GROUP BY c3))
  UNION 
        SELECT t.c1,t.c5
        FROM t LEFT JOIN t t1 ON t.c4 = t1.c2
        WHERE (FIND_IN_SET(t.c1,@a) AND t.c3 = '')) a;
 
drop table t;

#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x000055e54ab228e1 in my_write_core (sig=6) at /home/alice/git/10.2/mysys/stacktrace.c:477
#2  0x000055e54a3b2e5f in handle_fatal_signal (sig=6) at /home/alice/git/10.2/sql/signal_handler.cc:305
#3  <signal handler called>
#4  0x00007f199cdb2428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#5  0x00007f199cdb402a in __GI_abort () at abort.c:89
#6  0x00007f199cdf47ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f199cf0ded8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#7  0x00007f199cdfd37a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f199cf0dfe8 "double free or corruption (out)", action=3) at malloc.c:5006
#8  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3867
#9  0x00007f199ce0153c in __GI___libc_free (mem=<optimized out>) at malloc.c:2968
#10 0x000055e54ab1c6b4 in my_free (ptr=0x7f197c05fab8) at /home/alice/git/10.2/mysys/my_malloc.c:217
#11 0x000055e54a4768f1 in Item_subselect::cleanup (this=0x7f197c147ac0) at /home/alice/git/10.2/sql/item_subselect.cc:158
#12 0x000055e54a476ac9 in Item_in_subselect::cleanup (this=0x7f197c147ac0) at /home/alice/git/10.2/sql/item_subselect.cc:195
#13 0x000055e54a10798d in Item::delete_self (this=0x7f197c147ac0) at /home/alice/git/10.2/sql/item.h:1814
#14 0x000055e54a0fea82 in Query_arena::free_items (this=0x7f197c000ab0) at /home/alice/git/10.2/sql/sql_class.cc:3552
#15 0x000055e54a0f9f2b in THD::cleanup_after_query (this=0x7f197c000a98) at /home/alice/git/10.2/sql/sql_class.cc:2171
#16 0x000055e54a150208 in mysql_parse (thd=0x7f197c000a98, rawbuf=0x7f197c181bb0 "SELECT  * FROM\n(SELECT t.c1,t.c5 \nFROM t LEFT JOIN t t1 ON t.c4 = t1.c2\nWHERE(t.c1 IN (\nSELECT CAST(SUBSTRING_INDEX(GROUP_CONCAT(c1 ORDER BY c5, c1), ',', 1) AS SIGNED)\nFROM t\nWHERE (FIND_IN_SET(c1,@a"..., length=337, parser_state=0x7f19930581f0, is_com_multi=false, is_next_command=false) at /home/alice/git/10.2/sql/sql_parse.cc:7919
#17 0x000055e54a13dd6f in dispatch_command (command=COM_QUERY, thd=0x7f197c000a98, packet=0x7f197c02cd19 "", packet_length=337, is_com_multi=false, is_next_command=false) at /home/alice/git/10.2/sql/sql_parse.cc:1806
#18 0x000055e54a13c6cb in do_command (thd=0x7f197c000a98) at /home/alice/git/10.2/sql/sql_parse.cc:1360
#19 0x000055e54a28a809 in do_handle_one_connection (connect=0x55e54de91d48) at /home/alice/git/10.2/sql/sql_connect.cc:1335
#20 0x000055e54a28a589 in handle_one_connection (arg=0x55e54de91d48) at /home/alice/git/10.2/sql/sql_connect.cc:1241
#21 0x000055e54a5e7ec8 in pfs_spawn_thread (arg=0x55e54ddf1908) at /home/alice/git/10.2/storage/perfschema/pfs.cc:1862
#22 0x00007f199d9ef6ba in start_thread (arg=0x7f1993059700) at pthread_create.c:333
#23 0x00007f199ce8441d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

mdev15065.test

Comment by Elena Stepanova [ 2018-07-19 ]

Test case created by alice stopped causing the failure after the bugfix for MDEV-15755 which was released in 10.2.15:

commit 445ac662c369756c7465d687e40225ffcce28e74
Author: Sergei Golubchik <serg@mariadb.org>
Date:   Mon May 14 16:44:03 2018 +0200
 
    MDEV-15755 Query crashing MariaDB in cleanup_after_query
    
    set the pointer to NULL to avoid double-free
    when the item is cleaned up many times
    
    (once in JOIN_TAB::cleanup(): tmp->jtbm_subselect->cleanup()
    and once at the end of the query, with all other items)

ihanick, did you have a chance to upgrade to 10.2.15+? Can you confirm that the original problem is also gone?

Comment by Nickolay Ihalainen [ 2018-07-20 ]

the issue resolved in 10.3.8 and 10.2.16

Comment by Elena Stepanova [ 2018-07-20 ]

Thanks for the update.

Closing as fixed in scope of MDEV-15755.

Generated at Thu Feb 08 08:18:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.