[MDEV-4780] Crash while running SET GLOBAL key_cache_segments = 64 Created: 2013-07-11  Updated: 2015-06-02  Resolved: 2015-06-02

Status: Closed
Project: MariaDB Server
Component/s: Admin statements
Affects Version/s: 10.0.3, 5.5.31, 5.3.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Peter (Stig) Edwards Assignee: Unassigned
Resolution: Won't Fix Votes: 1
Labels: None


 Description   

Hello and thank you for mariadb-5.3.12-Linux-x86_64.tar.gz

I added a comment in https://mariadb.atlassian.net/browse/MDEV-4409 mentioning two crashes I had today while running "SET GLOBAL key_cache_segments = 64;" on mariadb-5.3.12-Linux-x86_64.tar.gz on RHEL5. Vladislav Vaintroub pointed me at https://bugs.launchpad.net/maria/+bug/1008293 and I found the following changes

http://bazaar.launchpad.net/~maria-captains/maria/5.2/revision/3159
http://bazaar.launchpad.net/~maria-captains/maria/5.2/revision/3115

I have a RQG reproducer

# cat 3.yy
query_init:
  SET GLOBAL key_buffer_size = 1024*1024;
thread1:
  SET GLOBAL key_cache_segments = _digit;
 
query:
  REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW());
 
# End of RQG grammar
 
# Run as
perl ./runall.pl \
--grammar=3.yy \
--duration=10 \
--queries=1M \
--threads=2 \
--basedir=<your basedir> \
--vardir=<your vardir>

Produces:

130711 20:08:18 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 5.3.12-MariaDB-log
key_buffer_size=1048575
read_buffer_size=131072
max_used_connections=3
max_threads=152
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61369 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x27d5c10
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f7c0b794e68 thread_stack 0x48000
/bin/mysqld(my_print_stacktrace+0x2e) [0x998a1e]
/bin/mysqld(handle_fatal_signal+0x3f9) [0x71ede9]
/lib64/libpthread.so.0() [0x3c1c60f500]
/bin/mysqld() [0x99ecc7]
/bin/mysqld(simple_key_cache_read+0x1bc) [0x99f3fc]
/bin/mysqld() [0x99f609]
/bin/mysqld(_mi_fetch_keypage+0x48) [0x7e1ff8]
/bin/mysqld() [0x7c64a6]
/bin/mysqld() [0x7c666b]
/bin/mysqld(_mi_ck_real_write_btree+0x60) [0x7c68e0]
/bin/mysqld(_mi_ck_write_btree+0x72) [0x7c6992]
/bin/mysqld(mi_write+0x320) [0x7c6ee0]
/bin/mysqld(handler::ha_write_row(unsigned char*)+0x37) [0x7137f7]
/bin/mysqld(write_record(THD*, st_table*, st_copy_info*)+0x34e) [0x691c4e]
/bin/mysqld(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)+0xa31) [0x696891]
/bin/mysqld(mysql_execute_command(THD*)+0xad8) [0x6048c8]
/bin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x299) [0x60a629]
/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xdf9) [0x60b879]
/bin/mysqld(do_command(THD*)+0x101) [0x60c0b1]
/bin/mysqld(handle_one_connection+0xfd) [0x5fd7bd]
/lib64/libpthread.so.0() [0x3c1c607851]
/lib64/libc.so.6(clone+0x6d) [0x3c1c2e767d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f7bc0004c48): REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW())
Connection ID (thread ID): 7
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on

With --debug

*** glibc detected *** /bin/mysqld: free(): corrupted unsorted chunks: 0x00007f43e800e450 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3c1c275366]
/lib64/libc.so.6[0x3c1c277e93]
/bin/mysqld[0x99ef65]
/bin/mysqld[0x99d300]
/bin/mysqld[0x99d5a7]
/bin/mysqld(repartition_key_cache+0x14)[0x99d654]
/bin/mysqld(_Z24ha_repartition_key_cacheP12st_key_cache+0x6a)[0x70feba]
/bin/mysqld(_ZN22sys_var_key_cache_long6updateEP3THDP7set_var+0x111)[0x6102f1]
/bin/mysqld(_ZN7set_var6updateEP3THD+0x4e)[0x60dc1e]
/bin/mysqld(_Z17sql_set_variablesP3THDP4ListI12set_var_baseE+0x7d)[0x619ced]
/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1fc6)[0x605db6]
/bin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x299)[0x60a629]
/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xdf9)[0x60b879]
/bin/mysqld(_Z10do_commandP3THD+0x101)[0x60c0b1]
/bin/mysqld(handle_one_connection+0xfd)[0x5fd7bd]
/lib64/libpthread.so.0[0x3c1c607851]
/lib64/libc.so.6(clone+0x6d)[0x3c1c2e767d]

*** glibc detected *** /bin/mysqld: malloc(): memory corruption: 0x00007f43e808cb80 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3c1c275366]
/lib64/libc.so.6[0x3c1c278de4]
/lib64/libc.so.6(__libc_malloc+0x71)[0x3c1c279b91]
/lib64/libc.so.6(__backtrace_symbols+0x119)[0x3c1c2fd899]
/bin/mysqld(my_print_stacktrace+0x50)[0x998a40]
/bin/mysqld(handle_fatal_signal+0x3f9)[0x71ede9]
/lib64/libpthread.so.0[0x3c1c60f500]
/lib64/libc.so.6(gsignal+0x35)[0x3c1c2328a5]
/lib64/libc.so.6(abort+0x175)[0x3c1c234085]
/lib64/libc.so.6[0x3c1c26fa37]
/lib64/libc.so.6[0x3c1c275366]
/lib64/libc.so.6[0x3c1c277e93]
/bin/mysqld[0x99ef65]
/bin/mysqld[0x99d300]
/bin/mysqld[0x99d5a7]
/bin/mysqld(repartition_key_cache+0x14)[0x99d654]
/bin/mysqld(_Z24ha_repartition_key_cacheP12st_key_cache+0x6a)[0x70feba]
/bin/mysqld(_ZN22sys_var_key_cache_long6updateEP3THDP7set_var+0x111)[0x6102f1]
/bin/mysqld(_ZN7set_var6updateEP3THD+0x4e)[0x60dc1e]
/bin/mysqld(_Z17sql_set_variablesP3THDP4ListI12set_var_baseE+0x7d)[0x619ced]
/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1fc6)[0x605db6]
/bin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x299)[0x60a629]
/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xdf9)[0x60b879]
/bin/mysqld(_Z10do_commandP3THD+0x101)[0x60c0b1]
/bin/mysqld(handle_one_connection+0xfd)[0x5fd7bd]
/lib64/libpthread.so.0[0x3c1c607851]
/lib64/libc.so.6(clone+0x6d)[0x3c1c2e767d]



 Comments   
Comment by Vladislav Vaintroub [ 2013-07-11 ]

How much do you need key cache segments to be dynamic variable? We can either continue to patch lke in https://bugs.launchpad.net/maria/+bug/1008293 , or if we're frank to ourselves, and admit that changing such things is hard (and does not bring much benefit) we make it static, fixed at the server startup,

Comment by Igor Babaev [ 2013-07-11 ]

V lad,
Re-partitioning (changing the number of key cache segments) should not be more complicated than key cache re-sizing that we had for years and that was a part of the key cache design introduced by Monty.

Comment by Vladislav Vaintroub [ 2013-07-11 ]

Ok, Igor, don't mind I assign this one to you , since similar bugs were already fixed. I freely admit obviously miss the elegance and simplicity of the solution.

Comment by Peter (Stig) Edwards [ 2013-07-12 ]

This is the first time I have used key_cache_segments, I was pleased when I saw it was listed as a dynamic variable:

https://kb.askmonty.org/en/server-system-variables/#key_cache_segments

because it means it can be changed without having to restart mysqld and this makes it easier to deploy the change.

Because the default value is 0, and I imagine there are users moving from mysqld to MariaDB who may only want to start enabling non compatible features once MariaDB has been stable with the unmodified my.cnf from the previous mysqld, if they don't have to incur a restart for setting key_cache_segments to non zero then it has benefit.

I have on occasion had to dynamically change the key_buffer_size so I think if using key_cache_segments it would be important to keep it and key_buffer_size dynamic.

The reproducer also works against the latest 5.2, building from:

revision-id: wlad@montyprogram.com-20130709202457-6n9syq59jdd2sy3b
date: 2013-07-09 22:24:57 +0200
build-date: 2013-07-11 18:33:31 -0400
revno: 3214
branch-nick: 5.2

The RQG grammar from the description above when run with the same options produces:

130712  1:31:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
key_buffer_size=1048572
read_buffer_size=131072
max_used_connections=3
max_threads=152
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61265 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x3373330
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f3a9821ee58 thread_stack 0x48000
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(my_print_stacktrace+0x2e) [0x96b02e]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(handle_fatal_signal+0x363) [0x6d2023]
/lib64/libpthread.so.0() [0x3c1c60f500]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld() [0x971156]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(simple_key_cache_read+0x1bb) [0x97328b]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld() [0x97357a]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(_mi_fetch_keypage+0x48) [0x775218]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld() [0x782b3a]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(_mi_ck_real_write_btree+0x5d) [0x782f3d]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(_mi_ck_write_btree+0x6f) [0x782fef]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(mi_write+0x303) [0x783523]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(handler::ha_write_row(unsigned char*)+0x37) [0x6c4a07]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(write_record(THD*, st_table*, st_copy_info*)+0x287) [0x650147]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)+0xed1) [0x653ca1]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(mysql_execute_command(THD*)+0x8fa) [0x5dadea]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2c9) [0x5dfe49]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x951) [0x5e07b1]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(do_command(THD*)+0xf2) [0x5e14d2]
mariadb-5.2.15-MariaDB-linux-x86_64/bin/mysqld(handle_one_connection+0x156) [0x5d2616]
/lib64/libpthread.so.0() [0x3c1c607851]
/lib64/libc.so.6(clone+0x6d) [0x3c1c2e767d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f3a4c004c48): REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW())
Connection ID (thread ID): 7
Status: NOT_KILLED

Thank you

Comment by Peter (Stig) Edwards [ 2013-07-12 ]

Running valgrind with 5.2 revno 3214 informs:

==535== Invalid read of size 8
==535==    at 0xAF49B7: safe_mutex_lock (thr_mutex.c:244)
==535==    by 0xAFD000: read_block (mf_keycache.c:2692)
==535==    by 0xAFD4D0: simple_key_cache_read (mf_keycache.c:2880)
==535==    by 0xB00FBE: partitioned_key_cache_read (mf_keycache.c:5451)
==535==    by 0xB01C75: key_cache_read (mf_keycache.c:6233)
==535==    by 0x88F69E: _mi_fetch_keypage (mi_page.c:33)
==535==    by 0x8A0E19: w_search (mi_write.c:362)
==535==    by 0x8A1298: w_search (mi_write.c:427)
==535==    by 0x8A0A7A: _mi_ck_real_write_btree (mi_write.c:305)
==535==    by 0x8A0961: _mi_ck_write_btree (mi_write.c:285)
==535==    by 0x8A0864: _mi_ck_write (mi_write.c:256)
==535==    by 0x8A0137: mi_write (mi_write.c:128)
==535==    by 0x87BC20: ha_myisam::write_row(unsigned char*) (ha_myisam.cc:807)
==535==    by 0x79161A: handler::ha_write_row(unsigned char*) (handler.cc:4965)
==535==    by 0x6EF3DA: write_record(THD*, st_table*, st_copy_info*) (sql_insert.cc:1422)
==535==    by 0x6ED3E5: mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) (sql_insert.cc:866)
==535==  Address 0x1eccd3f0 is 304 bytes inside a block of size 2,516 free'd
==535==    at 0x4A0595D: free (vg_replace_malloc.c:366)
==535==    by 0xAD0877: _myfree (safemalloc.c:337)
==535==    by 0xB00E6F: end_partitioned_key_cache (mf_keycache.c:5373)
==535==    by 0xB01B6B: end_key_cache_internal (mf_keycache.c:6141)
==535==    by 0xB01FD1: repartition_key_cache_internal (mf_keycache.c:6507)
==535==    by 0xB02075: repartition_key_cache (mf_keycache.c:6556)
==535==    by 0x78F6EC: ha_repartition_key_cache(st_key_cache*) (handler.cc:4037)
==535==    by 0x65D679: sys_var_key_cache_long::update(THD*, set_var*) (set_var.cc:2568)
==535==    by 0x65FDE8: set_var::update(THD*) (set_var.cc:3739)
==535==    by 0x65F874: sql_set_variables(THD*, List<set_var_base>*) (set_var.cc:3614)
==535==    by 0x64CF89: mysql_execute_command(THD*) (sql_parse.cc:3615)
==535==    by 0x654CAA: mysql_parse(THD*, char*, unsigned int, char const**) (sql_parse.cc:6216)
==535==    by 0x6469B2: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1248)
==535==    by 0x645C15: do_command(THD*) (sql_parse.cc:920)
==535==    by 0x642A2E: handle_one_connection (sql_connect.cc:1226)
==535==    by 0x3C1C607850: start_thread (pthread_create.c:301)
==535==

A backtrace from a core:

#0  __pthread_kill (threadid=<value optimized out>, signo=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63
        pd = <value optimized out>
        tid = 21950
        val = <value optimized out>
#1  0x0000000000aef17f in my_write_core (sig=6) at stacktrace.c:440
No locals.
#2  0x00000000007a602c in handle_fatal_signal (sig=6) at signal_handler.cc:255
        curr_time = 1373623633
        tm = {tm_sec = 13, tm_min = 7, tm_hour = 13, tm_mday = 12, tm_mon = 6, tm_year = 113, tm_wday = 5, tm_yday = 192, tm_isdst = 0, tm_gmtoff = 10800, tm_zone = 0x2050750 "GMT"}
        thd = 0x32e6718
#3  <signal handler called>
No symbol table info available.
#4  0x0000003c1c2328a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
        resultvar = 0
        pid = <value optimized out>
        selftid = 21950
#5  0x0000003c1c234085 in abort () at abort.c:92
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x7fe384534470, sa_sigaction = 0x7fe384534470}, sa_mask = {__val = {8482578528384, 22, 13689184, 62, 140615154358720, 258169924440, 140615573815296,
              140615154353216, 4294967295, 206158430256, 5, 17849688, 0, 0, 3, 0}}, sa_flags = 467722192, sa_restorer = 0x5}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#6  0x0000000000af4b70 in safe_mutex_lock (mp=0x7fe338161808, my_flags=0, file=0xd0e160 "mf_keycache.c", line=4382) at thr_mutex.c:306
        error = 22
        __PRETTY_FUNCTION__ = "safe_mutex_lock"
#7  0x0000000000affdee in flush_simple_key_cache_blocks (keycache=0x7fe338161758, file=45, file_extra=0x0, type=FLUSH_KEEP) at mf_keycache.c:4382
        res = 0
        _db_stack_frame_ = {func = 0x7fe3845344a0 " ES\204\343\177", file = 0xb06c29 "H\211E\370H\203}\370", level = 0, prev = 0x0}
#8  0x0000000000b013d7 in flush_partitioned_key_cache_blocks (keycache=0x7fe3380636d8, file=45, file_extra=0x7fe338024750, type=FLUSH_KEEP) at mf_keycache.c:5695
        partition = 0x7fe338161758
        i = 0
        partitions = 3
        err = 0
        dirty_part_map = 0x7fe338024750
        _db_stack_frame_ = {func = 0x7fe384534530 "`ES\204", file = 0xaf510d "\213E\354H\203\304@[A\\\311\303UH\211\345ATSH\201\354\340", level = 0, prev = 0x0}
#9  0x0000000000b01e0e in flush_key_blocks (keycache=0x2052d98, file=45, file_extra=0x7fe338024750, type=FLUSH_KEEP) at mf_keycache.c:6392
No locals.
#10 0x0000000000891d70 in mi_lock_database (info=0x7fe338030d78, lock_type=2) at mi_locking.c:76
        error = 0
        count = 0
        share = 0x7fe3380244c8
        _db_stack_frame_ = {func = 0x7fe3845345c0 "0FS\204\343\177", file = 0xb06a62 "H\211E\370H\203}\370", level = 0, prev = 0x0}
#11 0x000000000087f458 in ha_myisam::external_lock (this=0x7fe338022d70, thd=0x32e6718, lock_type=2) at ha_myisam.cc:1932
No locals.
#12 0x000000000079139c in handler::ha_external_lock (this=0x7fe338022d70, thd=0x32e6718, lock_type=2) at handler.cc:4926
        _db_stack_frame_ = {func = 0x7fe3845346a0 "", file = 0x33500b06cc1 <Address 0x33500b06cc1 out of bounds>, level = 0, prev = 0x0}
        dbug_violation_helper = {_entered = true}
        __PRETTY_FUNCTION__ = "int handler::ha_external_lock(THD*, int)"
        error = 0
#13 0x000000000062d8e5 in unlock_external (thd=0x32e6718, table=0x7fe33c006ea0, count=1) at lock.cc:829
        error = 0
        error_code = 0
        _db_stack_frame_ = {func = 0x7fe384534700 "", file = 0x18400b06a62 <Address 0x18400b06a62 out of bounds>, level = 0, prev = 0x0}
        dbug_violation_helper = {_entered = true}
#14 0x000000000062c87b in mysql_unlock_tables (thd=0x32e6718, sql_lock=0x7fe33c006e78) at lock.cc:390
---Type <return> to continue, or q <return> to quit---
        _db_stack_frame_ = {func = 0x7fe300000000 <Address 0x7fe300000000 out of bounds>, file = 0x0, level = 0, prev = 0x0}
        dbug_violation_helper = {_entered = true}
#15 0x000000000069573f in close_thread_tables (thd=0x32e6718) at sql_base.cc:1355
        dbug_violation_helper = {_entered = true}
        __PRETTY_FUNCTION__ = "void close_thread_tables(THD*)"
        table = 0x0
        prelocked_mode = NON_PRELOCKED
        _db_stack_frame_ = {func = 0x7fe300000000 <Address 0x7fe300000000 out of bounds>, file = 0x0, level = 0, prev = 0x0}
#16 0x0000000000647d44 in dispatch_command (command=COM_QUERY, thd=0x32e6718, packet=0x32ea099 "REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW())", packet_length=62) at sql_parse.cc:1693
        net = 0x32e6808
        dbug_violation_helper = {_entered = true}
        __FUNCTION__ = "dispatch_command"
        __PRETTY_FUNCTION__ = "bool dispatch_command(enum_server_command, THD*, char*, uint)"
        error = false
        _db_stack_frame_ = {func = 0x7fe384534ca0 "@MS\204\343\177", file = 0x7fe384534cee "", level = 0, prev = 0x0}
#17 0x0000000000645c16 in do_command (thd=0x32e6718) at sql_parse.cc:920
        return_value = false
        packet = 0x32ea098 "\003REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW())"
        packet_length = 63
        net = 0x32e6808
        command = COM_QUERY
        dbug_violation_helper = {_entered = true}
        __PRETTY_FUNCTION__ = "bool do_command(THD*)"
        _db_stack_frame_ = {func = 0x7fe384534e40 "\220NS\204\343\177", file = 0x642699 "\213\005I_\260", level = 0, prev = 0x0}
#18 0x0000000000642a2f in handle_one_connection (arg=0x32e6718) at sql_connect.cc:1226
        net = 0x32e6808
        create_user = true
        thd = 0x32e6718
#19 0x0000003c1c607851 in start_thread (arg=0x7fe384535700) at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7fe384535700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140615154358016, -1401933795574547793, 18118240, 140615154358720, 0, 3, 1390587551184190127, -1372634280569996625}, mask_was_saved = 0}}, priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#20 0x0000003c1c2e767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Comment by Peter (Stig) Edwards [ 2013-07-12 ]

[Updated this comment as I have seen a crash using when using FTWRL]

I am using this:

FLUSH TABLES WITH READ LOCK ; SET GLOBAL key_cache_segments = 64 ; UNLOCK TABLES;

The RQG grammar below failed to crash or hang 5.3.12. UPDATE - I have seen a crash against 5.3.11 when using FTWRL and a SELECT query was running.

cat 4.yy

query_init:
  SET GLOBAL key_buffer_size = 1024*1024;
thread1:
  FLUSH TABLES WITH READ LOCK ; SET GLOBAL key_cache_segments = _digit ; UNLOCK TABLES;
 
query:
  REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW());

Comment by Peter (Stig) Edwards [ 2013-07-15 ]

Just had a crash on a 5.3.11 instance (mariadb-5.3.11-Linux-x86_64.tar.gz) while using:

  FLUSH TABLES WITH READ LOCK ; SET GLOBAL key_cache_segments = 64 ; UNLOCK TABLES;

The stack trace and query differ, the query is a SELECT, I have included the explain plan and table definitions.

Thank you.

130715  3:20:58 [ERROR] mysqld got signal 8 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 5.3.11-MariaDB-log
key_buffer_size=1258291200
read_buffer_size=2097152
max_used_connections=214
max_threads=801
thread_count=18
connection_count=18
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6980843 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x2aab42ad1840
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x426cf0f8 thread_stack 0x48000
./bin/mysqld(my_print_stacktrace+0x2e) [0x9cb88e]
./bin/mysqld(handle_fatal_signal+0x3f9) [0x751eb9]
/lib64/libpthread.so.0 [0x3d52a0eb70]
./bin/mysqld(simple_key_cache_read+0xfa) [0x9d21aa]
./bin/mysqld(_mi_fetch_keypage+0x48) [0x814e18]
./bin/mysqld(_mi_search+0x7c) [0x80d48c]
./bin/mysqld(mi_rkey+0x16d) [0x7fdbad]
./bin/mysqld(ha_myisam::index_read_idx_map(unsigned char*, unsigned int, unsigned char const*, unsigned long, ha_rkey_function)+0x5d) [0x7e74ad]
./bin/mysqld [0x6a05f2]
./bin/mysqld [0x6a0974]
./bin/mysqld [0x6b6e8b]
./bin/mysqld(JOIN::optimize()+0xc87) [0x6b8b07]
./bin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0xd6) [0x6c0966]
./bin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x16f) [0x6c13ff]
./bin/mysqld [0x635c3e]
./bin/mysqld(mysql_execute_command(THD*)+0x3a43) [0x63b843]
./bin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x299) [0x63e629]
./bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd56) [0x63f7d6]
./bin/mysqld(do_command(THD*)+0x101) [0x640081]
./bin/mysqld(handle_one_connection+0xfd) [0x6317fd]
/lib64/libpthread.so.0 [0x3d52a0673d]
/lib64/libc.so.6(clone+0x6d) [0x3d51ed44bd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x2aabbe867e38): /* comment */ SELECT f, pcds.b FROM mpcds pcds JOIN mcbm fcbm USING(bf_id) WHERE fcbm.f = 123456 LIMIT 5000
Connection ID (thread ID): 46182543
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------+
| id | select_type | table | type  | possible_keys | key     | key_len | ref   | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------+
|  1 | SIMPLE      | fcbm  | const | PRIMARY       | PRIMARY | 4       | const |    1 |       |
|  1 | SIMPLE      | pcds  | const | PRIMARY       | PRIMARY | 10      | const |    1 |       |
+----+-------------+-------+-------+---------------+---------+---------+-------+------+-------+
 
CREATE TABLE `mpcds` (
  `bf_id` varchar(8) NOT NULL,
  `b` smallint(5) unsigned DEFAULT NULL,
  PRIMARY KEY (`bf_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 MAX_ROWS=999999999 PACK_KEYS=1
 
CREATE TABLE `mfcbm` (
  `f` int(10) unsigned NOT NULL,
  `c` char(9) DEFAULT NULL,
  `bf_id` varchar(8) DEFAULT NULL,
  PRIMARY KEY (`f`),
  KEY `compuid` (`c`,`f`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 MAX_ROWS=999999999 PACK_KEYS=1

Comment by Peter (Stig) Edwards [ 2013-07-22 ]

Just wanted to note that if the reproducer is changed to use a pseudo-random values between 0 and 64 for key_cache_segments and 64M key_buffer_size it seems to crash more readily.

query_init:
  SET GLOBAL key_buffer_size = 64*1024*1024;
thread1:
  SET GLOBAL key_cache_segments = { $prng->int(0,64) } ;
 
query:
  REPLACE INTO E(col_int_key, col_datetime_key) VALUES(1, NOW());

Comment by Sergei Golubchik [ 2014-01-26 ]

I couldn't repeat the crash. Neither in 5.5, nor in 5.3, with two of your grammars from above.

Comment by Elena Stepanova [ 2014-02-27 ]

I can easily reproduce it on 5.3, but not on 5.5 or 10.0.

From the history of the bug, it seems it had never been confirmed or seen by anyone on either 5.5 or 10.0; probably those versions crept into the 'Affected' list by some bulk update or something.
(I can't remove the versions from the list, as they appear as "archived", maybe you can).

So, I have set it up for 5.3 on perro in case you want to see it; but my vote is for not touching it in 5.3, at least unless/until somebody observes it on 5.5 or 10.0.

Comment by Elena Stepanova [ 2015-06-02 ]

Apparently it's not reproducible on 5.5 because during the test an attempt to set key_buffer_size causes the error:

MariaDB [(none)]> set global key_buffer_size = 64*1024*1024;
ERROR 1210 (HY000): Incorrect arguments to SET

I assume it's some kind of protection, although a bit ugly.

Current 5.3 still fails (only now for me it hangs, but it's all the same). However, it's certainly not the kind of a bug worth fixing in 5.3 now. So, I'm closing it as "won't fix"

Generated at Thu Feb 08 06:59:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.