As it might not be clear from the initial bug description, this issue could be encoutered on a production environement (its what happened to me).

As long as the key_cache_segment is not set to its default value (0), if you have a running INSERT query (likely to happen in an heavy workload or during a database import) and you do a GLOBAL SET on one of the problematic variables, you will hit the bug and lock/crash the table.

1. RQG grammar to reproduce (mdev4409.yy):

thread1_init:
SET GLOBAL key_cache_segments = 6; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (i INT, KEY) ENGINE=MyISAM;

thread2_init:
SELECT SLEEP(3);

thread1:
INSERT INTO t1 VALUES (1),(2),(3),(4),(5),(6); INSERT INTO t1 SELECT t1a.* FROM t1 t1a, t1 t1b, t1 t1c, t1 t1d, t1 t1e; DELETE FROM t1;

query:
SET GLOBAL key_buffer_size = 1073741824; SET GLOBAL key_buffer_size = 1048576;

1. End of RQG grammar

To reproduce,

• start server;
  -branch lp:randgen;
• cd randgen
• copy the grammar above into mdev4409.yy
• run
  perl ./gentest.pl --grammar=mdev4417.yy --threads=2 --duration=30 --reporters=Deadlock --dsn=dbi:mysql:host=127.0.0.1:port=3306:user=root:database=test

When the test passes, it ends in less than a minute saying, all at once,

1. Child worker process completed successfully.
2. Child worker process completed successfully.
3. Killing periodic reporting process with pid 11034...
4. Kill GenTest::ErrorFilter(11033)
5. Test completed successfully.

When the test hits the bug, it hangs, and in half a minute it says

1. Child worker process completed successfully.
   but only once, then it waits for another half a minute and says
2. Actual test duration (65 seconds) is more than 2 times the desired duration (30 seconds)

(the number might differ), and does not return.
The processlist shows INSERT hanging in 'Sending data' or 'Copying to tmp table' state.

Stack trace from the process (also might differ):

#2 0x0000000000c93e2a in inline_mysql_cond_wait (that=0x7f20f80008c8, mutex=0x7f20f800cfc0, src_file=0xe64520 "/home/elenst/bzr/5.5/mysys/mf_keycache.c", src_line=1138) at /home/elenst/bzr/5.5/include/mysql/psi/mysql_thread.h:984
#3 0x0000000000c951b1 in wait_on_queue (wqueue=0x7f20f800d068, mutex=0x7f20f800cfc0) at /home/elenst/bzr/5.5/mysys/mf_keycache.c:1138
#4 0x0000000000c9857f in simple_key_cache_read (keycache=0x7f20f800cf10, file=35, filepos=3072, level=3, buff=0x7f2149d5e7e0 "", length=1024, block_length=1024, return_buffer=0) at /home/elenst/bzr/5.5/mysys/mf_keycache.c:2800
#5 0x0000000000c9c05c in partitioned_key_cache_read (keycache=0x7f20f812d2d0, file=35, filepos=3072, level=3, buff=0x7f2149d5e7e0 "", length=1024, block_length=1024, return_buffer=0) at /home/elenst/bzr/5.5/mysys/mf_keycache.c:5423
#6 0x0000000000c9cd18 in key_cache_read (keycache=0x21edd70, file=35, filepos=3072, level=3, buff=0x7f2149d5e7e0 "", length=1024, block_length=1024, return_buffer=0) at /home/elenst/bzr/5.5/mysys/mf_keycache.c:6200
#7 0x0000000000bd1957 in _mi_fetch_keypage (info=0x7f20f801d110, keyinfo=0x7f20f800fcf8, page=3072, level=3, buff=0x7f2149d5e7e0 "", return_buffer=0) at /home/elenst/bzr/5.5/storage/myisam/mi_page.c:32
#8 0x0000000000bde507 in w_search (info=0x7f20f801d110, keyinfo=0x7f20f800fcf8, comp_flag=4, key=0x7f2149d60f50 "\001", key_length=5, page=3072, father_buff=0x7f2149d5fb10 "\200*", father_keypos=0x7f2149d5fb18 "\001", father_page=65536, insert_last=0 '\000') at /home/elenst/bzr/5.5/storage/myisam/mi_write.c:353
#9 0x0000000000bdea34 in w_search (info=0x7f20f801d110, keyinfo=0x7f20f800fcf8, comp_flag=4, key=0x7f2149d60f50 "\001", key_length=5, page=65536, father_buff=0x0, father_keypos=0x0, father_page=0, insert_last=0 '\000') at /home/elenst/bzr/5.5/storage/myisam/mi_write.c:417
#10 0x0000000000bde146 in _mi_ck_real_write_btree (info=0x7f20f801d110, keyinfo=0x7f20f800fcf8, key=0x7f2149d60f50 "\001", key_length=5, root=0x7f20f800fe80, comp_flag=4) at /home/elenst/bzr/5.5/storage/myisam/mi_write.c:295
#11 0x0000000000bde045 in _mi_ck_write_btree (info=0x7f20f801d110, keynr=0, key=0x7f2149d60f50 "\001", key_length=5) at /home/elenst/bzr/5.5/storage/myisam/mi_write.c:276
#12 0x0000000000be0b66 in keys_free (key=0x7f20f8049668 "\001", mode=free_free, param=0x7f20f8042880) at /home/elenst/bzr/5.5/storage/myisam/mi_write.c:953
#13 0x0000000000cbf4af in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f8049648) at /home/elenst/bzr/5.5/mysys/tree.c:183
#14 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f8049768) at /home/elenst/bzr/5.5/mysys/tree.c:181
#15 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f80499a8) at /home/elenst/bzr/5.5/mysys/tree.c:181
#16 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f8049e28) at /home/elenst/bzr/5.5/mysys/tree.c:181
#17 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f804a728) at /home/elenst/bzr/5.5/mysys/tree.c:181
#18 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f804b928) at /home/elenst/bzr/5.5/mysys/tree.c:181
#19 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f804dd28) at /home/elenst/bzr/5.5/mysys/tree.c:181
#20 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f8049678) at /home/elenst/bzr/5.5/mysys/tree.c:181
#21 0x0000000000cbf457 in delete_tree_element (tree=0x7f20f80425e0, element=0x7f20f80495b8) at /home/elenst/bzr/5.5/mysys/tree.c:181
#22 0x0000000000cbf355 in free_tree (tree=0x7f20f80425e0, free_flags=0) at /home/elenst/bzr/5.5/mysys/tree.c:151
#23 0x0000000000cbf3fe in delete_tree (tree=0x7f20f80425e0) at /home/elenst/bzr/5.5/mysys/tree.c:167
#24 0x0000000000be10dc in mi_end_bulk_insert (info=0x7f20f801d110) at /home/elenst/bzr/5.5/storage/myisam/mi_write.c:1044
#25 0x0000000000b9ae2a in ha_myisam::end_bulk_insert (this=0x7f20feeb5918) at /home/elenst/bzr/5.5/storage/myisam/ha_myisam.cc:1581
#26 0x00000000005fb508 in handler::ha_end_bulk_insert (this=0x7f20feeb5918) at /home/elenst/bzr/5.5/sql/handler.h:1932
#27 0x00000000005f923a in select_insert::send_eof (this=0x7f20f8008040) at /home/elenst/bzr/5.5/sql/sql_insert.cc:3651
#28 0x000000000066635b in do_select (join=0x7f20f8019ba8, fields=0x7f20f8019f30, table=0x0, procedure=0x0) at /home/elenst/bzr/5.5/sql/sql_select.cc:15995
#29 0x0000000000646256 in JOIN::exec (this=0x7f20f8019ba8) at /home/elenst/bzr/5.5/sql/sql_select.cc:2842
#30 0x0000000000646a53 in mysql_select (thd=0x32c86a0, rref_pointer_array=0x32cb6a8, tables=0x7f20f8006cb0, wild_num=1, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f20f8008040, unit=0x32cad78, select_lex=0x32cb450) at /home/elenst/bzr/5.5/sql/sql_select.cc:3062
#31 0x000000000063d5ca in handle_select (thd=0x32c86a0, lex=0x32cacc8, result=0x7f20f8008040, setup_tables_done_option=1073741824) at /home/elenst/bzr/5.5/sql/sql_select.cc:318
#32 0x0000000000611093 in mysql_execute_command (thd=0x32c86a0) at /home/elenst/bzr/5.5/sql/sql_parse.cc:3021
#33 0x0000000000618cbc in mysql_parse (thd=0x32c86a0, rawbuf=0x7f20f8006448 "INSERT INTO t1 SELECT t1a.* FROM t1 t1a, t1 t1b, t1 t1c, t1 t1d, t1 t1e", length=71, parser_state=0x7f2149d62550) at /home/elenst/bzr/5.5/sql/sql_parse.cc:5759
#34 0x000000000060c3bc in dispatch_command (command=COM_QUERY, thd=0x32c86a0, packet=0x33b7171 " INSERT INTO t1 SELECT t1a.* FROM t1 t1a, t1 t1b, t1 t1c, t1 t1d, t1 t1e", packet_length=72) at /home/elenst/bzr/5.5/sql/sql_parse.cc:1068
#35 0x000000000060b5fd in do_command (thd=0x32c86a0) at /home/elenst/bzr/5.5/sql/sql_parse.cc:794
#36 0x000000000071090d in do_handle_one_connection (thd_arg=0x32c86a0) at /home/elenst/bzr/5.5/sql/sql_connect.cc:1266
#37 0x00000000007102f4 in handle_one_connection (arg=0x32c86a0) at /home/elenst/bzr/5.5/sql/sql_connect.cc:1181
#38 0x00007f214936ee9a in start_thread (arg=0x7f2149d63700) at pthread_create.c:308

pushed to 5.2

I encountered a similar bug today while running "SET GLOBAL key_cache_segments = 64;" on 5.3.12, in both cases the stack trace mentions a heartbeat REPLACE query. The stack traces are very similar to the one in this bug report, but mysqld crashes. I am not sure if http://bazaar.launchpad.net/~maria-captains/maria/5.2/revision/3214 will also fix these, hopefully it will. Thank you.
Running mariadb-5.3.12-Linux-x86_64.tar.gz on RHEL5.

130711 6:28:05 [ERROR] mysqld got signal 8 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.3.12-MariaDB-log
key_buffer_size=10737418240
read_buffer_size=8388608
max_used_connections=188
max_threads=401
thread_count=13
connection_count=13
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 15829109 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x2b71d000cf60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x2b71aa2490f8 thread_stack 0x48000
./bin/mysqld(my_print_stacktrace+0x2e) [0x998a1e]
./bin/mysqld(handle_fatal_signal+0x3f9) [0x71ede9]
/lib64/libpthread.so.0 [0x3e7b40eca0]
./bin/mysqld(simple_key_cache_read+0xfa) [0x99f33a]
./bin/mysqld(_mi_fetch_keypage+0x48) [0x7e1ff8]
./bin/mysqld [0x7c64a6]
./bin/mysqld(_mi_ck_real_write_btree+0x60) [0x7c68e0]
./bin/mysqld(_mi_ck_write_btree+0x72) [0x7c6992]
./bin/mysqld(mi_write+0x320) [0x7c6ee0]
./bin/mysqld(handler::ha_write_row(unsigned char*)+0x37) [0x7137f7]
./bin/mysqld(write_record(THD*, st_table*, st_copy_info*)+0x34e) [0x691c4e]
./bin/mysqld(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)+0xa31) [0x696891]
./bin/mysqld(mysql_execute_command(THD*)+0xad8) [0x6048c8]
./bin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x299) [0x60a629]
./bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xdf9) [0x60b879]
./bin/mysqld(do_command(THD*)+0x101) [0x60c0b1]
./bin/mysqld(handle_one_connection+0xfd) [0x5fd7bd]
/lib64/libpthread.so.0 [0x3e7b40683d]
/lib64/libc.so.6(clone+0x6d) [0x3e7a8d503d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x306c9cb8): REPLACE INTO heartbeat_hostname_a(id, ts) VALUES(1, NOW())
Connection ID (thread ID): 1
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on

130711 6:56:53 [ERROR] mysqld got signal 8 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.3.12-MariaDB-log
key_buffer_size=4294967296
read_buffer_size=8388608
max_used_connections=94
max_threads=1001
thread_count=17
connection_count=17
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 17532709 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x10ee9d70
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x2b7ca13570f8 thread_stack 0x48000
./bin/mysqld(my_print_stacktrace+0x2e) [0x998a1e]
./bin/mysqld(handle_fatal_signal+0x3f9) [0x71ede9]
/lib64/libpthread.so.0 [0x2b7b04d88ca0]
./bin/mysqld(simple_key_cache_read+0xfa) [0x99f33a]
./bin/mysqld(_mi_fetch_keypage+0x48) [0x7e1ff8]
./bin/mysqld [0x7c64a6]
./bin/mysqld(_mi_ck_real_write_btree+0x60) [0x7c68e0]
./bin/mysqld(_mi_ck_write_btree+0x72) [0x7c6992]
./bin/mysqld(mi_write+0x320) [0x7c6ee0]
./bin/mysqld(handler::ha_write_row(unsigned char*)+0x37) [0x7137f7]
./bin/mysqld(write_record(THD*, st_table*, st_copy_info*)+0x34e) [0x691c4e]
./bin/mysqld(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)+0xa31) [0x696891]
./bin/mysqld(mysql_execute_command(THD*)+0xad8) [0x6048c8]
./bin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x299) [0x60a629]
./bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xdf9) [0x60b879]
./bin/mysqld(do_command(THD*)+0x101) [0x60c0b1]
./bin/mysqld(handle_one_connection+0xfd) [0x5fd7bd]
/lib64/libpthread.so.0 [0x2b7b04d8083d]
/lib64/libc.so.6(clone+0x6d) [0x2b7b057f903d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x10f067c8): REPLACE INTO heartbeat_hostname_b(id, ts) VALUES(1, NOW())
Connection ID (thread ID): 3
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on

Peter, it is not similar. The one is crash, the other is hang. The ones you have is similar to https://bugs.launchpad.net/maria/+bug/1008293

Thank you Vladislav, I have created https://mariadb.atlassian.net/browse/MDEV-4780