[MDEV-5905] Creating tmp. memory table kills the server Created: 2014-03-19  Updated: 2014-03-26  Resolved: 2014-03-26

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 5.5.36, 10.0.9
Fix Version/s: 5.5.37, 10.0.10

Type: Bug Priority: Critical
Reporter: Slawomir Pryczek Assignee: Michael Widenius
Resolution: Fixed Votes: 0
Labels: upstream
Environment:

FC 20, Tokudb installed from binary distribution found on tokudb website

Attachments: File _testing44.sql     File mdev5905.test     Text File mysql-dump.txt     Text File version .34 - dead completely.txt     Text File version .34 - restarted.txt     Text File version .35 - restarted.txt    

 Description   

Version: 5.5.36-tokudb-7.1.5-MariaDB-log
Don't know if that's the right bugtracker to submit it, as now we're trying mariadb with toku enabled.

It seems that's data or structure dependent, if i do SELECT `domain` instead of `id` it won't crash. Also when i switch to other table, and do SELECT `id` it won't crash.

Table:

CREATE TABLE `raw_stats_lookup` (
 
  `id` int(11) NOT NULL AUTO_INCREMENT,
 
  `network_id` tinyint(11) NOT NULL,
 
  `creative_id` int(11) NOT NULL,
 
  `domain` varchar(90) NOT NULL,
 
  `__last_used` date NOT NULL,
 
  PRIMARY KEY (`id`),
 
  UNIQUE KEY `_uk` (`domain`,`network_id`,`creative_id`) USING BTREE,
 
  KEY `creative_id` (`creative_id`)
 
) ENGINE=Aria AUTO_INCREMENT=763772 DEFAULT CHARSET=utf8 PAGE_CHECKSUM=0 TRANSACTIONAL=0;

After issuing this query the DB is killed

CREATE TEMPORARY TABLE ___tmp_raw_stats_lookup ENGINE=MEMORY
 
    SELECT id
 
    FROM `stats`.raw_stats_lookup
 
    WHERE domain = 'x.com'

I made some other steps:

  • duplicated the table and switched the engine to innodb (crashing again)
  • moved the table to another server (just the table, not all data), same hardware and settings (keeps crashing)
  • changed the ENGINE=MEMORY to ENGINE=MYISAM => no crash
  • moved the data to machine with normal mariadb (5.5.35-MariaDB-log) - it WONT crash

For DB dump please contact me on email if you like to see, sorry, there's some domain data so don't want to share in public. pslawek83 !a-t! o2.pl ... if you want to look into this (in dump it said to report errors to you).

Attaching dump...

BTW: We had problems with making mariadb with toku running on FC20, compilation error then we compiled using mysql_release (instead of Release) but we ended up with debug version. Anything you can suggest that's tested? (OS + compile? OS + binary?)

Thanks everyone as always



 Comments   
Comment by Slawomir Pryczek [ 2014-03-20 ]

We have nailed down the error to this config line.
set max_heap_table_size = 1024*1024*1024*19; -> works
set max_heap_table_size = 1024*1024*1024*21; -> works
set max_heap_table_size = 1024*1024*1024*20; -> crashes

After 20GB limit it seems that at random number the DB is crashing while creating memory table, like in attached log. The server has 128GB of RAM.

Comment by Slawomir Pryczek [ 2014-03-20 ]

Another update, after issuing that query it appears that it's retutning correctly, just the VERY NEXT query is killing the server (that's query to get status from navicat)...

In command line you need to issue the query and then issue
SHOW VARIABLES();
for the server to crash.

Comment by Elena Stepanova [ 2014-03-24 ]

Hi Slawomir,

If the problem only occurs on TokuDB build of MariaDB (the one that you downloaded from TokuDB site), but does not occur on the MariaDB release of the same number, then the better place to file the bug would probably be here: https://github.com/Tokutek/mariadb-5.5/issues .

But it makes sense to check first that it is indeed Tokutek-build-specific.
You mentioned that it is not reproducible on MariaDB 5.5.35; what about MariaDB 5.5.36?

Please note that if you just need MariaDB with TokuDB engine, and don't have any particular preferences for Tokutek builds, you can use the regular MariaDB release, it contains TokuDB engine, you just need to enable it.
It also answers your question about FC20.

Use the MariaDB yum repo as described in https://downloads.mariadb.org/mariadb/repositories/#mirror=mephi&distro=Fedora&distro_release=fedora20-amd64&version=5.5

After you started the server, run INSTALL SONAME 'ha_tokudb'. The TokuDB engine will be installed.
Alternatively, you can add plugin-load=ha_tokudb option to the cnf file.

Comment by Slawomir Pryczek [ 2014-03-25 ]

Error logs from other versions / distributions

Comment by Slawomir Pryczek [ 2014-03-25 ]

SQL File to test the bug

Comment by Slawomir Pryczek [ 2014-03-25 ]

Hi Elena, thank you for the answer...

Actually the problem wasn't reproduced on .35 because config was different on this server when i checked, sorry for confusion.

I was able to reproduce on .34 (debian wheezy from repo) and also .35 installed from mariadb, when config was changed in console.

Versions strings are:
5.5.34-MariaDB-1~wheezy-log (from debian wheezy repo)
5.5.35-MariaDB-log (from mariadb.org)

I created some very small table without indexes, and the bug is still there. SQL for testing:

SET max_heap_table_size = 1024*1024*1024*20;

CREATE TEMPORARY TABLE __somerandomtablename81872 ENGINE=MEMORY
SELECT id
FROM _testing44
WHERE domain = 'tester.com'

When i run the query both servers were killed or restarted... (sometimes it just restarts mysqld server, sometimes kills it - the process just hangs up and won't accept any new connections)
i wasn't able to get stack trace from debian (just "???") but there's trace for .35.

Im attaching all logs and SQL file to reproduce that.

Very important: i was only able to get this bug when i connected to server from remote machines. From local mysql console, it seems working normally. So may be networking related. I can give you access to the server with debian/.34 as it's my own dev machine, so you can see...

Comment by Slawomir Pryczek [ 2014-03-25 ]

Another quick comment... when i do:
SET max_heap_table_size = 1024*1024*1024*19;
or
SET max_heap_table_size = 1024*1024*1024*21;
the queries are working ok on all servers.

Comment by Elena Stepanova [ 2014-03-25 ]

Slawomir,

Thanks a lot for the test case. I was able to reproduce the crash with it.

Comment by Elena Stepanova [ 2014-03-25 ]

Hi Slawomir,

It turns out the bug is reproducible on Oracle MySQL. In such cases we also file reports at bugs.mysql.com to inform upstream about the problem. Do you prefer to do it yourself, or should we do it on your behalf?

Comment by Elena Stepanova [ 2014-03-25 ]

Even though it is an upstream bug, maybe we should not wait for the upstream fix.

I've attached mdev5905.test - the very same data/query as Slawomir reported, just put together in one file. It causes a crash, here is the stack trace from 5.5 revno 4115, revision-id revision-id: holyfoot@askmonty.org-20140323144448-ehflksuvchgcr49y:

#3  <signal handler called>
 
#4  0x000000000064331a in intern_plugin_unlock (lex=0x0, plugin=0x7fb58141d8c0) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1288
 
#5  0x0000000000643557 in plugin_unlock_list (thd=0x0, list=0x7fb58534ced8, count=0) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1330
 
#6  0x0000000000621b7c in lex_end (lex=0x7fb58534c2d0) at /home/elenst/bzr/5.5/sql/sql_lex.cc:551
 
#7  0x00000000005fd49b in THD::end_statement (this=0x7fb585349060) at /home/elenst/bzr/5.5/sql/sql_class.cc:3235
 
#8  0x000000000063ac85 in mysql_parse (thd=0x7fb585349060, rawbuf=0x7fb581548078 "CREATE TEMPORARY TABLE __somerandomtablename81872 ENGINE=MEMORY\nSELECT id\nFROM _testing44\nWHERE domain = 'tester.com'", length=117, parser_state=0x7fb581fb5630) at /home/elenst/bzr/5.5/sql/sql_parse.cc:5816
 
#9  0x000000000062eba9 in dispatch_command (command=COM_QUERY, thd=0x7fb585349060, packet=0x7fb5826fc061 "CREATE TEMPORARY TABLE __somerandomtablename81872 ENGINE=MEMORY\nSELECT id\nFROM _testing44\nWHERE domain = 'tester.com'", packet_length=117) at /home/elenst/bzr/5.5/sql/sql_parse.cc:1079
 
#10 0x000000000062dd35 in do_command (thd=0x7fb585349060) at /home/elenst/bzr/5.5/sql/sql_parse.cc:793
 
#11 0x000000000072f132 in do_handle_one_connection (thd_arg=0x7fb585349060) at /home/elenst/bzr/5.5/sql/sql_connect.cc:1266
 
#12 0x000000000072ebf1 in handle_one_connection (arg=0x7fb585349060) at /home/elenst/bzr/5.5/sql/sql_connect.cc:1181
 
#13 0x0000000000983ced in pfs_spawn_thread (arg=0x7fb5833eade0) at /home/elenst/bzr/5.5/storage/perfschema/pfs.cc:1015
 
#14 0x00007fb587da2b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
 
#15 0x00007fb5868f1a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

Here is also mdev5905-2.test - the minimal test case. It does not necessarily cause a crash, but produces valgrind warnings:

CREATE TABLE t1 (id INT);
 
INSERT INTO t1 VALUES (1);
 
INSERT INTO t1 VALUES (2);
 
 
 
SET max_heap_table_size = 1024*1024*1024*20;
 
 
 
CREATE TEMPORARY TABLE tmp ENGINE=MEMORY
 
  SELECT id FROM t1;
 

==30210== Thread 5:
 
==30210== Invalid write of size 8
 
==30210==    at 0x4C2AA3B: memcpy (mc_replace_strmem.c:883)
 
==30210==    by 0x96B514: heap_write (hp_write.c:56)
 
==30210==    by 0x9641C7: ha_heap::write_row(unsigned char*) (ha_heap.cc:251)
 
==30210==    by 0x7ECBFC: handler::ha_write_row(unsigned char*) (handler.cc:5201)
 
==30210==    by 0x6170BA: write_record(THD*, TABLE*, st_copy_info*) (sql_insert.cc:1843)
 
==30210==    by 0x61BA4E: select_insert::send_data(List<Item>&) (sql_insert.cc:3587)
 
==30210==    by 0x68BA3B: end_send(JOIN*, st_join_table*, bool) (sql_select.cc:17899)
 
==30210==    by 0x68983E: evaluate_join_record(JOIN*, st_join_table*, int) (sql_select.cc:17010)
 
==30210==    by 0x689307: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:16830)
 
==30210==    by 0x6889D5: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:16453)
 
==30210==    by 0x6679FE: JOIN::exec() (sql_select.cc:2859)
 
==30210==    by 0x668204: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3079)
 
==30210==    by 0x65EE1C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:319)
 
==30210==    by 0x63241A: mysql_execute_command(THD*) (sql_parse.cc:2621)
 
==30210==    by 0x63AB87: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:5800)
 
==30210==    by 0x62EBA8: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1079)
 
==30210==  Address 0xb81d390 is 0 bytes after a recently re-allocated block of size 112 alloc'd
 
==30210==    at 0xD2EB99: malloc (jemalloc.c:906)
 
==30210==    by 0xCE9E63: sf_malloc (safemalloc.c:105)
 
==30210==    by 0xCD6673: my_malloc (my_malloc.c:41)
 
==30210==    by 0x96C2EA: hp_get_new_block (hp_block.c:80)
 
==30210==    by 0x96BA25: next_free_record_pos (hp_write.c:165)
 
==30210==    by 0x96B459: heap_write (hp_write.c:45)
 
==30210==    by 0x9641C7: ha_heap::write_row(unsigned char*) (ha_heap.cc:251)
 
==30210==    by 0x7ECBFC: handler::ha_write_row(unsigned char*) (handler.cc:5201)
 
==30210==    by 0x6170BA: write_record(THD*, TABLE*, st_copy_info*) (sql_insert.cc:1843)
 
==30210==    by 0x61BA4E: select_insert::send_data(List<Item>&) (sql_insert.cc:3587)
 
==30210==    by 0x68BA3B: end_send(JOIN*, st_join_table*, bool) (sql_select.cc:17899)
 
==30210==    by 0x68983E: evaluate_join_record(JOIN*, st_join_table*, int) (sql_select.cc:17010)
 
==30210==    by 0x689177: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:16791)
 
==30210==    by 0x6889D5: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:16453)
 
==30210==    by 0x6679FE: JOIN::exec() (sql_select.cc:2859)
 
==30210==    by 0x668204: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3079)
 
==30210== Invalid write of size 1
 
==30210==    at 0x96B528: heap_write (hp_write.c:57)
 
==30210==    by 0x9641C7: ha_heap::write_row(unsigned char*) (ha_heap.cc:251)
 
==30210==    by 0x7ECBFC: handler::ha_write_row(unsigned char*) (handler.cc:5201)
 
==30210==    by 0x6170BA: write_record(THD*, TABLE*, st_copy_info*) (sql_insert.cc:1843)
 
==30210==    by 0x61BA4E: select_insert::send_data(List<Item>&) (sql_insert.cc:3587)
 
==30210==    by 0x68BA3B: end_send(JOIN*, st_join_table*, bool) (sql_select.cc:17899)
 
==30210==    by 0x68983E: evaluate_join_record(JOIN*, st_join_table*, int) (sql_select.cc:17010)
 
==30210==    by 0x689307: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:16830)
 
==30210==    by 0x6889D5: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:16453)
 
==30210==    by 0x6679FE: JOIN::exec() (sql_select.cc:2859)
 
==30210==    by 0x668204: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3079)
 
==30210==    by 0x65EE1C: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:319)
 
==30210==    by 0x63241A: mysql_execute_command(THD*) (sql_parse.cc:2621)
 
==30210==    by 0x63AB87: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:5800)
 
==30210==    by 0x62EBA8: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1079)
 
==30210==    by 0x62DD34: do_command(THD*) (sql_parse.cc:793)
 
==30210==  Address 0xb81d398 is 8 bytes after a recently re-allocated block of size 112 alloc'd
 
==30210==    at 0xD2EB99: malloc (jemalloc.c:906)
 
==30210==    by 0xCE9E63: sf_malloc (safemalloc.c:105)
 
==30210==    by 0xCD6673: my_malloc (my_malloc.c:41)
 
==30210==    by 0x96C2EA: hp_get_new_block (hp_block.c:80)
 
==30210==    by 0x96BA25: next_free_record_pos (hp_write.c:165)
 
==30210==    by 0x96B459: heap_write (hp_write.c:45)
 
==30210==    by 0x9641C7: ha_heap::write_row(unsigned char*) (ha_heap.cc:251)
 
==30210==    by 0x7ECBFC: handler::ha_write_row(unsigned char*) (handler.cc:5201)
 
==30210==    by 0x6170BA: write_record(THD*, TABLE*, st_copy_info*) (sql_insert.cc:1843)
 
==30210==    by 0x61BA4E: select_insert::send_data(List<Item>&) (sql_insert.cc:3587)
 
==30210==    by 0x68BA3B: end_send(JOIN*, st_join_table*, bool) (sql_select.cc:17899)
 
==30210==    by 0x68983E: evaluate_join_record(JOIN*, st_join_table*, int) (sql_select.cc:17010)
 
==30210==    by 0x689177: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:16791)
 
==30210==    by 0x6889D5: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:16453)
 
==30210==    by 0x6679FE: JOIN::exec() (sql_select.cc:2859)
 
==30210==    by 0x668204: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3079)
 
==30210== Invalid read of size 1
 
==30210==    at 0x96DE6B: heap_check_heap (_check.c:82)
 
==30210==    by 0x96CB8C: hp_close (hp_close.c:39)
 
==30210==    by 0x96CB0C: heap_close (hp_close.c:28)
 
==30210==    by 0x963DC8: ha_heap::close() (ha_heap.cc:150)
 
==30210==    by 0x7E75B3: handler::ha_close() (handler.cc:2317)
 
==30210==    by 0x6F72A9: closefrm(TABLE*, bool) (table.cc:2735)
 
==30210==    by 0x5D2F3A: close_temporary(TABLE*, bool, bool) (sql_base.cc:2284)
 
==30210==    by 0x5D1A97: close_temporary_tables(THD*) (sql_base.cc:1688)
 
==30210==    by 0x5F793F: THD::cleanup() (sql_class.cc:1395)
 
==30210==    by 0x585ADF: thd_cleanup(THD*) (mysqld.cc:2449)
 
==30210==    by 0x585B9B: unlink_thd(THD*) (mysqld.cc:2483)
 
==30210==    by 0x585F7C: one_thread_per_connection_end(THD*, bool) (mysqld.cc:2604)
 
==30210==    by 0x72F1D7: do_handle_one_connection(THD*) (sql_connect.cc:1277)
 
==30210==    by 0x72EBF0: handle_one_connection (sql_connect.cc:1181)
 
==30210==    by 0x983CEC: pfs_spawn_thread (pfs.cc:1015)
 
==30210==    by 0x4E35B4F: start_thread (pthread_create.c:304)
 
==30210==  Address 0xb81d398 is 8 bytes after a recently re-allocated block of size 112 alloc'd
 
==30210==    at 0xD2EB99: malloc (jemalloc.c:906)
 
==30210==    by 0xCE9E63: sf_malloc (safemalloc.c:105)
 
==30210==    by 0xCD6673: my_malloc (my_malloc.c:41)
 
==30210==    by 0x96C2EA: hp_get_new_block (hp_block.c:80)
 
==30210==    by 0x96BA25: next_free_record_pos (hp_write.c:165)
 
==30210==    by 0x96B459: heap_write (hp_write.c:45)
 
==30210==    by 0x9641C7: ha_heap::write_row(unsigned char*) (ha_heap.cc:251)
 
==30210==    by 0x7ECBFC: handler::ha_write_row(unsigned char*) (handler.cc:5201)
 
==30210==    by 0x6170BA: write_record(THD*, TABLE*, st_copy_info*) (sql_insert.cc:1843)
 
==30210==    by 0x61BA4E: select_insert::send_data(List<Item>&) (sql_insert.cc:3587)
 
==30210==    by 0x68BA3B: end_send(JOIN*, st_join_table*, bool) (sql_select.cc:17899)
 
==30210==    by 0x68983E: evaluate_join_record(JOIN*, st_join_table*, int) (sql_select.cc:17010)
 
==30210==    by 0x689177: sub_select(JOIN*, st_join_table*, bool) (sql_select.cc:16791)
 
==30210==    by 0x6889D5: do_select(JOIN*, List<Item>*, TABLE*, Procedure*) (sql_select.cc:16453)
 
==30210==    by 0x6679FE: JOIN::exec() (sql_select.cc:2859)
 
==30210==    by 0x668204: mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3079)

Also, both cases produce "Error: Freeing unallocated data or underrun buffer mysys/safemalloc.c:179, mysys/my_malloc.c:120, sql/sql_plugin.cc:1272, sql/sql_plugin.cc:1329, sql/sql_lex.cc:553, sql/sql_class.cc:3236, sql/sql_parse.cc:5817, sql/sql_parse.cc:1081" on a debug version

Both test cases should be re-checked after the fix.

Comment by Slawomir Pryczek [ 2014-03-25 ]

Hi Elena,
yes if you can please report to oracle's mysql bugtracker, i unfortunately don't have any machines with mysql servers i can kill

Thanks

Comment by Elena Stepanova [ 2014-03-26 ]

Refiled as http://bugs.mysql.com/bug.php?id=72128

Comment by Michael Widenius [ 2014-03-26 ]

Pushed into 5.5

Generated at Thu Feb 08 07:07:55 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.