[MDEV-14383] tokudb_bugs. tests failed in buildbot, lost connection to server Created: 2017-11-13  Updated: 2019-09-11  Resolved: 2019-09-01

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB, Tests
Affects Version/s: 5.5
Fix Version/s: 5.5.66

Type: Bug Priority: Blocker
Reporter: Alice Sherepa Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-14354 rpl-tokudb tests fail massively with ... Open
relates to MDEV-7550 TokuDB crashes in build tests on Laun... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos73-amd64/builds/2827/steps/mtr/logs/stdio

tokudb_alter_table.ai_part, tokudb_bugs.alter_part_tokudb_bug_155,
tokudb_bugs.db756_card_part_hash, tokudb_bugs.db756_card_part_hash_1,
tokudb_bugs.db756_card_part_hash_2 ,tokudb_bugs.db756_card_part_hash_2_pick,
tokudb_bugs.db756_card_part_hash_1_pick, tokudb_bugs.db757_part_alter_analyze,
tokudb_bugs.db766, tokudb_bugs.mdev4533

tokudb_bugs.mdev4533                     w3 [ fail ]
 
        Test ended at 2017-11-11 20:19:45
 
rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533
 
tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb
 
tokudb_bugs.5733_innodb
 
CURRENT_TEST: tokudb_bugs.mdev4533
 
Server [mysqld.1 - pid: 1437, winpid: 1437, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
171111 20:19:44 [Note] /usr/sbin/mysqld (mysqld 5.5.59-MariaDB) starting as process 1438 ...
 
171111 20:19:44 [Note] Plugin 'FEDERATED' is disabled.
 
171111 20:19:44 [Note] Plugin 'ARCHIVE' is disabled.
 
171111 20:19:45 [Note] Plugin 'InnoDB' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_RSEG' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_UNDO_LOGS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_TRX' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_CMP' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_CMPMEM' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_SYS_STATS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_TABLE_STATS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_INDEX_STATS' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_BUFFER_POOL_PAGES' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_BUFFER_POOL_PAGES_INDEX' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_BUFFER_POOL_PAGES_BLOB' is disabled.
 
171111 20:19:45 [Note] Plugin 'XTRADB_ADMIN_COMMAND' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
 
171111 20:19:45 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
 
171111 20:19:45 [Note] Plugin 'BLACKHOLE' is disabled.
 
171111 20:19:45 [Note] Plugin 'FEEDBACK' is disabled.
 
171111 20:19:45 [Note] Plugin 'partition' is disabled.
 
171111 20:19:45 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
 
171111 20:19:45 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
 
171111 20:19:45 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-interval=60'
 
171111 20:19:45 [Warning] /usr/sbin/mysqld: unknown variable 'loose-debug-sync-timeout=300'
 
171111 20:19:45 [Note] Server socket created on IP: '0.0.0.0'.
 
171111 20:19:45 [Note] Event Scheduler: Loaded 0 events
 
171111 20:19:45 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '5.5.59-MariaDB'  socket: '/dev/shm/var/tmp/3/mysqld.1.sock'  port: 16040  MariaDB Server
 
171111 20:19:45 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 5.5.59-MariaDB
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62494 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xac24be]
 
/usr/sbin/mysqld(handle_fatal_signal+0x390)[0x6e4860]
 
/lib64/libpthread.so.0(+0xf370)[0x7f5fbc1f6370]
 
/lib64/libpthread.so.0(pthread_mutex_lock+0x0)[0x7f5fbc1f0bd0]
 
/usr/sbin/mysqld[0xb31a06]
 
/usr/sbin/mysqld[0xb32324]
 
/usr/sbin/mysqld[0xb32583]
 
/lib64/libpthread.so.0(+0x7bc2)[0x7f5fbc1eebc2]
 
/lib64/libpthread.so.0(+0x7dd3)[0x7f5fbc1eedd3]
 
/lib64/libc.so.6(clone+0x6d)[0x7f5fbaa6773d]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file
 
----------SERVER LOG END-------------
 
mysqltest failed but provided no output
 
The result from queries just before the failure was:
 
< snip >
 
DROP TABLE IF EXISTS t1;
 
CREATE TABLE t1 (a INT(11), b CHAR(8)) ENGINE=TokuDB;
 
INSERT INTO t1 (a,b) VALUES (10000,'foobar'),(1,'a'),(2,'b'),(3,'c'),(4,'d'),(5,'e');
 
DELETE IGNORE FROM t1;
 
drop table t1;



 Comments   
Comment by Elena Stepanova [ 2017-11-13 ]

These tests are run with tcmalloc, see also MDEV-14354.

Comment by Elena Stepanova [ 2018-07-12 ]

It looks like it was a momentary problem on Nov 10-11 on CentOS 7.3, never happened again since then. The logs aren't of course available anymore, possibly it was related to us experimenting with tcmalloc test runs at the time.

Comment by Elena Stepanova [ 2019-07-23 ]

It's happening again, and now it seems persistent. Same story: 5.5, CentOS 7.3, tcmalloc, tokudb* tests.
serg, can I just disable TokuDB tests there, or do you want to get it fixed? I can't imagine anyone would care enough to fix TokuDB tests on 5.5.

Example:
http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos73-amd64/builds/11276

5.5 cc86a0bd11a436dfe4df0622481c571f83b1680a

 
tokudb_alter_table.ai_part               w4 [ fail ]
 
        Test ended at 2019-07-23 13:00:46
 
 
 
CURRENT_TEST: tokudb_alter_table.ai_part
 
mysqltest: At line 11: query 'ALTER TABLE foo ADD KEY(b)' failed: 2013: Lost connection to MySQL server during query
 
 
 
The result from queries just before the failure was:
 
SET DEFAULT_STORAGE_ENGINE='TokuDB';
 
DROP TABLE IF EXISTS foo;
 
SET SESSION tokudb_disable_slow_alter=1;
 
CREATE TABLE foo (a INT, b INT, PRIMARY KEY (a)) PARTITION BY HASH(a) PARTITIONS 2;
 
INSERT INTO foo VALUES (1,0),(2,0);
 
SHOW CREATE TABLE foo;
 
Table	Create Table
 
foo	CREATE TABLE `foo` (
 
  `a` int(11) NOT NULL DEFAULT '0',
 
  `b` int(11) DEFAULT NULL,
 
  PRIMARY KEY (`a`)
 
) ENGINE=TokuDB DEFAULT CHARSET=latin1
 
/*!50100 PARTITION BY HASH (a)
 
PARTITIONS 2 */
 
ALTER TABLE foo ADD KEY(b);
 
 
 
 
 
Server [mysqld.1 - pid: 7727, winpid: 7727, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
190723 13:00:43 [Note] /usr/sbin/mysqld (mysqld 5.5.65-MariaDB) starting as process 7728 ...
 
190723 13:00:43 [Note] Plugin 'ARCHIVE' is disabled.
 
190723 13:00:43 [Note] Plugin 'BLACKHOLE' is disabled.
 
190723 13:00:43 [Note] Plugin 'FEDERATED' is disabled.
 
190723 13:00:43 [Note] Plugin 'InnoDB' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_RSEG' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_UNDO_LOGS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_TRX' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_CMP' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_CMPMEM' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_SYS_STATS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_TABLE_STATS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_INDEX_STATS' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_BUFFER_POOL_PAGES' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_BUFFER_POOL_PAGES_INDEX' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_BUFFER_POOL_PAGES_BLOB' is disabled.
 
190723 13:00:43 [Note] Plugin 'XTRADB_ADMIN_COMMAND' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_CHANGED_PAGES' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
 
190723 13:00:43 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
 
190723 13:00:43 [Note] Plugin 'FEEDBACK' is disabled.
 
190723 13:00:43 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
 
190723 13:00:43 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
 
190723 13:00:43 [Warning] /usr/sbin/mysqld: unknown variable 'loose-feedback-debug-interval=60'
 
190723 13:00:43 [Warning] /usr/sbin/mysqld: unknown variable 'loose-debug-sync-timeout=300'
 
190723 13:00:43 [Note] Server socket created on IP: '0.0.0.0'.
 
190723 13:00:43 [Note] Event Scheduler: Loaded 0 events
 
190723 13:00:43 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '5.5.65-MariaDB'  socket: '/dev/shm/var/tmp/4/mysqld.1.sock'  port: 16040  MariaDB Server
 
190723 13:00:43 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 5.5.65-MariaDB
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=1
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62495 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7ff378640000
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7ff380679c90 thread_stack 0x48000
 
addr2line: Dwarf Error: Offset (411249) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (7500912) greater than or equal to .debug_info size (2145536).
 
addr2line: Dwarf Error: Unable to read alt ref 7500912.
 
sql/field.cc:2437(Field_decimal::cmp(unsigned char const*, unsigned char const*))[0xac72ee]
 
/usr/sbin/mysqld(handle_fatal_signal+0x390)[0x6e7970]
 
sigaction.c:0(__restore_rt)[0x7ff37feb8370]
 
:0(__GI___pthread_mutex_lock)[0x7ff37feb2bd0]
 
addr2line: Dwarf Error: Offset (411249) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (7500912) greater than or equal to .debug_info size (2145536).
 
addr2line: Dwarf Error: Unable to read alt ref 7500912.
 
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (1342584).
 
sql/item_func.h:119(Create_func_export_set::create_native(THD*, st_mysql_lex_string, List<Item>*))[0xb368e6]
 
sql/item.cc:4854(Item_field::fix_outer_field(THD*, Field**, Item**))[0xb08760]
 
sql/item.h:115(Item_func_in::Item_func_in(List<Item>&))[0xabc190]
 
tokudb/ha_tokudb.cc:241(ha_tokudb::external_lock(THD*, int))[0x7ff37b860485]
 
tokudb/ha_tokudb.cc:2212(ha_tokudb::__close())[0x7ff37b86071f]
 
tokudb/ha_tokudb.cc:2189(ha_tokudb::close())[0x7ff37b860836]
 
addr2line: Dwarf Error: Offset (411249) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (7500912) greater than or equal to .debug_info size (2145536).
 
addr2line: Dwarf Error: Unable to read alt ref 7500912.
 
sql/sql_list.h:206(convert_join_subqueries_to_semijoins(JOIN*))[0xa93180]
 
/usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x38)[0x63e3c8]
 
/usr/sbin/mysqld[0x558ec3]
 
/usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0x189)[0x55b1c9]
 
/usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_function+0x102)[0x5636e2]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P24st_ha_create_informationP10TABLE_LISTP10Alter_infojP8st_orderbb+0x604)[0x624734]
 
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (1050503) greater than or equal to .debug_str size (64251).
 
/usr/sbin/mysqld(_ZN21Alter_table_statement7executeEP3THD+0x4bf)[0x82f55f]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x18cb)[0x5a969b]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x11e)[0x5ae69e]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1754)[0x5b0624]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x20b)[0x6695eb]
 
/usr/sbin/mysqld(handle_one_connection+0x43)[0x6696b3]
 
addr2line: Dwarf Error: Offset (380080) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (50331690) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (50331690) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (935527246) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (22741810) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (2567111110) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (2567111110) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (62128393) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (62128393) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (380080) greater than or equal to .debug_str size (64251).
 
addr2line: Dwarf Error: Offset (939527433) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (939527433) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (127023181) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (127023181) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (2902655624) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (2902655624) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (28115738) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (28115738) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (2181235106) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (255143937) greater than or equal to .debug_str size (1342584).
 
addr2line: Dwarf Error: Offset (255143937) greater than or equal to .debug_str size (1342584).
 
sql/log_event.h:1219(Log_event::free_temp_buf())[0x93d421]
 
pthread_create.c:0(start_thread)[0x7ff37feb0dc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7ff37e6ad73d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7ff36641c018): ALTER TABLE foo ADD KEY(b)
 
Connection ID (thread ID): 2
 
Status: NOT_KILLED

Comment by Elena Stepanova [ 2019-07-23 ]

Raised to a blocker as a technicality, to get a decision and disable the tests, permanently or temporarily.

Generated at Thu Feb 08 08:13:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.