[MDEV-11143] TokuDB crash (PerconaFT/util/threadpool.cc:112 toku_thread_destroy) Created: 2016-10-26  Updated: 2017-07-31  Resolved: 2017-07-31

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Affects Version/s: 10.1.14
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: akrus Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: tokudb, upstream-wontfix
Environment:

Ubuntu 14.04.5

Attachments: File my.cnf     File table1.sql     File tokudb_crash.log     File tokudb_crash2.log    

 Description   

Hello!

We experienced MariaDB server crash last night. Unfortunately I don't have many details on this, but our server is almost purely TokuDB powered and at this time we had repartitioning event running (it started at 0:15).

 Comments   
Comment by Elena Stepanova [ 2016-11-02 ]

Was it a one-time failure, or are you getting it periodically?
What exactly does the repartitioning event do, which queries does it run?
How big is the database, how many tables are there?
Is it a production setup? What kind of data can you share (table definitions, table contents, ...)?
Did you try to report the bug to Percona? If you did, could you please paste the link to the report?
Please attach your cnf file(s).

Comment by akrus [ 2016-11-03 ]

It was one-time failure, but today this appeared on another server (MariaDB 10.1.11). Error message is different though.

First database is ~250G in size, second one is ~30G. It is a production setup, I can share table definitions, but not the contents.

Here are the logs from repartitioning script (db and table names changed):

DO localhost alter table `database1`.`table1` drop partition p20161011;
[ERR]: localhost Can't execute SQL statement: alter table `database1`.`table1` drop partition p20161011; on DB database1 : Lost connection to MySQL server during query
READY localhost alter table `database1`.`table1` drop partition p20161011;
DO localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);
[ERR]: localhost Can't execute SQL statement: alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB); on DB database1 : MySQL server has gone away
READY localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);

Comment by akrus [ 2016-11-03 ]

Reported to upstream now also:
https://bugs.launchpad.net/percona-server/+bug/1638806

Comment by Elena Stepanova [ 2016-11-03 ]

Great, thanks. Lets see what upstream says, maybe they've already seen it before.

Dec 2 update: no activity so far.
Jan 3 update: no activity so far.
Feb 1 update: no activity so far.
March 2 update: no activity so far.
May 2 update: no activity so far.

Comment by Elena Stepanova [ 2017-06-01 ]

The upstream bug report has been moved to https://jira.percona.com/browse/TDB-49 and then marked as incomplete due to the lack of information (and apparently even more so due to the lack of proof that the problem exists in non-MariaDB flavors). I guess there is no point waiting for upstream any further.

Comment by Elena Stepanova [ 2017-06-29 ]

akrus, are you still getting it on recent versions?

Comment by akrus [ 2017-06-30 ]

Well, I left that company, so I'm afraid I won't be able to update the status here anymore.

Comment by Elena Stepanova [ 2017-07-31 ]

I guess we'll have to wait till somebody else complains about a similar issue. For now, I'll put here the stack traces from attached error logs, to make them somewhat searchable:

161026  0:15:30 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.14-MariaDB-1~trusty
 
key_buffer_size=16777216
 
read_buffer_size=131072
 
max_used_connections=16
 
max_threads=602
 
thread_count=6
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4865990 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f8343c17008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f841b060df0 thread_stack 0x30000
 
/home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2)
 
: No such file or directory
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ab5fed4b5e]
 
Backtrace: (Note: toku_do_assert=0x0x7f841b11ecb0)
 
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f841b18065b]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7f841b1807e3]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7f841b11e8d7]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z24toku_thread_pool_destroyPP16toku_thread_pool+0x1bb)[0x7f841b161d7b]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z31toku_ft_serialize_layer_destroyv+0x10)[0x7f841b1650b0]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z21toku_ft_layer_destroyv+0x20)[0x7f841b1650e0]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x40199)[0x7f841b0a2199]
 
/lib64/ld-linux-x86-64.so.2(+0x1070a)[0x7f8420b3b70a]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x3c1a9)[0x7f841ecaf1a9]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x3c1f5)[0x7f841ecaf1f5]
 
/usr/sbin/mysqld(+0xa9893c)[0x55ab5fee893c]
 
/usr/sbin/mysqld(my_print_stacktrace+0x4c)[0x55ab5fed4b7c]
 
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330]
 
/usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b]
 
/usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664]
 
/usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94]
 
/usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09]
 
/usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135]
 
/usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66]
 
/usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180]
 
/usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3]
 
/usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7]
 
/usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873]
 
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12]
 
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]
 
Engine status:
 
BUILD_ID = 0
 
Env panic code: 2
 
Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2)


Memory usage:
 
Arena 0:
 
system bytes     =          0
 
in use bytes     =          0
 
Total (incl. mmap):
 
system bytes     =          0
 
in use bytes     =          0
 
max mmap regions =          0
 
max mmap bytes   =          0
 
Fatal signal 6 while backtracing
 
/usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b]
 
/usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664]
 
/usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94]
 
/usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09]
 
/usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135]
 
/usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66]
 
/usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180]
 
/usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3]
 
/usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7]
 
/usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873]
 
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12]
 
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f83bc579020): is an invalid pointer
 
Connection ID (thread ID): 8009371
 
Status: NOT_KILLED


/home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0)
 
: Success
 
Backtrace: (Note: toku_do_assert=0x0x7fe9c38e1cb0)
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fe9c394365b]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7fe9c39437e3]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66]
 
/usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8]
 
/usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071]
 
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c]
 
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]
 
Engine status:
 
BUILD_ID = 0
 
Env panic code: 18446744073709551615
 
Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0)


161103  0:15:05 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.11-MariaDB-1~trusty
 
key_buffer_size=16777216
 
read_buffer_size=131072
 
max_used_connections=21
 
max_threads=153
 
thread_count=7
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1248906 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7fe94efec008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fe9c85fcdf0 thread_stack 0x30000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fe9c80463fe]
 
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7fe9c7b73f9d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fe9c60cc340]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fe9c5723cc9]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9c57270d8]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e8)[0x7fe9c39437e8]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9]
 
/usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f]
 
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66]
 
/usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8]
 
/usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071]
 
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c]
 
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fe941020020): is an invalid pointer
 
Connection ID (thread ID): 687376
 
Status: NOT_KILLED

Generated at Thu Feb 08 07:47:38 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.