Details
-
Bug
-
Status: Closed (View Workflow)
-
Minor
-
Resolution: Cannot Reproduce
-
10.1.14
-
Ubuntu 14.04.5
Description
Hello!
We experienced MariaDB server crash last night. Unfortunately I don't have many details on this, but our server is almost purely TokuDB powered and at this time we had repartitioning event running (it started at 0:15).
Attachments
Issue Links
Activity
It was one-time failure, but today this appeared on another server (MariaDB 10.1.11). Error message is different though.
First database is ~250G in size, second one is ~30G. It is a production setup, I can share table definitions, but not the contents.
Here are the logs from repartitioning script (db and table names changed):
DO localhost alter table `database1`.`table1` drop partition p20161011;
[ERR]: localhost Can't execute SQL statement: alter table `database1`.`table1` drop partition p20161011; on DB database1 : Lost connection to MySQL server during query
READY localhost alter table `database1`.`table1` drop partition p20161011;
DO localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);
[ERR]: localhost Can't execute SQL statement: alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB); on DB database1 : MySQL server has gone away
READY localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);
Reported to upstream now also:
https://bugs.launchpad.net/percona-server/+bug/1638806
Great, thanks. Lets see what upstream says, maybe they've already seen it before.
Dec 2 update: no activity so far.
Jan 3 update: no activity so far.
Feb 1 update: no activity so far.
March 2 update: no activity so far.
May 2 update: no activity so far.
The upstream bug report has been moved to https://jira.percona.com/browse/TDB-49 and then marked as incomplete due to the lack of information (and apparently even more so due to the lack of proof that the problem exists in non-MariaDB flavors). I guess there is no point waiting for upstream any further.
Well, I left that company, so I'm afraid I won't be able to update the status here anymore.
I guess we'll have to wait till somebody else complains about a similar issue. For now, I'll put here the stack traces from attached error logs, to make them somewhat searchable:
161026 0:15:30 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.1.14-MariaDB-1~trusty
|
key_buffer_size=16777216
|
read_buffer_size=131072
|
max_used_connections=16
|
max_threads=602
|
thread_count=6
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4865990 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x7f8343c17008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f841b060df0 thread_stack 0x30000
|
/home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2)
|
: No such file or directory
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ab5fed4b5e]
|
Backtrace: (Note: toku_do_assert=0x0x7f841b11ecb0)
|
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f841b18065b]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7f841b1807e3]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7f841b11e8d7]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z24toku_thread_pool_destroyPP16toku_thread_pool+0x1bb)[0x7f841b161d7b]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z31toku_ft_serialize_layer_destroyv+0x10)[0x7f841b1650b0]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z21toku_ft_layer_destroyv+0x20)[0x7f841b1650e0]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x40199)[0x7f841b0a2199]
|
/lib64/ld-linux-x86-64.so.2(+0x1070a)[0x7f8420b3b70a]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x3c1a9)[0x7f841ecaf1a9]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x3c1f5)[0x7f841ecaf1f5]
|
/usr/sbin/mysqld(+0xa9893c)[0x55ab5fee893c]
|
/usr/sbin/mysqld(my_print_stacktrace+0x4c)[0x55ab5fed4b7c]
|
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330]
|
/usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b]
|
/usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664]
|
/usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94]
|
/usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09]
|
/usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135]
|
/usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66]
|
/usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180]
|
/usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3]
|
/usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7]
|
/usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873]
|
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590]
|
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12]
|
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66]
|
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0]
|
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a]
|
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]
|
Engine status:
|
BUILD_ID = 0
|
Env panic code: 2
|
Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2)
|
Memory usage:
|
Arena 0:
|
system bytes = 0
|
in use bytes = 0
|
Total (incl. mmap):
|
system bytes = 0
|
in use bytes = 0
|
max mmap regions = 0
|
max mmap bytes = 0
|
Fatal signal 6 while backtracing
|
/usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b]
|
/usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664]
|
/usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94]
|
/usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09]
|
/usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135]
|
/usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66]
|
/usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180]
|
/usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3]
|
/usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7]
|
/usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873]
|
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590]
|
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12]
|
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66]
|
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0]
|
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a]
|
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f83bc579020): is an invalid pointer
|
Connection ID (thread ID): 8009371
|
Status: NOT_KILLED
|
/home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0)
|
: Success
|
Backtrace: (Note: toku_do_assert=0x0x7fe9c38e1cb0)
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fe9c394365b]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7fe9c39437e3]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66]
|
/usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8]
|
/usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071]
|
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640]
|
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c]
|
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6]
|
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20]
|
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea]
|
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]
|
Engine status:
|
BUILD_ID = 0
|
Env panic code: 18446744073709551615
|
Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0)
|
161103 0:15:05 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.1.11-MariaDB-1~trusty
|
key_buffer_size=16777216
|
read_buffer_size=131072
|
max_used_connections=21
|
max_threads=153
|
thread_count=7
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1248906 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0x7fe94efec008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fe9c85fcdf0 thread_stack 0x30000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fe9c80463fe]
|
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7fe9c7b73f9d]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fe9c60cc340]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fe9c5723cc9]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9c57270d8]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e8)[0x7fe9c39437e8]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9]
|
/usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f]
|
/usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66]
|
/usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8]
|
/usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071]
|
/usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640]
|
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c]
|
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e]
|
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6]
|
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd]
|
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20]
|
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9]
|
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea]
|
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7fe941020020): is an invalid pointer
|
Connection ID (thread ID): 687376
|
Status: NOT_KILLED
|
Was it a one-time failure, or are you getting it periodically?
What exactly does the repartitioning event do, which queries does it run?
How big is the database, how many tables are there?
Is it a production setup? What kind of data can you share (table definitions, table contents, ...)?
Did you try to report the bug to Percona? If you did, could you please paste the link to the report?
Please attach your cnf file(s).