Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11143

TokuDB crash (PerconaFT/util/threadpool.cc:112 toku_thread_destroy)

Details

    Description

      Hello!

      We experienced MariaDB server crash last night. Unfortunately I don't have many details on this, but our server is almost purely TokuDB powered and at this time we had repartitioning event running (it started at 0:15).

      Attachments

        1. my.cnf
          2 kB
          akrus
        2. table1.sql
          3 kB
          akrus
        3. tokudb_crash.log
          23 kB
          akrus
        4. tokudb_crash2.log
          22 kB
          akrus

        Activity

          Was it a one-time failure, or are you getting it periodically?
          What exactly does the repartitioning event do, which queries does it run?
          How big is the database, how many tables are there?
          Is it a production setup? What kind of data can you share (table definitions, table contents, ...)?
          Did you try to report the bug to Percona? If you did, could you please paste the link to the report?
          Please attach your cnf file(s).

          elenst Elena Stepanova added a comment - Was it a one-time failure, or are you getting it periodically? What exactly does the repartitioning event do, which queries does it run? How big is the database, how many tables are there? Is it a production setup? What kind of data can you share (table definitions, table contents, ...)? Did you try to report the bug to Percona? If you did, could you please paste the link to the report? Please attach your cnf file(s).
          akrus akrus added a comment -

          It was one-time failure, but today this appeared on another server (MariaDB 10.1.11). Error message is different though.

          First database is ~250G in size, second one is ~30G. It is a production setup, I can share table definitions, but not the contents.

          Here are the logs from repartitioning script (db and table names changed):

          DO localhost alter table `database1`.`table1` drop partition p20161011;
          [ERR]: localhost Can't execute SQL statement: alter table `database1`.`table1` drop partition p20161011; on DB database1 : Lost connection to MySQL server during query
          READY localhost alter table `database1`.`table1` drop partition p20161011;
          DO localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);
          [ERR]: localhost Can't execute SQL statement: alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB); on DB database1 : MySQL server has gone away
          READY localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);

          akrus akrus added a comment - It was one-time failure, but today this appeared on another server (MariaDB 10.1.11). Error message is different though. First database is ~250G in size, second one is ~30G. It is a production setup, I can share table definitions, but not the contents. Here are the logs from repartitioning script (db and table names changed): DO localhost alter table `database1`.`table1` drop partition p20161011; [ERR] : localhost Can't execute SQL statement: alter table `database1`.`table1` drop partition p20161011; on DB database1 : Lost connection to MySQL server during query READY localhost alter table `database1`.`table1` drop partition p20161011; DO localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB); [ERR] : localhost Can't execute SQL statement: alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB); on DB database1 : MySQL server has gone away READY localhost alter table `database1`.`table1` reorganize partition pmax into (partition p20161117 VALUES LESS THAN(1479420000),PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = TokuDB);
          akrus akrus added a comment -
          akrus akrus added a comment - Reported to upstream now also: https://bugs.launchpad.net/percona-server/+bug/1638806
          elenst Elena Stepanova added a comment - - edited

          Great, thanks. Lets see what upstream says, maybe they've already seen it before.

          Dec 2 update: no activity so far.
          Jan 3 update: no activity so far.
          Feb 1 update: no activity so far.
          March 2 update: no activity so far.
          May 2 update: no activity so far.

          elenst Elena Stepanova added a comment - - edited Great, thanks. Lets see what upstream says, maybe they've already seen it before. Dec 2 update: no activity so far. Jan 3 update: no activity so far. Feb 1 update: no activity so far. March 2 update: no activity so far. May 2 update: no activity so far.

          The upstream bug report has been moved to https://jira.percona.com/browse/TDB-49 and then marked as incomplete due to the lack of information (and apparently even more so due to the lack of proof that the problem exists in non-MariaDB flavors). I guess there is no point waiting for upstream any further.

          elenst Elena Stepanova added a comment - The upstream bug report has been moved to https://jira.percona.com/browse/TDB-49 and then marked as incomplete due to the lack of information (and apparently even more so due to the lack of proof that the problem exists in non-MariaDB flavors). I guess there is no point waiting for upstream any further.

          akrus, are you still getting it on recent versions?

          elenst Elena Stepanova added a comment - akrus , are you still getting it on recent versions?
          akrus akrus added a comment -

          Well, I left that company, so I'm afraid I won't be able to update the status here anymore.

          akrus akrus added a comment - Well, I left that company, so I'm afraid I won't be able to update the status here anymore.

          I guess we'll have to wait till somebody else complains about a similar issue. For now, I'll put here the stack traces from attached error logs, to make them somewhat searchable:

          161026  0:15:30 [ERROR] mysqld got signal 11 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.
           
          To report this bug, see https://mariadb.com/kb/en/reporting-bugs
           
          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed, 
          something is definitely wrong and this may fail.
           
          Server version: 10.1.14-MariaDB-1~trusty
          key_buffer_size=16777216
          read_buffer_size=131072
          max_used_connections=16
          max_threads=602
          thread_count=6
          It is possible that mysqld could use up to 
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4865990 K  bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.
           
          Thread pointer: 0x0x7f8343c17008
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          stack_bottom = 0x7f841b060df0 thread_stack 0x30000
          /home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2)
          : No such file or directory
          /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ab5fed4b5e]
          Backtrace: (Note: toku_do_assert=0x0x7f841b11ecb0)
          /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd]
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f841b18065b]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7f841b1807e3]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7f841b11e8d7]
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z24toku_thread_pool_destroyPP16toku_thread_pool+0x1bb)[0x7f841b161d7b]
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z31toku_ft_serialize_layer_destroyv+0x10)[0x7f841b1650b0]
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z21toku_ft_layer_destroyv+0x20)[0x7f841b1650e0]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x40199)[0x7f841b0a2199]
          /lib64/ld-linux-x86-64.so.2(+0x1070a)[0x7f8420b3b70a]
          /lib/x86_64-linux-gnu/libc.so.6(+0x3c1a9)[0x7f841ecaf1a9]
          /lib/x86_64-linux-gnu/libc.so.6(+0x3c1f5)[0x7f841ecaf1f5]
          /usr/sbin/mysqld(+0xa9893c)[0x55ab5fee893c]
          /usr/sbin/mysqld(my_print_stacktrace+0x4c)[0x55ab5fed4b7c]
          /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330]
          /usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b]
          /usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664]
          /usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94]
          /usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09]
          /usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135]
          /usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66]
          /usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180]
          /usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3]
          /usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7]
          /usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873]
          /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590]
          /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12]
          /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330]
          /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66]
          /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d]
          /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0]
          /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89]
          /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a]
          /usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184]
          /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]
          Engine status:
          BUILD_ID = 0
          Env panic code: 2
          Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2)
          

          Memory usage:
          Arena 0:
          system bytes     =          0
          in use bytes     =          0
          Total (incl. mmap):
          system bytes     =          0
          in use bytes     =          0
          max mmap regions =          0
          max mmap bytes   =          0
          Fatal signal 6 while backtracing
          /usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b]
          /usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664]
          /usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94]
          /usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09]
          /usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135]
          /usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66]
          /usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180]
          /usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3]
          /usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7]
          /usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873]
          /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590]
          /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12]
          /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a]
          /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66]
          /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d]
          /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0]
          /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89]
          /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a]
          /usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184]
          /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]
           
          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (0x7f83bc579020): is an invalid pointer
          Connection ID (thread ID): 8009371
          Status: NOT_KILLED
          

          /home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0)
          : Success
          Backtrace: (Note: toku_do_assert=0x0x7fe9c38e1cb0)
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fe9c394365b]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7fe9c39437e3]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981]
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad]
          /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f]
          /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66]
          /usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8]
          /usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071]
          /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640]
          /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c]
          /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e]
          /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6]
          /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd]
          /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20]
          /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9]
          /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea]
          /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182]
          /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]
          Engine status:
          BUILD_ID = 0
          Env panic code: 18446744073709551615
          Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0)
          

          161103  0:15:05 [ERROR] mysqld got signal 6 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.
           
          To report this bug, see http://kb.askmonty.org/en/reporting-bugs
           
          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed, 
          something is definitely wrong and this may fail.
           
          Server version: 10.1.11-MariaDB-1~trusty
          key_buffer_size=16777216
          read_buffer_size=131072
          max_used_connections=21
          max_threads=153
          thread_count=7
          It is possible that mysqld could use up to 
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1248906 K  bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.
           
          Thread pointer: 0x0x7fe94efec008
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          stack_bottom = 0x7fe9c85fcdf0 thread_stack 0x30000
          /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fe9c80463fe]
          /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7fe9c7b73f9d]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fe9c60cc340]
          /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fe9c5723cc9]
          /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9c57270d8]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e8)[0x7fe9c39437e8]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981]
          /usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9]
          /usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad]
          /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f]
          /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66]
          /usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8]
          /usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071]
          /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640]
          /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c]
          /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e]
          /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6]
          /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd]
          /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20]
          /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9]
          /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea]
          /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0]
          /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182]
          /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]
           
          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (0x7fe941020020): is an invalid pointer
          Connection ID (thread ID): 687376
          Status: NOT_KILLED
          

          elenst Elena Stepanova added a comment - I guess we'll have to wait till somebody else complains about a similar issue. For now, I'll put here the stack traces from attached error logs, to make them somewhat searchable: 161026 0:15:30 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.14-MariaDB-1~trusty key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=16 max_threads=602 thread_count=6 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4865990 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x7f8343c17008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f841b060df0 thread_stack 0x30000 /home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2) : No such file or directory /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55ab5fed4b5e] Backtrace: (Note: toku_do_assert=0x0x7f841b11ecb0) /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd] /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f841b18065b] /usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7f841b1807e3] /usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7f841b11e8d7] /usr/lib/mysql/plugin/ha_tokudb.so(_Z24toku_thread_pool_destroyPP16toku_thread_pool+0x1bb)[0x7f841b161d7b] /usr/lib/mysql/plugin/ha_tokudb.so(_Z31toku_ft_serialize_layer_destroyv+0x10)[0x7f841b1650b0] /usr/lib/mysql/plugin/ha_tokudb.so(_Z21toku_ft_layer_destroyv+0x20)[0x7f841b1650e0] /usr/lib/mysql/plugin/ha_tokudb.so(+0x40199)[0x7f841b0a2199] /lib64/ld-linux-x86-64.so.2(+0x1070a)[0x7f8420b3b70a] /lib/x86_64-linux-gnu/libc.so.6(+0x3c1a9)[0x7f841ecaf1a9] /lib/x86_64-linux-gnu/libc.so.6(+0x3c1f5)[0x7f841ecaf1f5] /usr/sbin/mysqld(+0xa9893c)[0x55ab5fee893c] /usr/sbin/mysqld(my_print_stacktrace+0x4c)[0x55ab5fed4b7c] /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55ab5fa00abd] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330] /usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b] /usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664] /usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94] /usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09] /usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135] /usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66] /usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180] /usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3] /usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7] /usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873] /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590] /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12] /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f841f856330] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0] /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a] /usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d] Engine status: BUILD_ID = 0 Env panic code: 2 Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.14/storage/tokudb/PerconaFT/util/threadpool.cc:112 toku_thread_destroy: Assertion `r == 0 && ret == thread' failed (errno=2) Memory usage: Arena 0: system bytes = 0 in use bytes = 0 Total (incl. mmap): system bytes = 0 in use bytes = 0 max mmap regions = 0 max mmap bytes = 0 Fatal signal 6 while backtracing /usr/sbin/mysqld(+0xa5df0b)[0x55ab5feadf0b] /usr/sbin/mysqld(_ZN11TABLE_SHARE7destroyEv+0x24)[0x55ab5f922664] /usr/sbin/mysqld(+0x562d94)[0x55ab5f9b2d94] /usr/sbin/mysqld(_Z17tdc_release_shareP11TABLE_SHARE+0x269)[0x55ab5f9b3a09] /usr/sbin/mysqld(_Z8closefrmP5TABLEb+0x155)[0x55ab5f923135] /usr/sbin/mysqld(_Z18intern_close_tableP5TABLE+0x36)[0x55ab5f838b66] /usr/sbin/mysqld(_Z16tc_release_tableP5TABLE+0x190)[0x55ab5f9b3180] /usr/sbin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xf3)[0x55ab5f838ef3] /usr/sbin/mysqld(_Z25close_all_tables_for_nameP3THDP11TABLE_SHARE17ha_extra_functionP5TABLE+0x117)[0x55ab5f83a2c7] /usr/sbin/mysqld(+0x378873)[0x55ab5f7c8873] /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x7f0)[0x55ab5fb3a590] /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1352)[0x55ab5f90ae12] /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55ab5f951f7a] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55ab5f87fa66] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55ab5f88808d] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55ab5f88b3d0] /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55ab5f88bb89] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55ab5f94f24a] /usr/sbin/mysqld(handle_one_connection+0x40)[0x55ab5f94f420] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f841f84e184] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f841ed6d37d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f83bc579020): is an invalid pointer Connection ID (thread ID): 8009371 Status: NOT_KILLED /home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0) : Success Backtrace: (Note: toku_do_assert=0x0x7fe9c38e1cb0) /usr/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fe9c394365b] /usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e3)[0x7fe9c39437e3] /usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7] /usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d] /usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c] /usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981] /usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049] /usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9] /usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad] /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f] /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66] /usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8] /usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071] /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640] /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c] /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20] /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea] /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d] Engine status: BUILD_ID = 0 Env panic code: 18446744073709551615 Env panic string: /home/buildbot/buildbot/build/mariadb-10.1.11/storage/tokudb/PerconaFT/locktree/treenode.cc:74 destroy_root: Assertion `is_empty()' failed (errno=0) 161103 0:15:05 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.11-MariaDB-1~trusty key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=21 max_threads=153 thread_count=7 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1248906 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x7fe94efec008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fe9c85fcdf0 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fe9c80463fe] /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7fe9c7b73f9d] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fe9c60cc340] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7fe9c5723cc9] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe9c57270d8] /usr/lib/mysql/plugin/ha_tokudb.so(+0x11e7e8)[0x7fe9c39437e8] /usr/lib/mysql/plugin/ha_tokudb.so(+0xbc8d7)[0x7fe9c38e18d7] /usr/lib/mysql/plugin/ha_tokudb.so(+0x3f64d)[0x7fe9c386464d] /usr/lib/mysql/plugin/ha_tokudb.so(+0x10000c)[0x7fe9c392500c] /usr/lib/mysql/plugin/ha_tokudb.so(+0xb9981)[0x7fe9c38de981] /usr/lib/mysql/plugin/ha_tokudb.so(_Z13toku_db_closeP9__toku_db+0x59)[0x7fe9c3936049] /usr/lib/mysql/plugin/ha_tokudb.so(+0x1110a9)[0x7fe9c39360a9] /usr/lib/mysql/plugin/ha_tokudb.so(+0x4a6ad)[0x7fe9c386f6ad] /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb7__closeEv+0x10f)[0x7fe9c386fa3f] /usr/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb5closeEv+0x26)[0x7fe9c386fb66] /usr/sbin/mysqld(+0xa4e0f8)[0x7fe9c80130f8] /usr/sbin/mysqld(+0x6df071)[0x7fe9c7ca4071] /usr/sbin/mysqld(_Z26fast_alter_partition_tableP3THDP5TABLEP10Alter_infoP14HA_CREATE_INFOP10TABLE_LISTPcPKc+0x8d0)[0x7fe9c7cad640] /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1c9c)[0x7fe9c7a7fd6c] /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x62e)[0x7fe9c7ac602e] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1226)[0x7fe9c79f41f6] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x20d)[0x7fe9c79fc6dd] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fe9c79ffa20] /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fe9c7a001c9] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fe9c7ac32ea] /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fe9c7ac34c0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fe9c60c4182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe9c57e747d]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fe941020020): is an invalid pointer Connection ID (thread ID): 687376 Status: NOT_KILLED

          People

            Unassigned Unassigned
            akrus akrus
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.