[MDEV-9572] TokuDB plugin crash Created: 2016-02-17  Updated: 2016-06-22  Resolved: 2016-06-22

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Affects Version/s: 10.0.23
Fix Version/s: 10.0.26, 10.1.15

Type: Bug Priority: Major
Reporter: Karl E. Jørgensen Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: mariadb, tokudb
Environment:

Centos7, 64bit

Attachments: Text File crash-during-shutdown.txt     Text File crash.txt     Text File recovery.txt    
Issue Links:
PartOf
is part of MDEV-10224 10.0.26 merge Closed
Sprint: 10.0.26

 Description   

We had a crash - judging from the stack trace it appears to be TokuDB related:

/home/buildbot/buildbot/build/mariadb-10.0.23/storage/tokudb/PerconaFT/locktree/treenode.cc:338 remove: Assertion `child' failed (errno=2)
 
: No such file or directory
 
Backtrace: (Note: toku_do_assert=0x0x7fd2e2c93a30)
 
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fd2e2c901fb]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x8e323)[0x7fd2e2c90323]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x8e39a)[0x7fd2e2c9039a]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x117d17)[0x7fd2e2d19d17]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x117c1e)[0x7fd2e2d19c1e]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x117f6e)[0x7fd2e2d19f6e]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x119342)[0x7fd2e2d1b342]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x112635)[0x7fd2e2d14635]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x11292f)[0x7fd2e2d1492f]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x1133f5)[0x7fd2e2d153f5]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x1136eb)[0x7fd2e2d156eb]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z24toku_db_start_range_lockP9__toku_dbP13__toku_db_txnPK10__toku_dbtS5_N4toku12lock_request4typeEPS7_+0xdc)[0x7fd2e2d1ea6c]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z22toku_db_get_range_lockP9__toku_dbP13__toku_db_txnPK10__toku_dbtS5_N4toku12lock_request4typeE+0xe6)[0x7fd2e2d1ebf6]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z19env_update_multipleP13__toku_db_envP9__toku_dbP13__toku_db_txnP10__toku_dbtS6_S6_S6_jPS2_PjjP9DBT_ARRAYjSA_+0x838)[0x7fd2e2d241a8]
 
/usr/lib64/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb10update_rowEPKhPh+0x572)[0x7fd2e2c6b8a2]
 
/usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0x412)[0x745172]
 
/usr/sbin/mysqld(_ZN12multi_update9send_dataER4ListI4ItemE+0x2f9)[0x66c419]
 
/usr/sbin/mysqld[0x5fa69d]
 
/usr/sbin/mysqld[0x5fc9d7]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x176)[0x60b346]
 
/usr/sbin/mysqld[0x5fc9d7]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1bc)[0x60b38c]
 
/usr/sbin/mysqld[0x61686d]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa3a)[0x627caa]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x11)[0x629af1]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xc7)[0x6264f7]
 
/usr/sbin/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_select_lexPP12multi_update+0x1af)[0x66ea9f]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4e81)[0x5d6701]
 
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x34)[0x8527d4]
 
/usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x80)[0x859c40]
 
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x155)[0x85a065]
 
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x6be)[0x856d8e]
 
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x664)[0x858374]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1d41)[0x5d35c1]
 
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x34)[0x8527d4]
 
/usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x80)[0x859c40]
 
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x155)[0x85a065]
 
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x6be)[0x856d8e]
 
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x664)[0x858374]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1d41)[0x5d35c1]
 
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x34)[0x8527d4]
 
/usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x80)[0x859c40]
 
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x155)[0x85a065]
 
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x6be)[0x856d8e]
 
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x664)[0x858374]
 
/usr/sbin/mysqld(_ZN14Event_job_data7executeEP3THDb+0x799)[0x68dc79]
 
/usr/sbin/mysqld(event_worker_thread+0x109)[0x88de39]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7fd2ff446dc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7fd2fdcc321d]

The full log is attached in crash.txt and the subsequent recovery in recovery.txt

 Comments   
Comment by Karl E. Jørgensen [ 2016-02-17 ]

FYI: This seems to have been introduced since mariadb-10.0.21 : A previous incarnation of this server ran quite happily on that MariaDB version. Since it was switched to 10.0.23 we have had a couple of crashes (we skipped version 10.0.22).

Comment by Sergei Golubchik [ 2016-02-18 ]

Quite possible. MariaDB 10.0.21 had TokuDB 7.5.7.
MariaDB 10.0.23 uses TokuDB 5.6.26-74.0.

That is 10.0.21 still had the "old" TokuDB from Tokutek, it even used the Tokutek versioning scheme.
10.0.23 has the "new" TokuDB from Percona Server, it uses the Percona Server versioning.

Percona is doing quite a lot of changes in TokuDB, bugs are possible. But they are eager to fix them quickly.
Unfortunately, TokuDB from 5.6.27-76.0 and from 5.6.28-76.1 didn't work at all in MariaDB.
As soon as a new usable TokuDB version will be released, we'll put it in MariaDB at once.

Comment by Karl E. Jørgensen [ 2016-02-18 ]

When shutting down this instance, it crashed again - log attached in crash-during-shutdown.txt.

Annoyingly, the shutdown (using service mysql shutdown) claimed to be successful. But then mysqld_safe got involved and started it back up again!

Generated at Thu Feb 08 07:35:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.