[MDEV-5594] If TOKUDB crashes, no other engine starts, MariaDB does not start ! Created: 2014-01-30  Updated: 2014-04-07  Due: 2014-02-10  Resolved: 2014-04-07

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.0.7
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Joni-Pekka Kurronen Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: tokudb
Environment:

Ubuntu 12.4 LTS
MariaDB 10.0.7


 Description   

If TokuDB has at start crash you can not get MariaDB
up nor reapaisr TokuDB to get MariaDB up.

MariaDB could go up whit all other engines even tokuDB
or what so ever dose not start.

What I tried was to copy /var/lib/mysql and /media/SDD1/tokudb (tokudb database at sdd) to replication server /var/lib/mysql and /media/SDD2/tokudb (tokudb database at sdd, name is difrent at every disk sdd1, sdd2,... ext)

root@mpi1:/media/SDD1/tokudb# /opt/mariadb-10.0.7/bin/mysqld
 
140130 18:00:04 [Warning] option 'table_definition_cache': unsigned value 100 adjusted to 400
 
2014-01-30 18:00:04 7f33b133b780 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
 
140130 18:00:04 [Note] InnoDB: The InnoDB memory heap is disabled
 
140130 18:00:04 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
140130 18:00:04 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
 
140130 18:00:04 [Note] InnoDB: CPU does not support crc32 instructions
 
140130 18:00:04 [Note] InnoDB: Initializing buffer pool, size = 8.0M
 
140130 18:00:04 [Note] InnoDB: Completed initialization of buffer pool
 
140130 18:00:04 [Note] InnoDB: Highest supported file format is Barracuda.
 
140130 18:00:04 [Note] InnoDB: 128 rollback segment(s) are active.
 
140130 18:00:04 [Note] InnoDB: Waiting for purge to start
 
140130 18:00:04 [Note] InnoDB: 5.6.10 started; log sequence number 7062765079
 
140130 18:00:04 [Note] CONNECT: connect.ini=/var/lib/mysql/connect.ini
 
profile file /var/lib/mysql/connect.ini not found
 
Thu Jan 30 18:00:04 2014 Tokudb recovery starting in env /var/lib/mysql/
 
Thu Jan 30 18:00:04 2014 Tokudb recovery scanning backward from 17775
 
Thu Jan 30 18:00:04 2014 Tokudb recovery bw_end_checkpoint at 17775 timestamp 1391095211182665 xid 17771 (bw_newer)
 
Thu Jan 30 18:00:04 2014 Tokudb recovery bw_begin_checkpoint at 17771 timestamp 1391095211182654 (bw_between)
 
Thu Jan 30 18:00:04 2014 Tokudb recovery turning around at begin checkpoint 17771 time 11
 
Thu Jan 30 18:00:04 2014 Tokudb recovery starts scanning forward to 17775 from 17771 left 4 (fw_between)
 
/mpi3/S3/mariadb-10.0.7/storage/tokudb/ft-index/ft/ft-serialize.cc:712 toku_deserialize_ft_from: Assertion `!((r0==0 && checkpoint_lsn_0.lsn > max_acceptable_lsn.lsn) && (r1==0 && checkpoint_lsn_1.lsn > max_acceptable_lsn.lsn))' failed (errno=2)
 
: No such file or directory
 
Backtrace: (Note: toku_do_assert=0x0x7f33946d7120)
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c718)[0x7f33946d4718]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c8a0)[0x7f33946d48a0]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z24toku_deserialize_ft_fromi10__toku_lsnPP2ft+0x37b)[0x7f33947090db]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z35toku_read_ft_and_store_in_cachefileP9ft_handleP9cachefile10__toku_lsnPP2ft+0x90)[0x7f3394709210]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xb8c4e)[0x7f33946f0c4e]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z28toku_ft_handle_open_recoveryP9ft_handlePKciiP10cachetableP7tokutxn7FILENUM10__toku_lsn+0x28)[0x7f33946f1178]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z14tokudb_recoverP13__toku_db_envPFvS0_P7tokutxnEPFvS0_P10cachetableEP10tokuloggerPKcSC_PFiP9__toku_dbPK10__toku_dbtSH_EPFiSE_SH_SH_SH_PFvSH_PvESK_EPFiSE_SE_P9DBT_ARRAYSQ_SH_SH_EPFiSE_SE_SQ_SH_SH_Em+0x2193)[0x7f33946fc2a3]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xc5d09)[0x7f33946fdd09]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x47162)[0x7f339467f162]
 
/opt/mariadb-10.0.7/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5b)[0x6dff9b]
 
/opt/mariadb-10.0.7/bin/mysqld[0x5b9170]
 
/opt/mariadb-10.0.7/bin/mysqld(_Z11plugin_initPiPPci+0x64b)[0x5bd79b]
 
/opt/mariadb-10.0.7/bin/mysqld[0x53b775]
 
/opt/mariadb-10.0.7/bin/mysqld(_Z11mysqld_mainiPPc+0x3da)[0x53df7a]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f33af6e176d]
 
/opt/mariadb-10.0.7/bin/mysqld[0x534279]
 
Engine status function not available
 
Memory usage:
 
Arena 0:
 
system bytes     =          0
 
in use bytes     =          0
 
Total (incl. mmap):
 
system bytes     =          0
 
in use bytes     =          0
 
max mmap regions =          0
 
max mmap bytes   =          0
 
140130 18:00:04 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.7-MariaDB-log
 
key_buffer_size=16777216
 
read_buffer_size=1048576
 
max_used_connections=0
 
max_threads=102
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 227356 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x40000
 
mysys/stacktrace.c:247(my_print_stacktrace)[0xa7b526]
 
sql/signal_handler.cc:153(handle_fatal_signal)[0x6ddd88]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f33b08dfcb0]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f33af6f6425]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f33af6f9b8b]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c81b)[0x7f33946d481b]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c8a0)[0x7f33946d48a0]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z24toku_deserialize_ft_fromi10__toku_lsnPP2ft+0x37b)[0x7f33947090db]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z35toku_read_ft_and_store_in_cachefileP9ft_handleP9cachefile10__toku_lsnPP2ft+0x90)[0x7f3394709210]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xb8c4e)[0x7f33946f0c4e]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z28toku_ft_handle_open_recoveryP9ft_handlePKciiP10cachetableP7tokutxn7FILENUM10__toku_lsn+0x28)[0x7f33946f1178]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z14tokudb_recoverP13__toku_db_envPFvS0_P7tokutxnEPFvS0_P10cachetableEP10tokuloggerPKcSC_PFiP9__toku_dbPK10__toku_dbtSH_EPFiSE_SH_SH_SH_PFvSH_PvESK_EPFiSE_SE_P9DBT_ARRAYSQ_SH_SH_EPFiSE_SE_SQ_SH_SH_Em+0x2193)[0x7f33946fc2a3]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xc5d09)[0x7f33946fdd09]
 
/opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x47162)[0x7f339467f162]
 
sql/handler.cc:497(ha_initialize_handlerton(st_plugin_int*))[0x6dff9b]
 
sql/sql_plugin.cc:1358(plugin_initialize)[0x5b9170]
 
sql/sql_plugin.cc:1650(plugin_init(int*, char**, int))[0x5bd79b]
 
sql/mysqld.cc:4747(init_server_components)[0x53b775]
 
sql/mysqld.cc:5344(mysqld_main(int, char**))[0x53df7a]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f33af6e176d]
 
/opt/mariadb-10.0.7/bin/mysqld[0x534279]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
root@mpi1:/media/SDD1/tokudb#



 Comments   
Comment by Sergei Golubchik [ 2014-02-03 ]

There are two issues here.

1. TokuDB crash. TokuDB developers will comment on that.

2. MariaDB cannot continue. Currently plugins run in the server address space. To isolate plugin crashes, MariaDB would need to run every plugin in a sandbox, so that a plugin failure wouldn't affect the whole server. We currently have no plans of doing it, it would be a big a complex feature, and most probably it would negatively impact the performance of the server.

Comment by Sergei Golubchik [ 2014-02-03 ]

prohaska7, could you please comment on the TokuDB plugin crash above?

Comment by Joni-Pekka Kurronen [ 2014-02-03 ]

Reason for crash, how I got start up whitout crash:

before locking tables and get master status
I stopped slave and reseted it.

After that I made rsync copy. Directory names exaltly
like master for tokudb.

Then I could start whitout errors.

Comment by Sergei Golubchik [ 2014-04-07 ]

The bug as reported "if tokudb crashes, it takes the whole mysqld process down" will not be fixed anytime soon.

Generated at Thu Feb 08 07:05:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.