[MDEV-25246] MariaDB 10.4.18 replication after error Created: 2021-03-25  Updated: 2021-03-25  Resolved: 2021-03-25

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Affects Version/s: 10.4.18
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Na Yun Ho Assignee: Alice Sherepa
Resolution: Duplicate Votes: 0
Labels: None
Environment:

RHEL7.9


Issue Links:
Duplicate
is duplicated by MDEV-15174 Server crash in toku_memory_startup o... Open

 Description   

MariaDB Replication Error
Master Server

Log

2021-03-24 19:18:12 29 [Warning] Timeout waiting for reply of binlog (file: mariadb-bin.000001, pos: 447), semi-sync up to file , position 0.
2021-03-24 19:18:12 29 [Note] Semi-sync replication switched OFF.
2021-03-24 19:18:12 51 [Note] Start binlog_dump to slave_server(2), pos(, 4), using_gtid(1), gtid('')
2021-03-24 19:18:12 51 [Note] Start semi-sync binlog_dump to slave (server_id: 2), pos(/data/mariadb/log/mariadb-bin.000001, 4)
2021-03-24 19:18:12 6 [Note] Semi-sync replication switched ON with slave (server_id: 2) at (mariadb-bin.000001, 447)
/home/buildbot/buildbot/build/storage/tokudb/PerconaFT/portability/memory.cc:119 toku_memory_startup: Assertion `result == 0' failed (errno=2)
: No such file or directory
Backtrace: (Note: toku_do_assert=0x0x7f0ae59296f0)
/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f0ae592766b]
/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0xab7a3)[0x7f0ae59277a3]
/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0xab827)[0x7f0ae5927827]
/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0x5f41c)[0x7f0ae58db41c]
/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0x5f4f5)[0x7f0ae58db4f5]
/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f1b87d369c3]
/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f1b87d3b59e]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f1b87d367d4]
/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f1b87d3ab8b]
/lib64/libdl.so.2(+0xfab)[0x7f1b872c6fab]
/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f1b87d367d4]
/lib64/libdl.so.2(+0x15ad)[0x7f1b872c75ad]
/lib64/libdl.so.2(dlopen+0x31)[0x7f1b872c7041]
/opt/mariadb/mariadb/bin/mysqld(+0x741c76)[0x555c18d4fc76]
/opt/mariadb/mariadb/bin/mysqld(_Z17plugin_dl_foreachP3THDPK25st_mysql_const_lex_stringPFcS0_P13st_plugin_intPvES6_+0x30)[0x555c18d515a0]
/opt/mariadb/mariadb/bin/mysqld(_Z16fill_all_pluginsP3THDP10TABLE_LISTP4Item+0x10f)[0x555c18db548f]
/opt/mariadb/mariadb/bin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2bc)[0x555c18dbb6cc]
/opt/mariadb/mariadb/bin/mysqld(_ZN4JOIN10exec_innerEv+0x995)[0x555c18da06f5]
/opt/mariadb/mariadb/bin/mysqld(_ZN4JOIN4execEv+0x33)[0x555c18da0c03]
/opt/mariadb/mariadb/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x555c18d9ee36]
/opt/mariadb/mariadb/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1e7)[0x555c18d9f9b7]
/opt/mariadb/mariadb/bin/mysqld(+0x6257ff)[0x555c18c337ff]
/opt/mariadb/mariadb/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3b24)[0x555c18d43cb4]
/opt/mariadb/mariadb/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x34c)[0x555c18d4959c]
/opt/mariadb/mariadb/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1dfc)[0x555c18d4bb5c]
/opt/mariadb/mariadb/bin/mysqld(_Z10do_commandP3THD+0x112)[0x555c18d4d372]
/opt/mariadb/mariadb/bin/mysqld(_Z11tp_callbackP13TP_connection+0x1df)[0x555c18f2360f]
/opt/mariadb/mariadb/bin/mysqld(+0xad3040)[0x555c190e1040]
/lib64/libpthread.so.0(+0x7ea5)[0x7f1b87b12ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f1b869ec96d]
Engine status function not available
Memory usage:
Arena 0:
system bytes     = 1392422912
in use bytes     = 1391673856
Arena 1:
system bytes     =     368640
in use bytes     =       2448
Arena 2:
system bytes     =     307200
in use bytes     =      93296
Arena 3:
system bytes     =     299008
in use bytes     =      92224
Arena 4:
system bytes     =     307200
in use bytes     =       2448
Arena 5:
system bytes     =     352256
in use bytes     =       2496
Arena 6:
system bytes     =     352256
in use bytes     =       2496
Arena 7:
system bytes     =     352256
in use bytes     =       2496
Arena 8:
system bytes     =     352256
in use bytes     =       2496
Arena 9:
system bytes     =     360448
in use bytes     =       6864
Arena 10:
system bytes     =     352256
in use bytes     =       2496
Arena 11:
system bytes     =     352256
in use bytes     =       2496
Arena 12:
system bytes     =     135168
in use bytes     =       2224
Arena 13:
system bytes     =     135168
in use bytes     =       2224
Arena 14:
system bytes     =     135168
in use bytes     =       2224
Arena 15:
system bytes     =     135168
in use bytes     =       2224
Arena 16:
system bytes     =     135168
in use bytes     =       2224
Arena 17:
system bytes     =    2007040
in use bytes     =    1066624
Arena 18:
system bytes     =     352256
in use bytes     =       3088
Arena 19:
system bytes     =    4239360
in use bytes     =     579472
Arena 20:
system bytes     =    7012352
in use bytes     =    1793984
Arena 21:
system bytes     =    5779456
in use bytes     =     946704
Arena 22:
system bytes     =    3485696
in use bytes     =    2420224
Arena 23:
system bytes     =     352256
in use bytes     =       2496
Arena 24:
system bytes     =     360448
in use bytes     =       2496
Arena 25:
system bytes     =     352256
in use bytes     =       2496
Arena 26:
system bytes     =     352256
in use bytes     =     109664
Arena 27:
system bytes     =     360448
in use bytes     =      11296
Arena 28:
system bytes     =     360448
in use bytes     =       2496
Arena 29:
system bytes     =     360448
in use bytes     =       2496
Arena 30:
system bytes     =     352256
in use bytes     =       2496
Arena 31:
system bytes     =     372736
in use bytes     =      12480
Arena 32:
system bytes     =     352256
in use bytes     =       2496
Arena 33:
system bytes     =     352256
in use bytes     =       2496
Arena 34:
system bytes     =     360448
in use bytes     =       2496
Arena 35:
system bytes     =     352256
in use bytes     =       2496
Arena 36:
system bytes     =     348160
in use bytes     =       4816
Arena 37:
system bytes     =     352256
in use bytes     =       2496
Arena 38:
system bytes     =     364544
in use bytes     =       4864
Arena 39:
system bytes     =     352256
in use bytes     =       2448
Arena 40:
system bytes     =     303104
in use bytes     =       5184
Arena 41:
system bytes     =     360448
in use bytes     =       2448
Arena 42:
system bytes     =     352256
in use bytes     =       4000
Arena 43:
system bytes     =     307200
in use bytes     =       4816
Arena 44:
system bytes     =     352256
in use bytes     =      92000
Arena 45:
system bytes     =     352256
in use bytes     =       2496
Arena 46:
system bytes     =    4755456
in use bytes     =     435504
Arena 47:
system bytes     =     352256
in use bytes     =       2496
Arena 48:
system bytes     =     360448
in use bytes     =       2496
Arena 49:
system bytes     =     360448
in use bytes     =       2496
Arena 50:
system bytes     =     364544
in use bytes     =       3120
Arena 51:
system bytes     =    2146304
in use bytes     =      23824
Arena 52:
system bytes     =     352256
in use bytes     =       2496
Arena 53:
system bytes     =     360448
in use bytes     =       2496
Arena 54:
system bytes     =     438272
in use bytes     =      85344
Arena 55:
system bytes     =     352256
in use bytes     =       2496
Arena 56:
system bytes     =     352256
in use bytes     =       2496
Arena 57:
system bytes     =     352256
in use bytes     =       2496
Arena 58:
system bytes     =     352256
in use bytes     =       2832
Arena 59:
system bytes     =     352256
in use bytes     =       2496
Arena 60:
system bytes     =     372736
in use bytes     =      12144
Arena 61:
system bytes     =     352256
in use bytes     =       3776
Arena 62:
system bytes     =    4816896
in use bytes     =      93648
Arena 63:
system bytes     =     364544
in use bytes     =       4352
Arena 64:
system bytes     =     135168
in use bytes     =       2448
Arena 65:
system bytes     =     135168
in use bytes     =      10928
Arena 66:
system bytes     =     135168
in use bytes     =       2448
Arena 67:
system bytes     =     135168
in use bytes     =      42256
Arena 68:
system bytes     =     135168
in use bytes     =      31440
Arena 69:
system bytes     =     135168
in use bytes     =      31408
Arena 70:
system bytes     =     135168
in use bytes     =       2448
Arena 71:
system bytes     =     135168
in use bytes     =       2448
Arena 72:
system bytes     =     135168
in use bytes     =      31552
Arena 73:
system bytes     =     135168
in use bytes     =      31552
Arena 74:
system bytes     =     135168
in use bytes     =      31552
Arena 75:
system bytes     =     163840
in use bytes     =      46448
Arena 76:
system bytes     =     135168
in use bytes     =       2448
Arena 77:
system bytes     =     135168
in use bytes     =       2448
Arena 78:
system bytes     =     135168
in use bytes     =       2512
Arena 79:
system bytes     =     135168
in use bytes     =       2224
Arena 80:
system bytes     =     135168
in use bytes     =       2448
Arena 81:
system bytes     =     135168
in use bytes     =       2448
Arena 82:
system bytes     =     135168
in use bytes     =       2448
Arena 83:
system bytes     =     135168
in use bytes     =       2448
Arena 84:
system bytes     =     135168
in use bytes     =       2704
Arena 85:
system bytes     =     135168
in use bytes     =       2448
Total (incl. mmap):
system bytes     = 3875291136
in use bytes     = 3827283984
max mmap regions =        154
max mmap bytes   = 3560165376
210325  6:48:49 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.18-MariaDB-log
key_buffer_size=67108864
read_buffer_size=2097152
max_used_connections=6
max_threads=1025
thread_count=11
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4313445 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f0a18000a88
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0b100efe10 thread_stack 0x49000
mysys/stacktrace.c:175(my_print_stacktrace)[0x555c1954f8ae]
sql/signal_handler.cc:209(handle_fatal_signal)[0x555c18f51dbf]
sigaction.c:0(__restore_rt)[0x7f1b87b1a630]
:0(__GI_raise)[0x7f1b86924387]
:0(__GI_abort)[0x7f1b86925a78]
portability/toku_assert.cc:147(toku_do_backtrace_abort() [clone .107925])[0x7f0ae59277a8]
portability/toku_assert.cc:168(toku_do_assert_fail(char const*, char const*, char const*, int, int))[0x7f0ae5927827]
/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0x5f41c)[0x7f0ae58db41c]
util/context.cc:45(__static_initialization_and_destruction_0)[0x7f0ae58db4f5]
:0(_dl_init_internal)[0x7f1b87d369c3]
dl-open.c:0(dl_open_worker)[0x7f1b87d3b59e]
:0(_dl_catch_error)[0x7f1b87d367d4]
:0(_dl_open)[0x7f1b87d3ab8b]
dlopen.c:0(dlopen_doit)[0x7f1b872c6fab]
:0(_dl_catch_error)[0x7f1b87d367d4]
:0(_dlerror_run)[0x7f1b872c75ad]
:0(__dlopen_check)[0x7f1b872c7041]
sql/sql_plugin.cc:752(plugin_dl_add(st_mysql_const_lex_string const*, unsigned long))[0x555c18d4fc76]
sql/sql_plugin.cc:2527(plugin_dl_foreach(THD*, st_mysql_const_lex_string const*, char (*)(THD*, st_plugin_int*, void*), void*))[0x555c18d515a0]
sql/sql_class.h:3908(THD::get_stmt_da())[0x555c18db548f]
sql/sql_show.cc:8934(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x555c18dbb6cc]
sql/sql_select.cc:4442(JOIN::exec_inner())[0x555c18da06f5]
sql/sql_select.cc:4269(JOIN::exec())[0x555c18da0c03]
sql/sql_select.cc:4705(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x555c18d9ee36]
sql/sql_select.cc:422(handle_select(THD*, LEX*, select_result*, unsigned long))[0x555c18d9f9b7]
sql/sql_parse.cc:6419(execute_sqlcom_select(THD*, TABLE_LIST*))[0x555c18c337ff]
sql/sql_parse.cc:3937(mysql_execute_command(THD*))[0x555c18d43cb4]
sql/sql_parse.cc:7959(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x555c18d4959c]
sql/sql_audit.h:169(mysql_audit_general)[0x555c18d4bb5c]
sql/sql_parse.cc:1374(do_command(THD*))[0x555c18d4d372]
sql/threadpool_common.cc:373(threadpool_process_request)[0x555c18f2360f]
sql/threadpool_generic.cc:1605(worker_main(void*))[0x555c190e1040]
pthread_create.c:0(start_thread)[0x7f1b87b12ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f1b869ec96d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f0a1800ff50): SHOW plugins soname
 
Connection ID (thread ID): 21834
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /data/mariadb/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             65536                65536                processes 
Max open files            67567                67567                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       512538               512538               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
210325 06:48:50 mysqld_safe Number of processes running now: 0
210325 06:48:50 mysqld_safe mysqld restarted



 Comments   
Comment by Sergei Golubchik [ 2021-03-25 ]

It's a duplicate of MDEV-15174. As a workaround you can delete ha_tokudb.so if you don't need it or install MariaDB from RPM packages. Otherwise you can watch MDEV-15174 to get a notification when it's fixed.

Generated at Thu Feb 08 09:36:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.