MariaDB 10.4.18 replication after error

2021-03-24 19:18:12 29 [Warning] Timeout waiting for reply of binlog (file: mariadb-bin.000001, pos: 447), semi-sync up to file , position 0.

2021-03-24 19:18:12 29 [Note] Semi-sync replication switched OFF.

2021-03-24 19:18:12 51 [Note] Start binlog_dump to slave_server(2), pos(, 4), using_gtid(1), gtid('')

2021-03-24 19:18:12 51 [Note] Start semi-sync binlog_dump to slave (server_id: 2), pos(/data/mariadb/log/mariadb-bin.000001, 4)

2021-03-24 19:18:12 6 [Note] Semi-sync replication switched ON with slave (server_id: 2) at (mariadb-bin.000001, 447)

/home/buildbot/buildbot/build/storage/tokudb/PerconaFT/portability/memory.cc:119 toku_memory_startup: Assertion `result == 0' failed (errno=2)

: No such file or directory

Backtrace: (Note: toku_do_assert=0x0x7f0ae59296f0)

/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7f0ae592766b]

/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0xab7a3)[0x7f0ae59277a3]

/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0xab827)[0x7f0ae5927827]

/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0x5f41c)[0x7f0ae58db41c]

/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0x5f4f5)[0x7f0ae58db4f5]

/lib64/ld-linux-x86-64.so.2(+0xf9c3)[0x7f1b87d369c3]

/lib64/ld-linux-x86-64.so.2(+0x1459e)[0x7f1b87d3b59e]

/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f1b87d367d4]

/lib64/ld-linux-x86-64.so.2(+0x13b8b)[0x7f1b87d3ab8b]

/lib64/libdl.so.2(+0xfab)[0x7f1b872c6fab]

/lib64/ld-linux-x86-64.so.2(+0xf7d4)[0x7f1b87d367d4]

/lib64/libdl.so.2(+0x15ad)[0x7f1b872c75ad]

/lib64/libdl.so.2(dlopen+0x31)[0x7f1b872c7041]

/opt/mariadb/mariadb/bin/mysqld(+0x741c76)[0x555c18d4fc76]

/opt/mariadb/mariadb/bin/mysqld(_Z17plugin_dl_foreachP3THDPK25st_mysql_const_lex_stringPFcS0_P13st_plugin_intPvES6_+0x30)[0x555c18d515a0]

/opt/mariadb/mariadb/bin/mysqld(_Z16fill_all_pluginsP3THDP10TABLE_LISTP4Item+0x10f)[0x555c18db548f]

/opt/mariadb/mariadb/bin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2bc)[0x555c18dbb6cc]

/opt/mariadb/mariadb/bin/mysqld(_ZN4JOIN10exec_innerEv+0x995)[0x555c18da06f5]

/opt/mariadb/mariadb/bin/mysqld(_ZN4JOIN4execEv+0x33)[0x555c18da0c03]

/opt/mariadb/mariadb/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x555c18d9ee36]

/opt/mariadb/mariadb/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1e7)[0x555c18d9f9b7]

/opt/mariadb/mariadb/bin/mysqld(+0x6257ff)[0x555c18c337ff]

/opt/mariadb/mariadb/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3b24)[0x555c18d43cb4]

/opt/mariadb/mariadb/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x34c)[0x555c18d4959c]

/opt/mariadb/mariadb/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1dfc)[0x555c18d4bb5c]

/opt/mariadb/mariadb/bin/mysqld(_Z10do_commandP3THD+0x112)[0x555c18d4d372]

/opt/mariadb/mariadb/bin/mysqld(_Z11tp_callbackP13TP_connection+0x1df)[0x555c18f2360f]

/opt/mariadb/mariadb/bin/mysqld(+0xad3040)[0x555c190e1040]

/lib64/libpthread.so.0(+0x7ea5)[0x7f1b87b12ea5]

/lib64/libc.so.6(clone+0x6d)[0x7f1b869ec96d]

Engine status function not available

Memory usage:

Arena 0:

system bytes     = 1392422912

in use bytes     = 1391673856

Arena 1:

system bytes     =     368640

in use bytes     =       2448

Arena 2:

system bytes     =     307200

in use bytes     =      93296

Arena 3:

system bytes     =     299008

in use bytes     =      92224

Arena 4:

system bytes     =     307200

in use bytes     =       2448

Arena 5:

system bytes     =     352256

in use bytes     =       2496

Arena 6:

system bytes     =     352256

in use bytes     =       2496

Arena 7:

system bytes     =     352256

in use bytes     =       2496

Arena 8:

system bytes     =     352256

in use bytes     =       2496

Arena 9:

system bytes     =     360448

in use bytes     =       6864

Arena 10:

system bytes     =     352256

in use bytes     =       2496

Arena 11:

system bytes     =     352256

in use bytes     =       2496

Arena 12:

system bytes     =     135168

in use bytes     =       2224

Arena 13:

system bytes     =     135168

in use bytes     =       2224

Arena 14:

system bytes     =     135168

in use bytes     =       2224

Arena 15:

system bytes     =     135168

in use bytes     =       2224

Arena 16:

system bytes     =     135168

in use bytes     =       2224

Arena 17:

system bytes     =    2007040

in use bytes     =    1066624

Arena 18:

system bytes     =     352256

in use bytes     =       3088

Arena 19:

system bytes     =    4239360

in use bytes     =     579472

Arena 20:

system bytes     =    7012352

in use bytes     =    1793984

Arena 21:

system bytes     =    5779456

in use bytes     =     946704

Arena 22:

system bytes     =    3485696

in use bytes     =    2420224

Arena 23:

system bytes     =     352256

in use bytes     =       2496

Arena 24:

system bytes     =     360448

in use bytes     =       2496

Arena 25:

system bytes     =     352256

in use bytes     =       2496

Arena 26:

system bytes     =     352256

in use bytes     =     109664

Arena 27:

system bytes     =     360448

in use bytes     =      11296

Arena 28:

system bytes     =     360448

in use bytes     =       2496

Arena 29:

system bytes     =     360448

in use bytes     =       2496

Arena 30:

system bytes     =     352256

in use bytes     =       2496

Arena 31:

system bytes     =     372736

in use bytes     =      12480

Arena 32:

system bytes     =     352256

in use bytes     =       2496

Arena 33:

system bytes     =     352256

in use bytes     =       2496

Arena 34:

system bytes     =     360448

in use bytes     =       2496

Arena 35:

system bytes     =     352256

in use bytes     =       2496

Arena 36:

system bytes     =     348160

in use bytes     =       4816

Arena 37:

system bytes     =     352256

in use bytes     =       2496

Arena 38:

system bytes     =     364544

in use bytes     =       4864

Arena 39:

system bytes     =     352256

in use bytes     =       2448

Arena 40:

system bytes     =     303104

in use bytes     =       5184

Arena 41:

system bytes     =     360448

in use bytes     =       2448

Arena 42:

system bytes     =     352256

in use bytes     =       4000

Arena 43:

system bytes     =     307200

in use bytes     =       4816

Arena 44:

system bytes     =     352256

in use bytes     =      92000

Arena 45:

system bytes     =     352256

in use bytes     =       2496

Arena 46:

system bytes     =    4755456

in use bytes     =     435504

Arena 47:

system bytes     =     352256

in use bytes     =       2496

Arena 48:

system bytes     =     360448

in use bytes     =       2496

Arena 49:

system bytes     =     360448

in use bytes     =       2496

Arena 50:

system bytes     =     364544

in use bytes     =       3120

Arena 51:

system bytes     =    2146304

in use bytes     =      23824

Arena 52:

system bytes     =     352256

in use bytes     =       2496

Arena 53:

system bytes     =     360448

in use bytes     =       2496

Arena 54:

system bytes     =     438272

in use bytes     =      85344

Arena 55:

system bytes     =     352256

in use bytes     =       2496

Arena 56:

system bytes     =     352256

in use bytes     =       2496

Arena 57:

system bytes     =     352256

in use bytes     =       2496

Arena 58:

system bytes     =     352256

in use bytes     =       2832

Arena 59:

system bytes     =     352256

in use bytes     =       2496

Arena 60:

system bytes     =     372736

in use bytes     =      12144

Arena 61:

system bytes     =     352256

in use bytes     =       3776

Arena 62:

system bytes     =    4816896

in use bytes     =      93648

Arena 63:

system bytes     =     364544

in use bytes     =       4352

Arena 64:

system bytes     =     135168

in use bytes     =       2448

Arena 65:

system bytes     =     135168

in use bytes     =      10928

Arena 66:

system bytes     =     135168

in use bytes     =       2448

Arena 67:

system bytes     =     135168

in use bytes     =      42256

Arena 68:

system bytes     =     135168

in use bytes     =      31440

Arena 69:

system bytes     =     135168

in use bytes     =      31408

Arena 70:

system bytes     =     135168

in use bytes     =       2448

Arena 71:

system bytes     =     135168

in use bytes     =       2448

Arena 72:

system bytes     =     135168

in use bytes     =      31552

Arena 73:

system bytes     =     135168

in use bytes     =      31552

Arena 74:

system bytes     =     135168

in use bytes     =      31552

Arena 75:

system bytes     =     163840

in use bytes     =      46448

Arena 76:

system bytes     =     135168

in use bytes     =       2448

Arena 77:

system bytes     =     135168

in use bytes     =       2448

Arena 78:

system bytes     =     135168

in use bytes     =       2512

Arena 79:

system bytes     =     135168

in use bytes     =       2224

Arena 80:

system bytes     =     135168

in use bytes     =       2448

Arena 81:

system bytes     =     135168

in use bytes     =       2448

Arena 82:

system bytes     =     135168

in use bytes     =       2448

Arena 83:

system bytes     =     135168

in use bytes     =       2448

Arena 84:

system bytes     =     135168

in use bytes     =       2704

Arena 85:

system bytes     =     135168

in use bytes     =       2448

Total (incl. mmap):

system bytes     = 3875291136

in use bytes     = 3827283984

max mmap regions =        154

max mmap bytes   = 3560165376

210325  6:48:49 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 10.4.18-MariaDB-log

key_buffer_size=67108864

read_buffer_size=2097152

max_used_connections=6

max_threads=1025

thread_count=11

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4313445 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f0a18000a88

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f0b100efe10 thread_stack 0x49000

mysys/stacktrace.c:175(my_print_stacktrace)[0x555c1954f8ae]

sql/signal_handler.cc:209(handle_fatal_signal)[0x555c18f51dbf]

sigaction.c:0(__restore_rt)[0x7f1b87b1a630]

:0(__GI_raise)[0x7f1b86924387]

:0(__GI_abort)[0x7f1b86925a78]

portability/toku_assert.cc:147(toku_do_backtrace_abort() [clone .107925])[0x7f0ae59277a8]

portability/toku_assert.cc:168(toku_do_assert_fail(char const*, char const*, char const*, int, int))[0x7f0ae5927827]

/opt/mariadb/mariadb/lib64/plugin/ha_tokudb.so(+0x5f41c)[0x7f0ae58db41c]

util/context.cc:45(__static_initialization_and_destruction_0)[0x7f0ae58db4f5]

:0(_dl_init_internal)[0x7f1b87d369c3]

dl-open.c:0(dl_open_worker)[0x7f1b87d3b59e]

:0(_dl_catch_error)[0x7f1b87d367d4]

:0(_dl_open)[0x7f1b87d3ab8b]

dlopen.c:0(dlopen_doit)[0x7f1b872c6fab]

:0(_dl_catch_error)[0x7f1b87d367d4]

:0(_dlerror_run)[0x7f1b872c75ad]

:0(__dlopen_check)[0x7f1b872c7041]

sql/sql_plugin.cc:752(plugin_dl_add(st_mysql_const_lex_string const*, unsigned long))[0x555c18d4fc76]

sql/sql_plugin.cc:2527(plugin_dl_foreach(THD*, st_mysql_const_lex_string const*, char (*)(THD*, st_plugin_int*, void*), void*))[0x555c18d515a0]

sql/sql_class.h:3908(THD::get_stmt_da())[0x555c18db548f]

sql/sql_show.cc:8934(get_schema_tables_result(JOIN*, enum_schema_table_state))[0x555c18dbb6cc]

sql/sql_select.cc:4442(JOIN::exec_inner())[0x555c18da06f5]

sql/sql_select.cc:4269(JOIN::exec())[0x555c18da0c03]

sql/sql_select.cc:4705(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x555c18d9ee36]

sql/sql_select.cc:422(handle_select(THD*, LEX*, select_result*, unsigned long))[0x555c18d9f9b7]

sql/sql_parse.cc:6419(execute_sqlcom_select(THD*, TABLE_LIST*))[0x555c18c337ff]

sql/sql_parse.cc:3937(mysql_execute_command(THD*))[0x555c18d43cb4]

sql/sql_parse.cc:7959(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x555c18d4959c]

sql/sql_audit.h:169(mysql_audit_general)[0x555c18d4bb5c]

sql/sql_parse.cc:1374(do_command(THD*))[0x555c18d4d372]

sql/threadpool_common.cc:373(threadpool_process_request)[0x555c18f2360f]

sql/threadpool_generic.cc:1605(worker_main(void*))[0x555c190e1040]

pthread_create.c:0(start_thread)[0x7f1b87b12ea5]

/lib64/libc.so.6(clone+0x6d)[0x7f1b869ec96d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f0a1800ff50): SHOW plugins soname

Connection ID (thread ID): 21834

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /data/mariadb/data

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units     

Max cpu time              unlimited            unlimited            seconds   

Max file size             unlimited            unlimited            bytes     

Max data size             unlimited            unlimited            bytes     

Max stack size            8388608              unlimited            bytes     

Max core file size        0                    unlimited            bytes     

Max resident set          unlimited            unlimited            bytes     

Max processes             65536                65536                processes 

Max open files            67567                67567                files     

Max locked memory         65536                65536                bytes     

Max address space         unlimited            unlimited            bytes     

Max file locks            unlimited            unlimited            locks     

Max pending signals       512538               512538               signals   

Max msgqueue size         819200               819200               bytes     

Max nice priority         0                    0                    

Max realtime priority     0                    0                    

Max realtime timeout      unlimited            unlimited            us        

Core pattern: core

210325 06:48:50 mysqld_safe Number of processes running now: 0

210325 06:48:50 mysqld_safe mysqld restarted