Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23872

Crash in galera::TrxHandle::state()

    XMLWordPrintable

    Details

      Description

      Server crashes regularly:

      2020-09-30 19:01:16 1027 [Warning] Aborted connection 1027 to db: 'gleif_oais' user: 'oais.import' host: '10.138.101.50' (Got an error reading communication packets)
      200930 19:10:24 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.5-MariaDB-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=186
      max_threads=65546
      thread_count=271
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 344005449 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f6e44156ba8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fd58d909b18 thread_stack 0x49000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ac79a5bdee]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55ac794e6ec5]
      2020-09-30 19:10:28 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.6731S), skipping check
      /lib64/libpthread.so.0(+0x12dd0)[0x7fd799a18dd0]
      2020-09-30 19:10:31 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.82415S), skipping check
      /usr/lib64/galera-4/libgalera_smm.so(+0x1c772b)[0x7fd795c8872b]
      /usr/lib64/galera-4/libgalera_smm.so(+0x1de42e)[0x7fd795c9f42e]
      2020-09-30 19:10:34 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.89535S), skipping check
      /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v266replayERKNS_9ws_handleEPNS_21high_priority_serviceE+0x29)[0x55ac79af5df9]
      /usr/sbin/mariadbd(_ZN20Wsrep_client_service6replayEv+0xf6)[0x55ac79767136]
      /usr/sbin/mariadbd(_ZN5wsrep11transaction6replayERNS_11unique_lockINS_5mutexEEE+0x87)[0x55ac79af0ed7]
      /usr/sbin/mariadbd(_ZN5wsrep11transaction15after_statementEv+0x1f0)[0x55ac79af2290]
      /usr/sbin/mariadbd(_ZN5wsrep12client_state15after_statementEv+0xa3)[0x55ac79ad2c73]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1052)[0x55ac792f7547]
      /usr/sbin/mariadbd(_Z10do_commandP3THD+0x110)[0x55ac792f924f]
      /usr/sbin/mariadbd(_Z11tp_callbackP13TP_connection+0x68)[0x55ac79478678]
      /usr/sbin/mariadbd(+0xac5da0)[0x55ac79663da0]
      /usr/sbin/mariadbd(+0xb67d5a)[0x55ac79705d5a]
      2020-09-30 19:10:36 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.89475S), skipping check
      /lib64/libpthread.so.0(+0x82de)[0x7fd799a0e2de]
      2020-09-30 19:10:39 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.91365S), skipping check
      /lib64/libc.so.6(clone+0x43)[0x7fd79787be83]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): 
      Connection ID (thread ID): 1
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      

      and in the full backtrace from the core we see:

      Thread 1 (Thread 0x7ed4840a5700 (LWP 3200149)):
      #0 0x00007f3b2c9fda65 in pthread_kill () from /lib64/libpthread.so.0
      No symbol table info available.
      #1 0x000056049aec89a7 in my_write_core (sig=<optimized out>) at /usr/src/debug/MariaDB-10.5.5/src_0/mysys/stacktrace.c:519
      No locals.
      #2 0x000056049a953ee0 in handle_fatal_signal (sig=11) at /usr/src/debug/MariaDB-10.5.5/src_0/sql/signal_handler.cc:330
      curr_time = 1601573573
      tm = {tm_sec = 53, tm_min = 32, tm_hour = 19, tm_mday = 1, tm_mon = 9, tm_year = 120, tm_wday = 4, tm_yday = 274, tm_isdst = 1, tm_gmtoff = 7200, tm_zone = 0x56049c6642d0 "CEST"}
      thd = 0x7ed3541b26f8
      print_invalid_query_pointer = false
      #3 <signal handler called>
      No symbol table info available.
      #4 galera::TrxHandle::state (this=<optimized out>) at galera/src/trx_handle.hpp:197
      No locals.
      #5 galera::ReplicatorSMM::replay_trx (this=0x56049c6c9df0, trx=..., lock=..., trx_ctx=0x7ed4840a3c10) at galera/src/replicator_smm.cpp:1054
      tsp = {px = 0x0, pn = {pi_ = 0x0}}
      ts = <optimized out>
      __FUNCTION__ = "replay_trx"
      retval = WSREP_OK
      #6 0x00007f3b28c8742e in galera_replay_trx (gh=<optimized out>, trx_handle=<optimized out>, recv_ctx=0x7ed4840a3c10) at galera/src/wsrep_provider.cpp:311
      lock = {trx_ = @0x7ed4dc218560, locked_ = true}
      repl = 0x56049c6c9df0
      trx = 0x7ed4dc218560
      __FUNCTION__ = "galera_replay_trx"
      retval = <optimized out>
      #7 0x000056049af62df9 in wsrep::wsrep_provider_v26::replay (this=<optimized out>, ws_handle=..., reply_service=<optimized out>) at /usr/src/debug/MariaDB-10.5.5/src_0/wsrep-lib/src/wsrep_provider_v26.cpp:204
      mwsh = {ws_handle_ = @0x7ed3541b8850, native_ = {trx_id = 18583, opaque = 0x7ed4dc218560}}
      #8 0x000056049abd4136 in Wsrep_client_service::replay (this=<optimized out>) at /usr/src/debug/MariaDB-10.5.5/src_0/wsrep-lib/include/wsrep/transaction.hpp:219
      replayer_service = {<Wsrep_high_priority_service> = {<wsrep::high_priority_service> = {_vptr.high_priority_service = 0x56049b870460 <vtable for Wsrep_replayer_service+16>, server_state_ = @0x56049c694bd0,
      must_exit_ = false}, <wsrep::high_priority_context> = {_vptr.high_priority_context = 0x56049b870520 <vtable for Wsrep_replayer_service+208>, client_ = @0x7ed3541b8788, orig_mode_ = wsrep::client_state::m_local},
      m_thd = 0x7ed3541b26f8, m_rli = 0x7ed3541b8b20, m_rgi = 0x7ed3541bb860, m_shadow = {option_bits = 2147748608, server_status = 2, vio = 0x0, tx_isolation = 1, db = 0x0, db_length = 0, user_time = {val = 0},
      row_count_func = -1, wsrep_applier = true}}, m_orig_thd = 0x7ed354000d38, m_da_shadow = {status = Diagnostics_area::DA_OK_BULK, affected_rows = 0, last_insert_id = 0, message = '\000' <repeats 511 times>},
      m_replay_status = wsrep::provider::success}
      provider = <optimized out>
      replayer_thd = 0x7ed3541b26f8
      ret = <optimized out>
      #9 0x000056049af5ded7 in wsrep::transaction::replay (this=0x7ed354006e30, lock=...) at /usr/src/debug/MariaDB-10.5.5/src_0/wsrep-lib/src/transaction.cpp:1703
      ret = 0
      was_streaming = false
      replay_ret = <optimized out>
      #10 0x000056049af5f290 in wsrep::transaction::after_statement (this=this@entry=0x7ed354006e30) at /usr/src/debug/MariaDB-10.5.5/src_0/wsrep-lib/src/transaction.cpp:816
      ret = 0
      lock = {mutex_ = @0x7ed354006d90, locked_ = false}
      #11 0x000056049af3fc73 in wsrep::client_state::after_statement (this=0x7ed354006dc8) at /usr/src/debug/MariaDB-10.5.5/src_0/wsrep-lib/src/client_state.cpp:246
      lock = {mutex_ = @0x7ed354006d90, locked_ = false}
      #12 0x000056049a764547 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) () at /usr/src/debug/MariaDB-10.5.5/src_0/wsrep-lib/include/wsrep/client_state.hpp:854
      TIME_INVALID_DATES = {m_mode = date_conv_mode_t::INVALID_DATES}
      std::__ioinit = {static _S_refcount = <optimized out>, static _S_synced_with_stdio = <optimized out>}
      TIME_TIME_ONLY = {m_mode = date_conv_mode_t::TIME_ONLY}
      TIME_FUZZY_DATES = {m_mode = date_conv_mode_t::FUZZY_DATES}
      TIME_NO_ZEROS = <optimized out>
      TIME_INTERVAL_hhmmssff = <optimized out>
      TIME_INTERVAL_DAY = <optimized out>
      TIME_MODE_FOR_XXX_TO_DATE = <optimized out>
      TIME_FRAC_NONE = <optimized out>
      TIME_FRAC_TRUNCATE = <optimized out>
      TIME_FRAC_ROUND = <optimized out>
      sp_data_access_name = <optimized out>
      TIME_NO_ZERO_IN_DATE = <optimized out>
      TIME_NO_ZERO_DATE = <optimized out>
      TIME_CONV_NONE = <optimized out>
      any_db = 0x56049af872d7 "*any*"
      sql_command_flags = 0x56049b93b760 <sql_command_flags>
      server_command_flags = 0x56049b93b360 <server_command_flags>
      command_name = 0x56049b73dc40 <command_name>
      #13 0x000056049a76624f in do_command (thd=thd@entry=0x7ed354000d38) at /usr/src/debug/MariaDB-10.5.5/src_0/sql/sql_parse.cc:1348
      return_value = <optimized out>
      packet = 0x7ed35444b24c "\372\002"
      packet_length = <optimized out>
      net = 0x7ed354000fa8
      command = COM_STMT_BULK_EXECUTE
      __FUNCTION__ = "do_command"
      #14 0x000056049a8e5678 in threadpool_process_request (thd=0x7ed354000d38) at /usr/src/debug/MariaDB-10.5.5/src_0/sql/threadpool_common.cc:354
      vio = <optimized out>
      retval = <optimized out>
      retval = <optimized out>
      vio = <optimized out>
      #15 tp_callback (c=0x560664638080) at /usr/src/debug/MariaDB-10.5.5/src_0/sql/threadpool_common.cc:194
      worker_context = {psi_thread = 0x0, mysys_var = 0x7ed354000b60}
      thd = 0x7ed354000d38
      #16 0x000056049aad0da0 in worker_main (param=param@entry=0x56066457ccd8) at /usr/src/debug/MariaDB-10.5.5/src_0/sql/threadpool_generic.cc:1559
      connection = <optimized out>
      ts = {tv_sec = 1601573618, tv_nsec = 356159000}
      this_thread = {event_count = 81, thread_group = 0x56066457ccd8, next_in_list = 0x0, prev_in_list = 0x56066457cd40, cond = {m_cond = {__data = {{__wseq = 155, __wseq32 = {__low = 155, __high = 0}}, {__g1_start = 153,
      __g1_start32 = {__low = 153, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 0, __g_signals = {0, 0}},
      __size = "\233\000\000\000\000\000\000\000\231", '\000' <repeats 23 times>, "\004", '\000' <repeats 14 times>, __align = 155}, m_psi = 0x0}, woken = true}
      thread_group = 0x56066457ccd8
      last_thread = <optimized out>
      #17 0x000056049ab72d5a in pfs_spawn_thread (arg=0x5606646380f8) at /usr/src/debug/MariaDB-10.5.5/src_0/storage/perfschema/pfs.cc:2201
      typed_arg = 0x5606646380f8
      user_arg = 0x56066457ccd8
      user_start_routine = 0x56049aad0d10 <worker_main(void*)>
      pfs = <optimized out>
      klass = <optimized out>
      #18 0x00007f3b2c9f62de in start_thread () from /lib64/libpthread.so.0
      No symbol table info available.
      #19 0x00007f3b2a863e83 in clone () from /lib64/libc.so.6
      No symbol table info available.
      

        Attachments

          Activity

            People

            Assignee:
            jplindst Jan Lindström
            Reporter:
            valerii Valerii Kravchuk
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration