Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32363

when InnoDB gets an assertion failure, WSREP layer is not handled gracefully

    XMLWordPrintable

Details

    Description

      If InnoDB gets an assertion failure, the WSREP layer is not immediately notified and this causes all nodes to lose primary status as a consequence.

      You can see below that WSREP messages are mixed with the assertion stacktrace, and that WSREP tries to reconnect with peers even if the node is practically crashed and in fact it will die soon.

      When this happens, the entire cluster goes into non-primary and a cluster bootstrap is needed to recover, which is not what we expect from the situation – crashed node should just be evicted and cluster should continue normally.

      2023-07-07 15:50:19 11985573 [ERROR] InnoDB: We detected index corruption in an InnoDB type table. You have to dump + drop + reimport the table or, in a case of widespread corruption, dump all InnoDB tables and recreate the whole tablespace. If the mariadbd server crashes after the startup or when you dump the tables. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
      2023-07-07 15:50:19 11985573 [ERROR] mariadbd: Index for table 'failed_table' is corrupt; try to repair it
      2023-07-07 15:50:20 0x7f17f7766700  InnoDB: Assertion failure in file /home/jenkins/workspace/Build-Package/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE/storage/innobase/page/page0zip.cc line 4213
      InnoDB: Failing assertion: slot_rec
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mariadbd startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      230707 15:50:20 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.6.12-7-MariaDB-enterprise-log source revision: 8e2b75dad28995ab5f6e6acd436135420f7031c9
      key_buffer_size=268435456
      read_buffer_size=131072
      max_used_connections=2243
      max_threads=6002
      thread_count=1565
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 13479553 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7ef735ab51c8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f17f7765cb8 thread_stack 0x49000
      Can't start addr2line
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5608f6116c7e]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5608f5bc33a5]
      /lib64/libpthread.so.0(+0xf630)[0x7f2011c15630]
      /lib64/libc.so.6(gsignal+0x37)[0x7f2011060387]
      /lib64/libc.so.6(abort+0x148)[0x7f2011061a78]
      /usr/sbin/mariadbd(+0x694d97)[0x5608f5834d97]
      /usr/sbin/mariadbd(+0xdbfb05)[0x5608f5f5fb05]
      /usr/sbin/mariadbd(+0xdaf516)[0x5608f5f4f516]
      2023-07-07 15:50:32 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 844de70f-8aaf with addr tcp://10.10.1.103:4567 timed out, no messages seen in PT6S, socket stats: rtt: 766 rttvar: 579 rto: 201000 lost: 0 last_data_recv: 2567 cwnd: 10 last_queued_since: 8776161264 last_delivered_since: 11959172679 send_queue_length: 9 send_queue_bytes: 720 segment: 0 messages: 9
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.10.1.103:4567
      2023-07-07 15:50:32 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 5dda822d-b4c3 with addr tcp://10.10.1.104:4567 timed out, no messages seen in PT6S, socket stats: rtt: 6407 rttvar: 10277 rto: 207000 lost: 0 last_data_recv: 5900 cwnd: 10 last_queued_since: 307819 last_delivered_since: 8781038225 send_queue_length: 10 send_queue_bytes: 1080 segment: 0 messages: 10
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.10.1.104:42156
      2023-07-07 15:50:32 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 96c49f4b-8727 with addr tcp://10.10.1.101:4567 timed out, no messages seen in PT6S, socket stats: rtt: 5591 rttvar: 10094 rto: 206000 lost: 0 last_data_recv: 5924 cwnd: 10 last_queued_since: 10916 last_delivered_since: 8781705783 send_queue_length: 11 send_queue_bytes: 1292 segment: 0 messages: 11
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.10.1.101:33510
      2023-07-07 15:50:32 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.1.101:4567 tcp://10.10.1.103:4567 tcp://10.10.1.104:4567 
      2023-07-07 15:50:32 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT12.0655S), skipping check
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f17e8905e58
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer destruct
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7ef49da77b98
      2023-07-07 15:50:32 0 [Note] WSREP: Deferred close timer destruct
      /usr/sbin/mariadbd(+0xe62a65)[0x5608f6002a65]
      2023-07-07 15:50:33 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') reconnecting to 96c49f4b-8727 (tcp://10.10.1.101:4567), attempt 0
      2023-07-07 15:50:33 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') reconnecting to 844de70f-8aaf (tcp://10.10.1.103:4567), attempt 0
      2023-07-07 15:50:33 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') reconnecting to 5dda822d-b4c3 (tcp://10.10.1.104:4567), attempt 0
      2023-07-07 15:50:33 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.73771S), skipping check
      /usr/sbin/mariadbd(+0xe4e639)[0x5608f5fee639]
      /usr/sbin/mariadbd(+0xe5063b)[0x5608f5ff063b]
      /usr/sbin/mariadbd(+0xe62e98)[0x5608f6002e98]
      /usr/sbin/mariadbd(+0xde0227)[0x5608f5f80227]
      /usr/sbin/mariadbd(+0xde2da4)[0x5608f5f82da4]
      2023-07-07 15:50:41 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT6S, socket stats: rtt: 1000 rttvar: 500 rto: 201000 lost: 0 last_data_recv: 125408244 cwnd: 10 last_queued_since: 4421911460 last_delivered_since: 8715642835357354 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      2023-07-07 15:50:41 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.1.103:4567 timed out, no messages seen in PT6S, socket stats: rtt: 359 rttvar: 179 rto: 201000 lost: 0 last_data_recv: 7528 cwnd: 10 last_queued_since: 120133 last_delivered_since: 120133 send_queue_length: 0 send_queue_bytes: 0
      2023-07-07 15:50:41 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.1.101:4567 timed out, no messages seen in PT6S, socket stats: rtt: 437 rttvar: 218 rto: 201000 lost: 0 last_data_recv: 7532 cwnd: 10 last_queued_since: 4131087 last_delivered_since: 4131087 send_queue_length: 0 send_queue_bytes: 0
      2023-07-07 15:50:41 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.1.104:4567 timed out, no messages seen in PT6S, socket stats: rtt: 181 rttvar: 90 rto: 201000 lost: 0 last_data_recv: 7532 cwnd: 10 last_queued_since: 8715642839939685 last_delivered_since: 8715642839939685 send_queue_length: 0 send_queue_bytes: 0
      2023-07-07 15:50:41 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT7.53388S), skipping check
      /usr/sbin/mariadbd(+0xe151ab)[0x5608f5fb51ab]
      2023-07-07 15:50:43 0 [Note] WSREP: Deferred close timer handle_wait Success for 0x7f17ebc5b168
      2023-07-07 15:50:43 0 [Note] WSREP: Deferred close timer destruct
      2023-07-07 15:50:43 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.61329S), skipping check
      /usr/sbin/mariadbd(+0xe15869)[0x5608f5fb5869]
      2023-07-07 15:50:44 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.63124S), skipping check
      2023-07-07 15:50:46 0 [Note] WSREP: evs::proto(6c357751-8d5f, OPERATIONAL, view_id(REG,5dda822d-b4c3,50)) detected inactive node: 5dda822d-b4c3
      2023-07-07 15:50:46 0 [Note] WSREP: evs::proto(6c357751-8d5f, OPERATIONAL, view_id(REG,5dda822d-b4c3,50)) detected inactive node: 844de70f-8aaf
      2023-07-07 15:50:46 0 [Note] WSREP: evs::proto(6c357751-8d5f, OPERATIONAL, view_id(REG,5dda822d-b4c3,50)) detected inactive node: 96c49f4b-8727
      /usr/sbin/mariadbd(+0xdf28b2)[0x5608f5f928b2]
      /usr/sbin/mariadbd(+0xd43ca8)[0x5608f5ee3ca8]
      /usr/sbin/mariadbd(_ZN7handler13ha_update_rowEPKhS1_+0x232)[0x5608f5bd12b2]
      /usr/sbin/mariadbd(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_orderybPySA_+0x1a63)[0x5608f5a5cf33]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x263e)[0x5608f597d38e]
      /usr/sbin/mariadbd(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x38)[0x5608f58cb718]
      /usr/sbin/mariadbd(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x176)[0x5608f58d48b6]
      /usr/sbin/mariadbd(_ZN13sp_instr_stmt7executeEP3THDPj+0x5bc)[0x5608f58d529c]
      /usr/sbin/mariadbd(_ZN7sp_head7executeEP3THDb+0xa0c)[0x5608f58ceeac]
      2023-07-07 15:50:57 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT6S, socket stats: rtt: 0 rttvar: 0 rto: 200000 lost: 0 last_data_recv: 125424590 cwnd: 10 last_queued_since: 11598743653 last_delivered_since: 8715659181935888 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      2023-07-07 15:50:57 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 5dda822d-b4c3 with addr tcp://10.10.1.104:4567 timed out, no messages seen in PT6S, socket stats: rtt: 185 rttvar: 82 rto: 201000 lost: 0 last_data_recv: 5508 cwnd: 10 last_queued_since: 11598929590 last_delivered_since: 11598962474 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      2023-07-07 15:50:57 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 844de70f-8aaf with addr tcp://10.10.1.103:4567 timed out, no messages seen in PT6S, socket stats: rtt: 264 rttvar: 105 rto: 201000 lost: 0 last_data_recv: 5508 cwnd: 10 last_queued_since: 11599134047 last_delivered_since: 11599140957 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      2023-07-07 15:50:57 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 96c49f4b-8727 with addr tcp://10.10.1.101:4567 timed out, no messages seen in PT6S, socket stats: rtt: 384 rttvar: 151 rto: 201000 lost: 0 last_data_recv: 5678 cwnd: 10 last_queued_since: 11599486516 last_delivered_since: 11599494477 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      /usr/sbin/mariadbd(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x66a)[0x5608f58d093a]
      /usr/sbin/mariadbd(+0x7cfc17)[0x5608f596fc17]
      /usr/sbin/mariadbd(+0x7d3a68)[0x5608f5973a68]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x101a)[0x5608f597bd6a]
      2023-07-07 15:51:04 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr  timed out, no messages seen in PT6S, socket stats: rtt: 0 rttvar: 0 rto: 200000 lost: 0 last_data_recv: 125430994 cwnd: 10 last_queued_since: 221024 last_delivered_since: 8715665585967344 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
      2023-07-07 15:51:04 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT18.0029S), skipping check
      2023-07-07 15:51:04 0 [Note] WSREP: view(view_id(NON_PRIM,5dda822d-b4c3,50) memb {
      	6c357751-8d5f,0
      } joined {
      } left {
      } partitioned {
      	5dda822d-b4c3,0
      	844de70f-8aaf,0
      	96c49f4b-8727,0
      })
      2023-07-07 15:51:04 0 [Note] WSREP: view(view_id(NON_PRIM,6c357751-8d5f,51) memb {
      	6c357751-8d5f,0
      } joined {
      } left {
      } partitioned {
      	5dda822d-b4c3,0
      	844de70f-8aaf,0
      	96c49f4b-8727,0
      })
      2023-07-07 15:51:04 11994604 [Warning] WSREP: Send action {(nil), 139603616989752, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2023-07-07 15:51:04 0 [Note] WSREP: Flow-control interval: [240, 300]
      2023-07-07 15:51:04 0 [Note] WSREP: Received NON-PRIMARY.
      2023-07-07 15:51:04 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 6204240577)
      2023-07-07 15:51:04 11955214 [Warning] WSREP: Send action {(nil), 139599322023456, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11985855 [Warning] WSREP: Send action {(nil), 139603616990584, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11987046 [Warning] WSREP: Send action {(nil), 139599322023328, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11985820 [Warning] WSREP: Send action {(nil), 139599322023336, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 46 [Note] WSREP: ================================================
      View:
        id: c3a51458-b6fd-11eb-8a80-eb35c100e72c:6204240577
        status: non-primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 0
        members(1):
      	0: 6c357751-ce4f-11ed-8d5f-136e7094748b, PIXID-MDB-MASTER2
      =================================================
      2023-07-07 15:51:04 46 [Note] WSREP: Non-primary view
      2023-07-07 15:51:04 46 [Note] WSREP: Server status change synced -> connected
      2023-07-07 15:51:04 46 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-07-07 15:51:04 11997537 [Warning] WSREP: Send action {(nil), 139603616989760, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11996647 [Warning] WSREP: Send action {(nil), 139573552218680, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11986459 [Warning] WSREP: Send action {(nil), 139736760976944, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11997986 [Warning] WSREP: Send action {(nil), 139599322023552, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11985505 [Warning] WSREP: Send action {(nil), 139599322023336, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:04 11988311 [Warning] WSREP: Send action {(nil), 139607911957872, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11917365 [Warning] WSREP: Send action {(nil), 139586437121400, WRITESET} returned -107 (Transport endpoint is not connected)
      /usr/sbin/mariadbd(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x38)[0x5608f58cb718]
      2023-07-07 15:51:06 11985895 [Warning] WSREP: Send action {(nil), 139590732088096, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2023-07-07 15:51:06 0 [Note] WSREP: Flow-control interval: [240, 300]
      2023-07-07 15:51:06 0 [Note] WSREP: Received NON-PRIMARY.
      2023-07-07 15:51:06 11978506 [Warning] WSREP: Send action {(nil), 139599322023472, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11997530 [Warning] WSREP: Send action {(nil), 139599322023336, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11988302 [Warning] WSREP: Send action {(nil), 139599322023760, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11988512 [Warning] WSREP: Send action {(nil), 139736760977344, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11988290 [Warning] WSREP: Send action {(nil), 139595027055344, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11998006 [Warning] WSREP: Send action {(nil), 139599322023888, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11988294 [Warning] WSREP: Send action {(nil), 139603616990632, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11991973 [Warning] WSREP: Send action {(nil), 139599322023712, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11988288 [Warning] WSREP: Send action {(nil), 139595027057080, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11985860 [Warning] WSREP: Send action {(nil), 139577847186808, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11997914 [Warning] WSREP: Send action {(nil), 139599322023336, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11987069 [Warning] WSREP: Send action {(nil), 139599322023280, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11988354 [Warning] WSREP: Send action {(nil), 139736760976760, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11986489 [Warning] WSREP: Send action {(nil), 139564962285120, WRITESET} returned -107 (Transport endpoint is not connected)
      2023-07-07 15:51:06 11986139 [Warning] WSREP: Send action {(nil), 139582142155592, WRITESET} returned -107 (Transport endpoint is not connected)
      /usr/sbin/mariadbd(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x176)[0x5608f58d48b6]
      2023-07-07 15:51:07 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.43448S), skipping check
      /usr/sbin/mariadbd(_ZN13sp_instr_stmt7executeEP3THDPj+0x5bc)[0x5608f58d529c]
      /usr/sbin/mariadbd(_ZN7sp_head7executeEP3THDb+0xa0c)[0x5608f58ceeac]
      2023-07-07 15:51:10 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.93087S), skipping check
      /usr/sbin/mariadbd(_ZN7sp_head15execute_triggerEP3THDPK25st_mysql_const_lex_stringS4_P13st_grant_info+0x1df)[0x5608f58d008f]
      2023-07-07 15:51:11 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection established to 5dda822d-b4c3 tcp://10.10.1.104:4567
      2023-07-07 15:51:11 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection established to 844de70f-8aaf tcp://10.10.1.103:4567
      /usr/sbin/mariadbd(_ZN19Table_triggers_list16process_triggersEP3THD14trg_event_type20trg_action_time_typeb+0x104)[0x5608f5a40ec4]
      2023-07-07 15:51:13 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.87478S), skipping check
      /usr/sbin/mariadbd(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyyP13select_result+0xd99)[0x5608f5d33da9]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x24c8)[0x5608f597d218]
      2023-07-07 15:51:16 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.90666S), skipping check
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x20a)[0x5608f5980c9a]
      /usr/sbin/mariadbd(+0x7e1531)[0x5608f5981531]
      2023-07-07 15:51:19 0 [Note] WSREP: (6c357751-8d5f, 'tcp://0.0.0.0:4567') connection established to 96c49f4b-8727 tcp://10.10.1.101:4567
      2023-07-07 15:51:19 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.83971S), skipping check
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x29e1)[0x5608f5984c31]
      2023-07-07 15:51:20 0 [Warning] WSREP: evs::proto(6c357751-8d5f, GATHER, view_id(REG,6c357751-8d5f,51)) install timer expired
      evs::proto(evs::proto(6c357751-8d5f, GATHER, view_id(REG,6c357751-8d5f,51)), GATHER) {
      current_view=view(view_id(REG,6c357751-8d5f,51) memb {
      	6c357751-8d5f,0
      } joined {
      } left {
      } partitioned {
      }),
      input_map=evs::input_map: {aru_seq=2,safe_seq=2,node_index=node: {idx=0,range=[3,2],safe_seq=2} },
      fifo_seq=1874639154,
      last_sent=2,
      known:
      5dda822d-b4c3 at tcp://10.10.1.104:4567
      {o=1,s=0,i=0,fs=834192942,jm=
      {v=1,t=4,ut=255,o=1,s=122,sr=-1,as=122,f=4,src=5dda822d-b4c3,srcvid=view_id(REG,5dda822d-b4c3,52),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=834192942,nl=(
      	5dda822d-b4c3, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      	6c357751-8d5f, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,6c357751-8d5f,51),ss=2,ir=[3,2],}
      	844de70f-8aaf, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      	96c49f4b-8727, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      )
      },
      }
      6c357751-8d5f at 
      {o=1,s=0,i=0,fs=-1,jm=
      {v=1,t=4,ut=255,o=1,s=2,sr=-1,as=2,f=0,src=6c357751-8d5f,srcvid=view_id(REG,6c357751-8d5f,51),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=1874639154,nl=(
      	5dda822d-b4c3, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      	6c357751-8d5f, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,6c357751-8d5f,51),ss=2,ir=[3,2],}
      	844de70f-8aaf, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      	96c49f4b-8727, {o=0,s=0,e=0,ls=-1,vid=view_id(REG,00000000-0000,0),ss=-1,ir=[-1,-1],}
      )
      },
      }
      844de70f-8aaf at tcp://10.10.1.103:4567
      {o=1,s=0,i=0,fs=1475544355,jm=
      {v=1,t=4,ut=255,o=1,s=122,sr=-1,as=122,f=4,src=844de70f-8aaf,srcvid=view_id(REG,5dda822d-b4c3,52),insvid=view_id(UNKNOWN,00000000-0000,0),ru=00000000-0000,r=[-1,-1],fs=1475544355,nl=(
      	5dda822d-b4c3, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      	6c357751-8d5f, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,6c357751-8d5f,51),ss=2,ir=[3,2],}
      	844de70f-8aaf, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      	96c49f4b-8727, {o=1,s=0,e=0,ls=-1,vid=view_id(REG,5dda822d-b4c3,52),ss=122,ir=[123,122],}
      )
      },
      }
      96c49f4b-8727 at tcp://10.10.1.101:4567
      {o=0,s=0,i=0,fs=101154494,}
       }
      2023-07-07 15:51:20 0 [Note] WSREP: no install message received
      2023-07-07 15:51:20 0 [Note] WSREP: view(view_id(NON_PRIM,6c357751-8d5f,52) memb {
      	6c357751-8d5f,0
      } joined {
      } left {
      } partitioned {
      	5dda822d-b4c3,0
      	844de70f-8aaf,0
      	96c49f4b-8727,0
      })
      2023-07-07 15:51:20 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2023-07-07 15:51:20 0 [Note] WSREP: Flow-control interval: [240, 300]
      2023-07-07 15:51:20 0 [Note] WSREP: Received NON-PRIMARY.
      /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x132)[0x5608f5985942]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3b7)[0x5608f5aa2dd7]
      2023-07-07 15:51:23 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT4.34509S), skipping check
      /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x5608f5aa311d]
      2023-07-07 15:51:24 11986441 [Warning] WSREP: Send action {(nil), 139599322023528, WRITESET} returned -107 (Transport endpoint is not connected)
      /usr/sbin/mariadbd(+0xc839d2)[0x5608f5e239d2]
      /lib64/libpthread.so.0(+0x7ea5)[0x7f2011c0dea5]
      /lib64/libc.so.6(clone+0x6d)[0x7f2011128b0d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7ef692470578): UPDATE failed_schema.failed_table
      			SET TYPE_EVE =  NAME_CONST('V_TYPE_EVE',_utf8mb3'D' COLLATE 'utf8mb3_general_ci') , DATE_EFFECTIVE =  NAME_CONST('V_DATE_EFFECTIVE',TIMESTAMP'2023-07-07 15:50:20'), STATUT_DATAMART = NULL  WHERE EVT_ID =  NAME_CONST('V_EVT_ID',2051583478)
       
      Connection ID (thread ID): 11985593
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /data/mysql
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             805978               805978               processes 
      Max open files            1048576              1048576              files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       805978               805978               signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
      

      Attachments

        1. galera_crash_node.test
          4 kB
          Jan Lindström
        2. logs.tar.gz
          9.48 MB
          Ramesh Sivaraman

        Issue Links

          Activity

            People

              sysprg Julius Goryavsky
              rpizzi Rick Pizzi (Inactive)
              Votes:
              5 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.