Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32070

MariaDB instance crashes on a 3-node galera setup intermittently and comes back up in less than a minute

    XMLWordPrintable

Details

    Description

      We have a 3-node Galera set up with binlog replication happening to another 3-node galera cluster. On the primary Galera cluster, mariadb process is aborted and restarted automatically in under 30 seconds, and it comes up fine with no issues.

      Attaching the stack trace from the error log file:

      mariadbd: /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.11/wsrep-lib/include/wsrep/client_state.hpp:668: int wsrep::client_state::bf_abort(wsrep::seqno): Assertion `mode_ == m_local || transaction_.is_streaming()' failed.
      230831 19:18:32 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.

      Server version: 10.6.11-MariaDB-log
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=18
      max_threads=1002
      thread_count=12
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337543 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x7fd470000c58
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fdaf00fdbc8 thread_stack 0x49000
      Can't start addr2line
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5562edd0c33e]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5562ed7ec295]
      /lib64/libpthread.so.0(+0x12d40)[0x7fdb56753d40]
      /lib64/libc.so.6(gsignal+0x10f)[0x7fdb55aafb8f]
      /lib64/libc.so.6(abort+0x127)[0x7fdb55a82ea5]
      /lib64/libc.so.6(+0x21d79)[0x7fdb55a82d79]
      /lib64/libc.so.6(+0x474e6)[0x7fdb55aa84e6]
      /usr/sbin/mariadbd(Z14wsrep_bf_abortP3THDS0+0x5f9)[0x5562eda90949]
      /usr/sbin/mariadbd(wsrep_thd_bf_abort+0x19)[0x5562eda982a9]
      /usr/sbin/mariadbd(+0xc7d64c)[0x5562edaba64c]
      /usr/sbin/mariadbd(+0x6398fa)[0x5562ed4768fa]
      /usr/sbin/mariadbd(+0x639964)[0x5562ed476964]
      /usr/sbin/mariadbd(+0xd322af)[0x5562edb6f2af]
      /usr/sbin/mariadbd(+0xd34731)[0x5562edb71731]
      /usr/sbin/mariadbd(+0xc86da5)[0x5562edac3da5]
      /usr/sbin/mariadbd(_ZN7handler12ha_write_rowEPKh+0x21d)[0x5562ed7fa8bd]
      /usr/sbin/mariadbd(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x175)[0x5562ed8f9365]
      /usr/sbin/mariadbd(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x10d)[0x5562ed8f99ad]
      /usr/sbin/mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x27f)[0x5562ed8ee9ef]
      /usr/sbin/mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0x1dc)[0x5562eda8f71c]
      /usr/sbin/mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xa4)[0x5562eda75ff4]
      /usr/sbin/mariadbd(+0xf5d89a)[0x5562edd9a89a]
      /usr/sbin/mariadbd(_ZN5wsrep12server_state8on_applyERNS_21high_priority_serviceERKNS_9ws_handleERKNS_7ws_metaERKNS_12const_bufferE+0x15d)[0x5562edd9b42d]
      /usr/sbin/mariadbd(+0xf71f34)[0x5562eddaef34]
      /usr/lib64/galera-4/libgalera_smm.so(+0x57cef)[0x7fdb479e7cef]
      /usr/lib64/galera-4/libgalera_smm.so(+0x69200)[0x7fdb479f9200]
      /usr/lib64/galera-4/libgalera_smm.so(+0x6b03e)[0x7fdb479fb03e]
      /usr/lib64/galera-4/libgalera_smm.so(+0x974eb)[0x7fdb47a274eb]
      /usr/lib64/galera-4/libgalera_smm.so(+0x976a2)[0x7fdb47a276a2]
      /usr/lib64/galera-4/libgalera_smm.so(+0x98171)[0x7fdb47a28171]
      /usr/lib64/galera-4/libgalera_smm.so(+0x6bfc0)[0x7fdb479fbfc0]
      /usr/lib64/galera-4/libgalera_smm.so(+0x47538)[0x7fdb479d7538]
      /usr/sbin/mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0xe)[0x5562eddafd0e]
      /usr/sbin/mariadbd(+0xc546ed)[0x5562eda916ed]
      /usr/sbin/mariadbd(_Z15start_wsrep_THDPv+0x28c)[0x5562eda8119c]
      /usr/sbin/mariadbd(+0xbd113d)[0x5562eda0e13d]
      /lib64/libpthread.so.0(+0x81da)[0x7fdb567491da]
      /lib64/libc.so.6(clone+0x43)[0x7fdb55a9ae73]

      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7fd404bb5493): LOAD DATA LOCAL INFILE ':SOURCE:21fc815f20034519b1f4a8327db39126' INTO TABLE TBL_LINKED_ORDERS_INPUT CHARACTER SET utf8mb4 FIELDS TERMINATED BY ' ' ESCAPED BY '
      ' LINES TERMINATED BY '
      ' IGNORE 0 LINES (`LEAD_JIN`,`LEAD_TYPE`,`LEAD_DATE`,`LEAD_STATUS`,`CHILD_JIN`,`CHILD_TYPE`,`CHILD_DATE`,`CHILD_STATUS`,`WMQUEUE`)

      Connection ID (thread ID): 6
      Status: NOT_KILLED

      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /data/mariadata/mysql
      Resource Limits:
      Limit Soft Limit Hard Limit Units
      Max cpu time unlimited unlimited seconds
      Max file size unlimited unlimited bytes
      Max data size unlimited unlimited bytes
      Max stack size 8388608 unlimited bytes
      Max core file size 0 unlimited bytes
      Max resident set unlimited unlimited bytes
      Max processes 127401 127401 processes
      Max open files 32768 32768 files
      Max locked memory 65536 65536 bytes
      Max address space unlimited unlimited bytes
      Max file locks unlimited unlimited locks
      Max pending signals 127401 127401 signals
      Max msgqueue size 819200 819200 bytes
      Max nice priority 0 0
      Max realtime priority 0 0
      Max realtime timeout unlimited unlimited us
      Core pattern: |/opt/dynatrace/oneagent/agent/rdp -p %p -P %P -c %c -e %e -s %s -a %P %u %g %s %t %c %h %e
      Kernel version: Linux version 4.18.0-477.15.1.el8_8.x86_64 (mockbuild@host-100-100-224-19) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-18.0.2) (GCC)) #1 SMP Thu Jul 20 11:31:48 PDT 2023

      2023-08-31 19:18:47 0 [Note] WSREP: Loading provider /usr/lib64/galera-4/libgalera_smm.so initial position: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:47 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-4/libgalera_smm.so'
      2023-08-31 19:18:47 0 [Note] WSREP: wsrep_load(): Galera 26.4.13(rfe497aeb) by Codership Oy <info@codership.com> loaded successfully.
      2023-08-31 19:18:47 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
      2023-08-31 19:18:47 0 [Note] WSREP: Found saved state: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:-1, safe_to_bootstrap: 0
      2023-08-31 19:18:47 0 [Note] WSREP: GCache DEBUG: opened preamble:
      Version: 2
      UUID: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05
      Seqno: -1 - -1
      Offset: -1
      Synced: 0
      2023-08-31 19:18:47 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05, offset: -1
      2023-08-31 19:18:47 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/524288024 bytes) complete.
      2023-08-31 19:18:47 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (524288024/524288024 bytes) complete.
      2023-08-31 19:18:47 0 [Note] WSREP: Recovering GCache ring buffer: Recovery failed, need to do full reset.
      2023-08-31 19:18:47 0 [Note] WSREP: Passing config to GCS: base_dir = /data/mariadata/mysql/; base_host = 10.13.237.29; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data/mariadata/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 500M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 500M; gcache.recover = yes; gcache.size = 500M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcas
      2023-08-31 19:18:47 0 [Note] WSREP: Service thread queue flushed.
      2023-08-31 19:18:47 0 [Note] WSREP: ####### Assign initial position for certification: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854, protocol version: -1
      2023-08-31 19:18:47 0 [Note] WSREP: GCache history reset: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:0 -> c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:47 0 [Note] WSREP: Start replication
      2023-08-31 19:18:47 0 [Note] WSREP: Connecting with bootstrap option: 0
      2023-08-31 19:18:47 0 [Note] WSREP: Setting GCS initial position to c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:47 0 [Note] WSREP: protonet asio version 0
      2023-08-31 19:18:47 0 [Note] WSREP: Using CRC-32C for message checksums.
      2023-08-31 19:18:47 0 [Note] WSREP: backend: asio
      2023-08-31 19:18:47 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
      2023-08-31 19:18:47 0 [Note] WSREP: restore pc from disk successfully
      2023-08-31 19:18:47 0 [Note] WSREP: GMCast version 0
      2023-08-31 19:18:47 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      2023-08-31 19:18:47 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      2023-08-31 19:18:47 0 [Note] WSREP: EVS version 1
      2023-08-31 19:18:47 0 [Note] WSREP: gcomm: connecting to group 'galera-prod', peer '10.13.237.28:,10.13.237.29:,10.13.237.30:'
      2023-08-31 19:18:47 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.13.237.29:4567
      2023-08-31 19:18:47 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') connection established to a564b541-b726 tcp://10.13.237.28:4567
      2023-08-31 19:18:47 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      2023-08-31 19:18:47 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') connection established to 902a3da4-af60 tcp://10.13.237.30:4567
      2023-08-31 19:18:47 0 [Note] WSREP: EVS version upgrade 0 -> 1
      2023-08-31 19:18:47 0 [Note] WSREP: declaring 902a3da4-af60 at tcp://10.13.237.30:4567 stable
      2023-08-31 19:18:47 0 [Note] WSREP: declaring a564b541-b726 at tcp://10.13.237.28:4567 stable
      2023-08-31 19:18:47 0 [Note] WSREP: PC protocol upgrade 0 -> 1
      2023-08-31 19:18:47 0 [Note] WSREP: Node 902a3da4-af60 state prim
      2023-08-31 19:18:47 0 [Note] WSREP: view(view_id(PRIM,213ecf6f-a3aa,17) memb

      { 213ecf6f-a3aa,0 902a3da4-af60,0 a564b541-b726,0 }

      joined {
      } left {
      } partitioned {
      })
      2023-08-31 19:18:47 0 [Note] WSREP: save pc into disk
      2023-08-31 19:18:47 0 [Note] WSREP: clear restored view
      2023-08-31 19:18:48 0 [Note] WSREP: gcomm: connected
      2023-08-31 19:18:48 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      2023-08-31 19:18:48 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      2023-08-31 19:18:48 0 [Note] WSREP: Opened channel 'galera-prod'
      2023-08-31 19:18:48 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
      2023-08-31 19:18:48 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: d44be2bc-482a-11ee-ae66-3773224befaa
      2023-08-31 19:18:48 1 [Note] WSREP: Starting rollbacker thread 1
      2023-08-31 19:18:48 2 [Note] WSREP: Starting applier thread 2
      2023-08-31 19:18:48 0 [Note] WSREP: STATE EXCHANGE: sent state msg: d44be2bc-482a-11ee-ae66-3773224befaa
      2023-08-31 19:18:48 0 [Note] WSREP: STATE EXCHANGE: got state msg: d44be2bc-482a-11ee-ae66-3773224befaa from 0 (blp21152017)
      2023-08-31 19:18:48 0 [Note] WSREP: STATE EXCHANGE: got state msg: d44be2bc-482a-11ee-ae66-3773224befaa from 1 (blp21152018)
      2023-08-31 19:18:48 0 [Note] WSREP: STATE EXCHANGE: got state msg: d44be2bc-482a-11ee-ae66-3773224befaa from 2 (blp21152016)
      2023-08-31 19:18:48 0 [Note] WSREP: Quorum results:
      version = 6,
      component = PRIMARY,
      conf_id = 16,
      members = 2/3 (joined/total),
      act_id = 16049865,
      last_appl. = 16049855,
      protocols = 2/10/4 (gcs/repl/appl),
      vote policy= 0,
      group UUID = c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05
      2023-08-31 19:18:48 0 [Note] WSREP: Flow-control interval: [28, 28]
      2023-08-31 19:18:48 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 16049866)
      2023-08-31 19:18:48 2 [Note] WSREP: ####### processing CC 16049866, local, ordered
      2023-08-31 19:18:48 2 [Note] WSREP: Process first view: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05 my uuid: 213ecf6f-3f1b-11ee-a3aa-1b9e106b662b
      2023-08-31 19:18:48 2 [Note] WSREP: Server blp21152017 connected to cluster at position c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049866 with ID 213ecf6f-3f1b-11ee-a3aa-1b9e106b662b
      2023-08-31 19:18:48 2 [Note] WSREP: Server status change disconnected -> connected
      2023-08-31 19:18:48 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:48 2 [Note] WSREP: ####### My UUID: 213ecf6f-3f1b-11ee-a3aa-1b9e106b662b
      2023-08-31 19:18:48 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
      2023-08-31 19:18:48 0 [Note] WSREP: Service thread queue flushed.
      2023-08-31 19:18:48 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
      2023-08-31 19:18:48 2 [Note] WSREP: State transfer required:
      Group state: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049866
      Local state: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:48 2 [Note] WSREP: Server status change connected -> joiner
      2023-08-31 19:18:48 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:48 0 [Note] WSREP: Joiner monitor thread started to monitor
      2023-08-31 19:18:48 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.13.237.29' --datadir '/data/mariadata/mysql/' --parent 3808279 --progress 0 --binlog '/data/mariadata/binlogs/mariadb-bin' --mysqld-args --wsrep_start_position=c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854,5-1-16049240'
      WSREP_SST: [INFO] rsync SST started on joiner (20230831 19:18:48.299)
      2023-08-31 19:18:48 2 [Note] WSREP: ####### IST uuid:c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05 f: 16049855, l: 16049866, STRv: 3
      2023-08-31 19:18:48 2 [Note] WSREP: IST receiver addr using tcp://10.13.237.29:4568
      2023-08-31 19:18:48 2 [Note] WSREP: Prepared IST receiver for 16049855-16049866, listening at: tcp://10.13.237.29:4568
      2023-08-31 19:18:48 0 [Note] WSREP: Member 0.0 (blp21152017) requested state transfer from 'any'. Selected 2.0 (blp21152016)(SYNCED) as donor.
      2023-08-31 19:18:48 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 16049866)
      2023-08-31 19:18:48 2 [Note] WSREP: Requesting state transfer: success, donor: 2
      2023-08-31 19:18:48 0 [Note] WSREP: 2.0 (blp21152016): State transfer to 0.0 (blp21152017) complete.
      2023-08-31 19:18:48 0 [Note] WSREP: Member 2.0 (blp21152016) synced with group.
      WSREP_SST: [INFO] Galera co-ords from recovery: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854 5 (20230831 19:18:49.560)
      WSREP_SST: [INFO] rsync IST completed on joiner (20230831 19:18:49.563)
      WSREP_SST: [INFO] Joiner cleanup: rsync PID=3808409, stunnel PID=0 (20230831 19:18:49.567)
      WSREP_SST: [INFO] Joiner cleanup done. (20230831 19:18:50.111)
      2023-08-31 19:18:50 3 [Note] WSREP: SST received
      2023-08-31 19:18:50 3 [Note] WSREP: Server status change joiner -> initializing
      2023-08-31 19:18:50 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2023-08-31 19:18:50 0 [Note] InnoDB: Number of pools: 1
      2023-08-31 19:18:50 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2023-08-31 19:18:50 0 [Note] InnoDB: Using Linux native AIO
      2023-08-31 19:18:50 0 [Note] InnoDB: Initializing buffer pool, total size = 25769803776, chunk size = 134217728
      2023-08-31 19:18:50 0 [Note] InnoDB: Completed initialization of buffer pool
      2023-08-31 19:18:50 0 [Note] InnoDB: 128 rollback segments are active.
      2023-08-31 19:18:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2023-08-31 19:18:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2023-08-31 19:18:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      2023-08-31 19:18:50 0 [Note] InnoDB: 10.6.11 started; log sequence number 1484110312804; transaction id 60758841
      2023-08-31 19:18:50 0 [Note] InnoDB: Loading buffer pool(s) from /data/mariadata/mysql/ib_buffer_pool
      2023-08-31 19:18:50 0 [Note] Plugin 'FEEDBACK' is disabled.
      2023-08-31 19:18:50 0 [Note] Server socket created on IP: '0.0.0.0'.
      2023-08-31 19:18:50 0 [Note] WSREP: wsrep_init_schema_and_SR (nil)
      2023-08-31 19:18:50 0 [Note] WSREP: Server initialized
      2023-08-31 19:18:50 0 [Note] WSREP: Server status change initializing -> initialized
      2023-08-31 19:18:50 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:50 3 [Note] WSREP: Recovered position from storage: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:50 3 [Note] WSREP: Recovered view from SST:
      id: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16024761
      status: primary
      protocol_version: 4
      capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      final: no
      own_index: -1
      members(3):
      0: 213ecf6f-3f1b-11ee-a3a9-1b9e106b662b, blp21152017
      1: 902a3da4-3f15-11ee-af60-b706fe188821, blp21152018
      2: a564b541-3eca-11ee-b726-a2aa4392ef31, blp21152016

      2023-08-31 19:18:50 6 [Note] WSREP: Starting applier thread 6
      2023-08-31 19:18:50 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:50 0 [Note] /usr/sbin/mariadbd: ready for connections.
      Version: '10.6.11-MariaDB-log' socket: '/data/mariadata/mysql/mysql.sock' port: 3306 MariaDB Server
      2023-08-31 19:18:50 9 [Note] WSREP: Starting applier thread 9
      2023-08-31 19:18:50 11 [Note] WSREP: Recovered cluster id c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05
      2023-08-31 19:18:50 7 [Note] WSREP: Starting applier thread 7
      2023-08-31 19:18:50 3 [Note] WSREP: SST received: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:50 3 [Note] WSREP: SST succeeded for position c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:50 0 [Note] WSREP: Joiner monitor thread ended with total time 2 sec
      2023-08-31 19:18:50 2 [Note] WSREP: Installed new state from SST: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049854
      2023-08-31 19:18:50 2 [Note] WSREP: Receiving IST: 12 writesets, seqnos 16049855-16049866
      2023-08-31 19:18:50 0 [Note] WSREP: ####### IST applying starts with 16049855
      2023-08-31 19:18:50 0 [Note] WSREP: ####### IST current seqno initialized to 16049855
      2023-08-31 19:18:50 0 [Note] WSREP: Receiving IST... 0.0% ( 0/12 events) complete.
      2023-08-31 19:18:50 0 [Note] WSREP: IST preload starting at 16049855
      2023-08-31 19:18:50 0 [Note] WSREP: Service thread queue flushed.
      2023-08-31 19:18:50 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:16049854, protocol version: 5
      2023-08-31 19:18:50 0 [Note] InnoDB: Buffer pool(s) load completed at 230831 19:18:50
      2023-08-31 19:18:51 0 [Note] WSREP: (213ecf6f-a3aa, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2023-08-31 19:18:51 0 [Note] WSREP: REPL Protocols: 10 (5)
      2023-08-31 19:18:51 0 [Note] WSREP: ####### Adjusting cert position: 16049862 -> 16049863
      2023-08-31 19:18:51 0 [Note] WSREP: Service thread queue flushed.
      2023-08-31 19:18:51 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 16049855
      2023-08-31 19:18:51 0 [Note] WSREP: Min available from gcache for CC from ist: 16049855
      2023-08-31 19:18:51 9 [Note] WSREP: ================================================
      View:
      id: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049863
      status: primary
      protocol_version: 4
      capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      final: no
      own_index: -1
      members(2):
      0: 902a3da4-3f15-11ee-af60-b706fe188821, blp21152018
      1: a564b541-3eca-11ee-b726-a2aa4392ef31, blp21152016
      =================================================
      2023-08-31 19:18:51 9 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:51 0 [Note] WSREP: REPL Protocols: 10 (5)
      2023-08-31 19:18:51 0 [Note] WSREP: ####### Adjusting cert position: 16049865 -> 16049866
      2023-08-31 19:18:51 0 [Note] WSREP: Service thread queue flushed.
      2023-08-31 19:18:51 0 [Note] WSREP: Lowest cert index boundary for CC from ist: 16049855
      2023-08-31 19:18:51 0 [Note] WSREP: Min available from gcache for CC from ist: 16049855
      2023-08-31 19:18:51 0 [Note] WSREP: Receiving IST...100.0% (12/12 events) complete.
      2023-08-31 19:18:51 6 [Note] WSREP: ================================================
      View:
      id: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049866
      status: primary
      protocol_version: 4
      capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      final: no
      own_index: 0
      members(3):
      0: 213ecf6f-3f1b-11ee-a3aa-1b9e106b662b, blp21152017
      1: 902a3da4-3f15-11ee-af60-b706fe188821, blp21152018
      2: a564b541-3eca-11ee-b726-a2aa4392ef31, blp21152016
      =================================================
      2023-08-31 19:18:51 6 [Note] WSREP: Server status change initialized -> joined
      2023-08-31 19:18:51 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:51 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2023-08-31 19:18:51 2 [Note] WSREP: Draining apply monitors after IST up to 16049866
      2023-08-31 19:18:51 2 [Note] WSREP: IST received: c9a3c950-7d2f-11ed-9e25-5fc70c4f0d05:16049866
      2023-08-31 19:18:51 2 [Note] WSREP: Lowest cert index boundary for CC from sst: 16049855
      2023-08-31 19:18:51 2 [Note] WSREP: Min available from gcache for CC from sst: 16049855
      2023-08-31 19:18:51 0 [Note] WSREP: 0.0 (blp21152017): State transfer from 2.0 (blp21152016) complete.
      2023-08-31 19:18:51 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 16049867)
      2023-08-31 19:18:51 0 [Note] WSREP: Processing event queue:... -nan% (0/0 events) complete.
      2023-08-31 19:18:51 0 [Note] WSREP: Member 0.0 (blp21152017) synced with group.
      2023-08-31 19:18:51 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
      2023-08-31 19:18:51 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 16049867)
      2023-08-31 19:18:52 7 [Note] WSREP: Server blp21152017 synced with group
      2023-08-31 19:18:52 7 [Note] WSREP: Server status change joined -> synced
      2023-08-31 19:18:52 7 [Note] WSREP: Synchronized with group, ready for connections
      2023-08-31 19:18:52 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

      Attachments

        1. bug blp21152018 31st.txt
          25 kB
          PITTA NEELIMA
        2. bug blp21152018.txt
          24 kB
          PITTA NEELIMA
        3. bug blp21152017 31st.txt
          24 kB
          PITTA NEELIMA
        4. bug blp21152016 12th Sep.txt
          24 kB
          PITTA NEELIMA

        Issue Links

          Activity

            People

              sysprg Julius Goryavsky
              Neelima PITTA NEELIMA
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.