Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28256

Galera SST fails with 'InnoDB: Upgrade after a crash is not supported'

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.5.15
    • N/A
    • Galera SST
    • Ubuntu 20.04 LTS

    Description

      Hi, I'm trying to join a new 10.5 node to a 10.4 Galera cluster as part of a migration to new AWS instance types.

      The symptom is that the initial SST completes, but on the restart the joining node then fails with this error:

      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.
      

      After various Googling this seems to be related to crash recovery files ib_logfile0 and ib_logfile1, which don't exist on the empty JOINING node until the SST completes. The DONOR node was cleanly restarted prior to the sync. I'm not sure if this is a known issue?

      Things I have already tried:

      • Setting innodb_fast_shutdown=0 on DONOR and JOINING nodes
      • Deleting ib_logfile0 and ib_logfile1 on the JOINER and restarting

      Is there a known work-around for this? I'm cautious to try upgrading an existing node to 10.5 in case it has the same error.

      Galera Config

      [mysqld]
      binlog_format=ROW
      default-storage-engine=innodb
      innodb_autoinc_lock_mode=2
      query_cache_type=0
      query_cache_size=0
      bind-address=0.0.0.0
       
      [galera]
      wsrep_on=ON
       
      # Galera Provider Configuration
      wsrep_provider=/usr/lib/galera/libgalera_smm.so
      wsrep_provider_options="gcache.size=5G"
       
      # Galera Cluster Configuration
      wsrep_cluster_name="mariadb-cluster"
      wsrep_cluster_address="gcomm://172.31.34.27,172.31.35.130,172.31.37.249,172.31.40.66,172.31.37.12"
       
      # Galera Synchronization Congifuration
      wsrep_sst_method = rsync
      wsrep_sst_donor = ES5_AWS
       
      # Galera Node Configuration
      wsrep_node_name="ES6_AWS"
      wsrep_node_address="172.31.39.121"
      

      JOINER Logs

      Apr  7 10:35:46 ip-172-31-39-121 systemd[1]: Starting MariaDB 10.5.15 database server...
      Apr  7 10:35:47 ip-172-31-39-121 sh[129656]: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-1:10.5.15+maria~focal) starting as process 129747 ...
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: wsrep_load(): Galera 26.4.11(r67341d07) by Codership Oy <info@codership.com> loaded successfully.
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: CRC-32C: using hardware acceleration.
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GCache DEBUG: opened preamble:
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Version: 0
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: UUID: 00000000-0000-0000-0000-000000000000
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Seqno: -1 - -1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Offset: -1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: Synced: 0
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead.
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.39.121; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 5G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0;
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Start replication
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Connecting with bootstrap option: 0
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: protonet asio version 0
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Using CRC-32C for message checksums.
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: backend: asio
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: restore pc from disk failed
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: GMCast version 0
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: EVS version 1
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: gcomm: connecting to group 'mariadb-cluster', peer '172.31.34.27:,172.31.35.130:,172.31.37.249:,172.31.40.66:,172.31.37.12:'
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 95267912-98ce tcp://172.31.34.27:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.37.75:4567 tcp://172.31.9.176:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 98319f4a-8b85 tcp://172.31.35.130:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to da9e7474-8746 tcp://172.31.40.66:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to e4e23eea-bbd0 tcp://172.31.37.249:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to 44508de1-af53 tcp://172.31.37.75:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: (7c67946e-b045, 'tcp://0.0.0.0:4567') connection established to eb5f53de-8588 tcp://172.31.9.176:4567
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:39010
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0xaaab09f1f940
      Apr  7 10:35:47 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:47 0 [Note] WSREP: Deferred close timer destruct
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: EVS version upgrade 0 -> 1
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 44508de1-af53 at tcp://172.31.37.75:4567 stable
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
      Apr  7 10:35:48 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:48 0 [Note] WSREP: PC protocol upgrade 0 -> 1
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01144508de1-af53,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #0117c67946e-b045,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01195267912-98ce,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #01198319f4a-8b85,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011da9e7474-8746,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011e4e23eea-bbd0,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011eb5f53de-8588,0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } joined {
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } left {
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: } partitioned {
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: })
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: save pc into disk
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.37.12:4567
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: discarding pending addr proto entry 0xaaab09eff5a0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: gcomm: connected
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Opened channel 'mariadb-cluster'
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 7
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 1 [Note] WSREP: Starting rollbacker thread 1
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Starting applier thread 2
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Quorum results:
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011version    = 6,
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011component  = PRIMARY,
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011conf_id    = 363,
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011members    = 6/7 (joined/total),
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011act_id     = 5064032831,
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011last_appl. = 5064032726,
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011protocols  = 2/10/4 (gcs/repl/appl),
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011vote policy= 0,
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 5064032832)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Process first view: 1d4206f9-647d-11ea-a963-0ed98924f90b my uuid: 7c67946e-b65e-11ec-b045-eb2b92474255
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server ES6_AWS connected to cluster at position 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 with ID 7c67946e-b65e-11ec-b045-eb2b92474255
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change disconnected -> connected
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### My UUID: 7c67946e-b65e-11ec-b045-eb2b92474255
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Service thread queue flushed.
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: State transfer required:
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Group state: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: #011Local state: 00000000-0000-0000-0000-000000000000:-1
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Server status change connected -> joiner
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Joiner monitor thread started to monitor
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.39.121' --datadir '/var/lib/mysql/' --parent '129747' --mysqld-args --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'
      Apr  7 10:35:49 ip-172-31-39-121 rsyncd[129866]: rsyncd version 3.1.3 starting, listening on port 4444
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 5064032832, STRv: 3
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: IST receiver addr using tcp://172.31.39.121:4568
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Prepared IST receiver for 0-5064032832, listening at: tcp://172.31.39.121:4568
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5064032832)
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Requesting state transfer: success, donor: 0
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
      Apr  7 10:35:49 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:35:49 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
      Apr  7 10:35:50 ip-172-31-39-121 rsyncd[129878]: connect from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)
      Apr  7 10:35:50 ip-172-31-39-121 rsyncd[129878]: rsync to rsync_sst/ from ip-172-31-37-75.eu-west-1.compute.internal (172.31.37.75)
       
      ...
       
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup: rsync PID=129866, stunnel PID=0 (20220407 10:39:52.306)
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.
      Apr  7 10:39:52 ip-172-31-39-121 rsyncd[129866]: sent 0 bytes  received 0 bytes  total size 0
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129757]: WSREP_SST: [INFO] Joiner cleanup done. (20220407 10:39:52.825)
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: SST received
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: Server status change joiner -> initializing
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Uses event mutexes
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Number of pools: 1
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Using Linux native AIO
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Completed initialization of buffer pool
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.22.
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      Apr  7 10:39:52 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:52 0 [Note] InnoDB: Starting shutdown...
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' init function returned error.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [Note] Plugin 'FEEDBACK' is disabled.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Unknown/unsupported storage engine: innodb
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 2022-04-07 10:39:53 0 [ERROR] Aborting
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terminate called after throwing an instance of 'wsrep::runtime_error'
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]:   what():  State wait was interrupted
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: 220407 10:39:53 [ERROR] mysqld got signal 6 ;
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: This could be because you hit a bug. It is also possible that this binary
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or one of the libraries it was linked against is corrupt, improperly built,
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We will try our best to scrape up some info that will hopefully help
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: diagnose the problem, but since we have already crashed,
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: something is definitely wrong and this may fail.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Server version: 10.5.15-MariaDB-1:10.5.15+maria~focal
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size=134217728
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: read_buffer_size=131072
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_used_connections=0
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: max_threads=153
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: thread_count=4
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: It is possible that mysqld could use up to
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467879 K  bytes of memory
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Hope that's ok; if not, decrease some variables in the equation.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Thread pointer: 0xfffe1c002068
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Attempting backtrace. You can use the following information to find out
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: where mysqld died. If you see no messages after this, something went
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: terribly wrong...
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: stack_bottom = 0xfffe42579630 thread_stack 0x49000
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Printing to addr2line failed
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(my_print_stacktrace+0x30)[0xaaaacc7023f0]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(handle_fatal_signal+0x45c)[0xaaaacc1b137c]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff89d95794]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(gsignal+0xe0)[0xffff8942dc20]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(abort+0xfc)[0xffff8941aa14]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x19c)[0xffff896b88cc]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f21c)[0xffff896b621c]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f280)[0xffff896b6280]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/libstdc++.so.6(+0x9f574)[0xffff896b6574]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZNK5wsrep12server_state16wait_until_stateERSt11unique_lockINS_5mutexEENS0_5stateE+0x1dc)[0xaaaacc7808ec]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(_ZN5wsrep12server_state12sst_receivedERNS_14client_serviceEi+0xb04)[0xaaaacc78a5f4]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7b6bc)[0xaaaacc43b6bc]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb7c324)[0xaaaacc43c324]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /usr/sbin/mariadbd(+0xb0bca0)[0xaaaacc3cbca0]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libpthread.so.0(+0x73f0)[0xffff898173f0]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: /lib/aarch64-linux-gnu/atomics/libc.so.6(+0xd00dc)[0xffff894ca0dc]
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Trying to get some variables.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Some pointers may be invalid and cause the dump to abort.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query (0x0): (null)
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Connection ID (thread ID): 3
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Status: NOT_KILLED
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: information that should help you find out what is causing the crash.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: We think the query pointer is invalid, but we will try to print it anyway.
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Query:
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Writing a core file...
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Working directory at /media/storage/mysql
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Resource Limits:
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Limit                     Soft Limit           Hard Limit           Units
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max cpu time              unlimited            unlimited            seconds
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file size             unlimited            unlimited            bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max data size             unlimited            unlimited            bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max stack size            8388608              unlimited            bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max core file size        0                    unlimited            bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max resident set          unlimited            unlimited            bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max processes             125387               125387               processes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max open files            32768                32768                files
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max locked memory         65536                65536                bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max address space         unlimited            unlimited            bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max file locks            unlimited            unlimited            locks
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max pending signals       125387               125387               signals
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max msgqueue size         819200               819200               bytes
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max nice priority         0                    0
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime priority     0                    0
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Max realtime timeout      unlimited            unlimited            us
      Apr  7 10:39:53 ip-172-31-39-121 mariadbd[129747]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
      Apr  7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
      Apr  7 10:39:53 ip-172-31-39-121 systemd[1]: mariadb.service: Failed with result 'core-dump'.
      Apr  7 10:39:53 ip-172-31-39-121 systemd[1]: Failed to start MariaDB 10.5.15 database server.

      DONOR Logs

      Apr  7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567
      Apr  7 11:35:47 es5 mysqld[29230]: 2022-04-07 11:35:47 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection established to 7c67946e-b045 tcp://172.31.39.121:4567
      Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 7c67946e-b045 at tcp://172.31.39.121:4567 stable
      Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
      Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
      Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
      Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
      Apr  7 11:35:48 es5 mysqld[29230]: 2022-04-07 11:35:48 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Node 44508de1-af53 state prim
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,378) memb {
      Apr  7 11:35:49 es5 mysqld[29230]: #01144508de1-af53,0
      Apr  7 11:35:49 es5 mysqld[29230]: #0117c67946e-b045,0
      Apr  7 11:35:49 es5 mysqld[29230]: #01195267912-98ce,0
      Apr  7 11:35:49 es5 mysqld[29230]: #01198319f4a-8b85,0
      Apr  7 11:35:49 es5 mysqld[29230]: #011da9e7474-8746,0
      Apr  7 11:35:49 es5 mysqld[29230]: #011e4e23eea-bbd0,0
      Apr  7 11:35:49 es5 mysqld[29230]: #011eb5f53de-8588,0
      Apr  7 11:35:49 es5 mysqld[29230]: } joined {
      Apr  7 11:35:49 es5 mysqld[29230]: } left {
      Apr  7 11:35:49 es5 mysqld[29230]: } partitioned {
      Apr  7 11:35:49 es5 mysqld[29230]: })
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: save pc into disk
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 7
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7dcd956b-b65e-11ec-8734-924b8afeca4d
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 0 (ES5_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 2 (ES1_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 3 (ES2_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 4 (ES4_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 5 (ES3_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 6 (BACKUP_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: STATE EXCHANGE: got state msg: 7dcd956b-b65e-11ec-8734-924b8afeca4d from 1 (ES6_AWS)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Quorum results:
      Apr  7 11:35:49 es5 mysqld[29230]: #011version    = 6,
      Apr  7 11:35:49 es5 mysqld[29230]: #011component  = PRIMARY,
      Apr  7 11:35:49 es5 mysqld[29230]: #011conf_id    = 363,
      Apr  7 11:35:49 es5 mysqld[29230]: #011members    = 6/7 (joined/total),
      Apr  7 11:35:49 es5 mysqld[29230]: #011act_id     = 5064032831,
      Apr  7 11:35:49 es5 mysqld[29230]: #011last_appl. = 5064032726,
      Apr  7 11:35:49 es5 mysqld[29230]: #011protocols  = 2/10/4 (gcs/repl/appl),
      Apr  7 11:35:49 es5 mysqld[29230]: #011vote policy= 0,
      Apr  7 11:35:49 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flow-control interval: [42, 42]
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### processing CC 5064032832, local, ordered
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Skipping cert index reset
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: REPL Protocols: 10 (5)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ####### Adjusting cert position: 5064032831 -> 5064032832
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Service thread queue flushed.
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: ================================================
      Apr  7 11:35:49 es5 mysqld[29230]: View:
      Apr  7 11:35:49 es5 mysqld[29230]:   id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
      Apr  7 11:35:49 es5 mysqld[29230]:   status: primary
      Apr  7 11:35:49 es5 mysqld[29230]:   protocol_version: 4
      Apr  7 11:35:49 es5 mysqld[29230]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      Apr  7 11:35:49 es5 mysqld[29230]:   final: no
      Apr  7 11:35:49 es5 mysqld[29230]:   own_index: 0
      Apr  7 11:35:49 es5 mysqld[29230]:   members(7):
      Apr  7 11:35:49 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: #0111: 7c67946e-b65e-11ec-b045-eb2b92474255, ES6_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: #0112: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: #0113: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: #0114: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: #0115: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: #0116: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS
      Apr  7 11:35:49 es5 mysqld[29230]: =================================================
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Member 1.0 (ES6_AWS) requested state transfer from 'ES5_AWS'. Selected 0.0 (ES5_AWS)(SYNCED) as donor.
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 5064032832)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Detected STR version: 1, req_len: 125, req: STRv1
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Cert index preload: 5064032727 -> 5064032832
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: Server status change synced -> donor
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: async IST sender starting to serve tcp://172.31.39.121:4568 sending 5064032727-5064032832, preload starts from 5064032727
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Donor monitor thread started to monitor
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.31.39.121:4444/rsync_sst' --local-port '3306' --socket '/var/run/mysqld/mysqld.sock' --datadir '/media/storage/mysql/' --gtid '1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832' --gtid-domain-id '0' --mysqld-args --wsrep_start_position=1d4206f9-647d-11ea-a963-0ed98924f90b:5063284262'
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: IST sender 5064032727 -> 5064032832
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 2 [Note] WSREP: sst_donor_thread signaled with 0
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Flushing tables for SST...
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: pause
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Provider paused at 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832 (778520)
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Server paused at: 5064032832
      Apr  7 11:35:49 es5 mysqld[29230]: 2022-04-07 11:35:49 0 [Note] WSREP: Tables flushed.
      Apr  7 11:35:50 es5 mysqld[29230]: 2022-04-07 11:35:50 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off
      Apr  7 11:39:01 es5 CRON[10599]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
      Apr  7 11:39:05 es5 systemd[1]: Starting Clean php session files...
      Apr  7 11:39:06 es5 systemd[1]: Started Clean php session files.
      Apr  7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Warning] InnoDB: A long semaphore wait:
      Apr  7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:
      Apr  7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2
      Apr  7 11:39:51 es5 mysqld[29230]: 2022-04-07 11:39:51 0 [Note] InnoDB: A semaphore wait:
      Apr  7 11:39:51 es5 mysqld[29230]: --Thread 139702254601984 has waited at log0log.cc line 882 for 241.00 seconds the semaphore:
      Apr  7 11:39:51 es5 mysqld[29230]: Mutex at 0x55fd5cf32080, Mutex LOG_WRITE created log0log.cc:537, lock var 2
      Apr  7 11:39:51 es5 mysqld[29230]: InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
      Apr  7 11:39:51 es5 mysqld[29230]: InnoDB: Pending reads 0, writes 0
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resume
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: resuming provider at 778520
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Provider resumed.
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: SST sent: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064032832
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Server status change donor -> joined
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Donor monitor thread ended with total time 243 sec
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: 0.0 (ES5_AWS): State transfer to 1.0 (ES6_AWS) complete.
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 5064034794)
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Member 0.0 (ES5_AWS) synced with group.
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5064034794)
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server ES5_AWS synced with group
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Server status change joined -> synced
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: Synchronized with group, ready for connections
      Apr  7 11:39:52 es5 mysqld[29230]: 2022-04-07 11:39:52 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.39.121:4567
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 95267912-98ce at tcp://172.31.34.27:4567 stable
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring 98319f4a-8b85 at tcp://172.31.35.130:4567 stable
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring da9e7474-8746 at tcp://172.31.40.66:4567 stable
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring e4e23eea-bbd0 at tcp://172.31.37.249:4567 stable
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: declaring eb5f53de-8588 at tcp://172.31.9.176:4567 stable
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting off
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Node 44508de1-af53 state prim
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: view(view_id(PRIM,44508de1-af53,379) memb {
      Apr  7 11:39:53 es5 mysqld[29230]: #01144508de1-af53,0
      Apr  7 11:39:53 es5 mysqld[29230]: #01195267912-98ce,0
      Apr  7 11:39:53 es5 mysqld[29230]: #01198319f4a-8b85,0
      Apr  7 11:39:53 es5 mysqld[29230]: #011da9e7474-8746,0
      Apr  7 11:39:53 es5 mysqld[29230]: #011e4e23eea-bbd0,0
      Apr  7 11:39:53 es5 mysqld[29230]: #011eb5f53de-8588,0
      Apr  7 11:39:53 es5 mysqld[29230]: } joined {
      Apr  7 11:39:53 es5 mysqld[29230]: } left {
      Apr  7 11:39:53 es5 mysqld[29230]: } partitioned {
      Apr  7 11:39:53 es5 mysqld[29230]: #0117c67946e-b045,0
      Apr  7 11:39:53 es5 mysqld[29230]: })
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: save pc into disk
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: forgetting 7c67946e-b045 (tcp://172.31.39.121:4567)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 6
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0f078c2c-b65f-11ec-b06d-de758fe27da8
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 0 (ES5_AWS)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 1 (ES1_AWS)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 2 (ES2_AWS)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 3 (ES4_AWS)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 4 (ES3_AWS)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: STATE EXCHANGE: got state msg: 0f078c2c-b65f-11ec-b06d-de758fe27da8 from 5 (BACKUP_AWS)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Quorum results:
      Apr  7 11:39:53 es5 mysqld[29230]: #011version    = 6,
      Apr  7 11:39:53 es5 mysqld[29230]: #011component  = PRIMARY,
      Apr  7 11:39:53 es5 mysqld[29230]: #011conf_id    = 364,
      Apr  7 11:39:53 es5 mysqld[29230]: #011members    = 6/6 (joined/total),
      Apr  7 11:39:53 es5 mysqld[29230]: #011act_id     = 5064034794,
      Apr  7 11:39:53 es5 mysqld[29230]: #011last_appl. = 5064034689,
      Apr  7 11:39:53 es5 mysqld[29230]: #011protocols  = 2/10/4 (gcs/repl/appl),
      Apr  7 11:39:53 es5 mysqld[29230]: #011vote policy= 0,
      Apr  7 11:39:53 es5 mysqld[29230]: #011group UUID = 1d4206f9-647d-11ea-a963-0ed98924f90b
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Flow-control interval: [39, 39]
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### processing CC 5064034795, local, ordered
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### My UUID: 44508de1-b657-11ec-af53-dbbee42dcffe
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Skipping cert index reset
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: REPL Protocols: 10 (5)
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ####### Adjusting cert position: 5064034794 -> 5064034795
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: Service thread queue flushed.
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: ================================================
      Apr  7 11:39:53 es5 mysqld[29230]: View:
      Apr  7 11:39:53 es5 mysqld[29230]:   id: 1d4206f9-647d-11ea-a963-0ed98924f90b:5064034795
      Apr  7 11:39:53 es5 mysqld[29230]:   status: primary
      Apr  7 11:39:53 es5 mysqld[29230]:   protocol_version: 4
      Apr  7 11:39:53 es5 mysqld[29230]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      Apr  7 11:39:53 es5 mysqld[29230]:   final: no
      Apr  7 11:39:53 es5 mysqld[29230]:   own_index: 0
      Apr  7 11:39:53 es5 mysqld[29230]:   members(6):
      Apr  7 11:39:53 es5 mysqld[29230]: #0110: 44508de1-b657-11ec-af53-dbbee42dcffe, ES5_AWS
      Apr  7 11:39:53 es5 mysqld[29230]: #0111: 95267912-5df1-11ec-98ce-a66d469d045c, ES1_AWS
      Apr  7 11:39:53 es5 mysqld[29230]: #0112: 98319f4a-b5f9-11ec-8b85-e69cdab5035e, ES2_AWS
      Apr  7 11:39:53 es5 mysqld[29230]: #0113: da9e7474-5df2-11ec-8746-dfd73835a1c5, ES4_AWS
      Apr  7 11:39:53 es5 mysqld[29230]: #0114: e4e23eea-5df0-11ec-bbd0-8b132f61eb3d, ES3_AWS
      Apr  7 11:39:53 es5 mysqld[29230]: #0115: eb5f53de-882b-11ec-8588-1a528d6537ef, BACKUP_AWS
      Apr  7 11:39:53 es5 mysqld[29230]: =================================================
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Lowest cert index boundary for CC from group: 5064032727
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 2 [Note] WSREP: Min available from gcache for CC from group: 5063284333
      Apr  7 11:39:53 es5 mysqld[29230]: 2022-04-07 11:39:53 0 [Note] WSREP: async IST sender served
      Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') connection to peer eb5f53de-8588 with addr tcp://172.31.9.176:4567 timed out, no messages seen in PT3S, socket stats: rtt: 872 rttvar: 864 rto: 204000 lost: 0 last_data_recv: 3104 cwnd: 10 last_queued_since: 33142 last_delivered_since: 3103372544 send_queue_length: 1 send_queue_bytes: 804 segment: 0 messages: 1
      Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://172.31.9.176:60056
      Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: (44508de1-af53, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.9.176:4567
      Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x55fd61a9f5b0
      Apr  7 11:39:56 es5 mysqld[29230]: 2022-04-07 11:39:56 0 [Note] WSREP: Deferred close timer destruct

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              mattjones Matt Jones
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.