Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25739

Failed to create a new provider '/usr/lib/galera/libgalera_smm.so' with options

    XMLWordPrintable

    Details

      Description

      After upgrade to 10.5.10 and also on 10.5.9 we started to see following issue where node will never sync back to cluster and end in failed state

      jaro@cv-sqa-us-east4-k8s-lmgmt-a:~$ kubectl logs mysql-2 -n sde mysql
      2021/05/20 07:04:07 Peer list updated
      was []
      now [mysql-0.mysql.sde.svc.cluster.local mysql-1.mysql.sde.svc.cluster.local mysql-2.mysql.sde.svc.cluster.local]
      2021/05/20 07:04:07 execing: /opt/galera/on-start.sh with stdin: mysql-0.mysql.sde.svc.cluster.local
      mysql-1.mysql.sde.svc.cluster.local
      mysql-2.mysql.sde.svc.cluster.local
      2021/05/20 07:04:07 *** [Galera] Joining cluster: mysql-0.mysql.sde.svc.cluster.local,mysql-1.mysql.sde.svc.cluster.local
      2021/05/20 07:04:08 Peer finder exiting
      Galera - Determining recovery position...
      galera-recovery.sh: Attempting to recover GTID positon...
      2021-05-20  7:04:08 0 [Note] mysqld (mysqld 10.5.10-MariaDB-1:10.5.10+maria~focal) starting as process 48 ...
      galera-recovery.sh: Found WSREP position: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085:19951024
      Galera recovery position: --wsrep_start_position=6c9afcd7-96b7-11ea-96a5-76e81fcbb085:19951024
      2021-05-20  7:04:09 0 [Note] mysqld (mysqld 10.5.10-MariaDB-1:10.5.10+maria~focal) starting as process 1 ...
      2021-05-20  7:04:09 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085:19951024
      2021-05-20  7:04:09 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
      2021-05-20  7:04:09 0 [Note] WSREP: wsrep_load(): Galera 26.4.8(r902dd268) by Codership Oy <info@codership.com> loaded successfully.
      2021-05-20  7:04:09 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
      2021-05-20  7:04:09 0 [Note] WSREP: Found saved state: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085:-1, safe_to_bootstrap: 1
      2021-05-20  7:04:09 0 [Note] WSREP: GCache DEBUG: opened preamble:
      Version: 2
      UUID: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085
      Seqno: -1 - -1
      Offset: -1
      Synced: 0
      2021-05-20  7:04:09 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 6c9afcd7-96b7-11ea-96a5-76e81fcbb085, offset: -1
      2021-05-20  7:04:09 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
      2021-05-20  7:04:09 0 [ERROR] WSREP: deque::_M_new_elements_at_back
      2021-05-20  7:04:09 0 [ERROR] WSREP: Failed to create a new provider '/usr/lib/galera/libgalera_smm.so' with options '': Failed to initialize wsrep provider
      2021-05-20  7:04:09 0 [ERROR] WSREP: Failed to load provider
      2021-05-20  7:04:09 0 [ERROR] Aborting
      

      On the other nodes we also could see

      2021/05/20 07:37:08 Peer list updated
      was []
      now [mysql-0.mysql.default.svc.cluster.local mysql-1.mysql.default.svc.cluster.local mysql-2.mysql.default.svc.cluster.local]
      2021/05/20 07:37:08 execing: /opt/galera/on-start.sh with stdin: mysql-0.mysql.default.svc.cluster.local
      mysql-1.mysql.default.svc.cluster.local
      mysql-2.mysql.default.svc.cluster.local
      2021/05/20 07:37:08 *** [Galera] Joining cluster: mysql-1.mysql.default.svc.cluster.local,mysql-2.mysql.default.svc.cluster.local
      2021/05/20 07:37:09 Peer finder exiting
      Galera - Determining recovery position...
      galera-recovery.sh: Attempting to recover GTID positon...
      2021-05-20  7:37:09 0 [Note] mysqld (mysqld 10.5.9-MariaDB-1:10.5.9+maria~focal) starting as process 49 ...
      galera-recovery.sh: Found WSREP position: 8338b624-66eb-11eb-93e0-323a1dc8d4de:8714889
      Galera recovery position: --wsrep_start_position=8338b624-66eb-11eb-93e0-323a1dc8d4de:8714889
      2021-05-20  7:37:10 0 [Note] mysqld (mysqld 10.5.9-MariaDB-1:10.5.9+maria~focal) starting as process 1 ...
      2021-05-20  7:37:10 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 8338b624-66eb-11eb-93e0-323a1dc8d4de:8714889
      2021-05-20  7:37:10 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
      2021-05-20  7:37:10 0 [Note] WSREP: wsrep_load(): Galera 4.7(ree4f10fc) by Codership Oy <info@codership.com> loaded successfully.
      2021-05-20  7:37:10 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
      2021-05-20  7:37:10 0 [Note] WSREP: Found saved state: 8338b624-66eb-11eb-93e0-323a1dc8d4de:-1, safe_to_bootstrap: 1
      2021-05-20  7:37:10 0 [Note] WSREP: GCache DEBUG: opened preamble:
      Version: 2
      UUID: 8338b624-66eb-11eb-93e0-323a1dc8d4de
      Seqno: -1 - -1
      Offset: -1
      Synced: 0
      2021-05-20  7:37:10 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 8338b624-66eb-11eb-93e0-323a1dc8d4de, offset: -1
      2021-05-20  7:37:10 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
      2021-05-20  7:37:10 0 [ERROR] WSREP: std::bad_alloc
      2021-05-20  7:37:10 0 [ERROR] WSREP: Failed to create a new provider '/usr/lib/galera/libgalera_smm.so' with options '': Failed to initialize wsrep provider
      2021-05-20  7:37:10 0 [ERROR] WSREP: Failed to load provider
      2021-05-20  7:37:10 0 [ERROR] Aborting
      

      There is nothing we can do just to delete the disk for the node and let it fully resync. From linked tickets it seems the /var/lib/mysql/galera.cache got corrupted somehow and deleting it "solves" the issue.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Yurchenko Alexey
              Reporter:
              jaroslav Jaroslav
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.