[MDEV-25605] Failed to initialize wsrep provider - Galera Created: 2021-05-05  Updated: 2021-09-03  Resolved: 2021-08-16

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.19, 10.5.8, 10.5.9
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Hayden Seitz Assignee: Alexey
Resolution: Duplicate Votes: 0
Labels: galera
Environment:

Ubuntu 20.04
galera-4, 26.4.7-focal


Issue Links:
Relates
relates to MDEV-25739 Failed to create a new provider '/usr... Closed

 Description   

After recently running `apt full-upgrade -y` on one of my Galera cluster nodes, mariadb is unable to start and I am getting the below error:

2021-05-05 11:42:02 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf:874857343
2021-05-05 11:42:02 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2021-05-05 11:42:02 0 [Note] WSREP: wsrep_load(): Galera 4.7(ree4f10fc) by Codership Oy <info@codership.com> loaded successfully.
2021-05-05 11:42:02 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2021-05-05 11:42:02 0 [Note] WSREP: Found saved state: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf:-1, safe_to_bootstrap: 0
2021-05-05 11:42:02 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf
Seqno: -1 - -1
Offset: -1
Synced: 0
2021-05-05 11:42:02 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf, offset: -1
2021-05-05 11:42:02 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (         0/2147483672 bytes) complete.
2021-05-05 11:42:02 0 [ERROR] WSREP: deque::_M_new_elements_at_back
2021-05-05 11:42:02 0 [ERROR] WSREP: Failed to create a new provider '/usr/lib/galera/libgalera_smm.so' with options 'gcache.size=2G; gcache.page_size=2G; gcs.fc_limit=160; gcs.fc_factor=0.8; ': Failed to initialize wsrep provider
2021-05-05 11:42:02 0 [ERROR] WSREP: Failed to load provider
2021-05-05 11:42:02 0 [ERROR] Aborting

I get the same error when removing all wsrep_provider_options from any mariadb config file:

2021-05-05 11:46:33 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf:874857343
2021-05-05 11:46:33 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2021-05-05 11:46:33 0 [Note] WSREP: wsrep_load(): Galera 4.7(ree4f10fc) by Codership Oy <info@codership.com> loaded successfully.
2021-05-05 11:46:33 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2021-05-05 11:46:33 0 [Note] WSREP: Found saved state: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf:-1, safe_to_bootstrap: 0
2021-05-05 11:46:33 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf
Seqno: -1 - -1
Offset: -1
Synced: 0
2021-05-05 11:46:33 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 586b8c12-a6d2-11eb-a44d-263f7a10ffcf, offset: -1
2021-05-05 11:46:33 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
2021-05-05 11:46:33 0 [ERROR] WSREP: deque::_M_new_elements_at_back
2021-05-05 11:46:33 0 [ERROR] WSREP: Failed to create a new provider '/usr/lib/galera/libgalera_smm.so' with options '': Failed to initialize wsrep provider
2021-05-05 11:46:33 0 [ERROR] WSREP: Failed to load provider
2021-05-05 11:46:33 0 [ERROR] Aborting

Unfortunately I tried on 2 of the 3 nodes, so now I'm down to a single node



 Comments   
Comment by Hayden Seitz [ 2021-05-05 ]

I have manually removed /var/lib/mysql/galera.cache, and the other nodes have started and resynced. I will try to replicate the error again.

Generated at Thu Feb 08 09:38:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.