[MDEV-28627] "unknown variable" errors only detected after Galera startup already partially done Created: 2022-05-19  Updated: 2023-01-16  Resolved: 2023-01-16

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.24, 10.5.15, 10.6.7
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Hartmut Holzgraefe Assignee: Jan Lindström (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None


 Description   

When having an unknown variable in my.cnf or one of the option files included by it the server should just refuse to start (unless the loose-... prefix is being used).

Galera nodes do perform several initialization tasks before that already though, including potential IST / SST, and only fail later in the startup process.

Things get even worse with 10.4 and later where the starting server does not just exit after the "unknown variable" message, but has the critical signal handler catch a SIGABRT and perform crash handling including writing a core dump if configured.

This is the startup log output from 10.3 with an unknown variable in the config files. The node had already been part of the cluster, and has been stopped, the unknown variable added, and a restart attempted:

2022-05-19 14:23:39 0 [Note] mysqld (mysqld 10.3.34-MariaDB-1:10.3.34+maria~focal-log) starting as process 2938 ...
2022-05-19 14:23:39 0 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used;  This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=node-2' or '--log-bin=node-2-bin' to avoid this problem.
2022-05-19 14:23:39 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2022-05-19 14:23:39 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2022-05-19 14:23:39 0 [Note] WSREP: wsrep_load(): Galera 25.3.35(r545d0bf7) by Codership Oy <info@codership.com> loaded successfully.
2022-05-19 14:23:39 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2022-05-19 14:23:39 0 [Note] WSREP: Found saved state: a5f93485-d77e-11ec-946d-c23d83c7256f:0, safe_to_bootstrap: 0
2022-05-19 14:23:39 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = galera-node-2; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S
2022-05-19 14:23:39 0 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
2022-05-19 14:23:39 0 [Note] WSREP: wsrep_sst_grab()
2022-05-19 14:23:39 0 [Note] WSREP: Start replication
2022-05-19 14:23:39 0 [Note] WSREP: Setting initial position to a5f93485-d77e-11ec-946d-c23d83c7256f:0
2022-05-19 14:23:39 0 [Note] WSREP: protonet asio version 0
2022-05-19 14:23:39 0 [Note] WSREP: Using CRC-32C for message checksums.
2022-05-19 14:23:39 0 [Note] WSREP: backend: asio
2022-05-19 14:23:39 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 
2022-05-19 14:23:39 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2022-05-19 14:23:39 0 [Note] WSREP: restore pc from disk failed
2022-05-19 14:23:39 0 [Note] WSREP: GMCast version 0
2022-05-19 14:23:39 0 [Note] WSREP: (46ee191b, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2022-05-19 14:23:39 0 [Note] WSREP: (46ee191b, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2022-05-19 14:23:39 0 [Note] WSREP: EVS version 0
2022-05-19 14:23:39 0 [Note] WSREP: gcomm: connecting to group 'test_cluster', peer 'galera-node-1:,galera-node-2:'
2022-05-19 14:23:39 0 [Note] WSREP: (46ee191b, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.23.4.12:4567
2022-05-19 14:23:39 0 [Note] WSREP: (46ee191b, 'tcp://0.0.0.0:4567') connection established to a5f5c867 tcp://10.23.4.11:4567
2022-05-19 14:23:39 0 [Note] WSREP: (46ee191b, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2022-05-19 14:23:39 0 [Note] WSREP: declaring a5f5c867 at tcp://10.23.4.11:4567 stable
2022-05-19 14:23:39 0 [Note] WSREP: Node a5f5c867 state prim
2022-05-19 14:23:39 0 [Note] WSREP: view(view_id(PRIM,46ee191b,4) memb {
	46ee191b,0
	a5f5c867,0
} joined {
} left {
} partitioned {
})
2022-05-19 14:23:39 0 [Note] WSREP: save pc into disk
2022-05-19 14:23:39 0 [Note] WSREP: gcomm: connected
2022-05-19 14:23:39 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2022-05-19 14:23:39 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2022-05-19 14:23:39 0 [Note] WSREP: Opened channel 'test_cluster'
2022-05-19 14:23:39 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-05-19 14:23:39 0 [Note] WSREP: Waiting for SST to complete.
2022-05-19 14:23:39 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 473b320a-d77f-11ec-be60-67ccd75098fb
2022-05-19 14:23:39 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 473b320a-d77f-11ec-be60-67ccd75098fb
2022-05-19 14:23:39 0 [Note] WSREP: STATE EXCHANGE: got state msg: 473b320a-d77f-11ec-be60-67ccd75098fb from 0 (node-2)
2022-05-19 14:23:39 0 [Note] WSREP: STATE EXCHANGE: got state msg: 473b320a-d77f-11ec-be60-67ccd75098fb from 1 (node-1)
2022-05-19 14:23:39 0 [Note] WSREP: Quorum results:
	version    = 6,
	component  = PRIMARY,
	conf_id    = 3,
	members    = 2/2 (joined/total),
	act_id     = 0,
	last_appl. = -1,
	protocols  = 0/9/3 (gcs/repl/appl),
	group UUID = a5f93485-d77e-11ec-946d-c23d83c7256f
2022-05-19 14:23:39 0 [Note] WSREP: Flow-control interval: [23, 23]
2022-05-19 14:23:39 0 [Note] WSREP: Restored state OPEN -> JOINED (0)
2022-05-19 14:23:39 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2022-05-19 14:23:39 2 [Note] WSREP: New cluster view: global state: a5f93485-d77e-11ec-946d-c23d83c7256f:0, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3
2022-05-19 14:23:39 0 [Note] WSREP: Member 0.0 (node-2) synced with group.
2022-05-19 14:23:39 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
2022-05-19 14:23:39 0 [Note] WSREP: SST complete, seqno: 0
2022-05-19 14:23:39 0 [Note] InnoDB: Using Linux native AIO
2022-05-19 14:23:39 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-05-19 14:23:39 0 [Note] InnoDB: Uses event mutexes
2022-05-19 14:23:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-05-19 14:23:39 0 [Note] InnoDB: Number of pools: 1
2022-05-19 14:23:39 0 [Note] InnoDB: Using SSE2 crc32 instructions
2022-05-19 14:23:39 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2022-05-19 14:23:39 0 [Note] InnoDB: Completed initialization of buffer pool
2022-05-19 14:23:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-05-19 14:23:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2022-05-19 14:23:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-05-19 14:23:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-05-19 14:23:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-05-19 14:23:39 0 [Note] InnoDB: Waiting for purge to start
2022-05-19 14:23:39 0 [Note] InnoDB: 10.3.34 started; log sequence number 1626645; transaction id 20
2022-05-19 14:23:39 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-05-19 14:23:39 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-05-19 14:23:39 0 [ERROR] mysqld: unknown variable 'wsrep_ssl_mode=PROVIDER'
2022-05-19 14:23:39 0 [ERROR] Aborting

And this is the same with 10.4 with extra critical signal handler output. Later major versions up to 10.6 fail in a similar way (haven't tested 10.7 and 10.8 yet)

2022-05-19 14:08:00 0 [Note] mysqld (mysqld 10.4.24-MariaDB-1:10.4.24+maria~focal-log) starting as process 2813 ...
2022-05-19 14:08:00 0 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used;  This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=node-2' or '--log-bin=node-2-bin' to avoid this problem.
2022-05-19 14:08:00 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2022-05-19 14:08:00 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2022-05-19 14:08:00 0 [Note] WSREP: wsrep_load(): Galera 26.4.11(r67341d07) by Codership Oy <info@codership.com> loaded successfully.
2022-05-19 14:08:00 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2022-05-19 14:08:00 0 [Note] WSREP: Found saved state: ccf8501b-d77c-11ec-8321-ab80360be17f:2, safe_to_bootstrap: 0
2022-05-19 14:08:00 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: ccf8501b-d77c-11ec-8321-ab80360be17f
Seqno: 2 - 2
Offset: 1280
Synced: 1
2022-05-19 14:08:00 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: ccf8501b-d77c-11ec-8321-ab80360be17f, offset: 1280
2022-05-19 14:08:00 0 [Note] WSREP: GCache::RingBuffer initial scan...  0.0% (        0/134217752 bytes) complete.
2022-05-19 14:08:00 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
2022-05-19 14:08:00 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 2-2
2022-05-19 14:08:00 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...  0.0% (  0/208 bytes) complete.
2022-05-19 14:08:00 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers
2022-05-19 14:08:00 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 134217520/134217728
2022-05-19 14:08:00 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (208/208 bytes) complete.
2022-05-19 14:08:00 0 [Warning] WSREP: Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead. 
2022-05-19 14:08:00 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = galera-node-2; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0
2022-05-19 14:08:00 0 [Note] WSREP: Service thread queue flushed.
2022-05-19 14:08:00 0 [Note] WSREP: ####### Assign initial position for certification: ccf8501b-d77c-11ec-8321-ab80360be17f:2, protocol version: -1
2022-05-19 14:08:00 0 [Note] WSREP: Start replication
2022-05-19 14:08:00 0 [Note] WSREP: Connecting with bootstrap option: 0
2022-05-19 14:08:00 0 [Note] WSREP: Setting GCS initial position to ccf8501b-d77c-11ec-8321-ab80360be17f:2
2022-05-19 14:08:00 0 [Note] WSREP: protonet asio version 0
2022-05-19 14:08:00 0 [Note] WSREP: Using CRC-32C for message checksums.
2022-05-19 14:08:00 0 [Note] WSREP: backend: asio
2022-05-19 14:08:00 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 
2022-05-19 14:08:00 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2022-05-19 14:08:00 0 [Note] WSREP: restore pc from disk failed
2022-05-19 14:08:00 0 [Note] WSREP: GMCast version 0
2022-05-19 14:08:00 0 [Note] WSREP: (17624e50-9bd0, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2022-05-19 14:08:00 0 [Note] WSREP: (17624e50-9bd0, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2022-05-19 14:08:00 0 [Note] WSREP: EVS version 1
2022-05-19 14:08:00 0 [Note] WSREP: gcomm: connecting to group 'test_cluster', peer 'galera-node-1:,galera-node-2:'
2022-05-19 14:08:00 0 [Note] WSREP: (17624e50-9bd0, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.23.4.12:4567
2022-05-19 14:08:00 0 [Note] WSREP: (17624e50-9bd0, 'tcp://0.0.0.0:4567') connection established to ccf4d878-bbf3 tcp://10.23.4.11:4567
2022-05-19 14:08:00 0 [Note] WSREP: (17624e50-9bd0, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2022-05-19 14:08:00 0 [Note] WSREP: EVS version upgrade 0 -> 1
2022-05-19 14:08:00 0 [Note] WSREP: declaring ccf4d878-bbf3 at tcp://10.23.4.11:4567 stable
2022-05-19 14:08:00 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2022-05-19 14:08:00 0 [Note] WSREP: Node ccf4d878-bbf3 state prim
2022-05-19 14:08:00 0 [Note] WSREP: view(view_id(PRIM,17624e50-9bd0,4) memb {
	17624e50-9bd0,0
	ccf4d878-bbf3,0
} joined {
} left {
} partitioned {
})
2022-05-19 14:08:00 0 [Note] WSREP: save pc into disk
2022-05-19 14:08:01 0 [Note] WSREP: gcomm: connected
2022-05-19 14:08:01 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2022-05-19 14:08:01 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2022-05-19 14:08:01 0 [Note] WSREP: Opened channel 'test_cluster'
2022-05-19 14:08:01 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2022-05-19 14:08:01 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 17b51450-d77d-11ec-b1fd-3e239f91c17a
2022-05-19 14:08:01 1 [Note] WSREP: Starting applier thread 1
2022-05-19 14:08:01 2 [Note] WSREP: Starting rollbacker thread 2
2022-05-19 14:08:01 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 17b51450-d77d-11ec-b1fd-3e239f91c17a
2022-05-19 14:08:01 0 [Note] WSREP: STATE EXCHANGE: got state msg: 17b51450-d77d-11ec-b1fd-3e239f91c17a from 0 (node-2)
2022-05-19 14:08:01 0 [Note] WSREP: STATE EXCHANGE: got state msg: 17b51450-d77d-11ec-b1fd-3e239f91c17a from 1 (node-1)
2022-05-19 14:08:01 0 [Note] WSREP: Quorum results:
	version    = 6,
	component  = PRIMARY,
	conf_id    = 3,
	members    = 1/2 (joined/total),
	act_id     = 3,
	last_appl. = 0,
	protocols  = 2/10/4 (gcs/repl/appl),
	vote policy= 0,
	group UUID = ccf8501b-d77c-11ec-8321-ab80360be17f
2022-05-19 14:08:01 0 [Note] WSREP: Flow-control interval: [23, 23]
2022-05-19 14:08:01 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4)
2022-05-19 14:08:01 1 [Note] WSREP: ####### processing CC 4, local, ordered
2022-05-19 14:08:01 1 [Note] WSREP: Process first view: ccf8501b-d77c-11ec-8321-ab80360be17f my uuid: 17624e50-d77d-11ec-9bd0-a6be659d99bd
2022-05-19 14:08:01 1 [Note] WSREP: Server node-2 connected to cluster at position ccf8501b-d77c-11ec-8321-ab80360be17f:4 with ID 17624e50-d77d-11ec-9bd0-a6be659d99bd
2022-05-19 14:08:01 1 [Note] WSREP: Server status change disconnected -> connected
2022-05-19 14:08:01 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-05-19 14:08:01 1 [Note] WSREP: ####### My UUID: 17624e50-d77d-11ec-9bd0-a6be659d99bd
2022-05-19 14:08:01 1 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
2022-05-19 14:08:01 0 [Note] WSREP: Service thread queue flushed.
2022-05-19 14:08:01 1 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2022-05-19 14:08:01 1 [Note] WSREP: State transfer required: 
	Group state: ccf8501b-d77c-11ec-8321-ab80360be17f:4
	Local state: ccf8501b-d77c-11ec-8321-ab80360be17f:2
2022-05-19 14:08:01 1 [Note] WSREP: Server status change connected -> joiner
2022-05-19 14:08:01 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-05-19 14:08:01 0 [Note] WSREP: Joiner monitor thread started to monitor
2022-05-19 14:08:01 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'galera-node-2' --datadir '/var/lib/mysql/' --parent '2813' --binlog 'node-2-bin' --mysqld-args --user=mysql'
WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20220519 14:08:01.108)
WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20220519 14:08:01.188)
2022-05-19 14:08:01 1 [Note] WSREP: ####### IST uuid:ccf8501b-d77c-11ec-8321-ab80360be17f f: 3, l: 4, STRv: 3
2022-05-19 14:08:01 1 [Note] WSREP: IST receiver addr using tcp://galera-node-2:4568
2022-05-19 14:08:01 1 [Note] WSREP: Prepared IST receiver for 3-4, listening at: tcp://10.23.4.12:4568
2022-05-19 14:08:01 0 [Note] WSREP: Member 0.0 (node-2) requested state transfer from '*any*'. Selected 1.0 (node-1)(SYNCED) as donor.
2022-05-19 14:08:01 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4)
2022-05-19 14:08:01 1 [Note] WSREP: Requesting state transfer: success, donor: 1
2022-05-19 14:08:01 0 [Note] WSREP: 1.0 (node-1): State transfer to 0.0 (node-2) complete.
2022-05-19 14:08:01 0 [Note] WSREP: Member 1.0 (node-1) synced with group.
2022-05-19 14:08:01 3 [Note] WSREP: SST received
2022-05-19 14:08:01 3 [Note] WSREP: Server status change joiner -> initializing
2022-05-19 14:08:01 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-05-19 14:08:01 0 [Note] InnoDB: Using Linux native AIO
2022-05-19 14:08:01 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-05-19 14:08:01 0 [Note] InnoDB: Uses event mutexes
2022-05-19 14:08:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-05-19 14:08:01 0 [Note] InnoDB: Number of pools: 1
2022-05-19 14:08:01 0 [Note] InnoDB: Using SSE2 crc32 instructions
2022-05-19 14:08:01 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2022-05-19 14:08:01 0 [Note] InnoDB: Completed initialization of buffer pool
2022-05-19 14:08:01 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-05-19 14:08:01 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes
2022-05-19 14:08:01 0 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes
2022-05-19 14:08:01 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2022-05-19 14:08:01 0 [Note] InnoDB: New log files created, LSN=72752
2022-05-19 14:08:01 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2022-05-19 14:08:01 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-05-19 14:08:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-05-19 14:08:01 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-05-19 14:08:01 0 [Note] InnoDB: Waiting for purge to start
2022-05-19 14:08:01 0 [Note] InnoDB: 10.4.24 started; log sequence number 73228; transaction id 38
2022-05-19 14:08:01 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-05-19 14:08:01 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-05-19 14:08:01 0 [ERROR] mysqld: unknown variable 'wsrep_ssl_mode=PROVIDER'
2022-05-19 14:08:01 0 [ERROR] Aborting
terminate called after throwing an instance of 'wsrep::runtime_error'
  what():  State wait was interrupted
220519 14:08:01 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.4.24-MariaDB-1:10.4.24+maria~focal-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=9
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467769 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fa3180010d8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa331775b48 thread_stack 0x49000
addr2line: 'mysqld': No such file
mysqld(my_print_stacktrace+0x32)[0x55cc35be7f02]
mysqld(handle_fatal_signal+0x55d)[0x55cc3567f35d]
sigaction.c:0(__restore_rt)[0x7fa347c2c420]
addr2line: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fa34773000b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fa34770f859]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911)[0x7fa347ace911]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c)[0x7fa347ada38c]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7)[0x7fa347ada3f7]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa6a9)[0x7fa347ada6a9]
addr2line: 'mysqld': No such file
mysqld(+0x5dd2a4)[0x55cc353882a4]
mysqld(_ZN5wsrep12server_state12sst_receivedERNS_14client_serviceEi+0xe55)[0x55cc35c6aab5]
mysqld(+0xb63ebf)[0x55cc3590eebf]
mysqld(+0xb64dc1)[0x55cc3590fdc1]
mysqld(+0xb42b36)[0x55cc358edb36]
nptl/pthread_create.c:478(start_thread)[0x7fa347c20609]
addr2line: DWARF error: section .debug_info is larger than its filesize! (0x93ef57 vs 0x530ea0)
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fa34780c133]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 3
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
 
We think the query pointer is invalid, but we will try to print it anyway. 
Query: 
 
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             15588                15588                processes 
Max open files            32186                32186                files     
Max locked memory         67108864             67108864             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       15588                15588                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E

I'd expect a miss-configured server to fail right away on startup instead.



 Comments   
Comment by Hartmut Holzgraefe [ 2022-05-19 ]

Probably affects 10.7 and 10.8, too, did not get to testing these versions yet ...

Comment by Hartmut Holzgraefe [ 2022-05-31 ]

I can confirm it just fine, it is easy to verify:

  • create a test cluster with a valid my.cnf on the first node, bring up the first node with galera_new_cluster
  • create a 2nd node with an unknown variable in its my.cnf, try to start that node to make it join the cluster, observe how it goes through Galera initialization first, including SST, and only then (which may be much later, depending on how much data needs to be transferred for SST first) fails with "unknown variable", even though I'd expect to see such a problem being caught right away
Comment by Hartmut Holzgraefe [ 2022-05-31 ]

I now verified this on 10.7 and 10.8, too.

Generated at Thu Feb 08 10:02:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.