[MDEV-20104] MariaDB crashes when in galera-cluster behind ProxySQL 2.0 . Created: 2019-07-19  Updated: 2023-05-12  Resolved: 2023-05-12

Status: Closed
Project: MariaDB Server
Component/s: Galera, Server
Affects Version/s: 10.4.4, 10.4.5, 10.4.6
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Shravan Dwarka Assignee: Jan Lindström
Resolution: Incomplete Votes: 1
Labels: galera
Environment:

Ubuntu 18.04.2 (LTS)



 Description   

Hello,

I have a galera cluster setup with 3 nodes. This cluster is being used behind a ProxySQL v2.0.

When I execute a select query to view a parameter and exit the client interface, it causes the galera node currently ONLINE in the writer hostgroup to crash. For example, when I execute this query and exits, it crashes:

SELECT @@hostname;
+------------+
| @@hostname |
+------------+
| mariadb3   |
+------------+

The crash generates this error log:

2019-07-19 12:56:20 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.100.4:4567 
2019-07-19 12:56:21 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') reconnecting to b3b145cc (tcp://192.168.100.4:4567), attempt 0
2019-07-19 12:56:25 0 [Note] WSREP: evs::proto(f400fc1e, OPERATIONAL, view_id(REG,b3b145cc,78)) suspecting node: b3b145cc
2019-07-19 12:56:25 0 [Note] WSREP: evs::proto(f400fc1e, OPERATIONAL, view_id(REG,b3b145cc,78)) suspected node without join message, declaring inactive
2019-07-19 12:56:26 0 [Note] WSREP: declaring c9ed7fd8 at tcp://192.168.100.3:4567 stable
2019-07-19 12:56:26 0 [Note] WSREP: Node c9ed7fd8 state prim
2019-07-19 12:56:26 0 [Note] WSREP: view(view_id(PRIM,c9ed7fd8,79) memb {
	c9ed7fd8,0
	f400fc1e,0
} joined {
} left {
} partitioned {
	b3b145cc,0
})
2019-07-19 12:56:26 0 [Note] WSREP: save pc into disk
2019-07-19 12:56:26 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2019-07-19 12:56:26 0 [Note] WSREP: forgetting b3b145cc (tcp://192.168.100.4:4567)
2019-07-19 12:56:26 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 176378f2-aa03-11e9-a484-8a52dea1ff8c
2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: 176378f2-aa03-11e9-a484-8a52dea1ff8c from 0 (mariadb2)
2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: 176378f2-aa03-11e9-a484-8a52dea1ff8c from 1 (mariadb1)
2019-07-19 12:56:26 0 [Note] WSREP: Quorum results:
	version    = 5,
	component  = PRIMARY,
	conf_id    = 42,
	members    = 2/2 (joined/total),
	act_id     = 256,
	last_appl. = 0,
	protocols  = 1/10/4 (gcs/repl/appl),
	vote policy= 0,
	group UUID = a0e4ee53-a2ed-11e9-8785-77f49c647069
2019-07-19 12:56:26 0 [Note] WSREP: Writing down CC checksum: ef242699 07eacac7 fb0c42ea 3001f3d1 at offset 184
2019-07-19 12:56:26 0 [Note] WSREP: Flow-control interval: [23, 23]
2019-07-19 12:56:26 0 [Note] WSREP: Trying to continue unpaused monitor
2019-07-19 12:56:26 1 [Note] WSREP: ####### processing CC 257, local, ordered
2019-07-19 12:56:26 1 [Note] WSREP: ####### drain monitors upto 256
2019-07-19 12:56:26 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
2019-07-19 12:56:26 1 [Note] WSREP: ####### My UUID: f400fc1e-a6cf-11e9-ab61-eac7c8a52069
2019-07-19 12:56:26 1 [Note] WSREP: ####### ST not required
2019-07-19 12:56:26 1 [Note] WSREP: Skipping cert index reset
2019-07-19 12:56:26 1 [Note] WSREP: ####### Adjusting cert position: 256 -> 257
2019-07-19 12:56:26 0 [Note] WSREP: Service thread queue flushed.
2019-07-19 12:56:26 1 [Note] WSREP: ####### Setting monitor position to 257
2019-07-19 12:56:26 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 257
2019-07-19 12:56:26 1 [Note] WSREP: Min available from gcache for CC from group: 125
2019-07-19 12:56:26 1 [Note] WSREP: ================================================
View:
  id: a0e4ee53-a2ed-11e9-8785-77f49c647069:257
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 1
  members(2):
	0: c9ed7fd8-a2f6-11e9-b3a5-3e00a5379103, mariadb2
	1: f400fc1e-a6cf-11e9-ab61-eac7c8a52069, mariadb1
=================================================
2019-07-19 12:56:26 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-07-19 12:56:29 0 [Note] WSREP:  cleaning up b3b145cc (tcp://192.168.100.4:4567)
2019-07-19 12:56:30 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') connection established to b3b145cc tcp://192.168.100.4:4567
2019-07-19 12:56:30 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2019-07-19 12:56:30 0 [Note] WSREP: declaring b3b145cc at tcp://192.168.100.4:4567 stable
2019-07-19 12:56:30 0 [Note] WSREP: declaring c9ed7fd8 at tcp://192.168.100.3:4567 stable
2019-07-19 12:56:30 0 [Note] WSREP: Node c9ed7fd8 state prim
2019-07-19 12:56:30 0 [Note] WSREP: view(view_id(PRIM,b3b145cc,80) memb {
	b3b145cc,0
	c9ed7fd8,0
	f400fc1e,0
} joined {
} left {
} partitioned {
})
2019-07-19 12:56:30 0 [Note] WSREP: save pc into disk
2019-07-19 12:56:30 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af
2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af from 0 (mariadb3)
2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af from 1 (mariadb2)
2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af from 2 (mariadb1)
2019-07-19 12:56:30 0 [Note] WSREP: Quorum results:
	version    = 5,
	component  = PRIMARY,
	conf_id    = 43,
	members    = 2/3 (joined/total),
	act_id     = 257,
	last_appl. = 0,
	protocols  = 1/10/4 (gcs/repl/appl),
	vote policy= 0,
	group UUID = a0e4ee53-a2ed-11e9-8785-77f49c647069
2019-07-19 12:56:30 0 [Note] WSREP: Writing down CC checksum: 555e7bf5 849283ab a4fcdd10 db627a65 at offset 240
2019-07-19 12:56:30 0 [Note] WSREP: Flow-control interval: [28, 28]
2019-07-19 12:56:30 0 [Note] WSREP: Trying to continue unpaused monitor
2019-07-19 12:56:30 1 [Note] WSREP: ####### processing CC 258, local, ordered
2019-07-19 12:56:30 1 [Note] WSREP: ####### drain monitors upto 257
2019-07-19 12:56:30 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
2019-07-19 12:56:30 1 [Note] WSREP: ####### My UUID: f400fc1e-a6cf-11e9-ab61-eac7c8a52069
2019-07-19 12:56:30 1 [Note] WSREP: ####### ST not required
2019-07-19 12:56:30 1 [Note] WSREP: Skipping cert index reset
2019-07-19 12:56:30 1 [Note] WSREP: ####### Adjusting cert position: 257 -> 258
2019-07-19 12:56:30 0 [Note] WSREP: Service thread queue flushed.
2019-07-19 12:56:30 1 [Note] WSREP: ####### Setting monitor position to 258
2019-07-19 12:56:30 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 258
2019-07-19 12:56:30 1 [Note] WSREP: Min available from gcache for CC from group: 125
2019-07-19 12:56:30 1 [Note] WSREP: ================================================
View:
  id: a0e4ee53-a2ed-11e9-8785-77f49c647069:258
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 2
  members(3):
	0: b3b145cc-a3f3-11e9-8d5d-b6f514e33bcd, mariadb3
	1: c9ed7fd8-a2f6-11e9-b3a5-3e00a5379103, mariadb2
	2: f400fc1e-a6cf-11e9-ab61-eac7c8a52069, mariadb1
=================================================
2019-07-19 12:56:30 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-07-19 12:56:31 0 [Note] WSREP: Member 0.0 (mariadb3) requested state transfer from '*any*'. Selected 1.0 (mariadb2)(SYNCED) as donor.
2019-07-19 12:56:31 0 [Note] WSREP: 1.0 (mariadb2): State transfer to 0.0 (mariadb3) complete.
2019-07-19 12:56:31 0 [Note] WSREP: Member 1.0 (mariadb2) synced with group.
2019-07-19 12:56:33 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-07-19 12:56:34 0 [Note] WSREP: 0.0 (mariadb3): State transfer from 1.0 (mariadb2) complete.
2019-07-19 12:56:34 0 [Note] WSREP: Member 0.0 (mariadb3) synced with group.

I have already created an issue at proxysql [1] and galera [2] concerning this matter.

Regards,
Shravan

[1] - https://github.com/sysown/proxysql/issues/2132
[2] - https://github.com/codership/galera/issues/542



 Comments   
Comment by Jan Lindström [ 2023-04-11 ]

dshravan Can you please try with more recent version of MariaDB and Galera library. If issue reproduces please provide full error log, opened stack trace from core file and node configuration.

Generated at Thu Feb 08 08:56:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.