[MXS-2779] Maxscale cant choose a good server as a master after Server changed state Created: 2019-11-27  Updated: 2020-08-27  Resolved: 2020-08-27

Status: Closed
Project: MariaDB MaxScale
Component/s: N/A
Affects Version/s: 2.4.4
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: steven lin Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: None
Environment:

CentOS Linux release 7.7.1908 (Core) ,kernel:3.10.0-1062.1.1.el7.x86_64
Database: Percona XtrDB Cluster 5.7 (5 nodes Virtual Machines)


 Description   

Maxscale was abnormal,after it changed the status of my Master node .
it could't change Master to the original node,even the node was working normally.
So I restart the Maxscale service .
then everything was normal.

[maxscale]
 
threads=auto
 
 
 
# Servers
 
[node1]
 
type=server
 
address=10.1.0.176
 
port=3306
 
protocol=mariadbbackend
 
priority=1
 
 
 
[node2]
 
type=server
 
address=10.1.0.177
 
port=3306
 
protocol=mariadbbackend
 
priority=4
 
 
 
[node3]
 
type=server
 
address=10.1.0.178
 
port=3306
 
protocol=mariadbbackend
 
priority=2
 
 
 
[node4]
 
type=server
 
address=10.1.0.172
 
port=3306
 
protocol=mariadbbackend
 
priority=5
 
 
 
[node5]
 
type=server
 
address=10.1.0.173
 
port=3306
 
protocol=mariadbbackend
 
priority=3
 
 
 
[backup_server]
 
type=server
 
address=10.1.0.101
 
port=3306
 
protocol=mariadbbackend
 
# Monitoring for the servers
 
[PXC_Monitor]
 
type=monitor
 
module=galeramon
 
servers=node1,node2,node3,node4,node5,backup_server
 
user=maxscale_monitor
 
password=xxxxxxxxxxxxxxxxxxxxxxxxx
 
monitor_interval=3000ms
 
use_priority=true
 
#disable_master_failback=true
 
 
 
# Galera router service
 
[PXC_Service]
 
type=service
 
router=readwritesplit
 
#slave_selection_criteria=LEAST_CURRENT_OPERATIONS
 
slave_selection_criteria=LEAST_GLOBAL_CONNECTIONS
 
servers=node1,node2,node3,node4,node5
 
user=maxscale_monitor
 
password=xxxxxxxxxxxxxxxxxxxxxxxxx
 
enable_root_user=1
 
connection_timeout=7200s
 
max_connections=2000
 
max_slave_connections=1
 
log_auth_warnings=true
 
master_accept_reads=true
 
#master_failure_mode=fail_on_write
 
#master_reconnection=true
 
#max_sescmd_history=500
 
disable_sescmd_history=true
 
 
 
[BackUp_Mysql_Service]
 
type=service
 
router=readconnroute
 
router_options=running
 
servers=backup_server
 
user=maxscale_monitor
 
password=855077F67A3FC3F32228F836C40BFB3E
 
enable_root_user=1
 
 
 
# Galera cluster listener
 
[PXC_Listener]
 
type=listener
 
service=PXC_Service
 
protocol=mariadbclient
 
address=0.0.0.0
 
port=3306
 
 
 
[BackUp_Listener]
 
type=listener
 
service=BackUp_Mysql_Service
 
protocol=mariadbclient
 
address=0.0.0.0
 
port=3307

LOGS:

*2019-11-26 19:18:43   notice : Server changed state: node1[10.1.0.176:3306]: master_up. [Down] -> [Master, Synced, Running]
 
2019-11-26 19:18:43   notice : Server changed state: node3[10.1.0.178:3306]: new_slave. [Master, Synced, Running] -> [Slave, Synced, Running]*
 
{color:red}2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:{color}
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
 
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
........
 
2019-11-26 19:21:39   notice : Stopped MaxScale REST API
 
2019-11-26 19:21:39   notice : All workers have shut down.
 
2019-11-26 19:21:39   notice : MaxScale received signal SIGTERM. Exiting.
 
2019-11-26 19:21:39   notice : MaxScale is shutting down.
 
2019-11-26 19:21:39   notice : MaxScale shutdown completed.
 
2019-11-26 19:21:39   MariaDB MaxScale is shut down.
 
----------------------------------------------------
 
 
 
 
 
MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Nov 26 19:21:39 2019
 
----------------------------------------------------------------------------
 
2019-11-26 19:21:39   notice : syslog logging is enabled.
 
2019-11-26 19:21:39   notice : maxlog logging is enabled.
 
2019-11-26 19:21:39   notice : Using up to 1.15GiB of memory for query classifier cache
 
2019-11-26 19:21:39   notice : Working directory: /var/log/maxscale
 
2019-11-26 19:21:39   notice : The collection of SQLite memory allocation statistics turned off.
 
2019-11-26 19:21:39   notice : Threading mode of SQLite set to Multi-thread.
 
2019-11-26 19:21:39   notice : MariaDB MaxScale 2.4.4 started (Commit: 231f68b6dc70804b785df356679601ab3eb8e379)
 
2019-11-26 19:21:39   notice : MaxScale is running in process 13501
 
2019-11-26 19:21:39   notice : Configuration file: /etc/maxscale.cnf
 
2019-11-26 19:21:39   notice : Log directory: /var/log/maxscale
 
2019-11-26 19:21:39   notice : Data directory: /var/lib/maxscale
 
2019-11-26 19:21:39   notice : Module directory: /usr/lib64/maxscale
 
2019-11-26 19:21:39   notice : Service cache: /var/cache/maxscale
 
2019-11-26 19:21:39   notice : Worker message queue size: 1.00MiB
 
2019-11-26 19:21:39   notice : No query classifier specified, using default 'qc_sqlite'.
 
2019-11-26 19:21:39   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
 
2019-11-26 19:21:39   notice : Query classification results are cached and reused. Memory used per thread: 73.31MiB
 
2019-11-26 19:21:39   notice : The systemd watchdog is Enabled. Internal timeout = 30s
 
2019-11-26 19:21:39   notice : Loading /etc/maxscale.cnf.
 
2019-11-26 19:21:39   notice : /etc/maxscale.cnf.d does not exist, not reading.
 
2019-11-26 19:21:39   notice : Loaded module mariadbclient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
 
2019-11-26 19:21:39   notice : [readconnroute] Initialise readconnroute router module.
 
2019-11-26 19:21:39   notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so
 
2019-11-26 19:21:39   notice : [readwritesplit] Initializing statement-based read/write split router module.
 
2019-11-26 19:21:39   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
 
2019-11-26 19:21:39   notice : [galeramon] Initialise the MySQL Galera Monitor module.
 
2019-11-26 19:21:39   notice : Loaded module galeramon: V2.0.0 from /usr/lib64/maxscale/libgaleramon.so
 
2019-11-26 19:21:39   notice : Loaded module mariadbbackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
 
2019-11-26 19:21:39   notice : Loaded module mariadbbackendauth: V1.0.0 from /usr/lib64/maxscale/libmariadbbackendauth.so
 
2019-11-26 19:21:39   notice : Loaded module mariadbauth: V1.1.0 from /usr/lib64/maxscale/libmariadbauth.so
 
2019-11-26 19:21:39   notice : Started REST API on [127.0.0.1]:8989
 
2019-11-26 19:21:39   notice : MaxScale started with 16 worker threads, each with a stack size of 8388608 bytes.
 
2019-11-26 19:21:39   notice : Starting a total of 2 services...
 
2019-11-26 19:21:39   notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
 
2019-11-26 19:21:39   notice : Server 'node1' version: 5.7.27-30-57-log
 
2019-11-26 19:21:39   notice : Server 'node2' version: 5.7.27-30-57-log
 
2019-11-26 19:21:40   notice : Server 'node3' version: 5.7.27-30-57-log
 
2019-11-26 19:21:40   notice : Server 'node4' version: 5.7.27-30-57-log
 
2019-11-26 19:21:40   notice : Server 'node5' version: 5.7.27-30-57-log
 
2019-11-26 19:21:40   notice : [MariaDBAuth] [PXC_Service] Loaded 40 MySQL users for listener PXC_Listener from server node1.
 
2019-11-26 19:21:40   notice : Listening for connections at [0.0.0.0]:3306
 
2019-11-26 19:21:40   notice : Service 'PXC_Service' started (1/2)
 
2019-11-26 19:21:40   notice : Server 'backup_server' version: 5.6.26
 
2019-11-26 19:21:40   notice : [MariaDBAuth] [BackUp_Mysql_Service] Loaded 63 MySQL users for listener BackUp_Listener from server backup_server.
 
2019-11-26 19:21:40   notice : Listening for connections at [0.0.0.0]:3307
 
2019-11-26 19:21:40   notice : Service 'BackUp_Mysql_Service' started (2/2)
 
2019-11-26 19:21:40   notice : Loaded server states from journal file: /var/lib/maxscale/PXC_Monitor/monitor.dat



 Comments   
Comment by steven lin [ 2019-11-27 ]

2019-11-26 19:17:18 error : Failed to execute query on server 'node1' ([10.1.0.176]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:20 error : Failed to execute query on server 'node2' ([10.1.0.177]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:22 error : Failed to execute query on server 'node3' ([10.1.0.178]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:24 error : Failed to execute query on server 'node4' ([10.1.0.18
2]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:26 error : Failed to execute query on server 'node5' ([10.1.0.183]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:29 error : Failed to execute query on server 'node4' ([10.1.0.182]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:31 error : Failed to execute query on server 'node5' ([10.1.0.183]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:34 error : Failed to execute query on server 'node5' ([10.1.0.183]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:36 error : Failed to execute query on server 'node5' ([10.1.0.183]:3306): Lost connection to MySQL server during query
2019-11-26 19:17:39 error : Failed to execute query on server 'node5' ([10.1.0.183]:3306): Lost connection to MySQL server during query
2019-11-26 19:18:39 notice : Server changed state: node1[10.1.0.176:3306]: master_down. [Master, Synced, Running] -> [Down]
2019-11-26 19:18:39 notice : Server changed state: node3[10.1.0.178:3306]: new_master. [Slave, Synced, Running] -> [Master, Synced, Running]
2019-11-26 19:18:39 notice : Master switch detected: lost a master and gained a new one
2019-11-26 19:18:39 error : (23838) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while connection was idle. Connection has been idle for 2588.2 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (23781) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while connection was idle. Connection has been idle for 3517.9 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (23848) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while connection was idle. Connection has been idle for 406.3 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (33727) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while waiting for a result. Connection has been idle for 84.2 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (33726) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while waiting for a result. Connection has been idle for 85.2 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (33723) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while waiting for a result. Connection has been idle for 85.5 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (33331) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while connection was idle. Connection has been idle for 38.9 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (33728) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while waiting for a result. Connection has been idle for 83.9 seconds. Error caused by: #HY000: Lost connection to backend server. (subsequent similar messages suppressed for 10000 milliseconds)
2019-11-26 19:18:39 error : (33731) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while waiting for a result. Connection has been idle for 82.7 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:39 error : (33635) [readwritesplit] Lost connection to the master server 'node1', closing session. Lost connection to master server while waiting for a result. Connection has been idle for 110.1 seconds. Error caused by: #HY000: Lost connection to backend server.
2019-11-26 19:18:43 notice : Server changed state: node1[10.1.0.176:3306]: master_up. [Down] -> [Master, Synced, Running]
2019-11-26 19:18:43 notice : Server changed state: node3[10.1.0.178:3306]: new_slave. [Master, Synced, Running] -> [Slave, Synced, Running]
2019-11-26 19:18:43 error : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]

Comment by markus makela [ 2019-11-28 ]

I think this is happening because you have disable_sescmd_history=true and you have commented out master_reconnection=true. Try replacing disable_sescmd_history with prune_sescmd_history=true and uncomment master_reconnection=true.

Comment by steven lin [ 2019-12-02 ]

thanks for your suggestion.
I'll try prune_sescmd_history=true and master_reconnection=true
in my configuration file.

Thanks

Comment by Petros Moisiadis [ 2019-12-04 ]

We also started getting this loop of errors for the active master in a 3-node galera cluster. The master db server had crashed before with segfault and then these messages started to appear refering to that server. However, the 'list servers' command shows it normally as master. Later, we did a rolling upgrade of all galera nodes and in the process, the master flag changed to another server. Now, the 'Cannot choose server ... as master' message continues to appear, but it refers to the new master.

We are not setting explicitly any of the options mentioned above in the configuration file of maxscale, so the running configuration state for these is like this (the defaults I guess):
disable_sescmd_history: false,
master_reconnection: false,
prune_sescmd_history: false

Here is an example of the error line:

2019-12-04 21:01:06   error  : (3676379) [readwritesplit] Cannot choose server '<galera-host-2>' as the master because it is not in use and a new connection to it cannot be created. Connection status: 
name: [<galera-host-1>] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Wed Dec  4 19:15:49 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
 
name: [<galera-host-2>] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [Wed Dec  4 11:30:44 2019] last closed at: [Wed Dec  4 19:20:21 2019] last close reason: [] num sescmd: [0]
 
name: [<galera-host-3>] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]

Comment by markus makela [ 2019-12-05 ]

Ernest0x that is expected behavior with the default settings. Adding prune_sescmd_history=true and master_reconnection=true to the readwritesplit services should fix it.

Generated at Thu Feb 08 04:16:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.