MariaDB MaxScale /var/log/maxscale/maxscale.log Thu May 31 14:08:05 2018 ---------------------------------------------------------------------------- 2018-05-31 14:08:05 notice : The collection of SQLite memory allocation statistics turned off. 2018-05-31 14:08:05 notice : Threading mode of SQLite set to Multi-thread. 2018-05-31 14:08:05 notice : MariaDB MaxScale 2.2.5 started 2018-05-31 14:08:05 notice : MaxScale is running in process 171 2018-05-31 14:08:05 notice : Configuration file: /etc/maxscale.cnf 2018-05-31 14:08:05 notice : Log directory: /var/log/maxscale 2018-05-31 14:08:05 notice : Data directory: /var/lib/maxscale 2018-05-31 14:08:05 notice : Module directory: /usr/lib64/maxscale 2018-05-31 14:08:05 notice : Service cache: /var/cache/maxscale 2018-05-31 14:08:05 notice : Loading /etc/maxscale.cnf. 2018-05-31 14:08:05 notice : Loading /etc/maxscale.cnf.d/mdb-dc1-mariadb-1.cnf. 2018-05-31 14:08:05 notice : Loading /etc/maxscale.cnf.d/mdb-dc1-mariadb-2.cnf. 2018-05-31 14:08:05 notice : Loading /etc/maxscale.cnf.d/mdb-dc1-mariadb-0.cnf. 2018-05-31 14:08:05 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-05-31 14:08:05 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-05-31 14:08:05 notice : [cli] Initialise CLI router module 2018-05-31 14:08:05 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-05-31 14:08:05 notice : [readconnroute] Initialise readconnroute router module. 2018-05-31 14:08:05 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-05-31 14:08:05 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-05-31 14:08:05 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-05-31 14:08:05 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-05-31 14:08:05 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-05-31 14:08:05 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-05-31 14:08:05 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-05-31 14:08:05 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-05-31 14:08:05 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-05-31 14:08:05 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-05-31 14:08:05 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-05-31 14:08:05 notice : No query classifier specified, using default 'qc_sqlite'. 2018-05-31 14:08:05 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-05-31 14:08:05 warning: Removing stale journal file for monitor 'MariaDB-Monitor'. 2018-05-31 14:08:05 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-05-31 14:08:05 notice : Starting a total of 3 services... 2018-05-31 14:08:05 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-05-31 14:08:05 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-05-31 14:08:05 notice : Service 'RWSplit-Service' started (1/3) 2018-05-31 14:08:05 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-05-31 14:08:05 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-05-31 14:08:05 notice : Service 'RO-Service' started (2/3) 2018-05-31 14:08:05 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-05-31 14:08:05 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-05-31 14:08:05 notice : Service 'CLI-Service' started (3/3) 2018-05-31 14:08:05 notice : Started MaxScale log flusher. 2018-05-31 14:08:05 notice : Started REST API on [127.0.0.1]:8989 2018-05-31 14:08:05 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-05-31 14:08:06 notice : Server changed state: mdb-dc1-mariadb-1[192.168.1.132:3306]: new_slave. [Running] -> [Slave, Running] 2018-05-31 14:08:06 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.132]:3306 --event=new_slave --servers=[192.168.1.132]:3306,[192.168.1.134]:3306,[192.168.1.156]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.132]:3306,[192.168.1.156]:3306' on event 'new_slave' 2018-05-31 14:08:06 notice : Server changed state: mdb-dc1-mariadb-0[192.168.1.134:3306]: new_master. [Running] -> [Master, Slave of External Server, Running] 2018-05-31 14:08:06 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.134]:3306 --event=new_master --servers=[192.168.1.132]:3306,[192.168.1.134]:3306,[192.168.1.156]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.132]:3306,[192.168.1.156]:3306' on event 'new_master' 2018-05-31 14:08:06 notice : Server changed state: mdb-dc1-mariadb-2[192.168.1.156:3306]: new_slave. [Running] -> [Slave, Running] 2018-05-31 14:08:07 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.156]:3306 --event=new_slave --servers=[192.168.1.132]:3306,[192.168.1.134]:3306,[192.168.1.156]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.132]:3306,[192.168.1.156]:3306' on event 'new_slave' 2018-05-31 14:08:07 notice : [mariadbmon] A Master Server is now available: 192.168.1.134:3306 2018-05-31 14:08:08 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.229.115:3306 2018-05-31 14:09:29 error : Monitor was unable to connect to server [192.168.1.156]:3306 : "Can't connect to MySQL server on '192.168.1.156' (115)" 2018-05-31 14:09:29 notice : Server changed state: mdb-dc1-mariadb-2[192.168.1.156:3306]: slave_down. [Slave, Running] -> [Down] 2018-05-31 14:09:29 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.156]:3306 --event=slave_down --servers=[192.168.1.132]:3306,[192.168.1.134]:3306,[192.168.1.156]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.132]:3306' on event 'slave_down' 2018-05-31 14:10:51 notice : (104) Updated server 'mdb-dc1-mariadb-2': address=192.168.1.128 2018-05-31 14:10:59 notice : Server changed state: mdb-dc1-mariadb-2[192.168.1.128:3306]: slave_up. [Down] -> [Slave, Running] 2018-05-31 14:11:00 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.128]:3306 --event=slave_up --servers=[192.168.1.132]:3306,[192.168.1.134]:3306,[192.168.1.128]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.132]:3306,[192.168.1.128]:3306' on event 'slave_up' 2018-05-31 14:11:10 error : Monitor was unable to connect to server [192.168.1.132]:3306 : "Can't connect to MySQL server on '192.168.1.132' (115)" 2018-05-31 14:11:10 notice : Server changed state: mdb-dc1-mariadb-1[192.168.1.132:3306]: slave_down. [Slave, Running] -> [Down] 2018-05-31 14:11:10 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.132]:3306 --event=slave_down --servers=[192.168.1.132]:3306,[192.168.1.134]:3306,[192.168.1.128]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.128]:3306' on event 'slave_down' 2018-05-31 14:11:53 notice : (159) Updated server 'mdb-dc1-mariadb-1': address=192.168.1.78 2018-05-31 14:11:58 notice : Server changed state: mdb-dc1-mariadb-1[192.168.1.78:3306]: slave_up. [Down] -> [Slave, Running] 2018-05-31 14:11:58 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.78]:3306 --event=slave_up --servers=[192.168.1.78]:3306,[192.168.1.134]:3306,[192.168.1.128]:3306 --masters=[192.168.1.134]:3306 --slaves=[192.168.1.78]:3306,[192.168.1.128]:3306' on event 'slave_up' 2018-05-31 14:12:05 error : Monitor was unable to connect to server [192.168.1.134]:3306 : "Can't connect to MySQL server on '192.168.1.134' (115)" 2018-05-31 14:12:05 notice : [mariadbmon] Server [192.168.1.134]:3306 lost the master status. 2018-05-31 14:12:05 notice : Server changed state: mdb-dc1-mariadb-0[192.168.1.134:3306]: master_down. [Master, Slave of External Server, Running] -> [Down] 2018-05-31 14:12:06 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.134]:3306 --event=master_down --servers=[192.168.1.78]:3306,[192.168.1.134]:3306,[192.168.1.128]:3306 --masters= --slaves=[192.168.1.78]:3306,[192.168.1.128]:3306' on event 'master_down' 2018-05-31 14:12:06 warning: [mariadbmon] Master has failed. If master status does not change in 3 monitor passes, failover begins. 2018-05-31 14:12:06 error : [mariadbmon] No Master can be determined. Last known was 192.168.1.134:3306 2018-05-31 14:12:14 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:14 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:14 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:14 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:14 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:14 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:15 notice : [mariadbmon] Performing automatic failover to replace failed master 'mdb-dc1-mariadb-0'. 2018-05-31 14:12:15 warning: [mariadbmon] Slave 'mdb-dc1-mariadb-1' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2018-05-31 14:12:15 error : Failed to execute query on server 'mdb-dc1-mariadb-0' ([192.168.1.134]:3306): MySQL server has gone away 2018-05-31 14:12:15 warning: [mariadbmon] Slave 'mdb-dc1-mariadb-2' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2018-05-31 14:12:15 notice : [mariadbmon] Promoting server 'mdb-dc1-mariadb-1' to master. 2018-05-31 14:12:15 notice : [mariadbmon] New master starting replication from external master 10.254.229.115:3306. 2018-05-31 14:12:15 notice : [mariadbmon] Redirecting slaves to new master. 2018-05-31 14:12:16 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:16 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:16 notice : [mariadbmon] Slave 'mdb-dc1-mariadb-2' redirected to new master. 2018-05-31 14:12:16 warning: [mariadbmon] Replicating from external master, skipping final check. 2018-05-31 14:12:16 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:16 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:18 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:18 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:18 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-05-31 14:12:18 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-05-31 14:12:20 notice : Server changed state: mdb-dc1-mariadb-1[192.168.1.78:3306]: new_master. [Slave, Running] -> [Master, Slave of External Server, Running] 2018-05-31 14:12:22 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.78]:3306 --event=new_master --servers=[192.168.1.78]:3306,[192.168.1.134]:3306,[192.168.1.128]:3306 --masters=[192.168.1.78]:3306 --slaves=[192.168.1.128]:3306' on event 'new_master' 2018-05-31 14:12:22 notice : [mariadbmon] A Master Server is now available: 192.168.1.78:3306 2018-05-31 14:13:01 notice : (223) Updated server 'mdb-dc1-mariadb-0': address=192.168.1.218 2018-05-31 14:13:10 notice : Server changed state: mdb-dc1-mariadb-0[192.168.1.218:3306]: server_up. [Down] -> [Running] 2018-05-31 14:13:11 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.218]:3306 --event=server_up --servers=[192.168.1.78]:3306,[192.168.1.218]:3306,[192.168.1.128]:3306 --masters=[192.168.1.78]:3306 --slaves=[192.168.1.128]:3306' on event 'server_up' 2018-05-31 14:13:11 notice : [mariadbmon] Server 'mdb-dc1-mariadb-0' is replicating from a server other than 'mdb-dc1-mariadb-1', redirecting it to 'mdb-dc1-mariadb-1'. 2018-05-31 14:13:12 warning: [mariadbmon] Slave 'mdb-dc1-mariadb-0' redirection failed: 'Lost connection to MySQL server during query'. Query: 'CHANGE MASTER TO ...'. 2018-05-31 14:13:12 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-05-31 14:18:19 notice : Server changed state: mdb-dc1-mariadb-0[192.168.1.218:3306]: new_slave. [Running] -> [Slave, Running] 2018-05-31 14:18:19 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.218]:3306 --event=new_slave --servers=[192.168.1.78]:3306,[192.168.1.218]:3306,[192.168.1.128]:3306 --masters=[192.168.1.78]:3306 --slaves=[192.168.1.218]:3306,[192.168.1.128]:3306' on event 'new_slave'