MariaDB MaxScale /var/log/maxscale/maxscale.log Wed Feb 20 16:12:33 2019 ---------------------------------------------------------------------------- 2019-02-20 16:12:33 notice : syslog logging is enabled. 2019-02-20 16:12:33 notice : maxlog logging is enabled. 2019-02-20 16:12:33 notice : Using up to 1.48GiB of memory for query classifier cache 2019-02-20 16:12:33 notice : The collection of SQLite memory allocation statistics turned off. 2019-02-20 16:12:33 notice : Threading mode of SQLite set to Multi-thread. 2019-02-20 16:12:33 notice : MariaDB MaxScale 2.3.3 started (Commit: 28644e9626ad16cb30a2f715b8fa9e3a8df5583b) 2019-02-20 16:12:33 notice : MaxScale is running in process 572 2019-02-20 16:12:33 notice : Configuration file: /etc/maxscale.cnf 2019-02-20 16:12:33 notice : Log directory: /var/log/maxscale 2019-02-20 16:12:33 notice : Data directory: /var/lib/maxscale 2019-02-20 16:12:33 notice : Module directory: /usr/lib64/maxscale 2019-02-20 16:12:33 notice : Service cache: /var/cache/maxscale 2019-02-20 16:12:33 notice : No query classifier specified, using default 'qc_sqlite'. 2019-02-20 16:12:33 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2019-02-20 16:12:33 notice : Query classification results are cached and reused. Memory used per thread: 757.99MiB 2019-02-20 16:12:33 notice : Loading /etc/maxscale.cnf. 2019-02-20 16:12:33 notice : Loading /etc/maxscale.cnf.d/batavia-cmdb-mariadb-0.cnf. 2019-02-20 16:12:33 notice : Loading /etc/maxscale.cnf.d/batavia-cmdb-mariadb-1.cnf. 2019-02-20 16:12:33 notice : Loading /etc/maxscale.cnf.d/batavia-cmdb-mariadb-2.cnf. 2019-02-20 16:12:33 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2019-02-20 16:12:33 notice : Initialise the MariaDB Monitor module. 2019-02-20 16:12:33 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2019-02-20 16:12:33 warning: The parameter 'passwd' is deprecated: use 'password' instead 2019-02-20 16:12:33 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2019-02-20 16:12:33 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2019-02-20 16:12:33 warning: THE 'cli' MODULE AND 'maxadmin' ARE DEPRECATED: Use 'maxctrl' instead 2019-02-20 16:12:33 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2019-02-20 16:12:33 notice : Initialise readconnroute router module. 2019-02-20 16:12:33 notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so 2019-02-20 16:12:33 warning: The parameter 'passwd' is deprecated: use 'password' instead 2019-02-20 16:12:33 notice : Initializing statement-based read/write split router module. 2019-02-20 16:12:33 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2019-02-20 16:12:33 warning: The parameter 'passwd' is deprecated: use 'password' instead 2019-02-20 16:12:33 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2019-02-20 16:12:33 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2019-02-20 16:12:33 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2019-02-20 16:12:33 notice : Housekeeper thread started. 2019-02-20 16:12:33 warning: Removing stale journal file for monitor 'MariaDB-Monitor'. 2019-02-20 16:12:33 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2019-02-20 16:12:33 notice : Starting a total of 3 services... 2019-02-20 16:12:33 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2019-02-20 16:12:33 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2019-02-20 16:12:33 notice : Service 'CLI-Service' started (1/3) 2019-02-20 16:12:33 error : [RO-Service] Failed to connect to server 'batavia-cmdb-mariadb-2' ([192.168.1.84]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.84' (115) 2019-02-20 16:12:33 error : [RO-Service] Failed to connect to server 'batavia-cmdb-mariadb-1' ([192.168.1.6]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.6' (115) 2019-02-20 16:12:33 error : Monitor was unable to connect to server batavia-cmdb-mariadb-2[192.168.1.84:3306] : 'Can't connect to MySQL server on '192.168.1.84' (115)' 2019-02-20 16:12:33 error : Monitor was unable to connect to server batavia-cmdb-mariadb-1[192.168.1.6:3306] : 'Can't connect to MySQL server on '192.168.1.6' (115)' 2019-02-20 16:12:33 error : [RO-Service] Failed to connect to server 'batavia-cmdb-mariadb-0' ([192.168.1.206]:3306) when checking authentication user credentials and permissions: 1130 Host '192.168.1.132' is not allowed to connect to this MariaDB server 2019-02-20 16:12:33 error : Monitor was unable to connect to server batavia-cmdb-mariadb-0[192.168.1.206:3306] : 'Host '192.168.1.132' is not allowed to connect to this MariaDB server' 2019-02-20 16:12:33 warning: No valid master server found. 2019-02-20 16:12:33 warning: 'batavia-cmdb-mariadb-2' is not a valid master candidate because it is down. 'batavia-cmdb-mariadb-1' is not a valid master candidate because it is down. 'batavia-cmdb-mariadb-0' is not a valid master candidate because it is down. 2019-02-20 16:12:33 error : No Master can be determined 2019-02-20 16:12:33 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.84:3306]: server_down. [Running] -> [Down] 2019-02-20 16:12:33 error : Failure loading users data from backend [192.168.1.84:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.84' (115) 2019-02-20 16:12:33 error : Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-02-20 16:12:33 error : [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2019-02-20 16:12:33 notice : [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-02-20 16:12:33 notice : Listening for connections at [::]:3307 with protocol MySQL 2019-02-20 16:12:33 notice : Service 'RO-Service' started (2/3) 2019-02-20 16:12:33 error : [RWSplit-Service] Failed to connect to server 'batavia-cmdb-mariadb-2' ([192.168.1.84]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.84' (115) 2019-02-20 16:12:33 error : [RWSplit-Service] Failed to connect to server 'batavia-cmdb-mariadb-1' ([192.168.1.6]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.6' (115) 2019-02-20 16:12:33 error : [RWSplit-Service] Failed to connect to server 'batavia-cmdb-mariadb-0' ([192.168.1.206]:3306) when checking authentication user credentials and permissions: 1130 Host '192.168.1.132' is not allowed to connect to this MariaDB server 2019-02-20 16:12:33 notice : [RWSplit-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-02-20 16:12:33 notice : Listening for connections at [::]:3306 with protocol MySQL 2019-02-20 16:12:33 notice : Service 'RWSplit-Service' started (3/3) 2019-02-20 16:12:33 notice : [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-02-20 16:12:33 notice : [RWSplit-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-02-20 16:12:33 notice : Started REST API on [0.0.0.0]:8989 2019-02-20 16:12:33 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2019-02-20 16:12:33 notice : [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-02-20 16:12:33 notice : [RWSplit-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-02-20 16:12:33 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Server DOWN: [192.168.1.84]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:33.838Z", "type": "log"} 2019-02-20 16:12:33 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.84]:3306 --event=server_down --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters= --slaves=' on event 'server_down' 2019-02-20 16:12:33 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.6:3306]: server_down. [Running] -> [Down] 2019-02-20 16:12:34 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Server DOWN: [192.168.1.6]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:34.094Z", "type": "log"} 2019-02-20 16:12:34 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.6]:3306 --event=server_down --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters= --slaves=' on event 'server_down' 2019-02-20 16:12:34 notice : Server changed state: batavia-cmdb-mariadb-0[192.168.1.206:3306]: server_down. [Running] -> [Down] 2019-02-20 16:12:34 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Server DOWN: [192.168.1.206]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:34.346Z", "type": "log"} 2019-02-20 16:12:34 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.206]:3306 --event=server_down --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters= --slaves=' on event 'server_down' 2019-02-20 16:12:36 notice : (12) Updated 'passive' from 'true' to 'false' 2019-02-20 16:12:42 notice : Selecting new master server. 2019-02-20 16:12:42 warning: 'batavia-cmdb-mariadb-2' is not a valid master candidate because it is down. 'batavia-cmdb-mariadb-1' is not a valid master candidate because it is down. 2019-02-20 16:12:42 notice : Setting 'batavia-cmdb-mariadb-0' as master. 2019-02-20 16:12:42 notice : Server changed state: batavia-cmdb-mariadb-0[192.168.1.206:3306]: master_up. [Down] -> [Master, Running] 2019-02-20 16:12:42 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master UP: [192.168.1.206]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:42.723Z", "type": "log"} 2019-02-20 16:12:42 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "SQL [START SLAVE;] FAILED: (1200, 'Misconfigured slave: MASTER_HOST was not set; Fix in config file or with CHANGE MASTER TO')"}, "service": "maxscale", "level": "warning", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:42.797Z", "type": "log"} 2019-02-20 16:12:42 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for batavia-cmdb-mariadb-0 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:42.798Z", "type": "log"} 2019-02-20 16:12:42 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.206]:3306 --event=master_up --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters=[192.168.1.206]:3306 --slaves=' on event 'master_up' 2019-02-20 16:12:47 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:12:47 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.6:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 16:12:47 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.6]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:47.856Z", "type": "log"} 2019-02-20 16:12:47 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.6]:3306 --event=slave_up --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters=[192.168.1.206]:3306 --slaves=[192.168.1.6]:3306' on event 'slave_up' 2019-02-20 16:12:48 notice : (27) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:12:48 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:12:48 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:12:48 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.84:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 16:12:48 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.84]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:12:48.463Z", "type": "log"} 2019-02-20 16:12:48 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.84]:3306 --event=slave_up --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters=[192.168.1.206]:3306 --slaves=[192.168.1.84]:3306,[192.168.1.6]:3306' on event 'slave_up' 2019-02-20 16:13:01 notice : (44) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:13:01 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:13:01 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:13:15 notice : (62) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:13:15 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:13:15 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:13:29 notice : (80) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:13:29 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:13:29 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:13:43 notice : (97) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:13:43 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:13:43 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:13:57 notice : (115) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:13:57 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:13:57 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:14:10 notice : (133) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:14:10 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:14:10 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:14:24 notice : (150) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:14:24 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:14:24 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:14:38 notice : (168) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:14:38 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:14:38 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:14:51 notice : (186) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:14:51 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:14:51 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:15:05 notice : (203) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:15:05 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:15:05 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:15:19 notice : (221) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:15:19 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:15:19 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:15:33 notice : (239) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:15:33 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:15:33 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:15:46 notice : (257) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:15:46 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:15:46 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:16:00 notice : (274) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:16:00 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:16:00 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:16:14 notice : (292) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:16:14 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:16:14 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:16:28 notice : (310) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:16:28 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:16:28 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:16:41 notice : (327) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:16:41 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:16:41 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:16:55 notice : (345) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:16:55 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:16:55 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:17:09 notice : (363) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:17:09 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:17:09 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:17:23 notice : (381) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:17:23 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:17:23 error : The backend cluster does not support failover/switchover due to the following reason(s): Slave connection from batavia-cmdb-mariadb-2 to [192.168.1.206]:3306 is not using gtid-replication. Slave connection from batavia-cmdb-mariadb-1 to [192.168.1.206]:3306 is not using gtid-replication. Automatic failover/switchover has been disabled. They should only be enabled after the above issues have been resolved. To re-enable automatic failover, manually set 'auto_failover' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. To re-enable automatic switchover, manually set 'switchover_on_low_disk_space' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API, or restart MaxScale. 2019-02-20 16:17:36 notice : (400) Updated monitor 'MariaDB-Monitor': auto_failover=true 2019-02-20 16:17:36 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2019-02-20 16:22:10 error : Monitor was unable to connect to server batavia-cmdb-mariadb-2[192.168.1.84:3306] : 'Can't connect to MySQL server on '192.168.1.84' (115)' 2019-02-20 16:22:10 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.84:3306]: slave_down. [Slave, Running] -> [Down] 2019-02-20 16:22:11 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave DOWN: [192.168.1.84]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:22:11.041Z", "type": "log"} 2019-02-20 16:22:11 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.84]:3306 --event=slave_down --servers=[192.168.1.84]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters=[192.168.1.206]:3306 --slaves=[192.168.1.6]:3306' on event 'slave_down' 2019-02-20 16:24:28 notice : (808) Updated server 'batavia-cmdb-mariadb-2': address=192.168.1.207 2019-02-20 16:24:31 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.207:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 16:24:31 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.207]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:24:31.493Z", "type": "log"} 2019-02-20 16:24:31 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.207]:3306 --event=slave_up --servers=[192.168.1.207]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters=[192.168.1.206]:3306 --slaves=[192.168.1.207]:3306,[192.168.1.6]:3306' on event 'slave_up' 2019-02-20 16:25:08 error : Monitor was unable to connect to server batavia-cmdb-mariadb-0[192.168.1.206:3306] : 'Can't connect to MySQL server on '192.168.1.206' (115)' 2019-02-20 16:25:08 notice : Server changed state: batavia-cmdb-mariadb-0[192.168.1.206:3306]: master_down. [Master, Running] -> [Down] 2019-02-20 16:25:09 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master DOWN: [192.168.1.206]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:25:09.204Z", "type": "log"} 2019-02-20 16:25:09 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [] for batavia-cmdb-mariadb-0 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:25:09.206Z", "type": "log"} 2019-02-20 16:25:09 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.206]:3306 --event=master_down --servers=[192.168.1.207]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters= --slaves=[192.168.1.207]:3306,[192.168.1.6]:3306' on event 'master_down' 2019-02-20 16:25:09 warning: Master has failed. If master status does not change in 3 monitor passes, failover begins. 2019-02-20 16:25:11 notice : Selecting a server to promote and replace 'batavia-cmdb-mariadb-0'. Candidates are: 'batavia-cmdb-mariadb-2', 'batavia-cmdb-mariadb-1'. 2019-02-20 16:25:11 warning: Slave 'batavia-cmdb-mariadb-2' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 16:25:11 warning: Slave 'batavia-cmdb-mariadb-1' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 16:25:11 notice : Selected 'batavia-cmdb-mariadb-2'. 2019-02-20 16:25:11 notice : Performing automatic failover to replace failed master 'batavia-cmdb-mariadb-0'. 2019-02-20 16:25:11 notice : Executing sql queries from file '/etc/mariadb/promotion.sql' on server 'batavia-cmdb-mariadb-2'. 2019-02-20 16:25:11 notice : 1 queries executed successfully. 2019-02-20 16:25:11 notice : Redirecting 'batavia-cmdb-mariadb-1' to replicate from 'batavia-cmdb-mariadb-2' instead of 'batavia-cmdb-mariadb-0'. 2019-02-20 16:25:11 notice : All redirects successful. 2019-02-20 16:25:12 notice : All redirected slaves successfully started replication from 'batavia-cmdb-mariadb-2'. 2019-02-20 16:25:12 notice : Failover 'batavia-cmdb-mariadb-0' -> 'batavia-cmdb-mariadb-2' performed. 2019-02-20 16:25:15 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.207:3306]: new_master. [Slave, Running] -> [Master, Running] 2019-02-20 16:25:16 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] NEW Master: [192.168.1.207]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:25:16.153Z", "type": "log"} 2019-02-20 16:25:16 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for batavia-cmdb-mariadb-2 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:25:16.156Z", "type": "log"} 2019-02-20 16:25:16 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.207]:3306 --event=new_master --servers=[192.168.1.207]:3306,[192.168.1.6]:3306,[192.168.1.206]:3306 --masters=[192.168.1.207]:3306 --slaves=[192.168.1.6]:3306' on event 'new_master' 2019-02-20 16:27:29 notice : (911) Updated server 'batavia-cmdb-mariadb-0': address=192.168.1.208 2019-02-20 16:27:31 notice : Server changed state: batavia-cmdb-mariadb-0[192.168.1.208:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 16:27:31 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.208]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T16:27:31.872Z", "type": "log"} 2019-02-20 16:27:31 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.208]:3306 --event=slave_up --servers=[192.168.1.207]:3306,[192.168.1.6]:3306,[192.168.1.208]:3306 --masters=[192.168.1.207]:3306 --slaves=[192.168.1.6]:3306,[192.168.1.208]:3306' on event 'slave_up' 2019-02-20 21:20:24 error : Monitor was unable to connect to server batavia-cmdb-mariadb-2[192.168.1.207:3306] : 'Can't connect to MySQL server on '192.168.1.207' (115)' 2019-02-20 21:20:24 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.207:3306]: master_down. [Master, Running] -> [Down] 2019-02-20 21:20:25 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master DOWN: [192.168.1.207]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:20:25.089Z", "type": "log"} 2019-02-20 21:20:25 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [] for batavia-cmdb-mariadb-2 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:20:25.092Z", "type": "log"} 2019-02-20 21:20:25 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.207]:3306 --event=master_down --servers=[192.168.1.207]:3306,[192.168.1.6]:3306,[192.168.1.208]:3306 --masters= --slaves=[192.168.1.6]:3306,[192.168.1.208]:3306' on event 'master_down' 2019-02-20 21:20:25 warning: Master has failed. If master status does not change in 3 monitor passes, failover begins. 2019-02-20 21:20:30 notice : Selecting a server to promote and replace 'batavia-cmdb-mariadb-2'. Candidates are: 'batavia-cmdb-mariadb-1', 'batavia-cmdb-mariadb-0'. 2019-02-20 21:20:30 warning: Slave 'batavia-cmdb-mariadb-1' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:20:30 warning: Slave 'batavia-cmdb-mariadb-0' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:20:30 notice : Selected 'batavia-cmdb-mariadb-1'. 2019-02-20 21:20:30 notice : Performing automatic failover to replace failed master 'batavia-cmdb-mariadb-2'. 2019-02-20 21:20:30 notice : Executing sql queries from file '/etc/mariadb/promotion.sql' on server 'batavia-cmdb-mariadb-1'. 2019-02-20 21:20:30 notice : 1 queries executed successfully. 2019-02-20 21:20:30 notice : Redirecting 'batavia-cmdb-mariadb-0' to replicate from 'batavia-cmdb-mariadb-1' instead of 'batavia-cmdb-mariadb-2'. 2019-02-20 21:20:30 notice : All redirects successful. 2019-02-20 21:20:31 notice : All redirected slaves successfully started replication from 'batavia-cmdb-mariadb-1'. 2019-02-20 21:20:31 notice : Failover 'batavia-cmdb-mariadb-2' -> 'batavia-cmdb-mariadb-1' performed. 2019-02-20 21:20:34 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.6:3306]: new_master. [Slave, Running] -> [Master, Running] 2019-02-20 21:20:34 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] NEW Master: [192.168.1.6]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:20:34.659Z", "type": "log"} 2019-02-20 21:20:34 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for batavia-cmdb-mariadb-1 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:20:34.661Z", "type": "log"} 2019-02-20 21:20:34 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.6]:3306 --event=new_master --servers=[192.168.1.207]:3306,[192.168.1.6]:3306,[192.168.1.208]:3306 --masters=[192.168.1.6]:3306 --slaves=[192.168.1.208]:3306' on event 'new_master' 2019-02-20 21:22:44 notice : (23935) Updated server 'batavia-cmdb-mariadb-2': address=192.168.1.211 2019-02-20 21:22:47 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.211:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 21:22:47 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.211]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:22:47.488Z", "type": "log"} 2019-02-20 21:22:47 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.211]:3306 --event=slave_up --servers=[192.168.1.211]:3306,[192.168.1.6]:3306,[192.168.1.208]:3306 --masters=[192.168.1.6]:3306 --slaves=[192.168.1.211]:3306,[192.168.1.208]:3306' on event 'slave_up' 2019-02-20 21:28:38 error : Monitor was unable to connect to server batavia-cmdb-mariadb-1[192.168.1.6:3306] : 'Can't connect to MySQL server on '192.168.1.6' (115)' 2019-02-20 21:28:38 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.6:3306]: master_down. [Master, Running] -> [Down] 2019-02-20 21:28:38 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master DOWN: [192.168.1.6]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:28:38.268Z", "type": "log"} 2019-02-20 21:28:38 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [] for batavia-cmdb-mariadb-1 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:28:38.270Z", "type": "log"} 2019-02-20 21:28:38 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.6]:3306 --event=master_down --servers=[192.168.1.211]:3306,[192.168.1.6]:3306,[192.168.1.208]:3306 --masters= --slaves=[192.168.1.211]:3306,[192.168.1.208]:3306' on event 'master_down' 2019-02-20 21:28:38 warning: Master has failed. If master status does not change in 3 monitor passes, failover begins. 2019-02-20 21:28:44 notice : Selecting a server to promote and replace 'batavia-cmdb-mariadb-1'. Candidates are: 'batavia-cmdb-mariadb-2', 'batavia-cmdb-mariadb-0'. 2019-02-20 21:28:44 warning: Slave 'batavia-cmdb-mariadb-2' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:28:44 warning: Slave 'batavia-cmdb-mariadb-0' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:28:44 notice : Selected 'batavia-cmdb-mariadb-2'. 2019-02-20 21:28:44 notice : Performing automatic failover to replace failed master 'batavia-cmdb-mariadb-1'. 2019-02-20 21:28:44 notice : Executing sql queries from file '/etc/mariadb/promotion.sql' on server 'batavia-cmdb-mariadb-2'. 2019-02-20 21:28:44 notice : 1 queries executed successfully. 2019-02-20 21:28:44 notice : Redirecting 'batavia-cmdb-mariadb-0' to replicate from 'batavia-cmdb-mariadb-2' instead of 'batavia-cmdb-mariadb-1'. 2019-02-20 21:28:44 notice : All redirects successful. 2019-02-20 21:28:44 notice : All redirected slaves successfully started replication from 'batavia-cmdb-mariadb-2'. 2019-02-20 21:28:44 notice : Failover 'batavia-cmdb-mariadb-1' -> 'batavia-cmdb-mariadb-2' performed. 2019-02-20 21:28:47 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.211:3306]: new_master. [Slave, Running] -> [Master, Running] 2019-02-20 21:28:48 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] NEW Master: [192.168.1.211]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:28:48.335Z", "type": "log"} 2019-02-20 21:28:48 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for batavia-cmdb-mariadb-2 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:28:48.339Z", "type": "log"} 2019-02-20 21:28:48 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.211]:3306 --event=new_master --servers=[192.168.1.211]:3306,[192.168.1.6]:3306,[192.168.1.208]:3306 --masters=[192.168.1.211]:3306 --slaves=[192.168.1.208]:3306' on event 'new_master' 2019-02-20 21:30:58 notice : (24492) Updated server 'batavia-cmdb-mariadb-1': address=192.168.1.11 2019-02-20 21:31:00 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.11:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 21:31:01 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.11]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:31:01.117Z", "type": "log"} 2019-02-20 21:31:01 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.11]:3306 --event=slave_up --servers=[192.168.1.211]:3306,[192.168.1.11]:3306,[192.168.1.208]:3306 --masters=[192.168.1.211]:3306 --slaves=[192.168.1.11]:3306,[192.168.1.208]:3306' on event 'slave_up' 2019-02-20 21:35:43 error : Monitor was unable to connect to server batavia-cmdb-mariadb-2[192.168.1.211:3306] : 'Can't connect to MySQL server on '192.168.1.211' (115)' 2019-02-20 21:35:43 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.211:3306]: master_down. [Master, Running] -> [Down] 2019-02-20 21:35:44 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master DOWN: [192.168.1.211]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:35:44.088Z", "type": "log"} 2019-02-20 21:35:44 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [] for batavia-cmdb-mariadb-2 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:35:44.091Z", "type": "log"} 2019-02-20 21:35:44 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.211]:3306 --event=master_down --servers=[192.168.1.211]:3306,[192.168.1.11]:3306,[192.168.1.208]:3306 --masters= --slaves=[192.168.1.11]:3306,[192.168.1.208]:3306' on event 'master_down' 2019-02-20 21:35:44 warning: Master has failed. If master status does not change in 3 monitor passes, failover begins. 2019-02-20 21:35:49 notice : Selecting a server to promote and replace 'batavia-cmdb-mariadb-2'. Candidates are: 'batavia-cmdb-mariadb-1', 'batavia-cmdb-mariadb-0'. 2019-02-20 21:35:49 warning: Slave 'batavia-cmdb-mariadb-1' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:35:49 warning: Slave 'batavia-cmdb-mariadb-0' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:35:49 notice : Selected 'batavia-cmdb-mariadb-1'. 2019-02-20 21:35:49 notice : Performing automatic failover to replace failed master 'batavia-cmdb-mariadb-2'. 2019-02-20 21:35:49 notice : Executing sql queries from file '/etc/mariadb/promotion.sql' on server 'batavia-cmdb-mariadb-1'. 2019-02-20 21:35:49 notice : 1 queries executed successfully. 2019-02-20 21:35:49 notice : Redirecting 'batavia-cmdb-mariadb-0' to replicate from 'batavia-cmdb-mariadb-1' instead of 'batavia-cmdb-mariadb-2'. 2019-02-20 21:35:49 notice : All redirects successful. 2019-02-20 21:35:50 notice : All redirected slaves successfully started replication from 'batavia-cmdb-mariadb-1'. 2019-02-20 21:35:50 notice : Failover 'batavia-cmdb-mariadb-2' -> 'batavia-cmdb-mariadb-1' performed. 2019-02-20 21:35:53 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.11:3306]: new_master. [Slave, Running] -> [Master, Running] 2019-02-20 21:35:53 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] NEW Master: [192.168.1.11]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:35:53.621Z", "type": "log"} 2019-02-20 21:35:53 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for batavia-cmdb-mariadb-1 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:35:53.624Z", "type": "log"} 2019-02-20 21:35:53 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.11]:3306 --event=new_master --servers=[192.168.1.211]:3306,[192.168.1.11]:3306,[192.168.1.208]:3306 --masters=[192.168.1.11]:3306 --slaves=[192.168.1.208]:3306' on event 'new_master' 2019-02-20 21:38:03 notice : (24917) Updated server 'batavia-cmdb-mariadb-2': address=192.168.1.212 2019-02-20 21:38:06 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.212:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 21:38:06 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.212]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:38:06.420Z", "type": "log"} 2019-02-20 21:38:06 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.212]:3306 --event=slave_up --servers=[192.168.1.212]:3306,[192.168.1.11]:3306,[192.168.1.208]:3306 --masters=[192.168.1.11]:3306 --slaves=[192.168.1.212]:3306,[192.168.1.208]:3306' on event 'slave_up' 2019-02-20 21:53:42 error : Monitor was unable to connect to server batavia-cmdb-mariadb-1[192.168.1.11:3306] : 'Can't connect to MySQL server on '192.168.1.11' (115)' 2019-02-20 21:53:42 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.11:3306]: master_down. [Master, Running] -> [Down] 2019-02-20 21:53:42 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master DOWN: [192.168.1.11]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:53:42.783Z", "type": "log"} 2019-02-20 21:53:42 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [] for batavia-cmdb-mariadb-1 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:53:42.785Z", "type": "log"} 2019-02-20 21:53:42 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.11]:3306 --event=master_down --servers=[192.168.1.212]:3306,[192.168.1.11]:3306,[192.168.1.208]:3306 --masters= --slaves=[192.168.1.212]:3306,[192.168.1.208]:3306' on event 'master_down' 2019-02-20 21:53:42 warning: Master has failed. If master status does not change in 3 monitor passes, failover begins. 2019-02-20 21:53:50 notice : Selecting a server to promote and replace 'batavia-cmdb-mariadb-1'. Candidates are: 'batavia-cmdb-mariadb-2', 'batavia-cmdb-mariadb-0'. 2019-02-20 21:53:50 warning: Slave 'batavia-cmdb-mariadb-2' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:53:50 warning: Slave 'batavia-cmdb-mariadb-0' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-02-20 21:53:50 notice : Selected 'batavia-cmdb-mariadb-2'. 2019-02-20 21:53:50 notice : Performing automatic failover to replace failed master 'batavia-cmdb-mariadb-1'. 2019-02-20 21:53:50 notice : Executing sql queries from file '/etc/mariadb/promotion.sql' on server 'batavia-cmdb-mariadb-2'. 2019-02-20 21:53:50 notice : 1 queries executed successfully. 2019-02-20 21:53:50 notice : Redirecting 'batavia-cmdb-mariadb-0' to replicate from 'batavia-cmdb-mariadb-2' instead of 'batavia-cmdb-mariadb-1'. 2019-02-20 21:53:50 notice : All redirects successful. 2019-02-20 21:53:51 notice : All redirected slaves successfully started replication from 'batavia-cmdb-mariadb-2'. 2019-02-20 21:53:51 notice : Failover 'batavia-cmdb-mariadb-1' -> 'batavia-cmdb-mariadb-2' performed. 2019-02-20 21:53:54 notice : Server changed state: batavia-cmdb-mariadb-2[192.168.1.212:3306]: new_master. [Slave, Running] -> [Master, Running] 2019-02-20 21:53:54 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] NEW Master: [192.168.1.212]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:53:54.358Z", "type": "log"} 2019-02-20 21:53:54 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for batavia-cmdb-mariadb-2 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:53:54.360Z", "type": "log"} 2019-02-20 21:53:54 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.212]:3306 --event=new_master --servers=[192.168.1.212]:3306,[192.168.1.11]:3306,[192.168.1.208]:3306 --masters=[192.168.1.212]:3306 --slaves=[192.168.1.208]:3306' on event 'new_master' 2019-02-20 21:56:01 notice : (26174) Updated server 'batavia-cmdb-mariadb-1': address=192.168.1.12 2019-02-20 21:56:03 notice : Server changed state: batavia-cmdb-mariadb-1[192.168.1.12:3306]: slave_up. [Down] -> [Slave, Running] 2019-02-20 21:56:04 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [192.168.1.12]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "batavia-cmdb-maxscale-0", "version": "1.0", "time": "2019-02-20T21:56:04.082Z", "type": "log"} 2019-02-20 21:56:04 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.12]:3306 --event=slave_up --servers=[192.168.1.212]:3306,[192.168.1.12]:3306,[192.168.1.208]:3306 --masters=[192.168.1.212]:3306 --slaves=[192.168.1.12]:3306,[192.168.1.208]:3306' on event 'slave_up'