MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jan 31 16:17:17 2019 ---------------------------------------------------------------------------- 2019-01-31 16:17:17 notice : The collection of SQLite memory allocation statistics turned off. 2019-01-31 16:17:17 notice : Threading mode of SQLite set to Multi-thread. 2019-01-31 16:17:17 notice : MariaDB MaxScale 2.2.18 started (Commit: 136c0c3062eb0bb51be4f427ec24303b62b71a9e) 2019-01-31 16:17:17 notice : MaxScale is running in process 233 2019-01-31 16:17:17 notice : Configuration file: /etc/maxscale.cnf 2019-01-31 16:17:17 notice : Log directory: /var/log/maxscale 2019-01-31 16:17:17 notice : Data directory: /var/lib/maxscale 2019-01-31 16:17:17 notice : Module directory: /usr/lib64/maxscale 2019-01-31 16:17:17 notice : Service cache: /var/cache/maxscale 2019-01-31 16:17:17 notice : Loading /etc/maxscale.cnf. 2019-01-31 16:17:17 notice : Loading /etc/maxscale.cnf.d/dca-mariadb-1.cnf. 2019-01-31 16:17:17 notice : Loading /etc/maxscale.cnf.d/dca-mariadb-0.cnf. 2019-01-31 16:17:17 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2019-01-31 16:17:17 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2019-01-31 16:17:17 notice : [cli] Initialise CLI router module 2019-01-31 16:17:17 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2019-01-31 16:17:17 notice : [readconnroute] Initialise readconnroute router module. 2019-01-31 16:17:17 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2019-01-31 16:17:17 notice : [readwritesplit] Initializing statement-based read/write split router module. 2019-01-31 16:17:17 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2019-01-31 16:17:17 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2019-01-31 16:17:17 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2019-01-31 16:17:17 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2019-01-31 16:17:17 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2019-01-31 16:17:17 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2019-01-31 16:17:17 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2019-01-31 16:17:17 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2019-01-31 16:17:17 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2019-01-31 16:17:17 notice : No query classifier specified, using default 'qc_sqlite'. 2019-01-31 16:17:17 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2019-01-31 16:17:17 warning: Removing stale journal file for monitor 'MariaDB-Monitor'. 2019-01-31 16:17:17 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2019-01-31 16:17:17 error : [MariaDB-Monitor] Failed to connect to server 'dca-mariadb-1' ([172.17.0.4]:3306) when checking monitor user credentials and permissions: Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MariaDB-Monitor] Failed to connect to server 'dca-mariadb-0' ([172.17.0.3]:3306) when checking monitor user credentials and permissions: Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 notice : Starting a total of 3 services... 2019-01-31 16:17:17 warning: Service 'RWSplit-Service' has no listeners defined. 2019-01-31 16:17:17 notice : Service 'RWSplit-Service' started (1/3) 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Failed to connect to server 'dca-mariadb-1' ([172.17.0.4]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Failed to connect to server 'dca-mariadb-0' ([172.17.0.3]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.4:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.3:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Write-Listener listening at [::]:3306. 2019-01-31 16:17:17 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-01-31 16:17:17 notice : Listening for connections at [::]:3306 with protocol MySQL 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Failed to connect to server 'dca-mariadb-1' ([172.17.0.4]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Failed to connect to server 'dca-mariadb-0' ([172.17.0.3]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.4:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.3:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2019-01-31 16:17:17 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-01-31 16:17:17 notice : Listening for connections at [::]:3307 with protocol MySQL 2019-01-31 16:17:17 notice : Service 'RO-Service' started (2/3) 2019-01-31 16:17:17 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2019-01-31 16:17:17 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2019-01-31 16:17:17 notice : Service 'CLI-Service' started (3/3) 2019-01-31 16:17:17 notice : Started REST API on [127.0.0.1]:8989 2019-01-31 16:17:17 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.4:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.3:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Write-Listener listening at [::]:3306. 2019-01-31 16:17:17 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.4:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.3:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.3' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2019-01-31 16:17:17 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-01-31 16:17:17 notice : Started MaxScale log flusher. 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.4:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.4' (115) 2019-01-31 16:17:17 error : [MySQLAuth] Failure loading users data from backend [172.17.0.3:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '172.17.0.3' (115) (subsequent similar messages suppressed for 10000 milliseconds) 2019-01-31 16:17:17 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Write-Listener listening at [::]:3306. 2019-01-31 16:17:17 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-01-31 16:17:17 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2019-01-31 16:17:17 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2019-01-31 16:17:17 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2019-01-31 16:17:17 error : Monitor was unable to connect to server [172.17.0.4]:3306 : "Can't connect to MySQL server on '172.17.0.4' (115)" 2019-01-31 16:17:17 error : Monitor was unable to connect to server [172.17.0.3]:3306 : "Can't connect to MySQL server on '172.17.0.3' (115)" 2019-01-31 16:17:17 notice : Server changed state: dca-mariadb-1[172.17.0.4:3306]: server_down. [Running] -> [Down] 2019-01-31 16:17:17 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Server DOWN: [172.17.0.4]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:17:17.587Z", "type": "log"} 2019-01-31 16:17:17 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[172.17.0.4]:3306 --event=server_down --servers=[172.17.0.4]:3306,[172.17.0.3]:3306 --masters= --slaves=' on event 'server_down' 2019-01-31 16:17:17 notice : Server changed state: dca-mariadb-0[172.17.0.3:3306]: server_down. [Running] -> [Down] 2019-01-31 16:17:17 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Server DOWN: [172.17.0.3]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:17:17.876Z", "type": "log"} 2019-01-31 16:17:17 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[172.17.0.3]:3306 --event=server_down --servers=[172.17.0.4]:3306,[172.17.0.3]:3306 --masters= --slaves=' on event 'server_down' 2019-01-31 16:17:17 error : [mariadbmon] No Master can be determined 2019-01-31 16:17:20 notice : (12) Updated 'passive' from 'true' to 'false' 2019-01-31 16:17:27 warning: [mariadbmon] Setting standalone master, server 'dca-mariadb-0' is now the master. 2019-01-31 16:17:27 notice : Server changed state: dca-mariadb-0[172.17.0.3:3306]: master_up. [Down] -> [Master, Running] 2019-01-31 16:17:28 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master UP: [172.17.0.3]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:17:28.152Z", "type": "log"} 2019-01-31 16:17:28 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "SQL [START SLAVE;] FAILED: (1200, 'Misconfigured slave: MASTER_HOST was not set; Fix in config file or with CHANGE MASTER TO')"}, "service": "maxscale", "level": "warning", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:17:28.270Z", "type": "log"} 2019-01-31 16:17:28 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [yes] for dca-mariadb-0 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:17:28.272Z", "type": "log"} 2019-01-31 16:17:28 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[172.17.0.3]:3306 --event=master_up --servers=[172.17.0.4]:3306,[172.17.0.3]:3306 --masters=[172.17.0.3]:3306 --slaves=' on event 'master_up' 2019-01-31 16:17:33 notice : Server changed state: dca-mariadb-1[172.17.0.4:3306]: slave_up. [Down] -> [Slave, Running] 2019-01-31 16:17:33 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Slave UP: [172.17.0.4]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:17:33.589Z", "type": "log"} 2019-01-31 16:17:33 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[172.17.0.4]:3306 --event=slave_up --servers=[172.17.0.4]:3306,[172.17.0.3]:3306 --masters=[172.17.0.3]:3306 --slaves=[172.17.0.4]:3306' on event 'slave_up' 2019-01-31 16:17:39 notice : [mariadbmon] Cluster master server is replicating from an external master: 172.17.0.6:3306 2019-01-31 16:22:30 error : Monitor was unable to connect to server [172.17.0.3]:3306 : "Can't connect to MySQL server on '172.17.0.3' (115)" 2019-01-31 16:22:30 notice : [mariadbmon] Server [172.17.0.3]:3306 lost the master status. 2019-01-31 16:22:30 notice : Server changed state: dca-mariadb-0[172.17.0.3:3306]: master_down. [Master, Slave of External Server, Running] -> [Down] 2019-01-31 16:22:30 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "[NOTIFY] Master DOWN: [172.17.0.3]:3306"}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:22:30.689Z", "type": "log"} 2019-01-31 16:22:30 notice : /usr/lib/maxscale/maxscale_notify.py: {"log": {"message": "Setting mariadb-master label to [] for dca-mariadb-0 "}, "service": "maxscale", "level": "info", "process": "maxscale_notify.py", "host": "dcA-maxscale-0", "version": "1.0", "time": "2019-01-31T16:22:30.690Z", "type": "log"} 2019-01-31 16:22:30 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[172.17.0.3]:3306 --event=master_down --servers=[172.17.0.4]:3306,[172.17.0.3]:3306 --masters= --slaves=[172.17.0.4]:3306' on event 'master_down' 2019-01-31 16:22:30 warning: [mariadbmon] Master has failed. If master status does not change in 3 monitor passes, failover begins. 2019-01-31 16:22:33 notice : [mariadbmon] Performing automatic failover to replace failed master 'dca-mariadb-0'. 2019-01-31 16:22:33 warning: [mariadbmon] Slave 'dca-mariadb-1' has gtid_strict_mode disabled. Enabling this setting is recommended. For more information, see https://mariadb.com/kb/en/library/gtid/#gtid_strict_mode 2019-01-31 16:22:33 notice : [mariadbmon] Promoting server 'dca-mariadb-1' to master. 2019-01-31 16:22:33 notice : [mariadbmon] Executing sql queries from file '/etc/mariadb/promotion.sql' on server 'dca-mariadb-1'. 2019-01-31 16:22:33 notice : [mariadbmon] 2 queries executed successfully. 2019-01-31 16:22:33 notice : [mariadbmon] New master starting replication from external master 172.17.0.6:3306. 2019-01-31 16:22:33 notice : [mariadbmon] Redirecting slaves to new master. 2019-01-31 16:22:33 warning: [mariadbmon] Replicating from external master, skipping final check.