MariaDB MaxScale /var/log/maxscale/maxscale.log Tue Jun 5 07:10:04 2018 ---------------------------------------------------------------------------- 2018-06-05 07:10:04 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-05 07:10:04 notice : Threading mode of SQLite set to Multi-thread. 2018-06-05 07:10:04 notice : MariaDB MaxScale 2.2.5 started 2018-06-05 07:10:04 notice : MaxScale is running in process 780 2018-06-05 07:10:04 notice : Configuration file: /etc/maxscale.cnf 2018-06-05 07:10:04 notice : Log directory: /var/log/maxscale 2018-06-05 07:10:04 notice : Data directory: /var/lib/maxscale 2018-06-05 07:10:04 notice : Module directory: /usr/lib64/maxscale 2018-06-05 07:10:04 notice : Service cache: /var/cache/maxscale 2018-06-05 07:10:04 notice : Loading /etc/maxscale.cnf. 2018-06-05 07:10:04 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-05 07:10:04 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-05 07:10:04 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-05 07:10:04 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-05 07:10:04 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-05 07:10:04 notice : [cli] Initialise CLI router module 2018-06-05 07:10:04 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-05 07:10:04 notice : [readconnroute] Initialise readconnroute router module. 2018-06-05 07:10:04 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-05 07:10:04 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-05 07:10:04 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-05 07:10:04 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-05 07:10:04 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-05 07:10:04 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-05 07:10:04 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-05 07:10:04 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-05 07:10:04 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-05 07:10:04 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-05 07:10:04 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-05 07:10:04 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-05 07:10:04 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-05 07:10:04 warning: Removing stale journal file for monitor 'MariaDB-Monitor'. 2018-06-05 07:10:04 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-05 07:10:04 error : [MariaDB-Monitor] Failed to connect to server 'mdb-dc2-mariadb-2' ([192.168.1.46]:3306) when checking monitor user credentials and permissions: Can't connect to MySQL server on '192.168.1.46' (115) 2018-06-05 07:10:04 error : [MariaDB-Monitor] Failed to connect to server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306) when checking monitor user credentials and permissions: Can't connect to MySQL server on '192.168.1.124' (115) 2018-06-05 07:10:04 error : [MariaDB-Monitor] Failed to connect to server 'mdb-dc2-mariadb-1' ([192.168.1.118]:3306) when checking monitor user credentials and permissions: Can't connect to MySQL server on '192.168.1.118' (115) 2018-06-05 07:10:04 notice : Starting a total of 3 services... 2018-06-05 07:10:04 error : [MySQLAuth] [RWSplit-Service] Failed to connect to server 'mdb-dc2-mariadb-2' ([192.168.1.46]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.46' (115) 2018-06-05 07:10:04 error : [MySQLAuth] [RWSplit-Service] Failed to connect to server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.124' (115) 2018-06-05 07:10:04 error : [MySQLAuth] [RWSplit-Service] Failed to connect to server 'mdb-dc2-mariadb-1' ([192.168.1.118]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.118' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.46:3306] for service [RWSplit-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.46' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.124:3306] for service [RWSplit-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.124' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.118:3306] for service [RWSplit-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.118' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Unable to get user data from backend database for service [RWSplit-Service]. Failed to connect to any of the backend databases. 2018-06-05 07:10:04 error : [MySQLAuth] [RWSplit-Service] Unable to load users for listener Read-Write-Listener listening at [::]:3306. 2018-06-05 07:10:04 notice : [MySQLAuth] [RWSplit-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2018-06-05 07:10:04 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-05 07:10:04 notice : Service 'RWSplit-Service' started (1/3) 2018-06-05 07:10:04 error : [MySQLAuth] [RO-Service] Failed to connect to server 'mdb-dc2-mariadb-2' ([192.168.1.46]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.46' (115) 2018-06-05 07:10:04 error : [MySQLAuth] [RO-Service] Failed to connect to server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.124' (115) 2018-06-05 07:10:04 error : [MySQLAuth] [RO-Service] Failed to connect to server 'mdb-dc2-mariadb-1' ([192.168.1.118]:3306) when checking authentication user credentials and permissions: 2002 Can't connect to MySQL server on '192.168.1.118' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.46:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.46' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.124:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.124' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.118:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.118' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2018-06-05 07:10:04 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2018-06-05 07:10:04 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2018-06-05 07:10:04 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-05 07:10:04 notice : Service 'RO-Service' started (2/3) 2018-06-05 07:10:04 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-05 07:10:04 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-05 07:10:04 notice : Service 'CLI-Service' started (3/3) 2018-06-05 07:10:04 notice : Started MaxScale log flusher. 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.46:3306] for service [RWSplit-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.46' (115) 2018-06-05 07:10:04 notice : Started REST API on [127.0.0.1]:8989 2018-06-05 07:10:04 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.124:3306] for service [RWSplit-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.124' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.118:3306] for service [RWSplit-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.118' (115) 2018-06-05 07:10:04 error : [MySQLAuth] Unable to get user data from backend database for service [RWSplit-Service]. Failed to connect to any of the backend databases. 2018-06-05 07:10:04 error : [MySQLAuth] [RWSplit-Service] Unable to load users for listener Read-Write-Listener listening at [::]:3306. 2018-06-05 07:10:04 notice : [MySQLAuth] [RWSplit-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2018-06-05 07:10:04 error : [MySQLAuth] Failure loading users data from backend [192.168.1.46:3306] for service [RO-Service]. MySQL error 2002, Can't connect to MySQL server on '192.168.1.46' (115) (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-05 07:10:04 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2018-06-05 07:10:04 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2018-06-05 07:10:04 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2018-06-05 07:10:04 error : [MySQLAuth] Unable to get user data from backend database for service [RWSplit-Service]. Failed to connect to any of the backend databases. 2018-06-05 07:10:04 error : [MySQLAuth] [RWSplit-Service] Unable to load users for listener Read-Write-Listener listening at [::]:3306. 2018-06-05 07:10:04 notice : [MySQLAuth] [RWSplit-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2018-06-05 07:10:04 error : [MySQLAuth] Unable to get user data from backend database for service [RO-Service]. Failed to connect to any of the backend databases. 2018-06-05 07:10:04 error : [MySQLAuth] [RO-Service] Unable to load users for listener Read-Only-Listener listening at [::]:3307. 2018-06-05 07:10:04 notice : [MySQLAuth] [RO-Service] No users were loaded but 'inject_service_user' is enabled. Enabling service credentials for authentication until database users have been successfully loaded. 2018-06-05 07:10:05 error : Monitor was unable to connect to server [192.168.1.46]:3306 : "Can't connect to MySQL server on '192.168.1.46' (115)" 2018-06-05 07:10:05 error : Monitor was unable to connect to server [192.168.1.124]:3306 : "Can't connect to MySQL server on '192.168.1.124' (115)" 2018-06-05 07:10:05 error : Monitor was unable to connect to server [192.168.1.118]:3306 : "Can't connect to MySQL server on '192.168.1.118' (115)" 2018-06-05 07:10:05 notice : Server changed state: mdb-dc2-mariadb-2[192.168.1.46:3306]: server_down. [Running] -> [Down] 2018-06-05 07:10:05 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.46]:3306 --event=server_down --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=' on event 'server_down' 2018-06-05 07:10:05 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: server_down. [Running] -> [Down] 2018-06-05 07:10:05 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=server_down --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=' on event 'server_down' 2018-06-05 07:10:05 notice : Server changed state: mdb-dc2-mariadb-1[192.168.1.118:3306]: server_down. [Running] -> [Down] 2018-06-05 07:10:05 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.118]:3306 --event=server_down --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=' on event 'server_down' 2018-06-05 07:10:05 error : [mariadbmon] No Master can be determined 2018-06-05 07:10:06 notice : Server changed state: mdb-dc2-mariadb-1[192.168.1.118:3306]: slave_up. [Down] -> [Slave, Running] 2018-06-05 07:10:06 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.118]:3306 --event=slave_up --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.118]:3306' on event 'slave_up' 2018-06-05 07:10:07 error : Monitor was unable to connect to server [192.168.1.118]:3306 : "Can't connect to MySQL server on '192.168.1.118' (115)" 2018-06-05 07:10:07 notice : Server changed state: mdb-dc2-mariadb-1[192.168.1.118:3306]: slave_down. [Slave, Running] -> [Slave, Down] 2018-06-05 07:10:08 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.118]:3306 --event=slave_down --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.118]:3306' on event 'slave_down' 2018-06-05 07:10:10 notice : Server changed state: mdb-dc2-mariadb-2[192.168.1.46:3306]: slave_up. [Down] -> [Slave, Running] 2018-06-05 07:10:10 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.46]:3306 --event=slave_up --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.46]:3306,[192.168.1.118]:3306' on event 'slave_up' 2018-06-05 07:10:11 error : Monitor was unable to connect to server [192.168.1.46]:3306 : "Can't connect to MySQL server on '192.168.1.46' (115)" 2018-06-05 07:10:11 notice : Server changed state: mdb-dc2-mariadb-2[192.168.1.46:3306]: slave_down. [Slave, Running] -> [Slave, Down] 2018-06-05 07:10:11 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.46]:3306 --event=slave_down --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.46]:3306,[192.168.1.118]:3306' on event 'slave_down' 2018-06-05 07:10:14 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: slave_up. [Down] -> [Slave, Running] 2018-06-05 07:10:14 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=slave_up --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'slave_up' 2018-06-05 07:10:16 notice : Server changed state: mdb-dc2-mariadb-1[192.168.1.118:3306]: slave_up. [Slave, Down] -> [Slave, Running] 2018-06-05 07:10:16 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.118]:3306 --event=slave_up --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'slave_up' 2018-06-05 07:10:21 notice : Server changed state: mdb-dc2-mariadb-2[192.168.1.46:3306]: slave_up. [Slave, Down] -> [Slave, Running] 2018-06-05 07:10:21 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.46]:3306 --event=slave_up --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters= --slaves=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'slave_up' 2018-06-05 07:10:45 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-06-05 07:10:45 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-06-05 07:11:13 error : [readwritesplit] Couldn't find suitable Master from 3 candidates. 2018-06-05 07:11:13 error : Failed to create new router session for service 'RWSplit-Service'. See previous errors for more details. 2018-06-05 07:11:14 notice : Server changed state: mdb-dc2-mariadb-2[192.168.1.46:3306]: new_master. [Slave, Running] -> [Master, Slave of External Server, Running] 2018-06-05 07:11:15 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.46]:3306 --event=new_master --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_master' 2018-06-05 07:11:15 notice : [mariadbmon] A Master Server is now available: 192.168.1.46:3306 2018-06-05 07:11:16 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-05 09:29:26 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-05 09:29:26 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-05 09:29:26 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-05 09:29:26 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f67acd3b5e0]: sigaction.c:? 2018-06-05 09:29:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f67ad24d4aa]: ??:0 2018-06-05 09:29:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f67ad211311]: ??:0 2018-06-05 09:29:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f67ad221997]: ??:0 2018-06-05 09:29:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f67ad24e891]: ??:0 2018-06-05 09:29:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f67ad24eb02]: ??:0 2018-06-05 09:29:26 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-05 09:29:26 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f67aac40c05]: ??:? 2018-06-05 09:29:26 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Tue Jun 5 09:29:27 2018 ---------------------------------------------------------------------------- 2018-06-05 09:29:27 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-05 09:29:27 notice : Threading mode of SQLite set to Multi-thread. 2018-06-05 09:29:27 notice : MariaDB MaxScale 2.2.5 started 2018-06-05 09:29:27 notice : MaxScale is running in process 19022 2018-06-05 09:29:27 notice : Configuration file: /etc/maxscale.cnf 2018-06-05 09:29:27 notice : Log directory: /var/log/maxscale 2018-06-05 09:29:27 notice : Data directory: /var/lib/maxscale 2018-06-05 09:29:27 notice : Module directory: /usr/lib64/maxscale 2018-06-05 09:29:27 notice : Service cache: /var/cache/maxscale 2018-06-05 09:29:27 notice : Loading /etc/maxscale.cnf. 2018-06-05 09:29:27 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-05 09:29:27 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-05 09:29:27 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-05 09:29:27 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-05 09:29:27 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-05 09:29:27 notice : [cli] Initialise CLI router module 2018-06-05 09:29:27 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-05 09:29:27 notice : [readconnroute] Initialise readconnroute router module. 2018-06-05 09:29:27 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-05 09:29:27 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-05 09:29:27 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-05 09:29:27 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-05 09:29:27 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-05 09:29:27 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-05 09:29:27 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-05 09:29:27 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-05 09:29:27 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-05 09:29:27 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-05 09:29:27 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-05 09:29:27 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-05 09:29:27 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-05 09:29:27 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-05 09:29:27 notice : Starting a total of 3 services... 2018-06-05 09:29:27 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-05 09:29:27 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-05 09:29:27 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-05 09:29:27 notice : Service 'RWSplit-Service' started (1/3) 2018-06-05 09:29:27 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-05 09:29:27 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-05 09:29:27 notice : Service 'RO-Service' started (2/3) 2018-06-05 09:29:27 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-05 09:29:27 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-05 09:29:27 notice : Service 'CLI-Service' started (3/3) 2018-06-05 09:29:27 notice : Started MaxScale log flusher. 2018-06-05 09:29:27 notice : Started REST API on [127.0.0.1]:8989 2018-06-05 09:29:27 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-05 09:29:27 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-05 11:04:15 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-05 11:04:15 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-05 11:04:15 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-05 11:04:15 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-05 11:04:15 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-05 11:04:15 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 11:04:15 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-05 11:04:17 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-05 11:04:17 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-05 13:23:34 warning: [MySQLAuth] RWSplit-Service: login attempt for user 'cdp-reporting'@[::ffff:192.168.1.231]:40790, authentication failed. 2018-06-05 14:46:49 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-05 14:46:49 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-05 14:46:49 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-05 14:46:49 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-05 14:46:50 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-05 14:46:50 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-05 22:33:46 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-05 22:33:46 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-05 22:33:46 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-05 22:33:46 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f1447a8f5e0]: sigaction.c:? 2018-06-05 22:33:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f1447fa14aa]: ??:0 2018-06-05 22:33:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f1447f65311]: ??:0 2018-06-05 22:33:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f1447f75997]: ??:0 2018-06-05 22:33:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f1447fa2891]: ??:0 2018-06-05 22:33:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f1447fa2b02]: ??:0 2018-06-05 22:33:46 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-05 22:33:46 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f1445994c05]: ??:? 2018-06-05 22:33:46 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Tue Jun 5 22:33:47 2018 ---------------------------------------------------------------------------- 2018-06-05 22:33:47 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-05 22:33:47 notice : Threading mode of SQLite set to Multi-thread. 2018-06-05 22:33:47 notice : MariaDB MaxScale 2.2.5 started 2018-06-05 22:33:47 notice : MaxScale is running in process 10808 2018-06-05 22:33:47 notice : Configuration file: /etc/maxscale.cnf 2018-06-05 22:33:47 notice : Log directory: /var/log/maxscale 2018-06-05 22:33:47 notice : Data directory: /var/lib/maxscale 2018-06-05 22:33:47 notice : Module directory: /usr/lib64/maxscale 2018-06-05 22:33:47 notice : Service cache: /var/cache/maxscale 2018-06-05 22:33:47 notice : Loading /etc/maxscale.cnf. 2018-06-05 22:33:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-05 22:33:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-05 22:33:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-05 22:33:47 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-05 22:33:47 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-05 22:33:47 notice : [cli] Initialise CLI router module 2018-06-05 22:33:47 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-05 22:33:47 notice : [readconnroute] Initialise readconnroute router module. 2018-06-05 22:33:47 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-05 22:33:47 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-05 22:33:47 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-05 22:33:47 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-05 22:33:47 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-05 22:33:47 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-05 22:33:47 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-05 22:33:47 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-05 22:33:47 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-05 22:33:47 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-05 22:33:47 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-05 22:33:47 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-05 22:33:47 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-05 22:33:47 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-05 22:33:47 notice : Starting a total of 3 services... 2018-06-05 22:33:47 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-05 22:33:47 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-05 22:33:47 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-05 22:33:47 notice : Service 'RWSplit-Service' started (1/3) 2018-06-05 22:33:47 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-05 22:33:47 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-05 22:33:47 notice : Service 'RO-Service' started (2/3) 2018-06-05 22:33:47 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-05 22:33:47 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-05 22:33:47 notice : Service 'CLI-Service' started (3/3) 2018-06-05 22:33:47 notice : Started MaxScale log flusher. 2018-06-05 22:33:47 notice : Started REST API on [127.0.0.1]:8989 2018-06-05 22:33:47 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-05 22:33:47 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-05 22:48:16 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-05 22:48:16 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-05 22:48:16 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-05 22:48:16 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f5c2758b5e0]: sigaction.c:? 2018-06-05 22:48:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f5c27a9d4aa]: ??:0 2018-06-05 22:48:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f5c27a61311]: ??:0 2018-06-05 22:48:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f5c27a71997]: ??:0 2018-06-05 22:48:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f5c27a9e891]: ??:0 2018-06-05 22:48:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f5c27a9eb02]: ??:0 2018-06-05 22:48:16 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-05 22:48:16 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5c25490c05]: ??:? 2018-06-05 22:48:16 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Tue Jun 5 22:48:17 2018 ---------------------------------------------------------------------------- 2018-06-05 22:48:17 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-05 22:48:17 notice : Threading mode of SQLite set to Multi-thread. 2018-06-05 22:48:17 notice : MariaDB MaxScale 2.2.5 started 2018-06-05 22:48:17 notice : MaxScale is running in process 19666 2018-06-05 22:48:17 notice : Configuration file: /etc/maxscale.cnf 2018-06-05 22:48:17 notice : Log directory: /var/log/maxscale 2018-06-05 22:48:17 notice : Data directory: /var/lib/maxscale 2018-06-05 22:48:17 notice : Module directory: /usr/lib64/maxscale 2018-06-05 22:48:17 notice : Service cache: /var/cache/maxscale 2018-06-05 22:48:17 notice : Loading /etc/maxscale.cnf. 2018-06-05 22:48:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-05 22:48:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-05 22:48:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-05 22:48:17 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-05 22:48:17 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-05 22:48:17 notice : [cli] Initialise CLI router module 2018-06-05 22:48:17 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-05 22:48:17 notice : [readconnroute] Initialise readconnroute router module. 2018-06-05 22:48:17 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-05 22:48:17 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-05 22:48:17 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-05 22:48:17 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-05 22:48:17 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-05 22:48:17 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-05 22:48:17 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-05 22:48:17 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-05 22:48:17 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-05 22:48:17 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-05 22:48:17 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-05 22:48:17 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-05 22:48:17 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-05 22:48:17 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-05 22:48:17 notice : Starting a total of 3 services... 2018-06-05 22:48:17 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-05 22:48:17 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-05 22:48:17 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-05 22:48:17 notice : Service 'RWSplit-Service' started (1/3) 2018-06-05 22:48:17 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-05 22:48:17 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-05 22:48:17 notice : Service 'RO-Service' started (2/3) 2018-06-05 22:48:17 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-05 22:48:17 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-05 22:48:17 notice : Service 'CLI-Service' started (3/3) 2018-06-05 22:48:17 notice : Started MaxScale log flusher. 2018-06-05 22:48:17 notice : Started REST API on [127.0.0.1]:8989 2018-06-05 22:48:17 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-05 22:48:17 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-06 06:32:46 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-06 06:32:46 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-06 06:32:46 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-06 06:32:46 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f6b0262d5e0]: sigaction.c:? 2018-06-06 06:32:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f6b02b3f4aa]: ??:0 2018-06-06 06:32:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f6b02b03311]: ??:0 2018-06-06 06:32:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f6b02b13997]: ??:0 2018-06-06 06:32:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f6b02b40891]: ??:0 2018-06-06 06:32:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f6b02b40b02]: ??:0 2018-06-06 06:32:46 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-06 06:32:46 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f6b00532c05]: ??:? 2018-06-06 06:32:46 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Wed Jun 6 06:32:47 2018 ---------------------------------------------------------------------------- 2018-06-06 06:32:47 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-06 06:32:47 notice : Threading mode of SQLite set to Multi-thread. 2018-06-06 06:32:47 notice : MariaDB MaxScale 2.2.5 started 2018-06-06 06:32:47 notice : MaxScale is running in process 12444 2018-06-06 06:32:47 notice : Configuration file: /etc/maxscale.cnf 2018-06-06 06:32:47 notice : Log directory: /var/log/maxscale 2018-06-06 06:32:47 notice : Data directory: /var/lib/maxscale 2018-06-06 06:32:47 notice : Module directory: /usr/lib64/maxscale 2018-06-06 06:32:47 notice : Service cache: /var/cache/maxscale 2018-06-06 06:32:47 notice : Loading /etc/maxscale.cnf. 2018-06-06 06:32:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-06 06:32:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-06 06:32:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-06 06:32:47 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-06 06:32:47 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-06 06:32:47 notice : [cli] Initialise CLI router module 2018-06-06 06:32:47 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-06 06:32:47 notice : [readconnroute] Initialise readconnroute router module. 2018-06-06 06:32:47 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-06 06:32:47 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-06 06:32:47 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-06 06:32:47 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-06 06:32:47 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-06 06:32:47 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-06 06:32:47 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-06 06:32:47 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-06 06:32:47 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-06 06:32:47 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-06 06:32:47 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-06 06:32:47 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-06 06:32:47 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-06 06:32:47 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-06 06:32:47 notice : Starting a total of 3 services... 2018-06-06 06:32:47 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-06 06:32:47 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-06 06:32:47 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-06 06:32:47 notice : Service 'RWSplit-Service' started (1/3) 2018-06-06 06:32:47 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-06 06:32:47 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-06 06:32:47 notice : Service 'RO-Service' started (2/3) 2018-06-06 06:32:47 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-06 06:32:47 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-06 06:32:47 notice : Service 'CLI-Service' started (3/3) 2018-06-06 06:32:47 notice : Started MaxScale log flusher. 2018-06-06 06:32:47 notice : Started REST API on [127.0.0.1]:8989 2018-06-06 06:32:47 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-06 06:32:47 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-06 08:27:36 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-06 08:27:36 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-06 08:27:36 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-06 08:27:36 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-06 08:27:36 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-06 08:27:36 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:27:36 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-06 08:27:37 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-06 08:27:38 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-06 08:57:20 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-06 08:57:20 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-06 08:57:20 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-06 08:57:20 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-06 08:57:21 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-06 08:57:21 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-06 09:10:56 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-06 09:10:56 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-06 09:10:56 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-06 09:10:56 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f14259cc5e0]: sigaction.c:? 2018-06-06 09:10:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f1425ede4aa]: ??:0 2018-06-06 09:10:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f1425ea2311]: ??:0 2018-06-06 09:10:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f1425eb2997]: ??:0 2018-06-06 09:10:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f1425edf891]: ??:0 2018-06-06 09:10:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f1425edfb02]: ??:0 2018-06-06 09:10:56 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-06 09:10:56 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f14238d1c05]: ??:? 2018-06-06 09:10:56 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Wed Jun 6 09:10:57 2018 ---------------------------------------------------------------------------- 2018-06-06 09:10:57 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-06 09:10:57 notice : Threading mode of SQLite set to Multi-thread. 2018-06-06 09:10:57 notice : MariaDB MaxScale 2.2.5 started 2018-06-06 09:10:57 notice : MaxScale is running in process 12065 2018-06-06 09:10:57 notice : Configuration file: /etc/maxscale.cnf 2018-06-06 09:10:57 notice : Log directory: /var/log/maxscale 2018-06-06 09:10:57 notice : Data directory: /var/lib/maxscale 2018-06-06 09:10:57 notice : Module directory: /usr/lib64/maxscale 2018-06-06 09:10:57 notice : Service cache: /var/cache/maxscale 2018-06-06 09:10:57 notice : Loading /etc/maxscale.cnf. 2018-06-06 09:10:57 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-06 09:10:57 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-06 09:10:57 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-06 09:10:57 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-06 09:10:57 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-06 09:10:57 notice : [cli] Initialise CLI router module 2018-06-06 09:10:57 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-06 09:10:57 notice : [readconnroute] Initialise readconnroute router module. 2018-06-06 09:10:57 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-06 09:10:57 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-06 09:10:57 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-06 09:10:57 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-06 09:10:57 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-06 09:10:57 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-06 09:10:57 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-06 09:10:57 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-06 09:10:57 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-06 09:10:57 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-06 09:10:57 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-06 09:10:57 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-06 09:10:57 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-06 09:10:57 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-06 09:10:57 notice : Starting a total of 3 services... 2018-06-06 09:10:57 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-06 09:10:57 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-06 09:10:57 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-06 09:10:57 notice : Service 'RWSplit-Service' started (1/3) 2018-06-06 09:10:57 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-06 09:10:57 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-06 09:10:57 notice : Service 'RO-Service' started (2/3) 2018-06-06 09:10:57 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-06 09:10:57 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-06 09:10:57 notice : Service 'CLI-Service' started (3/3) 2018-06-06 09:10:57 notice : Started MaxScale log flusher. 2018-06-06 09:10:57 notice : Started REST API on [127.0.0.1]:8989 2018-06-06 09:10:57 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-06 09:10:57 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-06 15:19:46 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-06 15:19:46 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-06 15:19:46 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-06 15:19:46 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f4626bed5e0]: sigaction.c:? 2018-06-06 15:19:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f46270ff4aa]: ??:0 2018-06-06 15:19:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f46270c3311]: ??:0 2018-06-06 15:19:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f46270d3997]: ??:0 2018-06-06 15:19:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f4627100891]: ??:0 2018-06-06 15:19:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f4627100b02]: ??:0 2018-06-06 15:19:46 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-06 15:19:46 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4624af2c05]: ??:? 2018-06-06 15:19:46 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Wed Jun 6 15:19:47 2018 ---------------------------------------------------------------------------- 2018-06-06 15:19:47 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-06 15:19:47 notice : Threading mode of SQLite set to Multi-thread. 2018-06-06 15:19:47 notice : MariaDB MaxScale 2.2.5 started 2018-06-06 15:19:47 notice : MaxScale is running in process 11637 2018-06-06 15:19:47 notice : Configuration file: /etc/maxscale.cnf 2018-06-06 15:19:47 notice : Log directory: /var/log/maxscale 2018-06-06 15:19:47 notice : Data directory: /var/lib/maxscale 2018-06-06 15:19:47 notice : Module directory: /usr/lib64/maxscale 2018-06-06 15:19:47 notice : Service cache: /var/cache/maxscale 2018-06-06 15:19:47 notice : Loading /etc/maxscale.cnf. 2018-06-06 15:19:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-06 15:19:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-06 15:19:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-06 15:19:47 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-06 15:19:47 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-06 15:19:47 notice : [cli] Initialise CLI router module 2018-06-06 15:19:47 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-06 15:19:47 notice : [readconnroute] Initialise readconnroute router module. 2018-06-06 15:19:47 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-06 15:19:47 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-06 15:19:47 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-06 15:19:47 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-06 15:19:47 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-06 15:19:47 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-06 15:19:47 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-06 15:19:47 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-06 15:19:47 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-06 15:19:47 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-06 15:19:47 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-06 15:19:47 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-06 15:19:47 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-06 15:19:47 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-06 15:19:47 notice : Starting a total of 3 services... 2018-06-06 15:19:47 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-06 15:19:47 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-06 15:19:47 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-06 15:19:47 notice : Service 'RWSplit-Service' started (1/3) 2018-06-06 15:19:47 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-06 15:19:47 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-06 15:19:47 notice : Service 'RO-Service' started (2/3) 2018-06-06 15:19:47 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-06 15:19:47 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-06 15:19:47 notice : Service 'CLI-Service' started (3/3) 2018-06-06 15:19:47 notice : Started MaxScale log flusher. 2018-06-06 15:19:47 notice : Started REST API on [127.0.0.1]:8989 2018-06-06 15:19:47 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-06 15:19:47 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-06 22:44:26 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-06 22:44:26 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-06 22:44:26 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-06 22:44:26 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f5985a175e0]: sigaction.c:? 2018-06-06 22:44:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f5985f294aa]: ??:0 2018-06-06 22:44:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f5985eed311]: ??:0 2018-06-06 22:44:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f5985efd997]: ??:0 2018-06-06 22:44:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f5985f2a891]: ??:0 2018-06-06 22:44:26 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f5985f2ab02]: ??:0 2018-06-06 22:44:26 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-06 22:44:26 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f598391cc05]: ??:? 2018-06-06 22:44:26 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Wed Jun 6 22:44:27 2018 ---------------------------------------------------------------------------- 2018-06-06 22:44:27 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-06 22:44:27 notice : Threading mode of SQLite set to Multi-thread. 2018-06-06 22:44:27 notice : MariaDB MaxScale 2.2.5 started 2018-06-06 22:44:27 notice : MaxScale is running in process 24058 2018-06-06 22:44:27 notice : Configuration file: /etc/maxscale.cnf 2018-06-06 22:44:27 notice : Log directory: /var/log/maxscale 2018-06-06 22:44:27 notice : Data directory: /var/lib/maxscale 2018-06-06 22:44:27 notice : Module directory: /usr/lib64/maxscale 2018-06-06 22:44:27 notice : Service cache: /var/cache/maxscale 2018-06-06 22:44:27 notice : Loading /etc/maxscale.cnf. 2018-06-06 22:44:27 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-06 22:44:27 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-06 22:44:27 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-06 22:44:27 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-06 22:44:27 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-06 22:44:27 notice : [cli] Initialise CLI router module 2018-06-06 22:44:27 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-06 22:44:27 notice : [readconnroute] Initialise readconnroute router module. 2018-06-06 22:44:27 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-06 22:44:27 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-06 22:44:27 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-06 22:44:27 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-06 22:44:27 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-06 22:44:27 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-06 22:44:27 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-06 22:44:27 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-06 22:44:27 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-06 22:44:27 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-06 22:44:27 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-06 22:44:27 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-06 22:44:27 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-06 22:44:27 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-06 22:44:27 notice : Starting a total of 3 services... 2018-06-06 22:44:27 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-06 22:44:27 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-06 22:44:27 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-06 22:44:27 notice : Service 'RWSplit-Service' started (1/3) 2018-06-06 22:44:27 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-06 22:44:27 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-06 22:44:27 notice : Service 'RO-Service' started (2/3) 2018-06-06 22:44:27 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-06 22:44:27 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-06 22:44:27 notice : Service 'CLI-Service' started (3/3) 2018-06-06 22:44:27 notice : Started MaxScale log flusher. 2018-06-06 22:44:27 notice : Started REST API on [127.0.0.1]:8989 2018-06-06 22:44:27 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-06 22:44:27 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 01:25:16 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 01:25:16 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 01:25:16 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 01:25:16 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f2501c435e0]: sigaction.c:? 2018-06-07 01:25:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f25021554aa]: ??:0 2018-06-07 01:25:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f2502119311]: ??:0 2018-06-07 01:25:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f2502129997]: ??:0 2018-06-07 01:25:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f2502156891]: ??:0 2018-06-07 01:25:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f2502156b02]: ??:0 2018-06-07 01:25:16 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 01:25:16 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f24ffb48c05]: ??:? 2018-06-07 01:25:16 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 01:25:17 2018 ---------------------------------------------------------------------------- 2018-06-07 01:25:17 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 01:25:17 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 01:25:17 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 01:25:17 notice : MaxScale is running in process 25183 2018-06-07 01:25:17 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 01:25:17 notice : Log directory: /var/log/maxscale 2018-06-07 01:25:17 notice : Data directory: /var/lib/maxscale 2018-06-07 01:25:17 notice : Module directory: /usr/lib64/maxscale 2018-06-07 01:25:17 notice : Service cache: /var/cache/maxscale 2018-06-07 01:25:17 notice : Loading /etc/maxscale.cnf. 2018-06-07 01:25:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 01:25:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 01:25:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 01:25:17 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 01:25:17 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 01:25:17 notice : [cli] Initialise CLI router module 2018-06-07 01:25:17 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 01:25:17 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 01:25:17 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 01:25:17 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 01:25:17 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 01:25:17 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 01:25:17 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 01:25:17 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 01:25:17 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 01:25:17 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 01:25:17 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 01:25:17 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 01:25:17 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 01:25:17 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 01:25:17 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 01:25:17 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 01:25:17 notice : Starting a total of 3 services... 2018-06-07 01:25:17 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 01:25:17 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 01:25:17 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 01:25:17 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 01:25:17 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 01:25:17 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 01:25:17 notice : Service 'RO-Service' started (2/3) 2018-06-07 01:25:17 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 01:25:17 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 01:25:17 notice : Service 'CLI-Service' started (3/3) 2018-06-07 01:25:17 notice : Started MaxScale log flusher. 2018-06-07 01:25:17 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 01:25:17 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 01:25:17 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 04:32:43 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-07 04:32:43 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-07 04:32:43 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-07 04:32:43 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-07 04:32:43 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-07 04:32:43 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:32:43 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-07 04:32:44 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-07 04:32:44 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-07 04:35:42 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-07 04:35:42 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-07 04:35:42 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:42 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 04:35:44 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-07 04:35:44 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-07 04:35:44 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-07 04:36:16 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 04:36:16 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 04:36:16 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 04:36:16 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7fadb8ff15e0]: sigaction.c:? 2018-06-07 04:36:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7fadb95034aa]: ??:0 2018-06-07 04:36:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7fadb94c7311]: ??:0 2018-06-07 04:36:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7fadb94d7997]: ??:0 2018-06-07 04:36:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7fadb9504891]: ??:0 2018-06-07 04:36:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7fadb9504b02]: ??:0 2018-06-07 04:36:16 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 04:36:17 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fadb6ef6c05]: ??:? 2018-06-07 04:36:17 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-07 04:36:17 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-07 04:36:17 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 04:36:18 2018 ---------------------------------------------------------------------------- 2018-06-07 04:36:18 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 04:36:18 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 04:36:18 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 04:36:18 notice : MaxScale is running in process 12014 2018-06-07 04:36:18 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 04:36:18 notice : Log directory: /var/log/maxscale 2018-06-07 04:36:18 notice : Data directory: /var/lib/maxscale 2018-06-07 04:36:18 notice : Module directory: /usr/lib64/maxscale 2018-06-07 04:36:18 notice : Service cache: /var/cache/maxscale 2018-06-07 04:36:18 notice : Loading /etc/maxscale.cnf. 2018-06-07 04:36:18 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 04:36:18 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 04:36:18 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 04:36:18 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 04:36:18 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 04:36:18 notice : [cli] Initialise CLI router module 2018-06-07 04:36:18 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 04:36:18 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 04:36:18 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 04:36:18 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 04:36:18 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 04:36:18 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 04:36:18 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 04:36:18 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 04:36:18 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 04:36:18 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 04:36:18 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 04:36:18 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 04:36:18 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 04:36:18 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 04:36:18 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 04:36:18 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 04:36:18 notice : Starting a total of 3 services... 2018-06-07 04:36:18 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 04:36:18 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 04:36:18 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 04:36:18 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 04:36:18 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 04:36:18 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 04:36:18 notice : Service 'RO-Service' started (2/3) 2018-06-07 04:36:18 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 04:36:18 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 04:36:18 notice : Service 'CLI-Service' started (3/3) 2018-06-07 04:36:18 notice : Started MaxScale log flusher. 2018-06-07 04:36:18 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 04:36:18 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 04:36:18 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 09:13:12 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-07 09:13:12 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-07 09:13:12 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-07 09:13:12 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-07 09:13:13 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-07 09:13:13 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-07 09:13:13 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 09:13:14 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-07 09:13:14 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-07 11:49:36 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 11:49:36 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 11:49:36 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 11:49:36 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f884c4395e0]: sigaction.c:? 2018-06-07 11:49:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f884c94b4aa]: ??:0 2018-06-07 11:49:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f884c90f311]: ??:0 2018-06-07 11:49:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f884c91f997]: ??:0 2018-06-07 11:49:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f884c94c891]: ??:0 2018-06-07 11:49:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f884c94cb02]: ??:0 2018-06-07 11:49:36 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 11:49:36 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f884a33ec05]: ??:? 2018-06-07 11:49:36 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 11:49:37 2018 ---------------------------------------------------------------------------- 2018-06-07 11:49:37 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 11:49:37 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 11:49:37 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 11:49:37 notice : MaxScale is running in process 18019 2018-06-07 11:49:37 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 11:49:37 notice : Log directory: /var/log/maxscale 2018-06-07 11:49:37 notice : Data directory: /var/lib/maxscale 2018-06-07 11:49:37 notice : Module directory: /usr/lib64/maxscale 2018-06-07 11:49:37 notice : Service cache: /var/cache/maxscale 2018-06-07 11:49:37 notice : Loading /etc/maxscale.cnf. 2018-06-07 11:49:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 11:49:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 11:49:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 11:49:37 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 11:49:37 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 11:49:37 notice : [cli] Initialise CLI router module 2018-06-07 11:49:37 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 11:49:37 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 11:49:37 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 11:49:37 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 11:49:37 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 11:49:37 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 11:49:37 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 11:49:37 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 11:49:37 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 11:49:37 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 11:49:37 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 11:49:37 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 11:49:37 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 11:49:37 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 11:49:37 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 11:49:37 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 11:49:37 notice : Starting a total of 3 services... 2018-06-07 11:49:37 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 11:49:37 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 11:49:37 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 11:49:37 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 11:49:37 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 11:49:37 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 11:49:37 notice : Service 'RO-Service' started (2/3) 2018-06-07 11:49:37 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 11:49:37 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 11:49:37 notice : Service 'CLI-Service' started (3/3) 2018-06-07 11:49:37 notice : Started MaxScale log flusher. 2018-06-07 11:49:37 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 11:49:37 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 11:49:37 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 15:49:16 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 15:49:16 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 15:49:16 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 15:49:16 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f772a3495e0]: sigaction.c:? 2018-06-07 15:49:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f772a85b4aa]: ??:0 2018-06-07 15:49:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f772a81f311]: ??:0 2018-06-07 15:49:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f772a82f997]: ??:0 2018-06-07 15:49:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f772a85c891]: ??:0 2018-06-07 15:49:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f772a85cb02]: ??:0 2018-06-07 15:49:16 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 15:49:16 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f772824ec05]: ??:? 2018-06-07 15:49:16 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 15:49:17 2018 ---------------------------------------------------------------------------- 2018-06-07 15:49:17 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 15:49:17 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 15:49:17 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 15:49:17 notice : MaxScale is running in process 2252 2018-06-07 15:49:17 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 15:49:17 notice : Log directory: /var/log/maxscale 2018-06-07 15:49:17 notice : Data directory: /var/lib/maxscale 2018-06-07 15:49:17 notice : Module directory: /usr/lib64/maxscale 2018-06-07 15:49:17 notice : Service cache: /var/cache/maxscale 2018-06-07 15:49:17 notice : Loading /etc/maxscale.cnf. 2018-06-07 15:49:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 15:49:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 15:49:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 15:49:17 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 15:49:17 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 15:49:17 notice : [cli] Initialise CLI router module 2018-06-07 15:49:17 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 15:49:17 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 15:49:17 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 15:49:17 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 15:49:17 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 15:49:17 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 15:49:17 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 15:49:17 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 15:49:17 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 15:49:17 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 15:49:17 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 15:49:17 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 15:49:17 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 15:49:17 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 15:49:17 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 15:49:17 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 15:49:17 notice : Starting a total of 3 services... 2018-06-07 15:49:17 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 15:49:17 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 15:49:17 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 15:49:17 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 15:49:17 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 15:49:17 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 15:49:17 notice : Service 'RO-Service' started (2/3) 2018-06-07 15:49:17 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 15:49:17 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 15:49:17 notice : Service 'CLI-Service' started (3/3) 2018-06-07 15:49:17 notice : Started MaxScale log flusher. 2018-06-07 15:49:17 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 15:49:17 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 15:49:17 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 16:01:35 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-07 16:01:35 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-07 16:01:36 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-07 16:01:36 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-07 16:01:36 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-07 16:01:36 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 16:01:36 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-07 16:01:36 error : (3) Could not post DCB to worker. 2018-06-07 16:01:37 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-07 16:01:37 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-07 16:18:45 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 16:18:45 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 16:18:46 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 16:18:46 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f04d499c5e0]: sigaction.c:? 2018-06-07 16:18:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f04d4eae4aa]: ??:0 2018-06-07 16:18:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f04d4e72311]: ??:0 2018-06-07 16:18:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f04d4e82997]: ??:0 2018-06-07 16:18:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f04d4eaf891]: ??:0 2018-06-07 16:18:46 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f04d4eafb02]: ??:0 2018-06-07 16:18:46 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 16:18:46 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f04d28a1c05]: ??:? 2018-06-07 16:18:46 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 16:18:47 2018 ---------------------------------------------------------------------------- 2018-06-07 16:18:47 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 16:18:47 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 16:18:47 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 16:18:47 notice : MaxScale is running in process 20421 2018-06-07 16:18:47 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 16:18:47 notice : Log directory: /var/log/maxscale 2018-06-07 16:18:47 notice : Data directory: /var/lib/maxscale 2018-06-07 16:18:47 notice : Module directory: /usr/lib64/maxscale 2018-06-07 16:18:47 notice : Service cache: /var/cache/maxscale 2018-06-07 16:18:47 notice : Loading /etc/maxscale.cnf. 2018-06-07 16:18:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 16:18:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 16:18:47 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 16:18:47 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 16:18:47 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 16:18:47 notice : [cli] Initialise CLI router module 2018-06-07 16:18:47 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 16:18:47 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 16:18:47 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 16:18:47 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 16:18:47 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 16:18:47 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 16:18:47 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 16:18:47 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 16:18:47 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 16:18:47 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 16:18:47 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 16:18:47 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 16:18:47 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 16:18:47 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 16:18:47 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 16:18:47 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 16:18:47 notice : Starting a total of 3 services... 2018-06-07 16:18:47 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 16:18:47 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 16:18:47 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 16:18:47 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 16:18:47 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 16:18:47 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 16:18:47 notice : Service 'RO-Service' started (2/3) 2018-06-07 16:18:47 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 16:18:47 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 16:18:47 notice : Service 'CLI-Service' started (3/3) 2018-06-07 16:18:47 notice : Started MaxScale log flusher. 2018-06-07 16:18:47 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 16:18:47 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 16:18:47 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 19:22:56 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 19:22:56 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 19:22:56 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 19:22:56 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7fbef866b5e0]: sigaction.c:? 2018-06-07 19:22:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7fbef8b7d4aa]: ??:0 2018-06-07 19:22:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7fbef8b41311]: ??:0 2018-06-07 19:22:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7fbef8b51997]: ??:0 2018-06-07 19:22:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7fbef8b7e891]: ??:0 2018-06-07 19:22:56 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7fbef8b7eb02]: ??:0 2018-06-07 19:22:56 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 19:22:56 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fbef6570c05]: ??:? 2018-06-07 19:22:56 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 19:22:57 2018 ---------------------------------------------------------------------------- 2018-06-07 19:22:57 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 19:22:57 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 19:22:57 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 19:22:57 notice : MaxScale is running in process 3670 2018-06-07 19:22:57 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 19:22:57 notice : Log directory: /var/log/maxscale 2018-06-07 19:22:57 notice : Data directory: /var/lib/maxscale 2018-06-07 19:22:57 notice : Module directory: /usr/lib64/maxscale 2018-06-07 19:22:57 notice : Service cache: /var/cache/maxscale 2018-06-07 19:22:57 notice : Loading /etc/maxscale.cnf. 2018-06-07 19:22:57 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 19:22:57 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 19:22:57 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 19:22:57 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 19:22:57 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 19:22:57 notice : [cli] Initialise CLI router module 2018-06-07 19:22:57 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 19:22:57 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 19:22:57 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 19:22:57 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 19:22:57 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 19:22:57 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 19:22:57 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 19:22:57 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 19:22:57 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 19:22:57 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 19:22:57 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 19:22:57 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 19:22:57 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 19:22:57 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 19:22:57 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 19:22:57 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 19:22:57 notice : Starting a total of 3 services... 2018-06-07 19:22:57 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 19:22:57 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 19:22:57 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 19:22:57 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 19:22:57 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 19:22:57 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 19:22:57 notice : Service 'RO-Service' started (2/3) 2018-06-07 19:22:57 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 19:22:57 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 19:22:57 notice : Service 'CLI-Service' started (3/3) 2018-06-07 19:22:57 notice : Started MaxScale log flusher. 2018-06-07 19:22:57 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 19:22:57 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 19:22:57 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 22:27:15 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-07 22:27:16 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-07 22:27:16 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-07 22:27:16 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f57081b45e0]: sigaction.c:? 2018-06-07 22:27:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f57086c64aa]: ??:0 2018-06-07 22:27:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f570868a311]: ??:0 2018-06-07 22:27:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f570869a997]: ??:0 2018-06-07 22:27:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f57086c7891]: ??:0 2018-06-07 22:27:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f57086c7b02]: ??:0 2018-06-07 22:27:16 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-07 22:27:16 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f57060b9c05]: ??:? 2018-06-07 22:27:16 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Thu Jun 7 22:27:17 2018 ---------------------------------------------------------------------------- 2018-06-07 22:27:17 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-07 22:27:17 notice : Threading mode of SQLite set to Multi-thread. 2018-06-07 22:27:17 notice : MariaDB MaxScale 2.2.5 started 2018-06-07 22:27:17 notice : MaxScale is running in process 18172 2018-06-07 22:27:17 notice : Configuration file: /etc/maxscale.cnf 2018-06-07 22:27:17 notice : Log directory: /var/log/maxscale 2018-06-07 22:27:17 notice : Data directory: /var/lib/maxscale 2018-06-07 22:27:17 notice : Module directory: /usr/lib64/maxscale 2018-06-07 22:27:17 notice : Service cache: /var/cache/maxscale 2018-06-07 22:27:17 notice : Loading /etc/maxscale.cnf. 2018-06-07 22:27:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-07 22:27:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-07 22:27:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-07 22:27:17 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-07 22:27:17 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-07 22:27:17 notice : [cli] Initialise CLI router module 2018-06-07 22:27:17 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-07 22:27:17 notice : [readconnroute] Initialise readconnroute router module. 2018-06-07 22:27:17 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-07 22:27:17 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-07 22:27:17 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-07 22:27:17 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-07 22:27:17 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-07 22:27:17 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-07 22:27:17 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-07 22:27:17 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-07 22:27:17 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-07 22:27:17 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-07 22:27:17 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-07 22:27:17 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-07 22:27:17 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-07 22:27:17 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-07 22:27:17 notice : Starting a total of 3 services... 2018-06-07 22:27:17 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-07 22:27:17 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-07 22:27:17 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-07 22:27:17 notice : Service 'RWSplit-Service' started (1/3) 2018-06-07 22:27:17 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-07 22:27:17 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-07 22:27:17 notice : Service 'RO-Service' started (2/3) 2018-06-07 22:27:17 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-07 22:27:17 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-07 22:27:17 notice : Service 'CLI-Service' started (3/3) 2018-06-07 22:27:17 notice : Started MaxScale log flusher. 2018-06-07 22:27:17 notice : Started REST API on [127.0.0.1]:8989 2018-06-07 22:27:17 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-07 22:27:17 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-07 22:56:39 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.124]:3306): Lost connection to MySQL server during query 2018-06-07 22:56:39 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-07 22:56:39 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-07 22:56:39 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-07 22:56:40 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-07 22:56:40 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-07 22:56:40 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-07 22:56:41 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-07 22:56:41 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.124]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-08 02:47:07 error : Monitor was unable to connect to server [192.168.1.124]:3306 : "Can't connect to MySQL server on '192.168.1.124' (115)" 2018-06-08 02:47:07 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.124:3306]: slave_down. [Slave, Running] -> [Down] 2018-06-08 02:47:07 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.124]:3306 --event=slave_down --servers=[192.168.1.46]:3306,[192.168.1.124]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'slave_down' 2018-06-08 02:48:33 notice : (14200) Updated server 'mdb-dc2-mariadb-0': address=192.168.1.126 2018-06-08 02:48:36 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: server_up. [Down] -> [Running] 2018-06-08 02:48:36 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=server_up --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'server_up' 2018-06-08 02:48:37 error : Monitor was unable to connect to server [192.168.1.126]:3306 : "Can't connect to MySQL server on '192.168.1.126' (115)" 2018-06-08 02:48:37 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: server_down. [Running] -> [Down] 2018-06-08 02:48:37 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=server_down --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'server_down' 2018-06-08 02:48:43 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: server_up. [Down] -> [Running] 2018-06-08 02:48:43 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=server_up --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'server_up' 2018-06-08 02:48:44 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-08 02:48:45 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.126]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-08 03:48:36 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-08 03:48:36 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-08 03:48:36 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-08 03:48:36 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f298349c5e0]: sigaction.c:? 2018-06-08 03:48:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f29839ae4aa]: ??:0 2018-06-08 03:48:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f2983972311]: ??:0 2018-06-08 03:48:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f2983982997]: ??:0 2018-06-08 03:48:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f29839af891]: ??:0 2018-06-08 03:48:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f29839afb02]: ??:0 2018-06-08 03:48:36 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-08 03:48:36 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f29813a1c05]: ??:? 2018-06-08 03:48:36 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Fri Jun 8 03:48:37 2018 ---------------------------------------------------------------------------- 2018-06-08 03:48:37 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-08 03:48:37 notice : Threading mode of SQLite set to Multi-thread. 2018-06-08 03:48:37 notice : MariaDB MaxScale 2.2.5 started 2018-06-08 03:48:37 notice : MaxScale is running in process 20170 2018-06-08 03:48:37 notice : Configuration file: /etc/maxscale.cnf 2018-06-08 03:48:37 notice : Log directory: /var/log/maxscale 2018-06-08 03:48:37 notice : Data directory: /var/lib/maxscale 2018-06-08 03:48:37 notice : Module directory: /usr/lib64/maxscale 2018-06-08 03:48:37 notice : Service cache: /var/cache/maxscale 2018-06-08 03:48:37 notice : Loading /etc/maxscale.cnf. 2018-06-08 03:48:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 03:48:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-08 03:48:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-08 03:48:37 notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d' 2018-06-08 03:48:37 notice : Loading /var/lib/maxscale/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 03:48:37 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-08 03:48:37 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-08 03:48:37 notice : [cli] Initialise CLI router module 2018-06-08 03:48:37 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-08 03:48:37 notice : [readconnroute] Initialise readconnroute router module. 2018-06-08 03:48:37 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-08 03:48:37 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-08 03:48:37 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-08 03:48:37 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-08 03:48:37 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-08 03:48:37 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-08 03:48:37 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-08 03:48:37 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-08 03:48:37 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-08 03:48:37 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-08 03:48:37 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-08 03:48:37 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-08 03:48:37 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-08 03:48:37 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-08 03:48:37 notice : Starting a total of 3 services... 2018-06-08 03:48:37 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-08 03:48:37 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-08 03:48:37 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-08 03:48:37 notice : Service 'RWSplit-Service' started (1/3) 2018-06-08 03:48:37 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-08 03:48:37 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-08 03:48:37 notice : Service 'RO-Service' started (2/3) 2018-06-08 03:48:37 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-08 03:48:37 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-08 03:48:37 notice : Service 'CLI-Service' started (3/3) 2018-06-08 03:48:37 notice : Started MaxScale log flusher. 2018-06-08 03:48:37 notice : Started REST API on [127.0.0.1]:8989 2018-06-08 03:48:37 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-08 03:48:37 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-08 03:55:40 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.126]:3306): Lost connection to MySQL server during query 2018-06-08 03:55:40 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-08 03:55:41 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-08 03:55:41 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-08 03:55:41 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-08 03:55:41 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 03:55:41 error : Failed to write message: 11, Resource temporarily unavailable (subsequent similar messages suppressed for 10000 milliseconds) 2018-06-08 03:55:42 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: new_slave. [Running] -> [Slave, Running] 2018-06-08 03:55:42 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=new_slave --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.126]:3306,[192.168.1.118]:3306' on event 'new_slave' 2018-06-08 07:07:14 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-08 07:07:14 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-08 07:07:14 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-08 07:07:14 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f216411a5e0]: sigaction.c:? 2018-06-08 07:07:14 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f216462c4aa]: ??:0 2018-06-08 07:07:14 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f21645f0311]: ??:0 2018-06-08 07:07:14 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f2164600997]: ??:0 2018-06-08 07:07:14 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f216462d891]: ??:0 2018-06-08 07:07:14 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f216462db02]: ??:0 2018-06-08 07:07:14 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-08 07:07:14 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f216201fc05]: ??:? 2018-06-08 07:07:14 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Fri Jun 8 07:07:15 2018 ---------------------------------------------------------------------------- 2018-06-08 07:07:15 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-08 07:07:15 notice : Threading mode of SQLite set to Multi-thread. 2018-06-08 07:07:15 notice : MariaDB MaxScale 2.2.5 started 2018-06-08 07:07:15 notice : MaxScale is running in process 11829 2018-06-08 07:07:15 notice : Configuration file: /etc/maxscale.cnf 2018-06-08 07:07:15 notice : Log directory: /var/log/maxscale 2018-06-08 07:07:15 notice : Data directory: /var/lib/maxscale 2018-06-08 07:07:15 notice : Module directory: /usr/lib64/maxscale 2018-06-08 07:07:15 notice : Service cache: /var/cache/maxscale 2018-06-08 07:07:15 notice : Loading /etc/maxscale.cnf. 2018-06-08 07:07:15 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 07:07:15 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-08 07:07:15 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-08 07:07:15 notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d' 2018-06-08 07:07:15 notice : Loading /var/lib/maxscale/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 07:07:15 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-08 07:07:15 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-08 07:07:15 notice : [cli] Initialise CLI router module 2018-06-08 07:07:15 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-08 07:07:15 notice : [readconnroute] Initialise readconnroute router module. 2018-06-08 07:07:15 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-08 07:07:15 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-08 07:07:15 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-08 07:07:15 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-08 07:07:15 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-08 07:07:15 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-08 07:07:15 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-08 07:07:15 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-08 07:07:15 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-08 07:07:15 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-08 07:07:15 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-08 07:07:15 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-08 07:07:15 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-08 07:07:15 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-08 07:07:15 notice : Starting a total of 3 services... 2018-06-08 07:07:15 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-08 07:07:15 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-08 07:07:15 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-08 07:07:15 notice : Service 'RWSplit-Service' started (1/3) 2018-06-08 07:07:15 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-08 07:07:15 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-08 07:07:15 notice : Service 'RO-Service' started (2/3) 2018-06-08 07:07:15 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-08 07:07:15 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-08 07:07:15 notice : Service 'CLI-Service' started (3/3) 2018-06-08 07:07:15 notice : Started MaxScale log flusher. 2018-06-08 07:07:15 notice : Started REST API on [127.0.0.1]:8989 2018-06-08 07:07:15 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-08 07:07:15 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-08 08:19:36 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-08 08:19:36 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-08 08:19:36 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-08 08:19:36 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7fe1a042f5e0]: sigaction.c:? 2018-06-08 08:19:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7fe1a09414aa]: ??:0 2018-06-08 08:19:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7fe1a0905311]: ??:0 2018-06-08 08:19:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7fe1a0915997]: ??:0 2018-06-08 08:19:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7fe1a0942891]: ??:0 2018-06-08 08:19:36 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7fe1a0942b02]: ??:0 2018-06-08 08:19:36 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-08 08:19:36 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fe19e334c05]: ??:? 2018-06-08 08:19:36 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Fri Jun 8 08:19:37 2018 ---------------------------------------------------------------------------- 2018-06-08 08:19:37 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-08 08:19:37 notice : Threading mode of SQLite set to Multi-thread. 2018-06-08 08:19:37 notice : MariaDB MaxScale 2.2.5 started 2018-06-08 08:19:37 notice : MaxScale is running in process 23731 2018-06-08 08:19:37 notice : Configuration file: /etc/maxscale.cnf 2018-06-08 08:19:37 notice : Log directory: /var/log/maxscale 2018-06-08 08:19:37 notice : Data directory: /var/lib/maxscale 2018-06-08 08:19:37 notice : Module directory: /usr/lib64/maxscale 2018-06-08 08:19:37 notice : Service cache: /var/cache/maxscale 2018-06-08 08:19:37 notice : Loading /etc/maxscale.cnf. 2018-06-08 08:19:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 08:19:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-08 08:19:37 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-08 08:19:37 notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d' 2018-06-08 08:19:37 notice : Loading /var/lib/maxscale/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 08:19:37 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-08 08:19:37 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-08 08:19:37 notice : [cli] Initialise CLI router module 2018-06-08 08:19:37 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-08 08:19:37 notice : [readconnroute] Initialise readconnroute router module. 2018-06-08 08:19:37 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-08 08:19:37 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-08 08:19:37 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-08 08:19:37 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-08 08:19:37 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-08 08:19:37 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-08 08:19:37 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-08 08:19:37 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-08 08:19:37 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-08 08:19:37 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-08 08:19:37 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-08 08:19:37 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-08 08:19:37 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-08 08:19:37 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-08 08:19:37 notice : Starting a total of 3 services... 2018-06-08 08:19:37 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-08 08:19:37 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-08 08:19:37 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-08 08:19:37 notice : Service 'RWSplit-Service' started (1/3) 2018-06-08 08:19:37 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-08 08:19:37 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-08 08:19:37 notice : Service 'RO-Service' started (2/3) 2018-06-08 08:19:37 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-08 08:19:37 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-08 08:19:37 notice : Service 'CLI-Service' started (3/3) 2018-06-08 08:19:37 notice : Started MaxScale log flusher. 2018-06-08 08:19:37 notice : Started REST API on [127.0.0.1]:8989 2018-06-08 08:19:37 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-08 08:19:37 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-08 10:47:16 alert : Fatal: MaxScale 2.2.5 received fatal signal 11. Attempting backtrace. 2018-06-08 10:47:16 alert : Commit ID: d9e69413b4d18da24aa04e68e3ce9208159a267a System name: Linux Release string: CentOS Linux release 7.4.1708 (Core) 2018-06-08 10:47:16 alert : /usr/bin/maxscale() [0x4079d1]: ??:0 2018-06-08 10:47:16 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f340187d5e0]: sigaction.c:? 2018-06-08 10:47:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3getEi+0xa) [0x7f3401d8f4aa]: ??:0 2018-06-08 10:47:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(dcb_add_to_list+0x41) [0x7f3401d53311]: ??:0 2018-06-08 10:47:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12MessageQueue18handle_poll_eventsEij+0x117) [0x7f3401d63997]: ??:0 2018-06-08 10:47:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker15poll_waiteventsEv+0x1c1) [0x7f3401d90891]: ??:0 2018-06-08 10:47:16 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale6Worker3runEv+0x72) [0x7f3401d90b02]: ??:0 2018-06-08 10:47:16 alert : /usr/bin/maxscale(main+0x2437) [0x4067d7]: ??:0 2018-06-08 10:47:16 alert : /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f33ff782c05]: ??:? 2018-06-08 10:47:16 alert : /usr/bin/maxscale() [0x4069f5]: ??:0 MariaDB MaxScale /var/log/maxscale/maxscale.log Fri Jun 8 10:47:17 2018 ---------------------------------------------------------------------------- 2018-06-08 10:47:17 notice : The collection of SQLite memory allocation statistics turned off. 2018-06-08 10:47:17 notice : Threading mode of SQLite set to Multi-thread. 2018-06-08 10:47:17 notice : MariaDB MaxScale 2.2.5 started 2018-06-08 10:47:17 notice : MaxScale is running in process 16598 2018-06-08 10:47:17 notice : Configuration file: /etc/maxscale.cnf 2018-06-08 10:47:17 notice : Log directory: /var/log/maxscale 2018-06-08 10:47:17 notice : Data directory: /var/lib/maxscale 2018-06-08 10:47:17 notice : Module directory: /usr/lib64/maxscale 2018-06-08 10:47:17 notice : Service cache: /var/cache/maxscale 2018-06-08 10:47:17 notice : Loading /etc/maxscale.cnf. 2018-06-08 10:47:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 10:47:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-1.cnf. 2018-06-08 10:47:17 notice : Loading /etc/maxscale.cnf.d/mdb-dc2-mariadb-2.cnf. 2018-06-08 10:47:17 notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d' 2018-06-08 10:47:17 notice : Loading /var/lib/maxscale/maxscale.cnf.d/mdb-dc2-mariadb-0.cnf. 2018-06-08 10:47:17 notice : [mariadbmon] Initialise the MariaDB Monitor module. 2018-06-08 10:47:17 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so 2018-06-08 10:47:17 notice : [cli] Initialise CLI router module 2018-06-08 10:47:17 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so 2018-06-08 10:47:17 notice : [readconnroute] Initialise readconnroute router module. 2018-06-08 10:47:17 notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so 2018-06-08 10:47:17 notice : [readwritesplit] Initializing statement-based read/write split router module. 2018-06-08 10:47:17 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2018-06-08 10:47:17 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so 2018-06-08 10:47:17 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so 2018-06-08 10:47:17 notice : Monitor 'MariaDB-Monitor' is missing the 'journal_max_age' parameter, using default value of 28800 seconds. 2018-06-08 10:47:17 notice : Monitor 'MariaDB-Monitor' is missing the 'script_timeout' parameter, using default value of 90 seconds. 2018-06-08 10:47:17 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so 2018-06-08 10:47:17 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so 2018-06-08 10:47:17 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2018-06-08 10:47:17 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so 2018-06-08 10:47:17 notice : No query classifier specified, using default 'qc_sqlite'. 2018-06-08 10:47:17 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2018-06-08 10:47:17 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'. 2018-06-08 10:47:17 notice : Starting a total of 3 services... 2018-06-08 10:47:17 notice : Loaded server states from journal file: /var/lib/maxscale/MariaDB-Monitor/monitor.dat 2018-06-08 10:47:17 notice : [MySQLAuth] [RWSplit-Service] Loaded 11 MySQL users for listener Read-Write-Listener. 2018-06-08 10:47:17 notice : Listening for connections at [::]:3306 with protocol MySQL 2018-06-08 10:47:17 notice : Service 'RWSplit-Service' started (1/3) 2018-06-08 10:47:17 notice : [MySQLAuth] [RO-Service] Loaded 11 MySQL users for listener Read-Only-Listener. 2018-06-08 10:47:17 notice : Listening for connections at [::]:3307 with protocol MySQL 2018-06-08 10:47:17 notice : Service 'RO-Service' started (2/3) 2018-06-08 10:47:17 notice : Listening for connections at [/var/lib/maxscale/maxscale.sock]:0 with protocol MaxScale Admin 2018-06-08 10:47:17 notice : Listening for connections at [::]:6603 with protocol MaxScale Admin 2018-06-08 10:47:17 notice : Service 'CLI-Service' started (3/3) 2018-06-08 10:47:17 notice : Started MaxScale log flusher. 2018-06-08 10:47:17 notice : Started REST API on [127.0.0.1]:8989 2018-06-08 10:47:17 notice : MaxScale started with 2 worker threads, each with a stack size of 8388608 bytes. 2018-06-08 10:47:17 notice : [mariadbmon] Cluster master server is replicating from an external master: 10.254.16.19:3306 2018-06-08 13:10:24 error : Failed to execute query on server 'mdb-dc2-mariadb-0' ([192.168.1.126]:3306): Lost connection to MySQL server during query 2018-06-08 13:10:24 notice : Server changed state: mdb-dc2-mariadb-0[192.168.1.126:3306]: lost_slave. [Slave, Running] -> [Running] 2018-06-08 13:10:24 notice : Executed monitor script '/usr/lib/maxscale/maxscale_notify.py --initiator=[192.168.1.126]:3306 --event=lost_slave --servers=[192.168.1.46]:3306,[192.168.1.126]:3306,[192.168.1.118]:3306 --masters=[192.168.1.46]:3306 --slaves=[192.168.1.118]:3306' on event 'lost_slave' 2018-06-08 13:10:24 notice : [mariadbmon] Directing standalone server 'mdb-dc2-mariadb-0' to replicate from 'mdb-dc2-mariadb-2'. 2018-06-08 13:10:24 warning: [mariadbmon] Standalone server 'mdb-dc2-mariadb-0' failed to start replication: 'This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first'. Query: 'CHANGE MASTER TO ...'. 2018-06-08 13:10:24 error : [mariadbmon] A cluster join operation failed, disabling automatic rejoining. To re-enable, manually set 'auto_rejoin' to 'true' for monitor 'MariaDB-Monitor' via MaxAdmin or the REST API. 2018-06-08 13:10:24 error : Failed to write message: 11, Resource temporarily unavailable 2018-06-08 13:10:24 error : Failed to write message: 11, Resource temporarily unavailable