MariaDB MaxScale /data/clustrix/log/maxscale.log Thu Apr 8 20:51:05 2021 ----------------------------------------------------------------------------- 2021-04-08 20:51:05 notice : The collection of SQLite memory allocation statistics turned off. 2021-04-08 20:51:05 notice : Threading mode of SQLite set to Multi-thread. 2021-04-08 20:51:05 notice : Worker message queue size: 1MiB 2021-04-08 20:51:05 notice : The logging of informational messages has been enabled. 2021-04-08 20:51:05 notice : Using up to 7.05GiB of memory for query classifier cache 2021-04-08 20:51:05 notice : syslog logging is enabled. 2021-04-08 20:51:05 notice : maxlog logging is enabled. 2021-04-08 20:51:05 notice : Running OS: Linux@3.10.0-1127.19.1.el7.x86_64, #1 SMP Tue Aug 25 17:23:54 UTC 2020, x86_64 with 16 processor cores. 2021-04-08 20:51:05 notice : Total usable main memory: 47GiB. 2021-04-08 20:51:05 notice : MariaDB MaxScale 2.5.11 started (Commit: ab4c811e3dd4db26b30f15e29055cf81e9379e76) 2021-04-08 20:51:05 notice : MaxScale is running in process 9157 2021-04-08 20:51:05 notice : Configuration file: /etc/maxscale.conf 2021-04-08 20:51:05 notice : Log directory: /data/clustrix/log 2021-04-08 20:51:05 notice : Data directory: /var/lib/maxscale 2021-04-08 20:51:05 notice : Module directory: /usr/lib64/maxscale 2021-04-08 20:51:05 notice : Service cache: /var/cache/maxscale 2021-04-08 20:51:05 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2021-04-08 20:51:05 info : qc_sqlite loaded. 2021-04-08 20:51:05 notice : Query classification results are cached and reused. Memory used per thread: 451.2MiB 2021-04-08 20:51:05 notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140059134813952. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140059126421248. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140059118028544. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140059109635840. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140059101243136. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058749302528. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058740909824. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058732517120. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058724124416. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058715731712. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058707339008. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058698946304. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058212431616. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058204038912. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058195646208. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140058187253504. 2021-04-08 20:51:05 notice : MaxScale started with 16 worker threads, each with a stack size of 8388608 bytes. 2021-04-08 20:51:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140059283869440. 2021-04-08 20:51:05 notice : Loading /etc/maxscale.conf. 2021-04-08 20:51:05 notice : /etc/maxscale.conf.d does not exist, not reading. 2021-04-08 20:51:05 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2021-04-08 20:51:05 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2021-04-08 20:51:05 notice : Loaded module xpandmon: V1.0.0 from /usr/lib64/maxscale/libxpandmon.so 2021-04-08 20:51:05 notice : [xpandmon] sqlite3 database /var/lib/maxscale/Xpand/xpand_nodes-v1.db open/created and initialized. 2021-04-08 20:51:05 notice : (Splitter-Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib64/maxscale/libmariadbauth.so 2021-04-08 20:51:05 warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption. 2021-04-08 20:51:05 notice : Started REST API on [127.0.0.1]:8989 2021-04-08 20:51:05 info : Loaded server states from journal file: /var/lib/maxscale/Xpand/monitor.dat 2021-04-08 20:51:05 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node karma114:3306. 2021-04-08 20:51:05 notice : Created server '@@Xpand:node-15' at 10.2.15.59:3306 2021-04-08 20:51:05 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-08 20:51:05 notice : Created server '@@Xpand:node-31' at 10.2.13.175:3306 2021-04-08 20:51:05 info : [xpandmon] Updated Xpand node in bookkeeping: 31, '10.2.13.175', 3306, 3581. 2021-04-08 20:51:05 notice : Created server '@@Xpand:node-3' at 10.2.12.248:3306 2021-04-08 20:51:05 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-08 20:51:05 notice : Starting a total of 1 services... 2021-04-08 20:51:05 notice : (Splitter-Listener) Listening for connections at [0.0.0.0]:3306 2021-04-08 20:51:05 warning: Service 'Splitter-Service' has a listener but no servers 2021-04-08 20:51:05 notice : Service 'Splitter-Service' started (1/1) 2021-04-08 20:51:06 notice : '@@Xpand:node-15' sent version string '5.0.45-Xpand-5.3.15'. Detected type: 'Xpand', version: 5.0.45. 2021-04-08 20:51:06 warning: Service user 'maxscale' of service 'Splitter-Service' does not have 'SHOW DATABASES' or a similar global privilege on '@@Xpand:node-15'. This may cause authentication errors on clients logging in to a specific database. 2021-04-08 20:51:06 notice : Read 9 user@host entries from '@@Xpand:node-15' for service 'Splitter-Service'. 2021-04-08 20:51:41 info : (1) Found matching user 'maxscale'@'%' for client 'maxscale'@'10.2.16.29' with sufficient privileges. 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) Target connection counts: 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-15 Master, Running 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-31 Master, Running 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-3 Master, Running 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) Connected to '@@Xpand:node-15' 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) Selected Master: @@Xpand:node-15 2021-04-08 20:51:41 info : (1) Started Splitter-Service client session [1] for 'maxscale' from 10.2.16.29 2021-04-08 20:51:41 info : (1) Connected to '@@Xpand:node-15' with thread id 110607 2021-04-08 20:51:41 info : (1) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 37, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@version_comment limit 1 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:51:41 info : (1) Authentication to '@@Xpand:node-15' succeeded. 2021-04-08 20:51:41 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:42 info : (1) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 20, type: QUERY_TYPE_WRITE, stmt: truncate db1.t1 2021-04-08 20:51:42 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:43 info : (1) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 22, type: QUERY_TYPE_READ, stmt: SELECT DATABASE() 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:43 info : (1) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x02) COM_INIT_DB, plen: 8, type: QUERY_TYPE_SESSION_WRITE, stmt: 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-08 20:51:43 info : (1) (Splitter-Service) Execute sescmd #1 on '@@Xpand:node-15': [COM_INIT_DB] 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Will return response from '@@Xpand:node-15' to the client 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:43 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 33, type: QUERY_TYPE_SESSION_WRITE|QUERY_TYPE_GSYSVAR_WRITE|QUERY_TYPE_BEGIN_TRX|QUERY_TYPE_DISABLE_AUTOCOMMIT, stmt: set session autocommit=false 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-08 20:51:43 info : (1) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-15': [COM_QUERY] set session autocommit=false 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Will return response from '@@Xpand:node-15' to the client 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:43 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-15' 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: begin 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:43 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (1) 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (1) 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:43 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (2) 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (2) 2021-04-08 20:51:43 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:51:57 info : [xpandmon] Deleted Xpand node 3 from bookkeeping. 2021-04-08 20:51:57 info : [xpandmon] Deleted Xpand node 15 from bookkeeping. 2021-04-08 20:51:57 info : [xpandmon] Deleted Xpand node 31 from bookkeeping. 2021-04-08 20:51:59 error : [xpandmon] Xpand: Could not execute 'SELECT status FROM system.membership WHERE nid = gtmnid()' on karma114 via TCP/IP: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:51:59 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:51:59 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:51:59 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: Can't connect to MySQL server on '10.2.13.175' (115) 2021-04-08 20:51:59 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:51:59 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:51:59 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:01 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (3) 2021-04-08 20:52:01 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 1 2021-04-08 20:52:01 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:01 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:01 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:01 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:01 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:01 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:01 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:01 info : [xpandmon] Updated Xpand node in bookkeeping: 31, '10.2.13.175', 3306, 3581. 2021-04-08 20:52:02 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:02 info : [readwritesplit] Replacing old master '@@Xpand:node-15' with new master '@@Xpand:node-31' 2021-04-08 20:52:02 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:02 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:02 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:02 info : (1) Connected to '@@Xpand:node-31' with thread id 3073 2021-04-08 20:52:02 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:02 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:02 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 2 2021-04-08 20:52:02 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:03 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:03 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:03 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:03 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:03 info : (1) Connected to '@@Xpand:node-31' with thread id 4097 2021-04-08 20:52:03 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:03 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:03 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 3 2021-04-08 20:52:03 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:03 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:03 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:03 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:03 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:03 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:03 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:04 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:04 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:04 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:04 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:04 info : (1) Connected to '@@Xpand:node-31' with thread id 7169 2021-04-08 20:52:04 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:04 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:04 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 4 2021-04-08 20:52:04 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:05 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:05 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:05 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:05 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:05 info : (1) Connected to '@@Xpand:node-31' with thread id 8193 2021-04-08 20:52:05 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:05 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:05 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 5 2021-04-08 20:52:05 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:05 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:05 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:05 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:05 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:05 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:05 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:06 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:06 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:06 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:06 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:06 info : (1) Connected to '@@Xpand:node-31' with thread id 11265 2021-04-08 20:52:06 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:06 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:06 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 6 2021-04-08 20:52:06 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:07 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:07 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:07 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:07 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:07 info : (1) Connected to '@@Xpand:node-31' with thread id 12289 2021-04-08 20:52:07 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:07 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:07 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 7 2021-04-08 20:52:07 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:07 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:07 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:07 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:07 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:07 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:07 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:08 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:08 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:08 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:08 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:08 info : (1) Connected to '@@Xpand:node-31' with thread id 15361 2021-04-08 20:52:08 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:08 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:08 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 8 2021-04-08 20:52:08 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:09 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:09 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:09 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:09 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:09 info : (1) Connected to '@@Xpand:node-31' with thread id 16385 2021-04-08 20:52:09 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:09 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:09 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 9 2021-04-08 20:52:09 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:09 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:09 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:09 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:09 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:09 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:09 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:10 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:10 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:10 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:10 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:10 info : (1) Connected to '@@Xpand:node-31' with thread id 19457 2021-04-08 20:52:10 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:10 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:10 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 10 2021-04-08 20:52:10 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:11 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:11 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:52:11 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:52:11 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:52:11 info : (1) Connected to '@@Xpand:node-31' with thread id 20481 2021-04-08 20:52:11 error : (1) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:52:11 info : (1) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:52:11 info : (1) [readwritesplit] (Splitter-Service) Starting transaction replay 11 2021-04-08 20:52:11 info : (1) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:52:11 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node 10.2.12.248:3306. 2021-04-08 20:52:11 info : [xpandmon] Deleted Xpand node 31 from bookkeeping. 2021-04-08 20:52:11 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-08 20:52:11 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-08 20:52:12 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:12 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-3' 2021-04-08 20:52:12 info : Execute sescmd #1 on '@@Xpand:node-3': [COM_INIT_DB] 2021-04-08 20:52:12 info : [readwritesplit] Connected to '@@Xpand:node-3' 2021-04-08 20:52:12 info : [readwritesplit] Queuing query until '@@Xpand:node-3' completes session command 2021-04-08 20:52:12 info : (1) Connected to '@@Xpand:node-3' with thread id 73731 2021-04-08 20:52:12 info : (1) Authentication to '@@Xpand:node-3' succeeded. 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:12 info : (1) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-3': [COM_QUERY] set session autocommit=false 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) 1 session commands left on '@@Xpand:node-3' 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-08 20:52:12 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-3' 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: begin 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Replaying: insert into t1 values (1) 2021-04-08 20:52:12 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (1) 2021-04-08 20:52:12 info : [readwritesplit] Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (1) 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Replaying: insert into t1 values (2) 2021-04-08 20:52:12 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (2) 2021-04-08 20:52:12 info : [readwritesplit] Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (2) 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Checksums match, replay successful. 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Resuming execution: insert into t1 values (3) 2021-04-08 20:52:12 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (3) 2021-04-08 20:52:12 info : [readwritesplit] Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (3) 2021-04-08 20:52:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:20 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (4) 2021-04-08 20:52:20 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:20 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (4) 2021-04-08 20:52:20 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:24 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 11, type: QUERY_TYPE_COMMIT, stmt: commit 2021-04-08 20:52:24 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:24 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: commit 2021-04-08 20:52:24 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:24 info : (1) [readwritesplit] (Splitter-Service) Transaction complete 2021-04-08 20:52:28 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 20:52:28 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:52:28 info : (1) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-3' 2021-04-08 20:52:28 info : (1) [readwritesplit] (Splitter-Service) Adding to trx: select * from t1 2021-04-08 20:52:28 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:52:48 info : [xpandmon] Deleted Xpand node 3 from bookkeeping. 2021-04-08 20:52:48 info : [xpandmon] Deleted Xpand node 15 from bookkeeping. 2021-04-08 20:52:50 error : [xpandmon] Xpand: Could not execute 'SELECT status FROM system.membership WHERE nid = gtmnid()' on 10.2.12.248 via TCP/IP: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:50 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:50 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:50 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:50 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:50 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:52 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:52 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:52 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:52 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:52 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:54 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:54 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:54 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:54 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:54 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:56 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:56 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:56 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:56 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:56 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:52:58 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:58 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:58 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:52:58 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:52:58 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:53:00 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:00 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:00 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:00 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:53:00 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:53:02 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node 10.2.12.248:3306. 2021-04-08 20:53:02 notice : Created server '@@Xpand:node-32' at 10.2.13.175:3306 2021-04-08 20:53:02 info : [xpandmon] Updated Xpand node in bookkeeping: 32, '10.2.13.175', 3306, 3581. 2021-04-08 20:53:02 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-08 20:53:02 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-08 20:53:49 info : [xpandmon] Deleted Xpand node 3 from bookkeeping. 2021-04-08 20:53:49 info : [xpandmon] Deleted Xpand node 15 from bookkeeping. 2021-04-08 20:53:49 info : [xpandmon] Deleted Xpand node 32 from bookkeeping. 2021-04-08 20:53:51 error : [xpandmon] Xpand: Could not execute 'SELECT status FROM system.membership WHERE nid = gtmnid()' on 10.2.12.248 via TCP/IP: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:51 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:51 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:51 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:51 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:51 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:53:51 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:53:53 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:53 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:53 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:53 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:53 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:53:53 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:53:55 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:55 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:55 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:55 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:55 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:53:55 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:53:57 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:57 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:57 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:57 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:57 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:53:57 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:53:59 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:59 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:59 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:59 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:53:59 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:53:59 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:54:01 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:01 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:01 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:01 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:01 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:54:01 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:54:03 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:03 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:03 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:03 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:03 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:54:03 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:54:05 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:05 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:05 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:05 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:05 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:54:05 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:54:07 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:07 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:07 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:07 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:07 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:54:07 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:54:09 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:09 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:09 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:09 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:54:09 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:54:09 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:54:11 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node 10.2.12.248:3306. 2021-04-08 20:54:11 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-08 20:54:11 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-08 20:54:11 info : [xpandmon] Updated Xpand node in bookkeeping: 32, '10.2.13.175', 3306, 3581. 2021-04-08 20:54:27 info : (1) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x01) COM_QUIT, plen: 5, type: QUERY_TYPE_SESSION_WRITE, stmt: 2021-04-08 20:54:27 info : (1) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-08 20:54:27 info : (1) (Splitter-Service) Execute sescmd #3 on '@@Xpand:node-3': [COM_QUIT] 2021-04-08 20:54:27 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 2021-04-08 20:54:27 info : Stopped Splitter-Service client session [1] 2021-04-08 20:54:28 info : (2) Found matching user 'maxscale'@'%' for client 'maxscale'@'10.2.16.29' with sufficient privileges. 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) Target connection counts: 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-15 Master, Running 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-31 Master, Running 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-3 Master, Running 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-32 Master, Running 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) Connected to '@@Xpand:node-15' 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) Selected Master: @@Xpand:node-15 2021-04-08 20:54:28 info : (2) Started Splitter-Service client session [2] for 'maxscale' from 10.2.16.29 2021-04-08 20:54:28 info : (2) Connected to '@@Xpand:node-15' with thread id 223247 2021-04-08 20:54:28 info : (2) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 37, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@version_comment limit 1 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:54:28 info : (2) Authentication to '@@Xpand:node-15' succeeded. 2021-04-08 20:54:28 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 20, type: QUERY_TYPE_WRITE, stmt: truncate db1.t1 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 22, type: QUERY_TYPE_READ, stmt: SELECT DATABASE() 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x02) COM_INIT_DB, plen: 8, type: QUERY_TYPE_SESSION_WRITE, stmt: 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-08 20:55:17 info : (2) (Splitter-Service) Execute sescmd #1 on '@@Xpand:node-15': [COM_INIT_DB] 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Will return response from '@@Xpand:node-15' to the client 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 33, type: QUERY_TYPE_SESSION_WRITE|QUERY_TYPE_GSYSVAR_WRITE|QUERY_TYPE_BEGIN_TRX|QUERY_TYPE_DISABLE_AUTOCOMMIT, stmt: set session autocommit=false 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-08 20:55:17 info : (2) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-15': [COM_QUERY] set session autocommit=false 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Will return response from '@@Xpand:node-15' to the client 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-15' 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: begin 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (1) 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (1) 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:17 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (2) 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (2) 2021-04-08 20:55:17 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 20:55:38 info : [xpandmon] Deleted Xpand node 3 from bookkeeping. 2021-04-08 20:55:38 info : [xpandmon] Deleted Xpand node 15 from bookkeeping. 2021-04-08 20:55:38 info : [xpandmon] Deleted Xpand node 32 from bookkeeping. 2021-04-08 20:55:40 error : [xpandmon] Xpand: Could not execute 'SELECT status FROM system.membership WHERE nid = gtmnid()' on 10.2.12.248 via TCP/IP: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:40 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:40 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:40 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: Can't connect to MySQL server on '10.2.13.175' (115) 2021-04-08 20:55:40 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:40 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:40 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:41 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (3) 2021-04-08 20:55:41 info : (2) [readwritesplit] (Splitter-Service) Starting transaction migration to '@@Xpand:node-31' 2021-04-08 20:55:41 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 1 2021-04-08 20:55:41 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:42 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:42 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:42 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:42 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:42 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:42 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:42 info : [xpandmon] Updated Xpand node in bookkeeping: 32, '10.2.13.175', 3306, 3581. 2021-04-08 20:55:42 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:42 info : [readwritesplit] Replacing old master '@@Xpand:node-15' with new master '@@Xpand:node-31' 2021-04-08 20:55:42 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:42 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:42 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:42 info : (2) Connected to '@@Xpand:node-31' with thread id 2049 2021-04-08 20:55:42 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:42 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:42 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 2 2021-04-08 20:55:42 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:43 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:43 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:43 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:43 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:43 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:43 info : (2) Connected to '@@Xpand:node-32' with thread id 4097 2021-04-08 20:55:43 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:43 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:43 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 3 2021-04-08 20:55:43 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:44 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:44 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:44 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:44 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:44 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:44 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:44 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:44 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-31' 2021-04-08 20:55:44 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:44 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:44 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:44 info : (2) Connected to '@@Xpand:node-31' with thread id 6145 2021-04-08 20:55:44 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:44 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:44 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 4 2021-04-08 20:55:44 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:45 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:45 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:45 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:45 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:45 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:45 info : (2) Connected to '@@Xpand:node-32' with thread id 8193 2021-04-08 20:55:45 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:45 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:45 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 5 2021-04-08 20:55:45 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:46 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:46 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:46 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:46 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:46 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:46 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:46 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:46 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-31' 2021-04-08 20:55:46 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:46 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:46 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:46 info : (2) Connected to '@@Xpand:node-31' with thread id 10241 2021-04-08 20:55:46 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:46 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:46 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 6 2021-04-08 20:55:46 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:47 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:47 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:47 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:47 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:47 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:47 info : (2) Connected to '@@Xpand:node-32' with thread id 12289 2021-04-08 20:55:47 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:47 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:47 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 7 2021-04-08 20:55:47 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:48 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:48 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:48 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:48 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:48 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:48 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:48 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:48 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-31' 2021-04-08 20:55:48 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:48 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:48 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:48 info : (2) Connected to '@@Xpand:node-31' with thread id 14337 2021-04-08 20:55:48 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:48 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:48 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 8 2021-04-08 20:55:48 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:49 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:49 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:49 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:49 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:49 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:49 info : (2) Connected to '@@Xpand:node-32' with thread id 16385 2021-04-08 20:55:49 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:49 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:49 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 9 2021-04-08 20:55:49 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:50 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:50 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:50 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:50 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:50 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:50 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:50 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:50 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-31' 2021-04-08 20:55:50 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:50 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:50 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:50 info : (2) Connected to '@@Xpand:node-31' with thread id 18433 2021-04-08 20:55:50 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 10 2021-04-08 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:51 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:51 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:51 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:51 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:51 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:51 info : (2) Connected to '@@Xpand:node-32' with thread id 20481 2021-04-08 20:55:51 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:51 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:51 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 11 2021-04-08 20:55:51 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:52 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:52 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:52 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:52 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:52 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:52 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:52 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:52 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-31' 2021-04-08 20:55:52 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:52 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:52 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:52 info : (2) Connected to '@@Xpand:node-31' with thread id 22529 2021-04-08 20:55:52 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:52 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:52 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 12 2021-04-08 20:55:52 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:53 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:53 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:53 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:53 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:53 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:53 info : (2) Connected to '@@Xpand:node-32' with thread id 24577 2021-04-08 20:55:53 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:53 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:53 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 13 2021-04-08 20:55:53 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:54 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:54 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:54 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:54 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-08 20:55:54 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-08 20:55:54 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-08 20:55:54 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:54 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-31' 2021-04-08 20:55:54 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 20:55:54 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 20:55:54 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 20:55:54 info : (2) Connected to '@@Xpand:node-31' with thread id 26625 2021-04-08 20:55:54 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:54 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 20:55:54 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 14 2021-04-08 20:55:54 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:55 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:55 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 20:55:55 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 20:55:55 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 20:55:55 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 20:55:55 info : (2) Connected to '@@Xpand:node-32' with thread id 28673 2021-04-08 20:55:55 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 20:55:55 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 20:55:55 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 15 2021-04-08 20:55:55 info : (2) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-08 20:55:56 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node 10.2.12.248:3306. 2021-04-08 20:55:56 info : [xpandmon] Deleted Xpand node 32 from bookkeeping. 2021-04-08 20:55:56 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-08 20:55:56 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-08 20:55:56 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:56 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-3' 2021-04-08 20:55:56 info : Execute sescmd #1 on '@@Xpand:node-3': [COM_INIT_DB] 2021-04-08 20:55:56 info : [readwritesplit] Connected to '@@Xpand:node-3' 2021-04-08 20:55:56 info : [readwritesplit] Queuing query until '@@Xpand:node-3' completes session command 2021-04-08 20:55:56 info : (2) Connected to '@@Xpand:node-3' with thread id 139267 2021-04-08 20:55:56 info : (2) Authentication to '@@Xpand:node-3' succeeded. 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:55:56 info : (2) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-3': [COM_QUERY] set session autocommit=false 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) 1 session commands left on '@@Xpand:node-3' 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-08 20:55:56 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-3' 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: begin 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Replaying: insert into t1 values (1) 2021-04-08 20:55:56 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (1) 2021-04-08 20:55:56 info : [readwritesplit] Route query to master: @@Xpand:node-3 < 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (1) 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Replaying: insert into t1 values (2) 2021-04-08 20:55:56 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (2) 2021-04-08 20:55:56 info : [readwritesplit] Route query to master: @@Xpand:node-3 < 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (2) 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Checksums match, replay successful. 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Resuming execution: insert into t1 values (3) 2021-04-08 20:55:56 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (3) 2021-04-08 20:55:56 info : [readwritesplit] Route query to master: @@Xpand:node-3 < 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (3) 2021-04-08 20:55:56 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:56:10 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (4) 2021-04-08 20:56:10 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:56:10 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (4) 2021-04-08 20:56:10 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:56:14 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 11, type: QUERY_TYPE_COMMIT, stmt: commit 2021-04-08 20:56:14 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:56:14 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: commit 2021-04-08 20:56:14 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 20:56:14 info : (2) [readwritesplit] (Splitter-Service) Transaction complete 2021-04-08 20:56:21 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 20:56:21 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 20:56:21 info : (2) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-3' 2021-04-08 20:56:21 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: select * from t1 2021-04-08 20:56:21 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 21:01:23 info : Pinging '@@Xpand:node-3', idle for 301 seconds 2021-04-08 21:01:23 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:06:24 info : Pinging '@@Xpand:node-3', idle for 301 seconds 2021-04-08 21:06:24 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:11:26 info : Pinging '@@Xpand:node-3', idle for 301 seconds 2021-04-08 21:11:26 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:16:25 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-3' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-3) 2021-04-08 21:16:25 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 1 2021-04-08 21:16:25 info : (2) [readwritesplit] (Splitter-Service) Replaying: select * from t1 2021-04-08 21:16:26 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 21:16:26 info : [readwritesplit] Replacing old master '@@Xpand:node-3' with new master '@@Xpand:node-15' 2021-04-08 21:16:26 info : Execute sescmd #1 on '@@Xpand:node-15': [COM_INIT_DB] 2021-04-08 21:16:26 info : [readwritesplit] Connected to '@@Xpand:node-15' 2021-04-08 21:16:26 info : [readwritesplit] Queuing query until '@@Xpand:node-15' completes session command 2021-04-08 21:16:26 info : (2) Connected to '@@Xpand:node-15' with thread id 261135 2021-04-08 21:16:26 info : (2) Authentication to '@@Xpand:node-15' succeeded. 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 21:16:26 info : (2) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-15': [COM_QUERY] set session autocommit=false 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) 1 session commands left on '@@Xpand:node-15' 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-08 21:16:26 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-15' 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: select * from t1 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-08 21:16:26 info : (2) [readwritesplit] (Splitter-Service) Checksums match, replay successful. 2021-04-08 21:21:28 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-08 21:21:28 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:26:30 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-08 21:26:30 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:31:31 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-08 21:31:31 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:36:33 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-08 21:36:33 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-08 21:36:55 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-15' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-15) 2021-04-08 21:36:55 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 1 2021-04-08 21:36:55 info : (2) [readwritesplit] (Splitter-Service) Replaying: select * from t1 2021-04-08 21:36:56 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 21:36:56 info : [readwritesplit] Replacing old master '@@Xpand:node-15' with new master '@@Xpand:node-31' 2021-04-08 21:36:56 info : Execute sescmd #1 on '@@Xpand:node-31': [COM_INIT_DB] 2021-04-08 21:36:56 info : [readwritesplit] Connected to '@@Xpand:node-31' 2021-04-08 21:36:56 info : [readwritesplit] Queuing query until '@@Xpand:node-31' completes session command 2021-04-08 21:36:56 info : (2) Connected to '@@Xpand:node-31' with thread id 113665 2021-04-08 21:36:56 error : (2) Invalid authentication message from backend '@@Xpand:node-31'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 21:36:56 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-31' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-31) 2021-04-08 21:36:56 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 2 2021-04-08 21:36:56 info : (2) [readwritesplit] (Splitter-Service) Replaying: select * from t1 2021-04-08 21:36:57 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 21:36:57 info : [readwritesplit] Replacing old master '@@Xpand:node-31' with new master '@@Xpand:node-32' 2021-04-08 21:36:57 info : Execute sescmd #1 on '@@Xpand:node-32': [COM_INIT_DB] 2021-04-08 21:36:57 info : [readwritesplit] Connected to '@@Xpand:node-32' 2021-04-08 21:36:57 info : [readwritesplit] Queuing query until '@@Xpand:node-32' completes session command 2021-04-08 21:36:57 info : (2) Connected to '@@Xpand:node-32' with thread id 114689 2021-04-08 21:36:57 error : (2) Invalid authentication message from backend '@@Xpand:node-32'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-08 21:36:57 info : (2) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-32' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-32) 2021-04-08 21:36:57 info : (2) [readwritesplit] (Splitter-Service) Starting transaction replay 3 2021-04-08 21:36:57 info : (2) [readwritesplit] (Splitter-Service) Replaying: select * from t1 2021-04-08 21:36:58 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 21:36:58 info : [readwritesplit] Replacing old master '@@Xpand:node-32' with new master '@@Xpand:node-3' 2021-04-08 21:36:58 info : Execute sescmd #1 on '@@Xpand:node-3': [COM_INIT_DB] 2021-04-08 21:36:58 info : [readwritesplit] Connected to '@@Xpand:node-3' 2021-04-08 21:36:58 info : [readwritesplit] Queuing query until '@@Xpand:node-3' completes session command 2021-04-08 21:36:58 info : (2) Connected to '@@Xpand:node-3' with thread id 143363 2021-04-08 21:36:58 info : (2) Authentication to '@@Xpand:node-3' succeeded. 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 21:36:58 info : (2) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-3': [COM_QUERY] set session autocommit=false 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) 1 session commands left on '@@Xpand:node-3' 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-08 21:36:58 info : (2) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-3' 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Adding to trx: select * from t1 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-08 21:36:58 info : (2) [readwritesplit] (Splitter-Service) Checksum mismatch, transaction replay failed. Closing connection. 2021-04-08 21:36:58 info : Stopped Splitter-Service client session [2]