MariaDB MaxScale /data/clustrix/log/maxscale.log Tue Apr 13 20:55:05 2021 ----------------------------------------------------------------------------- 2021-04-13 20:55:05 notice : The collection of SQLite memory allocation statistics turned off. 2021-04-13 20:55:05 notice : Threading mode of SQLite set to Multi-thread. 2021-04-13 20:55:05 notice : Worker message queue size: 1MiB 2021-04-13 20:55:05 notice : The logging of informational messages has been enabled. 2021-04-13 20:55:05 notice : Using up to 7.05GiB of memory for query classifier cache 2021-04-13 20:55:05 notice : syslog logging is enabled. 2021-04-13 20:55:05 notice : maxlog logging is enabled. 2021-04-13 20:55:05 notice : Running OS: Linux@3.10.0-1127.19.1.el7.x86_64, #1 SMP Tue Aug 25 17:23:54 UTC 2020, x86_64 with 16 processor cores. 2021-04-13 20:55:05 notice : Total usable main memory: 47GiB. 2021-04-13 20:55:05 notice : MariaDB MaxScale 2.5.11 started (Commit: c177d1a999eb354698a16194ba905096ad985655) 2021-04-13 20:55:05 notice : MaxScale is running in process 17476 2021-04-13 20:55:05 notice : Configuration file: /etc/maxscale.conf 2021-04-13 20:55:05 notice : Log directory: /data/clustrix/log 2021-04-13 20:55:05 notice : Data directory: /var/lib/maxscale 2021-04-13 20:55:05 notice : Module directory: /usr/lib64/maxscale 2021-04-13 20:55:05 notice : Service cache: /var/cache/maxscale 2021-04-13 20:55:05 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so 2021-04-13 20:55:05 info : qc_sqlite loaded. 2021-04-13 20:55:05 notice : Query classification results are cached and reused. Memory used per thread: 451.2MiB 2021-04-13 20:55:05 notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : Epoll instance for listening sockets added to worker epoll instance. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428529444608. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428452034304. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428443641600. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428435248896. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428426856192. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428418463488. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428410070784. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428401678080. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427915163392. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427906770688. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427898377984. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427889985280. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427881592576. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427873199872. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427864807168. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140427378292480. 2021-04-13 20:55:05 notice : MaxScale started with 16 worker threads, each with a stack size of 8388608 bytes. 2021-04-13 20:55:05 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140428678500096. 2021-04-13 20:55:05 notice : Loading /etc/maxscale.conf. 2021-04-13 20:55:05 notice : /etc/maxscale.conf.d does not exist, not reading. 2021-04-13 20:55:05 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so 2021-04-13 20:55:05 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so 2021-04-13 20:55:05 notice : Loaded module xpandmon: V1.0.0 from /usr/lib64/maxscale/libxpandmon.so 2021-04-13 20:55:05 notice : [xpandmon] sqlite3 database /var/lib/maxscale/Xpand/xpand_nodes-v1.db open/created and initialized. 2021-04-13 20:55:05 notice : (Splitter-Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib64/maxscale/libmariadbauth.so 2021-04-13 20:55:05 warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption. 2021-04-13 20:55:05 notice : Started REST API on [127.0.0.1]:8989 2021-04-13 20:55:05 info : Loaded server states from journal file: /var/lib/maxscale/Xpand/monitor.dat 2021-04-13 20:55:05 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node karma114:3306. 2021-04-13 20:55:05 notice : Created server '@@Xpand:node-15' at 10.2.15.59:3306 2021-04-13 20:55:05 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-13 20:55:05 notice : Created server '@@Xpand:node-3' at 10.2.12.248:3306 2021-04-13 20:55:05 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-13 20:55:05 notice : Created server '@@Xpand:node-35' at 10.2.13.175:3306 2021-04-13 20:55:05 info : [xpandmon] Updated Xpand node in bookkeeping: 35, '10.2.13.175', 3306, 3581. 2021-04-13 20:55:05 notice : Starting a total of 1 services... 2021-04-13 20:55:05 notice : (Splitter-Listener) Listening for connections at [0.0.0.0]:3306 2021-04-13 20:55:05 warning: Service 'Splitter-Service' has a listener but no servers 2021-04-13 20:55:05 notice : Service 'Splitter-Service' started (1/1) 2021-04-13 20:55:06 notice : '@@Xpand:node-15' sent version string '5.0.45-Xpand-5.3.15'. Detected type: 'Xpand', version: 5.0.45. 2021-04-13 20:55:06 warning: Service user 'maxscale' of service 'Splitter-Service' does not have 'SHOW DATABASES' or a similar global privilege on '@@Xpand:node-15'. This may cause authentication errors on clients logging in to a specific database. 2021-04-13 20:55:06 notice : Read 9 user@host entries from '@@Xpand:node-15' for service 'Splitter-Service'. 2021-04-13 20:55:12 info : (1) Found matching user 'maxscale'@'%' for client 'maxscale'@'10.2.16.29' with sufficient privileges. 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) Target connection counts: 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-15 Master, Running 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-3 Master, Running 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-35 Master, Running 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) Connected to '@@Xpand:node-15' 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) Selected Master: @@Xpand:node-15 2021-04-13 20:55:12 info : (1) Started Splitter-Service client session [1] for 'maxscale' from 10.2.16.29 2021-04-13 20:55:12 info : (1) Connected to '@@Xpand:node-15' with thread id 539663 2021-04-13 20:55:12 info : (1) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 37, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@version_comment limit 1 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:55:12 info : (1) Authentication to '@@Xpand:node-15' succeeded. 2021-04-13 20:55:12 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:55:22 info : (1) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 22, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@hostname 2021-04-13 20:55:22 info : (1) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:55:22 info : (1) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:55:31 info : Accept authentication from 'admin', using password. Request: /v1/servers 2021-04-13 20:55:50 info : (2) Found matching user 'maxscale'@'%' for client 'maxscale'@'10.2.16.29' with sufficient privileges. 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Target connection counts: 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 1 in @@Xpand:node-15 Master, Running 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-3 Master, Running 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-35 Master, Running 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Connected to '@@Xpand:node-3' 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Selected Master: @@Xpand:node-3 2021-04-13 20:55:50 info : (2) Started Splitter-Service client session [2] for 'maxscale' from 10.2.16.29 2021-04-13 20:55:50 info : (2) Connected to '@@Xpand:node-3' with thread id 302083 2021-04-13 20:55:50 info : (2) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 37, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@version_comment limit 1 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-3 < 2021-04-13 20:55:50 info : (2) Authentication to '@@Xpand:node-3' succeeded. 2021-04-13 20:55:50 info : (2) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-3 2021-04-13 20:55:58 info : (3) Found matching user 'maxscale'@'%' for client 'maxscale'@'10.2.16.29' with sufficient privileges. 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) Target connection counts: 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) MaxScale connections : 1 in @@Xpand:node-15 Master, Running 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) MaxScale connections : 1 in @@Xpand:node-3 Master, Running 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) MaxScale connections : 0 in @@Xpand:node-35 Master, Running 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) Connected to '@@Xpand:node-35' 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) Selected Master: @@Xpand:node-35 2021-04-13 20:55:58 info : (3) Started Splitter-Service client session [3] for 'maxscale' from 10.2.16.29 2021-04-13 20:55:58 info : (3) Connected to '@@Xpand:node-35' with thread id 4131 2021-04-13 20:55:58 info : (3) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 37, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@version_comment limit 1 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:55:58 info : (3) Authentication to '@@Xpand:node-35' succeeded. 2021-04-13 20:55:58 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:08 info : (3) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 22, type: QUERY_TYPE_READ|QUERY_TYPE_SYSVAR_READ, stmt: select @@hostname 2021-04-13 20:56:08 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:56:08 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:40 info : (3) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 20, type: QUERY_TYPE_WRITE, stmt: truncate db1.t1 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:40 info : (3) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 22, type: QUERY_TYPE_READ, stmt: SELECT DATABASE() 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:40 info : (3) (Splitter-Service) > Autocommit: [enabled], trx is [not open], cmd: (0x02) COM_INIT_DB, plen: 8, type: QUERY_TYPE_SESSION_WRITE, stmt: 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-13 20:56:40 info : (3) (Splitter-Service) Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Will return response from '@@Xpand:node-35' to the client 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:40 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [not open], cmd: (0x03) COM_QUERY, plen: 33, type: QUERY_TYPE_SESSION_WRITE|QUERY_TYPE_GSYSVAR_WRITE|QUERY_TYPE_BEGIN_TRX|QUERY_TYPE_DISABLE_AUTOCOMMIT, stmt: set session autocommit=false 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Session write, routing to all servers. 2021-04-13 20:56:40 info : (3) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-35': [COM_QUERY] set session autocommit=false 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Will return response from '@@Xpand:node-35' to the client 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:40 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-35' 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: begin 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:40 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (1) 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (1) 2021-04-13 20:56:40 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:56:41 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (2) 2021-04-13 20:56:41 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-35 < 2021-04-13 20:56:41 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (2) 2021-04-13 20:56:41 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-35 2021-04-13 20:57:00 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:00 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 1 2021-04-13 20:57:00 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:01 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:01 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:01 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:01 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:01 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: network error (@@Xpand:node-35: 111, Connection refused) 2021-04-13 20:57:01 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 2 2021-04-13 20:57:01 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:02 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:02 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:02 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:02 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:02 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: network error (@@Xpand:node-35: 111, Connection refused) 2021-04-13 20:57:02 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 3 2021-04-13 20:57:02 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:02 info : [xpandmon] Deleted Xpand node 3 from bookkeeping. 2021-04-13 20:57:02 info : [xpandmon] Deleted Xpand node 15 from bookkeeping. 2021-04-13 20:57:02 info : [xpandmon] Deleted Xpand node 35 from bookkeeping. 2021-04-13 20:57:03 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:03 info : [readwritesplit] Delaying routing: begin 2021-04-13 20:57:04 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:04 info : [readwritesplit] Delaying routing: begin 2021-04-13 20:57:04 error : [xpandmon] Xpand: Could not execute 'SELECT status FROM system.membership WHERE nid = gtmnid()' on karma114 via TCP/IP: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:04 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:04 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:04 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: Can't connect to MySQL server on '10.2.13.175' (115) 2021-04-13 20:57:04 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:04 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:04 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:05 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:05 info : [readwritesplit] Delaying routing: begin 2021-04-13 20:57:05 info : (3) [readwritesplit] (Splitter-Service) New COM_QUERY received while transaction replay is active: insert into t1 values (3) 2021-04-13 20:57:06 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:06 info : [readwritesplit] Delaying routing: begin 2021-04-13 20:57:06 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:06 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:06 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:06 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:06 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:06 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:06 info : [xpandmon] Updated Xpand node in bookkeeping: 35, '10.2.13.175', 3306, 3581. 2021-04-13 20:57:07 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:07 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:07 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:07 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:07 info : (3) Connected to '@@Xpand:node-35' with thread id 2049 2021-04-13 20:57:07 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:07 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:07 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 4 2021-04-13 20:57:07 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:08 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:08 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:08 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:08 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:08 info : (3) Connected to '@@Xpand:node-35' with thread id 4097 2021-04-13 20:57:08 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:08 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:08 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 5 2021-04-13 20:57:08 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:08 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:08 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:08 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:08 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:08 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:08 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:09 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:09 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:09 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:09 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:09 info : (3) Connected to '@@Xpand:node-35' with thread id 6145 2021-04-13 20:57:09 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:09 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:09 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 6 2021-04-13 20:57:09 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:10 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:10 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:10 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:10 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:10 info : (3) Connected to '@@Xpand:node-35' with thread id 8193 2021-04-13 20:57:10 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:10 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:10 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 7 2021-04-13 20:57:10 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:10 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:10 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:10 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:10 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:10 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:10 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:11 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:11 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:11 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:11 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:11 info : (3) Connected to '@@Xpand:node-35' with thread id 12289 2021-04-13 20:57:11 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:11 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:11 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 8 2021-04-13 20:57:11 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:12 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:12 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:12 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:12 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:12 info : (3) Connected to '@@Xpand:node-35' with thread id 14337 2021-04-13 20:57:12 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:12 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:12 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 9 2021-04-13 20:57:12 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:12 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:12 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:12 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:12 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:12 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:12 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:13 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:13 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:13 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:13 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:13 info : (3) Connected to '@@Xpand:node-35' with thread id 16385 2021-04-13 20:57:13 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:13 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:13 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 10 2021-04-13 20:57:13 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:14 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:14 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:14 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:14 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:14 info : (3) Connected to '@@Xpand:node-35' with thread id 18433 2021-04-13 20:57:14 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:14 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:14 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 11 2021-04-13 20:57:14 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:14 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:14 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:14 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:14 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:14 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:14 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:15 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:15 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:15 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:15 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:15 info : (3) Connected to '@@Xpand:node-35' with thread id 20481 2021-04-13 20:57:15 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:15 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:15 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 12 2021-04-13 20:57:15 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:16 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:16 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:16 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:16 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:16 info : (3) Connected to '@@Xpand:node-35' with thread id 22529 2021-04-13 20:57:16 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:16 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:16 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 13 2021-04-13 20:57:16 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:16 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:16 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:16 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:16 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:16 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:16 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:17 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:17 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:17 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:17 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:17 info : (3) Connected to '@@Xpand:node-35' with thread id 24577 2021-04-13 20:57:17 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:17 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:17 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 14 2021-04-13 20:57:17 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:18 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:18 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:18 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:18 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:18 info : (3) Connected to '@@Xpand:node-35' with thread id 26625 2021-04-13 20:57:18 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:18 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:18 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 15 2021-04-13 20:57:18 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:18 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:18 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:18 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:18 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:18 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:18 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:19 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:19 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:19 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:19 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:19 info : (3) Connected to '@@Xpand:node-35' with thread id 28673 2021-04-13 20:57:19 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:19 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:19 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 16 2021-04-13 20:57:19 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:20 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:20 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:20 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:20 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:20 info : (3) Connected to '@@Xpand:node-35' with thread id 30721 2021-04-13 20:57:20 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:20 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:20 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 17 2021-04-13 20:57:20 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:20 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.12.248:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:20 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.15.59:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:20 error : [xpandmon] Xpand: Could either not ping or create connection to 10.2.13.175:3306: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:20 error : [xpandmon] Xpand: Could either not ping or create connection to karma114:3306: [16388] Group change during GTM operation: group change in progress, try restarting transaction 2021-04-13 20:57:20 notice : [xpandmon] Attempting to find a Xpand bootstrap node from one of the nodes used during the previous run of MaxScale. 2021-04-13 20:57:20 error : [xpandmon] Xpand: Could not connect to any server or no server that could be connected to was part of the quorum. 2021-04-13 20:57:21 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:21 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:21 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:21 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:21 info : (3) Connected to '@@Xpand:node-35' with thread id 32769 2021-04-13 20:57:21 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:21 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:21 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 18 2021-04-13 20:57:21 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:22 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:22 info : Execute sescmd #1 on '@@Xpand:node-35': [COM_INIT_DB] 2021-04-13 20:57:22 info : [readwritesplit] Connected to '@@Xpand:node-35' 2021-04-13 20:57:22 info : [readwritesplit] Queuing query until '@@Xpand:node-35' completes session command 2021-04-13 20:57:22 info : (3) Connected to '@@Xpand:node-35' with thread id 33793 2021-04-13 20:57:22 error : (3) Invalid authentication message from backend '@@Xpand:node-35'. Error code: 1045, Msg : #HY000: [39936] Access denied: for user 'maxscale'@'' (using password: YES) 2021-04-13 20:57:22 info : (3) [readwritesplit] (Splitter-Service) Master '@@Xpand:node-35' failed: #HY000: Lost connection to backend server: connection closed by peer (@@Xpand:node-35) 2021-04-13 20:57:22 info : (3) [readwritesplit] (Splitter-Service) Starting transaction replay 19 2021-04-13 20:57:22 info : (3) [readwritesplit] (Splitter-Service) Replaying: begin 2021-04-13 20:57:22 notice : [xpandmon] Xpand: Monitoring Xpand cluster state using node 10.2.12.248:3306. 2021-04-13 20:57:22 info : [xpandmon] Deleted Xpand node 35 from bookkeeping. 2021-04-13 20:57:22 info : [xpandmon] Updated Xpand node in bookkeeping: 3, '10.2.12.248', 3306, 3581. 2021-04-13 20:57:22 info : [xpandmon] Updated Xpand node in bookkeeping: 15, '10.2.15.59', 3306, 3581. 2021-04-13 20:57:23 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:23 info : [readwritesplit] Replacing old master '@@Xpand:node-35' with new master '@@Xpand:node-15' 2021-04-13 20:57:23 info : Execute sescmd #1 on '@@Xpand:node-15': [COM_INIT_DB] 2021-04-13 20:57:23 info : [readwritesplit] Connected to '@@Xpand:node-15' 2021-04-13 20:57:23 info : [readwritesplit] Queuing query until '@@Xpand:node-15' completes session command 2021-04-13 20:57:23 info : (3) Connected to '@@Xpand:node-15' with thread id 582671 2021-04-13 20:57:23 info : (3) Authentication to '@@Xpand:node-15' succeeded. 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:23 info : (3) (Splitter-Service) Execute sescmd #2 on '@@Xpand:node-15': [COM_QUERY] set session autocommit=false 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) 1 session commands left on '@@Xpand:node-15' 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-13 20:57:23 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 10, type: QUERY_TYPE_BEGIN_TRX, stmt: begin 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-15' 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) New COM_QUERY received while transaction replay is active: insert into t1 values (3) 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: begin 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Replaying: insert into t1 values (1) 2021-04-13 20:57:23 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (1) 2021-04-13 20:57:23 info : [readwritesplit] Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (1) 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Replaying: insert into t1 values (2) 2021-04-13 20:57:23 info : > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (2) 2021-04-13 20:57:23 info : [readwritesplit] Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (2) 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Checksums match, replay successful. 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) >>> Routing stored queries 2021-04-13 20:57:23 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (3) 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) <<< Stored queries routed 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (3) 2021-04-13 20:57:23 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:33 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 30, type: QUERY_TYPE_WRITE, stmt: insert into t1 values (4) 2021-04-13 20:57:33 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:33 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: insert into t1 values (4) 2021-04-13 20:57:33 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:37 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 11, type: QUERY_TYPE_COMMIT, stmt: commit 2021-04-13 20:57:37 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:37 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: commit 2021-04-13 20:57:37 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 20:57:37 info : (3) [readwritesplit] (Splitter-Service) Transaction complete 2021-04-13 20:57:43 info : (3) (Splitter-Service) > Autocommit: [disabled], trx is [open], cmd: (0x03) COM_QUERY, plen: 21, type: QUERY_TYPE_READ, stmt: select * from t1 2021-04-13 20:57:43 info : (3) [readwritesplit] (Splitter-Service) Route query to master: @@Xpand:node-15 < 2021-04-13 20:57:43 info : (3) [readwritesplit] (Splitter-Service) Transaction starting on '@@Xpand:node-15' 2021-04-13 20:57:43 info : (3) [readwritesplit] (Splitter-Service) Adding to trx: select * from t1 2021-04-13 20:57:43 info : (3) [readwritesplit] (Splitter-Service) Reply complete, last reply from @@Xpand:node-15 2021-04-13 21:00:24 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-13 21:00:24 info : (1) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:00:51 info : Pinging '@@Xpand:node-3', idle for 301 seconds 2021-04-13 21:00:51 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:02:44 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-13 21:02:44 info : (3) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:05:25 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-13 21:05:25 info : (1) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:05:52 info : Pinging '@@Xpand:node-3', idle for 301 seconds 2021-04-13 21:05:52 info : (2) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:07:46 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-13 21:07:46 info : (3) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:10:27 info : Pinging '@@Xpand:node-15', idle for 301 seconds 2021-04-13 21:10:27 info : (1) Response to COM_CHANGE_USER is OK, writing stored query 2021-04-13 21:10:54 info : Pinging '@@Xpand:node-3', idle for 301 seconds 2021-04-13 21:10:54 info : (2) Response to COM_CHANGE_USER is OK, writing stored query