CentOS 7.5.1804 with MariaDB 10.1.36 backend servers

Hi here is a debug log:

Sep 27 14:20:04 e4b01 maxscale[21615]: (824) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:04 e4b01 maxscale[21615]: (824) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:04 e4b01 maxscale[21615]: (824) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 4152280)
Sep 27 14:20:08 e4b01 maxscale[21615]: (905) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:08 e4b01 maxscale[21615]: (905) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:08 e4b01 maxscale[21615]: (905) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 5279267)
Sep 27 14:20:20 e4b01 maxscale[21615]: (1070) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db05' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:20 e4b01 maxscale[21615]: (1070) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:20 e4b01 maxscale[21615]: (1070) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 2368851)
Sep 27 14:20:20 e4b01 maxscale[21615]: (1073) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:20 e4b01 maxscale[21615]: (1073) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:20 e4b01 maxscale[21615]: (1073) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 3175738)
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 93212)
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db04' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) Stmt 4: SET NAMES utf8mb4
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) Stmt 3: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 93212)
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) Stmt 2: SELECT *#012 FROM `city_list`#012 WHERE (`ID` = 1680)#012 ORDER BY `area` ASC, `City` ASC
Sep 27 14:20:21 e4b01 maxscale[21615]: (1088) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 5275123)
Sep 27 14:20:21 e4b01 maxscale[21615]: (1096) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:21 e4b01 maxscale[21615]: (1096) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:21 e4b01 maxscale[21615]: (1096) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 3507012)
Sep 27 14:20:22 e4b01 maxscale[21615]: (1110) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db03' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:22 e4b01 maxscale[21615]: (1110) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:22 e4b01 maxscale[21615]: (1110) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 5282451)
Sep 27 14:20:23 e4b01 maxscale[21615]: (1158) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:23 e4b01 maxscale[21615]: (1158) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:23 e4b01 maxscale[21615]: (1158) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 5039575)
Sep 27 14:20:32 e4b01 maxscale[21615]: (1288) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:32 e4b01 maxscale[21615]: (1288) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:32 e4b01 maxscale[21615]: (1288) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 1816141)
Sep 27 14:20:41 e4b01 maxscale[21615]: (1492) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db03' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:41 e4b01 maxscale[21615]: (1492) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:41 e4b01 maxscale[21615]: (1492) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 2718299)
Sep 27 14:20:42 e4b01 maxscale[21615]: (1531) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db03' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:42 e4b01 maxscale[21615]: (1531) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:42 e4b01 maxscale[21615]: (1531) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserName` = 'Omardk87')#012 AND (`Deleted` = '0')
Sep 27 14:20:48 e4b01 maxscale[21615]: (1662) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:48 e4b01 maxscale[21615]: (1662) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:48 e4b01 maxscale[21615]: (1662) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 1247833)
Sep 27 14:20:49 e4b01 maxscale[21615]: (1683) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:49 e4b01 maxscale[21615]: (1683) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:49 e4b01 maxscale[21615]: (1683) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 5004007)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db05' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 1424640)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 5: SET NAMES utf8mb4
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 4: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 1424640)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 3: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 1575908)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 2: SELECT *#012 FROM `city_list`#012 WHERE (`ID` = 142)#012 ORDER BY `area` ASC, `City` ASC
Sep 27 14:20:58 e4b01 maxscale[21615]: (1781) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 94676)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1835) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db06' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:58 e4b01 maxscale[21615]: (1835) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:58 e4b01 maxscale[21615]: (1835) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 4114252)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1882) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db08' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:58 e4b01 maxscale[21615]: (1882) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:58 e4b01 maxscale[21615]: (1882) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 3421057)
Sep 27 14:20:58 e4b01 maxscale[21615]: (1883) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db08' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:20:58 e4b01 maxscale[21615]: (1883) Stmt 2: SET NAMES utf8mb4
Sep 27 14:20:58 e4b01 maxscale[21615]: (1883) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 3421057)
Sep 27 14:21:02 e4b01 maxscale[21615]: (1959) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
Sep 27 14:21:02 e4b01 maxscale[21615]: (1959) Stmt 2: SET NAMES utf8mb4
Sep 27 14:21:02 e4b01 maxscale[21615]: (1959) Stmt 1: SELECT *#012 FROM `users`#012 WHERE (`UserID` = 3749756)

The /etc/maxscale.cf as requested by Markus and Wagner on google groups

==============================
[maxscale]
threads = auto
admin_host=10.10.3.47

[splitter-service]
type=service
router=readwritesplit
servers=db01, db02, db03, db04, db05, db06, db07, db08
user=maxscale
passwd=[REMOVED]
router_options=master_accept_reads=false

[query-logger]
type=filter
module=qlafilter
filebase=/tmp/maxscale-queries

[splitter-listener-3309]
type=listener
service=splitter-service
protocol=MariaDBClient
port=3309

[db01]
type=server
address=10.10.3.59
port=3306
protocol=MariaDBBackend
priority=1
persistpoolmax=500
persistmaxtime=3600

[db02]
type=server
address=10.10.3.60
port=3306
protocol=MariaDBBackend
priority=2
persistpoolmax=250
persistmaxtime=3600

[db03]
type=server
address=10.10.3.61
port=3306
protocol=MariaDBBackend
priority=3
persistpoolmax=250
persistmaxtime=3600

[db04]
type=server
address=10.10.3.62
port=3306
protocol=MariaDBBackend
priority=4
persistpoolmax=250
persistmaxtime=3600

[db05]
type=server
address=10.10.3.63
port=3306
protocol=MariaDBBackend
priority=5
persistpoolmax=250
persistmaxtime=3600

[db06]
type=server
address=10.10.3.64
port=3306
protocol=MariaDBBackend
priority=6
persistpoolmax=250
persistmaxtime=3600

[db07]
type=server
address=10.10.3.65
port=3306
protocol=MariaDBBackend
priority=7
persistpoolmax=250
persistmaxtime=3600

[db08]
type=server
address=10.10.3.66
port=3306
protocol=MariaDBBackend
priority=8
persistpoolmax=250
persistmaxtime=3600

[galera-monitor]
type=monitor
module=galeramon
servers=db01, db02, db03, db04, db05, db06, db07, db08
user=maxscale
passwd=[REMOVED]
use_priority=true

[CLI]
type=service
router=cli

[CLI-listener]
type=listener
service=CLI
protocol=maxscaled
socket=default

Just to double check, are those logs from syslog or some other logging system? I'd expect that the #012 is some form of pretty-printing. Getting the logs from /var/log/maxscale/maxscale.log would rule this out.

Apart from that, running those queries did not cause the message to be logged.

Sorry they where from /var/log/messages

here is the same snapshot from /var/log/maxscale/maxscale.log

2018-09-27 14:20:04 error : (824) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:04 notice : (824) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:04 notice : (824) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 4152280)
2018-09-27 14:20:08 error : (905) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:08 notice : (905) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:08 notice : (905) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 5279267)
2018-09-27 14:20:20 error : (1070) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db05' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:20 notice : (1070) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:20 notice : (1070) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 2368851)
2018-09-27 14:20:20 error : (1073) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:20 notice : (1073) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:20 notice : (1073) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 3175738)
2018-09-27 14:20:21 error : (1088) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:21 notice : (1088) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:21 notice : (1088) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 93212)
2018-09-27 14:20:21 error : (1088) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db04' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:21 notice : (1088) Stmt 4: SET NAMES utf8mb4
2018-09-27 14:20:21 notice : (1088) Stmt 3: SELECT *
FROM `users`
WHERE (`UserID` = 93212)
2018-09-27 14:20:21 notice : (1088) Stmt 2: SELECT *
FROM `city_list`
WHERE (`ID` = 1680)
ORDER BY `area` ASC, `City` ASC
2018-09-27 14:20:21 notice : (1088) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 5275123)
2018-09-27 14:20:21 error : (1096) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:21 notice : (1096) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:21 notice : (1096) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 3507012)
2018-09-27 14:20:22 error : (1110) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db03' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:22 notice : (1110) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:22 notice : (1110) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 5282451)
2018-09-27 14:20:23 error : (1158) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:23 notice : (1158) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:23 notice : (1158) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 5039575)
2018-09-27 14:20:32 error : (1288) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:32 notice : (1288) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:32 notice : (1288) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 1816141)
2018-09-27 14:20:41 error : (1492) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db03' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:41 notice : (1492) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:41 notice : (1492) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 2718299)
2018-09-27 14:20:42 error : (1531) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db03' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:42 notice : (1531) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:42 notice : (1531) Stmt 1: SELECT *
FROM `users`
WHERE (`UserName` = 'Omardk87')
AND (`Deleted` = '0')
2018-09-27 14:20:48 error : (1662) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:48 notice : (1662) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:48 notice : (1662) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 1247833)
2018-09-27 14:20:49 error : (1683) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:49 notice : (1683) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:49 notice : (1683) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 5004007)
2018-09-27 14:20:58 error : (1781) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db05' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:58 notice : (1781) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:58 notice : (1781) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 1424640)
2018-09-27 14:20:58 error : (1781) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:58 notice : (1781) Stmt 5: SET NAMES utf8mb4
2018-09-27 14:20:58 notice : (1781) Stmt 4: SELECT *
FROM `users`
WHERE (`UserID` = 1424640)
2018-09-27 14:20:58 notice : (1781) Stmt 3: SELECT *
FROM `users`
WHERE (`UserID` = 1575908)
2018-09-27 14:20:58 notice : (1781) Stmt 2: SELECT *
FROM `city_list`
WHERE (`ID` = 142)
ORDER BY `area` ASC, `City` ASC
2018-09-27 14:20:58 notice : (1781) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 94676)
2018-09-27 14:20:58 error : (1835) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db06' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:58 notice : (1835) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:58 notice : (1835) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 4114252)
2018-09-27 14:20:58 error : (1882) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db08' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:58 notice : (1882) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:58 notice : (1882) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 3421057)
2018-09-27 14:20:58 error : (1883) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db08' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:20:58 notice : (1883) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:20:58 notice : (1883) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 3421057)
2018-09-27 14:21:02 error : (1959) [readwritesplit] Unexpected internal state: received response 0x03 from server 'db02' when no response was expected. Command: 0x03 Query: <not available>
2018-09-27 14:21:02 notice : (1959) Stmt 2: SET NAMES utf8mb4
2018-09-27 14:21:02 notice : (1959) Stmt 1: SELECT *
FROM `users`
WHERE (`UserID` = 3749756)

Just add a bit more, we have 2 maxscale servers setup in active/passive configuration and I get the same error log on both when they have active traffic.

We just released MaxScale 2.2.15 with a fix that might affect this. Can you try to upgrade and see if it still happens?

I am not seeing the errors now, out of curiosity, what was the problem.

Seems that it was caused by MXS-2066.

Appears to have been fixed by MXS-2066.