[MXS-1336] Cascaded MaxScales throwing errors after 10 Minutes Created: 2017-07-28  Updated: 2017-11-07  Resolved: 2017-11-07

Status: Closed
Project: MariaDB MaxScale
Component/s: readconnroute, schemarouter
Affects Version/s: 1.4.5
Fix Version/s: N/A

Type: Bug Priority: Blocker
Reporter: Stefan Koop Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Environment:

Ubuntu 16.04 LTS 64 Bit
MaxScale from MariaDB repositories
MariaDB 10.1.22 from MariaDB repositories
PHP 7.1 / PDO_MYSQL from Ubuntu PPA (ondrej)


 Description   

We are plannign to use a cascaded setup of MaxScales where our Application (Symfony/Doctrine) only knows one central MaxScale instance (maxscale0).

  • We have multiple Galera clusters
  • Each Galera cluster has its own MaxScale in front
  • maxscale0 has access to all other maxscale instances
  • We are using the schemarouter on maxscale0

Application -> maxscale0 -> maxscale(1-10)

Everything works very well and fast for 10 minutes. After 10 minutes doctrine is starting to throw errors.

"exception":{"class":"Doctrine\\ORM\\Internal\\Hydration\\HydrationException","message":"The discriminator column \"type\" is missing for \"WM\\UtilsBundle\\Entity\\ConfigTrackingAccount\" using the DQL alias \"r\"."}

With a message from MaxScale

SQLSTATE[HY000]: General error: 2003 Lost connection to backend server.

Even MaxScale in Debug mode does not show where the problem may come from.
It seems like, even on long running queries, the database gets lost.

Configurations

maxscale0

 
 
[MaxScale]
 
threads = 4
 
auth_connect_timeout = 5
 
auth_read_timeout = 10
 
auth_write_timeout = 5
 
...
 
 
 
...
 
[ShardAllDatabases]
 
type=service
 
router=schemarouter
 
servers=maxscale1,maxscale2,maxscale3
 
user=
 
passwd=
 
filters=FilterCreateDatabase|FilterDropDatabase
 
auth_all_servers=1
 
router_options=refresh_interval=5,refresh_databases=1
 
...
 
 
 
...
 
[maxscale1]
 
type=server
 
address=123.123.123.123
 
port=3306
 
protocol=MySQLBackend
 
...

maxscaleN

[MaxScale]
 
threads = 2
 
auth_connect_timeout = 5
 
auth_read_timeout = 10
 
auth_write_timeout = 5
 
...
 
 
 
[MySQL]
 
type=service
 
router=readconnroute
 
servers=mariadb1,mariadb2,mariadb3
 
enable_root_user=0
 
localhost_match_wildcard_host=1
 
version_string='MaxScale'
 
auth_all_servers=1
 
log_auth_warnings=1
 
router_options=synced
 
user=
 
passwd=
 
 
 
[mariadb1]
 
type=server
 
address=111.111.111.111
 
port=3306
 
protocol=MySQLBackend
 
...



 Comments   
Comment by markus makela [ 2017-07-28 ]

I would first recommend upgrading to MaxScale 2.1.4 and testing if this fixes your problem.

Comment by Stefan Koop [ 2017-07-28 ]

I would agree with that. But it's a licensing/costs issue. That's why we have to stick with 1.4.x.

Comment by markus makela [ 2017-07-28 ]

Please enable the info log level by adding log_info=true under the maxscale section and upload the resulting log file with all sensitive information removed. This should help us figure out why the connections are being closed.

Comment by Stefan Koop [ 2017-07-28 ]

Edit: I hope this is enough log output.

It seems, that MaxScale transports the errors through the instances and appends them to the current session.

Maybe there is a setting for that behaviour which i did not find out yet ...

2017-07-28 15:16:30   [4]  info   : schemarouter: returning reply [RSET] state [READY]  session [0x7fc738000ec0]
 
2017-07-28 15:16:30   debug  : 140493829052160 [dcb_write] Append to writequeue. 1800 writes buffered for dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5
 
2017-07-28 15:16:30   debug  : 140493829052160 [dcb_drain_writeq] Wrote 592 Bytes to dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5
 
2017-07-28 15:16:30   debug  : 140494456907648 [poll_waitevents] epoll_wait found 1 fds
 
2017-07-28 15:16:30   debug  : 140494456907648 [poll_waitevents] event 5 dcb 0x7fc738016f00 role DCB_ROLE_REQUEST_HANDLER
 
2017-07-28 15:16:30   debug  : 140494456907648 [gw_write_backend_event] wrote to dcb 0x7fc738016f00 fd 19, return 1
 
2017-07-28 15:16:30   debug  : 140494456907648 [poll_waitevents] Read in dcb 0x7fc738016f00 fd 19
 
2017-07-28 15:16:30   debug  : 140494456907648 [gw_read_backend_event] Read dcb 0x7fc738016f00 fd 19 protocol state 11, MYSQL_IDLE.
 
2017-07-28 15:16:30   debug  : 140494456907648 [dcb_read] Read 47 bytes from dcb 0x7fc738016f00 in state DCB_STATE_POLLING fd 19.
 
2017-07-28 15:16:30   debug  : 140494456907648 [protocol_get_srv_command] Read command UNKNOWN MYSQL PACKET TYPE for fd 19.
 
2017-07-28 15:16:30   debug  : schemarouter: Reply from [maxscale3] session [0x7fc738000ec0] mapping [false] queries queued [none]
 
2017-07-28 15:16:30   [4]  info   : schemarouter: returning reply [ERR] state [READY]  session [0x7fc738000ec0]
 
2017-07-28 15:16:30   debug  : 140494456907648 [dcb_write] Append to writequeue. 1801 writes buffered for dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5
 
2017-07-28 15:16:30   debug  : 140494456907648 [dcb_drain_writeq] Wrote 47 Bytes to dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5
 
2017-07-28 15:16:30   debug  : 140494456907648 [poll_waitevents] epoll_wait found 1 fds
 
2017-07-28 15:16:30   debug  : 140494456907648 [poll_waitevents] event 5 dcb 0x7fc738012d50 role DCB_ROLE_REQUEST_HANDLER
 
2017-07-28 15:16:30   debug  : 140494456907648 [gw_write_backend_event] wrote to dcb 0x7fc738012d50 fd 16, return 1
 
2017-07-28 15:16:30   debug  : 140494456907648 [poll_waitevents] Read in dcb 0x7fc738012d50 fd 16
 
2017-07-28 15:16:30   debug  : 140494456907648 [gw_read_backend_event] Read dcb 0x7fc738012d50 fd 16 protocol state 11, MYSQL_IDLE.
 
2017-07-28 15:16:30   debug  : 140494456907648 [dcb_read] Read 47 bytes from dcb 0x7fc738012d50 in state DCB_STATE_POLLING fd 16.
 
2017-07-28 15:16:30   debug  : 140494456907648 [protocol_get_srv_command] Read command UNKNOWN MYSQL PACKET TYPE for fd 16.
 
2017-07-28 15:16:30   debug  : schemarouter: Reply from [maxscale44] session [0x7fc738000ec0] mapping [false] queries queued [none]
 
2017-07-28 15:16:30   [4]  info   : schemarouter: returning reply [ERR] state [READY]  session [0x7fc738000ec0]
 
2017-07-28 15:16:30   debug  : 140494456907648 [dcb_write] Append to writequeue. 1802 writes buffered for dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5
 
2017-07-28 15:16:30   debug  : 140494456907648 [dcb_drain_writeq] Wrote 47 Bytes to dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5
 
2017-07-28 15:16:31   debug  : 140493845837568 [poll_waitevents] epoll_wait found 1 fds
 
2017-07-28 15:16:31   debug  : 140493845837568 [poll_waitevents] event 5 dcb 0x7fc738000ac0 role DCB_ROLE_REQUEST_HANDLER
 
2017-07-28 15:16:31   debug  : 140493845837568 [poll_waitevents] Read in dcb 0x7fc738000ac0 fd 5
 
2017-07-28 15:16:31   debug  : 140493845837568 [dcb_read] Read 27 bytes from dcb 0x7fc738000ac0 in state DCB_STATE_POLLING fd 5.
 
2017-07-28 15:16:31   debug  : 140493845837568 [resolve_query_type] Item CONNECTION_ID():FUNC_ITEM
 
2017-07-28 15:16:31   debug  : 140493845837568 [resolve_query_type] functype UNKNOWN_FUNC, typically some system function.
 
2017-07-28 15:16:31   [4]  info   : > Cmd: COM_QUERY, type: QUERY_TYPE_READ, stmt: SELECT CONNECTION_ID() 
2017-07-28 15:16:31   [4]  info   : schemarouter: Using active database 'my_database'
 
2017-07-28 15:16:31   [4]  info   : Route query to      111.111.111.111:3306 <

The result are multiple errormessages where one is expected.

Comment by Johan Wikman [ 2017-11-07 ]

We do not actively develop 1.4 anymore

Generated at Thu Feb 08 04:05:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.