[MXS-1163] Log flood using binlog server on Ubuntu Yakkety Yak Created: 2017-03-04  Updated: 2017-10-09  Resolved: 2017-10-09

Status: Closed
Project: MariaDB MaxScale
Component/s: binlogrouter
Affects Version/s: 2.0.4, 2.1.0
Fix Version/s: 2.1.10, 2.2.0

Type: Bug Priority: Major
Reporter: VAROQUI Stephane Assignee: Massimiliano Pinto (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
includes MXS-1155 Define a set of features for better i... Closed
Sprint: 2017-42, 2017-43

 Description   

maxsclale log is lofin around 500M per day of such information,

The only way to get read of it is to disable the binlog router
This happen as well if nothing is connecting to maxscale

2017-03-04 13:26:45 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:46 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:47 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:48 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:50 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:51 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:52 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:54 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:55 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:56 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:26:58 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017-03-04 13:27:00 error : Write to dcb 0x7c89b0 in state DCB_STATE_ALLOC fd 18 failed due errno 104, Connection reset by peer
2017
Os used
Linux version 4.8.0-30-generic (buildd@lcy01-08) (gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12) ) #32-Ubuntu SMP Fri Dec 2 03:43:27 UTC 2016
Config used
deb used
maxscale-2.0.4-1.ubuntu.xenial.x86_64.deb
cat /etc/debian_version
lsb_release -r
Release: 16.10

stretch/sid
[maxscale]
threads=1

[MaxInfo]
type=service
router=maxinfo
user=monitor
passwd=EBD2F49C3B375812A8CDEBA632ED8BBC

[MaxInfo JSON Listener]
type=listener
service=MaxInfo
protocol=HTTPD
port=4002

[Replication]
type=service
router=binlogrouter
version_string=5.6.17-log
user=skysql
passwd=skyvodka
router_options==send_slave_heartbeat=on,transaction_safety=on,semisync=1,mariadb10-compatibility=1,server-id=999,user=skysql,password=skyvodka,binlogdir=/var/cache/maxscale/
#router_options=send_slave_heartbeat=on,transaction_safety=on,semisync=1,mariadb10-compatibility=1,server-id=999,user=skysql,password=skyvodka,binlogdir=/var/lib/maxscale/

[Replication Listener]
type=listener
service=Replication
protocol=MySQLClient
port=3306

[MySQL Monitor]
type=monitor
module=mysqlmon
servers=server1,server2
user=skysql
passwd=skyvodka
monitor_interval=2000
detect_stale_master=true

  1. Service definitions
    #
  2. Service Definition for a read-only service and
  3. a read/write splitting service.
    #
  1. ReadConnRoute documentation:
  2. https://github.com/mariadb-corporation/MaxScale/blob/master/Documentation/Routers/ReadConnRoute.md

[Read-Only Service]
type=service
router=readconnroute
servers=server1
user=skysql
passwd=skyvodka
router_options=slave

  1. ReadWriteSplit documentation:
  2. https://github.com/mariadb-corporation/MaxScale/blob/master/Documentation/Routers/ReadWriteSplit.md

[Read-Write Service]
type=service
router=readwritesplit
servers=server1
user=skysql
passwd=skyvodka
max_slave_connections=100%

  1. This service enables the use of the MaxAdmin interface
  2. MaxScale administration guide:
  3. https://github.com/mariadb-corporation/MaxScale/blob/master/Documentation/Reference/MaxAdmin.md

[MaxAdmin Service]
type=service
router=cli

  1. Listener definitions for the services
    #
  2. These listeners represent the ports the
  3. services will listen on.
    #

[Write Connection Router]
type=service
router=readconnroute
router_options=master
servers=server1,server2
user=skysql
passwd=skyvodka
enable_root_user=true

[Read-Only Listener]
type=listener
service=Read-Only Service
protocol=MySQLClient
port=4008

[Write-Only Listener]
type=listener
service=Write Connection Router
protocol=MySQLClient
port=4007

[Read-Write Listener]
type=listener
service=Read-Write Service
protocol=MySQLClient
port=4006

#[MaxAdmin Listener]
#type=listener
#service=MaxAdmin Service
#protocol=maxscaled
#socket=default

  1. Internet Socket
    [CLI Inet Listener]
    type=listener
    service=MaxAdmin Service
    protocol=maxscaled
    address=0.0.0.0
    port= 3307

[server2]
type=server
address=192.168.0.41
port=5056
protocol=MySQLBackend

[server1]
type=server
address=192.168.0.41
port=5055
protocol=MySQLBackend



 Comments   
Comment by Dipti Joshi (Inactive) [ 2017-03-04 ]

stephane@skysql.com You are using MaxScale 2.0.4

Please try 2.1.0 beta which has a new global setting log_throttling to throttle flooding of log file

Comment by VAROQUI Stephane [ 2017-03-04 ]

Thanks Dipty , I'll try this and report

Comment by VAROQUI Stephane [ 2017-03-05 ]

issue still the same with MaxScale 2.1.0 Beta
03-05 12:17:59 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:00 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:01 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:02 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:03 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:04 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:05 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:06 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:07 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:08 error : Write to Client DCB 192.168.0.41 in state DCB_STATE_ALLOC failed due errno 104, Connection reset by peer
2017-03-05 12:18:09 error

Comment by VAROQUI Stephane [ 2017-03-05 ]

Still can reproduce with minimum conf file
[maxscale]
threads=1

[Replication]
type=service
router=binlogrouter
version_string=5.6.17-log
user=skysql
passwd=skyvodka
router_options==send_slave_heartbeat=on,transaction_safety=on,semisync=1,mariadb10-compatibility=1,server-id=999,user=skysql,password=skyvodka,binlogdir=/var/cache/maxscale/

[Replication Listener]
type=listener
service=Replication
protocol=MySQLClient
address=0.0.0.0
port=3306

[MaxAdmin Service]
type=service
router=cli

[CLI Inet Listener]
type=listener
service=MaxAdmin Service
protocol=maxscaled
address=0.0.0.0
port= 3307

[server2]
type=server
address=192.168.0.41
port=5056
protocol=MySQLBackend

[server1]
type=server
address=192.168.0.41
port=5055
protocol=MySQLBackend

Comment by Johan Wikman [ 2017-03-06 ]

Those errors are logged too rarely to be throttled. By default, a message that is logged 10 times in one second will be throttled for 10 seconds.

https://github.com/mariadb-corporation/MaxScale/blob/2.1/Documentation/Getting-Started/Configuration-Guide.md#log_throttling

Comment by Marco Fretz [ 2017-05-11 ]

what do those errors actually mean? is there something wrong?

we have several maxscale / galera installations with mariadb 10.1 on Ubuntu 16.04. We're using maxscale 2.0.5 and I see those errors on only one of our cluster setups:

2017-05-11 15:35:54   error  : Write to dcb 0x7f10000126c0 in state DCB_STATE_POLLING fd 16 failed due errno 104, Connection reset by peer
2017-05-11 15:35:56   error  : Write to dcb 0x7f10000143b0 in state DCB_STATE_POLLING fd 16 failed due errno 104, Connection reset by peer
2017-05-11 15:35:58   error  : Write to dcb 0x1af75b0 in state DCB_STATE_POLLING fd 13 failed due errno 104, Connection reset by peer
2017-05-11 15:35:58   error  : Write to dcb 0x7f1008011f40 in state DCB_STATE_POLLING fd 16 failed due errno 104, Connection reset by peer
2017-05-11 15:36:08   error  : Write to dcb 0x1af75b0 in state DCB_STATE_POLLING fd 18 failed due errno 104, Connection reset by peer
2017-05-11 15:36:12   error  : Write to dcb 0x1afa930 in state DCB_STATE_POLLING fd 15 failed due errno 104, Connection reset by peer
2017-05-11 15:36:20   error  : Write to dcb 0x1afb8b0 in state DCB_STATE_POLLING fd 18 failed due errno 104, Connection reset by peer
2017-05-11 15:36:22   error  : Write to dcb 0x7f1008011b50 in state DCB_STATE_POLLING fd 18 failed due errno 104, Connection reset by peer
2017-05-11 15:36:26   error  : Write to dcb 0x7f1000011b50 in state DCB_STATE_POLLING fd 15 failed due errno 104, Connection reset by peer
2017-05-11 15:36:30   error  : Write to dcb 0x1afb020 in state DCB_STATE_POLLING fd 15 failed due errno 104, Connection reset by peer
2017-05-11 15:36:36   error  : Write to dcb 0x1afa930 in state DCB_STATE_POLLING fd 17 failed due errno 104, Connection reset by peer
2017-05-11 15:36:36   error  : Write to dcb 0x1afa480 in state DCB_STATE_POLLING fd 15 failed due errno 104, Connection reset by peer
2017-05-11 15:36:38   error  : Write to dcb 0x7f1000015250 in state DCB_STATE_POLLING fd 13 failed due errno 104, Connection reset by peer

Unfortunately I cannot upgrade this affected system to a beta version (2.1).

any ideas?
thank you very much

Comment by Marco Fretz [ 2017-05-16 ]

an update from my side: i tested the latest 2.1.x package I found for Xenial 16.04 and the errors are the same, looks like they are not related to connection to maxscale, possibly coming from the galera monitor?

any news on this?

Comment by Massimiliano Pinto (Inactive) [ 2017-09-27 ]

stephane@skysql.com

Is binlog server configured or there is just the entry in the config file?
If configured please share master.ini file (located in binlogdir) with all master server details.
We don't see any information about master server: could you add more details?
More informations are needed in order to check for any binlog router issue.

Comment by Massimiliano Pinto (Inactive) [ 2017-10-09 ]

MXS-1458: Mark backend server as inactive if router is not configured.

The un-configured backend server is no longer checked for db users auth loading.

Generated at Thu Feb 08 04:04:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.