[MXS-1244] MySQL monitor "detect_replication_lag=true" doesn't work with "mysql51_replication=true" Created: 2017-04-20  Updated: 2017-05-23  Resolved: 2017-04-21

Status: Closed
Project: MariaDB MaxScale
Component/s: mariadbmon
Affects Version/s: 2.0.0
Fix Version/s: 2.0.6, 2.1.3

Type: Bug Priority: Major
Reporter: Massimiliano Pinto (Inactive) Assignee: Massimiliano Pinto (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MXS-1260 MaxAdmin show service missing stats w... Closed
relates to MXS-1272 MySQL Monitor with a Replication Clus... Closed

 Description   

The call to function set_master_heartbeat requires both arguments handle and root_master not to be null for being able to succeed.

but handle->master is always null and there's no chance for set_master_heartbeat to create the database maxscale_schema and table replication_heartbeat.

it seems an intrinsic bug of MaxScale not being able to cope detect_replication_lag=true when mysql51_replication=true.

 Comments   
Comment by Andrea Barbieri [ 2017-04-21 ]

relevant snippets in /etc/maxscale.cnf

#
 
[MySQL Monitor]
 
type=monitor
 
module=mysqlmon
 
servers=server1,server2,server3
 
user=maxmon
 
password=HASH_XXX
 
monitor_interval=5000
 
mysql51_replication=true
 
detect_stale_master=true
 
detect_replication_lag=true
 
 
 
#
 
[server1]
 
type=server
 
address=10.10.81.191
 
port=3306
 
protocol=MySQLBackend
 
 
 
[server2]
 
# minkar
 
address=10.10.81.192
 
port=3306
 
protocol=MySQLBackend
 
 
 
[server3]
 
type=server
 
address=10.10.81.193
 
port=3306
 
protocol=MySQLBackend

and entries in log file when starting the service:

MariaDB Corporation MaxScale    /var/log/maxscale/maxscale1.log Thu Apr 20 13:39:48 2017
 
-----------------------------------------------------------------------
 
2017-04-20 13:39:48   notice : Working directory: /var/log/maxscale
 
2017-04-20 13:39:48   notice : MariaDB MaxScale 2.0.5 started
 
2017-04-20 13:39:48   notice : MaxScale is running in process 15480
 
2017-04-20 13:39:48   notice : Configuration file: /etc/maxscale.cnf
 
2017-04-20 13:39:48   notice : Log directory: /var/log/maxscale
 
2017-04-20 13:39:48   notice : Data directory: /var/lib/maxscale
 
2017-04-20 13:39:48   notice : Module directory: /usr/lib64/maxscale
 
2017-04-20 13:39:48   notice : Service cache: /var/cache/maxscale
 
2017-04-20 13:39:48   notice : Initializing statemend-based read/write split router module.
 
2017-04-20 13:39:48   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
 
2017-04-20 13:39:48   notice : Initialise readconnroute router module V1.1.0.
 
2017-04-20 13:39:48   notice : Loaded module readconnroute: V1.1.0 from /usr/lib64/maxscale/libreadconnroute.so
 
2017-04-20 13:39:48   notice : Initialise CLI router module V1.0.0.
 
2017-04-20 13:39:48   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
 
2017-04-20 13:39:48   notice : Initialise the MySQL Monitor module V1.4.0.
 
2017-04-20 13:39:48   notice : Loaded module mysqlmon: V1.4.0 from /usr/lib64/maxscale/libmysqlmon.so
 
2017-04-20 13:39:48   notice : No query classifier specified, using default 'qc_sqlite'.
 
2017-04-20 13:39:48   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
 
2017-04-20 13:39:48   notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
 
2017-04-20 13:39:48   notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so
 
2017-04-20 13:39:48   notice : Listening connections at /tmp/maxadmin.sock with protocol MaxScale Admin
 
2017-04-20 13:39:49   notice : Server changed state: server1[10.10.81.191:3306]: new_slave. [Running] -> [Slave, Running]
 
2017-04-20 13:39:49   notice : Server changed state: server2[10.10.81.192:3306]: new_master. [Running] -> [Master, Running]
 
2017-04-20 13:39:49   notice : Server changed state: server3[10.10.81.193:3306]: new_slave. [Running] -> [Slave, Running]
 
2017-04-20 13:39:49   notice : A Master Server is now available: 10.10.81.192:3306
 
2017-04-20 13:39:49   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:39:49   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:39:49   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:39:49   notice : Loaded 15 MySQL Users for service [ReadConnRouter_Slave].
 
2017-04-20 13:39:49   notice : Loaded module MySQLClient: V1.1.0 from /usr/lib64/maxscale/libMySQLClient.so
 
2017-04-20 13:39:49   notice : Listening connections at 10.10.81.190:4307 with protocol MySQL
 
2017-04-20 13:39:49   notice : Loaded 15 MySQL Users for service [ReadConnRouter_Master].
 
2017-04-20 13:39:49   notice : Listening connections at 10.10.81.190:4308 with protocol MySQL
 
2017-04-20 13:39:49   notice : Loaded 15 MySQL Users for service [RWRC].
 
2017-04-20 13:39:49   notice : Listening connections at 10.10.81.190:5307 with protocol MySQL
 
2017-04-20 13:39:49   notice : Started MaxScale log flusher.
 
2017-04-20 13:39:49   notice : MaxScale started with 2 server threads.
 
2017-04-20 13:39:54   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:39:54   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:39:54   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:39:59   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:39:59   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:39:59   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:40:04   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:40:04   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:40:04   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:40:09   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:40:09   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:40:09   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:40:14   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:40:14   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:40:14   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
...
 
2017-04-20 13:41:06   notice : Loaded module MaxAdminAuth: V2.0.0 from /usr/lib64/maxscale/libMaxAdminAuth.so
 
2017-04-20 13:41:09   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:41:09   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:41:09   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:41:14   error  : [mysql_mon]: set_master_heartbeat called without an available Master server
 
2017-04-20 13:41:14   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
2017-04-20 13:41:14   error  : [mysql_mon]: set_slave_heartbeat called without an available Master server
 
...

and this is the show servers report:

[root@gienah ~]# maxadmin show servers
 
Server 0xde0770 (server1)
 
	Server:                              10.10.81.191
 
	Status:                              Slave, Running
 
	Protocol:                            MySQLBackend
 
	Port:                                3306
 
	Server Version:                      5.1.62-MariaDB-mariadb115-log
 
	Node Id:                             191
 
	Master Id:                           192
 
	Slave Ids:                           
	Repl Depth:                          -1
 
	Number of connections:               0
 
	Current no. of conns:                0
 
	Current no. of operations:           0
 
Server 0xde0290 (server2)
 
	Server:                              10.10.81.192
 
	Status:                              Master, Running
 
	Protocol:                            MySQLBackend
 
	Port:                                3306
 
	Server Version:                      5.1.62-MariaDB-mariadb115-log
 
	Node Id:                             192
 
	Master Id:                           -1
 
	Slave Ids:                           193, 191 
	Repl Depth:                          -1
 
	Number of connections:               0
 
	Current no. of conns:                0
 
	Current no. of operations:           0
 
Server 0xddfdb0 (server3)
 
	Server:                              10.10.81.193
 
	Status:                              Slave, Running
 
	Protocol:                            MySQLBackend
 
	Port:                                3306
 
	Server Version:                      5.1.62-MariaDB-mariadb115-log
 
	Node Id:                             193
 
	Master Id:                           192
 
	Slave Ids:                           
	Repl Depth:                          -1
 
	Number of connections:               0
 
	Current no. of conns:                0
 
	Current no. of operations:           0

the current topology is indeed:

  • server1 Slave (server-id=191)
  • server2 Master (server-id=192)
  • server3 Slave (server-id=193)
Comment by Andrea Barbieri [ 2017-04-21 ]

I also find this intriguing

Repl Depth:                          -1

in this simple case it should be:

  • 0 for Master
  • 1 for Slaves
Comment by Massimiliano Pinto (Inactive) [ 2017-04-21 ]

I confirm that build_mysql51_replication_tree() routine doesn't set handle->master

Additionally server's field "depth" is not set.

Accordingly to MaxScale 2.0.5 code:

https://github.com/mariadb-corporation/MaxScale/blob/2.0.5/server/modules/monitor/mysql_mon.c

I tested a quick fix for the missing handle->master

after line 578
add 2 lines:

MYSQL_MONITOR *handle = mon->handle;
handle->master = database;

If you are able to recompile your maxscale code base you can give it a try.

If not you might need to wait for a new 2.0 packages in the next days.

Massimiliano

Comment by Andrea Barbieri [ 2017-04-21 ]

I have just done so earlier today.

  • checked out v2.0.5
  • patched server/modules/monitor/mysql_mon.c
  • built the RPM package (I'm using CentOS 7.3)
  • used the extracted /usr/lib64/maxscale/libmysqlmon.so.1.4.0

and after restarting the maxscale service with the patched libmysqlmon.so.1.4.0 the show servers now reports:

Server 0x22ed770 (server1)
 
    Server:                              10.10.81.191
 
    Status:                              Slave, Running
 
    Protocol:                            MySQLBackend
 
    Port:                                3306
 
    Server Version:                      5.1.62-MariaDB-mariadb115-log
 
    Node Id:                             191
 
    Master Id:                           192
 
    Slave Ids:                           
    Repl Depth:                          -1
 
    Slave delay:                         0
 
    Last Repl Heartbeat:                 Fri Apr 21 13:58:50 2017
 
    Number of connections:               0
 
    Current no. of conns:                0
 
    Current no. of operations:           0
 
Server 0x22ed290 (server2)
 
    Server:                              10.10.81.192
 
    Status:                              Master, Running
 
    Protocol:                            MySQLBackend
 
    Port:                                3306
 
    Server Version:                      5.1.62-MariaDB-mariadb115-log
 
    Node Id:                             192
 
    Master Id:                           -1
 
    Slave Ids:                           193, 191 
    Repl Depth:                          -1
 
    Last Repl Heartbeat:                 Fri Apr 21 13:58:55 2017
 
    Number of connections:               0
 
    Current no. of conns:                0
 
    Current no. of operations:           0
 
Server 0x22ecdb0 (server3)
 
    Server:                              10.10.81.193
 
    Status:                              Slave, Running
 
    Protocol:                            MySQLBackend
 
    Port:                                3306
 
    Server Version:                      5.1.62-MariaDB-mariadb115-log
 
    Node Id:                             193
 
    Master Id:                           192
 
    Slave Ids:                           
    Repl Depth:                          -1
 
    Slave delay:                         0
 
    Last Repl Heartbeat:                 Fri Apr 21 13:58:55 2017
 
    Number of connections:               0
 
    Current no. of conns:                0
 
    Current no. of operations:           0

Ideally to complete MariaDB 5.1 compatibility the server's field "depth" should be set depending on the role and cluster topology.

Comment by Andrea Barbieri [ 2017-04-21 ]

I also noticed when explicitly setting the parameter detect_stale_slave I experience the following error:

2017-04-21 14:04:07   error  : Unexpected parameter 'detect_stale_slave' for object 'MySQL Monitor' of type 'monitor'.

MySQL Monitor v2.x documentation seems to suggest it is a valid monitor section parameter.

Comment by Massimiliano Pinto (Inactive) [ 2017-04-24 ]

We added repl depth:

https://github.com/mariadb-corporation/MaxScale/commit/e6b34ea85cb1bae47f51ac2c79129ea9b1910d13

Server 0x63e1c0 (server1)
Server: 127.0.0.1
Status: Master, Running
Protocol: MySQLBackend
Port: 10116
Server Version: 10.1.16-MariaDB
Node Id: 10116
Master Id: -1
Slave Ids: 93, 102
Repl Depth: 0
...

Server 0x63c6e0 (server3)
Server: 127.0.0.1
Status: Slave, Running
Protocol: MySQLBackend
Port: 25233
Server Version: 10.0.21-MariaDB-log
Node Id: 102
Master Id: 10116
Slave Ids:
Repl Depth: 1

Comment by Massimiliano Pinto (Inactive) [ 2017-04-24 ]

Parameter 'detect_stale_slave' is missing in server/core/config.c.

The fix is:
add

"detect_stale_slave",

after "detect_stale_master",

static char *monitor_params[] =

{ ... "detect_replication_lag", "detect_stale_master", "detect_stale_slave", .... }
Comment by Andrea Barbieri [ 2017-04-24 ]

confirmed, applied fix and now parameter detect_stale_slave is recognised in the monitor section.

Generated at Thu Feb 08 04:05:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.