[MXS-1468] Using dynamic commands to create readwritesplit configs fail after restart Created: 2017-10-10  Updated: 2017-10-17  Resolved: 2017-10-14

Status: Closed
Project: MariaDB MaxScale
Component/s: mariadbmon, maxadmin, readconnroute, readwritesplit
Affects Version/s: 2.1.9
Fix Version/s: 2.1.10

Type: Bug Priority: Major
Reporter: Wagner Bianchi (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 1
Labels: None
Environment:

Linux, running CentOS 7.2
3.10.0-327.4.5.el7.x86_64


 Description   

Hello Folks,

Following up with a setup of Maxscale 2.1.9 when a ReadWriteSplit is on the radar to be implemented, I started Maxscale on the box with a very basic configuration file with the intention to create everything dynamically and in separated files which are located at /var/lib/maxscale/maxscale.cnf.d/, making it way more organized then having just one giant maxscale.cnf file.

I followed the below:

#: basic configuration file
 
[root@maxscale maxscale_configs]# cat basic.cnf
 
[maxscale]
 
threads=auto
 
 
 
[rwsplit-service]
 
type=service
 
router=readwritesplit
 
user=maxuser
 
passwd=FD08F35D87C442AABA518CFB7471D691
 
 
 
[CLI]
 
type=service
 
router=cli
 
 
 
[CLI Listener]
 
type=listener
 
service=CLI
 
protocol=maxscaled
 
socket=default

And then, I started maxscale, entered the following commands, killed its process and start again with the basic configuration, which should give me a maxscale with basic configurations + what I created on the command line:

[root@maxscale maxscale_configs]# maxscale -f basic.cnf
 
 
 
[root@maxscale maxscale_configs]# maxadmin create monitor cluster-monitor mysqlmon
 
Created monitor 'cluster-monitor'
 
 
 
[root@maxscale maxscale_configs]# maxadmin alter monitor cluster-monitor user=maxuser password=FD08F35D87C442AABA518CFB7471D691 monitor_interval=1000
 
 
 
[root@maxscale maxscale_configs]# maxadmin restart monitor cluster-monitor
 
 
 
[root@maxscale maxscale_configs]# maxadmin create listener rwsplit-service rwsplit-listener 0.0.0.0 4007
 
Listener 'rwsplit-listener' created
 
 
 
[root@maxscale maxscale_configs]# maxadmin list listeners
 
Listeners.
 
---------------------+---------------------+--------------------+-----------------+-------+--------
 
Name                 | Service Name        | Protocol Module    | Address         | Port  | State
 
---------------------+---------------------+--------------------+-----------------+-------+--------
 
rwsplit-listener     | rwsplit-service     | MySQLClient        | 0.0.0.0         |  4007 | Running
 
CLI Listener         | CLI                 | maxscaled          | default         |     0 | Running
 
---------------------+---------------------+--------------------+-----------------+-------+--------
 
 
 
[root@maxscale maxscale_configs]# maxadmin create server prod_mysql01 192.168.50.11 3306
 
Created server 'prod_mysql01'
 
[root@maxscale maxscale_configs]# maxadmin create server prod_mysql02 192.168.50.12 3306
 
Created server 'prod_mysql02'
 
[root@maxscale maxscale_configs]# maxadmin create server prod_mysql03 192.168.50.13 3306
 
Created server 'prod_mysql03'
 
 
 
[root@maxscale maxscale_configs]# maxadmin list servers
 
Servers.
 
-------------------+-----------------+-------+-------------+--------------------
 
Server             | Address         | Port  | Connections | Status
 
-------------------+-----------------+-------+-------------+--------------------
 
prod_mysql03       | 192.168.50.13   |  3306 |           0 | Running
 
prod_mysql02       | 192.168.50.12   |  3306 |           0 | Running
 
prod_mysql01       | 192.168.50.11   |  3306 |           0 | Running
 
-------------------+-----------------+-------+-------------+--------------------
 
 
 
[root@maxscale maxscale_configs]# maxadmin add server prod_mysql02 cluster-monitor rwsplit-service
 
Added server 'prod_mysql02' to 'cluster-monitor'
 
Added server 'prod_mysql02' to 'rwsplit-service'
 
 
 
[root@maxscale maxscale_configs]# maxadmin add server prod_mysql01 cluster-monitor rwsplit-service
 
Added server 'prod_mysql01' to 'cluster-monitor'
 
Added server 'prod_mysql01' to 'rwsplit-service'
 
 
 
[root@maxscale maxscale_configs]# maxadmin add server prod_mysql03 cluster-monitor rwsplit-service
 
Added server 'prod_mysql03' to 'cluster-monitor'
 
Added server 'prod_mysql03' to 'rwsplit-service'
 
 
 
[root@maxscale maxscale_configs]# maxadmin list servers
 
Servers.
 
-------------------+-----------------+-------+-------------+--------------------
 
Server             | Address         | Port  | Connections | Status
 
-------------------+-----------------+-------+-------------+--------------------
 
prod_mysql03       | 192.168.50.13   |  3306 |           0 | Slave, Running
 
prod_mysql02       | 192.168.50.12   |  3306 |           0 | Slave, Running
 
prod_mysql01       | 192.168.50.11   |  3306 |           0 | Master, Running
 
-------------------+-----------------+-------+-------------+--------------------
 
 
 
[root@maxscale maxscale_configs]# ps aux | grep maxscale
 
root      4146  0.1  1.7 358152 10936 ?        Ssl  15:18   0:18 maxscale -f maxscale_configs/basic.cnf
 
root      4489  0.0  0.1 112644   956 pts/2    R+   19:14   0:00 grep --color=auto maxscale
 
 
 
[root@maxscale maxscale_configs]# kill 4146
 
Shutting down MaxScale
 
 
 
[root@maxscale maxscale_configs]# maxscale -f basic.cnf
 
* Error: Failed to open, read or process the MaxScale configuration file. Exiting. See the error log for details.

The logs prior the shutdown was like below:

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Oct 10 19:11:17 2017
 
----------------------------------------------------------------------------
 
2017-10-10 19:11:17   notice : Working directory: /var/log/maxscale
 
2017-10-10 19:11:17   notice : MariaDB MaxScale 2.1.9 started
 
2017-10-10 19:11:17   notice : MaxScale is running in process 4464
 
2017-10-10 19:11:17   notice : Configuration file: /root/maxscale_configs/basic.cnf
 
2017-10-10 19:11:17   notice : Log directory: /var/log/maxscale
 
2017-10-10 19:11:17   notice : Data directory: /var/lib/maxscale
 
2017-10-10 19:11:17   notice : Module directory: /usr/lib64/maxscale
 
2017-10-10 19:11:17   notice : Service cache: /var/cache/maxscale
 
2017-10-10 19:11:17   notice : Loading /root/maxscale_configs/basic.cnf.
 
2017-10-10 19:11:17   notice : /root/maxscale_configs/basic.cnf.d does not exist, not reading.
 
2017-10-10 19:11:17   notice : [cli] Initialise CLI router module
 
2017-10-10 19:11:17   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
 
2017-10-10 19:11:17   notice : [readwritesplit] Initializing statement-based read/write split router module.
 
2017-10-10 19:11:17   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
 
2017-10-10 19:11:17   notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so
 
2017-10-10 19:11:17   notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libMaxAdminAuth.so
 
2017-10-10 19:11:17   notice : No query classifier specified, using default 'qc_sqlite'.
 
2017-10-10 19:11:17   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
 
2017-10-10 19:11:17   warning: Service 'rwsplit-service' has no listeners defined.
 
2017-10-10 19:11:17   notice : Listening for connections at [/tmp/maxadmin.sock]:0 with protocol MaxScale Admin
 
2017-10-10 19:11:17   notice : MaxScale started with 1 server threads.
 
2017-10-10 19:11:17   notice : Started MaxScale log flusher.
 
2017-10-10 19:11:41   notice : (2) [mysqlmon] Initialise the MySQL Monitor module.
 
2017-10-10 19:11:41   notice : (2) Loaded module mysqlmon: V1.5.0 from /usr/lib64/maxscale/libmysqlmon.so
 
2017-10-10 19:11:41   notice : (2) Created monitor 'cluster-monitor'
 
2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': user=maxuser
 
2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': password=FD08F35D87C442AABA518CFB7471D691
 
2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': monitor_interval=1000
 
2017-10-10 19:11:59   error  : [mysqlmon] No Master can be determined
 
2017-10-10 19:12:08   notice : (5) Loaded module MySQLClient: V1.1.0 from /usr/lib64/maxscale/libMySQLClient.so
 
2017-10-10 19:12:08   notice : (5) Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libMySQLAuth.so
 
2017-10-10 19:12:08   notice : (5) Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
 
2017-10-10 19:12:08   error  : (5) [MySQLAuth] Unable to get user data from backend database for service [rwsplit-service]. Failed to connect to any of the backend databases.
 
2017-10-10 19:12:08   error  : (5) [MySQLAuth] [rwsplit-service] Unable to load users for listener rwsplit-listener listening at [0.0.0.0]:4006.
 
2017-10-10 19:13:04   error  : (9) [MySQLAuth] [rwsplit-service] Unable to load users for listener rwsplit-listener listening at [0.0.0.0]:4007.
 
2017-10-10 19:13:04   notice : (9) Listening for connections at [0.0.0.0]:4007 with protocol MySQL
 
2017-10-10 19:13:04   notice : (9) Created listener 'rwsplit-listener' at 0.0.0.0:4007 for service 'rwsplit-service'
 
2017-10-10 19:13:44   notice : (12) Loaded module MySQLBackend: V2.0.0 from /usr/lib64/maxscale/libMySQLBackend.so
 
2017-10-10 19:13:44   notice : (12) Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so
 
2017-10-10 19:13:44   notice : (12) Created server 'prod_mysql01' at 192.168.50.11:3306
 
2017-10-10 19:13:49   notice : (13) Created server 'prod_mysql02' at 192.168.50.12:3306
 
2017-10-10 19:13:54   notice : (14) Created server 'prod_mysql03' at 192.168.50.13:3306
 
^@2017-10-10 19:14:16   notice : (16) Added server 'prod_mysql02' to monitor 'cluster-monitor'
 
2017-10-10 19:14:16   notice : (16) Added server 'prod_mysql02' to service 'rwsplit-service'
 
2017-10-10 19:14:16   notice : Server changed state: prod_mysql02[192.168.50.12:3306]: new_master. [Running] -> [Master, Running]
 
2017-10-10 19:14:16   notice : [mysqlmon] A Master Server is now available: 192.168.50.12:3306
 
2017-10-10 19:14:22   notice : (17) Added server 'prod_mysql01' to monitor 'cluster-monitor'
 
2017-10-10 19:14:22   notice : (17) Added server 'prod_mysql01' to service 'rwsplit-service'
 
2017-10-10 19:14:22   notice : Server changed state: prod_mysql02[192.168.50.12:3306]: new_slave. [Master, Running] -> [Slave, Running]
 
2017-10-10 19:14:22   notice : Server changed state: prod_mysql01[192.168.50.11:3306]: new_master. [Running] -> [Master, Running]
 
2017-10-10 19:14:22   notice : [mysqlmon] A Master Server is now available: 192.168.50.11:3306
 
2017-10-10 19:14:30   notice : (18) Added server 'prod_mysql03' to monitor 'cluster-monitor'
 
2017-10-10 19:14:30   notice : (18) Added server 'prod_mysql03' to service 'rwsplit-service'
 
2017-10-10 19:14:30   notice : Server changed state: prod_mysql03[192.168.50.13:3306]: new_slave. [Running] -> [Slave, Running]
 
^@2017-10-10 19:15:14   notice : Waiting for housekeeper to shut down.
 
2017-10-10 19:15:15   notice : Housekeeper shutting down.
 
2017-10-10 19:15:15   notice : Housekeeper has shut down.
 
2017-10-10 19:15:15   notice : Finished MaxScale log flusher.
 
2017-10-10 19:15:15   notice : MaxScale received signal SIGTERM. Exiting.
 
2017-10-10 19:15:15   notice : MaxScale is shutting down.
 
2017-10-10 19:15:15   notice : MaxScale shutdown completed.
 
2017-10-10 19:15:15   MariaDB MaxScale is shut down.
 
----------------------------------------------------

The logs after the kill <pid> is like:

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Oct 10 19:15:26 2017
 
----------------------------------------------------------------------------
 
2017-10-10 19:15:26   notice : Working directory: /var/log/maxscale
 
2017-10-10 19:15:26   notice : MariaDB MaxScale 2.1.9 started
 
2017-10-10 19:15:26   notice : MaxScale is running in process 4491
 
2017-10-10 19:15:26   notice : Configuration file: /root/maxscale_configs/basic.cnf
 
2017-10-10 19:15:26   notice : Log directory: /var/log/maxscale
 
2017-10-10 19:15:26   notice : Data directory: /var/lib/maxscale
 
2017-10-10 19:15:26   notice : Module directory: /usr/lib64/maxscale
 
2017-10-10 19:15:26   notice : Service cache: /var/cache/maxscale
 
2017-10-10 19:15:26   notice : Loading /root/maxscale_configs/basic.cnf.
 
2017-10-10 19:15:26   notice : /root/maxscale_configs/basic.cnf.d does not exist, not reading.
 
2017-10-10 19:15:26   notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d'
 
2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/cluster-monitor.cnf.
 
2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/rwsplit-listener.cnf.
 
2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql01.cnf.
 
2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql02.cnf.
 
2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql03.cnf.
 
2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/rwsplit-service.cnf.
 
2017-10-10 19:15:26   notice : [cli] Initialise CLI router module
 
2017-10-10 19:15:26   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
 
2017-10-10 19:15:26   notice : [readwritesplit] Initializing statement-based read/write split router module.
 
2017-10-10 19:15:26   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
 
2017-10-10 19:15:26   notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so
 
2017-10-10 19:15:26   error  : Configuration object 'cluster-monitor' has no type.
 
2017-10-10 19:15:26   error  : 1 errors were encountered while processing the configuration file '/root/maxscale_configs/basic.cnf'.
 
2017-10-10 19:15:26   warning: One or more generated configurations were found at '/var/lib/maxscale/maxscale.cnf.d'. If the error relates to any of the files located there, remove the offending configurations from this directory.
 
2017-10-10 19:15:26   error  : Failed to open, read or process the MaxScale configuration file /root/maxscale_configs/basic.cnf. Exiting.
 
2017-10-10 19:15:26   MariaDB MaxScale is shut down.
 
----------------------------------------------------

When I first seen this error, I resolved to tweak the files at /var/lib/maxscale/maxscale.cnf.d/

[root@maxscale maxscale_configs]# cd /var/lib/maxscale/maxscale.cnf.d/
 
[root@maxscale maxscale.cnf.d]# ls -lh
 
total 24K
 
-rw-r--r-- 1 root root  65 Oct 10 19:14 cluster-monitor.cnf
 
-rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql01.cnf
 
-rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql02.cnf
 
-rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql03.cnf
 
-rw-r--r-- 1 root root 128 Oct 10 19:13 rwsplit-listener.cnf
 
-rw-r--r-- 1 root root  65 Oct 10 19:14 rwsplit-service.cnf
 
 
 
[root@maxscale maxscale.cnf.d]# cat cluster-monitor.cnf
 
[cluster-monitor]
 
servers=prod_mysql02,prod_mysql01,prod_mysql03

Really, the file defining the cluster-monitor, in this example, using the MySQLMon, does not have a type definition. The command accepts the type=monitor, but, that type issued together with the command does not go to the file:

maxadmin alter monitor cluster-monitor type=monitor user=maxuser password=FD08F35D87C442AABA518CFB7471D691 monitor_interval=1000

If I add type=monitor manually, the log will show me another missing stuff:

2017-10-10 19:34:39   error  : Monitor 'cluster-monitor' is missing the required 'module' parameter.

Not sure if it all is covered by the command to create the monitor:

MaxScale> help create monitor
 
create monitor - Create a new monitor
 
 
 
Usage: create monitor NAME MODULE
 
 
 
Parameters:
 
NAME    Monitor name
 
MODULE  Monitor module
 
 
 
Example: create monitor my-monitor mysqlmon

So, you may ask after all this, how to reproduce the case?

1. Setup 3 vms, 1 maxscale, 1 master, 1 slave
2. Configure replication on master and slave for the sake of the readwritesplit router
3. On maxscale, start it with the basic configuration I put together above
4. Repeat commands as I put together or following https://mariadb.com/resources/blog/making-maxscale-dynamic-new-mariadb-maxscale-21
5. check logs on every action

 Comments   
Comment by Wagner Bianchi (Inactive) [ 2017-10-10 ]

If I tweak more and more the files, more stuff will appearing as you go through:

2017-10-10 18:54:49   error  : No router defined for service 'rwsplit-service'.
 
2017-10-10 18:54:49   error  : Configuration object 'cluster-monitor' has no type
 
2017-10-10 18:56:25   error  : Service 'rwsplit-service' is missing the 'user' parameter and the 'password' or 'passwd' parameter.

Looking further to another template I've got here with me for readwritesplit, I'm seeing that the readwritesplit-service create by the dynamic command is missing a lot of things in comparison to what you can see below:

[readwritesplit_router]
 
type=service
 
router=readwritesplit
 
servers=box01,box02,box03
 
user=maxuser
 
passwd=FD08F35D87C442AABA518CFB7471D691

Comment by markus makela [ 2017-10-11 ]

Adding new servers to a monitor that was created at runtime would overwrite the base configuration with only the server configuration.

Comment by Wagner Bianchi (Inactive) [ 2017-10-15 ]

Markus,

I am seeing this bug as fixed, wasn't it a bug? If it's a bug, it';s going to be release on the next 2.1 release?

Cheers!

Comment by Wagner Bianchi (Inactive) [ 2017-10-17 ]

Ok, 2.1.10, good, thanks Markus!

Generated at Thu Feb 08 04:06:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.