[MXS-1468] Using dynamic commands to create readwritesplit configs fail after restart - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 2.1.9
Fix Version/s: 2.1.10
Component/s: mariadbmon, maxadmin, readconnroute, readwritesplit
Labels:
None
Environment:
Linux, running CentOS 7.2
3.10.0-327.4.5.el7.x86_64

Description

Hello Folks,

Following up with a setup of Maxscale 2.1.9 when a ReadWriteSplit is on the radar to be implemented, I started Maxscale on the box with a very basic configuration file with the intention to create everything dynamically and in separated files which are located at /var/lib/maxscale/maxscale.cnf.d/, making it way more organized then having just one giant maxscale.cnf file.

I followed the below:

#: basic configuration file

[root@maxscale maxscale_configs]# cat basic.cnf

[maxscale]

threads=auto

[rwsplit-service]

type=service

router=readwritesplit

user=maxuser

passwd=FD08F35D87C442AABA518CFB7471D691

[CLI]

type=service

router=cli

[CLI Listener]

type=listener

service=CLI

protocol=maxscaled

socket=default

And then, I started maxscale, entered the following commands, killed its process and start again with the basic configuration, which should give me a maxscale with basic configurations + what I created on the command line:

[root@maxscale maxscale_configs]# maxscale -f basic.cnf

[root@maxscale maxscale_configs]# maxadmin create monitor cluster-monitor mysqlmon

Created monitor 'cluster-monitor'

[root@maxscale maxscale_configs]# maxadmin alter monitor cluster-monitor user=maxuser password=FD08F35D87C442AABA518CFB7471D691 monitor_interval=1000

[root@maxscale maxscale_configs]# maxadmin restart monitor cluster-monitor

[root@maxscale maxscale_configs]# maxadmin create listener rwsplit-service rwsplit-listener 0.0.0.0 4007

Listener 'rwsplit-listener' created

[root@maxscale maxscale_configs]# maxadmin list listeners

Listeners.

---------------------+---------------------+--------------------+-----------------+-------+--------

Name                 | Service Name        | Protocol Module    | Address         | Port  | State

---------------------+---------------------+--------------------+-----------------+-------+--------

rwsplit-listener     | rwsplit-service     | MySQLClient        | 0.0.0.0         |  4007 | Running

CLI Listener         | CLI                 | maxscaled          | default         |     0 | Running

---------------------+---------------------+--------------------+-----------------+-------+--------

[root@maxscale maxscale_configs]# maxadmin create server prod_mysql01 192.168.50.11 3306

Created server 'prod_mysql01'

[root@maxscale maxscale_configs]# maxadmin create server prod_mysql02 192.168.50.12 3306

Created server 'prod_mysql02'

[root@maxscale maxscale_configs]# maxadmin create server prod_mysql03 192.168.50.13 3306

Created server 'prod_mysql03'

[root@maxscale maxscale_configs]# maxadmin list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status

-------------------+-----------------+-------+-------------+--------------------

prod_mysql03       | 192.168.50.13   |  3306 |           0 | Running

prod_mysql02       | 192.168.50.12   |  3306 |           0 | Running

prod_mysql01       | 192.168.50.11   |  3306 |           0 | Running

-------------------+-----------------+-------+-------------+--------------------

[root@maxscale maxscale_configs]# maxadmin add server prod_mysql02 cluster-monitor rwsplit-service

Added server 'prod_mysql02' to 'cluster-monitor'

Added server 'prod_mysql02' to 'rwsplit-service'

[root@maxscale maxscale_configs]# maxadmin add server prod_mysql01 cluster-monitor rwsplit-service

Added server 'prod_mysql01' to 'cluster-monitor'

Added server 'prod_mysql01' to 'rwsplit-service'

[root@maxscale maxscale_configs]# maxadmin add server prod_mysql03 cluster-monitor rwsplit-service

Added server 'prod_mysql03' to 'cluster-monitor'

Added server 'prod_mysql03' to 'rwsplit-service'

[root@maxscale maxscale_configs]# maxadmin list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status

-------------------+-----------------+-------+-------------+--------------------

prod_mysql03       | 192.168.50.13   |  3306 |           0 | Slave, Running

prod_mysql02       | 192.168.50.12   |  3306 |           0 | Slave, Running

prod_mysql01       | 192.168.50.11   |  3306 |           0 | Master, Running

-------------------+-----------------+-------+-------------+--------------------

[root@maxscale maxscale_configs]# ps aux | grep maxscale

root      4146  0.1  1.7 358152 10936 ?        Ssl  15:18   0:18 maxscale -f maxscale_configs/basic.cnf

root      4489  0.0  0.1 112644   956 pts/2    R+   19:14   0:00 grep --color=auto maxscale

[root@maxscale maxscale_configs]# kill 4146

Shutting down MaxScale

[root@maxscale maxscale_configs]# maxscale -f basic.cnf

* Error: Failed to open, read or process the MaxScale configuration file. Exiting. See the error log for details.

The logs prior the shutdown was like below:

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Oct 10 19:11:17 2017

----------------------------------------------------------------------------

2017-10-10 19:11:17   notice : Working directory: /var/log/maxscale

2017-10-10 19:11:17   notice : MariaDB MaxScale 2.1.9 started

2017-10-10 19:11:17   notice : MaxScale is running in process 4464

2017-10-10 19:11:17   notice : Configuration file: /root/maxscale_configs/basic.cnf

2017-10-10 19:11:17   notice : Log directory: /var/log/maxscale

2017-10-10 19:11:17   notice : Data directory: /var/lib/maxscale

2017-10-10 19:11:17   notice : Module directory: /usr/lib64/maxscale

2017-10-10 19:11:17   notice : Service cache: /var/cache/maxscale

2017-10-10 19:11:17   notice : Loading /root/maxscale_configs/basic.cnf.

2017-10-10 19:11:17   notice : /root/maxscale_configs/basic.cnf.d does not exist, not reading.

2017-10-10 19:11:17   notice : [cli] Initialise CLI router module

2017-10-10 19:11:17   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so

2017-10-10 19:11:17   notice : [readwritesplit] Initializing statement-based read/write split router module.

2017-10-10 19:11:17   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so

2017-10-10 19:11:17   notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so

2017-10-10 19:11:17   notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libMaxAdminAuth.so

2017-10-10 19:11:17   notice : No query classifier specified, using default 'qc_sqlite'.

2017-10-10 19:11:17   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so

2017-10-10 19:11:17   warning: Service 'rwsplit-service' has no listeners defined.

2017-10-10 19:11:17   notice : Listening for connections at [/tmp/maxadmin.sock]:0 with protocol MaxScale Admin

2017-10-10 19:11:17   notice : MaxScale started with 1 server threads.

2017-10-10 19:11:17   notice : Started MaxScale log flusher.

2017-10-10 19:11:41   notice : (2) [mysqlmon] Initialise the MySQL Monitor module.

2017-10-10 19:11:41   notice : (2) Loaded module mysqlmon: V1.5.0 from /usr/lib64/maxscale/libmysqlmon.so

2017-10-10 19:11:41   notice : (2) Created monitor 'cluster-monitor'

2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': user=maxuser

2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': password=FD08F35D87C442AABA518CFB7471D691

2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': monitor_interval=1000

2017-10-10 19:11:59   error  : [mysqlmon] No Master can be determined

2017-10-10 19:12:08   notice : (5) Loaded module MySQLClient: V1.1.0 from /usr/lib64/maxscale/libMySQLClient.so

2017-10-10 19:12:08   notice : (5) Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libMySQLAuth.so

2017-10-10 19:12:08   notice : (5) Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.

2017-10-10 19:12:08   error  : (5) [MySQLAuth] Unable to get user data from backend database for service [rwsplit-service]. Failed to connect to any of the backend databases.

2017-10-10 19:12:08   error  : (5) [MySQLAuth] [rwsplit-service] Unable to load users for listener rwsplit-listener listening at [0.0.0.0]:4006.

2017-10-10 19:13:04   error  : (9) [MySQLAuth] [rwsplit-service] Unable to load users for listener rwsplit-listener listening at [0.0.0.0]:4007.

2017-10-10 19:13:04   notice : (9) Listening for connections at [0.0.0.0]:4007 with protocol MySQL

2017-10-10 19:13:04   notice : (9) Created listener 'rwsplit-listener' at 0.0.0.0:4007 for service 'rwsplit-service'

2017-10-10 19:13:44   notice : (12) Loaded module MySQLBackend: V2.0.0 from /usr/lib64/maxscale/libMySQLBackend.so

2017-10-10 19:13:44   notice : (12) Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so

2017-10-10 19:13:44   notice : (12) Created server 'prod_mysql01' at 192.168.50.11:3306

2017-10-10 19:13:49   notice : (13) Created server 'prod_mysql02' at 192.168.50.12:3306

2017-10-10 19:13:54   notice : (14) Created server 'prod_mysql03' at 192.168.50.13:3306

^@2017-10-10 19:14:16   notice : (16) Added server 'prod_mysql02' to monitor 'cluster-monitor'

2017-10-10 19:14:16   notice : (16) Added server 'prod_mysql02' to service 'rwsplit-service'

2017-10-10 19:14:16   notice : Server changed state: prod_mysql02[192.168.50.12:3306]: new_master. [Running] -> [Master, Running]

2017-10-10 19:14:16   notice : [mysqlmon] A Master Server is now available: 192.168.50.12:3306

2017-10-10 19:14:22   notice : (17) Added server 'prod_mysql01' to monitor 'cluster-monitor'

2017-10-10 19:14:22   notice : (17) Added server 'prod_mysql01' to service 'rwsplit-service'

2017-10-10 19:14:22   notice : Server changed state: prod_mysql02[192.168.50.12:3306]: new_slave. [Master, Running] -> [Slave, Running]

2017-10-10 19:14:22   notice : Server changed state: prod_mysql01[192.168.50.11:3306]: new_master. [Running] -> [Master, Running]

2017-10-10 19:14:22   notice : [mysqlmon] A Master Server is now available: 192.168.50.11:3306

2017-10-10 19:14:30   notice : (18) Added server 'prod_mysql03' to monitor 'cluster-monitor'

2017-10-10 19:14:30   notice : (18) Added server 'prod_mysql03' to service 'rwsplit-service'

2017-10-10 19:14:30   notice : Server changed state: prod_mysql03[192.168.50.13:3306]: new_slave. [Running] -> [Slave, Running]

^@2017-10-10 19:15:14   notice : Waiting for housekeeper to shut down.

2017-10-10 19:15:15   notice : Housekeeper shutting down.

2017-10-10 19:15:15   notice : Housekeeper has shut down.

2017-10-10 19:15:15   notice : Finished MaxScale log flusher.

2017-10-10 19:15:15   notice : MaxScale received signal SIGTERM. Exiting.

2017-10-10 19:15:15   notice : MaxScale is shutting down.

2017-10-10 19:15:15   notice : MaxScale shutdown completed.

2017-10-10 19:15:15   MariaDB MaxScale is shut down.

----------------------------------------------------

The logs after the kill <pid> is like:

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Oct 10 19:15:26 2017

----------------------------------------------------------------------------

2017-10-10 19:15:26   notice : Working directory: /var/log/maxscale

2017-10-10 19:15:26   notice : MariaDB MaxScale 2.1.9 started

2017-10-10 19:15:26   notice : MaxScale is running in process 4491

2017-10-10 19:15:26   notice : Configuration file: /root/maxscale_configs/basic.cnf

2017-10-10 19:15:26   notice : Log directory: /var/log/maxscale

2017-10-10 19:15:26   notice : Data directory: /var/lib/maxscale

2017-10-10 19:15:26   notice : Module directory: /usr/lib64/maxscale

2017-10-10 19:15:26   notice : Service cache: /var/cache/maxscale

2017-10-10 19:15:26   notice : Loading /root/maxscale_configs/basic.cnf.

2017-10-10 19:15:26   notice : /root/maxscale_configs/basic.cnf.d does not exist, not reading.

2017-10-10 19:15:26   notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d'

2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/cluster-monitor.cnf.

2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/rwsplit-listener.cnf.

2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql01.cnf.

2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql02.cnf.

2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql03.cnf.

2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/rwsplit-service.cnf.

2017-10-10 19:15:26   notice : [cli] Initialise CLI router module

2017-10-10 19:15:26   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so

2017-10-10 19:15:26   notice : [readwritesplit] Initializing statement-based read/write split router module.

2017-10-10 19:15:26   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so

2017-10-10 19:15:26   notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so

2017-10-10 19:15:26   error  : Configuration object 'cluster-monitor' has no type.

2017-10-10 19:15:26   error  : 1 errors were encountered while processing the configuration file '/root/maxscale_configs/basic.cnf'.

2017-10-10 19:15:26   warning: One or more generated configurations were found at '/var/lib/maxscale/maxscale.cnf.d'. If the error relates to any of the files located there, remove the offending configurations from this directory.

2017-10-10 19:15:26   error  : Failed to open, read or process the MaxScale configuration file /root/maxscale_configs/basic.cnf. Exiting.

2017-10-10 19:15:26   MariaDB MaxScale is shut down.

----------------------------------------------------

When I first seen this error, I resolved to tweak the files at /var/lib/maxscale/maxscale.cnf.d/

[root@maxscale maxscale_configs]# cd /var/lib/maxscale/maxscale.cnf.d/

[root@maxscale maxscale.cnf.d]# ls -lh

total 24K

-rw-r--r-- 1 root root  65 Oct 10 19:14 cluster-monitor.cnf

-rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql01.cnf

-rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql02.cnf

-rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql03.cnf

-rw-r--r-- 1 root root 128 Oct 10 19:13 rwsplit-listener.cnf

-rw-r--r-- 1 root root  65 Oct 10 19:14 rwsplit-service.cnf

[root@maxscale maxscale.cnf.d]# cat cluster-monitor.cnf

[cluster-monitor]

servers=prod_mysql02,prod_mysql01,prod_mysql03

Really, the file defining the cluster-monitor, in this example, using the MySQLMon, does not have a type definition. The command accepts the type=monitor, but, that type issued together with the command does not go to the file:

maxadmin alter monitor cluster-monitor type=monitor user=maxuser password=FD08F35D87C442AABA518CFB7471D691 monitor_interval=1000

If I add type=monitor manually, the log will show me another missing stuff:

2017-10-10 19:34:39   error  : Monitor 'cluster-monitor' is missing the required 'module' parameter.

Not sure if it all is covered by the command to create the monitor:

MaxScale> help create monitor

create monitor - Create a new monitor

Usage: create monitor NAME MODULE

Parameters:

NAME    Monitor name

MODULE  Monitor module

Example: create monitor my-monitor mysqlmon

So, you may ask after all this, how to reproduce the case?

1. Setup 3 vms, 1 maxscale, 1 master, 1 slave
2. Configure replication on master and slave for the sake of the readwritesplit router
3. On maxscale, start it with the basic configuration I put together above
4. Repeat commands as I put together or following https://mariadb.com/resources/blog/making-maxscale-dynamic-new-mariadb-maxscale-21
5. check logs on every action

Attachments

Activity

People

Assignee:: markus makela

Reporter:: Wagner Bianchi (Inactive)

Votes:: 1 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2017-10-10 18:53

Updated:: 2017-10-17 08:41

Resolved:: 2017-10-14 21:08

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.