Uploaded image for project: 'MaxScale'
  1. MaxScale
  2. MXS-1468

Using dynamic commands to create readwritesplit configs fail after restart

    Details

      Description

      Hello Folks,

      Following up with a setup of Maxscale 2.1.9 when a ReadWriteSplit is on the radar to be implemented, I started Maxscale on the box with a very basic configuration file with the intention to create everything dynamically and in separated files which are located at /var/lib/maxscale/maxscale.cnf.d/, making it way more organized then having just one giant maxscale.cnf file.

      I followed the below:

      #: basic configuration file
      [root@maxscale maxscale_configs]# cat basic.cnf
      [maxscale]
      threads=auto
       
      [rwsplit-service]
      type=service
      router=readwritesplit
      user=maxuser
      passwd=FD08F35D87C442AABA518CFB7471D691
       
      [CLI]
      type=service
      router=cli
       
      [CLI Listener]
      type=listener
      service=CLI
      protocol=maxscaled
      socket=default
      

      And then, I started maxscale, entered the following commands, killed its process and start again with the basic configuration, which should give me a maxscale with basic configurations + what I created on the command line:

      [root@maxscale maxscale_configs]# maxscale -f basic.cnf
       
      [root@maxscale maxscale_configs]# maxadmin create monitor cluster-monitor mysqlmon
      Created monitor 'cluster-monitor'
       
      [root@maxscale maxscale_configs]# maxadmin alter monitor cluster-monitor user=maxuser password=FD08F35D87C442AABA518CFB7471D691 monitor_interval=1000
       
      [root@maxscale maxscale_configs]# maxadmin restart monitor cluster-monitor
       
      [root@maxscale maxscale_configs]# maxadmin create listener rwsplit-service rwsplit-listener 0.0.0.0 4007
      Listener 'rwsplit-listener' created
       
      [root@maxscale maxscale_configs]# maxadmin list listeners
      Listeners.
      ---------------------+---------------------+--------------------+-----------------+-------+--------
      Name                 | Service Name        | Protocol Module    | Address         | Port  | State
      ---------------------+---------------------+--------------------+-----------------+-------+--------
      rwsplit-listener     | rwsplit-service     | MySQLClient        | 0.0.0.0         |  4007 | Running
      CLI Listener         | CLI                 | maxscaled          | default         |     0 | Running
      ---------------------+---------------------+--------------------+-----------------+-------+--------
       
      [root@maxscale maxscale_configs]# maxadmin create server prod_mysql01 192.168.50.11 3306
      Created server 'prod_mysql01'
      [root@maxscale maxscale_configs]# maxadmin create server prod_mysql02 192.168.50.12 3306
      Created server 'prod_mysql02'
      [root@maxscale maxscale_configs]# maxadmin create server prod_mysql03 192.168.50.13 3306
      Created server 'prod_mysql03'
       
      [root@maxscale maxscale_configs]# maxadmin list servers
      Servers.
      -------------------+-----------------+-------+-------------+--------------------
      Server             | Address         | Port  | Connections | Status
      -------------------+-----------------+-------+-------------+--------------------
      prod_mysql03       | 192.168.50.13   |  3306 |           0 | Running
      prod_mysql02       | 192.168.50.12   |  3306 |           0 | Running
      prod_mysql01       | 192.168.50.11   |  3306 |           0 | Running
      -------------------+-----------------+-------+-------------+--------------------
       
      [root@maxscale maxscale_configs]# maxadmin add server prod_mysql02 cluster-monitor rwsplit-service
      Added server 'prod_mysql02' to 'cluster-monitor'
      Added server 'prod_mysql02' to 'rwsplit-service'
       
      [root@maxscale maxscale_configs]# maxadmin add server prod_mysql01 cluster-monitor rwsplit-service
      Added server 'prod_mysql01' to 'cluster-monitor'
      Added server 'prod_mysql01' to 'rwsplit-service'
       
      [root@maxscale maxscale_configs]# maxadmin add server prod_mysql03 cluster-monitor rwsplit-service
      Added server 'prod_mysql03' to 'cluster-monitor'
      Added server 'prod_mysql03' to 'rwsplit-service'
       
      [root@maxscale maxscale_configs]# maxadmin list servers
      Servers.
      -------------------+-----------------+-------+-------------+--------------------
      Server             | Address         | Port  | Connections | Status
      -------------------+-----------------+-------+-------------+--------------------
      prod_mysql03       | 192.168.50.13   |  3306 |           0 | Slave, Running
      prod_mysql02       | 192.168.50.12   |  3306 |           0 | Slave, Running
      prod_mysql01       | 192.168.50.11   |  3306 |           0 | Master, Running
      -------------------+-----------------+-------+-------------+--------------------
       
      [root@maxscale maxscale_configs]# ps aux | grep maxscale
      root      4146  0.1  1.7 358152 10936 ?        Ssl  15:18   0:18 maxscale -f maxscale_configs/basic.cnf
      root      4489  0.0  0.1 112644   956 pts/2    R+   19:14   0:00 grep --color=auto maxscale
       
      [root@maxscale maxscale_configs]# kill 4146
      Shutting down MaxScale
       
      [root@maxscale maxscale_configs]# maxscale -f basic.cnf
      * Error: Failed to open, read or process the MaxScale configuration file. Exiting. See the error log for details.
      

      The logs prior the shutdown was like below:

      MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Oct 10 19:11:17 2017
      ----------------------------------------------------------------------------
      2017-10-10 19:11:17   notice : Working directory: /var/log/maxscale
      2017-10-10 19:11:17   notice : MariaDB MaxScale 2.1.9 started
      2017-10-10 19:11:17   notice : MaxScale is running in process 4464
      2017-10-10 19:11:17   notice : Configuration file: /root/maxscale_configs/basic.cnf
      2017-10-10 19:11:17   notice : Log directory: /var/log/maxscale
      2017-10-10 19:11:17   notice : Data directory: /var/lib/maxscale
      2017-10-10 19:11:17   notice : Module directory: /usr/lib64/maxscale
      2017-10-10 19:11:17   notice : Service cache: /var/cache/maxscale
      2017-10-10 19:11:17   notice : Loading /root/maxscale_configs/basic.cnf.
      2017-10-10 19:11:17   notice : /root/maxscale_configs/basic.cnf.d does not exist, not reading.
      2017-10-10 19:11:17   notice : [cli] Initialise CLI router module
      2017-10-10 19:11:17   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
      2017-10-10 19:11:17   notice : [readwritesplit] Initializing statement-based read/write split router module.
      2017-10-10 19:11:17   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
      2017-10-10 19:11:17   notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so
      2017-10-10 19:11:17   notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libMaxAdminAuth.so
      2017-10-10 19:11:17   notice : No query classifier specified, using default 'qc_sqlite'.
      2017-10-10 19:11:17   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
      2017-10-10 19:11:17   warning: Service 'rwsplit-service' has no listeners defined.
      2017-10-10 19:11:17   notice : Listening for connections at [/tmp/maxadmin.sock]:0 with protocol MaxScale Admin
      2017-10-10 19:11:17   notice : MaxScale started with 1 server threads.
      2017-10-10 19:11:17   notice : Started MaxScale log flusher.
      2017-10-10 19:11:41   notice : (2) [mysqlmon] Initialise the MySQL Monitor module.
      2017-10-10 19:11:41   notice : (2) Loaded module mysqlmon: V1.5.0 from /usr/lib64/maxscale/libmysqlmon.so
      2017-10-10 19:11:41   notice : (2) Created monitor 'cluster-monitor'
      2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': user=maxuser
      2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': password=FD08F35D87C442AABA518CFB7471D691
      2017-10-10 19:11:52   notice : (3) Updated monitor 'cluster-monitor': monitor_interval=1000
      2017-10-10 19:11:59   error  : [mysqlmon] No Master can be determined
      2017-10-10 19:12:08   notice : (5) Loaded module MySQLClient: V1.1.0 from /usr/lib64/maxscale/libMySQLClient.so
      2017-10-10 19:12:08   notice : (5) Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libMySQLAuth.so
      2017-10-10 19:12:08   notice : (5) Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
      2017-10-10 19:12:08   error  : (5) [MySQLAuth] Unable to get user data from backend database for service [rwsplit-service]. Failed to connect to any of the backend databases.
      2017-10-10 19:12:08   error  : (5) [MySQLAuth] [rwsplit-service] Unable to load users for listener rwsplit-listener listening at [0.0.0.0]:4006.
      2017-10-10 19:13:04   error  : (9) [MySQLAuth] [rwsplit-service] Unable to load users for listener rwsplit-listener listening at [0.0.0.0]:4007.
      2017-10-10 19:13:04   notice : (9) Listening for connections at [0.0.0.0]:4007 with protocol MySQL
      2017-10-10 19:13:04   notice : (9) Created listener 'rwsplit-listener' at 0.0.0.0:4007 for service 'rwsplit-service'
      2017-10-10 19:13:44   notice : (12) Loaded module MySQLBackend: V2.0.0 from /usr/lib64/maxscale/libMySQLBackend.so
      2017-10-10 19:13:44   notice : (12) Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so
      2017-10-10 19:13:44   notice : (12) Created server 'prod_mysql01' at 192.168.50.11:3306
      2017-10-10 19:13:49   notice : (13) Created server 'prod_mysql02' at 192.168.50.12:3306
      2017-10-10 19:13:54   notice : (14) Created server 'prod_mysql03' at 192.168.50.13:3306
      ^@2017-10-10 19:14:16   notice : (16) Added server 'prod_mysql02' to monitor 'cluster-monitor'
      2017-10-10 19:14:16   notice : (16) Added server 'prod_mysql02' to service 'rwsplit-service'
      2017-10-10 19:14:16   notice : Server changed state: prod_mysql02[192.168.50.12:3306]: new_master. [Running] -> [Master, Running]
      2017-10-10 19:14:16   notice : [mysqlmon] A Master Server is now available: 192.168.50.12:3306
      2017-10-10 19:14:22   notice : (17) Added server 'prod_mysql01' to monitor 'cluster-monitor'
      2017-10-10 19:14:22   notice : (17) Added server 'prod_mysql01' to service 'rwsplit-service'
      2017-10-10 19:14:22   notice : Server changed state: prod_mysql02[192.168.50.12:3306]: new_slave. [Master, Running] -> [Slave, Running]
      2017-10-10 19:14:22   notice : Server changed state: prod_mysql01[192.168.50.11:3306]: new_master. [Running] -> [Master, Running]
      2017-10-10 19:14:22   notice : [mysqlmon] A Master Server is now available: 192.168.50.11:3306
      2017-10-10 19:14:30   notice : (18) Added server 'prod_mysql03' to monitor 'cluster-monitor'
      2017-10-10 19:14:30   notice : (18) Added server 'prod_mysql03' to service 'rwsplit-service'
      2017-10-10 19:14:30   notice : Server changed state: prod_mysql03[192.168.50.13:3306]: new_slave. [Running] -> [Slave, Running]
      ^@2017-10-10 19:15:14   notice : Waiting for housekeeper to shut down.
      2017-10-10 19:15:15   notice : Housekeeper shutting down.
      2017-10-10 19:15:15   notice : Housekeeper has shut down.
      2017-10-10 19:15:15   notice : Finished MaxScale log flusher.
      2017-10-10 19:15:15   notice : MaxScale received signal SIGTERM. Exiting.
      2017-10-10 19:15:15   notice : MaxScale is shutting down.
      2017-10-10 19:15:15   notice : MaxScale shutdown completed.
      2017-10-10 19:15:15   MariaDB MaxScale is shut down.
      ----------------------------------------------------
      

      The logs after the kill <pid> is like:

      MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Oct 10 19:15:26 2017
      ----------------------------------------------------------------------------
      2017-10-10 19:15:26   notice : Working directory: /var/log/maxscale
      2017-10-10 19:15:26   notice : MariaDB MaxScale 2.1.9 started
      2017-10-10 19:15:26   notice : MaxScale is running in process 4491
      2017-10-10 19:15:26   notice : Configuration file: /root/maxscale_configs/basic.cnf
      2017-10-10 19:15:26   notice : Log directory: /var/log/maxscale
      2017-10-10 19:15:26   notice : Data directory: /var/lib/maxscale
      2017-10-10 19:15:26   notice : Module directory: /usr/lib64/maxscale
      2017-10-10 19:15:26   notice : Service cache: /var/cache/maxscale
      2017-10-10 19:15:26   notice : Loading /root/maxscale_configs/basic.cnf.
      2017-10-10 19:15:26   notice : /root/maxscale_configs/basic.cnf.d does not exist, not reading.
      2017-10-10 19:15:26   notice : Loading generated configuration files from '/var/lib/maxscale/maxscale.cnf.d'
      2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/cluster-monitor.cnf.
      2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/rwsplit-listener.cnf.
      2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql01.cnf.
      2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql02.cnf.
      2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/prod_mysql03.cnf.
      2017-10-10 19:15:26   notice : Loading /var/lib/maxscale/maxscale.cnf.d/rwsplit-service.cnf.
      2017-10-10 19:15:26   notice : [cli] Initialise CLI router module
      2017-10-10 19:15:26   notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
      2017-10-10 19:15:26   notice : [readwritesplit] Initializing statement-based read/write split router module.
      2017-10-10 19:15:26   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
      2017-10-10 19:15:26   notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so
      2017-10-10 19:15:26   error  : Configuration object 'cluster-monitor' has no type.
      2017-10-10 19:15:26   error  : 1 errors were encountered while processing the configuration file '/root/maxscale_configs/basic.cnf'.
      2017-10-10 19:15:26   warning: One or more generated configurations were found at '/var/lib/maxscale/maxscale.cnf.d'. If the error relates to any of the files located there, remove the offending configurations from this directory.
      2017-10-10 19:15:26   error  : Failed to open, read or process the MaxScale configuration file /root/maxscale_configs/basic.cnf. Exiting.
      2017-10-10 19:15:26   MariaDB MaxScale is shut down.
      ----------------------------------------------------
      

      When I first seen this error, I resolved to tweak the files at /var/lib/maxscale/maxscale.cnf.d/

      [root@maxscale maxscale_configs]# cd /var/lib/maxscale/maxscale.cnf.d/
      [root@maxscale maxscale.cnf.d]# ls -lh
      total 24K
      -rw-r--r-- 1 root root  65 Oct 10 19:14 cluster-monitor.cnf
      -rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql01.cnf
      -rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql02.cnf
      -rw-r--r-- 1 root root 112 Oct 10 19:13 prod_mysql03.cnf
      -rw-r--r-- 1 root root 128 Oct 10 19:13 rwsplit-listener.cnf
      -rw-r--r-- 1 root root  65 Oct 10 19:14 rwsplit-service.cnf
       
      [root@maxscale maxscale.cnf.d]# cat cluster-monitor.cnf
      [cluster-monitor]
      servers=prod_mysql02,prod_mysql01,prod_mysql03
      

      Really, the file defining the cluster-monitor, in this example, using the MySQLMon, does not have a type definition. The command accepts the type=monitor, but, that type issued together with the command does not go to the file:

      maxadmin alter monitor cluster-monitor type=monitor user=maxuser password=FD08F35D87C442AABA518CFB7471D691 monitor_interval=1000
      

      If I add type=monitor manually, the log will show me another missing stuff:

      2017-10-10 19:34:39   error  : Monitor 'cluster-monitor' is missing the required 'module' parameter.
      

      Not sure if it all is covered by the command to create the monitor:

      MaxScale> help create monitor
      create monitor - Create a new monitor
       
      Usage: create monitor NAME MODULE
       
      Parameters:
      NAME    Monitor name
      MODULE  Monitor module
       
      Example: create monitor my-monitor mysqlmon
      

      So, you may ask after all this, how to reproduce the case?

      1. Setup 3 vms, 1 maxscale, 1 master, 1 slave
      2. Configure replication on master and slave for the sake of the readwritesplit router
      3. On maxscale, start it with the basic configuration I put together above
      4. Repeat commands as I put together or following https://mariadb.com/resources/blog/making-maxscale-dynamic-new-mariadb-maxscale-21
      5. check logs on every action

        Attachments

          Activity

            People

            • Assignee:
              markus makela markus makela
              Reporter:
              wagnerbianchi Wagner Bianchi
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: