Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-1610

Changing masters does not close readwritesplit session immediately

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 2.1.13
    • 2.3.0
    • readwritesplit
    • None
    • ubuntu 14.04

    Description

      With long sessions applications, like zabbix, when master is going down, readwritesplit service correctly changes the backend server. But when master going back, readwritesplit service does not change master back and in log appears messages like:

      error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      

      Example

      Start server:

      2018-01-16 12:05:43   notice : MaxScale started with 4 server threads.
      

      Shuting down master:

      2018-01-16 12:14:23   error  : Monitor was unable to connect to server [MASTER_IP_ADDRESS]:3306 : "Can't connect to MySQL server on 'MASTER_IP_ADDRESS' (107)"
      2018-01-16 12:14:23   notice : Server changed state: MASTER_SERVER[MASTER_IP_ADDRESS:3306]: master_down. [Master, Synced, Running] -> [Down]
      2018-01-16 12:14:23   notice : Server changed state: SLAVE_SERVER[SLAVE_IP_ADDRESS:3306]: new_master. [Slave, Synced, Running] -> [Master, Synced, Running]
      

      Master changed, no errors:

      Start master:

      2018-01-16 12:15:26   notice : Server changed state: MASTER_SERVER[MASTER_IP_ADDRESS:3306]: master_up. [Down] -> [Master, Synced, Running]
      2018-01-16 12:15:26   notice : Server changed state: SLAVE_SERVER[SLAVE_IP_ADDRESS:3306]: new_slave. [Master, Synced, Running] -> [Slave, Synced, Running]
      

      Errors in log appears:

      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:26   error  : (2831) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE (subsequent similar messages suppressed for 10000 milliseconds)
      2018-01-16 12:15:27   warning: (2595) [readwritesplit] [Splitter Service] Write query received from zabbix@::ffff:xxx.xxx.xxx.xxx. Could not find a valid master connection. Closing client connection.
      2018-01-16 12:15:27   error  : (2595) [MySQLClient] Routing the query failed. Session will be closed.
      2018-01-16 12:15:30   warning: (2606) [readwritesplit] [Splitter Service] Write query received from zabbix@::ffff:xxx.xxx.xxx.xxx. Could not find a valid master connection. Closing client connection.
      2018-01-16 12:15:30   error  : (2606) [MySQLClient] Routing the query failed. Session will be closed.
      2018-01-16 12:15:33   warning: (2599) [readwritesplit] [Splitter Service] Write query received from zabbix@::ffff:xxx.xxx.xxx.xxx. Could not find a valid master connection. Closing client connection.
      2018-01-16 12:15:33   error  : (2599) [MySQLClient] Routing the query failed. Session will be closed.
      2018-01-16 12:15:38   error  : (2675) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2573) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2573) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2600) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2600) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2585) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2585) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:39   error  : (2585) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:40   error  : (2605) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:40   error  : (2571) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:40   error  : (2571) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2648) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2626) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2648) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2626) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2648) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2626) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2648) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:41   error  : (2626) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE (subsequent similar messages suppressed for 10000 milliseconds)
      2018-01-16 12:15:52   error  : (2594) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2594) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2571) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2576) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2571) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2576) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2600) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:52   error  : (2600) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      2018-01-16 12:15:53   error  : (2600) [readwritesplit] Could not find master among the backend servers. Previous master's state : RUNNING SLAVE
      

      Аnd this errors continues to appear until the zabbix service reboots.

      maxscale config:

      [maxscale]
      threads=4
       
      [MASTER]
      type=server
      address=xxx.xxx.xxx.xxx
      port=3306
      protocol=MySQLBackend
      priority=1
       
      [SLAVE]
      type=server
      address=xxx.xxx.xxx.xxx
      port=3306
      protocol=MySQLBackend
      priority=2
       
      [SLAVE2]
      type=server
      address=xxx.xxx.xxx.xxx
      port=3306
      protocol=MySQLBackend
      priority=3
       
      [Galera Monitor]
      type=monitor
      module=galeramon
      servers=MASTER,SLAVE,SLAVE2
      user=xxx
      passwd=yyy
      monitor_interval=10000
      available_when_donor=true
      use_priority=1
       
      [Splitter Service]
      type=service
      router=readwritesplit
      servers=MASTER, SLAVE2, SLAVE
      user=xxx
      passwd=yyy
      max_slave_connections=1
      router_options=master_accept_reads=true,slave_selection_criteria=LEAST_GLOBAL_CONNECTIONS
      use_sql_variables_in=master
       
      [Splitter Listener]
      type=listener
      service=Splitter Service
      protocol=MySQLClient
      port=3306
      

      Attachments

        Activity

          People

            markus makela markus makela
            uname Petrov Aleksey
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.