Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2779

Maxscale cant choose a good server as a master after Server changed state

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Not a Bug
    • 2.4.4
    • N/A
    • N/A
    • None
    • CentOS Linux release 7.7.1908 (Core) ,kernel:3.10.0-1062.1.1.el7.x86_64
      Database: Percona XtrDB Cluster 5.7 (5 nodes Virtual Machines)

    Description

      Maxscale was abnormal,after it changed the status of my Master node .
      it could't change Master to the original node,even the node was working normally.
      So I restart the Maxscale service .
      then everything was normal.

      [maxscale]
      threads=auto
       
      # Servers
      [node1]
      type=server
      address=10.1.0.176
      port=3306
      protocol=mariadbbackend
      priority=1
       
      [node2]
      type=server
      address=10.1.0.177
      port=3306
      protocol=mariadbbackend
      priority=4
       
      [node3]
      type=server
      address=10.1.0.178
      port=3306
      protocol=mariadbbackend
      priority=2
       
      [node4]
      type=server
      address=10.1.0.172
      port=3306
      protocol=mariadbbackend
      priority=5
       
      [node5]
      type=server
      address=10.1.0.173
      port=3306
      protocol=mariadbbackend
      priority=3
       
      [backup_server]
      type=server
      address=10.1.0.101
      port=3306
      protocol=mariadbbackend
      # Monitoring for the servers
      [PXC_Monitor]
      type=monitor
      module=galeramon
      servers=node1,node2,node3,node4,node5,backup_server
      user=maxscale_monitor
      password=xxxxxxxxxxxxxxxxxxxxxxxxx
      monitor_interval=3000ms
      use_priority=true
      #disable_master_failback=true
       
      # Galera router service
      [PXC_Service]
      type=service
      router=readwritesplit
      #slave_selection_criteria=LEAST_CURRENT_OPERATIONS
      slave_selection_criteria=LEAST_GLOBAL_CONNECTIONS
      servers=node1,node2,node3,node4,node5
      user=maxscale_monitor
      password=xxxxxxxxxxxxxxxxxxxxxxxxx
      enable_root_user=1
      connection_timeout=7200s
      max_connections=2000
      max_slave_connections=1
      log_auth_warnings=true
      master_accept_reads=true
      #master_failure_mode=fail_on_write
      #master_reconnection=true
      #max_sescmd_history=500
      disable_sescmd_history=true
       
      [BackUp_Mysql_Service]
      type=service
      router=readconnroute
      router_options=running
      servers=backup_server
      user=maxscale_monitor
      password=855077F67A3FC3F32228F836C40BFB3E
      enable_root_user=1
       
      # Galera cluster listener
      [PXC_Listener]
      type=listener
      service=PXC_Service
      protocol=mariadbclient
      address=0.0.0.0
      port=3306
       
      [BackUp_Listener]
      type=listener
      service=BackUp_Mysql_Service
      protocol=mariadbclient
      address=0.0.0.0
      port=3307
      

      LOGS:

      *2019-11-26 19:18:43   notice : Server changed state: node1[10.1.0.176:3306]: master_up. [Down] -> [Master, Synced, Running]
      2019-11-26 19:18:43   notice : Server changed state: node3[10.1.0.178:3306]: new_slave. [Master, Synced, Running] -> [Slave, Synced, Running]*
      {color:red}2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:{color}
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node3] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node4] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node5] status: [Slave, Synced, Running] state: [IN_USE] last opened at: [Tue Nov 26 19:18:41 2019] last closed at: [not closed] last close reason: [] num sescmd: [0]
      2019-11-26 19:18:43   error  : (33974) [readwritesplit] Cannot choose server 'node1' as the master because it is not in use and a new connection to it cannot be created. Connection status:
      name: [node1] status: [Master, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      name: [node2] status: [Slave, Synced, Running] state: [NOT_IN_USE] last opened at: [not opened] last closed at: [not closed] last close reason: [] num sescmd: [0]
      ........
      2019-11-26 19:21:39   notice : Stopped MaxScale REST API
      2019-11-26 19:21:39   notice : All workers have shut down.
      2019-11-26 19:21:39   notice : MaxScale received signal SIGTERM. Exiting.
      2019-11-26 19:21:39   notice : MaxScale is shutting down.
      2019-11-26 19:21:39   notice : MaxScale shutdown completed.
      2019-11-26 19:21:39   MariaDB MaxScale is shut down.
      ----------------------------------------------------
       
       
      MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Nov 26 19:21:39 2019
      ----------------------------------------------------------------------------
      2019-11-26 19:21:39   notice : syslog logging is enabled.
      2019-11-26 19:21:39   notice : maxlog logging is enabled.
      2019-11-26 19:21:39   notice : Using up to 1.15GiB of memory for query classifier cache
      2019-11-26 19:21:39   notice : Working directory: /var/log/maxscale
      2019-11-26 19:21:39   notice : The collection of SQLite memory allocation statistics turned off.
      2019-11-26 19:21:39   notice : Threading mode of SQLite set to Multi-thread.
      2019-11-26 19:21:39   notice : MariaDB MaxScale 2.4.4 started (Commit: 231f68b6dc70804b785df356679601ab3eb8e379)
      2019-11-26 19:21:39   notice : MaxScale is running in process 13501
      2019-11-26 19:21:39   notice : Configuration file: /etc/maxscale.cnf
      2019-11-26 19:21:39   notice : Log directory: /var/log/maxscale
      2019-11-26 19:21:39   notice : Data directory: /var/lib/maxscale
      2019-11-26 19:21:39   notice : Module directory: /usr/lib64/maxscale
      2019-11-26 19:21:39   notice : Service cache: /var/cache/maxscale
      2019-11-26 19:21:39   notice : Worker message queue size: 1.00MiB
      2019-11-26 19:21:39   notice : No query classifier specified, using default 'qc_sqlite'.
      2019-11-26 19:21:39   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
      2019-11-26 19:21:39   notice : Query classification results are cached and reused. Memory used per thread: 73.31MiB
      2019-11-26 19:21:39   notice : The systemd watchdog is Enabled. Internal timeout = 30s
      2019-11-26 19:21:39   notice : Loading /etc/maxscale.cnf.
      2019-11-26 19:21:39   notice : /etc/maxscale.cnf.d does not exist, not reading.
      2019-11-26 19:21:39   notice : Loaded module mariadbclient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
      2019-11-26 19:21:39   notice : [readconnroute] Initialise readconnroute router module.
      2019-11-26 19:21:39   notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so
      2019-11-26 19:21:39   notice : [readwritesplit] Initializing statement-based read/write split router module.
      2019-11-26 19:21:39   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
      2019-11-26 19:21:39   notice : [galeramon] Initialise the MySQL Galera Monitor module.
      2019-11-26 19:21:39   notice : Loaded module galeramon: V2.0.0 from /usr/lib64/maxscale/libgaleramon.so
      2019-11-26 19:21:39   notice : Loaded module mariadbbackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
      2019-11-26 19:21:39   notice : Loaded module mariadbbackendauth: V1.0.0 from /usr/lib64/maxscale/libmariadbbackendauth.so
      2019-11-26 19:21:39   notice : Loaded module mariadbauth: V1.1.0 from /usr/lib64/maxscale/libmariadbauth.so
      2019-11-26 19:21:39   notice : Started REST API on [127.0.0.1]:8989
      2019-11-26 19:21:39   notice : MaxScale started with 16 worker threads, each with a stack size of 8388608 bytes.
      2019-11-26 19:21:39   notice : Starting a total of 2 services...
      2019-11-26 19:21:39   notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
      2019-11-26 19:21:39   notice : Server 'node1' version: 5.7.27-30-57-log
      2019-11-26 19:21:39   notice : Server 'node2' version: 5.7.27-30-57-log
      2019-11-26 19:21:40   notice : Server 'node3' version: 5.7.27-30-57-log
      2019-11-26 19:21:40   notice : Server 'node4' version: 5.7.27-30-57-log
      2019-11-26 19:21:40   notice : Server 'node5' version: 5.7.27-30-57-log
      2019-11-26 19:21:40   notice : [MariaDBAuth] [PXC_Service] Loaded 40 MySQL users for listener PXC_Listener from server node1.
      2019-11-26 19:21:40   notice : Listening for connections at [0.0.0.0]:3306
      2019-11-26 19:21:40   notice : Service 'PXC_Service' started (1/2)
      2019-11-26 19:21:40   notice : Server 'backup_server' version: 5.6.26
      2019-11-26 19:21:40   notice : [MariaDBAuth] [BackUp_Mysql_Service] Loaded 63 MySQL users for listener BackUp_Listener from server backup_server.
      2019-11-26 19:21:40   notice : Listening for connections at [0.0.0.0]:3307
      2019-11-26 19:21:40   notice : Service 'BackUp_Mysql_Service' started (2/2)
      2019-11-26 19:21:40   notice : Loaded server states from journal file: /var/lib/maxscale/PXC_Monitor/monitor.dat
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            xman13a steven lin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.