Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2463

maxctrl shows stale information on changing bootstrap node

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: xpandmon
    • Labels:
      None
    • Environment:
      MaxScale server karma136:
      OS: CentOS 7
      Version: built from develop branch, MariaDB MaxScale 2.3.7 (Commit: 820ff756a76bdd28301a0bd072eb959cba7c35bc)
      Clustrix nodes:
      OS: CentOS 7
      Version: 9.1.4

      Description

      This is the working as expected config and maxctrl o/p:

      [root@karma016 ~]# clx s
      Cluster Name:    cl0281322e37e3d4b8
      Cluster Version: 9.1.4
      Cluster Status:   OK
      Cluster Size:    3 nodes - 16 CPUs per Node
      Current Node:    karma016 - nid 5
       
      nid |  Hostname | Status |  IP Address  | TPS |       Used      |  Total
      ----+-----------+--------+--------------+-----+-----------------+--------
        3 |  karma049 |    OK  |  10.2.15.126 |   0 |  499.3M (0.06%) |  767.0G
        4 |  karma055 |    OK  |  10.2.15.144 |   0 |  496.7M (0.06%) |  767.0G
        5 |  karma016 |    OK  |   10.2.15.19 |  11 |  496.9M (0.06%) |  767.0G
      ----+-----------+--------+--------------+-----+-----------------+--------
                                                 11 |    1.5G (0.06%) |    2.2T
      Conf file bootstrap server entry using karma016 IP:
      [Bootstrap1-karma016]
      type=server
      address=10.2.15.19
      #karma016
      port=3306
      protocol=mariadbbackend
      ssl=required
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      [root@karma136 ~]# maxctrl list servers
      ┌─────────────────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐
      │ Server              │ Address     │ Port │ Connections │ State           │ GTID │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ @@Clustrix:node-4   │ 10.2.15.144 │ 3306 │ 0           │ Master, Running │      │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ @@Clustrix:node-3   │ 10.2.15.126 │ 3306 │ 0           │ Master, Running │      │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ @@Clustrix:node-5   │ 10.2.15.19  │ 3306 │ 0           │ Master, Running │      │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ Bootstrap1-karma016 │ 10.2.15.19  │ 3306 │ 0           │ Master, Running │      │
      └─────────────────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
      

      Now, change the bootstrap server entry to a server that does not have clustrix (karma197, IP 10.2.13.97):

       
      Conf file bootstrap server entry using karma016 IP:
      [Bootstrap1-karma197]
      type=server
      address=10.2.13.97
      #karma197
      port=3306
      protocol=mariadbbackend
      ssl=required
      ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
      ssl_key=/etc/my.cnf.d/certs/client-key.pem
      ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
       
      [root@karma136 ~]# maxctrl list servers
      ┌─────────────────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐
      │ Server              │ Address     │ Port │ Connections │ State           │ GTID │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ @@Clustrix:node-4   │ 10.2.15.144 │ 3306 │ 0           │ Master, Running │      │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ @@Clustrix:node-3   │ 10.2.15.126 │ 3306 │ 0           │ Master, Running │      │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ @@Clustrix:node-5   │ 10.2.15.19  │ 3306 │ 0           │ Master, Running │      │
      ├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
      │ Bootstrap1-karma197 │ 10.2.13.97  │ 3306 │ 0           │ Down            │      │
      └─────────────────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
      

      The first 3 entries for @@Clustrix:node-3/4/5 are for the old bootstrap server (karma016) which is no more part of the config file.

      Expected:
      We should try to find out nodes from current bootstrap nodes only instead of all previously discovered nodes, if using current bootstrap node fails.
      The maxctrl output should only have one entry like:

      │ Server              │ Address     │ Port │ Connections │ State           │ GTID │
      │ Bootstrap1-karma197 │ 10.2.13.97  │ 3306 │ 0           │ Down            │      │
      

      Relevant logs:

      2019-05-01 20:01:40   error  : [clustrixmon] Clustrix: Could either not ping or create connection to 10.2.13.97:3306: Can't connect to MySQL server on '10.2.13.97' (115)
      2019-05-01 20:01:40   notice : [clustrixmon] Attempting to find a Clustrix bootstrap node from one of the nodes used during the previous run of MaxScale.
      2019-05-01 20:01:40   notice : [clustrixmon] Trying to find out cluster nodes from 10.2.15.19:3306.
      2019-05-01 20:01:40   notice : Created server '@@Clustrix:node-5' at 10.2.15.19:3306
      2019-05-01 20:01:40   info   : [clustrixmon] Updated Clustrix node in bookkeeping: 5, '10.2.15.19', 3306, 3581.
      2019-05-01 20:01:40   notice : Created server '@@Clustrix:node-4' at 10.2.15.144:3306
      2019-05-01 20:01:40   info   : [clustrixmon] Updated Clustrix node in bookkeeping: 4, '10.2.15.144', 3306, 3581.
      2019-05-01 20:01:40   notice : Created server '@@Clustrix:node-3' at 10.2.15.126:3306
      2019-05-01 20:01:40   info   : [clustrixmon] Updated Clustrix node in bookkeeping: 3, '10.2.15.126', 3306, 3581.
      2019-05-01 20:01:40   notice : [clustrixmon] Cluster nodes refreshed.
      2019-05-01 20:01:40   notice : [clustrixmon] Clustrix: Monitoring Clustrix cluster state using node 10.2.15.126:3306.
      2019-05-01 20:01:40   info   : In-memory sqlite database successfully opened for thread 140382505428736.
      2019-05-01 20:01:40   info   : In-memory sqlite database successfully opened for thread 140382497036032.
      2019-05-01 20:01:40   info   : In-memory sqlite database successfully opened for thread 140382281135872.
      2019-05-01 20:01:40   notice : Server changed state: Bootstrap1-karma197[10.2.13.97:3306]: server_down. [Running] -> [Down]
      

      Full logs attached.

        Attachments

          Activity

            People

            Assignee:
            johan.wikman Johan Wikman
            Reporter:
            rahul.joshi@mariadb.com Rahul Joshi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration