Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
None
-
MaxScale server karma136:
OS: CentOS 7
Version: built from develop branch, MariaDB MaxScale 2.3.7 (Commit: 820ff756a76bdd28301a0bd072eb959cba7c35bc)
Clustrix nodes:
OS: CentOS 7
Version: 9.1.4
Description
This is the working as expected config and maxctrl o/p:
[root@karma016 ~]# clx s
|
Cluster Name: cl0281322e37e3d4b8
|
Cluster Version: 9.1.4
|
Cluster Status: OK
|
Cluster Size: 3 nodes - 16 CPUs per Node
|
Current Node: karma016 - nid 5
|
|
|
nid | Hostname | Status | IP Address | TPS | Used | Total
|
----+-----------+--------+--------------+-----+-----------------+--------
|
3 | karma049 | OK | 10.2.15.126 | 0 | 499.3M (0.06%) | 767.0G
|
4 | karma055 | OK | 10.2.15.144 | 0 | 496.7M (0.06%) | 767.0G
|
5 | karma016 | OK | 10.2.15.19 | 11 | 496.9M (0.06%) | 767.0G
|
----+-----------+--------+--------------+-----+-----------------+--------
|
11 | 1.5G (0.06%) | 2.2T
|
Conf file bootstrap server entry using karma016 IP:
|
[Bootstrap1-karma016]
|
type=server
|
address=10.2.15.19
|
#karma016
|
port=3306
|
protocol=mariadbbackend
|
ssl=required
|
ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
|
ssl_key=/etc/my.cnf.d/certs/client-key.pem
|
ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
|
|
|
[root@karma136 ~]# maxctrl list servers
|
┌─────────────────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐
|
│ Server │ Address │ Port │ Connections │ State │ GTID │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ @@Clustrix:node-4 │ 10.2.15.144 │ 3306 │ 0 │ Master, Running │ │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ @@Clustrix:node-3 │ 10.2.15.126 │ 3306 │ 0 │ Master, Running │ │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ @@Clustrix:node-5 │ 10.2.15.19 │ 3306 │ 0 │ Master, Running │ │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ Bootstrap1-karma016 │ 10.2.15.19 │ 3306 │ 0 │ Master, Running │ │
|
└─────────────────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
|
Now, change the bootstrap server entry to a server that does not have clustrix (karma197, IP 10.2.13.97):
|
|
Conf file bootstrap server entry using karma016 IP:
|
[Bootstrap1-karma197]
|
type=server
|
address=10.2.13.97
|
#karma197
|
port=3306
|
protocol=mariadbbackend
|
ssl=required
|
ssl_cert=/etc/my.cnf.d/certs/client-cert.pem
|
ssl_key=/etc/my.cnf.d/certs/client-key.pem
|
ssl_ca_cert=/etc/my.cnf.d/certs/ca-cert.pem
|
|
|
[root@karma136 ~]# maxctrl list servers
|
┌─────────────────────┬─────────────┬──────┬─────────────┬─────────────────┬──────┐
|
│ Server │ Address │ Port │ Connections │ State │ GTID │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ @@Clustrix:node-4 │ 10.2.15.144 │ 3306 │ 0 │ Master, Running │ │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ @@Clustrix:node-3 │ 10.2.15.126 │ 3306 │ 0 │ Master, Running │ │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ @@Clustrix:node-5 │ 10.2.15.19 │ 3306 │ 0 │ Master, Running │ │
|
├─────────────────────┼─────────────┼──────┼─────────────┼─────────────────┼──────┤
|
│ Bootstrap1-karma197 │ 10.2.13.97 │ 3306 │ 0 │ Down │ │
|
└─────────────────────┴─────────────┴──────┴─────────────┴─────────────────┴──────┘
|
The first 3 entries for @@Clustrix:node-3/4/5 are for the old bootstrap server (karma016) which is no more part of the config file.
Expected:
We should try to find out nodes from current bootstrap nodes only instead of all previously discovered nodes, if using current bootstrap node fails.
The maxctrl output should only have one entry like:
│ Server │ Address │ Port │ Connections │ State │ GTID │
|
│ Bootstrap1-karma197 │ 10.2.13.97 │ 3306 │ 0 │ Down │ │
|
Relevant logs:
2019-05-01 20:01:40 error : [clustrixmon] Clustrix: Could either not ping or create connection to 10.2.13.97:3306: Can't connect to MySQL server on '10.2.13.97' (115)
|
2019-05-01 20:01:40 notice : [clustrixmon] Attempting to find a Clustrix bootstrap node from one of the nodes used during the previous run of MaxScale.
|
2019-05-01 20:01:40 notice : [clustrixmon] Trying to find out cluster nodes from 10.2.15.19:3306.
|
2019-05-01 20:01:40 notice : Created server '@@Clustrix:node-5' at 10.2.15.19:3306
|
2019-05-01 20:01:40 info : [clustrixmon] Updated Clustrix node in bookkeeping: 5, '10.2.15.19', 3306, 3581.
|
2019-05-01 20:01:40 notice : Created server '@@Clustrix:node-4' at 10.2.15.144:3306
|
2019-05-01 20:01:40 info : [clustrixmon] Updated Clustrix node in bookkeeping: 4, '10.2.15.144', 3306, 3581.
|
2019-05-01 20:01:40 notice : Created server '@@Clustrix:node-3' at 10.2.15.126:3306
|
2019-05-01 20:01:40 info : [clustrixmon] Updated Clustrix node in bookkeeping: 3, '10.2.15.126', 3306, 3581.
|
2019-05-01 20:01:40 notice : [clustrixmon] Cluster nodes refreshed.
|
2019-05-01 20:01:40 notice : [clustrixmon] Clustrix: Monitoring Clustrix cluster state using node 10.2.15.126:3306.
|
2019-05-01 20:01:40 info : In-memory sqlite database successfully opened for thread 140382505428736.
|
2019-05-01 20:01:40 info : In-memory sqlite database successfully opened for thread 140382497036032.
|
2019-05-01 20:01:40 info : In-memory sqlite database successfully opened for thread 140382281135872.
|
2019-05-01 20:01:40 notice : Server changed state: Bootstrap1-karma197[10.2.13.97:3306]: server_down. [Running] -> [Down]
|
Full logs attached.