[MXS-4219] Settings of bootstrap servers are not correctly propagated to dynamic servers Created: 2022-07-27  Updated: 2023-07-26  Resolved: 2022-08-12

Status: Closed
Project: MariaDB MaxScale
Component/s: xpandmon
Affects Version/s: 6.4.2
Fix Version/s: 6.4.2, 22.08.1

Type: Bug Priority: Major
Reporter: Daman Saini (Inactive) Assignee: Johan Wikman
Resolution: Fixed Votes: 0
Labels: None
Environment:

3 Node Xpand : Xpand-mainline1-17678
Maxscale Version : MaxScale 6.4.2 - 302d0d88ee2524cdc53f4c4584f1ed3ef0caffa1

Issue Links:
Relates
relates to MXS-4683 ssl parameters specified on the boots... Closed
relates to MXS-4213 Proxy Protocol : access denied Withou... Closed
Sprint: MXS-SPRINT-163

 Description   

When the Xpand monitor creates a dynamic server, the settings of the bootstrap servers are not correctly propagated to the dynamic server.

Further, the Xpand monitor does not detect if the bootstrap servers have different settings, leading to a situation where it is unclear what settings the dynamic servers should have. The monitor should detect such a situation and refuse to start.

Original description:

Issue :
====
Issue is more specific to Bootstrap1's proxy_protocol parameter which is always getting set as True in below cases

Case1 : proxy_protocol=off explicitly mentioned under Bootstrap1 

[root@karma075 ~]# cat /etc/maxscale.cnf
 
[maxscale]
 
log_info=1
 
threads=auto
 
logdir=/data/clustrix/log
 
 
 
[Bootstrap1]
 
type=server
 
address=10.2.15.126
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=off
 
 
 
[Clustrix]
 
type=monitor
 
module=xpandmon
 
servers=Bootstrap1
 
user=maxscale
 
password=maxscale_pw
 
cluster_monitor_interval=10000ms
 
 
 
[RCR]
 
type=service
 
router=readconnroute
 
user=maxscale
 
password=maxscale_pw
 
cluster=Clustrix
 
 
 
[RCR-Listener]
 
type=listener
 
service=RCR
 
protocol=MariaDBClient
 
address=0.0.0.0
 
port=4008

Case2 : No proxy_protocol mentioned under Bootstrap1 section of maxscale.cnf . So
just remove "proxy_protocol=off" from Case1 maxscale.cnf

Output of maxctrl in both Case1 and Case2

[root@karma075 ~]#  maxctrl show servers | grep 'Server\|proxy_protocol'
 
│ Server              │ Bootstrap1                                      │
 
│                     │     "proxy_protocol": true,                     │
 
│ Server              │ @@Clustrix:node-3                            │
 
│                     │     "proxy_protocol": true,                  │
 
│ Server              │ @@Clustrix:node-1                            │
 
│                     │     "proxy_protocol": true,                  │
 
│ Server              │ @@Clustrix:node-2                            │
 
│                     │     "proxy_protocol": true,                  │
 
[root@karma075 ~]#

Imp Note : There are interesting Case3 and 4 too

Case3 : Declare all 3 nodes of Xpand as Bootstrap1 , Bootstrap2 and Bootstrap3 with proxy_protocol=off 

[root@karma075 ~]# cat /etc/maxscale.cnf
 
[maxscale]
 
log_info=1
 
threads=auto
 
logdir=/data/clustrix/log
 
 
 
[Bootstrap1]
 
type=server
 
address=10.2.15.126
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=off
 
 
 
[Bootstrap2]
 
type=server
 
address=10.2.15.143
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=off
 
 
 
[Bootstrap3]
 
type=server
 
address=10.2.12.232
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=off
 
 
 
[Clustrix]
 
type=monitor
 
module=xpandmon
 
servers=Bootstrap1,Bootstrap2,Bootstrap3
 
user=maxscale
 
password=maxscale_pw
 
cluster_monitor_interval=10000ms
 
 
 
[RCR]
 
type=service
 
router=readconnroute
 
user=maxscale
 
password=maxscale_pw
 
cluster=Clustrix
 
 
 
[RCR-Listener]
 
type=listener
 
service=RCR
 
protocol=MariaDBClient
 
address=0.0.0.0
 
port=4008

Case4 : No proxy_protocol mentioned under Bootstrap1 ,2, 3 sections of maxscale.cnf. So
just remove "proxy_protocol=off" from Case3 maxscale.cnf

Output of maxctrl in both Case3 and Case4

[root@karma075 ~]#  maxctrl show servers | grep 'Server\|proxy_protocol'
 
│ Server              │ Bootstrap2                                   │
 
 
 
│                     │     "proxy_protocol": false,                 │
 
│ Server              │ Bootstrap3                                   │
 
 
 
│                     │     "proxy_protocol": false,                 │
 
│ Server              │ Bootstrap1                                      │
 
 
 
│                     │     "proxy_protocol": true,                     │
 
│ Server              │ @@Clustrix:node-1                            │
 
 
 
│                     │     "proxy_protocol": false,                 │
 
│ Server              │ @@Clustrix:node-3                            │
 
 
 
│                     │     "proxy_protocol": false,                 │
 
│ Server              │ @@Clustrix:node-2                            │
 
 
 
│                     │     "proxy_protocol": false,                 │
 
[root@karma075 ~]#

Expected Fix :
==========
Correct value of proxy_protocol should be set for Bootstrap1 which in turn set correct value for the dynamically detected nodes



 Comments   
Comment by Daman Saini (Inactive) [ 2022-07-27 ]

markus makela
Feel Free to use my setup
Xpand : karma049 + 2 nodes
Maxscale : Karma075

Case1 maxscale file on Karma075 is /etc/maxscale.cnf.MXS-4219-Case1
Case3 File is /etc/maxscale.cnf.MXS-4219-Case3

Comment by Johan Wikman [ 2022-08-03 ]

damansaini There are a couple of things that needs to be addressed.

I suppose that as far as Xpand goes, the proxy_protocol setting is cluster-wide? That is, either all Xpand nodes of a cluster will have proxy_protocol enabled, or none will have? So, I think MaxScale should require that the proxy_protocol setting of all bootstrap servers is identical.

Further, the bootstrap servers are basically only used at first startup. Thereafter, MaxScale will use information that it has probed at runtime. However, it will detect whether something has changed in the bootstrap configuration and re-probe but I'm not sure whether currently a change in the proxy_protocol setting will trigger that.

Comment by Manjinder Nijjar [ 2022-08-05 ]

Hi johan.wikman ,

1. Yes, the proxy_protocol setting is cluster-wide. On xpand proxy protocol is enabled when you set global proxy_protocol_nets to a ip subnet. There is no other setting on xpand. By default this global is empty i.e. proxy protocol is not enabled.

2.Makes sense. So does that mean today maxscale is able to read proxy_protocol_nets from xpand and can respond accordingly? We were not aware of this and also from our testing it did not see this happening.
+emurray who was dev on xpand for this feature.

Comment by Johan Wikman [ 2022-08-09 ]

2. No, I was only referring to changes make to the bootstrap configuration in the MaxScale configuration file.

If proxy protocol is enabled in Xpand, then MaxScale cannot connect without providing the proxy protocol string (right?). That is, if MaxScale connects without providing the proxy protocol string and it succeeds, then MaxScale knows the proxy protocol is not required for its IP and if it connects and provides the proxy protocol string and it succeeds, then MaxScale knows the proxy protocol is required. However, a failure to connect does not as such tell indicate whether it is related to the presence/absence of the proxy protocol string. Anyway, basically MaxScale does not need to read value of proxy_protocol_nets to find out whether it applies to MaxScale's IP. Butdoing that probing is somewhat messy, so initially it will have to be specified in the MaxScale configuration file.

Can the value of proxy_protocol_nets be changed at any time and if so, what's the effect on existing connections?

Comment by Daman Saini (Inactive) [ 2022-08-09 ]

emurray Please ans above queries.

Comment by Elizabeth Murray (Inactive) [ 2022-08-10 ]

Existing connections are unaffected by changing the proxy protocol configuration on xpand.

If proxy protocol is enabled, the header is required. I think what you said about a way to check for proxy protocol being enabled on xpand makes sense.

Comment by Johan Wikman [ 2022-08-10 ]

Ok, good to know that the probing should work. However, initially the proxy protocol setting will have to be configured in MaxScale as well. Probing the setting and automatically adapting to whatever it happens to be in Xpand is for later.

Comment by Johan Wikman [ 2022-08-25 ]

When the Xpand monitor creates a dynamic server, the values for max_routing_connections, persistmaxtime, persistpoolmax, and proxy_protocol are now copied from the bootstrap servers.

At startup the Xpand monitor checks that the values of those settings are identical and if they are not, the monitor will not start. When a dynamic server is created, a check is again made and if they are not identical a warning will be logged and the last know good consistent set of settings will be used.

Generated at Thu Feb 08 04:27:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.