[MXS-4213] Proxy Protocol : access denied Without dynamic_node_detection=false Created: 2022-07-20  Updated: 2022-07-27  Resolved: 2022-07-25

Status: Closed
Project: MariaDB MaxScale
Component/s: xpandmon
Affects Version/s: 6.4.1
Fix Version/s: 22.08.0

Type: Bug Priority: Major
Reporter: Christine Lieu (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Attachments: Text File maxscale_trace.txt     Text File xpand_trace.txt    
Issue Links:
Relates
relates to MXS-4219 Settings of bootstrap servers are not... Closed

 Description   

Issue
====
Default value of dynamic_node_detection is true
If we don't put dynamic_node_detection=false for the xpandmon monitor in
MaxScale config then the login attempts using proxy fails.

Backend issue appears to be that dynamically detected nodes are having
proxy_protocol off so users are getting access denied.

Below example is where karma075 is MaxScale and user1@clientIP, user2@clientIP
are Xpand users.

[root@vqc006a ~]#  mysql -A -h karma075 -u user1 -P 4008 -p
 
Enter password:
 
ERROR 1045 (28000): Access denied for user 'user1'@'10.2.14.177' (using
 
password: YES)
 
[root@vqc006a ~]#
 
[root@vqc006a ~]#
 
[root@vqc006a ~]#  mysql -A -h karma075 -u user2 -P 4008
 
ERROR 1045 (28000): Access denied for user 'user2'@'10.2.14.177' (using
 
password: NO)
 
[root@vqc006a ~]#

Expected Fix
=========
If our MaxScale understanding is right that one need atleast 1 bootstrap server
configuration to be defined by customer for “dynamic_node_detection”
feature to work and If the customer define "proxy_protocol=on" under server definition (in MaxScale
config) then any dynamic detection for that server nodes should have
proxy_protocol=on and
if customer Do Not put "proxy_protocol=on" then dynamic detection for that
server nodes should have proxy_protocol=off.

This way its conditional, more safe and will not effect non proxy environments.

Detailed Analysis
============
After checking traces from Maxscale and Xpand sides, its revealed that there is
no "proxy" packet.

When checked through maxctrl cmd, found that dynamically detected servers as
shown below have "proxy_protocol" set to off/false .

[root@karma075 ~]# maxctrl show servers | grep 'Server|proxy_protocol'

│ Server              │ Bootstrap1                                   │
 
│                     │     "proxy_protocol": true,                  │
 
│ Server              │ Bootstrap2                                   │
 
│                     │     "proxy_protocol": true,                  │
 
│ Server              │ Bootstrap3                                   │
 
│                     │     "proxy_protocol": true,                  │
 
│ Server              │ @@Clustrix:node-1                            │
 
│                     │     "proxy_protocol": false,                 │
 
│ Server              │ @@Clustrix:node-2                            │
 
│                     │     "proxy_protocol": false,                 │
 
│ Server              │ @@Clustrix:node-3                            │
 
│                     │     "proxy_protocol": false,                 │

Since these server entries are detected at the runtime, there is no way
customer can manually change this parameter's value. We tried with maxctrl and
it failed 

[root@karma075 ~]# maxctrl alter server @@Clustrix:node-1 proxy_protocol true
 
Error: Server at http://127.0.0.1:8989 responded with 400 Bad Request to `PATCH
 
servers/@@Clustrix:node-1`
 
{
 
    "errors": [
 
        {
 
            "detail": "Cannot update server '@@Clustrix:node-1' to '[10.2.15.126]:3306', server 'Bootstrap1' exists there already."
 
        }
 
    ]
 
}
 
[root@karma075 ~]#



 Comments   
Comment by Daman Saini (Inactive) [ 2022-07-21 ]

Xpand Version : 5.0.45-Xpand-mainline1-17678

[root@karma075 ~]# maxscale -V

MaxScale 6.4.1 - ef87eb2c4f054a82d37e79eb472050e28a359ff5
 
CMake flags:   -DBUILD_TESTS=Y -DCMAKE_BUILD_TYPE=Debug -DBUILD_MMMON=Y -DBUILD_AVRO=Y -DBUILD_CDC=Y -DWITH_ASAN=N -DBUILD_GUI=Y -DPACKAGE=Y -DDISTRIB_SUFFIX=rhel.7
 
[root@karma075 ~]#

[root@karma075 ~]# cat /etc/maxscale.cnf

[maxscale]
 
log_info=1
 
threads=auto
 
logdir=/data/clustrix/log
 
 
 
[Bootstrap1]
 
type=server
 
address=10.2.15.126
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=on
 
 
 
[Bootstrap2]
 
type=server
 
address=10.2.15.143
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=on
 
 
 
[Bootstrap3]
 
type=server
 
address=10.2.12.232
 
port=3306
 
protocol=mariadbbackend
 
proxy_protocol=on
 
 
 
[Clustrix]
 
type=monitor
 
module=xpandmon
 
servers=Bootstrap1,Bootstrap2,Bootstrap3
 
user=maxscale
 
password=maxscale_pw
 
cluster_monitor_interval=10000ms
 
# dynamic_node_detection=false
 
 
 
[RCR]
 
type=service
 
router=readconnroute
 
user=maxscale
 
password=maxscale_pw
 
cluster=Clustrix
 
#servers=Bootstrap1,Bootstrap2,Bootstrap3
 
 
 
[RCR-Listener]
 
type=listener
 
service=RCR
 
protocol=MariaDBClient
 
address=0.0.0.0
 
port=4008
 
# authenticator_options=skip_authentication=true

Comment by markus makela [ 2022-07-25 ]

The configuration options from the bootstrap nodes are now copied over to the dynamically generated ones. With the addition of the proxy_protocol support for internal connections, this should work with the 22.08.0 release of MaxScale.

Comment by Daman Saini (Inactive) [ 2022-07-27 ]

While testing the build generated by markus makela, found an issue which is being tracked as MXS-4219
For "proxy_protocol=on" scenarios, the fix worked fine.

Scenario 1: Any 1 node (only) of Xpand with "proxy_protocol=on"
http://paste/RqMh (All logs of the testing and working maxscale.cfg is here)

Scenario 2: Declare all nodes of the Xpand with "proxy_protocol=on"
http://paste/Owhk (All logs of the testing and working maxscale.cfg is here)

Generated at Thu Feb 08 04:27:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.