[MDEV-20627] Galera4 not able to report proper wsrep_incoming_addresses Created: 2019-09-18  Updated: 2022-05-31  Resolved: 2022-05-30

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.8, 10.5
Fix Version/s: 10.4.26, 10.5.17, 10.6.9, 10.7.5, 10.8.4

Type: Bug Priority: Blocker
Reporter: Marcelo Altmann Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 7
Labels: None

Issue Links:
Relates
relates to MDEV-28295 wsrep_server_incoming_address inc_add... Closed

 Description   

Galera4 is not able to report proper address of members from the cluster:

How to reproduce
1. Strat maria with galera4:

mysqld --no-defaults --wsrep_on=ON --wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so --wsrep_sst_method=mariabackup --user=mysql --binlog_format=ROW --wsrep_cluster_address=gcomm:// --wsrep_node_address=10.126.47.243 --bind_address=10.126.47.243

2. Check wsrep_incoming_addresses variable:

mysql -e "SHOW GLOBAL STATUS LIKE 'wsrep_inc%'"

Result

+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| wsrep_incoming_addresses | AUTO  |
+--------------------------+-------+



 Comments   
Comment by staf wagemakers [ 2020-04-23 ]

We had the same issue when we upgrade to galera 4, we used the value of wsrep_incoming_addresses in our monitoring scripts.
Work-around is set also 'wsrep_node_incoming_address' variable.

Comment by Rob Brown [ 2020-05-18 ]

Yes, all of our monitoring applications suddenly got ruined after upgrading to Galera-4 due to wsrep_incoming_addresses just saying "AUTO" all the sudden. For example, here is what is getting passed to my wsrep_notify_cmd:

 --status joined
 --uuid 0db74c70-184d-11e8-81f3-2f1afb15c1c9
 --primary yes
 --index -1
 --members 528b829e-97dd-11ea-b29e-4b751ad2b256/node130/AUTO,df15aa3f-97de-11ea-af73-c728f9115f6b/node112/AUTO,f864ece2-97dd-11ea-b207-02205a497b45/node121/AUTO

Now I'm not even sure what "--index -1" means, but if that were more accurate, then at least it would be possible to determine what my own node_name was from this list of members. There really just needs to be some way to link the node_name to the IP Address for our monitoring application to function properly. Right now this seems impossible.

THANK YOU for the work-around of hacking wsrep_node_incoming_address to force it to show something more useful to prevent the monitor handler from crashing.

But I recommend please NEVER show "AUTO" for the "address" field! If "wsrep_node_incoming_address" is set to "AUTO" then it shall use "wsrep_node_address" or something more useful.

I noticed that "wsrep_node_incoming_address" is also set to "AUTO" in Galera-3, but everything worked much better somehow.

Comment by Aurélien LEQUOY [ 2020-09-09 ]

Hello, do you have any idea when it's will be fixed ?

it's really problematic for our monitoring too.

Comment by Aurélien LEQUOY [ 2020-10-02 ]

https://galeracluster.com/library/documentation/mysql-wsrep-options.html#wsrep-node-incoming-address

documentation say it too :

This parameter defines the IP address and port number at which the node should expect to receive client connections. It’s intended for integration with load balancers. For now, it’s otherwise unused by the node.

You can execute the SHOW VARIABLES statement with the LIKE operator as shown below to get the IP address and port setting of this parameter:

SHOW VARIABLES LIKE 'wsrep_node_incoming_address';
+-----------------------------+------------------+
| Variable_name               | Value            |
+-----------------------------+------------------+
| wsrep_node_incoming_address | 192.168.1.1:3306 |
+-----------------------------+------------------+

it's the only one way to know the others members of the cluster

Comment by Aurélien LEQUOY [ 2020-10-14 ]

and know when there is a SST who joiner

Comment by George [ 2021-01-02 ]

Careful. There are two parameters here, one is wsrep_incoming_address the other is wsrep_incoming_addresses.

The bug centers around the parameter wsrep_incoming_addresses, where instead reporting the IP addresses of the other acrive cluster nodes, it will report back as AUTO.

Comment by Vincent Milum Jr [ 2022-04-20 ]

Why was this closed without any comment at all? This most certainly IS a bug, it is a feature regression, and it goes against the documented functionality of how this SHOULD work. Please re-open this ticket.

Comment by Daniel Black [ 2022-04-20 ]

Although MDEV-28295 was filed as remove the dropped code, maybe it should actually be used.

Comment by Jan Lindström (Inactive) [ 2022-04-21 ]

I tested this on latest 10.3 and with following config

[mysqld.1]
wsrep-on=ON
wsrep-cluster-address=gcomm://
wsrep-provider=@ENV.WSREP_PROVIDER
binlog-format=ROW
wsrep_node_address=127.0.0.1

and I see following:

show variables like '%addres%';
Variable_name	Value
bind_address	127.0.0.1
wsrep_cluster_address	gcomm://
wsrep_node_address	127.0.0.1
wsrep_node_incoming_address	AUTO
wsrep_sst_receive_address	AUTO

darkain Can you provide your config and output from Galera 3 where it behaves differently i.e. gives real addresses on wsrep_node_incoming_address variable.

Comment by Brian [ 2022-05-18 ]

Same issue after upgrading to Mariadb10.5 / Galera 4:

MariaDB [(none)]> show status like 'wsrep_incoming_addresses';
+--------------------------+----------------+
| Variable_name            | Value          |
+--------------------------+----------------+
| wsrep_incoming_addresses | AUTO,AUTO,AUTO |
+--------------------------+----------------+

Comment by Vincent Milum Jr [ 2022-05-23 ]

Why does this ticket keep on getting closed? This is STILL an unresolved regression, and I'm not sure what more feedback is needed. This is a pretty simple issue of "value reported IP addresses before, now only reports the word AUTO"

Comment by Jan Lindström (Inactive) [ 2022-05-24 ]

darkain Because we have not been able to find a version and config where wsrep_incoming_addresses would be something else than AUTO, changing it would mean external user change.

Comment by staf wagemakers [ 2022-05-24 ]

For our centos 6 with MariaDB 10.1 is was set to the ip addresses or node name automatically.
When we upgraded to centos 7 with 10.4 we need change the MariaDB configuration and set the wsrep_node_incoming_address in the mariadb configuration.

[galera]
wsrep_node_address = x.x.x.x
wsrep_node_incoming_address = x.x.x.x

  1. mariadb
    Welcome to the MariaDB monitor. Commands end with ; or \g.
    Your MariaDB connection id is 1620710
    Server version: 10.4.12-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show status like 'wsrep_incoming_addresses';
-------------------------------------+

Variable_name Value

-------------------------------------+

wsrep_incoming_addresses x.x.x.x

-------------------------------------+
1 row in set (0.001 sec)

MariaDB [(none)]>

A lot of monitoring scripts use this value, so it'd be great if the old behaviour cloud be restored or it cloud be documented.

Hope this helps to reproduce this issue...

Comment by Vincent Milum Jr [ 2022-05-24 ]

Jan Lindström, as others have said, go back and look at the behavior of MariaDB 10.1 and 10.2 using a version of Galera that matched those versions rather than a current version of Galera. This has been broken for quite some time and this ticket has been open since 2019, so yes, you need to find a MariaDB version older than the ticket. Only testing relatively "recent" versions of MariaDB of course has the change from IP addresses to "AUTO". There are examples in this bug report and version numbers listed in several of the comments.

Comment by Brian [ 2022-05-24 ]

Fully agree, I am facing this issue since an update from MariaDB10.3/Galera 3 to Mariadb10.5/Galera4. As staf wagemakers said, we also use this for monitoring purposes.

Comment by Jan Lindström (Inactive) [ 2022-05-25 ]

ramesh Can you test with 10.1 and 10.2.

Comment by Ramesh Sivaraman [ 2022-05-25 ]

jplindst 10.1 and 10.2 show the IP addresses in the wsrep_incoming_addresses status variable, even if you did not specify the IP address in the wsrep_node_incoming_address variable through the cnf file or the Galera startup command.

node1:root@localhost> select @@version;
+-----------------------+
| @@version             |
+-----------------------+
| 10.2.44-MariaDB-debug |
+-----------------------+
1 row in set (0.00 sec)
 
node1:root@localhost> 
Status variable
node1:root@localhost> SHOW GLOBAL STATUS LIKE 'wsrep_incoming_addresses';
+--------------------------+---------------------------------+
| Variable_name            | Value                           |
+--------------------------+---------------------------------+
| wsrep_incoming_addresses | 127.0.0.1:11531,127.0.0.1:10979 |
+--------------------------+---------------------------------+
1 row in set (0.00 sec)
 
node1:root@localhost>
 
 
10.1.49-opt>select @@version;
+-----------------+
| @@version       |
+-----------------+
| 10.1.49-MariaDB |
+-----------------+
1 row in set (0.00 sec)
 
10.1.49-opt>SHOW GLOBAL STATUS LIKE 'wsrep_incoming_addresses';
+--------------------------+---------------------------------+
| Variable_name            | Value                           |
+--------------------------+---------------------------------+
| wsrep_incoming_addresses | 127.0.0.1:10491,127.0.0.1:10913 |
+--------------------------+---------------------------------+
1 row in set (0.00 sec)
 
10.1.49-opt>

Comment by Jan Lindström (Inactive) [ 2022-05-25 ]

Because based on testing this is regression from 10.3 to 10.4, I will set this bug as a blocker.

Comment by staf wagemakers [ 2022-05-30 ]

Thx Jan Lindström !

Generated at Thu Feb 08 09:00:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.