[MXS-5636] Maxctrl reports phantom connections - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Incomplete
Affects Version/s: 23.08.7
Fix Version/s: N/A
Component/s: N/A
Labels:
None

Description

This behavior has been seen multiple times on a customer's production instance, but we do not currently have a way to reproduce. Adding this info to a bug report to save time reinvestigating down the road.

Customer Setup
Two maxscales on version 23.08.7. These are in an active/passive configuration, with a virtual IP managed by keepalived
dbproxy1
dbproxy2

Three database servers in async replication. db1 is the primary and db2 and db3 are replicas:
db1
db2
db3

Behavior
Before patching, dbproxy1 was the active maxscale. There were no connections through dbproxy2.jpn:

dbproxy2.jpn

┌─────────┬───────────────┬──────┬─────────────┬─────────────────┬──────────────┬─────────────────┐

│ Server  │ Address       │ Port │ Connections │ State           │ GTID         │ Monitor         │

├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┼─────────────────┤

│ db1.jpn │ 10.60.120.241 │ 3306 │ 0           │ Master, Running │ 0-1-43412200 │ MariaDB-Monitor │

├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┼─────────────────┤

│ db2.jpn │ 10.60.120.249 │ 3306 │ 0           │ Slave, Running  │ 0-1-43412200 │ MariaDB-Monitor │

├─────────┼───────────────┼──────┼─────────────┼─────────────────┼──────────────┼─────────────────┤

│ db3.jpn │ 10.60.120.248 │ 3306 │ 0           │ Slave, Running  │ 0-1-43412200 │ MariaDB-Monitor │

└─────────┴───────────────┴──────┴─────────────┴─────────────────┴──────────────┴─────────────────┘

Connection to dbproxy2.jpn closed.

After patching and rebooting, dbproxy2.jpn reported 14 connections, including connections to a server in maintenance mode:

PRD [srv-kyle.turner@dbproxy2-jpn.mfg.aristanetworks.com ~]$ maxctrl list servers

┌─────────┬───────────────┬──────┬─────────────┬──────────────────────┬──────────────┬─────────────────┐

│ Server  │ Address       │ Port │ Connections │ State                │ GTID         │ Monitor         │

├─────────┼───────────────┼──────┼─────────────┼──────────────────────┼──────────────┼─────────────────┤

│ db1.jpn │ 10.60.120.241 │ 3306 │ 14          │ Master, Running      │ 0-1-49794523 │ MariaDB-Monitor │

├─────────┼───────────────┼──────┼─────────────┼──────────────────────┼──────────────┼─────────────────┤

│ db2.jpn │ 10.60.120.249 │ 3306 │ 14          │ Maintenance, Running │ 0-1-49794523 │ MariaDB-Monitor │

├─────────┼───────────────┼──────┼─────────────┼──────────────────────┼──────────────┼─────────────────┤

│ db3.jpn │ 10.60.120.248 │ 3306 │ 14          │ Slave, Running       │ 0-1-49794523 │ MariaDB-Monitor │

└─────────┴───────────────┴──────┴─────────────┴──────────────────────┴──────────────┴─────────────────┘

All sessions appeared as soon as maxscale started, and were in an idle state. I couldn’t find any evidence of these connections on the backend server:

PRD [srv-kyle.turner@dbproxy2-jpn.mfg.aristanetworks.com ~]$ maxctrl list sessions

┌─────┬──────────┬───────────────┬───────────────────────┬───────┬────────────────────┬────────┬──────────────┐

│ Id  │ User     │ Host          │ Connected             │ Idle  │ Service            │ Memory │ I/O-Activity │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 103 │ arastra2 │ 10.60.121.33  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 69162  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 6   │ arastra2 │ 10.60.121.102 │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 68987  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 19  │ arastra2 │ 10.60.121.120 │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 68987  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 4   │ arastra2 │ 10.60.121.35  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 68987  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 80  │ arastra2 │ 10.60.121.34  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 3378   │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 3   │ arastra2 │ 10.60.121.42  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 68930  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 102 │ arastra2 │ 10.60.121.68  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 69162  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 11  │ arastra2 │ 10.60.121.58  │ 3/25/2025, 9:24:26 PM │ 260   │ Read-Write-Service │ 3435   │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 10  │ arastra2 │ 10.60.121.80  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 68987  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 101 │ arastra2 │ 10.60.122.45  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 69162  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 5   │ arastra2 │ 10.60.121.68  │ 3/25/2025, 9:24:26 PM │ 260   │ Read-Write-Service │ 3435   │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 35  │ arastra2 │ 10.60.122.45  │ 3/25/2025, 9:24:26 PM │ 260   │ Read-Write-Service │ 68987  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 41  │ arastra2 │ 10.60.121.33  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 68987  │ 0            │

├─────┼──────────┼───────────────┼───────────────────────┼───────┼────────────────────┼────────┼──────────────┤

│ 97  │ arastra2 │ 10.60.121.35  │ 3/25/2025, 9:24:26 PM │ 259.9 │ Read-Write-Service │ 69162  │ 0            │

└─────┴──────────┴───────────────┴───────────────────────┴───────┴────────────────────┴────────┴──────────────┘

Over time, the connections started to drop off, but maxscale still overreported the number of sessions. Here’s an example of the processlist when maxscale reported 7 connections.

Sessions:

PRD [srv-kyle.turner@dbproxy2-jpn.mfg.aristanetworks.com ~]$ maxctrl list sessions

┌────┬──────────┬───────────────┬───────────────────────┬────────┬────────────────────┬────────┬──────────────┐

│ Id │ User     │ Host          │ Connected             │ Idle   │ Service            │ Memory │ I/O-Activity │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 6  │ arastra2 │ 10.60.121.102 │ 3/25/2025, 9:24:26 PM │ 1215.1 │ Read-Write-Service │ 68987  │ 0            │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 4  │ arastra2 │ 10.60.121.35  │ 3/25/2025, 9:24:26 PM │ 1215.1 │ Read-Write-Service │ 68987  │ 0            │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 80 │ arastra2 │ 10.60.121.34  │ 3/25/2025, 9:24:26 PM │ 1215.1 │ Read-Write-Service │ 3435   │ 0            │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 11 │ arastra2 │ 10.60.121.58  │ 3/25/2025, 9:24:26 PM │ 1215.2 │ Read-Write-Service │ 3435   │ 0            │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 5  │ arastra2 │ 10.60.121.68  │ 3/25/2025, 9:24:26 PM │ 1215.2 │ Read-Write-Service │ 3435   │ 0            │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 35 │ arastra2 │ 10.60.122.45  │ 3/25/2025, 9:24:26 PM │ 1215.2 │ Read-Write-Service │ 68987  │ 0            │

├────┼──────────┼───────────────┼───────────────────────┼────────┼────────────────────┼────────┼──────────────┤

│ 41 │ arastra2 │ 10.60.121.33  │ 3/25/2025, 9:24:26 PM │ 1215.1 │ Read-Write-Service │ 68987  │ 0            │

└────┴──────────┴───────────────┴───────────────────────┴────────┴────────────────────┴────────┴──────────────┘

vs Processlist:

MariaDB [(none)]> show processlist;

+------------+-------------+---------------------+------+-----------+------+--------------------------------------------------------+------------------+----------+

| Id         | User        | Host                | db   | Command   | Time | State                                                  | Info             | Progress |

+------------+-------------+---------------------+------+-----------+------+--------------------------------------------------------+------------------+----------+

|     147061 | maxscale    | 10.60.120.253:57308 | NULL | Sleep     |    0 |                                                        | NULL             |    0.000 |

| 1002866361 | monyogmon   | 10.90.194.13:41832  | NULL | Sleep     |    0 |                                                        | NULL             |    0.000 |

| 1002867462 | monyogmon   | 10.90.194.13:41896  | NULL | Sleep     |   50 |                                                        | NULL             |    0.000 |

| 1022986714 | system user |                     | NULL | Slave_IO  | 2473 | Waiting for master to send event                       | NULL             |    0.000 |

| 1022986715 | system user |                     | NULL | Slave_SQL |    0 | Slave has read all relay log; waiting for more updates | NULL             |    0.000 |

| 1023595799 | maxscale    | 10.60.120.245:49310 | NULL | Sleep     |    0 |                                                        | NULL             |    0.000 |

| 1023745598 | root        | localhost           | NULL | Query     |    0 | starting                                               | show processlist |    0.000 |

+------------+-------------+---------------------+------+-----------+------+--------------------------------------------------------+------------------+----------+

maxctrl output:

PRD [srv-kyle.turner@dbproxy2-jpn.mfg.aristanetworks.com ~]$ maxctrl list servers

┌─────────┬───────────────┬──────┬─────────────┬──────────────────────┬──────────────┬─────────────────┐

│ Server  │ Address       │ Port │ Connections │ State                │ GTID         │ Monitor         │

├─────────┼───────────────┼──────┼─────────────┼──────────────────────┼──────────────┼─────────────────┤

│ db1.jpn │ 10.60.120.241 │ 3306 │ 7           │ Master, Running      │ 0-1-49797185 │ MariaDB-Monitor │

├─────────┼───────────────┼──────┼─────────────┼──────────────────────┼──────────────┼─────────────────┤

│ db2.jpn │ 10.60.120.249 │ 3306 │ 7           │ Maintenance, Running │ 0-1-49797185 │ MariaDB-Monitor │

├─────────┼───────────────┼──────┼─────────────┼──────────────────────┼──────────────┼─────────────────┤

│ db3.jpn │ 10.60.120.248 │ 3306 │ 7           │ Slave, Running       │ 0-1-49797185 │ MariaDB-Monitor │

└─────────┴───────────────┴──────┴─────────────┴──────────────────────┴──────────────┴─────────────────┘

Finally, I've attached two additional snippets. The first shows maxctrl show sessions output on the nonexistent connections. The second shows startup logs just before maxscale started reporting phantom connections.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

dbproxy1_jpn_report
2025-06-25 22:20
8.60 MB
J Kyle Turner
maxctrl_show_sessions.txt
2025-06-05 19:33
49 kB
J Kyle Turner
maxscale_start_logs.txt
2025-06-05 19:33
6 kB
J Kyle Turner

Activity

People

Assignee:: markus makela

Reporter:: J Kyle Turner

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2025-06-05 19:27

Updated:: 2025-08-22 07:05

Resolved:: 2025-08-22 07:05

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.