Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Not a Bug
-
2.4.5
-
CentOS 7.7 x86_64 in VMWare
MariaDB 10.3.21
Description
I'm working on setting up a MariaDB 3 Node Cluster and using Maxscale as the proxy. I had set up a practice config on some local KVM machines, worked without a hitch. So I went to spin up the production servers and I'm getting an error I can't make sense of. If I run any command in maxctrl at all it throws the same error:
ERROR
|
The requested URL could not be retrieved
|
The following error was encountered while trying to retrieve the URL: http://localhost:8989/v1/maxscale/modules/mariadbmon/ |
Connection to ::1 failed. |
The system returned: (99) Cannot assign requested address |
The remote host or network may be down. Please try the request again. |
Ok so it sounds like something was using port 8989 before Maxscale, let's check with lsof -i -P -n | grep 89:
maxscale 1117 maxscale 23u IPv4 19765 0t0 TCP 127.0.0.1:8989 (LISTEN)
|
SELinux is set to Permissive for testing, Firewalld is off for testing.
Someone suggested it might be an IPv6 issue since it says connection to ::1 but I can't see what the difference would be between my test and pro machines as they both have the same default loopback adapter settings in `lo` and both have the same aliases in `/etc/hosts`
Suggestions for debugging?
Some other steps I performed:
1) Maxscale logs, here's everything up until the listener claims to be started:
MariaDB MaxScale /var/log/maxscale/maxscale.log Sun Feb 2 21:31:23 2020
|
----------------------------------------------------------------------------
|
2020-02-02 21:31:23 notice : syslog logging is enabled.
|
2020-02-02 21:31:23 notice : maxlog logging is enabled.
|
2020-02-02 21:31:23 notice : Using up to 3.51GiB of memory for query classifier cache
|
2020-02-02 21:31:23 notice : Working directory: /var/log/maxscale
|
2020-02-02 21:31:23 notice : The collection of SQLite memory allocation statistics turned off.
|
2020-02-02 21:31:23 notice : Threading mode of SQLite set to Multi-thread.
|
2020-02-02 21:31:23 notice : MariaDB MaxScale 2.4.5 started (Commit: 61b8bbf7f63c38ca9c408674e66f3627a0b2192e)
|
2020-02-02 21:31:23 notice : MaxScale is running in process 8036
|
2020-02-02 21:31:23 notice : Configuration file: /etc/maxscale.cnf
|
2020-02-02 21:31:23 notice : Log directory: /var/log/maxscale
|
2020-02-02 21:31:23 notice : Data directory: /var/lib/maxscale
|
2020-02-02 21:31:23 notice : Module directory: /usr/lib64/maxscale
|
2020-02-02 21:31:23 notice : Service cache: /var/cache/maxscale
|
2020-02-02 21:31:23 notice : Worker message queue size: 1.00MiB
|
2020-02-02 21:31:23 notice : No query classifier specified, using default 'qc_sqlite'.
|
2020-02-02 21:31:23 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
|
2020-02-02 21:31:23 notice : Query classification results are cached and reused. Memory used per thread: 449.02MiB
|
2020-02-02 21:31:23 notice : The systemd watchdog is Enabled. Internal timeout = 30s
|
2020-02-02 21:31:23 notice : Loading /etc/maxscale.cnf.
|
2020-02-02 21:31:23 notice : /etc/maxscale.cnf.d does not exist, not reading.
|
2020-02-02 21:31:23 notice : Loaded module MariaDBClient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
|
2020-02-02 21:31:23 notice : [readwritesplit] Initializing statement-based read/write split router module.
|
2020-02-02 21:31:23 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
|
2020-02-02 21:31:23 notice : [readconnroute] Initialise readconnroute router module.
|
2020-02-02 21:31:23 notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so
|
2020-02-02 21:31:23 notice : [mariadbmon] Initialise the MariaDB Monitor module.
|
2020-02-02 21:31:23 notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so
|
2020-02-02 21:31:23 notice : Loaded module MariaDBBackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
|
2020-02-02 21:31:23 notice : Loaded module mariadbbackendauth: V1.0.0 from /usr/lib64/maxscale/libmariadbbackendauth.so
|
2020-02-02 21:31:23 notice : Using encrypted passwords. Encryption key: '/var/lib/maxscale/.secrets'.
|
2020-02-02 21:31:23 notice : Loaded module mariadbauth: V1.1.0 from /usr/lib64/maxscale/libmariadbauth.so
|
2020-02-02 21:31:23 notice : Started REST API on [127.0.0.1]:8989
|
2020-02-02 21:31:23 notice : MaxScale started with 8 worker threads, each with a stack size of 8388608 bytes.
|
2020-02-02 21:31:23 notice : Starting a total of 2 services...
|
2020-02-02 21:31:23 notice : Server 'server1' version: 10.3.21-MariaDB-log
|
2020-02-02 21:31:23 notice : Server 'server2' version: 10.3.21-MariaDB-log
|
2) curl localhost:8989/v1/maxscale returns the 99 error code as above. If I do curl 127.0.0.1:8989/v1/maxscale it returns a different 111 error.
<blockquote id="error">
|
<p><b>Connection to 127.0.0.1 failed.</b></p>
|
</blockquote>
|
 |
<p id="sysmsg">The system returned: <i>(111) Connection refused</i></p>
|
3) tcpdump shows that absolutely nothing is coming across the wire, which is really weird. I tried `tcpdump -v -i ens192 'port 8989'` and `tcpdump -v -i lo 'port 8989'` and then both curl methods as above, and get the same result:
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
|
0 packets captured
|
0 packets received by filter
|
0 packets dropped by kernel
|
Finally I have attached the MariaDB and Maxscale config files (with obfuscation) to aid in finding out what the issue is.