RHEL v8.2
VMware vCenter v7
MariaDB v10.7 (3 node Galera cluster)
Maxcsale v23.02.3 (one instance of each per MariaDB VM for redundancy)

Have you tried upgrading to the latest release?

Hi,

Yesterday we upgraded to v23.02.3. Issue re-occurred this morning, please refer to the attached log & and server state changes graph.

Thanks.

MXS-4711_maxscale-logs-01.txt

One possibility where this can happen if a reverse DNS lookup done during authentication is taking a long time. To rule this out, you can add skip_name_resolve=true under the [maxscale] section. At the same time, I think it'd be a good idea to enable GDB stacktraces by adding debug=gdb-stacktrace in the same [maxscale] section. After that you'll need to install GDB on the MaxScale server which it will then use to generate a very detailed stacktrace from all threads at the time the problem occurs. This way if the problem does reoccur, we'll have all the information we need to fix it.

Hi There,

We updated the config as per recommendation, in place for +30 hours. A few "lost connection" events occurred this morning and a master DB failover, but no load balancer crash. Will continue to monitor.

Thanks.

MXS-4711_maxscale-logs-02.txt

Hi There,

We had a crash on stand-by load balancer #2, (#1 is/has been active for the last few days). Please see attached, let us know if you need further info.

Thanks.

MXS-4711_maxscale-logs-03.txt

Odd, it looks like all of the worker threads are idle and systemd should not kill the MaxScale process in that case. Can you verify that the systemd journal has a message for the maxscale service about the watchdog timeout being exceeded?

Hi Markus,

Yes we see several messages. Please refer to the attached. FYI you will see a few keepalived messages, we have configured keepalived as a VIP on each node in the cluster.

Thanks.

MXS-4711_maxscale-logs-04.txt MXS-4711_keepalived-config-01.txt

If this is the standby MaxScale instance that's causing problems, it could be that the kafkacdc router is the source of the problems as it'll process data even if there's no client traffic. You can confirm this by removing the service and seeing if the watchdog timeouts stop on the standby MaxScale. Given the nature of the kafkacdc router, I wouldn't expect you to need two instances to be up at all times since they'll both push duplicate events to Kafka.

Has the other MaxScale with skip_name_resolve=true had any problems so far? If this seems to make the problem go away, the explanation could be a slow reverse name lookup but this would not explain why the standby MaxScale is behaving the way it is.

Node viexh-session-usage-mdb-01 is the active node for the load balancer, this latest issue is occurring on the active node. We've experienced issues across all three nodes.

We are aware of the KafkaCDC repeated records produced across the cluster, we can live with for now.

We configured skip_name_resolve=true across the cluster, we're seeing far fewer load balancer crashes since then.

There's an internal thread in MaxScale that monitors the state of all other threads in MaxScale to make sure they aren't stuck. The upcoming 22.08.8 release will contain some improvements that will log the name of the thread that is stuck if a stuck thread is detected. Once the release is out, you could upgrade the MaxScale instances and we should see which thread is stuck.

Hello Markus,

We'll upgrade to that version when available.

The issue we have is both vSAN & vMotion traffic share the same allocated bandwidth through the physical host NICs, so whenever a vMotion occurs the associated data transfer chokes everything else. We're currently running the two host 10gb NICs as active/standby, we will configure them as active/active thereby doubling throughput to 20gb as see if that helps. Please close the the ticket if you need to.

Thanks.

OK, I think that means that it's most likely a DNS request that's blocking MaxScale and it's just not fast enough to respond to the SystemD watchdog requests to be considered alive. Do you notice a slowdown in the client applications whenever this happens? If you do, this would be supporting evidence to the theory of DNS lookups causing it.

Hello Markus,

It's more of a complete stop after these events, when the failover usually occurs connections flip to the next available load balancer, Unsure if this is related to a DNS issue, what we observe after most failover events is the Java apps are unable to get to the DB backends and report "Exhausted to Serve". At this point, most times the connection count remains low, and even when connection count does remain high, the only way to resolve this is by restarting the apps;

{{2023-08-26 07:19:33,991 ERROR com.telstra.mds.extrahop.kafka.consumer.AccountingListener [mds-cpvnf-extrahop-10-C-1] All Retry-Attempts=201 Exhausted to Serve AcctRecord=

}}

Thanks

I'll close this ticket now that most of the problems have been solved. There's an open issue (MXS-4710) for fixing the cases where a slow DNS server can cause the MaxScale process to be killed. I filed MXS-4778 for improving the handling of the case where the DNS lookups are indeed the cause of the aborts.