[MXS-4870] Maxscale dump in RoutingWorker::execute_concurrently Created: 2023-11-22  Updated: 2024-01-22  Resolved: 2024-01-22

Status: Closed
Project: MariaDB MaxScale
Component/s: N/A
Affects Version/s: 22.08.3
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Jiri Srba Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: crash
Environment:

production



 Description   

The maxscale process failed with the following backtrace.

This is the performance issue. The one maxscale is capable up to 5k connections. Over the 5k connection, the response is quite slow .. and after few minutes the maxscale crash.

2023-11-21 11:57:17 alert : MaxScale 22.08.4 received fatal signal 6. Commit ID: 9be47263ae2ff2c7bdf4c3d1334fb86a7de53b77 System name: Li
nux Release string: undefined
2023-11-21 11:57:17 alert : Statement currently being classified: none/unknown
2023-11-21 11:57:17 notice : For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscale] section.
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11174): ??:?
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11278): ??:?
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker20execute_concurrentlyERKSt8functionIFvvEE+0x52)
: maxutils/maxbase/include/maxbase/semaphore.hh:146
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale12WorkerGlobalISt13unordered_mapIjmSt4hashIjESt8equal_toIjESaIS
t4pairIKjmEEEE6assignERKSA_0xdb): /usr/include/c+/10/bits/std_function.h:244
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(0x2ae40d): /usr/include/c+/10/bits/hashtable.h:421
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x79): maxutils/maxbase/src/worker.cc:738
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase17EventMessageQueue18handle_poll_eventsEPNS_6WorkerEjNS_8Pollable7ContextE+0x73): maxutils/maxbase/src/messagequeue.cc:220
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14deliver_eventsEmNSt6chrono10time_pointINS1_3_V212steady_clockENS1_8durationIlSt5ratioILl1ELl1000000000EEEEEEPNS_8PollableEjNSA_7ContextE+0x8d): maxutils/maxbase/src/worker.cc:1043
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x344): maxutils/maxbase/src/worker.cc:1168
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x4f): maxutils/maxbase/src/worker.cc:822
/usr/bin/maxscale(main+0x26fd): server/core/gateway.cc:2243
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea): ??:0
/usr/bin/maxscale(_start+0x2a): ??:?
alert : Writing core dump.

The current maxscale configuration

[maxscale]
threads=auto
 
[db1]
type=server
address=db-1
port=3306
protocol=MariaDBBackend
 
[db2]
type=server
address=db-2
port=3306
protocol=MariaDBBackend
 
[db3]
type=server
address=db-3
port=3306
protocol=MariaDBBackend
 
[Read-Write-Service]
type=service
router=readwritesplit
servers=db1,db2,db3
user=maxscale
password=<redacted>
connection_keepalive=300s
master_accept_reads=true
max_slave_replication_lag=2s
filters=Hint
 
[Read-Write-Listener]
type=listener
service=Read-Write-Service
protocol=MariaDBClient
port=3306



 Comments   
Comment by Johan Wikman [ 2023-11-22 ]

Signal 6 typically means that MaxScale was killed by the systemd watchdog.

You could try to replace the domain names (db-1 etc) with their real IPs, as domain name lookup may take a while and have been known to cause problems.

Also follow the instructions from the log

2023-11-21 11:57:17 notice : For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscale] section.

as that will provide a more informative stack-trace.

Comment by markus makela [ 2023-12-04 ]

You can also try to install MaxScale 23.08 which has a built-in functionality for dumping stacktraces of all threads.

Comment by Jiri Srba [ 2023-12-04 ]

upgraded to maxscale 23.08 version, thanks

Comment by markus makela [ 2024-01-08 ]

jiri.srba any updates?

Comment by markus makela [ 2024-01-22 ]

I'll close this as Incomplete due to no feedback with the latest version. If this occurs with the latest version again, please report back and we'll reopen it.

Generated at Thu Feb 08 04:31:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.