[MXS-2721] MaxScale 2.4.2 crashes on different lxc containers with similar error Created: 2019-10-09  Updated: 2019-10-29  Resolved: 2019-10-10

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 2.4.2
Fix Version/s: 2.3.13, 2.4.3

Type: Bug Priority: Critical
Reporter: Anton A. Grishin Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS Linux release 7.7.1908 in lxc container.



 Description   

Hello.

I noticed periodic service fails on different servers with similar settings. Below are the traces from different servers at different times.

server1:
 
2019-10-09 15:43:15   alert  : Fatal: MaxScale 2.4.2 received fatal signal 11. Attempting backtrace.
2019-10-09 15:43:15   alert  : Commit ID: aad4148d77bf2dfbaa0042bc45abda30c101cad2 System name: Linux Release string: CentOS Linux release 7.7.1908 (Core)
2019-10-09 15:43:15   alert  :   /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(_ZN11LocalClient11queue_queryEP5GWBUF+0): server/modules/protocol/MySQL/mariadb_client.cc:52
2019-10-09 15:43:15   alert  :   /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(+0x738b): server/modules/protocol/MySQL/mysql_common.cc:1425
2019-10-09 15:43:15   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:291
2019-10-09 15:43:15   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x196): maxutils/maxbase/src/worker.cc:858
2019-10-09 15:43:15   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:559
2019-10-09 15:43:15   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1ad83f): thread48.o:?
2019-10-09 15:43:15   alert  :   /lib64/libpthread.so.0(+0x7e65): pthread_create.c:?
2019-10-09 15:43:15   alert  :   /lib64/libc.so.6(clone+0x6d): ??:?
 
server2:
 
2019-10-09 09:25:14   alert  : Commit ID: aad4148d77bf2dfbaa0042bc45abda30c101cad2 System name: Linux Release string: NAME="CentOS Linux"
2019-10-09 09:25:14   alert  :   /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(_ZN11LocalClient11queue_queryEP5GWBUF+0): server/modules/protocol/MySQL/mariadb_client.cc:52
2019-10-09 09:25:14   alert  :   /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(+0x738b): server/modules/protocol/MySQL/mysql_common.cc:1425
2019-10-09 09:25:14   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:291
2019-10-09 09:25:14   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x196): maxutils/maxbase/src/worker.cc:858
2019-10-09 09:25:14   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:559
2019-10-09 09:25:15   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1ad83f): thread48.o:?
2019-10-09 09:25:15   alert  :   /lib64/libpthread.so.0(+0x7dd5): pthread_create.c:?
2019-10-09 09:25:15   alert  :   /lib64/libc.so.6(clone+0x6d): ??:?
 
 
server3:
 
2019-10-07 06:14:23   alert  : Fatal: MaxScale 2.4.2 received fatal signal 11. Attempting backtrace.
2019-10-07 06:14:23   alert  : Commit ID: aad4148d77bf2dfbaa0042bc45abda30c101cad2 System name: Linux Release string: CentOS Linux release 7.7.1908 (Core)
2019-10-07 06:14:23   alert  :   /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(_ZN11LocalClient11queue_queryEP5GWBUF+0): server/modules/protocol/MySQL/mariadb_client.cc:52
2019-10-07 06:14:23   alert  :   /usr/lib64/maxscale/libmysqlcommon.so.2.0.0(+0x738b): server/modules/protocol/MySQL/mysql_common.cc:1425
2019-10-07 06:14:23   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:291
2019-10-07 06:14:23   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x196): maxutils/maxbase/src/worker.cc:858
2019-10-07 06:14:23   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:559
2019-10-07 06:14:23   alert  :   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1ad83f): thread48.o:?
2019-10-07 06:14:23   alert  :   /lib64/libpthread.so.0(+0x7e65): pthread_create.c:?
2019-10-07 06:14:23   alert  :   /lib64/libc.so.6(clone+0x6d): ??:?

all servers have similar config:

[maxscale]
threads=8
query_retries=3
 
[server-1]
type=server
address=server-1
port=2306
protocol=mariadbbackend
 
[server-2]
type=server
address=server-2
port=2306
protocol=mariadbbackend
 
[TheMonitor]
type=monitor
module=mariadbmon
servers=server-1,server-2
user=maxscale
password=xxxxxxxxx
monitor_interval=1000
backend_connect_timeout=10
backend_write_timeout=10
backend_read_timeout=10
detect_stale_master=true
 
[Write-Service]
type=service
router=readconnroute
router_options=master
servers=server-1,server-2
 
[Write-Listener]
type=listener
service=Write-Service
protocol=MySQLClient
port=3306



 Comments   
Comment by markus makela [ 2019-10-10 ]

Looks like the crash is happening when a query is being killed by another connection. Have you noticed any network errors at the time of the crash? Looking at the code it looks like that might cause it.

Generated at Thu Feb 08 04:16:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.