[MXS-2727] Maxscale 2.4.2 crash Created: 2019-10-18  Updated: 2019-11-29  Resolved: 2019-11-12

Status: Closed
Project: MariaDB MaxScale
Component/s: cache
Affects Version/s: None
Fix Version/s: 2.4.4

Type: Bug Priority: Critical
Reporter: steven lin Assignee: Johan Wikman
Resolution: Fixed Votes: 0
Labels: None
Environment:

OS:CentOS Linux release 7.7.1908 (Core)
Maxscale: Linux maxscale_189 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

MaxScale 2.4.2
ram 8G
cpu 16 core

Sprint: MXS-SPRINT-93, MXS-SPRINT-94

 Description   

error in maxscale.log
Maybe this is because I enable maxscale cache for my Percona XtrDB Cluster 5.7
So I closed maxscale cache now.

==========================
 
2019-10-01 21:38:52   alert  : (79707) Fatal: MaxScale 2.4.2 received fatal signal 11. Attempting backtrace.
 
2019-10-01 21:38:52   alert  : (79707) Commit ID: aad4148d77bf2dfbaa0042bc45abda30c101cad2 System name: Linux Release string: NAME="CentOS Linux"
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libcache.so(_ZN10LRUStorage12do_put_valueERK9cache_keyPK5GWBUF+0xc8): server/modules/filter/cache/lrustorage.hh:201
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libcache.so(_ZN12LRUStorageMT9put_valueERK9cache_keyPK5GWBUF+0x47): /opt/rh/devtoolset-7/root/usr/include/c++/7/x86_64-redhat-linux/bits/gthr-default.h:777
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libcache.so(_ZN18CacheFilterSession12store_resultEv+0x39): server/modules/filter/cache/cachefiltersession.cc:774
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libcache.so(_ZN18CacheFilterSession21handle_expecting_rowsEv+0x14e): server/modules/filter/cache/cachefiltersession.cc:634
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE11clientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUF+0x14): include/maxscale/filter.hh:550
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_Z19session_route_replyP11MXS_SESSIONP5GWBUF+0x15): server/core/session.cc:573
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFP3DCB+0x4a1): server/modules/routing/readwritesplit/rwsplitsession.cc:836
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFP3DCB+0x27): include/maxscale/router.hh:477
 
2019-10-01 21:38:52   alert  : (79707)   /usr/lib64/maxscale/libmariadbbackend.so(+0x4683): server/modules/protocol/MySQL/mariadbbackend/mysql_backend.cc:1044
 
2019-10-01 21:38:53   alert  : (79707)   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x9d487): server/core/dcb.cc:2701
 
2019-10-01 21:38:53   alert  : (79707)   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x9d511): server/core/dcb.cc:2793
 
2019-10-01 21:38:53   alert  : (79707)   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x196): maxutils/maxbase/src/worker.cc:858
 
2019-10-01 21:38:53   alert  : (79707)   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:559
 
2019-10-01 21:38:53   alert  : (79707)   /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1ad83f): thread48.o:?
 
2019-10-01 21:38:53   alert  : (79707)   /lib64/libpthread.so.0(+0x7e65): pthread_create.c:?
 
2019-10-01 21:38:53   alert  : (79707)   /lib64/libc.so.6(clone+0x6d): ??:?
 
============================


/etc/maxscale.cnf
 
[maxscale]
 
threads=auto
 
 
 
[node1]
 
type=server
 
address=10.1.0.176
 
port=3306
 
protocol=mariadbbackend
 
 
 
[node2]
 
type=server
 
address=10.1.0.177
 
port=3306
 
protocol=mariadbbackend
 
 
 
[node3]
 
type=server
 
address=10.1.0.178
 
port=3306
 
protocol=mariadbbackend
 
 
 
[node4]
 
type=server
 
address=10.1.0.179
 
port=3306
 
protocol=mariadbbackend
 
 
 
[node5]
 
type=server
 
address=10.1.0.180
 
port=3306
 
protocol=mariadbbackend
 
 
 
[backup_server]
 
type=server
 
address=10.1.0.121
 
port=3306
 
protocol=mariadbbackend
 
 
 
[PXC_Monitor]
 
type=monitor
 
module=galeramon
 
servers=node1,node2,node3,node4,node5,backup_server
 
user=maxscale_monitor
 
password=xxxxxxxxxxxxxxxxxxxxxx
 
monitor_interval=2000ms
 
 
 
 
 
[MaxScaleCache]
 
type=filter
 
module=cache
 
storage=storage_inmemory
 
soft_ttl=60s
 
hard_ttl=90s
 
cached_data=shared
 
max_size=256Mi
 
max_resultset_rows=1000
 
selects=verify_cacheable
 
enabled=true
 
 
 
[PXC_Service]
 
type=service
 
router=readwritesplit
 
slave_selection_criteria=LEAST_GLOBAL_CONNECTIONS
 
servers=node1,node2,node3,node4,node5
 
user=maxscale_monitor
 
password=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
enable_root_user=1
 
connection_timeout=7200s
 
max_connections=2000
 
max_slave_connections=1
 
filters=MaxScaleCache
 
log_auth_warnings=true
 
master_accept_reads=true
 
disable_sescmd_history=true
 
 
 
[BackUp_Mysql_Service]
 
type=service
 
router=readconnroute
 
router_options=running
 
servers=backup_server
 
user=maxscale_monitor
 
password=xxxxxxxxxxxxxxxxxxx
 
enable_root_user=1
 
 
 
[PXC_Listener]
 
type=listener
 
service=PXC_Service
 
protocol=mariadbclient
 
address=0.0.0.0
 
port=3306
 
 
 
[BackUp_Listener]
 
type=listener
 
service=BackUp_Mysql_Service
 
protocol=mariadbclient
 
address=0.0.0.0
 
port=3307



 Comments   
Comment by Johan Wikman [ 2019-10-22 ]

Does it immediately crash for you, or only after a while?
Can you say something about the load when the crash occurs?
If it is easily repeatable, could you try using a thread specific cache:

cached_data=thread_specific

Does it make a difference?
Could you consider providing a core-file?

Comment by steven lin [ 2019-10-24 ]

No, It crashed occasionally .
Linux rebooted maxscale automatically. The loading was not heavy.
This is our production server. so I can't modify the configuration for you
and I can not provide you a core-file.

Comment by Johan Wikman [ 2019-11-12 ]

For the crash to occur, the following must hold.

  • There is a entry in the cache.
  • That entry is updated with a value that is too large to be stored in the cache.

That will lead to a situation where a SIGSEGV occurs.

Generated at Thu Feb 08 04:16:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.