[MXS-2570]  Fatal: MaxScale 2.3.8 received fatal signal 11. Attempting backtrace. Created: 2019-06-21  Updated: 2020-08-27  Resolved: 2020-08-27

Status: Closed
Project: MariaDB MaxScale
Component/s: cache, ccrfilter
Affects Version/s: 2.3.8
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Ihor Horhul Assignee: Johan Wikman
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

Debian 9

Sprint: MXS-SPRINT-85

 Description   

Maxscale crashes with signal 11 If i am using cache mode:

2019-06-21 07:37:10   alert  : (42082954) Fatal: MaxScale 2.3.8 received fatal signal 11. Attempting backtrace.
 
2019-06-21 07:37:10   alert  : (42082954) Commit ID: 814ae196e9e6b2823e5e3ffac174ff5270e7d9d5 System name: Linux Release string: undefined
 
2019-06-21 07:37:14   alert  : (42082954)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x5579de3c0c1b]: /home/vagrant/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
 
2019-06-21 07:37:18   alert  : (42082954)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x39) [0x5579de3c0f89]: /usr/include/c++/6/functional:1689
 
2019-06-21 07:37:20   alert  : (42082954)   /usr/bin/maxscale(+0x9756) [0x5579de3bd756]: /home/vagrant/MaxScale/server/core/gateway.cc:437
 
2019-06-21 07:37:22   alert  : (42082954)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7fd7df99d0e0]: ??:?
 
2019-06-21 07:37:26   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN11CacheSimple12do_refreshedERK9cache_keyPK18CacheFilterSession+0x1c) [0x7fd7d40a9e8c]: /usr/include/c++/6/bits/hashtable_policy.h:446
 
2019-06-21 07:37:30   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN7CacheMT9refreshedERK9cache_keyPK18CacheFilterSession+0x47) [0x7fd7d40a7bb7]: /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:777
 
2019-06-21 07:37:34   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession12store_resultEv+0x5a) [0x7fd7d40a5dfa]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:790
 
2019-06-21 07:37:38   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession21handle_expecting_rowsEv+0x126) [0x7fd7d40a6f86]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:634
 
2019-06-21 07:37:42   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession23handle_expecting_fieldsEv+0xa0) [0x7fd7d40a7050]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:477
 
2019-06-21 07:37:46   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession25handle_expecting_responseEv+0xb6) [0x7fd7d40a7116]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:588
 
2019-06-21 07:37:47   alert  : (42083006) Fatal: MaxScale 2.3.8 received fatal signal 6. Attempting backtrace.
 
2019-06-21 07:37:47   alert  : (42083006) Commit ID: 814ae196e9e6b2823e5e3ffac174ff5270e7d9d5 System name: Linux Release string: undefined
 
2019-06-21 07:37:50   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE11clientReplyEP10mxs_filterP18mxs_filter_sessionP5gwbuf+0x14) [0x7fd7d40a4a94]: ??:0
 
2019-06-21 07:37:51   alert  : (42083006)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x5579de3c0c1b]: /home/vagrant/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
 
2019-06-21 07:37:54   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(session_route_reply+0x15) [0x7fd7dfee1a75]: /home/vagrant/MaxScale/server/core/session.cc:688
 
2019-06-21 07:37:56   alert  : (42083006)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x39) [0x5579de3c0f89]: /usr/include/c++/6/functional:1689
 
2019-06-21 07:37:58   alert  : (42083006)   /usr/bin/maxscale(+0x9756) [0x5579de3bd756]: /home/vagrant/MaxScale/server/core/gateway.cc:437
 
2019-06-21 07:37:59   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5gwbufP3dcb+0x295) [0x7fd7d4af67e5]: /home/vagrant/MaxScale/server/modules/routing/readwritesplit/rwsplitsession.cc:773
 
2019-06-21 07:38:00   alert  : (42083006)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7fd7df99d0e0]: ??:?
 
2019-06-21 07:38:02   alert  : (42083006)   /lib/x86_64-linux-gnu/libpthread.so.0(+0xff7c) [0x7fd7df99bf7c]: ??:?
 
2019-06-21 07:38:03   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5gwbufP3dcb+0x27) [0x7fd7d4aef9d7]: /home/vagrant/MaxScale/include/maxscale/router.hh:206
 
2019-06-21 07:38:05   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x4131) [0x7fd7d46d4131]: /home/vagrant/MaxScale/server/modules/protocol/MySQL/mariadbbackend/mysql_backend.cc:1041
 
2019-06-21 07:38:06   alert  : (42083006)   /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x75) [0x7fd7df995bb5]: ??:?
 
2019-06-21 07:38:07   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x81877) [0x7fd7dfe96877]: /home/vagrant/MaxScale/server/core/dcb.cc:3143
 
2019-06-21 07:38:09   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x81931) [0x7fd7dfe96931]: /home/vagrant/MaxScale/server/core/dcb.cc:3235
 
2019-06-21 07:38:10   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN7CacheMT12must_refreshERK9cache_keyPK18CacheFilterSession+0x33) [0x7fd7d40a7a93]: /usr/include/c++/6/bits/std_mutex.h:106
 
2019-06-21 07:38:13   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1a6) [0x7fd7dfeeff46]: /home/vagrant/MaxScale/maxutils/maxbase/src/worker.cc:844
 
2019-06-21 07:38:14   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession12route_SELECTENS_14cache_action_tERK10CacheRulesP5gwbuf+0x2c8) [0x7fd7d40a6808]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:1075
 
2019-06-21 07:38:17   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53) [0x7fd7dfef0163]: /home/vagrant/MaxScale/maxutils/maxbase/src/worker.cc:545
 
2019-06-21 07:38:18   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession10routeQueryEP5gwbuf+0x153) [0x7fd7d40a6ad3]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:347
 
2019-06-21 07:38:19   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9e6f) [0x7fd7de8c8e6f]: ??:?
 
2019-06-21 07:38:21   alert  : (42082954)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4) [0x7fd7df9934a4]: ??:?
 
2019-06-21 07:38:22   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE10routeQueryEP10mxs_filterP18mxs_filter_sessionP5gwbuf+0x14) [0x7fd7d40a4b84]: /home/vagrant/MaxScale/include/maxscale/filter.hh:292
 
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
 
2019-06-21 07:38:26   alert  : (42082954)   /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fd7dd790d0f]: ??:0

Config:

[maxscale]
 
threads=12
 
 
 
[Cache]
 
type=filter
 
module=cache
 
hard_ttl=30
 
soft_ttl=10
 
cached_data=shared
 
selects=verify_cacheable
 
#max_size=64G
 
 
 
[CCRFilter]
 
type=filter
 
module=ccrfilter
 
time=1
 
 
 
[db-1]
 
type=server
 
address=10.0.0.2
 
port=3306
 
protocol=mariadbbackend
 
serversize=4
 
 
 
[db-2]
 
type=server
 
address=10.0.0.3
 
port=3306
 
protocol=mariadbbackend
 
serversize=1
 
 
 
[db-3]
 
type=server
 
address=10.0.0.5
 
port=3306
 
protocol=mariadbbackend
 
serversize=4
 
 
 
[db-5]
 
type=server
 
address=10.0.0.9
 
port=3306
 
protocol=mariadbbackend
 
serversize=1
 
 
 
[db-7]
 
type=server
 
address=10.0.0.11
 
port=3306
 
protocol=mariadbbackend
 
serversize=1
 
 
 
[Galera-Monitor]
 
type=monitor
 
module=galeramon
 
servers=db-1, db-2, db-3, db-5, db-7
 
user=maxscale
 
password=123
 
monitor_interval=2000
 
 
 
[RW]
 
type=service
 
router=readwritesplit
 
servers=db-1, db-2, db-3, db-5, db-7
 
master_accept_reads=true
 
user=maxscale
 
password=gnGRCYerxrEOJWFm
 
filters=Cache | CCRFilter
 
#filters=CCRFilter
 
enable_root_user=1
 
weightby=serversize
 
 
 
#[Splitter-Listener]
 
#type=listener
 
#service=RW
 
#protocol=mariadbclient
 
#port=3306
 
#address=10.0.0.12
 
 
 
[Splitter-Listener-diff]
 
type=listener
 
service=RW
 
protocol=mariadbclient
 
port=3308
 
address=10.0.0.12



 Comments   
Comment by markus makela [ 2019-06-21 ]

Looks like two stacktraces in one log.

First:

2019-06-21 07:37:10   alert  : (42082954) Fatal: MaxScale 2.3.8 received fatal signal 11. Attempting backtrace.
 
2019-06-21 07:37:10   alert  : (42082954) Commit ID: 814ae196e9e6b2823e5e3ffac174ff5270e7d9d5 System name: Linux Release string: undefined
 
2019-06-21 07:37:14   alert  : (42082954)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x5579de3c0c1b]: /home/vagrant/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
 
2019-06-21 07:37:18   alert  : (42082954)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x39) [0x5579de3c0f89]: /usr/include/c++/6/functional:1689
 
2019-06-21 07:37:20   alert  : (42082954)   /usr/bin/maxscale(+0x9756) [0x5579de3bd756]: /home/vagrant/MaxScale/server/core/gateway.cc:437
 
2019-06-21 07:37:22   alert  : (42082954)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7fd7df99d0e0]: ??:?
 
2019-06-21 07:37:26   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN11CacheSimple12do_refreshedERK9cache_keyPK18CacheFilterSession+0x1c) [0x7fd7d40a9e8c]: /usr/include/c++/6/bits/hashtable_policy.h:446
 
2019-06-21 07:37:30   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN7CacheMT9refreshedERK9cache_keyPK18CacheFilterSession+0x47) [0x7fd7d40a7bb7]: /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:777
 
2019-06-21 07:37:34   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession12store_resultEv+0x5a) [0x7fd7d40a5dfa]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:790
 
2019-06-21 07:37:38   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession21handle_expecting_rowsEv+0x126) [0x7fd7d40a6f86]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:634
 
2019-06-21 07:37:42   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession23handle_expecting_fieldsEv+0xa0) [0x7fd7d40a7050]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:477
 
2019-06-21 07:37:46   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession25handle_expecting_responseEv+0xb6) [0x7fd7d40a7116]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:588
 
2019-06-21 07:37:50   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE11clientReplyEP10mxs_filterP18mxs_filter_sessionP5gwbuf+0x14) [0x7fd7d40a4a94]: ??:0
 
2019-06-21 07:37:54   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(session_route_reply+0x15) [0x7fd7dfee1a75]: /home/vagrant/MaxScale/server/core/session.cc:688
 
2019-06-21 07:37:59   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5gwbufP3dcb+0x295) [0x7fd7d4af67e5]: /home/vagrant/MaxScale/server/modules/routing/readwritesplit/rwsplitsession.cc:773
 
2019-06-21 07:38:03   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5gwbufP3dcb+0x27) [0x7fd7d4aef9d7]: /home/vagrant/MaxScale/include/maxscale/router.hh:206
 
2019-06-21 07:38:05   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x4131) [0x7fd7d46d4131]: /home/vagrant/MaxScale/server/modules/protocol/MySQL/mariadbbackend/mysql_backend.cc:1041
 
2019-06-21 07:38:07   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x81877) [0x7fd7dfe96877]: /home/vagrant/MaxScale/server/core/dcb.cc:3143
 
2019-06-21 07:38:09   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x81931) [0x7fd7dfe96931]: /home/vagrant/MaxScale/server/core/dcb.cc:3235
 
2019-06-21 07:38:13   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1a6) [0x7fd7dfeeff46]: /home/vagrant/MaxScale/maxutils/maxbase/src/worker.cc:844
 
2019-06-21 07:38:17   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53) [0x7fd7dfef0163]: /home/vagrant/MaxScale/maxutils/maxbase/src/worker.cc:545
 
2019-06-21 07:38:19   alert  : (42082954)   /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9e6f) [0x7fd7de8c8e6f]: ??:?
 
2019-06-21 07:38:21   alert  : (42082954)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4) [0x7fd7df9934a4]: ??:?
 
2019-06-21 07:38:26   alert  : (42082954)   /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fd7dd790d0f]: ??:0

Second:

2019-06-21 07:37:47   alert  : (42083006) Fatal: MaxScale 2.3.8 received fatal signal 6. Attempting backtrace.
 
2019-06-21 07:37:47   alert  : (42083006) Commit ID: 814ae196e9e6b2823e5e3ffac174ff5270e7d9d5 System name: Linux Release string: undefined
 
2019-06-21 07:37:51   alert  : (42083006)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x5579de3c0c1b]: /home/vagrant/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
 
2019-06-21 07:37:56   alert  : (42083006)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x39) [0x5579de3c0f89]: /usr/include/c++/6/functional:1689
 
2019-06-21 07:37:58   alert  : (42083006)   /usr/bin/maxscale(+0x9756) [0x5579de3bd756]: /home/vagrant/MaxScale/server/core/gateway.cc:437
 
2019-06-21 07:38:00   alert  : (42083006)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7fd7df99d0e0]: ??:?
 
2019-06-21 07:38:02   alert  : (42083006)   /lib/x86_64-linux-gnu/libpthread.so.0(+0xff7c) [0x7fd7df99bf7c]: ??:?
 
2019-06-21 07:38:06   alert  : (42083006)   /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x75) [0x7fd7df995bb5]: ??:?
 
2019-06-21 07:38:10   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN7CacheMT12must_refreshERK9cache_keyPK18CacheFilterSession+0x33) [0x7fd7d40a7a93]: /usr/include/c++/6/bits/std_mutex.h:106
 
2019-06-21 07:38:14   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession12route_SELECTENS_14cache_action_tERK10CacheRulesP5gwbuf+0x2c8) [0x7fd7d40a6808]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:1075
 
2019-06-21 07:38:18   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession10routeQueryEP5gwbuf+0x153) [0x7fd7d40a6ad3]: /home/vagrant/MaxScale/server/modules/filter/cache/cachefiltersession.cc:347
 
2019-06-21 07:38:22   alert  : (42083006)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE10routeQueryEP10mxs_filterP18mxs_filter_sessionP5gwbuf+0x14) [0x7fd7d40a4b84]: /home/vagrant/MaxScale/include/maxscale/filter.hh:292

Comment by Johan Wikman [ 2019-06-24 ]

xXxSPYxXx Thanks for reporting.

Does MaxScale always crash for you with this configuration or just occasionally?

Comment by Ihor Horhul [ 2019-06-24 ]

@Johan Wikman
If i am set "max_size=32G" - its crash frequent, if more or unlimit (by default) - once per day.

Comment by Johan Wikman [ 2019-07-04 ]

xXxSPYxXx Are there any cache related or other errors in the log before the crash?

With or without errors, could you provide more of the log? Please ensure there is no sensitive data in it.

Comment by markus makela [ 2020-07-03 ]

xXxSPYxXx any updates on this issue? Does it still happen with the latest 2.4 and 2.3 releases?

Comment by markus makela [ 2020-08-27 ]

Closing issue as Cannot Reproduce. If this still happens with the latest release, please let us know and we'll reopen the issue.

Generated at Thu Feb 08 04:15:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.