Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
2.2.16, 2.3.4, 2.3.6, 2.3.9, 2.4.1, 2.4.4, 2.4.6, 2.4.10
-
None
-
Ubuntu 16.04 LTS, Ubuntu 18.04 LTS
-
MXS-SPRINT-96, MXS-SPRINT-98, MXS-SPRINT-101
Description
I am experiencing an issue that MaxScale is periodically dying with the following backtrace:
2019-07-02 15:25:02 alert : (169770) Fatal: MaxScale 2.3.6 received fatal signal 11. Attempting backtrace.
|
2019-07-02 15:25:02 alert : (169770) Commit ID: 03dc969cf203decb4d0bc65280fe4f901da87830 System name: Linux Release string: Ubuntu 18.04.2 LTS
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:02 alert : (169770) /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x3a) [0x558bd60300da]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:02 alert : (169770) /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x48) [0x558bd60304b8]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:02 alert : (169770) /usr/bin/maxscale(+0x99f0) [0x558bd602c9f0]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:02 alert : (169770) /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fc2afbc7890]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:03 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage12do_put_valueERK9cache_keyPK5gwbuf+0xe8) [0x7fc2a6281178]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:03 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession12store_resultEv+0x39) [0x7fc2a627b049]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:03 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession21handle_expecting_rowsEv+0xea) [0x7fc2a627c2ca]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:03 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE11clientReplyEP10mxs_filterP18mxs_filter_sessionP5gwbuf+0x14) [0x7fc2a6279c44]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:03 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(session_route_reply+0x15) [0x7fc2b0392db5]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:04 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5gwbufP3dcb+0x127) [0x7fc2a68be077]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:04 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5gwbufP3dcb+0x27) [0x7fc2a68b7687]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:04 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x4138) [0x7fc2a649c138]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:04 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x82f77) [0x7fc2b0344f77]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:04 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x83031) [0x7fc2b0345031]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:05 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1b6) [0x7fc2b03a1ce6]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:05 alert : (169770) /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x4f) [0x7fc2b03a1eff]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:05 alert : (169770) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd66f) [0x7fc2af16866f]:
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:05 alert : (169770) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc2afbbc6db]:
|
sh: 1: nm: not found
|
sh: 1: addr2line: not found
|
2019-07-02 15:25:05 alert : (169770) /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc2ae4c088f]:
|
Right after that monit detects, that the service isn't running and starts MaxScale again.
The issue is surely situated around the memory consumption through caching. But I limited it at the configuration. Anyway it keeps dying. I tried different max_sizes as same threaded/shared operation mode. At the beginning of my investigation of that issue the system was running OOM. But I through that it should not happen based on the hard and soft TTL set.
I attached the maxscale configuration as same as the only one used filter_rule.
System itself has 8 cores/threads with 16 Gb of RAM and is virtualized with Qemu/KVM.
Kernel-Version: 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Backend database Servers are: Server version: 10.2.18-MariaDB-10.2.18+maria~xenial
Looking forward for any hints.
–
Greetings