Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2585

MaxScale dies with fatal signal 11

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.2.16, 2.3.4, 2.3.6, 2.3.9, 2.4.1, 2.4.4, 2.4.6, 2.4.10
    • Fix Version/s: 2.4.11
    • Component/s: cache
    • Labels:
    • Environment:
      Ubuntu 16.04 LTS, Ubuntu 18.04 LTS
    • Sprint:
      MXS-SPRINT-96, MXS-SPRINT-98, MXS-SPRINT-101

      Description

      I am experiencing an issue that MaxScale is periodically dying with the following backtrace:

      2019-07-02 15:25:02   alert  : (169770) Fatal: MaxScale 2.3.6 received fatal signal 11. Attempting backtrace.
      2019-07-02 15:25:02   alert  : (169770) Commit ID: 03dc969cf203decb4d0bc65280fe4f901da87830 System name: Linux Release string: Ubuntu 18.04.2 LTS
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:02   alert  : (169770)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x3a) [0x558bd60300da]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:02   alert  : (169770)   /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x48) [0x558bd60304b8]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:02   alert  : (169770)   /usr/bin/maxscale(+0x99f0) [0x558bd602c9f0]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:02   alert  : (169770)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fc2afbc7890]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:03   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage12do_put_valueERK9cache_keyPK5gwbuf+0xe8) [0x7fc2a6281178]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:03   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession12store_resultEv+0x39) [0x7fc2a627b049]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:03   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession21handle_expecting_rowsEv+0xea) [0x7fc2a627c2ca]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:03   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE11clientReplyEP10mxs_filterP18mxs_filter_sessionP5gwbuf+0x14) [0x7fc2a6279c44]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:03   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(session_route_reply+0x15) [0x7fc2b0392db5]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:04   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5gwbufP3dcb+0x127) [0x7fc2a68be077]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:04   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5gwbufP3dcb+0x27) [0x7fc2a68b7687]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:04   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libmariadbbackend.so(+0x4138) [0x7fc2a649c138]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:04   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x82f77) [0x7fc2b0344f77]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:04   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x83031) [0x7fc2b0345031]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:05   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1b6) [0x7fc2b03a1ce6]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:05   alert  : (169770)   /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x4f) [0x7fc2b03a1eff]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:05   alert  : (169770)   /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd66f) [0x7fc2af16866f]: 
      sh: 1: addr2line: not found
      2019-07-02 15:25:05   alert  : (169770)   /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fc2afbbc6db]: 
      sh: 1: nm: not found
      sh: 1: addr2line: not found
      2019-07-02 15:25:05   alert  : (169770)   /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fc2ae4c088f]: 
      

      Right after that monit detects, that the service isn't running and starts MaxScale again.

      The issue is surely situated around the memory consumption through caching. But I limited it at the configuration. Anyway it keeps dying. I tried different max_sizes as same threaded/shared operation mode. At the beginning of my investigation of that issue the system was running OOM. But I through that it should not happen based on the hard and soft TTL set.

      I attached the maxscale configuration as same as the only one used filter_rule.

      System itself has 8 cores/threads with 16 Gb of RAM and is virtualized with Qemu/KVM.
      Kernel-Version: 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

      Backend database Servers are: Server version: 10.2.18-MariaDB-10.2.18+maria~xenial

      Looking forward for any hints.

      –
      Greetings

        Attachments

        1. caching_rules.json
          0.1 kB
        2. maxscale.cnf
          1 kB
        3. maxscale.log
          29 kB
        4. maxscale-2-4-10-verbose-log.txt
          103 kB
        5. maxscale-verbose-2-4-6-1-debug.log
          15 kB
        6. maxscale-verbose-log.cnf
          1 kB

          Activity

            People

            Assignee:
            markus makela markus makela
            Reporter:
            Marckardt Daniel
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: