Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-3694

MXS - crash when cache is used with invalidate and hard_ttl

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.14
    • Fix Version/s: 2.5.15
    • Component/s: cache
    • Labels:
    • Environment:
      Ubuntu 20.04, Maxscale 2.5.14, Mariadb10.5
    • Sprint:
      MXS-SPRINT-137, MXS-SPRINT-138

      Description

      When using a local inmemory cachefilter with the option invalidate=current used, combined with a hard_ttl, maxscale crashes whenever the hard_ttl expires.

      [Galera_Service]
      type=service
      router=readwritesplit
      ...
      filters=Galera_service_cache
       
       
      [Galera_service_cache]
      type=filter
      module=cache
      max_size=128M
      max_resultset_size=2M
      soft_ttl=10s
      hard_ttl=60s
      cached_data=shared
      invalidate=current
      storage=storage_inmemory
      enabled=true
      

      When the options hard_ttl OR invalidate is removed, maxscale no longer crashes.
      It's vital to have a hard_ttl because of the use of multiple maxscale instances in front of a Galera cluster. We cannot share the local cache over multiple maxscale instances and we cannot use an invalidate function with memcached or redis.

      Maxscale.log:

      2021-07-28 12:46:02   error  : (59) [cache] (Galera_Service) Item in LRU list was not found in key mapping.
      alert  : MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
       
       
      2021-07-28 12:46:02   alert  : (59) (Galera_Service) MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
      2021-07-28 12:46:02   alert  : (59) (Galera_Service) Statement currently being classified: none/unknown
      nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
      alert  :   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage11remove_nodeEPNS_4NodeE+0x23): server/modules/filter/cache/lrustorage.hh:218
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage9free_nodeEPNS_4NodeE+0x19): server/modules/filter/cache/lrustorage.cc:814
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage10invalidateEPNS_4NodeENS_7ContextE+0x3e): server/modules/filter/cache/lrustorage.cc:1019
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage14LRUInvalidator10invalidateERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x249): server/modules/filter/cache/lrustorage.cc:205
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage13do_invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE+0x44): server/modules/filter/cache/lrustorage.cc:538
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN12LRUStorageMT10invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKSt8functionIFvjEE+0x4b): server/modules/filter/cache/lrustoragemt.cc:83
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x761): server/modules/filter/cache/sessioncache.hh:117
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE14apiClientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x1f): include/maxscale/filter.hh:504
        /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x1c2): server/modules/routing/readwritesplit/rwsplitsession.cc:661
        /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x2a): include/maxscale/router.hh:455
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbb): maxutils/maxbase/include/maxbase/log.hh:92
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN14ServerEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbe): maxutils/maxbase/include/maxbase/log.hh:92
        /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so.2.0.0(_ZN24MariaDBBackendConnection11normal_readEv+0x288): server/modules/protocol/MariaDB/mariadb_backend.cc:841
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1300
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x25): server/core/dcb.cc:1351
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x20e): maxutils/maxbase/src/worker.cc:879
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x57): maxutils/maxbase/src/worker.cc:574
        /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609): /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:478 (discriminator 6)
        /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0
       
       
      2021-07-28 12:46:03   alert  : (59) (Galera_Service) 
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage11remove_nodeEPNS_4NodeE+0x23): server/modules/filter/cache/lrustorage.hh:218
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage9free_nodeEPNS_4NodeE+0x19): server/modules/filter/cache/lrustorage.cc:814
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage10invalidateEPNS_4NodeENS_7ContextE+0x3e): server/modules/filter/cache/lrustorage.cc:1019
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage14LRUInvalidator10invalidateERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x249): server/modules/filter/cache/lrustorage.cc:205
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage13do_invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE+0x44): server/modules/filter/cache/lrustorage.cc:538
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN12LRUStorageMT10invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKSt8functionIFvjEE+0x4b): server/modules/filter/cache/lrustoragemt.cc:83
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x761): server/modules/filter/cache/sessioncache.hh:117
        /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE14apiClientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x1f): include/maxscale/filter.hh:504
        /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x1c2): server/modules/routing/readwritesplit/rwsplitsession.cc:661
        /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x2a): include/maxscale/router.hh:455
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbb): maxutils/maxbase/include/maxbase/log.hh:92
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN14ServerEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbe): maxutils/maxbase/include/maxbase/log.hh:92
        /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so.2.0.0(_ZN24MariaDBBackendConnection11normal_readEv+0x288): server/modules/protocol/MariaDB/mariadb_backend.cc:841
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1300
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x25): server/core/dcb.cc:1351
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x20e): maxutils/maxbase/src/worker.cc:879
        /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x57): maxutils/maxbase/src/worker.cc:574
        /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609): /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:478 (discriminator 6)
        /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0
      alert  : Writing core dump.
       
       
      MariaDB MaxScale  /var/log/maxscale/maxscale.log  Wed Jul 28 12:46:03 2021
      ----------------------------------------------------------------------------
      2021-07-28 12:46:04   notice : The systemd watchdog is Enabled. Internal timeout = 30s
      2021-07-28 12:46:04   notice : Worker message queue size: 1MiB
      2021-07-28 12:46:04   notice : Using up to 590.35MiB of memory for query classifier cache
      2021-07-28 12:46:04   notice : syslog logging is enabled.
      2021-07-28 12:46:04   notice : maxlog logging is enabled.
      2021-07-28 12:46:04   notice : Running OS: Linux@5.4.0-80-generic, #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021, x86_64 with 4 processor cores.
      2021-07-28 12:46:04   notice : Total usable main memory: 3.84GiB.
      2021-07-28 12:46:04   notice : MariaDB MaxScale 2.5.14 started (Commit: e63d226705ff236ace9cd8b135fbab0d67f14a13)
      2021-07-28 12:46:04   notice : MaxScale is running in process 358170
      2021-07-28 12:46:04   notice : Configuration file: /etc/maxscale.cnf
      2021-07-28 12:46:04   notice : Log directory: /var/log/maxscale
      2021-07-28 12:46:04   notice : Data directory: /var/lib/maxscale
      2021-07-28 12:46:04   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
      2021-07-28 12:46:04   notice : Service cache: /var/cache/maxscale
      2021-07-28 12:46:04   notice : Working directory: /var/log/maxscale
      2021-07-28 12:46:04   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so
      2021-07-28 12:46:04   notice : Query classification results are cached and reused. Memory used per thread: 590.35MiB
      2021-07-28 12:46:04   notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
      2021-07-28 12:46:04   notice : MaxScale started with 1 worker threads, each with a stack size of 8388608 bytes.
      2021-07-28 12:46:04   notice : Loading /etc/maxscale.cnf.
      2021-07-28 12:46:04   notice : /etc/maxscale.cnf.d does not exist, not reading.
      2021-07-28 12:46:04   notice : Loaded module cache: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libcache.so
      2021-07-28 12:46:04   notice : Loaded module galeramon: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libgaleramon.so
      2021-07-28 12:46:04   warning: Protocol module 'mysqlclient' has been deprecated, use 'mariadbclient' instead.
      2021-07-28 12:46:04   notice : Loaded module mariadbclient: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so
      2021-07-28 12:46:04   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so
      2021-07-28 12:46:04   notice : [cache] Creating shared cache.
      2021-07-28 12:46:04   notice : [storage_inmemory] Storage module created.
      2021-07-28 12:46:04   notice : [cache] Created multi threaded LRU storage.
      2021-07-28 12:46:04   notice : [cache] Created multi threaded cache.
      2021-07-28 12:46:04   notice : (Galera_Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbauth.so
      2021-07-28 12:46:04   warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption.
      2021-07-28 12:46:04   notice : Started REST API on [127.0.0.1]:8989
      2021-07-28 12:46:04   notice : 's1' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
      2021-07-28 12:46:04   notice : Server 's1' charset: latin1
      2021-07-28 12:46:04   notice : 's2' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
      2021-07-28 12:46:04   notice : Server 's2' charset: latin1
      2021-07-28 12:46:04   notice : 's3' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
      2021-07-28 12:46:04   notice : Server 's3' charset: latin1
      2021-07-28 12:46:04   notice : Starting a total of 1 services...
      2021-07-28 12:46:04   notice : (Galera_Listener) Listening for connections at [::]:3306
      2021-07-28 12:46:04   notice : Service 'Galera_Service' started (1/1)
      2021-07-28 12:46:04   notice : Read 14 user@host entries from 's1' for service 'Galera_Service'.
      alert  : MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
      

        Attachments

          Activity

            People

            Assignee:
            johan.wikman Johan Wikman
            Reporter:
            avannorden Arnold van Norden
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration