Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
2.5.14
-
Ubuntu 20.04, Maxscale 2.5.14, Mariadb10.5
-
MXS-SPRINT-137, MXS-SPRINT-138
Description
When using a local inmemory cachefilter with the option invalidate=current used, combined with a hard_ttl, maxscale crashes whenever the hard_ttl expires.
[Galera_Service]
|
type=service
|
router=readwritesplit
|
...
|
filters=Galera_service_cache
|
|
|
|
|
[Galera_service_cache]
|
type=filter
|
module=cache
|
max_size=128M
|
max_resultset_size=2M
|
soft_ttl=10s
|
hard_ttl=60s
|
cached_data=shared
|
invalidate=current
|
storage=storage_inmemory
|
enabled=true
|
When the options hard_ttl OR invalidate is removed, maxscale no longer crashes.
It's vital to have a hard_ttl because of the use of multiple maxscale instances in front of a Galera cluster. We cannot share the local cache over multiple maxscale instances and we cannot use an invalidate function with memcached or redis.
Maxscale.log:
2021-07-28 12:46:02 error : (59) [cache] (Galera_Service) Item in LRU list was not found in key mapping.
|
alert : MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
|
|
|
|
|
2021-07-28 12:46:02 alert : (59) (Galera_Service) MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
|
2021-07-28 12:46:02 alert : (59) (Galera_Service) Statement currently being classified: none/unknown
|
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
|
alert : /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage11remove_nodeEPNS_4NodeE+0x23): server/modules/filter/cache/lrustorage.hh:218
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage9free_nodeEPNS_4NodeE+0x19): server/modules/filter/cache/lrustorage.cc:814
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage10invalidateEPNS_4NodeENS_7ContextE+0x3e): server/modules/filter/cache/lrustorage.cc:1019
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage14LRUInvalidator10invalidateERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x249): server/modules/filter/cache/lrustorage.cc:205
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage13do_invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE+0x44): server/modules/filter/cache/lrustorage.cc:538
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN12LRUStorageMT10invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKSt8functionIFvjEE+0x4b): server/modules/filter/cache/lrustoragemt.cc:83
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x761): server/modules/filter/cache/sessioncache.hh:117
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE14apiClientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x1f): include/maxscale/filter.hh:504
|
/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x1c2): server/modules/routing/readwritesplit/rwsplitsession.cc:661
|
/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x2a): include/maxscale/router.hh:455
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbb): maxutils/maxbase/include/maxbase/log.hh:92
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN14ServerEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbe): maxutils/maxbase/include/maxbase/log.hh:92
|
/usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so.2.0.0(_ZN24MariaDBBackendConnection11normal_readEv+0x288): server/modules/protocol/MariaDB/mariadb_backend.cc:841
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1300
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x25): server/core/dcb.cc:1351
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x20e): maxutils/maxbase/src/worker.cc:879
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x57): maxutils/maxbase/src/worker.cc:574
|
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609): /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:478 (discriminator 6)
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0
|
|
|
|
|
2021-07-28 12:46:03 alert : (59) (Galera_Service)
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage11remove_nodeEPNS_4NodeE+0x23): server/modules/filter/cache/lrustorage.hh:218
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage9free_nodeEPNS_4NodeE+0x19): server/modules/filter/cache/lrustorage.cc:814
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage10invalidateEPNS_4NodeENS_7ContextE+0x3e): server/modules/filter/cache/lrustorage.cc:1019
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage14LRUInvalidator10invalidateERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x249): server/modules/filter/cache/lrustorage.cc:205
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage13do_invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE+0x44): server/modules/filter/cache/lrustorage.cc:538
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN12LRUStorageMT10invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKSt8functionIFvjEE+0x4b): server/modules/filter/cache/lrustoragemt.cc:83
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x761): server/modules/filter/cache/sessioncache.hh:117
|
/usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE14apiClientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x1f): include/maxscale/filter.hh:504
|
/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x1c2): server/modules/routing/readwritesplit/rwsplitsession.cc:661
|
/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x2a): include/maxscale/router.hh:455
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbb): maxutils/maxbase/include/maxbase/log.hh:92
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN14ServerEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbe): maxutils/maxbase/include/maxbase/log.hh:92
|
/usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so.2.0.0(_ZN24MariaDBBackendConnection11normal_readEv+0x288): server/modules/protocol/MariaDB/mariadb_backend.cc:841
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1300
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x25): server/core/dcb.cc:1351
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x20e): maxutils/maxbase/src/worker.cc:879
|
/usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x57): maxutils/maxbase/src/worker.cc:574
|
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609): /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:478 (discriminator 6)
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0
|
alert : Writing core dump.
|
|
|
|
|
MariaDB MaxScale /var/log/maxscale/maxscale.log Wed Jul 28 12:46:03 2021
|
----------------------------------------------------------------------------
|
2021-07-28 12:46:04 notice : The systemd watchdog is Enabled. Internal timeout = 30s
|
2021-07-28 12:46:04 notice : Worker message queue size: 1MiB
|
2021-07-28 12:46:04 notice : Using up to 590.35MiB of memory for query classifier cache
|
2021-07-28 12:46:04 notice : syslog logging is enabled.
|
2021-07-28 12:46:04 notice : maxlog logging is enabled.
|
2021-07-28 12:46:04 notice : Running OS: Linux@5.4.0-80-generic, #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021, x86_64 with 4 processor cores.
|
2021-07-28 12:46:04 notice : Total usable main memory: 3.84GiB.
|
2021-07-28 12:46:04 notice : MariaDB MaxScale 2.5.14 started (Commit: e63d226705ff236ace9cd8b135fbab0d67f14a13)
|
2021-07-28 12:46:04 notice : MaxScale is running in process 358170
|
2021-07-28 12:46:04 notice : Configuration file: /etc/maxscale.cnf
|
2021-07-28 12:46:04 notice : Log directory: /var/log/maxscale
|
2021-07-28 12:46:04 notice : Data directory: /var/lib/maxscale
|
2021-07-28 12:46:04 notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
|
2021-07-28 12:46:04 notice : Service cache: /var/cache/maxscale
|
2021-07-28 12:46:04 notice : Working directory: /var/log/maxscale
|
2021-07-28 12:46:04 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so
|
2021-07-28 12:46:04 notice : Query classification results are cached and reused. Memory used per thread: 590.35MiB
|
2021-07-28 12:46:04 notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
|
2021-07-28 12:46:04 notice : MaxScale started with 1 worker threads, each with a stack size of 8388608 bytes.
|
2021-07-28 12:46:04 notice : Loading /etc/maxscale.cnf.
|
2021-07-28 12:46:04 notice : /etc/maxscale.cnf.d does not exist, not reading.
|
2021-07-28 12:46:04 notice : Loaded module cache: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libcache.so
|
2021-07-28 12:46:04 notice : Loaded module galeramon: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libgaleramon.so
|
2021-07-28 12:46:04 warning: Protocol module 'mysqlclient' has been deprecated, use 'mariadbclient' instead.
|
2021-07-28 12:46:04 notice : Loaded module mariadbclient: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so
|
2021-07-28 12:46:04 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so
|
2021-07-28 12:46:04 notice : [cache] Creating shared cache.
|
2021-07-28 12:46:04 notice : [storage_inmemory] Storage module created.
|
2021-07-28 12:46:04 notice : [cache] Created multi threaded LRU storage.
|
2021-07-28 12:46:04 notice : [cache] Created multi threaded cache.
|
2021-07-28 12:46:04 notice : (Galera_Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbauth.so
|
2021-07-28 12:46:04 warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption.
|
2021-07-28 12:46:04 notice : Started REST API on [127.0.0.1]:8989
|
2021-07-28 12:46:04 notice : 's1' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
|
2021-07-28 12:46:04 notice : Server 's1' charset: latin1
|
2021-07-28 12:46:04 notice : 's2' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
|
2021-07-28 12:46:04 notice : Server 's2' charset: latin1
|
2021-07-28 12:46:04 notice : 's3' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
|
2021-07-28 12:46:04 notice : Server 's3' charset: latin1
|
2021-07-28 12:46:04 notice : Starting a total of 1 services...
|
2021-07-28 12:46:04 notice : (Galera_Listener) Listening for connections at [::]:3306
|
2021-07-28 12:46:04 notice : Service 'Galera_Service' started (1/1)
|
2021-07-28 12:46:04 notice : Read 14 user@host entries from 's1' for service 'Galera_Service'.
|
alert : MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
|