[MXS-3694] MXS - crash when cache is used with invalidate and hard_ttl Created: 2021-07-28  Updated: 2021-10-26  Resolved: 2021-08-17

Status: Closed
Project: MariaDB MaxScale
Component/s: cache
Affects Version/s: 2.5.14
Fix Version/s: 2.5.15

Type: Bug Priority: Critical
Reporter: Arnold van Norden Assignee: Johan Wikman
Resolution: Fixed Votes: 0
Labels: crash, performance
Environment:

Ubuntu 20.04, Maxscale 2.5.14, Mariadb10.5


Sprint: MXS-SPRINT-137, MXS-SPRINT-138

 Description   

When using a local inmemory cachefilter with the option invalidate=current used, combined with a hard_ttl, maxscale crashes whenever the hard_ttl expires.

[Galera_Service]
type=service
router=readwritesplit
...
filters=Galera_service_cache
 
 
[Galera_service_cache]
type=filter
module=cache
max_size=128M
max_resultset_size=2M
soft_ttl=10s
hard_ttl=60s
cached_data=shared
invalidate=current
storage=storage_inmemory
enabled=true

When the options hard_ttl OR invalidate is removed, maxscale no longer crashes.
It's vital to have a hard_ttl because of the use of multiple maxscale instances in front of a Galera cluster. We cannot share the local cache over multiple maxscale instances and we cannot use an invalidate function with memcached or redis.

Maxscale.log:

2021-07-28 12:46:02   error  : (59) [cache] (Galera_Service) Item in LRU list was not found in key mapping.
alert  : MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
 
 
2021-07-28 12:46:02   alert  : (59) (Galera_Service) MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
2021-07-28 12:46:02   alert  : (59) (Galera_Service) Statement currently being classified: none/unknown
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
alert  :   /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage11remove_nodeEPNS_4NodeE+0x23): server/modules/filter/cache/lrustorage.hh:218
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage9free_nodeEPNS_4NodeE+0x19): server/modules/filter/cache/lrustorage.cc:814
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage10invalidateEPNS_4NodeENS_7ContextE+0x3e): server/modules/filter/cache/lrustorage.cc:1019
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage14LRUInvalidator10invalidateERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x249): server/modules/filter/cache/lrustorage.cc:205
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage13do_invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE+0x44): server/modules/filter/cache/lrustorage.cc:538
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN12LRUStorageMT10invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKSt8functionIFvjEE+0x4b): server/modules/filter/cache/lrustoragemt.cc:83
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x761): server/modules/filter/cache/sessioncache.hh:117
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE14apiClientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x1f): include/maxscale/filter.hh:504
  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x1c2): server/modules/routing/readwritesplit/rwsplitsession.cc:661
  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x2a): include/maxscale/router.hh:455
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbb): maxutils/maxbase/include/maxbase/log.hh:92
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN14ServerEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbe): maxutils/maxbase/include/maxbase/log.hh:92
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so.2.0.0(_ZN24MariaDBBackendConnection11normal_readEv+0x288): server/modules/protocol/MariaDB/mariadb_backend.cc:841
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1300
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x25): server/core/dcb.cc:1351
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x20e): maxutils/maxbase/src/worker.cc:879
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x57): maxutils/maxbase/src/worker.cc:574
  /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609): /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:478 (discriminator 6)
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0
 
 
2021-07-28 12:46:03   alert  : (59) (Galera_Service) 
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage11remove_nodeEPNS_4NodeE+0x23): server/modules/filter/cache/lrustorage.hh:218
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK10LRUStorage9free_nodeEPNS_4NodeE+0x19): server/modules/filter/cache/lrustorage.cc:814
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage10invalidateEPNS_4NodeENS_7ContextE+0x3e): server/modules/filter/cache/lrustorage.cc:1019
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage14LRUInvalidator10invalidateERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x249): server/modules/filter/cache/lrustorage.cc:205
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN10LRUStorage13do_invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EE+0x44): server/modules/filter/cache/lrustorage.cc:538
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN12LRUStorageMT10invalidateEPN7Storage5TokenERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS9_EERKSt8functionIFvjEE+0x4b): server/modules/filter/cache/lrustoragemt.cc:83
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN18CacheFilterSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x761): server/modules/filter/cache/sessioncache.hh:117
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN8maxscale6FilterI11CacheFilter18CacheFilterSessionE14apiClientReplyEP10mxs_filterP18mxs_filter_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x1f): include/maxscale/filter.hh:504
  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN14RWSplitSession11clientReplyEP5GWBUFRKSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0x1c2): server/modules/routing/readwritesplit/rwsplitsession.cc:661
  /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(_ZN8maxscale6RouterI7RWSplit14RWSplitSessionE11clientReplyEP10mxs_routerP18mxs_router_sessionP5GWBUFRKSt6vectorIPNS_8EndpointESaISC_EERKNS_5ReplyE+0x2a): include/maxscale/router.hh:455
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN15ServiceEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbb): maxutils/maxbase/include/maxbase/log.hh:92
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN14ServerEndpoint11clientReplyEP5GWBUFRSt6vectorIPN8maxscale8EndpointESaIS5_EERKNS3_5ReplyE+0xbe): maxutils/maxbase/include/maxbase/log.hh:92
  /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so.2.0.0(_ZN24MariaDBBackendConnection11normal_readEv+0x288): server/modules/protocol/MariaDB/mariadb_backend.cc:841
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1300
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x25): server/core/dcb.cc:1351
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x20e): maxutils/maxbase/src/worker.cc:879
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x57): maxutils/maxbase/src/worker.cc:574
  /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4): ??:?
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609): /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:478 (discriminator 6)
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43): ??:0
alert  : Writing core dump.
 
 
MariaDB MaxScale  /var/log/maxscale/maxscale.log  Wed Jul 28 12:46:03 2021
----------------------------------------------------------------------------
2021-07-28 12:46:04   notice : The systemd watchdog is Enabled. Internal timeout = 30s
2021-07-28 12:46:04   notice : Worker message queue size: 1MiB
2021-07-28 12:46:04   notice : Using up to 590.35MiB of memory for query classifier cache
2021-07-28 12:46:04   notice : syslog logging is enabled.
2021-07-28 12:46:04   notice : maxlog logging is enabled.
2021-07-28 12:46:04   notice : Running OS: Linux@5.4.0-80-generic, #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021, x86_64 with 4 processor cores.
2021-07-28 12:46:04   notice : Total usable main memory: 3.84GiB.
2021-07-28 12:46:04   notice : MariaDB MaxScale 2.5.14 started (Commit: e63d226705ff236ace9cd8b135fbab0d67f14a13)
2021-07-28 12:46:04   notice : MaxScale is running in process 358170
2021-07-28 12:46:04   notice : Configuration file: /etc/maxscale.cnf
2021-07-28 12:46:04   notice : Log directory: /var/log/maxscale
2021-07-28 12:46:04   notice : Data directory: /var/lib/maxscale
2021-07-28 12:46:04   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
2021-07-28 12:46:04   notice : Service cache: /var/cache/maxscale
2021-07-28 12:46:04   notice : Working directory: /var/log/maxscale
2021-07-28 12:46:04   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so
2021-07-28 12:46:04   notice : Query classification results are cached and reused. Memory used per thread: 590.35MiB
2021-07-28 12:46:04   notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2021-07-28 12:46:04   notice : MaxScale started with 1 worker threads, each with a stack size of 8388608 bytes.
2021-07-28 12:46:04   notice : Loading /etc/maxscale.cnf.
2021-07-28 12:46:04   notice : /etc/maxscale.cnf.d does not exist, not reading.
2021-07-28 12:46:04   notice : Loaded module cache: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libcache.so
2021-07-28 12:46:04   notice : Loaded module galeramon: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libgaleramon.so
2021-07-28 12:46:04   warning: Protocol module 'mysqlclient' has been deprecated, use 'mariadbclient' instead.
2021-07-28 12:46:04   notice : Loaded module mariadbclient: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbclient.so
2021-07-28 12:46:04   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so
2021-07-28 12:46:04   notice : [cache] Creating shared cache.
2021-07-28 12:46:04   notice : [storage_inmemory] Storage module created.
2021-07-28 12:46:04   notice : [cache] Created multi threaded LRU storage.
2021-07-28 12:46:04   notice : [cache] Created multi threaded cache.
2021-07-28 12:46:04   notice : (Galera_Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib/x86_64-linux-gnu/maxscale/libmariadbauth.so
2021-07-28 12:46:04   warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption.
2021-07-28 12:46:04   notice : Started REST API on [127.0.0.1]:8989
2021-07-28 12:46:04   notice : 's1' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
2021-07-28 12:46:04   notice : Server 's1' charset: latin1
2021-07-28 12:46:04   notice : 's2' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
2021-07-28 12:46:04   notice : Server 's2' charset: latin1
2021-07-28 12:46:04   notice : 's3' sent version string '10.5.11-MariaDB-1:10.5.11+maria~focal'. Detected type: 'MariaDB', version: 10.5.11.
2021-07-28 12:46:04   notice : Server 's3' charset: latin1
2021-07-28 12:46:04   notice : Starting a total of 1 services...
2021-07-28 12:46:04   notice : (Galera_Listener) Listening for connections at [::]:3306
2021-07-28 12:46:04   notice : Service 'Galera_Service' started (1/1)
2021-07-28 12:46:04   notice : Read 14 user@host entries from 's1' for service 'Galera_Service'.
alert  : MaxScale 2.5.14 received fatal signal 11. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS



 Comments   
Comment by markus makela [ 2021-07-28 ]

I think I might have reproduced this with ASAN enabled:

=================================================================
==3232734==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000ab4be0 at pc 0x7fffeaa00975 bp 0x7fffec29c1b0 sp 0x7fffec29c1a8
READ of size 8 at 0x606000ab4be0 thread T8
    #0 0x7fffeaa00974 in LRUStorage::Node::key() const /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.hh:152
    #1 0x7fffea9fd10f in LRUStorage::free_node_data(LRUStorage::Node*, LRUStorage::Context) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:748
    #2 0x7fffeaa00175 in LRUStorage::invalidate(LRUStorage::Node*, LRUStorage::Context) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:1015
    #3 0x7fffeaa02b14 in LRUStorage::FullInvalidator::invalidate_node(LRUStorage::Node*) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:274
    #4 0x7fffeaa0221b in LRUStorage::LRUInvalidator::invalidate(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:205
    #5 0x7fffea9fbb01 in LRUStorage::do_invalidate(Storage::Token*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:538
    #6 0x7fffeaa10b6e in LRUStorageMT::invalidate(Storage::Token*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::function<void (unsigned int)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/lrustoragemt.cc:83
    #7 0x7fffea9f4078 in CacheSimple::invalidate(Storage::Token*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::function<void (unsigned int)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachesimple.cc:75
    #8 0x7fffea9d2db8 in SessionCache::invalidate(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::function<void (unsigned int)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/sessioncache.hh:117
    #9 0x7fffea9c7c1a in CacheFilterSession::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:567
    #10 0x7fffea9c2f45 in maxscale::Filter<CacheFilter, CacheFilterSession>::apiClientReply(mxs_filter*, mxs_filter_session*, GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/include/maxscale/filter.hh:501
    #11 0x7ffff7168dbc in maxscale::RouterSession::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/core/router.cc:43
    #12 0x7fffea5593ac in RWSplitSession::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/routing/readwritesplit/rwsplitsession.cc:658
    #13 0x7fffea525a1d in maxscale::Router<RWSplit, RWSplitSession>::clientReply(mxs_router*, mxs_router_session*, GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) (/home/markusjm/build-develop/lib64/maxscale/libreadwritesplit.so+0xbba1d)
    #14 0x7ffff71eca57 in ServiceEndpoint::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> >&, maxscale::Reply const&) /home/markusjm/MaxScale/server/core/service.cc:1458
    #15 0x7ffff71aca4b in ServerEndpoint::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> >&, maxscale::Reply const&) /home/markusjm/MaxScale/server/core/server.cc:999
    #16 0x7fffea7ec4b3 in MariaDBBackendConnection::normal_read() /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_backend.cc:834
    #17 0x7fffea7e9760 in MariaDBBackendConnection::ready_for_reading(DCB*) /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_backend.cc:381
    #18 0x7ffff7044d13 in DCB::process_events(unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1292
    #19 0x7ffff7045322 in DCB::event_handler(DCB*, unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1344
    #20 0x7ffff704553f in DCB::poll_handler(MXB_POLL_DATA*, MXB_WORKER*, unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1383
    #21 0x7ffff7299324 in maxbase::Worker::poll_waitevents() /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:877
    #22 0x7ffff729740f in maxbase::Worker::run(maxbase::Semaphore*) /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:570
    #23 0x7ffff7298448 in maxbase::Worker::thread_main(maxbase::Worker*, maxbase::Semaphore*) /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:696
    #24 0x7ffff72a4c08 in void std::__invoke_impl<void, void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>(std::__invoke_other, void (*&&)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*&&, maxbase::Semaphore*&&) /usr/include/c++/11/bits/invoke.h:61
    #25 0x7ffff72a4ad9 in std::__invoke_result<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>::type std::__invoke<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>(void (*&&)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*&&, maxbase::Semaphore*&&) /usr/include/c++/11/bits/invoke.h:96
    #26 0x7ffff72a4a0c in void std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/11/bits/std_thread.h:253
    #27 0x7ffff72a49a9 in std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> >::operator()() /usr/include/c++/11/bits/std_thread.h:260
    #28 0x7ffff72a484d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211
    #29 0x7ffff6373c83 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xd9c83)
    #30 0x7ffff6811298 in start_thread (/lib64/libpthread.so.0+0x9298)
    #31 0x7ffff5e4a352 in clone (/lib64/libc.so.6+0x100352)
 
0x606000ab4be0 is located 0 bytes inside of 56-byte region [0x606000ab4be0,0x606000ab4c18)
freed by thread T2 here:
    #0 0x7ffff768ef27 in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb0f27)
    #1 0x7fffea9fdd07 in LRUStorage::free_node(LRUStorage::Node*) const /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:814
    #2 0x7fffea9fdfc3 in LRUStorage::free_node(std::__detail::_Node_iterator<std::pair<CacheKey const, LRUStorage::Node*>, false, true>&) const /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:827
    #3 0x7fffea9fcbcc in LRUStorage::access_value(LRUStorage::access_approach_t, CacheKey const&, unsigned int, unsigned int, unsigned int, GWBUF**) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:656
    #4 0x7fffea9fa597 in LRUStorage::do_get_value(Storage::Token*, CacheKey const&, unsigned int, unsigned int, unsigned int, GWBUF**) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:433
    #5 0x7fffeaa107a6 in LRUStorageMT::get_value(Storage::Token*, CacheKey const&, unsigned int, unsigned int, unsigned int, GWBUF**, std::function<void (unsigned int, GWBUF*)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/lrustoragemt.cc:54
    #6 0x7fffea9f3e43 in CacheSimple::get_value(Storage::Token*, CacheKey const&, unsigned int, unsigned int, unsigned int, GWBUF**, std::function<void (unsigned int, GWBUF*)> const&) const /home/markusjm/MaxScale/server/modules/filter/cache/cachesimple.cc:52
    #7 0x7fffea9d2b6a in SessionCache::get_value(CacheKey const&, unsigned int, unsigned int, unsigned int, GWBUF**, std::function<void (unsigned int, GWBUF*)> const&) const /home/markusjm/MaxScale/server/modules/filter/cache/sessioncache.hh:88
    #8 0x7fffea9cca3b in CacheFilterSession::route_SELECT(CacheFilterSession::cache_action_t, CacheRules const&, GWBUF*) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:1214
    #9 0x7fffea9cbfea in CacheFilterSession::route_COM_QUERY(GWBUF*) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:1140
    #10 0x7fffea9c6ee3 in CacheFilterSession::routeQuery(GWBUF*) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:452
    #11 0x7fffea9c2efa in maxscale::Filter<CacheFilter, CacheFilterSession>::apiRouteQuery(mxs_filter*, mxs_filter_session*, GWBUF*) /home/markusjm/MaxScale/include/maxscale/filter.hh:487
    #12 0x7ffff71ec693 in ServiceEndpoint::routeQuery(GWBUF*) /home/markusjm/MaxScale/server/core/service.cc:1451
    #13 0x7ffff7226fb8 in Session::routeQuery(GWBUF*) /home/markusjm/MaxScale/server/core/session.cc:1338
    #14 0x7fffea7bdd29 in MariaDBClientConnection::route_statement(maxscale::Buffer&&) /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_client.cc:1264
    #15 0x7fffea7c8fd7 in MariaDBClientConnection::process_normal_packet(maxscale::Buffer&&) /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_client.cc:2449
    #16 0x7fffea7be0d7 in MariaDBClientConnection::process_normal_read() /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_client.cc:1321
    #17 0x7fffea7beb59 in MariaDBClientConnection::ready_for_reading(DCB*) /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_client.cc:1440
    #18 0x7ffff7044d13 in DCB::process_events(unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1292
    #19 0x7ffff7045322 in DCB::event_handler(DCB*, unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1344
    #20 0x7ffff704553f in DCB::poll_handler(MXB_POLL_DATA*, MXB_WORKER*, unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1383
    #21 0x7ffff7299324 in maxbase::Worker::poll_waitevents() /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:877
    #22 0x7ffff729740f in maxbase::Worker::run(maxbase::Semaphore*) /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:570
    #23 0x7ffff7298448 in maxbase::Worker::thread_main(maxbase::Worker*, maxbase::Semaphore*) /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:696
    #24 0x7ffff72a4c08 in void std::__invoke_impl<void, void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>(std::__invoke_other, void (*&&)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*&&, maxbase::Semaphore*&&) /usr/include/c++/11/bits/invoke.h:61
    #25 0x7ffff72a4ad9 in std::__invoke_result<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>::type std::__invoke<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>(void (*&&)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*&&, maxbase::Semaphore*&&) /usr/include/c++/11/bits/invoke.h:96
    #26 0x7ffff72a4a0c in void std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/11/bits/std_thread.h:253
    #27 0x7ffff72a49a9 in std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> >::operator()() /usr/include/c++/11/bits/std_thread.h:260
    #28 0x7ffff72a484d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211
    #29 0x7ffff6373c83 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xd9c83)
 
previously allocated by thread T5 here:
    #0 0x7ffff768e35f in operator new(unsigned long, std::nothrow_t const&) (/lib64/libasan.so.6+0xb035f)
    #1 0x7fffea9ff9d5 in LRUStorage::get_new_node(CacheKey const&, GWBUF const*, std::__detail::_Node_iterator<std::pair<CacheKey const, LRUStorage::Node*>, false, true>*, LRUStorage::Node**) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:978
    #2 0x7fffea9fa8c3 in LRUStorage::do_put_value(Storage::Token*, CacheKey const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, GWBUF const*) /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.cc:458
    #3 0x7fffeaa108fe in LRUStorageMT::put_value(Storage::Token*, CacheKey const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, GWBUF const*, std::function<void (unsigned int)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/lrustoragemt.cc:65
    #4 0x7fffea9f3f10 in CacheSimple::put_value(Storage::Token*, CacheKey const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, GWBUF const*, std::function<void (unsigned int)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachesimple.cc:61
    #5 0x7fffea9d2c40 in SessionCache::put_value(CacheKey const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, GWBUF const*, std::function<void (unsigned int)> const&) /home/markusjm/MaxScale/server/modules/filter/cache/sessioncache.hh:99
    #6 0x7fffea9c9ea4 in CacheFilterSession::store_and_prepare_response(std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:792
    #7 0x7fffea9c9134 in CacheFilterSession::handle_storing_response(std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:715
    #8 0x7fffea9c7074 in CacheFilterSession::client_reply_post_process(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:482
    #9 0x7fffea9c7dfb in CacheFilterSession::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/filter/cache/cachefiltersession.cc:612
    #10 0x7fffea9c2f45 in maxscale::Filter<CacheFilter, CacheFilterSession>::apiClientReply(mxs_filter*, mxs_filter_session*, GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/include/maxscale/filter.hh:501
    #11 0x7ffff7168dbc in maxscale::RouterSession::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/core/router.cc:43
    #12 0x7fffea5593ac in RWSplitSession::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) /home/markusjm/MaxScale/server/modules/routing/readwritesplit/rwsplitsession.cc:658
    #13 0x7fffea525a1d in maxscale::Router<RWSplit, RWSplitSession>::clientReply(mxs_router*, mxs_router_session*, GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> > const&, maxscale::Reply const&) (/home/markusjm/build-develop/lib64/maxscale/libreadwritesplit.so+0xbba1d)
    #14 0x7ffff71eca57 in ServiceEndpoint::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> >&, maxscale::Reply const&) /home/markusjm/MaxScale/server/core/service.cc:1458
    #15 0x7ffff71aca4b in ServerEndpoint::clientReply(GWBUF*, std::vector<maxscale::Endpoint*, std::allocator<maxscale::Endpoint*> >&, maxscale::Reply const&) /home/markusjm/MaxScale/server/core/server.cc:999
    #16 0x7fffea7ec4b3 in MariaDBBackendConnection::normal_read() /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_backend.cc:834
    #17 0x7fffea7e9760 in MariaDBBackendConnection::ready_for_reading(DCB*) /home/markusjm/MaxScale/server/modules/protocol/MariaDB/mariadb_backend.cc:381
    #18 0x7ffff7044d13 in DCB::process_events(unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1292
    #19 0x7ffff7045322 in DCB::event_handler(DCB*, unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1344
    #20 0x7ffff704553f in DCB::poll_handler(MXB_POLL_DATA*, MXB_WORKER*, unsigned int) /home/markusjm/MaxScale/server/core/dcb.cc:1383
    #21 0x7ffff7299324 in maxbase::Worker::poll_waitevents() /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:877
    #22 0x7ffff729740f in maxbase::Worker::run(maxbase::Semaphore*) /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:570
    #23 0x7ffff7298448 in maxbase::Worker::thread_main(maxbase::Worker*, maxbase::Semaphore*) /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:696
    #24 0x7ffff72a4c08 in void std::__invoke_impl<void, void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>(std::__invoke_other, void (*&&)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*&&, maxbase::Semaphore*&&) /usr/include/c++/11/bits/invoke.h:61
    #25 0x7ffff72a4ad9 in std::__invoke_result<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>::type std::__invoke<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*>(void (*&&)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*&&, maxbase::Semaphore*&&) /usr/include/c++/11/bits/invoke.h:96
    #26 0x7ffff72a4a0c in void std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/11/bits/std_thread.h:253
    #27 0x7ffff72a49a9 in std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> >::operator()() /usr/include/c++/11/bits/std_thread.h:260
    #28 0x7ffff72a484d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(maxbase::Worker*, maxbase::Semaphore*), maxbase::Worker*, maxbase::Semaphore*> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211
    #29 0x7ffff6373c83 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xd9c83)
 
Thread T8 created by T0 here:
    #0 0x7ffff76348d6 in pthread_create (/lib64/libasan.so.6+0x568d6)
    #1 0x7ffff6373f39 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xd9f39)
    #2 0x7ffff7297a77 in maxbase::Worker::start() /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:597
    #3 0x7ffff716aaa9 in maxscale::RoutingWorker::start_workers() /home/markusjm/MaxScale/server/core/routingworker.cc:374
    #4 0x41c5c2 in main /home/markusjm/MaxScale/server/core/gateway.cc:2128
    #5 0x7ffff5d71b74 in __libc_start_main (/lib64/libc.so.6+0x27b74)
 
Thread T2 created by T0 here:
    #0 0x7ffff76348d6 in pthread_create (/lib64/libasan.so.6+0x568d6)
    #1 0x7ffff6373f39 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xd9f39)
    #2 0x7ffff7297a77 in maxbase::Worker::start() /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:597
    #3 0x7ffff716aaa9 in maxscale::RoutingWorker::start_workers() /home/markusjm/MaxScale/server/core/routingworker.cc:374
    #4 0x41c5c2 in main /home/markusjm/MaxScale/server/core/gateway.cc:2128
    #5 0x7ffff5d71b74 in __libc_start_main (/lib64/libc.so.6+0x27b74)
 
Thread T5 created by T0 here:
    #0 0x7ffff76348d6 in pthread_create (/lib64/libasan.so.6+0x568d6)
    #1 0x7ffff6373f39 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xd9f39)
    #2 0x7ffff7297a77 in maxbase::Worker::start() /home/markusjm/MaxScale/maxutils/maxbase/src/worker.cc:597
    #3 0x7ffff716aaa9 in maxscale::RoutingWorker::start_workers() /home/markusjm/MaxScale/server/core/routingworker.cc:374
    #4 0x41c5c2 in main /home/markusjm/MaxScale/server/core/gateway.cc:2128
    #5 0x7ffff5d71b74 in __libc_start_main (/lib64/libc.so.6+0x27b74)
 
SUMMARY: AddressSanitizer: heap-use-after-free /home/markusjm/MaxScale/server/modules/filter/cache/lrustorage.hh:152 in LRUStorage::Node::key() const
Shadow bytes around the buggy address:
  0x0c0c8014e920: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c8014e930: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c8014e940: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c8014e950: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c8014e960: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
=>0x0c0c8014e970: fd fd fd fd fd fd fd fd fa fa fa fa[fd]fd fd fd
  0x0c0c8014e980: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c8014e990: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c8014e9a0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c8014e9b0: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c8014e9c0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3232734==ABORTING

Comment by Arnold van Norden [ 2021-07-28 ]

Thanks Markus,

So, it's a memory issue then?
Or a bug in the LRU?

Comment by markus makela [ 2021-07-28 ]

Looks like a bug in the shared cache when invalidation is enabled. You could confirm this by using cached_data=thread_specific which I think might also prevent this.

Now that we've managed to reproduce this, I think we'll be able to create a fix for it.

Comment by Arnold van Norden [ 2021-07-29 ]

Thank you for your fast reply Markus,
Changing the cached_data to thread_specific doesn't solve the issue. Maxscale still crashes. I'm looking forward to your fix.
Thanks in advance.

Maxscale.log:

2021-07-29 09:06:01 notice : [cache] Creating thread specific cache.
2021-07-29 09:06:01 notice : [storage_inmemory] Storage module created.
2021-07-29 09:06:01 notice : [cache] Created single threaded LRU storage.
2021-07-29 09:06:01 notice : [cache] Created single threaded cache.
2021-07-29 09:06:01 notice : [cache] Created cache per thread.2021-07-29 09:06:01 notice : [cache] Creating thread specific cache.
2021-07-29 09:06:01 notice : [storage_inmemory] Storage module created.
2021-07-29 09:06:01 notice : [cache] Created single threaded LRU storage.
2021-07-29 09:06:01 notice : [cache] Created single threaded cache.
2021-07-29 09:06:01 notice : [cache] Created cache per thread.
...
...
021-07-29 09:08:02 error : (119) [cache] (Galera_Service) Item in LRU list was not found in key mapping.
free(): double free detected in tcache 2
alert : MaxScale 2.5.14 received fatal signal 6. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS

2021-07-29 09:08:02 alert : (119) (Galera_Service) MaxScale 2.5.14 received fatal signal 6. Commit ID: e63d226705ff236ace9cd8b135fbab0d67f14a13 System name: Linux Release string: Ubuntu 20.04.2 LTS
2021-07-29 09:08:02 alert : (119) (Galera_Service) Statement currently being classified: none/unknown

Comment by markus makela [ 2021-07-29 ]

OK, that actually simplifies things as it means that there's no concurrency involved in the bug.

Comment by markus makela [ 2021-07-29 ]

I think I found a way to reproduce this:

  1. Configure cache with a hard TTL of 5 second and enable invalidation and configure MaxScale to use only one thread
  2. Repeatedly execute SELECT * FROM test.t1 in one window
  3. Execute UPDATE test.t1 SET id = id + SLEEP(10) in another window

When the update returns, it should try to free a node via invalidation that was already freed due to the hard TTL being reached.

Generated at Thu Feb 08 04:23:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.