[MXS-4563] when create cache filter MaxScale 23.02.1 received fatal signal 11 Created: 2023-03-24  Updated: 2023-05-22  Resolved: 2023-05-22

Status: Closed
Project: MariaDB MaxScale
Component/s: cache
Affects Version/s: None
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: igor kis Assignee: Johan Wikman
Resolution: Cannot Reproduce Votes: 0
Labels: None


 Description   

MaxScale crashing after creating cache filter and restarting. But, everytime I try to login in GUI it crashes.
also maxctrl list filters does the same thing
root@... ~ # maxctrl list filters
Error: socket hang up. If MaxScale is configured to use HTTPS, use the --secure option.

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Fri Mar 24 07:52:09 2023
----------------------------------------------------------------------------
2023-03-24 07:52:09   notice : Module 'mariadbmon' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libmariadbmon.so'.
2023-03-24 07:52:09   notice : Module 'masking' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libmasking.so'.
2023-03-24 07:52:09   notice : Module 'readconnroute' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libreadconnroute.so'.
2023-03-24 07:52:09   notice : Module 'readwritesplit' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so'.
2023-03-24 07:52:09   notice : Module 'cache' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libcache.so'.
2023-03-24 07:52:09   notice : The logging of info messages has been enabled.
2023-03-24 07:52:09   notice : Using up to 37.74GiB of memory for query classifier cache
2023-03-24 07:52:09   notice : syslog logging is disabled.
2023-03-24 07:52:09   notice : maxlog logging is enabled.
2023-03-24 07:52:09   notice : Host: 'PapaBase' OS: Linux@5.10.0-18-amd64, #1 SMP Debian 5.10.140-1 (2022-09-02), x86_64 with 12 processor cores (12.00 available).
2023-03-24 07:52:09   notice : Total main memory: 251.6GiB (251.6GiB usable).
2023-03-24 07:52:09   notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5)
2023-03-24 07:52:09   notice : MaxScale is running in process 3384064
2023-03-24 07:52:09   notice : Configuration file: /etc/maxscale.cnf
2023-03-24 07:52:09   notice : Log directory: /var/log/maxscale
2023-03-24 07:52:09   notice : Data directory: /var/lib/maxscale
2023-03-24 07:52:09   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
2023-03-24 07:52:09   notice : Service cache: /var/cache/maxscale
2023-03-24 07:52:09   notice : Working directory: /var/log/maxscale
2023-03-24 07:52:09   notice : Module 'qc_sqlite' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so'.
2023-03-24 07:52:09   info   : qc_sqlite loaded.
2023-03-24 07:52:09   notice : Query classification results are cached and reused. Memory used per thread: 3.15GiB
2023-03-24 07:52:09   notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2023-03-24 07:52:09   notice : The systemd watchdog is Enabled. Internal timeout = 30s
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525978312192.
2023-03-24 07:52:09   info   : No 'auto_tune' parameters specified, no auto tuning will be performed.
2023-03-24 07:52:09   notice : [masking] Masking filter [Obfuscates-card] created.
2023-03-24 07:52:09   notice : [masking] The parameter 'treat_string_arg_as_field' is enabled for Obfuscates-card, disabling the query classifier cache.
2023-03-24 07:52:09   notice : Query classifier cache disabled.
2023-03-24 07:52:09   notice : [cache] Creating thread specific cache.
2023-03-24 07:52:09   notice : [cache] Created cache per thread.
2023-03-24 07:52:09   notice : Using HS256 for JWT signatures
2023-03-24 07:52:09   notice : Started REST API on [0.0.0.0]:8989
2023-03-24 07:52:09   notice : 'server1' sent version string '10.5.15-MariaDB-0+deb11u1'. Detected type: 'MariaDB', version: 10.5.15.
2023-03-24 07:52:09   notice : Server 'server1' charset: utf8mb4_general_ci
2023-03-24 07:52:09   notice : Starting a total of 2 services...
2023-03-24 07:52:09   notice : (Read-Write-Listener); Listening for connections at [::]:4006
2023-03-24 07:52:09   notice : Service 'Read-Write-Service' started (1/2)
2023-03-24 07:52:09   notice : (Read-Only-Listener); Listening for connections at [::]:4008
2023-03-24 07:52:09   notice : Service 'Read-Only-Service' started (2/2)
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525558560512.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525541775104.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525524989696.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525508204288.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525281732352.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525264946944.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140525248161536.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140524954580736.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140524937795328.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140524921009920.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140524904224512.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 140524677752576.
2023-03-24 07:52:09   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:52:09   notice : MaxScale started with 12 worker threads.
2023-03-24 07:52:10   notice : Read 10 user@host entries from 'server1' for service 'Read-Write-Service'.
2023-03-24 07:52:10   notice : Read 10 user@host entries from 'server1' for service 'Read-Only-Service'.
2023-03-24 07:52:11   info   : Variables have changed on 'server1', next check in 10 seconds: 'session_track_system_variables = autocommit,character_set_client,character_set_connection,character_set_results,time_zone', 'wait_timeout = 28800'
2023-03-24 07:54:39   info   : Accept authentication from 'admin', using password. Request: /v1/filters
MaxScale 23.02.1 received fatal signal 11. Commit ID: 5de96ecc39da80a992fc8f362d22a030a844bef5, System name: Linux, Release string: undefined, Thread: maxscale
 
2023-03-24 07:54:39   notice : For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscale] section.
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZNK8maxscale13RoutingWorker5indexEv+0): server/core/routingworker.cc:835
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZN7CachePT12worker_cacheEv+0x4d): server/modules/filter/cache/cachept.cc:209
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK7CachePT9all_rulesEv+0x18): server/modules/filter/cache/cachept.hh:86
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK5Cache11do_get_infoEj+0x5a): /usr/include/c++/10/bits/shared_ptr_base.h:1325
  /usr/lib/x86_64-linux-gnu/maxscale/libcache.so(_ZNK7CachePT8get_infoEj+0x1b): server/modules/filter/cache/cachept.cc:106
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZNK9FilterDef9json_dataEPKc+0x14c): server/core/filter.cc:239
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN9FilterDef19filter_list_to_jsonEPKc+0x5b): server/core/filter.cc:276
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x2aa909): server/core/resource.cc:745
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x16): server/core/resource.cc:153
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x2b0fa5): server/core/resource.cc:1717
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x79): maxutils/maxbase/src/worker.cc:778
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase17EventMessageQueue18handle_poll_eventsEPNS_6WorkerEjNS_8Pollable7ContextE+0x73): maxutils/maxbase/src/messagequeue.cc:222
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14deliver_eventsEmNSt6chrono10time_pointINS1_3_V212steady_clockENS1_8durationIlSt5ratioILl1ELl1000000000EEEEEEPNS_8PollableEjNSA_7ContextE+0x8d): maxutils/maxbase/src/worker.cc:1100
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x33c): maxutils/maxbase/src/worker.cc:1221
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x55): maxutils/maxbase/src/worker.cc:865
  /usr/bin/maxscale(main+0x289c): server/core/gateway.cc:2133
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea): ??:0
  /usr/bin/maxscale(_start+0x2a): ??:?
Writing core dump.
 
 
MariaDB MaxScale  /var/log/maxscale/maxscale.log  Fri Mar 24 07:54:41 2023
----------------------------------------------------------------------------
2023-03-24 07:54:41   notice : Module 'mariadbmon' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libmariadbmon.so'.
2023-03-24 07:54:41   notice : Module 'masking' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libmasking.so'.
2023-03-24 07:54:41   notice : Module 'readconnroute' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libreadconnroute.so'.
2023-03-24 07:54:41   notice : Module 'readwritesplit' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so'.
2023-03-24 07:54:41   notice : Module 'cache' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libcache.so'.
2023-03-24 07:54:41   notice : The logging of info messages has been enabled.
2023-03-24 07:54:41   notice : Using up to 37.74GiB of memory for query classifier cache
2023-03-24 07:54:41   notice : syslog logging is disabled.
2023-03-24 07:54:41   notice : maxlog logging is enabled.
2023-03-24 07:54:41   notice : Host: 'PapaBase' OS: Linux@5.10.0-18-amd64, #1 SMP Debian 5.10.140-1 (2022-09-02), x86_64 with 12 processor cores (12.00 available).
2023-03-24 07:54:41   notice : Total main memory: 251.6GiB (251.6GiB usable).
2023-03-24 07:54:41   notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5)
2023-03-24 07:54:41   notice : MaxScale is running in process 3384649
2023-03-24 07:54:41   notice : Configuration file: /etc/maxscale.cnf
2023-03-24 07:54:41   notice : Log directory: /var/log/maxscale
2023-03-24 07:54:41   notice : Data directory: /var/lib/maxscale
2023-03-24 07:54:41   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
2023-03-24 07:54:41   notice : Service cache: /var/cache/maxscale
2023-03-24 07:54:41   notice : Working directory: /var/log/maxscale
2023-03-24 07:54:41   notice : Module 'qc_sqlite' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so'.
2023-03-24 07:54:41   info   : qc_sqlite loaded.
2023-03-24 07:54:41   notice : Query classification results are cached and reused. Memory used per thread: 3.15GiB
2023-03-24 07:54:41   notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2023-03-24 07:54:41   notice : The systemd watchdog is Enabled. Internal timeout = 30s
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987977842176.
2023-03-24 07:54:41   info   : No 'auto_tune' parameters specified, no auto tuning will be performed.
2023-03-24 07:54:41   notice : [masking] Masking filter [Obfuscates-card] created.
2023-03-24 07:54:41   notice : [masking] The parameter 'treat_string_arg_as_field' is enabled for Obfuscates-card, disabling the query classifier cache.
2023-03-24 07:54:41   notice : Query classifier cache disabled.
2023-03-24 07:54:41   notice : [cache] Creating thread specific cache.
2023-03-24 07:54:41   notice : [cache] Created cache per thread.
2023-03-24 07:54:41   notice : Using HS256 for JWT signatures
2023-03-24 07:54:41   notice : Started REST API on [0.0.0.0]:8989
2023-03-24 07:54:41   notice : 'server1' sent version string '10.5.15-MariaDB-0+deb11u1'. Detected type: 'MariaDB', version: 10.5.15.
2023-03-24 07:54:41   notice : Server 'server1' charset: utf8mb4_general_ci
2023-03-24 07:54:41   notice : Starting a total of 2 services...
2023-03-24 07:54:41   notice : (Read-Write-Listener); Listening for connections at [::]:4006
2023-03-24 07:54:41   notice : Service 'Read-Write-Service' started (1/2)
2023-03-24 07:54:41   notice : (Read-Only-Listener); Listening for connections at [::]:4008
2023-03-24 07:54:41   notice : Service 'Read-Only-Service' started (2/2)
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987898930944.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987538405120.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987521619712.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987504834304.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987211253504.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987194468096.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987177682688.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139987160897280.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139986934425344.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139986917639936.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139986900854528.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   info   : [qc_sqlite] In-memory sqlite database successfully opened for thread 139986607273728.
2023-03-24 07:54:41   info   : Epoll instance for listening sockets added to worker epoll instance.
2023-03-24 07:54:41   notice : MaxScale started with 12 worker threads.
2023-03-24 07:54:42   notice : Read 10 user@host entries from 'server1' for service 'Read-Only-Service'.
2023-03-24 07:54:42   notice : Read 10 user@host entries from 'server1' for service 'Read-Write-Service'.
2023-03-24 07:54:43   info   : Variables have changed on 'server1', next check in 10 seconds: 'session_track_system_variables = autocommit,character_set_client,character_set_connection,character_set_results,time_zone', 'wait_timeout = 28800'



 Comments   
Comment by igor kis [ 2023-03-24 ]

also if add
soft_ttl=10
hard_ttl=10
to chache filter cfg file and try to restart got

systemctl restart maxscale
Job for maxscale.service failed because the control process exited with error code.
See "systemctl status maxscale.service" and "journalctl -xe" for details.

Comment by Johan Wikman [ 2023-03-24 ]

Could you paste your MaxScale configuration. Please remove/obfuscate passwords.

Johan

Comment by igor kis [ 2023-03-24 ]

Im sorry.
I've cleared main config from comments,
then restarted and find errors
2023-03-24 09:08:26 error : hard_ttl: Invalid duration: 10
2023-03-24 09:08:26 error : soft_ttl: Invalid duration: 10
changed to cache rule manually
[cache]
rules=/var/lib/maxscale/rules/cache.json
type=filter
module=cache
storage=storage_inmemory
cached_data=shared
soft_ttl=5m
hard_ttl=5m

and now all work!

Comment by Johan Wikman [ 2023-03-24 ]

Good that you got it to work!

I would be quite interested, though, in the original config and the exact steps need to cause MaxScale to crash, because no matter what, MaxScale should not crash.

Generated at Thu Feb 08 04:29:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.