[MXS-3796] maxscale 6.1.2 killed by SystemD watchdog Created: 2021-09-28  Updated: 2021-11-15  Resolved: 2021-09-30

Status: Closed
Project: MariaDB MaxScale
Component/s: readconnroute, Router
Affects Version/s: 6.1.2
Fix Version/s: 6.1.3

Type: Bug Priority: Blocker
Reporter: Massimo Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

Ubuntu 20.04.3 LTS


Attachments: Text File gdb_tfli.txt    
Issue Links:
Relates
relates to MXS-3866 Crash in MaxScale Closed

 Description   

Maxscale keep crashing with signal 6

MariaDB MaxScale  /var/log/maxscale/maxscale.log  Tue Sep 28 15:20:39 2021
----------------------------------------------------------------------------
2021-09-28 15:20:39   notice : The systemd watchdog is Enabled. Internal timeout = 30s
2021-09-28 15:20:39   notice : Worker message queue size: 1MiB
2021-09-28 15:20:39   notice : The logging of info messages has been disabled.
2021-09-28 15:20:39   notice : Using up to 2.32GiB of memory for query classifier cache
2021-09-28 15:20:39   notice : syslog logging is enabled.
2021-09-28 15:20:39   notice : maxlog logging is enabled.
2021-09-28 15:20:39   notice : Host: 'ip-172-31-16-68' OS: Linux@5.11.0-1016-aws, #17~20.04.1-Ubuntu SMP Thu Aug 12 05:39:36 UTC 2021, x86_64 with 4 processor cores.
2021-09-28 15:20:39   notice : Total usable main memory: 15.5GiB.
2021-09-28 15:20:39   notice : MariaDB MaxScale 6.1.2 started (Commit: 1479685b53546a0758137c81b864539a80b13bbc)
2021-09-28 15:20:39   notice : MaxScale is running in process 110746
2021-09-28 15:20:39   notice : Configuration file: /etc/maxscale.cnf
2021-09-28 15:20:39   notice : Log directory: /var/log/maxscale
2021-09-28 15:20:39   notice : Data directory: /var/lib/maxscale
2021-09-28 15:20:39   notice : Module directory: /usr/lib/x86_64-linux-gnu/maxscale
2021-09-28 15:20:39   notice : Service cache: /var/cache/maxscale
2021-09-28 15:20:39   notice : Working directory: /var/log/maxscale
2021-09-28 15:20:39   notice : Module 'qc_sqlite' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libqc_sqlite.so'.
2021-09-28 15:20:39   notice : Query classification results are cached and reused. Memory used per thread: 595.02MiB
2021-09-28 15:20:39   notice : Using encrypted passwords. Encryption key read from '/var/lib/maxscale/.secrets'.
2021-09-28 15:20:39   notice : MaxScale started with 4 worker threads.
2021-09-28 15:20:39   notice : Loading /etc/maxscale.cnf.
2021-09-28 15:20:39   notice : /etc/maxscale.cnf.d does not exist, not reading.
2021-09-28 15:20:39   notice : Runtime configuration changes have been done to MaxScale. Loading persisted configuration files and applying them on top of the main configuration file. These changes can overri
de the values of the main configuration file: To revert them, remove all the files in '/var/lib/maxscale/maxscale.cnf.d'.
2021-09-28 15:20:39   notice : Loading /var/lib/maxscale/maxscale.cnf.d/MariaDB-Monitor.cnf.
2021-09-28 15:20:39   warning: Found static and runtime configurations for [MariaDB-Monitor], ignoring static configuration. Move the runtime changes into the static configuration file and remove the generate
d file in '/var/lib/maxscale/maxscale.cnf.d' to remove this warning.
2021-09-28 15:20:39   notice : Loading /var/lib/maxscale/maxscale.cnf.d/Write-Service.cnf.
2021-09-28 15:20:39   warning: Found static and runtime configurations for [Write-Service], ignoring static configuration. Move the runtime changes into the static configuration file and remove the generated file in '/var/lib/maxscale/maxscale.cnf.d' to remove this warning.
2021-09-28 15:20:39   notice : Loading /var/lib/maxscale/maxscale.cnf.d/Read-Write-Service.cnf.
2021-09-28 15:20:39   warning: Found static and runtime configurations for [Read-Write-Service], ignoring static configuration. Move the runtime changes into the static configuration file and remove the generated file in '/var/lib/maxscale/maxscale.cnf.d' to remove this warning.
2021-09-28 15:20:39   notice : Module 'readwritesplit' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so'.
2021-09-28 15:20:39   notice : Module 'readconnroute' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libreadconnroute.so'.
2021-09-28 15:20:39   notice : Module 'mariadbmon' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libmariadbmon.so'.
2021-09-28 15:20:39   warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption.
2021-09-28 15:20:39   notice : Started REST API on [127.0.0.1]:8989
 2021-09-28 15:20:39   notice : 'db-rep02' sent version string '10.5.9-6-MariaDB-enterprise-log'. Detected type: 'MariaDB', version: 10.5.9.
2021-09-28 15:20:39   notice : 'db-rep03' sent version string '10.5.10-7-MariaDB-enterprise-log'. Detected type: 'MariaDB', version: 10.5.10.
2021-09-28 15:20:39   notice : Server 'db-rep02' charset: utf8mb4
2021-09-28 15:20:39   notice : Server 'db-rep03' charset: utf8mb4
2021-09-28 15:20:39   notice : [mariadbmon] 'MariaDB-Monitor' acquired the exclusive lock on a majority of its servers. Automatic cluster manipulation operations such as failover will be enabled in 5 monitor ticks.
2021-09-28 15:20:39   notice : Starting a total of 3 services...
2021-09-28 15:20:39   notice : (Read-Only-Listener); Listening for connections at [0.0.0.0]:4008
2021-09-28 15:20:39   notice : Service 'Read-Only-Service' started (1/3)
2021-09-28 15:20:39   notice : (Write-Listener); Listening for connections at [0.0.0.0]:4006
2021-09-28 15:20:39   notice : Service 'Write-Service' started (2/3)
2021-09-28 15:20:39   notice : (Read-Write-Listener); Listening for connections at [0.0.0.0]:4007
2021-09-28 15:20:39   notice : Service 'Read-Write-Service' started (3/3)
2021-09-28 15:20:40   notice : Read 247 user@host entries from 'db-rep02' for service 'Write-Service'.
2021-09-28 15:20:40   notice : Read 247 user@host entries from 'db-rep03' for service 'Read-Only-Service'.
2021-09-28 15:20:40   notice : Read 247 user@host entries from 'db-rep02' for service 'Read-Write-Service'.
alert  : MaxScale 6.1.2 received fatal signal 6. Commit ID: 1479685b53546a0758137c81b864539a80b13bbc System name: Linux Release string: Ubuntu 20.04.3 LTS
 
 
nm: /lib/x86_64-linux-gnu/libc.so.6: no symbols
alert  :   /lib/x86_64-linux-gnu/libpthread.so.0(+0x133f4): ??:?
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x134e8): ??:?
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZNK7maxbase9Semaphore4waitENS0_17signal_approach_tE+0x1f): maxutils/maxbase/include/maxbase/semaphore.hh:144
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker16execute_seriallyERN7maxbase10WorkerTaskE+0x143): server/core/routingworker.cc:1132
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_Z11dcb_foreachPFbP3DCBPvES1_+0xdf): server/core/dcb.cc:2096
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_Z20session_list_to_jsonPKcb+0x42): server/core/session.cc:463
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x53514a): server/core/resource.cc:755
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x2f): server/core/resource.cc:152
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x539f18): server/core/resource.cc:1541 (discriminator 1)
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x53b751): server/core/resource.cc:1739
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x53be57): server/core/resource.cc:1801 (discriminator 1)
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x53c3b6): /usr/include/c++/9/bits/std_function.h:302
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZNKSt8functionIFvvEEclEv+0x36): /usr/include/c++/9/bits/std_function.h:688
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x662904): maxutils/maxbase/src/worker.cc:495
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x122): maxutils/maxbase/src/worker.cc:656
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x1f1): maxutils/maxbase/src/messagequeue.cc:307
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue12poll_handlerEP13MXB_POLL_DATAP10MXB_WORKERj+0x35): maxutils/maxbase/src/messagequeue.cc:343
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x4fc): maxutils/maxbase/src/worker.cc:863
  /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x12d): maxutils/maxbase/src/worker.cc:560
  /usr/bin/maxscale(_ZN7maxbase6Worker3runEv+0x21): maxutils/maxbase/include/maxbase/worker.hh:451
  /usr/bin/maxscale(main+0x172d): server/core/gateway.cc:2213
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3): ??:0
  /usr/bin/maxscale(_start+0x2e): ??:?
 
 
alert  : Writing core dump.



 Comments   
Comment by Massimo [ 2021-09-29 ]

I have attach the output of gdb

Comment by markus makela [ 2021-09-30 ]

Turns out this was a simple case of use-after-free and the way the buffer was freed ended up what appeared to be an empty buffer.

Generated at Thu Feb 08 04:24:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.