[MXS-3437] SIGABRT when cleaning up persistent connections (SystemD watchdog) Created: 2021-03-12  Updated: 2021-05-03  Resolved: 2021-05-03

Status: Closed
Project: MariaDB MaxScale
Component/s: readwritesplit
Affects Version/s: 2.5.8
Fix Version/s: 2.5.9

Type: Bug Priority: Major
Reporter: Allen Lee (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None


 Description   

Customer hit the same fatal as https://jira.mariadb.org/browse/MXS-3425.

A crash appears to have happened when the persistent connections are being removed:

/lib64/libpthread.so.0(+0xdbcf): sem_wait.c:?
/lib64/libpthread.so.0(sem_wait+0x2b): ??:0
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker20execute_concurrentlyESt8functionIFvvEE+0x62): maxutils/maxbase/include/maxbase/semaphore.hh:146
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server30cleanup_persistent_connectionsEv+0x31): /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:275
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server15json_attributesEv+0x184): include/maxscale/target.hh:316
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server12to_json_dataEPKc+0x73): server/core/server.cc:842
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager29server_to_json_data_relationsEPK6ServerPKc+0x1c8): server/core/servermanager.cc:223
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1585a5): server/core/servermanager.cc:186
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager19server_list_to_jsonEPKc+0xb4): server/core/servermanager.cc:47
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x133cf8): server/core/resource.cc:606
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x14): server/core/resource.cc:151
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x137eb2): server/core/resource.cc:1313
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x139a10): server/core/resource.cc:1343
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x159): maxutils/maxbase/src/worker.cc:490
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:307
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be): maxutils/maxbase/src/worker.cc:879
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
/usr/bin/maxscale(main+0x1dbb): maxutils/maxbase/include/maxbase/log.h:168
/lib64/libc.so.6(__libc_start_main+0xf5): ??:?
/usr/bin/maxscale(): ??:?

Original log:

Mar 12 08:00:49 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 434809100 bytes sent.
Mar 12 08:01:00 ibs-reportdbproxy-dc1 systemd: maxscale.service watchdog timeout (limit 1min)!
Mar 12 08:01:00 ibs-reportdbproxy-dc1 maxscale[15759]: MaxScale 2.5.8 received fatal signal 6. Commit ID: 8a610d4dad58a710669ab137be91f3c9376c3e9a System name: Linux Release string: NAME="CentOS Linux"
Mar 12 08:01:00 ibs-reportdbproxy-dc1 maxscale[15759]: Statement currently being classified: none/unknown
Mar 12 08:01:01 ibs-reportdbproxy-dc1 systemd: Started Session 1007 of user root.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771521600 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771525700 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771529800 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771533900 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771538000 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771542100 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771546200 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192678) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192711) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 707368900 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: 
#012  /lib64/libpthread.so.0(+0xdbcf): sem_wait.c:?
#012  /lib64/libpthread.so.0(sem_wait+0x2b): ??:0
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker20execute_concurrentlyESt8functionIFvvEE+0x62): maxutils/maxbase/include/maxbase/semaphore.hh:146
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server30cleanup_persistent_connectionsEv+0x31): /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:275
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server15json_attributesEv+0x184): include/maxscale/target.hh:316
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server12to_json_dataEPKc+0x73): server/core/server.cc:842
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager29server_to_json_data_relationsEPK6ServerPKc+0x1c8): server/core/servermanager.cc:223
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1585a5): server/core/servermanager.cc:186
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager19server_list_to_jsonEPKc+0xb4): server/core/servermanager.cc:47
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x133cf8): server/core/resource.cc:606
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x14): server/core/resource.cc:151
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x137eb2): server/core/resource.cc:1313
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x139a10): server/core/resource.cc:1343
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x159): maxutils/maxbase/src/worker.cc:490
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:307
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be): maxutils/maxbase/src/worker.cc:879
#012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
#012  /usr/bin/maxscale(main+0x1dbb): maxutils/maxbase/include/maxbase/log.h:168
#012  /lib64/libc.so.6(__libc_start_main+0xf5): ??:?
#012  /usr/bin/maxscale(): ??:?
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192711) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771550300 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192678) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 908703500 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192678) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 616529300 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771554400 bytes sent.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: maxscale.service: main process exited, code=killed, status=6/ABRT
Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: Unit maxscale.service entered failed state.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: maxscale.service failed.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: maxscale.service holdoff time over, scheduling restart.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: Stopped MariaDB MaxScale Database Proxy.
Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: Starting MariaDB MaxScale Database Proxy...
Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: The 'log_debug' option has no effect in release mode.
Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: Found static and runtime configurations for [maxscale], ignoring static configuration. Move the runtime changes into the static configuration file and remove the generated file in '/var/lib/maxscale/maxscale.cnf.d' to remove this warning.
Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: Missing slashes (/) around a regular expression is deprecated: 'exclude=select,show'.
Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: Missing quotes (") around a quoted string is deprecated: 'separator=,'.
Mar 12 08:01:03 ibs-reportdbproxy-dc1 systemd: Started MariaDB MaxScale Database Proxy.
Mar 12 08:01:19 ibs-reportdbproxy-dc1 journal: Suppressed 592865 messages from /system.slice/maxscale.service
Mar 12 08:01:19 ibs-reportdbproxy-dc1 maxscale[32441]: (17) Low water mark hit for 'report'@'::ffff:10.1.12.107', accepting new data



 Comments   
Comment by markus makela [ 2021-03-12 ]

This is fixed in 2.5.9.

Generated at Thu Feb 08 04:21:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.