Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-3437

SIGABRT when cleaning up persistent connections (SystemD watchdog)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.8
    • Fix Version/s: 2.5.9
    • Component/s: readwritesplit
    • Labels:
      None

      Description

      Customer hit the same fatal as https://jira.mariadb.org/browse/MXS-3425.

      A crash appears to have happened when the persistent connections are being removed:

      /lib64/libpthread.so.0(+0xdbcf): sem_wait.c:?
      /lib64/libpthread.so.0(sem_wait+0x2b): ??:0
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker20execute_concurrentlyESt8functionIFvvEE+0x62): maxutils/maxbase/include/maxbase/semaphore.hh:146
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server30cleanup_persistent_connectionsEv+0x31): /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:275
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server15json_attributesEv+0x184): include/maxscale/target.hh:316
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server12to_json_dataEPKc+0x73): server/core/server.cc:842
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager29server_to_json_data_relationsEPK6ServerPKc+0x1c8): server/core/servermanager.cc:223
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1585a5): server/core/servermanager.cc:186
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager19server_list_to_jsonEPKc+0xb4): server/core/servermanager.cc:47
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x133cf8): server/core/resource.cc:606
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x14): server/core/resource.cc:151
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x137eb2): server/core/resource.cc:1313
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x139a10): server/core/resource.cc:1343
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x159): maxutils/maxbase/src/worker.cc:490
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:307
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be): maxutils/maxbase/src/worker.cc:879
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
      /usr/bin/maxscale(main+0x1dbb): maxutils/maxbase/include/maxbase/log.h:168
      /lib64/libc.so.6(__libc_start_main+0xf5): ??:?
      /usr/bin/maxscale(): ??:?
      

      Original log:

      Mar 12 08:00:49 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 434809100 bytes sent.
      Mar 12 08:01:00 ibs-reportdbproxy-dc1 systemd: maxscale.service watchdog timeout (limit 1min)!
      Mar 12 08:01:00 ibs-reportdbproxy-dc1 maxscale[15759]: MaxScale 2.5.8 received fatal signal 6. Commit ID: 8a610d4dad58a710669ab137be91f3c9376c3e9a System name: Linux Release string: NAME="CentOS Linux"
      Mar 12 08:01:00 ibs-reportdbproxy-dc1 maxscale[15759]: Statement currently being classified: none/unknown
      Mar 12 08:01:01 ibs-reportdbproxy-dc1 systemd: Started Session 1007 of user root.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771521600 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771525700 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771529800 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771533900 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771538000 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771542100 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771546200 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192678) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192711) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 707368900 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: 
      #012  /lib64/libpthread.so.0(+0xdbcf): sem_wait.c:?
      #012  /lib64/libpthread.so.0(sem_wait+0x2b): ??:0
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN8maxscale13RoutingWorker20execute_concurrentlyESt8functionIFvvEE+0x62): maxutils/maxbase/include/maxbase/semaphore.hh:146
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server30cleanup_persistent_connectionsEv+0x31): /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:275
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server15json_attributesEv+0x184): include/maxscale/target.hh:316
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK6Server12to_json_dataEPKc+0x73): server/core/server.cc:842
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager29server_to_json_data_relationsEPK6ServerPKc+0x1c8): server/core/servermanager.cc:223
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x1585a5): server/core/servermanager.cc:186
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN13ServerManager19server_list_to_jsonEPKc+0xb4): server/core/servermanager.cc:47
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x133cf8): server/core/resource.cc:606
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZNK8Resource4callERK11HttpRequest+0x14): server/core/resource.cc:151
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x137eb2): server/core/resource.cc:1313
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x139a10): server/core/resource.cc:1343
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker14handle_messageERNS_12MessageQueueERKNS_19MessageQueueMessageE+0x159): maxutils/maxbase/src/worker.cc:490
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase12MessageQueue18handle_poll_eventsEPNS_6WorkerEj+0x138): maxutils/maxbase/src/messagequeue.cc:307
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be): maxutils/maxbase/src/worker.cc:879
      #012  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
      #012  /usr/bin/maxscale(main+0x1dbb): maxutils/maxbase/include/maxbase/log.h:168
      #012  /lib64/libc.so.6(__libc_start_main+0xf5): ??:?
      #012  /usr/bin/maxscale(): ??:?
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192711) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771550300 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192678) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 908703500 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192678) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 616529300 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192725) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) (RW-Service-report) > Processing LOAD DATA LOCAL INFILE: 771554400 bytes sent.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 maxscale[15759]: (192700) [readwritesplit] (RW-Service-report) Route query to master: ibs-reportdb-dc1 <
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: maxscale.service: main process exited, code=killed, status=6/ABRT
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: Unit maxscale.service entered failed state.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: maxscale.service failed.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: maxscale.service holdoff time over, scheduling restart.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: Stopped MariaDB MaxScale Database Proxy.
      Mar 12 08:01:02 ibs-reportdbproxy-dc1 systemd: Starting MariaDB MaxScale Database Proxy...
      Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: The 'log_debug' option has no effect in release mode.
      Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: Found static and runtime configurations for [maxscale], ignoring static configuration. Move the runtime changes into the static configuration file and remove the generated file in '/var/lib/maxscale/maxscale.cnf.d' to remove this warning.
      Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: Missing slashes (/) around a regular expression is deprecated: 'exclude=select,show'.
      Mar 12 08:01:03 ibs-reportdbproxy-dc1 maxscale[32441]: Missing quotes (") around a quoted string is deprecated: 'separator=,'.
      Mar 12 08:01:03 ibs-reportdbproxy-dc1 systemd: Started MariaDB MaxScale Database Proxy.
      Mar 12 08:01:19 ibs-reportdbproxy-dc1 journal: Suppressed 592865 messages from /system.slice/maxscale.service
      Mar 12 08:01:19 ibs-reportdbproxy-dc1 maxscale[32441]: (17) Low water mark hit for 'report'@'::ffff:10.1.12.107', accepting new data
      
      

        Attachments

          Activity

            People

            Assignee:
            markus makela markus makela
            Reporter:
            allen.lee@mariadb.com Allen Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration