Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-4594

Maxscale service is killed while running the data migration

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 23.02.1
    • N/A
    • None
    • None
    • MXS-SPRINT-182

    Description

      While doing the data migration activity from PostgreSQL to MariaDB then maxscale is killing itself.
      OS: Centos7
      RAM: 16GB

      Even though we have created one test schema with simple table but still we are getting the below issue. While fetching objects from source then we are getting the below issue.

      NOTE: I did not find any details from the Maxscale logs.
      Please find the error details from the system logs.

      Apr 25 03:50:49 test201 maxscale: The 'log_debug' option has no effect in release mode.
      Apr 25 03:50:49 test201 maxscale: The logging of debug messages has been enabled.
      Apr 25 03:50:49 test201 maxscale: The logging of info messages has been enabled.
      Apr 25 03:50:49 test201 maxscale: The logging of notice messages has been enabled.
      Apr 25 03:50:49 test201 maxscale: The logging of warning messages has been enabled.
      Apr 25 03:50:49 test201 maxscale: Using up to 2.32GiB of memory for query classifier cache
      Apr 25 03:50:49 test201 systemd: Started MariaDB MaxScale Database Proxy.
      Apr 25 04:00:11 test201 systemd: maxscale.service: main process exited, code=killed, status=11/SEGV
      Apr 25 04:00:11 test201 systemd: Unit maxscale.service entered failed state.
      Apr 25 04:00:11 test201 systemd: maxscale.service failed.
      Apr 25 04:00:11 test201 systemd: maxscale.service holdoff time over, scheduling restart.
      Apr 25 04:00:11 test201 systemd: Stopped MariaDB MaxScale Database Proxy.
      Apr 25 04:00:11 test201 systemd: Starting MariaDB MaxScale Database Proxy...
      Apr 25 04:00:11 test201 maxscale: /etc/maxscale.cnf.d does not exist, not reading.
      Apr 25 04:00:11 test201 maxscale: The 'log_debug' option has no effect in release mode.
      Apr 25 04:00:11 test201 maxscale: The logging of debug messages has been enabled.
      Apr 25 04:00:11 test201 maxscale: The logging of info messages has been enabled.
      Apr 25 04:00:11 test201 maxscale: The logging of notice messages has been enabled.
      Apr 25 04:00:11 test201 maxscale: The logging of warning messages has been enabled.
      Apr 25 04:00:11 test201 maxscale: Using up to 2.32GiB of memory for query classifier cache
      Apr 25 04:00:11 test201 systemd: Started MariaDB MaxScale Database Proxy.
      Apr 25 04:01:01 test201 systemd: Created slice User Slice of root.
      Apr 25 04:01:01 test201 systemd: Started Session 6862 of user root.
      Apr 25 04:01:01 test201 systemd: Removed slice User Slice of root.
      Apr 25 04:04:10 test201 systemd: maxscale.service: main process exited, code=killed, status=11/SEGV
      Apr 25 04:04:10 test201 systemd: Unit maxscale.service entered failed state.
      Apr 25 04:04:10 test201 systemd: maxscale.service failed.
      Apr 25 04:04:10 test201 systemd: maxscale.service holdoff time over, scheduling restart.
      Apr 25 04:04:10 test201 systemd: Stopped MariaDB MaxScale Database Proxy.
      Apr 25 04:04:10 test201 systemd: Starting MariaDB MaxScale Database Proxy...
      Apr 25 04:04:10 test201 maxscale: /etc/maxscale.cnf.d does not exist, not reading.
      Apr 25 04:04:10 test201 maxscale: The 'log_debug' option has no effect in release mode.
      Apr 25 04:04:10 test201 maxscale: The logging of debug messages has been enabled.
      Apr 25 04:04:10 test201 maxscale: The logging of info messages has been enabled.
      Apr 25 04:04:10 test201 maxscale: The logging of notice messages has been enabled.
      Apr 25 04:04:10 test201 maxscale: The logging of warning messages has been enabled.
      Apr 25 04:04:10 test201 maxscale: Using up to 2.32GiB of memory for query classifier cache
      Apr 25 04:04:10 test201 systemd: Started MariaDB MaxScale Database Proxy.

      Attachments

        Activity

          Still unable to generate core dump file.

          [root@testans401 ~]# sudo /usr/bin/maxscale -d -lstdout -U maxscale &
          [1] 83549
          [root@testans401 ~]# 2023-04-25 21:28:01 notice : MaxScale will be run in the terminal process.
          2023-04-25 21:28:01 notice : /etc/maxscale.cnf.d does not exist, not reading.
          2023-04-25 21:28:01 warning: The 'log_debug' option has no effect in release mode.
          2023-04-25 21:28:01 notice : The logging of debug messages has been enabled.
          2023-04-25 21:28:01 notice : The logging of info messages has been enabled.
          2023-04-25 21:28:01 notice : The logging of notice messages has been enabled.
          2023-04-25 21:28:01 notice : The logging of warning messages has been enabled.
          2023-04-25 21:28:01 notice : Using up to 2.32GiB of memory for query classifier cache
          2023-04-25 21:28:01 notice : syslog logging is disabled.
          2023-04-25 21:28:01 notice : maxlog logging is enabled.
          2023-04-25 21:28:01 notice : Host: 'testans401.corp.copart.com' OS: Linux@3.10.0-1160.71.1.el7.x86_64, #1 SMP Tue Jun 28 15:37:28 UTC 2022, x86_64 with 4 processor cores (4.00 available).
          2023-04-25 21:28:01 notice : Total main memory: 15.49GiB (15.49GiB usable).
          2023-04-25 21:28:01 notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5)
          2023-04-25 21:28:01 notice : MaxScale is running in process 83550

          Configuration file : /etc/maxscale.cnf
          Log directory : /var/log/maxscale
          Data directory : /var/lib/maxscale
          Module directory : /usr/lib64/maxscale
          Service cache : /var/cache/maxscale

          2023-04-25 21:28:01 notice : Configuration file: /etc/maxscale.cnf
          2023-04-25 21:28:01 notice : Log directory: /var/log/maxscale
          2023-04-25 21:28:01 notice : Data directory: /var/lib/maxscale
          2023-04-25 21:28:01 notice : Module directory: /usr/lib64/maxscale
          2023-04-25 21:28:01 notice : Service cache: /var/cache/maxscale
          2023-04-25 21:28:01 notice : Module 'qc_sqlite' loaded from '/usr/lib64/maxscale/libqc_sqlite.so'.
          2023-04-25 21:28:01 info : qc_sqlite loaded.
          2023-04-25 21:28:01 notice : Query classification results are cached and reused. Memory used per thread: 595MiB
          2023-04-25 21:28:01 notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
          2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610602904704.
          2023-04-25 21:28:01 info : No 'auto_tune' parameters specified, no auto tuning will be performed.
          2023-04-25 21:28:01 notice : Using HS256 for JWT signatures
          2023-04-25 21:28:01 notice : Started REST API on [testans401]:8080
          2023-04-25 21:28:01 notice : No services defined in any of the configuration files
          2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610431850240.
          2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
          2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610423457536.
          2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
          2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610415064832.
          2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
          2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610406672128.
          2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
          2023-04-25 21:28:01 notice : MaxScale started with 4 worker threads.
          2023-04-25 21:28:09 notice : Message from PAM api when authenticating 'nachandra': 'Your password will expire in 11 days.
          '
          2023-04-25 21:28:09 error : Unknown PAM message type '5'.
          2023-04-25 21:28:09 info : Accept authentication from 'nachandra', using password. Request: /auth

          [1]+ Segmentation fault sudo /usr/bin/maxscale -d -lstdout -U maxscale
          [root@testans401 ~]#
          [root@testans401 ~]#
          [root@testans401 ~]# ps -ef|grep maxscale
          root 83691 82112 0 21:30 pts/6 00:00:00 grep --color=auto maxscale
          [root@testans401 ~]# coredumpctl
          No coredumps found.
          [root@testans401 ~]# sysctl kernel.core_pattern
          kernel.core_pattern = core
          [root@testans401 ~]# ls -lrth /usr/lib/systemd/systemd-coredump*
          -rwxr-xr-x 1 root root 130K Jan 13 2022 /usr/lib/systemd/systemd-coredump
          [root@testans401 ~]#
          [root@testans401 ~]#
          [root@testans401 ~]# ls -lrht /usr/bin/maxscale/
          ls: cannot access /usr/bin/maxscale/: Not a directory
          [root@testans401 ~]#
          [root@testans401 ~]# ls -lrht /var/lib/systemd/coredump/
          total 0
          [root@testans401 ~]#

          naresh.chandra@copart.com Naresh Chandra added a comment - Still unable to generate core dump file. [root@testans401 ~] # sudo /usr/bin/maxscale -d -lstdout -U maxscale & [1] 83549 [root@testans401 ~] # 2023-04-25 21:28:01 notice : MaxScale will be run in the terminal process. 2023-04-25 21:28:01 notice : /etc/maxscale.cnf.d does not exist, not reading. 2023-04-25 21:28:01 warning: The 'log_debug' option has no effect in release mode. 2023-04-25 21:28:01 notice : The logging of debug messages has been enabled. 2023-04-25 21:28:01 notice : The logging of info messages has been enabled. 2023-04-25 21:28:01 notice : The logging of notice messages has been enabled. 2023-04-25 21:28:01 notice : The logging of warning messages has been enabled. 2023-04-25 21:28:01 notice : Using up to 2.32GiB of memory for query classifier cache 2023-04-25 21:28:01 notice : syslog logging is disabled. 2023-04-25 21:28:01 notice : maxlog logging is enabled. 2023-04-25 21:28:01 notice : Host: 'testans401.corp.copart.com' OS: Linux@3.10.0-1160.71.1.el7.x86_64, #1 SMP Tue Jun 28 15:37:28 UTC 2022, x86_64 with 4 processor cores (4.00 available). 2023-04-25 21:28:01 notice : Total main memory: 15.49GiB (15.49GiB usable). 2023-04-25 21:28:01 notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5) 2023-04-25 21:28:01 notice : MaxScale is running in process 83550 Configuration file : /etc/maxscale.cnf Log directory : /var/log/maxscale Data directory : /var/lib/maxscale Module directory : /usr/lib64/maxscale Service cache : /var/cache/maxscale 2023-04-25 21:28:01 notice : Configuration file: /etc/maxscale.cnf 2023-04-25 21:28:01 notice : Log directory: /var/log/maxscale 2023-04-25 21:28:01 notice : Data directory: /var/lib/maxscale 2023-04-25 21:28:01 notice : Module directory: /usr/lib64/maxscale 2023-04-25 21:28:01 notice : Service cache: /var/cache/maxscale 2023-04-25 21:28:01 notice : Module 'qc_sqlite' loaded from '/usr/lib64/maxscale/libqc_sqlite.so'. 2023-04-25 21:28:01 info : qc_sqlite loaded. 2023-04-25 21:28:01 notice : Query classification results are cached and reused. Memory used per thread: 595MiB 2023-04-25 21:28:01 notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext. 2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610602904704. 2023-04-25 21:28:01 info : No 'auto_tune' parameters specified, no auto tuning will be performed. 2023-04-25 21:28:01 notice : Using HS256 for JWT signatures 2023-04-25 21:28:01 notice : Started REST API on [testans401] :8080 2023-04-25 21:28:01 notice : No services defined in any of the configuration files 2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610431850240. 2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance. 2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610423457536. 2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance. 2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610415064832. 2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance. 2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610406672128. 2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance. 2023-04-25 21:28:01 notice : MaxScale started with 4 worker threads. 2023-04-25 21:28:09 notice : Message from PAM api when authenticating 'nachandra': 'Your password will expire in 11 days. ' 2023-04-25 21:28:09 error : Unknown PAM message type '5'. 2023-04-25 21:28:09 info : Accept authentication from 'nachandra', using password. Request: /auth [1] + Segmentation fault sudo /usr/bin/maxscale -d -lstdout -U maxscale [root@testans401 ~] # [root@testans401 ~] # [root@testans401 ~] # ps -ef|grep maxscale root 83691 82112 0 21:30 pts/6 00:00:00 grep --color=auto maxscale [root@testans401 ~] # coredumpctl No coredumps found. [root@testans401 ~] # sysctl kernel.core_pattern kernel.core_pattern = core [root@testans401 ~] # ls -lrth /usr/lib/systemd/systemd-coredump* -rwxr-xr-x 1 root root 130K Jan 13 2022 /usr/lib/systemd/systemd-coredump [root@testans401 ~] # [root@testans401 ~] # [root@testans401 ~] # ls -lrht /usr/bin/maxscale/ ls: cannot access /usr/bin/maxscale/: Not a directory [root@testans401 ~] # [root@testans401 ~] # ls -lrht /var/lib/systemd/coredump/ total 0 [root@testans401 ~] #
          markus makela markus makela added a comment - - edited

          Could you try doing the test again but this time with the debug packages? You can find the debug version of the 23.02.1 release here: https://dlm.mariadb.com/2900358/MaxScale/23.02.1-debug/packages/rhel/7/x86_64/maxscale-23.02.1-1.rhel.7.x86_64.rpm

          Another option is to run MaxScale under gdb and see if that catches the problem. Use this to start GDB:

          sudo gdb /usr/bin/maxscale
          

          Then use this GDB command to start MaxScale:

          r -d -lstdout -U maxscale
          

          Once it hits that problem, run this command to print the stacktrace:

          bt
          

          The stacktrace should show where the problem originates from.

          markus makela markus makela added a comment - - edited Could you try doing the test again but this time with the debug packages? You can find the debug version of the 23.02.1 release here: https://dlm.mariadb.com/2900358/MaxScale/23.02.1-debug/packages/rhel/7/x86_64/maxscale-23.02.1-1.rhel.7.x86_64.rpm Another option is to run MaxScale under gdb and see if that catches the problem. Use this to start GDB: sudo gdb /usr/bin/maxscale Then use this GDB command to start MaxScale: r -d -lstdout -U maxscale Once it hits that problem, run this command to print the stacktrace: bt The stacktrace should show where the problem originates from.
          markus makela markus makela added a comment - naresh.chandra@copart.com any updates?
          markus makela markus makela added a comment -

          naresh.chandra@copart.com have you been able to reproduce this outside of a production environment? If you have, can you provide a test case that we can use to reproduce this?

          markus makela markus makela added a comment - naresh.chandra@copart.com have you been able to reproduce this outside of a production environment? If you have, can you provide a test case that we can use to reproduce this?
          markus makela markus makela added a comment -

          We'll need information on how to reproduce this or at least a stacktrace to see where the problem is. Right now there's not much to go on so I closed this as Incomplete. Once we have some information or it can be reproduced on another system, we can reopen this issue.

          markus makela markus makela added a comment - We'll need information on how to reproduce this or at least a stacktrace to see where the problem is. Right now there's not much to go on so I closed this as Incomplete. Once we have some information or it can be reproduced on another system, we can reopen this issue.

          People

            markus makela markus makela
            naresh.chandra@copart.com Naresh Chandra
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.