|
Looks like a segmentation fault, are you sure there's nothing in /var/log/maxscale/maxscale.log? Have you disabled logging into this file by adding maxlog=false?
Does coredumpctl report anything?
|
|
We have not enabled maxlog=false parameter in the max config.
Here is maxscale config:
[maxscale]
threads = 4
admin_host = testans401
admin_port = 8080
auth_read_timeout = 10s
admin_secure_gui = false
admin_pam_readonly_service = maxscale
log_info = 1
log_notice = 1
log_warning = 1
log_debug = 1
[MariaDB_TEST]
type = server
address = testsan302
port = 3306
protocol = mariadbclient
I don't see coredump file from the below locations:
[root@rnpans401 ~]# cd /var/lib/maxscale
[root@rnpans401 maxscale]# ls
data14435 maxadmin-users maxscale.lock MyMonitor_journal.json passwords.json
LV-RO-Monitor_journal.json maxscale.cnf.d MyMonitor passwd
[root@testans401 maxscale]# ls -lrht
total 24K
drwxr-xr-x 2 maxscale maxscale 6 Jan 28 2022 maxscale.cnf.d
rw-rr- 1 maxscale maxscale 209 Jan 28 2022 passwords.json
rw-rr- 1 maxscale maxscale 678 Feb 7 2022 MyMonitor_journal.json
rw-r---- 1 maxscale maxscale 147 Feb 7 2022 maxadmin-users
drwxr-r- 2 maxscale maxscale 25 Feb 7 2022 MyMonitor
rw------ 1 maxscale maxscale 568 Oct 25 03:02 passwd
rw-rr- 1 maxscale maxscale 722 Apr 25 03:50 LV-RO-Monitor_journal.json
-rwxr-xr-x 1 maxscale maxscale 5 Apr 25 04:34 maxscale.lock
drwxr-xr-x 2 maxscale maxscale 6 Apr 25 04:34 data14435
[root@testans401 maxscale]# ls -lrht data14435
total 0
[root@testans401 maxscale]# ls -lrht /var/lib/systemd/coredump/
total 0
[root@testans401 maxscale]# ls -lrht /usr/bin/maxscale/
ls: cannot access /usr/bin/maxscale/: Not a directory
[root@testans401 maxscale]# ls -lrht /var/lib/systemd/coredump/
total 0
[root@testans401 maxscale]#
Can you please share me the core dump path?
|
|
Does /var/log/maxscale/maxscale.log exists? If it does, is there anything in it?
The location of the coredumps depends on your OS configuration. If you use systemd, I'd recommend starting off by looking at the output of coredumpctl command and seeing if there's anything MaxScale related. If not, the output of sysctl kernel.core_pattern will show you where coredump are stored. If it's something like kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h it means that coredumps are handled by systemd and coredumpctl should show them.
|
|
I mean there is know much info related to crash, its just showing below information from the maxscale log.
[root@testans401 ~]# sysctl kernel.core_pattern
kernel.core_pattern = core
[root@testans401 ~]# coredumpctl
No coredumps found.
From the Maxscale log file:
2023-04-25 04:00:11
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:00:11
notice : MaxScale started with 4 worker threads.
2023-04-25 04:00:18
notice : Message from PAM api when authenticating 'testans': 'Your password will expire in 11 days.
2023-04-25 04:00:18
error : Unknown PAM message type '5'.
2023-04-25 04:00:18
info : Accept authentication from 'testans', using password. Request: /auth
2023-04-25 04:04:10
notice : /etc/maxscale.cnf.d does not exist, not reading.
2023-04-25 04:04:10
warning: The 'log_debug' option has no effect in release mode.
2023-04-25 04:04:10
notice : The logging of debug messages has been enabled.
2023-04-25 04:04:10
notice : The logging of info messages has been enabled.
2023-04-25 04:04:10
notice : The logging of notice messages has been enabled.
2023-04-25 04:04:10
notice : The logging of warning messages has been enabled.
2023-04-25 04:04:10
notice : Using up to 2.32GiB of memory for query classifier cache
2023-04-25 04:04:10
notice : syslog logging is disabled.
2023-04-25 04:04:10
notice : maxlog logging is enabled.
2023-04-25 04:04:10
notice : Host: 'testans401.corp.copart.com' OS: Linux@3.10.0-1160.71.1.el7.x86_64, #1 SMP Tue Jun 28 15:37:28 UTC 2022, x86_64 with 4 processor cores (4.00 available).
2023-04-25 04:04:10
notice : Total main memory: 15.49GiB (15.49GiB usable).
2023-04-25 04:04:10
notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5)
2023-04-25 04:04:10
notice : MaxScale is running in process 12499
2023-04-25 04:04:10
notice : Configuration file: /etc/maxscale.cnf
2023-04-25 04:04:10
notice : Log directory: /var/log/maxscale
2023-04-25 04:04:10
notice : Data directory: /var/lib/maxscale
2023-04-25 04:04:10
notice : Module directory: /usr/lib64/maxscale
2023-04-25 04:04:10
notice : Service cache: /var/cache/maxscale
2023-04-25 04:04:10
notice : Working directory: /var/log/maxscale
2023-04-25 04:04:10
notice : Module 'qc_sqlite' loaded from '/usr/lib64/maxscale/libqc_sqlite.so'.
2023-04-25 04:04:10
info : qc_sqlite loaded.
2023-04-25 04:04:10
notice : Query classification results are cached and reused. Memory used per thread: 595MiB
2023-04-25 04:04:10
notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2023-04-25 04:04:10
notice : The systemd watchdog is Enabled. Internal timeout = 30s
2023-04-25 04:04:10
info : In-memory sqlite database successfully opened for thread 139925527870656.
2023-04-25 04:04:10
info : No 'auto_tune' parameters specified, no auto tuning will be performed.
2023-04-25 04:04:10
notice : Using HS256 for JWT signatures
2023-04-25 04:04:10
notice : Started REST API on [testans401]:8080
2023-04-25 04:04:10
notice : No services defined in any of the configuration files
2023-04-25 04:04:10
info : In-memory sqlite database successfully opened for thread 139925320083200.
2023-04-25 04:04:10
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:04:10
info : In-memory sqlite database successfully opened for thread 139925294933760.
2023-04-25 04:04:10
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:04:10
info : In-memory sqlite database successfully opened for thread 139925269767936.
2023-04-25 04:04:10
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:04:10
info : In-memory sqlite database successfully opened for thread 139925244602112.
2023-04-25 04:04:10
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:04:10
notice : MaxScale started with 4 worker threads.
2023-04-25 04:05:22
notice : Message from PAM api when authenticating 'testans': 'Your password will expire in 11 days.
2023-04-25 04:05:22
error : Unknown PAM message type '5'.
2023-04-25 04:05:22
info : Accept authentication from 'testans', using password. Request: /auth
2023-04-25 04:10:38
info : Accept authentication from 'admin', using password. Request: /auth
2023-04-25 04:34:51
notice : /etc/maxscale.cnf.d does not exist, not reading.
2023-04-25 04:34:51
warning: The 'log_debug' option has no effect in release mode.
2023-04-25 04:34:51
notice : The logging of debug messages has been enabled.
2023-04-25 04:34:51
notice : The logging of info messages has been enabled.
2023-04-25 04:34:51
notice : The logging of notice messages has been enabled.
2023-04-25 04:34:51
notice : The logging of warning messages has been enabled.
2023-04-25 04:34:51
notice : Using up to 2.32GiB of memory for query classifier cache
2023-04-25 04:34:51
notice : syslog logging is disabled.
2023-04-25 04:34:51
notice : maxlog logging is enabled.
2023-04-25 04:34:51
notice : Host: 'testans401.corp.copart.com' OS: Linux@3.10.0-1160.71.1.el7.x86_64, #1 SMP Tue Jun 28 15:37:28 UTC 2022, x86_64 with 4 processor cores (4.00 available).
2023-04-25 04:34:51
notice : Total main memory: 15.49GiB (15.49GiB usable).
2023-04-25 04:34:51
notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5)
2023-04-25 04:34:51
notice : MaxScale is running in process 14435
2023-04-25 04:34:51
notice : Configuration file: /etc/maxscale.cnf
2023-04-25 04:34:51
notice : Log directory: /var/log/maxscale
2023-04-25 04:34:51
notice : Data directory: /var/lib/maxscale
2023-04-25 04:34:51
notice : Module directory: /usr/lib64/maxscale
2023-04-25 04:34:51
notice : Service cache: /var/cache/maxscale
2023-04-25 04:34:51
notice : Working directory: /var/log/maxscale
2023-04-25 04:34:51
notice : Module 'qc_sqlite' loaded from '/usr/lib64/maxscale/libqc_sqlite.so'.
2023-04-25 04:34:51
info : qc_sqlite loaded.
2023-04-25 04:34:51
notice : Query classification results are cached and reused. Memory used per thread: 595MiB
2023-04-25 04:34:51
notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2023-04-25 04:34:51
notice : The systemd watchdog is Enabled. Internal timeout = 30s
2023-04-25 04:34:51
info : In-memory sqlite database successfully opened for thread 139810499599552.
2023-04-25 04:34:51
info : No 'auto_tune' parameters specified, no auto tuning will be performed.
2023-04-25 04:34:51
notice : Using HS256 for JWT signatures
2023-04-25 04:34:51
notice : Started REST API on [testans401]:8080
2023-04-25 04:34:51
notice : No services defined in any of the configuration files
2023-04-25 04:34:51
info : In-memory sqlite database successfully opened for thread 139810291296000.
2023-04-25 04:34:51
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:34:51
info : In-memory sqlite database successfully opened for thread 139810266146560.
2023-04-25 04:34:51
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:34:51
info : In-memory sqlite database successfully opened for thread 139810240980736.
2023-04-25 04:34:51
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:34:51
info : In-memory sqlite database successfully opened for thread 139810215814912.
2023-04-25 04:34:51
info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 04:34:51
notice : MaxScale started with 4 worker threads.
2023-04-25 04:36:06
notice : Message from PAM api when authenticating 'testans': 'Your password will expire in 11 days.
2023-04-25 04:36:06
error : Unknown PAM message type '5'.
2023-04-25 04:36:06
info : Accept authentication from 'testans', using password. Request: /auth
|
|
You could try to run MaxScale manually from the command line and then try to reproduce the problem. This command should allow you to run MaxScale on the command line:
sudo /usr/bin/maxscale -d -lstdout -U maxscale
|
|
|
Still unable to generate core dump file.
[root@testans401 ~]# sudo /usr/bin/maxscale -d -lstdout -U maxscale &
[1] 83549
[root@testans401 ~]# 2023-04-25 21:28:01 notice : MaxScale will be run in the terminal process.
2023-04-25 21:28:01 notice : /etc/maxscale.cnf.d does not exist, not reading.
2023-04-25 21:28:01 warning: The 'log_debug' option has no effect in release mode.
2023-04-25 21:28:01 notice : The logging of debug messages has been enabled.
2023-04-25 21:28:01 notice : The logging of info messages has been enabled.
2023-04-25 21:28:01 notice : The logging of notice messages has been enabled.
2023-04-25 21:28:01 notice : The logging of warning messages has been enabled.
2023-04-25 21:28:01 notice : Using up to 2.32GiB of memory for query classifier cache
2023-04-25 21:28:01 notice : syslog logging is disabled.
2023-04-25 21:28:01 notice : maxlog logging is enabled.
2023-04-25 21:28:01 notice : Host: 'testans401.corp.copart.com' OS: Linux@3.10.0-1160.71.1.el7.x86_64, #1 SMP Tue Jun 28 15:37:28 UTC 2022, x86_64 with 4 processor cores (4.00 available).
2023-04-25 21:28:01 notice : Total main memory: 15.49GiB (15.49GiB usable).
2023-04-25 21:28:01 notice : MariaDB MaxScale 23.02.1 started (Commit: 5de96ecc39da80a992fc8f362d22a030a844bef5)
2023-04-25 21:28:01 notice : MaxScale is running in process 83550
Configuration file : /etc/maxscale.cnf
Log directory : /var/log/maxscale
Data directory : /var/lib/maxscale
Module directory : /usr/lib64/maxscale
Service cache : /var/cache/maxscale
2023-04-25 21:28:01 notice : Configuration file: /etc/maxscale.cnf
2023-04-25 21:28:01 notice : Log directory: /var/log/maxscale
2023-04-25 21:28:01 notice : Data directory: /var/lib/maxscale
2023-04-25 21:28:01 notice : Module directory: /usr/lib64/maxscale
2023-04-25 21:28:01 notice : Service cache: /var/cache/maxscale
2023-04-25 21:28:01 notice : Module 'qc_sqlite' loaded from '/usr/lib64/maxscale/libqc_sqlite.so'.
2023-04-25 21:28:01 info : qc_sqlite loaded.
2023-04-25 21:28:01 notice : Query classification results are cached and reused. Memory used per thread: 595MiB
2023-04-25 21:28:01 notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610602904704.
2023-04-25 21:28:01 info : No 'auto_tune' parameters specified, no auto tuning will be performed.
2023-04-25 21:28:01 notice : Using HS256 for JWT signatures
2023-04-25 21:28:01 notice : Started REST API on [testans401]:8080
2023-04-25 21:28:01 notice : No services defined in any of the configuration files
2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610431850240.
2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610423457536.
2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610415064832.
2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 21:28:01 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 140610406672128.
2023-04-25 21:28:01 info : Epoll instance for listening sockets added to worker epoll instance.
2023-04-25 21:28:01 notice : MaxScale started with 4 worker threads.
2023-04-25 21:28:09 notice : Message from PAM api when authenticating 'nachandra': 'Your password will expire in 11 days.
'
2023-04-25 21:28:09 error : Unknown PAM message type '5'.
2023-04-25 21:28:09 info : Accept authentication from 'nachandra', using password. Request: /auth
[1]+ Segmentation fault sudo /usr/bin/maxscale -d -lstdout -U maxscale
[root@testans401 ~]#
[root@testans401 ~]#
[root@testans401 ~]# ps -ef|grep maxscale
root 83691 82112 0 21:30 pts/6 00:00:00 grep --color=auto maxscale
[root@testans401 ~]# coredumpctl
No coredumps found.
[root@testans401 ~]# sysctl kernel.core_pattern
kernel.core_pattern = core
[root@testans401 ~]# ls -lrth /usr/lib/systemd/systemd-coredump*
-rwxr-xr-x 1 root root 130K Jan 13 2022 /usr/lib/systemd/systemd-coredump
[root@testans401 ~]#
[root@testans401 ~]#
[root@testans401 ~]# ls -lrht /usr/bin/maxscale/
ls: cannot access /usr/bin/maxscale/: Not a directory
[root@testans401 ~]#
[root@testans401 ~]# ls -lrht /var/lib/systemd/coredump/
total 0
[root@testans401 ~]#
|
|
Could you try doing the test again but this time with the debug packages? You can find the debug version of the 23.02.1 release here: https://dlm.mariadb.com/2900358/MaxScale/23.02.1-debug/packages/rhel/7/x86_64/maxscale-23.02.1-1.rhel.7.x86_64.rpm
Another option is to run MaxScale under gdb and see if that catches the problem. Use this to start GDB:
sudo gdb /usr/bin/maxscale
|
Then use this GDB command to start MaxScale:
r -d -lstdout -U maxscale
|
Once it hits that problem, run this command to print the stacktrace:
The stacktrace should show where the problem originates from.
|
|
naresh.chandra@copart.com any updates?
|
|
naresh.chandra@copart.com have you been able to reproduce this outside of a production environment? If you have, can you provide a test case that we can use to reproduce this?
|
|
We'll need information on how to reproduce this or at least a stacktrace to see where the problem is. Right now there's not much to go on so I closed this as Incomplete. Once we have some information or it can be reproduced on another system, we can reopen this issue.
|