[MXS-164] setting log_debug=1 triggers signal 11 crash Created: 2015-05-20  Updated: 2015-11-25  Resolved: 2015-11-25

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 1.1.0
Fix Version/s: 1.3.0

Type: Bug Priority: Minor
Reporter: Glenn Plas Assignee: Massimiliano Pinto (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: logging
Environment:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"

Attachments: File MaxScale.cnf    
Issue Links:
Blocks
blocks MXS-175 sudden signal 6 abort after running f... Closed

 Description   

Without any log_debug settings, maxscale actually seems to run fine. Once you set this flag to 1, it will not start and crash with signal 11.

It happens, nomatter what config you use, below is a tracer of an unfinished setup but I have a semi-production setup that does exactly the same. All ubuntu 14.04

May 20 11:55:48 ttextlb1 MaxScale[18055]: Home directory : /usr/local/mariadb-maxscale
May 20 11:55:48 ttextlb1 MaxScale[18055]: Data directory : /usr/local/mariadb-maxscale/data/data18055
May 20 11:55:48 ttextlb1 MaxScale[18055]: Log directory : /usr/local/mariadb-maxscale/log
May 20 11:55:48 ttextlb1 MaxScale[18055]: Configuration file : /usr/local/mariadb-maxscale/etc/MaxScale.cnf
May 20 11:55:48 ttextlb1 MaxScale[18055]: Initialise CLI router module V1.0.0.
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module cli: V1.0.0 from /usr/local/mariadb-maxscale/modules/libcli.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Initialise debug CLI router module V1.1.1.
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module debugcli: V1.1.1 from /usr/local/mariadb-maxscale/modules/libdebugcli.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Initializing statemend-based read/write split router module.
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module readwritesplit: V1.0.2 from /usr/local/mariadb-maxscale/modules/libreadwritesplit.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Error: Unable to find server 'slave2' that is configured as part of service 'RW Split Router'.
May 20 11:55:48 ttextlb1 MaxScale[18055]: Initialise the MySQL Monitor module V1.4.0.
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module mysqlmon: V1.4.0 from /usr/local/mariadb-maxscale/modules/libmysqlmon.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Error: Unable to find server 'slave2' that is configured in the monitor 'MySQL Monitor'.
May 20 11:55:48 ttextlb1 MaxScale[18055]: MariaDB Corporation MaxScale 1.1.0 (C) MariaDB Corporation Ab 2013-2014
May 20 11:55:48 ttextlb1 MaxScale[18055]: MaxScale is running in process 18055
May 20 11:55:48 ttextlb1 MaxScale[18055]: Encrypted password file /usr/local/mariadb-maxscale/etc/.secrets can't be accessed (No such file or directory). Password encryption is not used.
May 20 11:55:48 ttextlb1 MaxScale[18055]: Error : Monitor was unable to connect to server 192.168.128.89:3306 : "Access denied for user 'lb_mon'@'ttextlb1' (using password: YES)"
May 20 11:55:48 ttextlb1 MaxScale[18055]: Warning: Duplicate MySQL user found for service [RW Split Router]: brenden@% for database: ticketing_archive_master
May 20 11:55:48 ttextlb1 MaxScale[18055]: Warning: Duplicate MySQL user found for service [RW Split Router]: brenden@% for database: ticketing_unittest_master
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded 33 MySQL Users for service [RW Split Router].
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module MySQLClient: V1.0.0 from /usr/local/mariadb-maxscale/modules/libMySQLClient.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Listening MySQL connections at 0.0.0.0:4006
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module telnetd: V1.0.1 from /usr/local/mariadb-maxscale/modules/libtelnetd.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Listening telnet connections at 127.0.0.1:4442
May 20 11:55:48 ttextlb1 MaxScale[18055]: Loaded module maxscaled: V1.0.0 from /usr/local/mariadb-maxscale/modules/libmaxscaled.so
May 20 11:55:48 ttextlb1 MaxScale[18055]: Listening maxscale connections at 0.0.0.0:6603
May 20 11:55:48 ttextlb1 MaxScale[18055]: Started MaxScale log flusher.
May 20 11:55:48 ttextlb1 MaxScale[18055]: MaxScale started with 4 server threads.
May 20 11:56:14 ttextlb1 MaxScale[18055]: MaxScale received signal SIGTERM. Exiting.
May 20 11:56:14 ttextlb1 MaxScale[18055]: Finished MaxScale log flusher.
May 20 11:56:14 ttextlb1 MaxScale[18055]: MaxScale is shutting down.
May 20 11:56:14 ttextlb1 MaxScale[18055]: MaxScale shutdown completed.
May 20 11:56:32 ttextlb1 MaxScale[18196]: Home directory : /usr/local/mariadb-maxscale
May 20 11:56:32 ttextlb1 MaxScale[18196]: Data directory : /usr/local/mariadb-maxscale/data/data18196
May 20 11:56:32 ttextlb1 MaxScale[18196]: Log directory : /usr/local/mariadb-maxscale/log
May 20 11:56:32 ttextlb1 MaxScale[18196]: Configuration file : /usr/local/mariadb-maxscale/etc/MaxScale.cnf
May 20 11:56:32 ttextlb1 MaxScale[18196]: Fatal: MaxScale received fatal signal 11. Attempting backtrace.
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale() [0x5483e3]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f4b23861340]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /lib/x86_64-linux-gnu/libc.so.6(+0x97d3a) [0x7f4b224f9d3a]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale() [0x55caca]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale(ini_parse_file+0x388) [0xae0d03]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale(ini_parse+0x50) [0xae0dcd]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale(config_load+0xed) [0x55cca9]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale(main+0x17b3) [0x54ac79]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4b22483ec5]
May 20 11:56:32 ttextlb1 MaxScale[18196]: /usr/local/mariadb-maxscale/bin/maxscale() [0x54722a]
May 20 11:56:46 ttextlb1 MaxScale[18212]: Home directory : /usr/local/mariadb-maxscale
May 20 11:56:46 ttextlb1 MaxScale[18212]: Data directory : /usr/local/mariadb-maxscale/data/data18212
May 20 11:56:46 ttextlb1 MaxScale[18212]: Log directory : /usr/local/mariadb-maxscale/log
May 20 11:56:46 ttextlb1 MaxScale[18212]: Configuration file : /usr/local/mariadb-maxscale/etc/MaxScale.cnf
May 20 11:56:46 ttextlb1 MaxScale[18212]: Fatal: MaxScale received fatal signal 11. Attempting backtrace.
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale() [0x5483e3]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f73f3b3f340]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /lib/x86_64-linux-gnu/libc.so.6(+0x97d3a) [0x7f73f27d7d3a]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale() [0x55caca]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale(ini_parse_file+0x388) [0xae0d03]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale(ini_parse+0x50) [0xae0dcd]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale(config_load+0xed) [0x55cca9]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale(main+0x17b3) [0x54ac79]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f73f2761ec5]
May 20 11:56:46 ttextlb1 MaxScale[18212]: /usr/local/mariadb-maxscale/bin/maxscale() [0x54722a]

 Comments   
Comment by Massimiliano Pinto (Inactive) [ 2015-06-11 ]

gplv2 In the attached maxScale.cnf I see:

#log_debug=1

Is the issue about starting maxscale with that option set or changing the value via maxadmin ?

Did you try with MaxScale 1.1.1 as well?

Could you please start MaxScale whith gdb ?

#gdb ./maxscale
(gdb) run -d -c ../

and if the issue is found, please type in gdb

(gdb) where

This way we get full stack trace

Thanks

Comment by Glenn Plas [ 2015-06-11 ]

"Once you set this flag to 1, it will not start" -> maybe I wasn't too clear on this.

it's commented out in the config due the the sequence of troubleshooting these events, but it's a startup issue for me. It will launch but immediately crash and syslog this info.

Will test 1.1.1 and get back to you. Not sure if I can do the gdb anytime soon and trigger it. It takes days between crashes and we kind of need this to be up. I will try though.

Comment by Glenn Plas [ 2015-06-12 ]

Hi,

I've tested switching on trace and debug log using maxadmin. This does not cause a crash so far. So once it is running it seems to be working. I got you the stack trace as this was reproducable.

root@ttextlb2:/etc/maxscale/etc# gdb /usr/local/mariadb-maxscale/bin/maxscale
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/mariadb-maxscale/bin/maxscale...done.
(gdb) run -d -c ../
Starting program: /usr/local/mariadb-maxscale/bin/maxscale -d -c ../
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Info : MaxScale will be run in the terminal process.
Using Home directory command-line argument as MAXSCALE_HOME = /usr/local/mariadb-maxscale
[New Thread 0x7ffff1c04700 (LWP 15461)]
Home directory : /usr/local/mariadb-maxscale
Configuration file : /usr/local/mariadb-maxscale/etc/MaxScale.cnf
Log directory : /usr/local/mariadb-maxscale/log
Data directory : /usr/local/mariadb-maxscale/data/data15457

[New Thread 0x7ffff1403700 (LWP 15462)]
[Thread 0x7ffff1403700 (LWP 15462) exited]
[New Thread 0x7ffff1403700 (LWP 15463)]

Program received signal SIGSEGV, Segmentation fault.
__strcmp_sse2_unaligned () at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:29
29 ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S: No such file or directory.
(gdb) where
#0 __strcmp_sse2_unaligned () at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:29
#1 0x000000000055caca in handler (userdata=0x7fffffffb180, section=0x7fffffffaf90 "", name=0x7fffffffb010 "log_debug", value=0x7fffffffb01a "1") at /home/ec2-user/workspace/server/core/config.c:173
#2 0x0000000000ae0d03 in ini_parse_file (file=0x1a15510, handler=0x55c967 <handler>, user=0x7fffffffb180) at /home/ec2-user/workspace/server/inih/ini.c:145
#3 0x0000000000ae0dcd in ini_parse (filename=0x19de570 "/usr/local/mariadb-maxscale/etc/MaxScale.cnf", handler=0x55c967 <handler>, user=0x7fffffffb180) at /home/ec2-user/workspace/server/inih/ini.c:173
#4 0x000000000055cca9 in config_load (file=0x19de570 "/usr/local/mariadb-maxscale/etc/MaxScale.cnf") at /home/ec2-user/workspace/server/core/config.c:228
#5 0x000000000054ac79 in main (argc=4, argv=0x7fffffffe628) at /home/ec2-user/workspace/server/core/gateway.c:1771

Comment by Glenn Plas [ 2015-06-12 ]

to avoid confusion with paths in the above , I have softlinked the maxscale directory under /etc

lrwxrwxrwx 1 root root 27 May 11 00:37 maxscale -> /usr/local/mariadb-maxscale

Comment by Dipti Joshi (Inactive) [ 2015-06-16 ]

gplv2 Can you clarify whether you reproduced this with 1.1.1 version of MaxScale ?

Comment by Dipti Joshi (Inactive) [ 2015-07-17 ]

gplv2 1.2.0 is out, can you try to reproduce this with 1.2.0 now ?

Comment by markus makela [ 2015-11-25 ]

We have not been able to reproduce this so we are closing this for now. If the problem still appears with 1.3.0, this bug can be reopened.

Generated at Thu Feb 08 03:57:16 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.