szemlyanoy Can you please attach the core-dump file ?

Sorry, where can I get it?

You need do following to enable coredump

ulimit -c unlimited

export DAEMON_COREFILE_LIMIT='unlimited'

echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

echo 2 > /proc/sys/fs/suid_dumpable

service maxscale restart

After this run your traffic that causes fatal singal 11, you should find the core file in /tmp directory. It will be named core-maxscale-*
Find the latest such core file generated and attach it to this jira item.

Guys,

Attaching of coredump would be cumbersome since it's about 400mb.
I opened generated coredump and attached the output to the ticket. Please check if it's enough or let me know to provide more details.
Looking forward for hearing from you

szemlyanoy What is the exact OS version of your system ? We need to give you debug build for your OS version- so that the coredump will be useful.

Hi,

CentOS release 6.5 (Final)

tturenko, markus makela Please generate a debug build of 1.1.1 on CentOS 6.5 (Final) - so that we can give to Sergey to produce coredump.

szemlyanoy Here you can download MaxScale debug build rpm from http://maxscale-jenkins.mariadb.com/repository/1.1.1-debug/mariadb-maxscale/yum/centos6/x86_64/

Change repo config in your /etc/yum.repo.d/maxscale.repo

[maxscale]

name=maxscale

baseurl= http://maxscale-jenkins.mariadb.com/repository/1.1.1-debug/mariadb-maxscale/yum/centos/6/x86_64

enabled=1

gpgcheck=false

Then do following to reinstall maxscale

yum reinstall maxscale

Next ,enable coredump as per earlier instructions, run traffic to cause crash and generate coredump.

If you could upload the generated coredump somewhere and provide us a link - we will download from there.

Also include all the log files for MaxScale.

please check it now

Was the core generated with the debug build?

szemlyanoyThanks for providing these core dumps - unfortunately the debug RPM that we provided you earlier had some build issues and hence was missing debug symbols. We have fixed it now. If you could do following again,

yum reinstall maxscale

Then run the traffic to cause the crash, and provide us the new core dump file please.

Thanks for providing us with these coredumps.

The core dump doesn't seem to work with the debug RPM. The command issued was gdb --core=<path to core> /usr/local/mariadb-maxscale/bin/maxscale. This did load the executable which had the debugging symbols but the core file still did not display correctly with debug info.

You can confirm that the binary has debug symbols by starting GDB with the maxscale binary as the parameter:
gdb /usr/local/mariadb-maxscale/bin/maxscale

GDB should output some information and as the last line it should print:
Reading symbols from /usr/local/mariadb-maxscale/bin/maxscale...done.

If there are no debugging symbols, the following is printed:
Reading symbols from /usr/local/mariadb-maxscale/bin/maxscale...(no debugging symbols found)...done.

Here is the GDB output of the latest core file with the latest RPM:

Reading symbols from /usr/local/mariadb-maxscale/modules/libcli.so...done.

Loaded symbols for /usr/local/mariadb-maxscale/modules/libcli.so

Reading symbols from /usr/local/mariadb-maxscale/modules/libdebugcli.so...done.

Loaded symbols for /usr/local/mariadb-maxscale/modules/libdebugcli.so

Reading symbols from /usr/local/mariadb-maxscale/modules/libreadconnroute.so...done.

Loaded symbols for /usr/local/mariadb-maxscale/modules/libreadconnroute.so

Reading symbols from /usr/local/mariadb-maxscale/modules/libmysqlmon.so...done.

Loaded symbols for /usr/local/mariadb-maxscale/modules/libmysqlmon.so

Core was generated by `/usr/local/mariadb-maxscale/bin/maxscale'.

Program terminated with signal 11, Segmentation fault.

#0  0x00000033b92371b2 in ?? ()

Missing separate debuginfos, use: debuginfo-install maxscale-1.1.1-2.x86_64

(gdb) bt

#0  0x00000033b92371b2 in ?? ()

#1  0x00007f314c0051a0 in ?? ()

#2  0x0000000000000000 in ?? ()

(gdb) thr appl all bt

Thread 4 (Thread 0x7f3167c7a7e0 (LWP 39473)):

#0  0x00000033b960e75d in fcntl () from /lib64/libpthread.so.0

#1  0x0000000003726050 in ?? ()

#2  0x0000000003726050 in ?? ()

#3  0x00007fff112b7d40 in ?? ()

#4  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f3167c78700 (LWP 39476)):

#0  0x00000033b960b5bc in __condvar_cleanup1 () from /lib64/libpthread.so.0

#1  0x00007f3167ea2a8f in skygw_message_wait (mes=0x364c4d0) at /home/ec2-user/workspace/utils/skygw_utils.cc:1615

#2  0x00007f3167e99b75 in thr_filewriter_fun (data=0x364d690) at /home/ec2-user/workspace/log_manager/log_manager.cc:2855

#3  0x00000033b96079d1 in start_thread () from /lib64/libpthread.so.0

#4  0x00000033b92e8b5d in ?? ()

#5  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f3167277700 (LWP 39478)):

#0  0x00000033b960b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000000000092d964 in inline_mysql_cond_timedwait (control=0x11b9970, sleep_time=<value optimized out>) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1020

#2  my_service_thread_sleep (control=0x11b9970, sleep_time=<value optimized out>) at /home/buildbot/buildbot/build/storage/maria/ma_servicethread.c:118

#3  0x0000000000926626 in ma_checkpoint_background (arg=<value optimized out>) at /home/buildbot/buildbot/build/storage/maria/ma_checkpoint.c:705

#4  0x00000033b96079d1 in start_thread () from /lib64/libpthread.so.0

#5  0x00000033b92e8b5d in ?? ()

#6  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f3151a5d700 (LWP 39479)):

#0  0x00000033b92371b2 in ?? ()

#1  0x00007f314c0051a0 in ?? ()

#2  0x0000000000000000 in ?? ()

Another option is to run MaxScale under GDB until the segmentation fault appears. After the crash has happened either generate the core dump manually with the command generate--core-file or examine the backtrace with the command backtrace.

szemlyanoy Looks like the your reinstall did not pick up the updated debug binaries.

With your /etc/yum.repo.d/maxscale.repo looking like this

[maxscale]

name=maxscale

baseurl= http://maxscale-jenkins.mariadb.com/repository/1.1.1-debug/mariadb-maxscale/yum/centos/6/x86_64

enabled=1

gpgcheck=false

Please do following

yum remove maxscale

yum install maxscale

And then generate the core. Thanks for working with us on this.

Reproduced!

sql_queries test with Maxscale 1.1.1 and MySLQ 5.1 backend.

Program terminated with signal 11, Segmentation fault.
#0 0x00007fa7b683e132 in ____strtoll_l_internal () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install maxscale-1.1.1-2.x86_64
(gdb) bt
#0 0x00007fa7b683e132 in ____strtoll_l_internal () from /lib64/libc.so.6
#1 0x00007fa7b14947dd in monitorDatabase (handle=0x24e0160, database=0x24e24e0) at /home/ec2-user/workspace/server/modules/monitor/mysql_mon.c:570
#2 0x00007fa7b1494a48 in monitorMain (arg=0x24e0160) at /home/ec2-user/workspace/server/modules/monitor/mysql_mon.c:667
#3 0x00007fa7b7f7d851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fa7b68ef90d in clone () from /lib64/libc.so.6

Thanks tturenko !

Please send coredump with debug builds to markus makela for analysis !

szemlyanoy We have reproduced and identified the issue. We are working on a fix. Once the fix is ready we will let you know.

Hi guys,

Thanks for update. Looking forward for a fix

Hi guys,
If it's possible could you please provide any ETA on fix release?

Cheers
Sergey

The fix has been implemented and is under testing. To my knowledge this will be in the 1.2 release.

Hi

Can someone provide me with yum repository with recently released maxscale v1.2 ?

Thanks

Please find it here https://mariadb.com/my_portal/download - under "MariaDB Enterprise Repository" click on RedHat/CentOS

but there is no maxscale rpms.
I can manually obtain maxscale rpms on https://mariadb.com/my_portal/download/maxscale, but I'm looking for yum repo to avoid manual downloading.

When you install "mariadb-enterprise-repository.rpm" per Step 1- it configures the /etc/yum/repo.d file with mariadb-enterprise as well as maxscale

And then in step 2 you do "yum install maxscale"

Then going forward your maxscale will stay uptodate

Thanks!