GeoffMontee Please attach MaxScale.cnf

Massimiliano Pinto, markus makela Please analyze this ticket.

Comment by matt131:
I think we are seeing this issue happen at the same time a backup kicks off. Maybe this could have something to do with wsrep_desync turning on and off? We've hadd maxscale fail 3 times in the last 12 hours.

Just to confirm, we've disabled backups and the problem has stopped. This issue has something to do with how backups were being ran.

Do we have an insight from the application team?

-Matt

matt131 Can you please locate the core file of crashed MaxScale from your environment and provide it to us ?

Hi,

Do you have instructions on finding that file?

-Matt

I tested with MaxScale and a four node Galera cluster. I set wsrep_desync manually to ON on master and slave nodes and could not reproduce the crash. I also tried doing a backup on master and slave nodes with both xtrabackup and mariadb-backup and could not reproduce the crash.

If you can provide more information about the method of backup that would be much appreciated.

Repeated the previous tests while also blocking network traffic to various nodes. I could not repeat the crash.

Following info is needed:

• Maxscale configuration (MaxScale.cnf file)
• Linux distribution version
• backend configuration (is it Master/slave or Galera?)
• backend servers parameters (my.cnf)
• backend MariaDB/MySQL version

tturenko Please check in eventum 9510

Hello, what is the status of this issue? Is it fixed in 1.2?

I've experienced this issue several times as well. Here is the request info:

• All servers are running Ubuntu 14.04
• MaxScale 1.1.1
• MariaDB Galera Server 10.0.17
• The backend configuration is a 3 node Galera cluster
• Using the R/W Splitter

The crash MaxScale and MariaDB configs are attached as well as the relevant
snippet of the MaxScale error log.

bryantraywick , Can you install a debug version of the build to reproduce the crash ?

Here is what you need to do

(1) uninstall current version of MaxScale and install debug version

service maxscale stop

sudo apt-get remove maxscale

deb http://downloads.mariadb.com/enterprise/f8rm-9k90/mariadb-maxscale/1.2-debug/ubuntu/  trusty main

(2) Configure maxscale per your environment

(3) Enable coredump

ulimit -c unlimited

export DAEMON_COREFILE_LIMIT='unlimited'

echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

echo 2 > /proc/sys/fs/suid_dumpable

service maxscale restart

(3) Run your traffic that leads to crash

Now the coredump will be in /tmp and will have name patterned as core-maxscale-*

Please collect the coredump as well as config file and log files and attach to this Jira.

The coredump will help us easily identify the root cause and fix it.

A report from a crash while running a debug build would be extremely helpful for making progress with this fault. Please note that the debug and trace logs should configured off for production running of a debug build, since otherwise you will suffer excessive log size. This is done through the MaxScale configuration file, something like:

[maxscale]

threads=6

log_debug=0

log_trace=0

markus makela based on stacktrace please identify the line number where this is crashing in readwrite split.

The failure looks likely to have been at what was line 1651 in release 1.1.1 (1652 in release 1.2) which was:

data = (MYSQL_session*)master_dcb->session->data;

This could be caused by either master_dcb or session being NULL, and the latter is believed to be more likely. MXS-329 is working to guarantee that the session pointer in a DCB cannot be NULL.

NULL value checks were added to the temporary table functions and the calls to those functions are now properly done under a spinlock. As was stated before, this is a treatment to the symptoms of NULL DCB/Session pointers and the real fix will be done in MXS-329.