[MDEV-10809] Logrotates flush-logs process never ends Created: 2016-09-14  Updated: 2017-12-13  Resolved: 2017-12-13

Status: Closed
Project: MariaDB Server
Component/s: Galera, Replication
Affects Version/s: 10.1.17
Fix Version/s: 10.1.30

Type: Bug Priority: Major
Reporter: André Bauer Assignee: Sachin Setiya (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Environment:

Ubuntu 14.04 & 16.06
Galera Cluster

Issue Links:
Relates
relates to MDEV-9510 Segmentation fault in binlog thread c... Closed
relates to MDEV-11610 Logrotate to only FLUSH LOCAL ERROR L... Closed

 Description   

Every day logrotate trys to flush the mariadb logs but the process never stops so that it just starts a new one the next day.

After some time your server would crash because of this. Only way to solve the problem is to restart mariadb server.

ps faux example:

root      1227  0.0  0.0  23648   896 ?        Ss   Aug30   0:01 cron
 
root     24631  0.0  0.0  59632  1444 ?        S    Sep10   0:00  \_ CRON
 
root     24632  0.0  0.0   4440   648 ?        Ss   Sep10   0:00  |   \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
 
root     24633  0.0  0.0   4332   612 ?        S    Sep10   0:00  |       \_ run-parts --report /etc/cron.daily
 
root     28256  0.0  0.0   4440   648 ?        S    Sep10   0:00  |           \_ /bin/sh /etc/cron.daily/logrotate
 
root     28261  0.0  0.0  35696  1316 ?        S    Sep10   0:00  |               \_ /usr/sbin/logrotate /etc/logrotate.conf
 
root     28265  0.0  0.0   4440   652 ?        S    Sep10   0:00  |                   \_ sh -c            test -x /usr/bin/mysqladmin || exit 0            if [ -f `my_pri
 
root     28269  0.0  0.0  36400  2544 ?        S    Sep10   0:00  |                       \_ mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
 
root       948  0.0  0.0  59632  1448 ?        S    Sep11   0:00  \_ CRON
 
root       949  0.0  0.0   4440   648 ?        Ss   Sep11   0:00  |   \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
 
root       950  0.0  0.0   4332   608 ?        S    Sep11   0:00  |       \_ run-parts --report /etc/cron.daily
 
root      3887  0.0  0.0   4440   652 ?        S    Sep11   0:00  |           \_ /bin/sh /etc/cron.daily/logrotate
 
root      3892  0.0  0.0  35696  1316 ?        S    Sep11   0:00  |               \_ /usr/sbin/logrotate /etc/logrotate.conf
 
root      3893  0.0  0.0   4440   652 ?        S    Sep11   0:00  |                   \_ sh -c            test -x /usr/bin/mysqladmin || exit 0            if [ -f `my_pri
 
root      3897  0.0  0.0  36400  2544 ?        S    Sep11   0:00  |                       \_ mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
 
root      9959  0.0  0.0  59632  1448 ?        S    Sep12   0:00  \_ CRON
 
root      9960  0.0  0.0   4440   648 ?        Ss   Sep12   0:00  |   \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
 
root      9961  0.0  0.0   4332   608 ?        S    Sep12   0:00  |       \_ run-parts --report /etc/cron.daily
 
root     12427  0.0  0.0   4440   652 ?        S    Sep12   0:00  |           \_ /bin/sh /etc/cron.daily/logrotate
 
root     12432  0.0  0.0  35696  1320 ?        S    Sep12   0:00  |               \_ /usr/sbin/logrotate /etc/logrotate.conf
 
root     12433  0.0  0.0   4440   652 ?        S    Sep12   0:00  |                   \_ sh -c            test -x /usr/bin/mysqladmin || exit 0            if [ -f `my_pri
 
root     12437  0.0  0.0  36400  2544 ?        S    Sep12   0:00  |                       \_ mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
 
root     17501  0.0  0.0  59632  1448 ?        S    Sep13   0:00  \_ CRON
 
root     17502  0.0  0.0   4440   644 ?        Ss   Sep13   0:00  |   \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
 
root     17503  0.0  0.0   4332   608 ?        S    Sep13   0:00  |       \_ run-parts --report /etc/cron.daily
 
root     19805  0.0  0.0   4440   648 ?        S    Sep13   0:00  |           \_ /bin/sh /etc/cron.daily/logrotate
 
root     19810  0.0  0.0  35696  1316 ?        S    Sep13   0:00  |               \_ /usr/sbin/logrotate /etc/logrotate.conf
 
root     19811  0.0  0.0   4440   648 ?        S    Sep13   0:00  |                   \_ sh -c            test -x /usr/bin/mysqladmin || exit 0            if [ -f `my_pri
 
root     19815  0.0  0.0  36400  2540 ?        S    Sep13   0:00  |                       \_ mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs
 
root     25774  0.0  0.0  59632  1448 ?        S    06:25   0:00  \_ CRON
 
root     25775  0.0  0.0   4440   652 ?        Ss   06:25   0:00      \_ /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
 
root     25776  0.0  0.0   4332   608 ?        S    06:25   0:00          \_ run-parts --report /etc/cron.daily
 
root     29098  0.0  0.0   4440   648 ?        S    06:51   0:00              \_ /bin/sh /etc/cron.daily/logrotate
 
root     29103  0.0  0.0  35696  1316 ?        S    06:51   0:00                  \_ /usr/sbin/logrotate /etc/logrotate.conf
 
root     29107  0.0  0.0   4440   652 ?        S    06:51   0:00                      \_ sh -c            test -x /usr/bin/mysqladmin || exit 0            if [ -f `my_pri
 
root     29111  0.0  0.0  36400  2540 ?        S    06:51   0:00                          \_ mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logs



 Comments   
Comment by Elena Stepanova [ 2016-09-14 ]

Could you please

  • run install soname 'metadata_lock_info' from the client;
  • wait till there is another hanging FLUSH;
  • when it happens, connect to the server;
  • run 

    SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST;
    		 
    SELECT * FROM INFORMATION_SCHEMA.METADATA_LOCK_INFO;

  • paste the output of both commands, and also attach your cnf file.

If you have gdb on the server, maybe you could also run
gdb --batch --eval-command="thread apply all bt" <path to mysqld> <mysqld PID>.

Comment by André Bauer [ 2016-09-22 ]

OK, restarted all mariadb servers and installed the plugin.

I, ll give feedback when i have another hanging logrotate...

Comment by André Bauer [ 2016-09-23 ]

This is the output of: SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST;

MariaDB [(none)]> SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST;
 
+------+------------------+-------------------------+------+---------+-------+----------------------+----------------------------------------------+--------------+-------+-----------+----------+-------------+---------------+----------+----------------------------------------------+-------+
 
| ID   | USER             | HOST                    | DB   | COMMAND | TIME  | STATE                | INFO                                         | TIME_MS      | STAGE | MAX_STAGE | PROGRESS | MEMORY_USED | EXAMINED_ROWS | QUERY_ID | INFO_BINARY                                  | TID   |
 
+------+------------------+-------------------------+------+---------+-------+----------------------+----------------------------------------------+--------------+-------+-----------+----------+-------------+---------------+----------+----------------------------------------------+-------+
 
| 3354 | root             | localhost               | NULL | Query   |     0 | Filling schema table | SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST |        0.306 |     0 |         0 |    0.000 |       84576 |             0 |  2708042 | SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST | 20043 |
 
| 3351 | kix              | some.mysql.client:39434 | kix  | Sleep   |     6 |                      | NULL                                         |     6816.411 |     0 |         0 |    0.000 |       67464 |             0 |  2707802 | NULL                                         | 19224 |
 
| 3350 | otrs_search      | some.mysql.client:38911 | kix  | Sleep   |    42 |                      | NULL                                         |    42514.773 |     0 |         0 |    0.000 |       67464 |             0 |  2706423 | NULL                                         | 19495 |
 
| 3348 | kix              | some.mysql.client:38716 | kix  | Sleep   |     4 |                      | NULL                                         |     4725.912 |     0 |         0 |    0.000 |       67464 |             0 |  2707874 | NULL                                         | 19496 |
 
| 3332 | otrs_search      | some.mysql.client:59844 | kix  | Sleep   |   203 |                      | NULL                                         |   203703.748 |     0 |         0 |    0.000 |       67464 |             0 |  2698487 | NULL                                         | 12152 |
 
| 3329 | kix              | some.mysql.client:59240 | kix  | Sleep   |     1 |                      | NULL                                         |     1325.749 |     0 |         0 |    0.000 |       67464 |             0 |  2707924 | NULL                                         | 19466 |
 
| 3310 | kix              | some.mysql.client:55720 | kix  | Sleep   |     2 |                      | NULL                                         |     2243.492 |     0 |         0 |    0.000 |       67464 |             0 |  2707919 | NULL                                         | 19465 |
 
| 3301 | kix              | some.mysql.client:54290 | kix  | Sleep   |    19 |                      | NULL                                         |    19419.194 |     0 |         0 |    0.000 |       67464 |             0 |  2707252 | NULL                                         | 19213 |
 
| 3299 | itsm             | some.mysql.client:54237 | itsm | Sleep   |    18 |                      | NULL                                         |    18549.472 |     0 |         0 |    0.000 |       67464 |             0 |  2707320 | NULL                                         | 19212 |
 
| 3257 | kix              | some.mysql.client:47398 | kix  | Sleep   |     3 |                      | NULL                                         |     3550.323 |     0 |         0 |    0.000 |       67464 |             0 |  2707912 | NULL                                         | 19215 |
 
| 3251 | otrs_search      | some.mysql.client:46529 | kix  | Sleep   |    39 |                      | NULL                                         |    39601.288 |     0 |         0 |    0.000 |       67464 |             0 |  2706635 | NULL                                         | 19497 |
 
| 3244 | kix              | some.mysql.client:44585 | kix  | Sleep   |     0 |                      | NULL                                         |        9.886 |     0 |         0 |    0.000 |       67464 |             0 |  2708041 | NULL                                         | 26878 |
 
| 2961 | debian-sys-maint | localhost               | NULL | Query   | 12272 | init                 | flush logs                                   | 12272972.642 |     0 |         0 |    0.000 |       69120 |             0 |  2460364 | flush logs                                   | 20044 |
 
|   52 | maxscale         | some.mysql.client:33052 | NULL | Sleep   |     1 |                      | NULL                                         |     1726.428 |     0 |         0 |    0.000 |       67464 |             0 |  2707921 | NULL                                         | 19211 |
 
|   51 | maxscale         | some.mysql.client:34879 | NULL | Sleep   |     7 |                      | NULL                                         |     7308.289 |     0 |         0 |    0.000 |       67464 |             0 |  2707801 | NULL                                         | 19143 |
 
|   18 | system user      |                         | NULL | Sleep   |     2 | committed 81498151   | NULL                                         |     2745.287 |     0 |         0 |    0.000 |       52120 |             0 |  2707914 | NULL                                         | 19139 |
 
|   17 | system user      |                         | NULL | Sleep   |     6 | committed 81498138   | NULL                                         |     6174.088 |     0 |         0 |    0.000 |       52120 |             0 |  2707830 | NULL                                         | 19137 |
 
|   16 | system user      |                         | NULL | Sleep   |     4 | committed 81498144   | NULL                                         |     4808.828 |     0 |         0 |    0.000 |       52120 |             0 |  2707865 | NULL                                         | 19126 |
 
|   15 | system user      |                         | NULL | Sleep   |     7 | committed 81498134   | NULL                                         |     7784.357 |     0 |         0 |    0.000 |       51064 |             0 |  2707797 | NULL                                         | 19136 |
 
|   14 | system user      |                         | NULL | Sleep   |     7 | committed 81498133   | NULL                                         |     7787.316 |     0 |         0 |    0.000 |       52120 |             0 |  2707796 | NULL                                         | 19130 |
 
|   13 | system user      |                         | NULL | Sleep   |     4 | committed 81498147   | NULL                                         |     4252.400 |     0 |         0 |    0.000 |       52120 |             0 |  2707894 | NULL                                         | 19135 |
 
|   12 | system user      |                         | NULL | Sleep   |    11 | committed 81498087   | NULL                                         |    11236.887 |     0 |         0 |    0.000 |       52120 |             0 |  2707568 | NULL                                         | 19125 |
 
|   11 | system user      |                         | NULL | Sleep   |     7 | committed 81498135   | NULL                                         |     7781.535 |     0 |         0 |    0.000 |       52120 |             0 |  2707798 | NULL                                         | 19134 |
 
|   10 | system user      |                         | NULL | Sleep   |     7 | committed 81498132   | NULL                                         |     7790.557 |     0 |         0 |    0.000 |       52120 |             0 |  2707795 | NULL                                         | 19129 |
 
|    9 | system user      |                         | NULL | Sleep   |     4 | committed 81498143   | NULL                                         |     4811.543 |     0 |         0 |    0.000 |       52120 |             0 |  2707864 | NULL                                         | 19133 |
 
|    8 | system user      |                         | NULL | Sleep   |     2 | committed 81498152   | NULL                                         |     2742.134 |     0 |         0 |    0.000 |       51064 |             0 |  2707915 | NULL                                         | 19138 |
 
|    7 | system user      |                         | NULL | Sleep   |     2 | committed 81498153   | NULL                                         |     2734.068 |     0 |         0 |    0.000 |       52120 |             0 |  2707916 | NULL                                         | 19127 |
 
|    6 | system user      |                         | NULL | Sleep   |    11 | committed 81498086   | NULL                                         |    11250.099 |     0 |         0 |    0.000 |       52120 |             0 |  2707567 | NULL                                         | 19132 |
 
|    5 | system user      |                         | NULL | Sleep   |     6 | committed 81498139   | NULL                                         |     6170.581 |     0 |         0 |    0.000 |       52120 |             0 |  2707831 | NULL                                         | 19128 |
 
|    4 | system user      |                         | NULL | Sleep   |    11 | committed 81498085   | NULL                                         |    11253.073 |     0 |         0 |    0.000 |       52120 |             0 |  2707566 | NULL                                         | 19131 |
 
|    2 | system user      |                         | NULL | Sleep   |     7 | committed 81498136   | NULL                                         |     7778.601 |     0 |         0 |    0.000 |       52120 |             0 |  2707799 | NULL                                         | 19014 |
 
|    1 | system user      |                         | NULL | Sleep   | 86933 | wsrep aborter idle   | NULL                                         | 86933410.467 |     0 |         0 |    0.000 |       52120 |             0 |        0 | NULL                                         | 19013 |
 
+------+------------------+-------------------------+------+---------+-------+----------------------+----------------------------------------------+--------------+-------+-----------+----------+-------------+---------------+----------+----------------------------------------------+-------+
 
32 rows in set (0.00 sec)

This is the output of: SELECT * FROM INFORMATION_SCHEMA.METADATA_LOCK_INFO;

MariaDB [(none)]> SELECT * FROM INFORMATION_SCHEMA.METADATA_LOCK_INFO;
 
Empty set (0.00 sec)

my.cnf:

# puppet
 
# MariaDB database server configuration file.
 
#
 
# You can copy this file to one of:
 
# - "/etc/mysql/my.cnf" to set global options,
 
# - "~/.my.cnf" to set user-specific options.
 
# 
# One can use all long options that the program supports.
 
# Run program with --help to get a list of available options and with
 
# --print-defaults to see which it would actually understand and use.
 
#
 
# For explanations see
 
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
 
 
 
# This will be passed to all mysql clients
 
# It has been reported that passwords should be enclosed with ticks/quotes
 
# escpecially if they contain "#" chars...
 
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
 
[client]
 
port            = 3306
 
socket          = /var/run/mysqld/mysqld.sock
 
 
 
# Here is entries for some specific programs
 
# The following values assume you have at least 32M ram
 
 
 
# This was formally known as [safe_mysqld]. Both versions are currently parsed.
 
[mysqld_safe]
 
socket          = /var/run/mysqld/mysqld.sock
 
nice            = 0
 
 
 
[mysqld]
 
#
 
# * Basic Settings
 
#
 
user            = mysql
 
pid-file        = /var/run/mysqld/mysqld.pid
 
socket          = /var/run/mysqld/mysqld.sock
 
port            = 3306
 
basedir         = /usr
 
datadir         = /var/lib/mysql
 
tmpdir          = /tmp
 
lc_messages_dir = /usr/share/mysql
 
lc_messages     = en_US
 
skip-external-locking
 
 
 
#abauer
 
ignore_db_dirs=lost+found
 
 
 
#
 
# Instead of skip-networking the default is now to listen only on
 
# localhost which is more compatible and is not less secure.
 
bind-address            = 0.0.0.0
 
#
 
# * Fine Tuning
 
#
 
max_connections         = 50
 
connect_timeout         = 5
 
wait_timeout            = 600
 
max_allowed_packet      = 20M
 
thread_cache_size       = 128
 
sort_buffer_size        = 32M
 
bulk_insert_buffer_size = 16M
 
tmp_table_size          = 32M
 
max_heap_table_size     = 32M
 
#
 
# * MyISAM
 
#
 
# This replaces the startup script and checks MyISAM tables if needed
 
# the first time they are touched. On error, make copy and try a repair.
 
myisam-recover-options  = BACKUP
 
key_buffer_size         = 128M
 
myisam_sort_buffer_size = 512M
 
concurrent_insert       = 2
 
read_buffer_size        = 2M
 
read_rnd_buffer_size    = 1M
 
#
 
# * Query Cache Configuration
 
#
 
# Cache only tiny result sets, so we can fit more in the query cache.
 
query_cache_limit               = 128K
 
query_cache_size                = 64M
 
# for more write intensive setups, set to DEMAND or OFF
 
#query_cache_type               = DEMAND
 
#
 
# * Logging and Replication
 
#
 
# Both location gets rotated by the cronjob.
 
# Be aware that this log type is a performance killer.
 
# As of 5.1 you can enable the log at runtime!
 
#general_log_file        = /var/log/mysql/mysql.log
 
#general_log             = 1
 
#
 
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
 
#
 
# we do want to know about network errors and such
 
log_warnings            = 2
 
#
 
# Enable the slow query log to see queries with especially long duration
 
#slow_query_log[={0|1}]
 
slow_query_log_file     = /var/log/mysql/mariadb-slow.log
 
long_query_time = 10
 
#log_slow_rate_limit    = 1000
 
log_slow_verbosity      = query_plan
 
 
 
#log-queries-not-using-indexes
 
#log_slow_admin_statements
 
#
 
# The following can be used as easy to replay backup logs or for replication.
 
# note: if you are setting up a replication slave, see README.Debian about
 
#       other settings you may need to change.
 
#server-id              = 1
 
#report_host            = master1
 
#auto_increment_increment = 2
 
#auto_increment_offset  = 1
 
log_bin                 = /var/log/mysql/mariadb-bin
 
log_bin_index           = /var/log/mysql/mariadb-bin.index
 
# not fab for performance, but safer
 
#sync_binlog            = 1
 
expire_logs_days        = 10
 
max_binlog_size         = 100M
 
# slaves
 
#relay_log              = /var/log/mysql/relay-bin
 
#relay_log_index        = /var/log/mysql/relay-bin.index
 
#relay_log_info_file    = /var/log/mysql/relay-bin.info
 
#log_slave_updates
 
#read_only
 
#
 
# If applications support it, this stricter sql_mode prevents some
 
# mistakes like inserting invalid dates etc.
 
#sql_mode               = NO_ENGINE_SUBSTITUTION,TRADITIONAL
 
#
 
# * InnoDB
 
#
 
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
 
# Read the manual for more InnoDB related options. There are many!
 
default_storage_engine  = InnoDB
 
# you can't just change log file size, requires special procedure
 
innodb_log_file_size   = 512M
 
innodb_buffer_pool_size = 11GB
 
innodb_buffer_pool_instances = 11
 
innodb_log_buffer_size  = 8M
 
innodb_file_per_table   = 1
 
innodb_open_files       = 400
 
innodb_io_capacity      = 400
 
innodb_flush_method     = O_DIRECT
 
#
 
# * Security Features
 
#
 
# Read the manual, too, if you want chroot!
 
# chroot = /var/lib/mysql/
 
#
 
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
 
#
 
# ssl-ca=/etc/mysql/cacert.pem
 
# ssl-cert=/etc/mysql/server-cert.pem
 
# ssl-key=/etc/mysql/server-key.pem
 
 
 
[mysqldump]
 
quick
 
quote-names
 
max_allowed_packet      = 16M
 
 
 
[mysql]
 
#no-auto-rehash # faster start of mysql but no tab completition
 
 
 
[isamchk]
 
key_buffer              = 16M
 
 
 
#
 
# * IMPORTANT: Additional settings that can override those from this file!
 
#   The files must end with '.cnf', otherwise they'll be ignored.
 
#
 
!includedir /etc/mysql/conf.d/

Comment by André Bauer [ 2016-10-07 ]

2 days ago i moved all Galera nodes to freshly installed Ubuntu Xenial with Mariadb 10.1.18.
Used the same config. Problem still exists. First node has zombie flush logs process now:

"| 9113 | debian-sys-maint | localhost | NULL | Query | 12573 | init | flush logs | 0.000 |"

Is it safe to just kill the process from mariadb console?
Any way to debug this further ?

Comment by Elena Stepanova [ 2016-10-18 ]

nirbhay_c,
Have you seen anything like this before in regard to Galera? I'm not quite sure it's related to the cluster, but I don't remember similar complaints about a standalone server.

Comment by André Bauer [ 2016-10-24 ]

If i try to kill the processes in the mysql shell the linux processes are closed but "show processlist;" still has them with status "killed":

MariaDB [(none)]> show processlist;
 
+-------+------------------+-------------------------+--------+---------+--------+--------------------+------------------+----------+
 
| Id    | User             | Host                    | db     | Command | Time   | State              | Info             | Progress |
 
+-------+------------------+-------------------------+--------+---------+--------+--------------------+------------------+----------+
 
|     1 | system user      |                         | NULL   | Sleep   | 251113 | wsrep aborter idle | NULL             |    0.000 |
 
|     2 | system user      |                         | NULL   | Sleep   |      2 | committed 88464697 | NULL             |    0.000 |
 
|     4 | system user      |                         | NULL   | Sleep   |      2 | committed 88464702 | NULL             |    0.000 |
 
|     5 | system user      |                         | NULL   | Sleep   |      2 | committed 88464699 | NULL             |    0.000 |
 
|     6 | system user      |                         | NULL   | Sleep   |      2 | committed 88464698 | NULL             |    0.000 |
 
|     7 | system user      |                         | NULL   | Sleep   |      2 | committed 88464701 | NULL             |    0.000 |
 
|     8 | system user      |                         | NULL   | Sleep   |      1 | committed 88464706 | NULL             |    0.000 |
 
|     9 | system user      |                         | NULL   | Sleep   |      2 | committed 88464694 | NULL             |    0.000 |
 
|    10 | system user      |                         | NULL   | Sleep   |      2 | committed 88464696 | NULL             |    0.000 |
 
|    11 | system user      |                         | NULL   | Sleep   |      1 | committed 88464704 | NULL             |    0.000 |
 
|    12 | system user      |                         | NULL   | Sleep   |      1 | committed 88464707 | NULL             |    0.000 |
 
|    13 | system user      |                         | NULL   | Sleep   |      2 | committed 88464700 | NULL             |    0.000 |
 
|    14 | system user      |                         | NULL   | Sleep   |      2 | committed 88464692 | NULL             |    0.000 |
 
|    15 | system user      |                         | NULL   | Sleep   |      2 | committed 88464703 | NULL             |    0.000 |
 
|    16 | system user      |                         | NULL   | Sleep   |      2 | committed 88464695 | NULL             |    0.000 |
 
|    17 | system user      |                         | NULL   | Sleep   |      1 | committed 88464705 | NULL             |    0.000 |
 
|    18 | system user      |                         | NULL   | Sleep   |      2 | committed 88464693 | NULL             |    0.000 |
 
|    20 | maxscale         | lb1.domain.local:55245 | NULL   | Sleep   |      0 |                    | NULL             |    0.000 |
 
|    21 | maxscale         | lb2.domain.local:33631 | NULL   | Sleep   |      2 |                    | NULL             |    0.000 |
 
|  3338 | debian-sys-maint | localhost               | NULL   | Killed  | 187539 | init               | flush logs       |    0.000 |
 
|  8207 | debian-sys-maint | localhost               | NULL   | Killed  | 101138 | init               | flush logs       |    0.000 |
 
| 11910 | debian-sys-maint | localhost               | NULL   | Killed  |  14738 | init               | flush logs       |    0.000 |
 
| 12362 | kix              | lb1.domain.local:40513 | kix    | Sleep   |     20 |                    | NULL             |    0.000 |
 
| 12625 | kix              | lb1.domain.local:41122 | kix    | Sleep   |     11 |                    | NULL             |    0.000 |
 
| 12626 | itsm             | lb1.domain.local:41123 | itsm   | Sleep   |     11 |                    | NULL             |    0.000 |
 
| 12683 | kix              | lb1.domain.local:46169 | kix    | Sleep   |     20 |                    | NULL             |    0.000 |
 
| 12731 | zammad           | lb1.domain.local:50735 | zammad | Sleep   |      5 |                    | NULL             |    0.000 |
 
| 12734 | itsm             | lb1.domain.local:50900 | itsm   | Sleep   |      0 |                    | NULL             |    0.000 |
 
| 12735 | zammad           | lb1.domain.local:50945 | zammad | Sleep   |     23 |                    | NULL             |    0.000 |
 
| 12736 | zammad           | lb1.domain.local:51076 | zammad | Sleep   |   2055 |                    | NULL             |    0.000 |
 
| 12748 | kix              | lb1.domain.local:52247 | kix    | Sleep   |     22 |                    | NULL             |    0.000 |
 
| 12761 | kix              | lb1.domain.local:52904 | kix    | Sleep   |     20 |                    | NULL             |    0.000 |
 
| 12762 | otrs_search      | lb1.domain.local:53115 | kix    | Sleep   |    199 |                    | NULL             |    0.000 |
 
| 12809 | otrs_search      | lb1.domain.local:34423 | kix    | Sleep   |     52 |                    | NULL             |    0.000 |
 
| 12833 | root             | localhost               | NULL   | Query   |      0 | init               | show processlist |    0.000 |
 
| 12837 | kix              | lb1.domain.local:39972 | kix    | Sleep   |      0 |                    | NULL             |    0.000 |
 
| 12839 | zammad           | lb1.domain.local:40120 | zammad | Sleep   |     34 |                    | NULL             |    0.000 |
 
| 12843 | itsm             | lb1.domain.local:40410 | itsm   | Sleep   |      0 |                    | NULL             |    0.000 |
 
+-------+------------------+-------------------------+--------+---------+--------+--------------------+------------------+----------+
 
 

Comment by Florian Engelmann [ 2016-12-16 ]

Hello,

we have got exactly the same problem. My assumption was slow-logs could be involved but even after setting slow_query_log to 0 flush-logs got stuck. We are running MariaDB 10.1.19 in a 5 node Galera cluster. Randomly one or more of those nodes crash at logrotation with:

Dec 15 02:22:53 xxxxxx-xxxxx-prod mysqld: 161215 2:22:53 [ERROR] mysqld got signal 11 ;

It doesn't happen in the test environment (settings are the same). And it doesn't happen if flush-logs is triggered frequently enough. It only happens if the DB is running on load for a certain amount of queries (or time?).

This is a show stopper to us!!!! Is anyone working on this issue right now?

All the best,
Florian

Comment by Nirbhay Choubey (Inactive) [ 2016-12-16 ]

engel75: This problem is hard to repeat. Could you enable --wsrep-debug=ON and share the full error log of the crashing node including the stack trace of signal 11?

Comment by Florian Engelmann [ 2016-12-16 ]

Something with the binary logrotation seems to mess up things... I tried to limit the binlog size to 100K by setting:

[mysqld]
max_binlog_size = 100K

but this caused mariadb to crash without any flush-log after a couple of minutes:

Dec 16 16:33:01 galera01-some-prod mysqld: 161216 16:33:01 [ERROR] mysqld got signal 11 ;
 
Dec 16 16:33:01 galera01-some-prod mysqld: This could be because you hit a bug. It is also possible that this binary
 
Dec 16 16:33:01 galera01-some-prod mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 16 16:33:01 galera01-some-prod mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: We will try our best to scrape up some info that will hopefully help
 
Dec 16 16:33:01 galera01-some-prod mysqld: diagnose the problem, but since we have already crashed, 
Dec 16 16:33:01 galera01-some-prod mysqld: something is definitely wrong and this may fail.
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: Server version: 10.1.19-MariaDB-1~trusty
 
Dec 16 16:33:01 galera01-some-prod mysqld: key_buffer_size=134217728
 
Dec 16 16:33:01 galera01-some-prod mysqld: read_buffer_size=2097152
 
Dec 16 16:33:01 galera01-some-prod mysqld: max_used_connections=21
 
Dec 16 16:33:01 galera01-some-prod mysqld: max_threads=262
 
Dec 16 16:33:01 galera01-some-prod mysqld: thread_count=24
 
Dec 16 16:33:01 galera01-some-prod mysqld: It is possible that mysqld could use up to 
Dec 16 16:33:01 galera01-some-prod mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2819312 K  bytes of memory
 
Dec 16 16:33:01 galera01-some-prod mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: Thread pointer: 0x0x7fdea2012008
 
Dec 16 16:33:01 galera01-some-prod mysqld: Attempting backtrace. You can use the following information to find out
 
Dec 16 16:33:01 galera01-some-prod mysqld: where mysqld died. If you see no messages after this, something went
 
Dec 16 16:33:01 galera01-some-prod mysqld: terribly wrong...
 
Dec 16 16:33:01 galera01-some-prod mysqld: stack_bottom = 0x7fe2362f1210 thread_stack 0x48400
 
Dec 16 16:33:01 galera01-some-prod mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fe235ce867e]
 
Dec 16 16:33:01 galera01-some-prod mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x7fe23580bba5]
 
Dec 16 16:33:01 galera01-some-prod mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fe233b57330]
 
Dec 16 16:33:01 galera01-some-prod mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0xe8)[0x7fe2358cde88]
 
Dec 16 16:33:01 galera01-some-prod mysqld: /usr/sbin/mysqld(binlog_background_thread+0x2c5)[0x7fe2358ce525]
 
Dec 16 16:33:01 galera01-some-prod mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fe233b4f184]
 
Dec 16 16:33:01 galera01-some-prod mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe23306e37d]
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: Trying to get some variables.
 
Dec 16 16:33:01 galera01-some-prod mysqld: Some pointers may be invalid and cause the dump to abort.
 
Dec 16 16:33:01 galera01-some-prod mysqld: Query (0x0): is an invalid pointer
 
Dec 16 16:33:01 galera01-some-prod mysqld: Connection ID (thread ID): 3
 
Dec 16 16:33:01 galera01-some-prod mysqld: Status: NOT_KILLED
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Dec 16 16:33:01 galera01-some-prod mysqld: information that should help you find out what is causing the crash.
 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld: We think the query pointer is invalid, but we will try to print it anyway. 
Dec 16 16:33:01 galera01-some-prod mysqld: Query: 
Dec 16 16:33:01 galera01-some-prod mysqld: 
Dec 16 16:33:01 galera01-some-prod mysqld_safe: Number of processes running now: 0
 
Dec 16 16:33:01 galera01-some-prod mysqld_safe: WSREP: not restarting wsrep node automatically
 
Dec 16 16:33:01 galera01-some-prod mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

Comment by Florian Engelmann [ 2016-12-16 ]

I restarted node1 and enabled wsrep_debug. So let's see what happenn on the next crash.

Comment by Florian Engelmann [ 2016-12-17 ]

as expected the node crashed like follows:

Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (2466)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_modified AS last_modified_3, t0.segmentation AS segmentation_4, t0.toplogo_file AS toplogo_file_5, t0.industry AS industry_6, t0.url AS url_7, t0.user_defined_child_ids AS user_defined_child_ids_8, t0.visible AS visible_9, t0.deletedAt AS deletedAt_10, t0.parent_id AS parent_id_11, t12.xing_profile_url AS xing_profile_url_13, t12.xing_poster_url AS xing_poster_url_14, t12.employees AS employees_15, t12.turnover AS turnover_16, t12.kununu_id AS kununu_id_17, t12.kununu_url AS kununu_url_18, t12.kununu_rating AS kununu_rating_19, t12.kununu_top AS kununu_top_20, t12.kununu_open AS kununu_open_21, t12.stock_symbol AS stock_symbol_22, t12.stock_url AS stock_url_23, t12.home_office_friendly AS home_office_friendly_24, t12.gptw AS gptw_25, t12.fws AS fws_26, t12.tfc AS tfc_27, t12.id AS id_28, t12.company_id AS company_id_29 FROM company t0 LEFT JOIN company_portrait t12 ON t12.co
 
Dec 16 21:16:33 galera01-some2-prod mysqld: mpany_id = t0.id WHERE t0.id = 7228 AND ((t
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (7228)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (41317)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (2466)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (7228)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (41317)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 41317
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 41317
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 41317 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.type AS type_2 FROM product t0 WHERE t0.id = 4
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (2466)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 2466
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 2466
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 2466 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (7228)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 7228
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 7228
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892068727552 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 7228 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_modified AS last_modified_3, t0.segmentation AS segmentation_4, t0.toplogo_file AS toplogo_file_5, t0.industry AS industry_6, t0.url AS url_7, t0.user_defined_child_ids AS user_defined_child_ids_8, t0.visible AS visible_9, t0.deletedAt AS deletedAt_10, t0.parent_id AS parent_id_11, t12.xing_profile_url AS xing_profile_url_13, t12.xing_poster_url AS xing_poster_url_14, t12.employees AS employees_15, t12.turnover AS turnover_16, t12.kununu_id AS kununu_id_17, t12.kununu_url AS kununu_url_18, t12.kununu_rating AS kununu_rating_19, t12.kununu_top AS kununu_top_20, t12.kununu_open AS kununu_open_21, t12.stock_symbol AS stock_symbol_22, t12.stock_url AS stock_url_23, t12.home_office_friendly AS home_office_friendly_24, t12.gptw AS gptw_25, t12.fws AS fws_26, t12.tfc AS tfc_27, t12.id AS id_28, t12.company_id AS company_id_29 FROM company t0 LEFT JOIN company_portrait t12 ON t12.co
 
Dec 16 21:16:33 galera01-some2-prod mysqld: mpany_id = t0.id WHERE t0.id = '769' AND t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (769)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (769)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.description AS description_1, t0.id AS id_2, t0.locale AS locale_3, t0.company_id AS company_id_4 FROM company_portrait_description t0 WHERE t0.company_id = 769
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (769)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 769 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.url AS url_1, t0.position AS position_2, t0.id AS id_3, t0.company_id AS company_id_4 FROM company_portrait_url t0 WHERE t0.company_id = 769 ORDER BY t0.position ASC
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 769
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892132285184 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 769
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892131982080 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892131982080 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.expires_at AS expires_at_1, t0.scope AS scope_2, t0.id AS id_3, t0.token AS token_4, t0.client_id AS client_id_5, t0.user_id AS user_id_6 FROM oauth_access_token t0 WHERE t0.token = 'YjZjNGY2NTRmNjgwZDdjNmMyMTg5NGVhODUxZWZhMDkxMWY5NDcxZjlhYjk3YTVhMTA5YTYwMDNiNDMyMmM4Mg' LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.email AS email_2, t0.password AS password_3, t0.language AS language_4, t0.email_address_verified AS email_address_verified_5, t0.emails_deliverable AS emails_deliverable_6, t0.created_at AS created_at_7, t0.last_login AS last_login_8, t0.push_at_hour AS push_at_hour_9, t0.legacy_id AS legacy_id_10, t0.legacy_password AS legacy_password_11, t0.legacy_salt AS legacy_salt_12, t0.search_profiles_migrated AS search_profiles_migrated_13, t0.source_id AS source_id_14, t0.discriminator FROM user t0 WHERE t0.id = '2edb1bf5-a8fb-4406-a878-7508aae39b9c' AND t0.discriminator IN ('someuser')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_modified AS last_modified_3, t0.segmentation AS segmentation_4, t0.toplogo_file AS toplogo_file_5, t0.industry AS industry_6, t0.url AS url_7, t0.user_defined_child_ids AS user_defined_child_ids_8, t0.visible AS visible_9, t0.deletedAt AS deletedAt_10, t0.parent_id AS parent_id_11, t12.xing_profile_url AS xing_profile_url_13, t12.xing_poster_url AS xing_poster_url_14, t12.employees AS employees_15, t12.turnover AS turnover_16, t12.kununu_id AS kununu_id_17, t12.kununu_url AS kununu_url_18, t12.kununu_rating AS kununu_rating_19, t12.kununu_top AS kununu_top_20, t12.kununu_open AS kununu_open_21, t12.stock_symbol AS stock_symbol_22, t12.stock_url AS stock_url_23, t12.home_office_friendly AS home_office_friendly_24, t12.gptw AS gptw_25, t12.fws AS fws_26, t12.tfc AS tfc_27, t12.id AS id_28, t12.company_id AS company_id_29 FROM company t0 LEFT JOIN company_portrait t12 ON t12.co
 
Dec 16 21:16:33 galera01-some2-prod mysqld: mpany_id = t0.id WHERE t0.id = '20684' AND 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.user_id AS user_id_6, t0.type FROM bookmark t0 WHERE t0.user_id = '2edb1bf5-a8fb-4406-a878-7508aae39b9c' AND t0.bookmarked_entity_id = '20684' AND t0.type IN ('companybookmark') LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (20684)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (20684)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.description AS description_1, t0.id AS id_2, t0.locale AS locale_3, t0.company_id AS company_id_4 FROM company_portrait_description t0 WHERE t0.company_id = 20684
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (20684)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 20684 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.url AS url_1, t0.position AS position_2, t0.id AS id_3, t0.company_id AS company_id_4 FROM company_portrait_url t0 WHERE t0.company_id = 20684 ORDER BY t0.position ASC
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 20684
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069030656 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 20684
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069333760 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521937152 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0 WHERE t0.id = 31529
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069333760 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_modified AS last_modified_3, t0.segmentation AS segmentation_4, t0.toplogo_file AS toplogo_file_5, t0.industry AS industry_6, t0.url AS url_7, t0.user_defined_child_ids AS user_defined_child_ids_8, t0.visible AS visible_9, t0.deletedAt AS deletedAt_10, t0.parent_id AS parent_id_11, t12.xing_profile_url AS xing_profile_url_13, t12.xing_poster_url AS xing_poster_url_14, t12.employees AS employees_15, t12.turnover AS turnover_16, t12.kununu_id AS kununu_id_17, t12.kununu_url AS kununu_url_18, t12.kununu_rating AS kununu_rating_19, t12.kununu_top AS kununu_top_20, t12.kununu_open AS kununu_open_21, t12.stock_symbol AS stock_symbol_22, t12.stock_url AS stock_url_23, t12.home_office_friendly AS home_office_friendly_24, t12.gptw AS gptw_25, t12.fws AS fws_26, t12.tfc AS tfc_27, t12.id AS id_28, t12.company_id AS company_id_29 FROM company t0 LEFT JOIN company_portrait t12 ON t12.co
 
Dec 16 21:16:33 galera01-some2-prod mysqld: mpany_id = t0.id WHERE t0.id = '5465' AND t
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (5465)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521937152 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT p0_.id AS id_0 FROM product_relation p0_ INNER JOIN product p1_ ON p0_.product_id = p1_.id AND (p1_.type = 'product_company_featured') INNER JOIN company c2_ ON p0_.parent_id = c2_.id AND (c2_.deletedAt IS NULL) AND (c2_.visible = 1) WHERE ((((p0_.start_date IS NULL OR p0_.start_date <= '2016-12-16 00:00:00') AND (p0_.end_date IS NULL OR p0_.end_date >= '2016-12-16 00:00:00')) AND p0_.active = 1 AND p0_.locale = 'fr' AND p0_.platform_id IN (31529)) AND (p0_.deletedAt IS NULL)) AND p0_.discriminator IN ('companyproduct')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521937152 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT p0_.id AS id_0, p0_.locale AS locale_1, p0_.start_date AS start_date_2, p0_.end_date AS end_date_3, p0_.legacy_id AS legacy_id_4, p0_.active AS active_5, p0_.deletedAt AS deletedAt_6, p0_.discriminator AS discriminator_7, p0_.product_id AS product_id_8, p0_.category_id AS category_id_9, p0_.region_id AS region_id_10, p0_.platform_id AS platform_id_11, p0_.parent_id AS parent_id_12 FROM product_relation p0_ WHERE ((p0_.id IN (NULL)) AND (p0_.deletedAt IS NULL)) AND p0_.discriminator IN ('abstractproductrelation', 'companyproduct')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (5465)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.description AS description_1, t0.id AS id_2, t0.locale AS locale_3, t0.company_id AS company_id_4 FROM company_portrait_description t0 WHERE t0.company_id = 5465
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (5465)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 5465 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.url AS url_1, t0.position AS position_2, t0.id AS id_3, t0.company_id AS company_id_4 FROM company_portrait_url t0 WHERE t0.company_id = 5465 ORDER BY t0.position ASC
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 5465
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521027840 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 5465
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521634048 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522543360 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0 WHERE t0.id = 31529
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897521634048 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0 WHERE t0.id = 3
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_modified AS last_modified_3, t0.segmentation AS segmentation_4, t0.toplogo_file AS toplogo_file_5, t0.industry AS industry_6, t0.url AS url_7, t0.user_defined_child_ids AS user_defined_child_ids_8, t0.visible AS visible_9, t0.deletedAt AS deletedAt_10, t0.parent_id AS parent_id_11, t12.xing_profile_url AS xing_profile_url_13, t12.xing_poster_url AS xing_poster_url_14, t12.employees AS employees_15, t12.turnover AS turnover_16, t12.kununu_id AS kununu_id_17, t12.kununu_url AS kununu_url_18, t12.kununu_rating AS kununu_rating_19, t12.kununu_top AS kununu_top_20, t12.kununu_open AS kununu_open_21, t12.stock_symbol AS stock_symbol_22, t12.stock_url AS stock_url_23, t12.home_office_friendly AS home_office_friendly_24, t12.gptw AS gptw_25, t12.fws AS fws_26, t12.tfc AS tfc_27, t12.id AS id_28, t12.company_id AS company_id_29 FROM company t0 LEFT JOIN company_portrait t12 ON t12.co
 
Dec 16 21:16:33 galera01-some2-prod mysqld: mpany_id = t0.id WHERE t0.id = 71656 AND t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (71656)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (71656)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.description AS description_1, t0.id AS id_2, t0.locale AS locale_3, t0.company_id AS company_id_4 FROM company_portrait_description t0 WHERE t0.company_id = 71656
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 71656 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 71656
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.url AS url_1, t0.position AS position_2, t0.id AS id_3, t0.company_id AS company_id_4 FROM company_portrait_url t0 WHERE t0.company_id = 71656 ORDER BY t0.position ASC
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522543360 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT p0_.id AS id_0 FROM product_relation p0_ INNER JOIN product p1_ ON p0_.product_id = p1_.id AND (p1_.type = 'product_company_featured') INNER JOIN company c2_ ON p0_.parent_id = c2_.id AND (c2_.deletedAt IS NULL) AND (c2_.visible = 1) WHERE ((((p0_.start_date IS NULL OR p0_.start_date <= '2016-12-16 00:00:00') AND (p0_.end_date IS NULL OR p0_.end_date >= '2016-12-16 00:00:00')) AND p0_.active = 1 AND p0_.locale = 'fr' AND p0_.platform_id IN (31529)) AND (p0_.deletedAt IS NULL)) AND p0_.discriminator IN ('companyproduct')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (71656)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522543360 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT p0_.id AS id_0, p0_.locale AS locale_1, p0_.start_date AS start_date_2, p0_.end_date AS end_date_3, p0_.legacy_id AS legacy_id_4, p0_.active AS active_5, p0_.deletedAt AS deletedAt_6, p0_.discriminator AS discriminator_7, p0_.product_id AS product_id_8, p0_.category_id AS category_id_9, p0_.region_id AS region_id_10, p0_.platform_id AS platform_id_11, p0_.parent_id AS parent_id_12 FROM product_relation p0_ WHERE ((p0_.id IN (NULL)) AND (p0_.deletedAt IS NULL)) AND p0_.discriminator IN ('abstractproductrelation', 'companyproduct')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0 WHERE t0.id = 62913
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897522240256 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.random_id AS random_id_1, t0.redirect_uris AS redirect_uris_2, t0.secret AS secret_3, t0.allowed_grant_types AS allowed_grant_types_4, t0.id AS id_5, t0.operating_system_provider AS operating_system_provider_6, t0.application_name AS application_name_7, t0.logo_url AS logo_url_8, t0.allowed_scopes AS allowed_scopes_9, t0.source_id AS source_id_10 FROM oauth_client t0 WHERE t0.id = 4
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.random_id AS random_id_1, t0.redirect_uris AS redirect_uris_2, t0.secret AS secret_3, t0.allowed_grant_types AS allowed_grant_types_4, t0.id AS id_5, t0.operating_system_provider AS operating_system_provider_6, t0.application_name AS application_name_7, t0.logo_url AS logo_url_8, t0.allowed_scopes AS allowed_scopes_9, t0.source_id AS source_id_10 FROM oauth_client t0 WHERE t0.id = '4' AND t0.random_id = '3dle02gasr8ko80gows4480ok8gs8c88o4s0w0gkg4go4c0ww8' LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2 FROM source t0 WHERE t0.id = 4
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.random_id AS random_id_1, t0.redirect_uris AS redirect_uris_2, t0.secret AS secret_3, t0.allowed_grant_types AS allowed_grant_types_4, t0.id AS id_5, t0.operating_system_provider AS operating_system_provider_6, t0.application_name AS application_name_7, t0.logo_url AS logo_url_8, t0.allowed_scopes AS allowed_scopes_9, t0.source_id AS source_id_10 FROM oauth_client t0 WHERE t0.id = '4' AND t0.random_id = '3dle02gasr8ko80gows4480ok8gs8c88o4s0w0gkg4go4c0ww8' LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.expires_at AS expires_at_1, t0.scope AS scope_2, t0.id AS id_3, t0.token AS token_4, t0.client_id AS client_id_5, t0.user_id AS user_id_6 FROM oauth_refresh_token t0 WHERE t0.token = 'M2ZlNjU2NzJjNWJhZWRhMjViODU5ZTg1ODM4YjMzN2FjODNmZGFhNDZiZDc0ZDA1OWRlOTJmNTNmY2Q0N2FmMA' LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892131072768 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.expires_at AS expires_at_1, t0.scope AS scope_2, t0.id AS id_3, t0.token AS token_4, t0.client_id AS client_id_5, t0.user_id AS user_id_6 FROM oauth_refresh_token t0 WHERE t0.token = 'M2ZlNjU2NzJjNWJhZWRhMjViODU5ZTg1ODM4YjMzN2FjODNmZGFhNDZiZDc0ZDA1OWRlOTJmNTNmY2Q0N2FmMA' LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.expires_at AS expires_at_1, t0.scope AS scope_2, t0.id AS id_3, t0.token AS token_4, t0.client_id AS client_id_5, t0.user_id AS user_id_6 FROM oauth_access_token t0 WHERE t0.token = 'ZDdjNjY3ODJkZjc5ZDNlZGU1ZDc4YzgyNmQ3YjEzNGFhYmI1MjNiZGUyYzU2ODUwZjc2MzM4ZjBmOTM5YzE3ZQ' LIMIT 1
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.email AS email_2, t0.password AS password_3, t0.language AS language_4, t0.email_address_verified AS email_address_verified_5, t0.emails_deliverable AS emails_deliverable_6, t0.created_at AS created_at_7, t0.last_login AS last_login_8, t0.push_at_hour AS push_at_hour_9, t0.legacy_id AS legacy_id_10, t0.legacy_password AS legacy_password_11, t0.legacy_salt AS legacy_salt_12, t0.search_profiles_migrated AS search_profiles_migrated_13, t0.source_id AS source_id_14, t0.discriminator FROM user t0 WHERE t0.id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.discriminator IN ('someuser')
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: MYSQL_BIN_LOG::open(): Adding new xid_list_entry for mariadb-bin.000023 (20)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139892069636864 [Note] WSREP: TC_LOG_BINLOG::mark_xid_done(): Removing xid_list_entry for mariadb-bin.000022 (19)
 
Dec 16 21:16:33 galera01-some2-prod mysqld: 2016-12-16 21:16:33 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_modified AS last_modified_3, t0.segmentation AS segmentation_4, t0.toplogo_file AS toplogo_file_5, t0.industry AS industry_6, t0.url AS url_7, t0.user_defined_child_ids AS user_defined_child_ids_8, t0.visible AS visible_9, t0.deletedAt AS deletedAt_10, t0.parent_id AS parent_id_11, t12.xing_profile_url AS xing_profile_url_13, t12.xing_poster_url AS xing_poster_url_14, t12.employees AS employees_15, t12.turnover AS turnover_16, t12.kununu_id AS kununu_id_17, t12.kununu_url AS kununu_url_18, t12.kununu_rating AS kununu_rating_19, t12.kununu_top AS kununu_top_20, t12.kununu_open AS kununu_open_21, t12.stock_symbol AS stock_symbol_22, t12.stock_url AS stock_url_23, t12.home_office_friendly AS home_office_friendly_24, t12.gptw AS gptw_25, t12.fws AS fws_26, t12.tfc AS tfc_27, t12.id AS id_28, t12.company_id AS company_id_29 FROM company t0 LEFT JOIN company_portrait t12 ON t12.co
 
Dec 16 21:16:34 galera01-some2-prod mysqld: mpany_id = t0.id WHERE t0.id = '912' AND t0
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (912)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (912)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.description AS description_1, t0.id AS id_2, t0.locale AS locale_3, t0.company_id AS company_id_4 FROM company_portrait_description t0 WHERE t0.company_id = 912
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT c0_.id AS id_0 FROM company c0_ WHERE (c0_.parent_id IN (912)) AND (c0_.deletedAt IS NULL)
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.street1 AS street1_1, t0.street2 AS street2_2, t0.department AS department_3, t0.city AS city_4, t0.zip_code AS zip_code_5, t0.country_code AS country_code_6, t0.telephone1 AS telephone1_7, t0.telephone2 AS telephone2_8, t0.fax AS fax_9, t0.firstname AS firstname_10, t0.lastname AS lastname_11, t0.email AS email_12, t0.latitude AS latitude_13, t0.longitude AS longitude_14, t0.id AS id_15, t0.company_id AS company_id_16, t0.type FROM company_address t0 WHERE t0.company_id = 912 AND t0.type IN ('address', 'contact', 'billing')
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.url AS url_1, t0.position AS position_2, t0.id AS id_3, t0.company_id AS company_id_4 FROM company_portrait_url t0 WHERE t0.company_id = 912 ORDER BY t0.position ASC
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.context AS context_1, t0.file_name AS file_name_2, t0.width AS width_3, t0.height AS height_4, t0.type AS type_5, t0.updated AS updated_6, t0.id AS id_7, t0.company_id AS company_id_8 FROM company_image t0 WHERE t0.company_id = 912
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897520724736 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.vimeo_id AS vimeo_id_1, t0.vimeo_thumbnail_url AS vimeo_thumbnail_url_2, t0.description AS description_3, t0.standard AS standard_4, t0.id AS id_5, t0.company_id AS company_id_6 FROM company_video t0 WHERE t0.company_id = 912
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.expires_at AS expires_at_1, t0.scope AS scope_2, t0.id AS id_3, t0.token AS token_4, t0.client_id AS client_id_5, t0.user_id AS user_id_6 FROM oauth_access_token t0 WHERE t0.token = 'ZDdjNjY3ODJkZjc5ZDNlZGU1ZDc4YzgyNmQ3YjEzNGFhYmI1MjNiZGUyYzU2ODUwZjc2MzM4ZjBmOTM5YzE3ZQ' LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.email AS email_2, t0.password AS password_3, t0.language AS language_4, t0.email_address_verified AS email_address_verified_5, t0.emails_deliverable AS emails_deliverable_6, t0.created_at AS created_at_7, t0.last_login AS last_login_8, t0.push_at_hour AS push_at_hour_9, t0.legacy_id AS legacy_id_10, t0.legacy_password AS legacy_password_11, t0.legacy_salt AS legacy_salt_12, t0.search_profiles_migrated AS search_profiles_migrated_13, t0.source_id AS source_id_14, t0.discriminator FROM user t0 WHERE t0.id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.discriminator IN ('someuser')
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT s0_.id AS id_0, s0_.name AS name_1, s0_.last_usage AS last_usage_2, s0_.last_sent_jobmail AS last_sent_jobmail_3, s0_.last_sent_push_notification AS last_sent_push_notification_4, s0_.jobmail_interval AS jobmail_interval_5, s0_.created_at AS created_at_6, s0_.updated_at AS updated_at_7, j1_.hash AS hash_8, j1_.query AS query_9, j1_.location AS location_10, j1_.employment_grade_min AS employment_grade_min_11, j1_.employment_grade_max AS employment_grade_max_12, j1_.top_left_latitude AS top_left_latitude_13, j1_.top_left_longitude AS top_left_longitude_14, j1_.bottom_right_latitude AS bottom_right_latitude_15, j1_.bottom_right_longitude AS bottom_right_longitude_16, u2_.id AS id_17, u2_.email AS email_18, u2_.password AS password_19, u2_.language AS language_20, u2_.email_address_verified AS email_address_verified_21, u2_.emails_deliverable AS emails_deliverable_22, u2_.created_at AS created_at_23, u2_.last_login AS l
 
Dec 16 21:16:34 galera01-some2-prod mysqld: ast_login_24, u2_.push_at_hour AS push_at_h
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.category_id AS category_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_category t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.industry_id AS industry_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_industry t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.company_segment AS company_segment_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_company_segment t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.employment_position_id AS employment_position_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_employment_position t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.employment_type_id AS employment_type_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_employment_type t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.region_id AS region_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_region t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.company_id AS company_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_company t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.platform_id AS platform_id_1, t0.job_search_hash AS job_search_hash_2 FROM job_search_platform t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.latitude AS latitude_2, t0.longitude AS longitude_3, t0.job_search_hash AS job_search_hash_4 FROM job_search_geo_shape_point t0 WHERE t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c3'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897521937152 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.host AS host_1, t0.schemes AS schemes_2, t0.methods AS methods_3, t0.defaults AS defaults_4, t0.requirements AS requirements_5, t0.options AS options_6, t0.condition_expr AS condition_expr_7, t0.variablePattern AS variablePattern_8, t0.staticPrefix AS staticPrefix_9, t0.name AS name_10, t0.position AS position_11, t0.id AS id_12 FROM orm_routes t0
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_usage AS last_usage_3, t0.last_sent_jobmail AS last_sent_jobmail_4, t0.last_sent_push_notification AS last_sent_push_notification_5, t0.jobmail_interval AS jobmail_interval_6, t0.created_at AS created_at_7, t0.updated_at AS updated_at_8, t0.job_search_hash AS job_search_hash_9, t10.hash AS hash_11, t10.query AS query_12, t10.location AS location_13, t10.employment_grade_min AS employment_grade_min_14, t10.employment_grade_max AS employment_grade_max_15, t10.top_left_latitude AS top_left_latitude_16, t10.top_left_longitude AS top_left_longitude_17, t10.bottom_right_latitude AS bottom_right_latitude_18, t10.bottom_right_longitude AS bottom_right_longitude_19, t0.user_id AS user_id_20 FROM search_profile t0 LEFT JOIN job_search t10 ON t0.job_search_hash = t10.hash WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.job_search_hash = '7e307e2652941c59226686b2fca5d5c
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 3' LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_usage AS last_usage_3, t0.last_sent_jobmail AS last_sent_jobmail_4, t0.last_sent_push_notification AS last_sent_push_notification_5, t0.jobmail_interval AS jobmail_interval_6, t0.created_at AS created_at_7, t0.updated_at AS updated_at_8, t0.job_search_hash AS job_search_hash_9, t10.hash AS hash_11, t10.query AS query_12, t10.location AS location_13, t10.employment_grade_min AS employment_grade_min_14, t10.employment_grade_max AS employment_grade_max_15, t10.top_left_latitude AS top_left_latitude_16, t10.top_left_longitude AS top_left_longitude_17, t10.bottom_right_latitude AS bottom_right_latitude_18, t10.bottom_right_longitude AS bottom_right_longitude_19, t0.user_id AS user_id_20 FROM search_profile t0 LEFT JOIN job_search t10 ON t0.job_search_hash = t10.hash WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.name = 'Jobs in mehreren Fachbereichen'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897521937152 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT p0_.id AS id_0, p0_.locale AS locale_1, p0_.start_date AS start_date_2, p0_.end_date AS end_date_3, p0_.legacy_id AS legacy_id_4, p0_.active AS active_5, p0_.deletedAt AS deletedAt_6, p0_.discriminator AS discriminator_7, p0_.product_id AS product_id_8, p0_.category_id AS category_id_9, p0_.region_id AS region_id_10, p0_.platform_id AS platform_id_11, p0_.parent_id AS parent_id_12 FROM product_relation p0_ WHERE ((p0_.id IN (41517, 41722, 40772)) AND (p0_.deletedAt IS NULL)) AND p0_.discriminator IN ('abstractproductrelation', 'companyproduct')
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139897521937152 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.type AS type_2 FROM product t0 WHERE t0.id = 5
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.state AS state_6, t0.user_id AS user_id_7, t0.type FROM bookmark t0 WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.type IN ('jobbookmark') ORDER BY t0.created_at DESC
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 161216 21:16:34 [ERROR] mysqld got signal 11 ;
 
Dec 16 21:16:34 galera01-some2-prod mysqld: This could be because you hit a bug. It is also possible that this binary
 
Dec 16 21:16:34 galera01-some2-prod mysqld: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec 16 21:16:34 galera01-some2-prod mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: We will try our best to scrape up some info that will hopefully help
 
Dec 16 21:16:34 galera01-some2-prod mysqld: diagnose the problem, but since we have already crashed, 
Dec 16 21:16:34 galera01-some2-prod mysqld: something is definitely wrong and this may fail.
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: Server version: 10.1.19-MariaDB-1~trusty
 
Dec 16 21:16:34 galera01-some2-prod mysqld: key_buffer_size=134217728
 
Dec 16 21:16:34 galera01-some2-prod mysqld: read_buffer_size=2097152
 
Dec 16 21:16:34 galera01-some2-prod mysqld: max_used_connections=17
 
Dec 16 21:16:34 galera01-some2-prod mysqld: max_threads=262
 
Dec 16 21:16:34 galera01-some2-prod mysqld: thread_count=19
 
Dec 16 21:16:34 galera01-some2-prod mysqld: It is possible that mysqld could use up to 
Dec 16 21:16:34 galera01-some2-prod mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2819312 K  bytes of memory
 
Dec 16 21:16:34 galera01-some2-prod mysqld: Hope that's ok; if not, decrease some variables in the equation.
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: Thread pointer: 0x0x7f38df412008
 
Dec 16 21:16:34 galera01-some2-prod mysqld: Attempting backtrace. You can use the following information to find out
 
Dec 16 21:16:34 galera01-some2-prod mysqld: where mysqld died. If you see no messages after this, something went
 
Dec 16 21:16:34 galera01-some2-prod mysqld: terribly wrong...
 
Dec 16 21:16:34 galera01-some2-prod mysqld: stack_bottom = 0x7f3c730e8210 thread_stack 0x48400
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f3c72adf67e]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.state AS state_6, t0.user_id AS user_id_7, t0.type FROM bookmark t0 WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.bookmarked_entity_id = '6667415' AND t0.type IN ('jobbookmark') LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.state AS state_6, t0.user_id AS user_id_7, t0.type FROM bookmark t0 WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.bookmarked_entity_id = '6662129' AND t0.type IN ('jobbookmark') LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.slug AS slug_2, t0.title AS title_3 FROM platform t0 WHERE t0.id = 3
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT j0_.id AS id_0, j0_.uuid AS uuid_1, j0_.locale AS locale_2, j0_.job_id AS job_id_3, j0_.gender AS gender_4, j0_.firstname AS firstname_5, j0_.lastname AS lastname_6, j0_.phone AS phone_7, j0_.email AS email_8, j0_.content AS content_9, j0_.application_management AS application_management_10, j0_.member_id AS member_id_11, j0_.member_sent_cv AS member_sent_cv_12, j0_.user_id AS user_id_13, j0_.device AS device_14, j0_.last_occupation AS last_occupation_15, j0_.state AS state_16, j0_.created_at AS created_at_17, j0_.bucket_id AS bucket_id_18, j0_.platform_id AS platform_id_19 FROM job_application j0_ WHERE j0_.platform_id = 3 AND j0_.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND j0_.state = 'sent_to_employer' AND j0_.job_id IN ('6667415', '6662129') ORDER BY j0_.created_at DESC
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_usage AS last_usage_3, t0.last_sent_jobmail AS last_sent_jobmail_4, t0.last_sent_push_notification AS last_sent_push_notification_5, t0.jobmail_interval AS jobmail_interval_6, t0.created_at AS created_at_7, t0.updated_at AS updated_at_8, t0.job_search_hash AS job_search_hash_9, t10.hash AS hash_11, t10.query AS query_12, t10.location AS location_13, t10.employment_grade_min AS employment_grade_min_14, t10.employment_grade_max AS employment_grade_max_15, t10.top_left_latitude AS top_left_latitude_16, t10.top_left_longitude AS top_left_longitude_17, t10.bottom_right_latitude AS bottom_right_latitude_18, t10.bottom_right_longitude AS bottom_right_longitude_19, t0.user_id AS user_id_20 FROM search_profile t0 LEFT JOIN job_search t10 ON t0.job_search_hash = t10.hash WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.job_search_hash = '480f77a51c5fd4b91e6e2286bdb6ca9
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 7' LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_usage AS last_usage_3, t0.last_sent_jobmail AS last_sent_jobmail_4, t0.last_sent_push_notification AS last_sent_push_notification_5, t0.jobmail_interval AS jobmail_interval_6, t0.created_at AS created_at_7, t0.updated_at AS updated_at_8, t0.job_search_hash AS job_search_hash_9, t10.hash AS hash_11, t10.query AS query_12, t10.location AS location_13, t10.employment_grade_min AS employment_grade_min_14, t10.employment_grade_max AS employment_grade_max_15, t10.top_left_latitude AS top_left_latitude_16, t10.top_left_longitude AS top_left_longitude_17, t10.bottom_right_latitude AS bottom_right_latitude_18, t10.bottom_right_longitude AS bottom_right_longitude_19, t0.user_id AS user_id_20 FROM search_profile t0 LEFT JOIN job_search t10 ON t0.job_search_hash = t10.hash WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.name = 'Alle Jobs von some.ch'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x7f3c72602ba5]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_usage AS last_usage_3, t0.last_sent_jobmail AS last_sent_jobmail_4, t0.last_sent_push_notification AS last_sent_push_notification_5, t0.jobmail_interval AS jobmail_interval_6, t0.created_at AS created_at_7, t0.updated_at AS updated_at_8, t0.job_search_hash AS job_search_hash_9, t10.hash AS hash_11, t10.query AS query_12, t10.location AS location_13, t10.employment_grade_min AS employment_grade_min_14, t10.employment_grade_max AS employment_grade_max_15, t10.top_left_latitude AS top_left_latitude_16, t10.top_left_longitude AS top_left_longitude_17, t10.bottom_right_latitude AS bottom_right_latitude_18, t10.bottom_right_longitude AS bottom_right_longitude_19, t0.user_id AS user_id_20 FROM search_profile t0 LEFT JOIN job_search t10 ON t0.job_search_hash = t10.hash WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.job_search_hash = '480f77a51c5fd4b91e6e2286bdb6ca9
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 7' LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.name AS name_2, t0.last_usage AS last_usage_3, t0.last_sent_jobmail AS last_sent_jobmail_4, t0.last_sent_push_notification AS last_sent_push_notification_5, t0.jobmail_interval AS jobmail_interval_6, t0.created_at AS created_at_7, t0.updated_at AS updated_at_8, t0.job_search_hash AS job_search_hash_9, t10.hash AS hash_11, t10.query AS query_12, t10.location AS location_13, t10.employment_grade_min AS employment_grade_min_14, t10.employment_grade_max AS employment_grade_max_15, t10.top_left_latitude AS top_left_latitude_16, t10.top_left_longitude AS top_left_longitude_17, t10.bottom_right_latitude AS bottom_right_latitude_18, t10.bottom_right_longitude AS bottom_right_longitude_19, t0.user_id AS user_id_20 FROM search_profile t0 LEFT JOIN job_search t10 ON t0.job_search_hash = t10.hash WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.name = 'Alle Jobs von some.ch'
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.user_id AS user_id_6, t0.type FROM bookmark t0 WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.type IN ('companybookmark') ORDER BY t0.created_at DESC
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f3c7094e330]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb+0xe8)[0x7f3c726c4e88]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /usr/sbin/mysqld(binlog_background_thread+0x2c5)[0x7f3c726c5525]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f3c70946184]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.user_id AS user_id_6, t0.type FROM bookmark t0 WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.bookmarked_entity_id = '20673' AND t0.type IN ('companybookmark') LIMIT 1
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 2016-12-16 21:16:34 139892131375872 [Note] WSREP: cleanup transaction for LOCAL_STATE: SELECT t0.id AS id_1, t0.bookmarked_entity_id AS bookmarked_entity_id_2, t0.created_at AS created_at_3, t0.updated_at AS updated_at_4, t0.note AS note_5, t0.user_id AS user_id_6, t0.type FROM bookmark t0 WHERE t0.user_id = '3601185e-ef53-437a-91b2-4035ac4ed858' AND t0.type IN ('consultantbookmark') ORDER BY t0.created_at DESC
 
Dec 16 21:16:34 galera01-some2-prod mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f3c6fe6537d]
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: Trying to get some variables.
 
Dec 16 21:16:34 galera01-some2-prod mysqld: Some pointers may be invalid and cause the dump to abort.
 
Dec 16 21:16:34 galera01-some2-prod mysqld: Query (0x0): is an invalid pointer
 
Dec 16 21:16:34 galera01-some2-prod mysqld: Connection ID (thread ID): 3
 
Dec 16 21:16:34 galera01-some2-prod mysqld: Status: NOT_KILLED
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Dec 16 21:16:34 galera01-some2-prod mysqld: information that should help you find out what is causing the crash.
 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld: We think the query pointer is invalid, but we will try to print it anyway. 
Dec 16 21:16:34 galera01-some2-prod mysqld: Query: 
Dec 16 21:16:34 galera01-some2-prod mysqld: 
Dec 16 21:16:34 galera01-some2-prod mysqld_safe: Number of processes running now: 0
 
Dec 16 21:16:34 galera01-some2-prod mysqld_safe: WSREP: not restarting wsrep node automatically
 
Dec 16 21:16:34 galera01-some2-prod mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

Comment by Florian Engelmann [ 2016-12-17 ]

the node crashed after a binlog rotation (no logrotate or any other external command was involved)

rw-rw--- 1 mysql adm 102458 Dec 16 20:33 mariadb-bin.000018
rw-rw--- 1 mysql adm 102511 Dec 16 20:46 mariadb-bin.000019
rw-rw--- 1 mysql adm 102727 Dec 16 20:56 mariadb-bin.000020
rw-rw--- 1 mysql adm 113520 Dec 16 21:00 mariadb-bin.000021
rw-rw--- 1 mysql adm 102933 Dec 16 21:16 mariadb-bin.000022
rw-rw--- 1 mysql adm 370 Dec 16 21:16 mariadb-bin.000023

  1. stat mariadb-bin.000022
    File: 'mariadb-bin.000022'
    Size: 102933 Blocks: 216 IO Block: 4096 regular file
    Device: fc16h/64534d Inode: 672012 Links: 1
    Access: (0660/rw-rw---) Uid: ( 107/ mysql) Gid: ( 4/ adm)
    Access: 2016-12-17 04:21:56.974904275 +0100
    Modify: 2016-12-16 21:16:33.000000000 +0100
    Change: 2016-12-16 23:07:52.789561273 +0100
    Birth: -
  1. stat mariadb-bin.000023
    File: 'mariadb-bin.000023'
    Size: 370 Blocks: 8 IO Block: 4096 regular file
    Device: fc16h/64534d Inode: 672014 Links: 1
    Access: (0660/rw-rw---) Uid: ( 107/ mysql) Gid: ( 4/ adm)
    Access: 2016-12-17 04:21:56.974904275 +0100
    Modify: 2016-12-16 21:16:33.000000000 +0100
    Change: 2016-12-16 23:07:52.953563685 +0100
    Birth: -
Comment by Florian Engelmann [ 2016-12-18 ]

As expected some of the Galera Cluster nodes got more than one hanging flush logs (initiated by logrotation):

xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node2
474688 debian-sys-maint localhost NULL Query 147641 init flush logs 0.000
1750598 debian-sys-maint localhost NULL Query 61032 init flush logs 0.000
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node3
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node4
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node5
1079667 debian-sys-maint localhost NULL Query 147108 init flush logs 0.000
2263433 debian-sys-maint localhost NULL Query 61224 init flush logs 0.000

Comment by Florian Engelmann [ 2016-12-18 ]

mount options (/var/lib/mysql) are:

ext4 nouser_xattr,noacl,nodiratime,noatime,data=writeback,commit=60,nodev

Comment by Florian Engelmann [ 2016-12-19 ]

To make things even worth the backup script using xtrabackup seems to hang as well (node1):

xxxxxxxxxxxxxxxxxxxxxxxxxxxxx 1
783989 root localhost NULL Query 32889 init FLUSH BINARY LOGS 0.000
794756 debian-sys-maint localhost NULL Query 24959 init flush logs 0.000
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2
474688 debian-sys-maint localhost NULL Query 197724 init flush logs 0.000
1750598 debian-sys-maint localhost NULL Query 111115 init flush logs 0.000
3080494 debian-sys-maint localhost NULL Query 25435 init flush logs 0.000
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx 3
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx 4
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx 5
1079667 debian-sys-maint localhost NULL Query 197192 init flush logs 0.000
2263433 debian-sys-maint localhost NULL Query 111308 init flush logs 0.000
3495292 debian-sys-maint localhost NULL Query 25987 init flush logs 0.000

Comment by Florian Engelmann [ 2016-12-19 ]

I changed the mount options on node1 to see if the problem is related to those.

Comment by Daniel Black [ 2016-12-19 ]

engel75, sorry to say changing the mount options is highly unlikely to help. Getting the my.cnf file prepared to capture a core file will help a lot.
https://mariadb.com/kb/en/mariadb/how-to-produce-a-full-stack-trace-for-mysqld/. Your problem is also a crash unlike the original bug report so its probably related to another bug report https://jira.mariadb.org/issues/?jql=text%20~%20%22_ZN13MYSQL_BIN_LOG13mark_xid_doneEmb%22

Comment by Florian Engelmann [ 2016-12-20 ]

@Daniel Black

true - the mount options didn't change anything:

xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node1
471025 root localhost NULL Query 38700 init FLUSH BINARY LOGS 0.000
471544 debian-sys-maint localhost NULL Query 31587 init flush logs 0.000
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node2
1104107 debian-sys-maint localhost NULL Query 31777 init flush logs 0.000
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node3
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node4
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx node5
1079667 debian-sys-maint localhost NULL Query 289409 init flush logs 0.000
2263433 debian-sys-maint localhost NULL Query 203525 init flush logs 0.000
3495292 debian-sys-maint localhost NULL Query 118204 init flush logs 0.000
5106069 debian-sys-maint localhost NULL Query 31333 init flush logs 0.000

I've installed 10.1.20 on node1 today to test the new release. If the issue isn't solved I will compile a debugging binary to get a core dump and a full stack trace.

Comment by Daniel Black [ 2016-12-20 ]

Even a core dump and full stack trace from a non-debug build might help.

Comment by Florian Engelmann [ 2016-12-21 ]

Ok one node is running with:

[mysqld-safe]
 
--core-file-size=unlimited
 
 
 
[mysqld]
 
#max_binlog_size         = 100K
 
stack-trace
 
core-file
 
disable-gdb

Comment by Florian Engelmann [ 2016-12-22 ]

4 of the 5 nodes had a stuck flush logs process only the one running with those settings from above was fine. I guess ist will take some time to get a core-dump.

Comment by Florian Engelmann [ 2017-01-16 ]

I've got some core dumps:

node02:
rw------ 1 mysql mysql 9.7G Jan 13 02:22 /var/tmp/core.mysqld.62302.11.1484270553
node03:
rw------ 1 mysql mysql 9.9G Jan 10 02:20 /var/tmp/core.mysqld.26218.11.1484011188
node04:
rw------ 1 mysql mysql 9.5G Jan 16 08:42 /var/tmp/core.mysqld.19041.6.1484552521
rw------ 1 mysql mysql 9.5G Jan 16 09:06 /var/tmp/core.mysqld.26623.6.1484553963
rw------ 1 mysql mysql 9.4G Jan 16 09:26 /var/tmp/core.mysqld.51632.6.1484555202
rw------ 1 mysql mysql 11G Jan 15 02:24 /var/tmp/core.mysqld.54607.11.1484443336

Compressed those are about ~ 2 GB I guess. Where to upload those files?

Comment by Nirbhay Choubey (Inactive) [ 2017-01-16 ]

You can upload them here : ftp.askmonty.org/private
More details : https://mariadb.com/kb/en/mariadb/reporting-bugs/

Comment by Nirbhay Choubey (Inactive) [ 2017-01-16 ]

engel75 Do you also have general query log and error log enabled? If so, and if possible, please also upload them as well.

Comment by Florian Engelmann [ 2017-01-16 ]

file names are:
MDEV-10809-syslog.log
MDEV-10809-core.mysqld.54607.11.1484443336.tar.xz

Query log does not exist.

Please let me know if you need anything else.

Comment by Florian Engelmann [ 2017-01-19 ]

Dear nirbhay_c

any update on this? Can I do anything to support you?

All the best,
Florian

Comment by Florian Engelmann [ 2017-01-25 ]

Dear nirbhay_c

could you please link this issue to MDEV-9510?

All the best,
Florian

Comment by Sachin Setiya (Inactive) [ 2017-04-28 ]

Hi Florian!

I tried the core-dump you provide , Mariadb-10.1.20 , ubuntu 14.04 , release build ,

But stack-trace I got was corrupted ,
(gdb) bt
#0 0x00007f3add8e4611 in ?? ()
#1 0x00007f3adf59cb3c in ?? ()
#2 0x000000000000000f in ?? ()
#3 0x00007f3741421d88 in ?? ()
#4 0x00000000587acec7 in ?? ()
#5 0x0000000b0000001e in ?? ()
#6 0x0000000000000000 in ?? ()

Can you open log and print the output of

info locals
thread apply all bt
Or can you upload a newer stack trace

Regards
sachin

Comment by Florian Engelmann [ 2017-04-28 ]

Dear sachin.setiya.007

I will try to get a fresh core dump. Currently we do run:

mysql  Ver 15.1 Distrib 10.1.21-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

The problem does still exist. I will upload a fresh dump asap.

Comment by Sachin Setiya (Inactive) [ 2017-04-28 ]

Hi Florian!
Thanks!
Which version of debian so that I can download it now

regards
sachin

Comment by Florian Engelmann [ 2017-04-28 ]

it's still Ubuntu:
Ubuntu 14.04.5 LTS

Comment by Florian Engelmann [ 2017-05-03 ]

Hi sachin.setiya.007,

I was not able to produce a fresh core dump till now but I noticed something else. We are running that five node galera cluster on five HP servers having 16 SAS drives as raid 10. The write cache of the raid controller failed two days ago on one of those hosts and since then the mariadb instance in that host dies two times. We experienced the same "behaviour" when all the trouble started. One of the Hosts had a failed cache battery and the mariadb instance on that host dies a couple of times. So to me it looks like all this is IO load related? We have not been able to reproduce the "flush logs" problem on our test environment but the test environment is running almost idle. So that's another indicator this could be load related.

All the best,
Florian

Comment by Sachin Setiya (Inactive) [ 2017-05-03 ]

Hi engel75,

May be it is related to load , because on our test system we were not able to reproduce it.
Trying with galera cluster on 16.04 on 3 node virtual box vm

Regards
sachin

Comment by Florian Engelmann [ 2017-05-03 ]

Dear sachin.setiya.007,

I was wrong - there was a fresh core dump created last night. I uploaded it packed with xz including the binary to ftp.askmonty.org/private

MDEV-10809_florian_engelmann-20170503-001.tar

The core dump is quite big: 9.6GB

All the best,
Florian

Comment by Sachin Setiya (Inactive) [ 2017-05-03 ]

Hi engel75,

Thanks, I will try to download it, It will take some time because I dont have that much fast internet .
btw you can try uploading compressed core , if it is possible.

Regards
sachin

Comment by Florian Engelmann [ 2017-05-03 ]

Dear sachin.setiya.007,

it is compressed by xz. The tar was created to add the binary to the package. So it's just ~900MB compressed but 9.6GB uncompressed.

All the best,
Florian

Comment by Sachin Setiya (Inactive) [ 2017-05-03 ]

Hi engel75,

Ohh okay, thanks for your help.

Regards
sachin

Comment by Andrei Elkin [ 2017-12-13 ]

Fixed by MDEV-9510.

Generated at Thu Feb 08 07:45:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.