Fully updated Centos 5 Openvz container, running on a fully updated Centos 6 Hardware Node.

1. cat /proc/version
   Linux version 2.6.32-042stab084.26 (root@rh6-build-x64) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Mon Feb 17 21:00:14 MSK 2014

Is it possible that you have disk problems?

No it is not. The two log files are from two different boxes with a RAID 10 array thoroughly tested for errors from time to time.

On the same affected virtual machine/hardware node we switched back to Oracle MySQL 5.5.36 and did not get these crashes. We have also seen intermittent crashes when running MariaDB >= 5.5.35 on CentOS 6 x86_64 containers under OpenVZ CentOS 6. So it appears to affect both CentOS 5 and CentOS 6.

I will try and get the particular query that is causing these issues.

We got a new crash on a fully updated centos 6 container and Host Node. Bellow you can find what was logged to syslog:

Mar 21 09:15:33 rps-ppcdb02 mysqld: 140321 9:15:33 InnoDB: Assertion failure in thread 140642103580416 in file os0sync.c line 434
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: Failing assertion: event
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: We intentionally generate a memory trap.
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: If you get repeated assertion failures or crashes, even
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: immediately after the mysqld startup, there may be
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Mar 21 09:15:33 rps-ppcdb02 mysqld: InnoDB: about forcing recovery.
Mar 21 09:15:33 rps-ppcdb02 mysqld: 140321 9:15:33 [ERROR] mysqld got signal 6 ;
Mar 21 09:15:33 rps-ppcdb02 mysqld: This could be because you hit a bug. It is also possible that this binary
Mar 21 09:15:33 rps-ppcdb02 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Mar 21 09:15:33 rps-ppcdb02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Mar 21 09:15:33 rps-ppcdb02 mysqld:
Mar 21 09:15:33 rps-ppcdb02 mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Mar 21 09:15:33 rps-ppcdb02 mysqld:
Mar 21 09:15:33 rps-ppcdb02 mysqld: We will try our best to scrape up some info that will hopefully help
Mar 21 09:15:33 rps-ppcdb02 mysqld: diagnose the problem, but since we have already crashed,
Mar 21 09:15:33 rps-ppcdb02 mysqld: something is definitely wrong and this may fail.
Mar 21 09:15:33 rps-ppcdb02 mysqld:
Mar 21 09:15:33 rps-ppcdb02 mysqld: Server version: 5.5.36-MariaDB
Mar 21 09:15:33 rps-ppcdb02 mysqld: key_buffer_size=134217728
Mar 21 09:15:33 rps-ppcdb02 mysqld: read_buffer_size=131072
Mar 21 09:15:33 rps-ppcdb02 mysqld: max_used_connections=111
Mar 21 09:15:33 rps-ppcdb02 mysqld: max_threads=1002
Mar 21 09:15:33 rps-ppcdb02 mysqld: thread_count=4
Mar 21 09:15:33 rps-ppcdb02 mysqld: It is possible that mysqld could use up to
Mar 21 09:15:33 rps-ppcdb02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2329316 K bytes of memory
Mar 21 09:15:33 rps-ppcdb02 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Mar 21 09:15:33 rps-ppcdb02 mysqld:
Mar 21 09:15:33 rps-ppcdb02 mysqld: Thread pointer: 0x0x0
Mar 21 09:15:33 rps-ppcdb02 mysqld: Attempting backtrace. You can use the following information to find out
Mar 21 09:15:33 rps-ppcdb02 mysqld: where mysqld died. If you see no messages after this, something went
Mar 21 09:15:33 rps-ppcdb02 mysqld: terribly wrong...
Mar 21 09:15:33 rps-ppcdb02 mysqld: stack_bottom = 0x0 thread_stack 0x48000
Mar 21 09:15:33 rps-ppcdb02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa6c18b]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6cdb98]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /lib64/libpthread.so.0(+0xf710)[0x7fedb3140710]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /lib64/libc.so.6(gsignal+0x35)[0x7fedb19a1925]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /lib64/libc.so.6(abort+0x175)[0x7fedb19a3105]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /usr/sbin/mysqld[0x92467c]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /usr/sbin/mysqld[0x848dcf]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /usr/sbin/mysqld[0x841c35]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /lib64/libpthread.so.0(+0x79d1)[0x7fedb31389d1]
Mar 21 09:15:33 rps-ppcdb02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fedb1a57b6d]
Mar 21 09:15:33 rps-ppcdb02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Mar 21 09:15:33 rps-ppcdb02 mysqld: information that should help you find out what is causing the crash.
Mar 21 09:15:35 rps-ppcdb02 mysqld_safe: Number of processes running now: 0
Mar 21 09:15:35 rps-ppcdb02 mysqld_safe: mysqld restarted
Mar 21 09:15:35 rps-ppcdb02 mysqld: 140321 9:15:35 InnoDB: The InnoDB memory heap is disabled
Mar 21 09:15:35 rps-ppcdb02 mysqld: 140321 9:15:35 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Mar 21 09:15:35 rps-ppcdb02 mysqld: 140321 9:15:35 InnoDB: Compressed tables use zlib 1.2.3
Mar 21 09:15:35 rps-ppcdb02 mysqld: 140321 9:15:35 InnoDB: Using Linux native AIO
Mar 21 09:15:35 rps-ppcdb02 mysqld: 140321 9:15:35 InnoDB: Initializing buffer pool, size = 14.0G
Mar 21 09:15:36 rps-ppcdb02 mysqld: 140321 9:15:36 InnoDB: Completed initialization of buffer pool
Mar 21 09:15:36 rps-ppcdb02 mysqld: 140321 9:15:36 InnoDB: highest supported file format is Barracuda.
Mar 21 09:15:36 rps-ppcdb02 mysqld: InnoDB: Log scan progressed past the checkpoint lsn 5327544896371
Mar 21 09:15:36 rps-ppcdb02 mysqld: 140321 9:15:36 InnoDB: Database was not shut down normally!
Mar 21 09:15:36 rps-ppcdb02 mysqld: InnoDB: Starting crash recovery.
Mar 21 09:15:36 rps-ppcdb02 mysqld: InnoDB: Reading tablespace information from the .ibd files...
Mar 21 09:16:11 rps-ppcdb02 mysqld: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 21 09:16:11 rps-ppcdb02 mysqld: InnoDB: buffer...
Mar 21 09:16:11 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5327544906430
Mar 21 09:16:11 rps-ppcdb02 mysqld: InnoDB: 1 transaction(s) which must be rolled back or cleaned up
Mar 21 09:16:11 rps-ppcdb02 mysqld: InnoDB: in total 82588 row operations to undo
Mar 21 09:16:11 rps-ppcdb02 mysqld: InnoDB: Trx id counter is 49CE8C00
Mar 21 09:16:11 rps-ppcdb02 mysqld: 140321 9:16:11 InnoDB: Starting an apply batch of log records to the database...
Mar 21 09:16:13 rps-ppcdb02 mysqld: InnoDB: Progress in percents: 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
Mar 21 09:16:13 rps-ppcdb02 mysqld: InnoDB: Apply batch completed
Mar 21 09:16:13 rps-ppcdb02 mysqld: InnoDB: Starting in background the rollback of uncommitted transactions
Mar 21 09:16:13 rps-ppcdb02 mysqld: 140321 9:16:13 InnoDB: Rolling back trx with id 49CE8A0A, 82588 rows to undo
Mar 21 09:16:13 rps-ppcdb02 mysqld:
Mar 21 09:16:13 rps-ppcdb02 mysqld: InnoDB: Progress in percents: 1140321 9:16:13 InnoDB: Waiting for the background threads to start
Mar 21 09:16:14 rps-ppcdb02 mysqld: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33140321 9:16:14 Percona XtraDB (http://www.percona.com) 5.5.36-MariaDB-33.0 started; log sequence number 5327544906430
Mar 21 09:16:14 rps-ppcdb02 mysqld: 34 35 36 37 38140321 9:16:14 [Note] Plugin 'FEEDBACK' is disabled.
Mar 21 09:16:14 rps-ppcdb02 mysqld: 39 40140321 9:16:14 [Note] Server socket created on IP: '::'.
Mar 21 09:16:15 rps-ppcdb02 mysqld: 41 42 43 44 45 46 47140321 9:16:15 [Note] Event Scheduler: Loaded 0 events
Mar 21 09:16:15 rps-ppcdb02 mysqld: 140321 9:16:15 [Note] /usr/sbin/mysqld: ready for connections.
Mar 21 09:16:15 rps-ppcdb02 mysqld: Version: '5.5.36-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
Mar 21 09:16:16 rps-ppcdb02 mysqld: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
Mar 21 09:16:16 rps-ppcdb02 mysqld: InnoDB: Rolling back of trx id 49CE8A0A completed
Mar 21 09:16:16 rps-ppcdb02 mysqld: 140321 9:16:16 InnoDB: Rollback of non-prepared transactions completed

Could you please attach your cnf files from both servers?

Thanks Elena, we will get those for you.

In both case these crashes seem to be being caused by some sort of locking issue.

The query being run in this case was taking much longer than it normally does, and on our previous server several simple insert queries were taking much longer than normal.

When I saw a locked query, if I ran show engine innodb status Maria would immediately crash hard.

The kernel version is:

Linux 2.6.32-042stab084.26 #1 SMP Mon Feb 17 21:00:14 MSK 2014 x86_64 x86_64 x86_64 GNU/Linux

We are using the official CentOS 6 RPMs supplied by MariaDB:

rpm -qi MariaDB-server
Name : MariaDB-server Relocations: /usr
Version : 5.5.36 Vendor: Monty Program AB
Release : 1.el6 Build Date: Mon 24 Feb 2014 17:14:26 UTC
Install Date: Mon 17 Mar 2014 09:46:41 UTC Build Host: centos6-amd64
Group : Applications/Databases Source RPM: MariaDB-server-5.5.36-1.el6.src.rpm
Size : 179364728 License: GPL
Signature : DSA/SHA1, Mon 24 Feb 2014 18:25:36 UTC, Key ID cbcb082a1bb943db
URL : http://mariadb.org
Summary : MariaDB: a very fast and robust SQL database server

The config files are exactly the same for both servers. You can find them pasted bellow:

[root@rps-ppcdb02 ~]# cat /etc/my.cnf
#

1. This group is read both both by the client and the server
2. use it for options that affect everything
   #
   [client-server]

#

1. include all files from the config directory
   #
   !includedir /etc/my.cnf.d

[root@rps-ppcdb02 ~]# cat /etc/my.cnf.d/client.cnf
#

1. These two groups are read by the client library
2. Use it for options that affect all clients, but not the server
   #

[client]

1. This group is not read by mysql client library,
2. If you use the same .cnf file for MySQL and MariaDB,
3. use it for MariaDB-only client options
   [client-mariadb]

[root@rps-ppcdb02 ~]# cat /etc/my.cnf.d/init.cnf
[mysqld]
init-file=/var/lib/mysql/init.sql

[root@rps-ppcdb02 ~]# cat /etc/my.cnf.d/mysql-clients.cnf
#

1. These groups are read by MariaDB command-line tools
2. Use it for options that affect only one utility
   #

[mysql]

[mysql_upgrade]

[mysqladmin]

[mysqlbinlog]

[mysqlcheck]

[mysqldump]

[mysqlimport]

[mysqlshow]

[mysqlslap]

[root@rps-ppcdb02 ~]# cat /etc/my.cnf.d/server.cnf
#

1. These groups are read by MariaDB server.
2. Use it for options that only the server (but not clients) should see
   #
3. See the examples of server my.cnf files in /usr/share/mysql/
   #

1. this is read by the standalone daemon and embedded servers
   [server]

1. this is only for the mysqld standalone daemon
   [mysqld]

#Infinity settings
symbolic-links=0
local-infile=0
innodb_file_per_table=1
max_connections=1000
innodb_buffer_pool_size=256M
innodb_file_format=Barracuda
innodb_old_blocks_time=1000
innodb_flush_log_at_trx_commit=2
bind-address=::

[mysqld_safe]
syslog

1. this is only for embedded server
   [embedded]

1. This group is only read by MariaDB-5.5 servers.
2. If you use the same .cnf file for MariaDB of different versions,
3. use this group for options that older servers don't understand
   [mysqld-5.5]

1. These two groups are only read by MariaDB servers, not by MySQL.
2. If you use the same .cnf file for MySQL and MariaDB,
3. you can put MariaDB-only options here
   [mariadb]

[mariadb-5.5]

[root@rps-ppcdb02 ~]# cat /etc/my.cnf.d/tokudb.cnf
[mariadb]

1. See https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/
2. for instructions how to enable TokuDB
   #
3. See https://mariadb.com/kb/en/tokudb-differences/ for differences
4. between TokuDB in MariaDB and TokuDB from http://www.tokutek.com/

1. Note, that only one plugin-load option will be recognized by the
2. MariaDB server. If you uncomment the line below, make sure that
3. you don't also have plugin-load options in your other
4. my.cnf files. Otherwise, add ha_tokudb.so to the existing
5. plugin-load option, instead of uncommenting the line below.

#plugin-load=ha_tokudb.so

[root@rps-ppcdb02 ~]# cat /etc/my.cnf.d/x-role.cnf
#

1. Overrides for PPC database servers
   #

1. this is only for the mysqld standalone daemon
   [mysqld]

innodb_log_file_size=50M
innodb_buffer_pool_size=14G
innodb_buffer_pool_instances=10

tmp_table_size=1G
max_heap_table_size=1G

Hi,

We had another crash on this server yesterday around the same time:

Mar 22 10:13:27 rps-ppcdb02 mysqld: InnoDB: Free list record 10699 is above rec heap top 7010
Mar 22 10:13:27 rps-ppcdb02 mysqld: 140322 10:13:27 InnoDB: Page dump in ascii and hex (16384 bytes):
Mar 22 10:13:27 rps-ppcdb02 mysqld: len 16384; hex 55eb0d9e00000bdc00000bc400000bf3000004d9b849106645bf0000000000000000001cf3d900061b62801a29cb093200000002000f00160000000000000000000000000000001c345f00000000000000000000000000000000000000000100020916696e66696d756d0005000b000073757072656d756d01080a090a030a000010035e6c61737432384461797367617736393834393739313130313331383730313637383131363533353532373938393930393632700000003f000049e02d6370000271c12c96001c818252000000000000002700000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a20001800006c6173743238446179736761773639383439373931313031333138373031363738313136353335363437313130303533363236326200000000000049b6a1ea0900029dfc3c85001c818500000000000000000100000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a200020fee06c617374323844617973676177363938343937393131303133313837303136373831313635333536343731313030353336323632620000003f000049b6a1ea0900029dfc3d4a001c81850000000000000000010000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001080a090a030a060028011e6c6173743238446179736761773639383439373931
Mar 22 10:13:27 rps-ppcdb02 mysqld: 31303133313837303136373831313635333536343731323237333033306200000000000049e02d6370000271c12d5c001c81820a00000000000003ca000000000000000e8000001924000000000000000380000000008000005a0000008000000000000080000008000000800001e000000080000003000000800001e000000080000005000000800000000000008000000000000080000000000000800000000000008000000000000080000008000000800001e000000080000003000000800001e0000000800000012c8000000151800000390e800000031180000023478000000507800000152b800000082d800001c640800007644a800000221d01080a090a030a000030011e6c61737432384461797367617736393834393739313130313331383730313637383131363533353634373132323733303330620000003f000049e02d6370000271c12e22001c81820a00000000000003ca000000000000000e8000001924000000000000000380000000008000005a0000008000000000000080000008000000800001e000000080000003000000800001e000000080000005000000800000000000008000000000000080000000000000800000000000008000000000000080000008000000800001e000000080000003000000800001e0000000800000012c8000000151800000390e8000000311800000234780000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 00507800000152b800000082d800001c640800007644a800000221d01080a090a030a000038011e6c617374323844617973676177363938343937393131303133313837303136373831313635333536343731343330313938326500000000000049e02d6370000271c12ee8001c81830000000000000000020000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001080a090a030a000040011e6c61737432384461797367617736393834393739313130313331383730313637383131363533353634373134333031393832650000003f000049e02d6370000271c12fae001c8183000000000000000002000000000000000080000000000000000000000000800000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 0000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a000048f8166c61737432384461797367617736393834393739313130313331383730313637383131363533353634373135353232393331336200000000000049e02d6370000271c13074001c81825500000000000000060000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a000050011f6c6173743238446179736761773639383439373931313031333138373031363738313136353335353237323838373539313338650000003f000049e02d6370000271c1200c001c81850000000000000000010000000000000000800000000000000000000000008000000000800000000000008000000000000080000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 0000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a00005801206c6173743238446179736761773639383439373931313031333138373031363738313136353335353237333136373432393732316200000000000049e02d6370000271c120d4001c818521000000000000000300000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a00006001206c617374323844617973676177363938343937393131303133313837303136373831313635333535323733313637343239373231620000003f000049e0
Mar 22 10:13:27 rps-ppcdb02 mysqld: 2d6370000271c1219e001c81852100000000000000030000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a040068011f6c61737432384461797367617736393834393739313130313331383730313637383131363533353532373333313930313137357000000000000049e02d6370000271c12268001c81812100000000000000030000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a00007
Mar 22 10:13:27 rps-ppcdb02 mysqld: 0011f6c6173743238446179736761773639383439373931313031333138373031363738313136353335353237333331393031313735700000003f000049e02d6370000271c12330001c81812100000000000000030000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a000078011f6c61737432384461797367617736393834393739313130313331383730313637383131363533353532373333323036333533356200000000000049e02d6370000271c123f8001c818309000000000000002d0000000000000002800000010200000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 00042c80000000338000000000800000000080000000008000000000800000000080000000007ffffffefd8000000000800000000001090a090a030a000080011f6c6173743238446179736761773639383439373931313031333138373031363738313136353335353237333332303633353335620000003f000049e02d6370000271c124c0001c818309000000000000002d000000000000000280000001020000000000000000800000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000042c80000000338000000000800000000080000000008000000000800000000080000000007ffffffefd8000000000800000000001090a090a030a040088011f6c61737432384461797367617736393834393739313130313331383730313637383131363533353532373335343034303435376500000000000049e02d6370000271c12588001c818300000000000000000900000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 00000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a000090011f6c6173743238446179736761773639383439373931313031333138373031363738313136353335353237333534303430343537650000003f000049e02d6370000271c12650001c818300000000000000000900000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a00009801206c6173743238446179736761773639383439373931313031333138373031363738313136353335353237333632393933393031326200000000000049e02d6370000271c12718001c818260000000000000002c0000000000000000800000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 0000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a0000a001206c617374323844617973676177363938343937393131303133313837303136373831313635333535323733363239393339303132620000003f000049e02d6370000271c127e2001c818260000000000000002c0000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a0400a8011f6c617374323844617973676177363938343937393131303133313837303136373
Mar 22 10:13:27 rps-ppcdb02 mysqld: 83131363533353532373534353433383438387000000000000049e02d6370000271c128ac001c81820000000000000000010000000000000000800000000000000000000000008000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001090a090a030a0000b0011f6c6173743238446179736761773639383439373931313031333138373031363738313136353335353237353435343338343838700000003f000049e02d6370000271c12974001c818200000000000000000100000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000800000000080000000008000000000010a0a090a030a0000b801206c6173743238446179736761773639383439373931313031333138373031363738313136353335353237393030333031323839336500000000000049e02d6370000271c12a3c001c818300000000000000000100000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000010a0a090a030a0000c001206c617374323844617973676177363938343937393131303133313837303136373831313635333535323739303033303132383933650000003f000049e02d6370000271c12b06001c818300000000000000000100000000000000008000000000000000000000000080000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008
Mar 22 10:13:27 rps-ppcdb02 mysqld: 00000000000008000000000000080000000000000800000000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000001080a090a030a0000c8e6346c617374323844617973676177363938343937393131303133313837303136373831313635333535323739383939303936327000000000000049e02d6370000271c12bd0001c8182520000000000000027000000000000000080000000000000000000000000800000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000000008000000000000080000000000000800000000080000000008000000000800000000080000000008000000000800000000080000000008000000000800000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:27 rps-ppcdb02 mysqld: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Mar 22 10:13:28 rps-ppcdb02 mysqld: 00000000007003e215d211540cd80063844134ffb8491066; asc U I fE b ( X 4_ infimum supremum ^last28Daysgaw6984979110131870167811653552798990962p ? I -cp q , R ' last28Daysgaw698497911013187016781165356471100536262b I < last28Daysgaw698497911013187016781165356471100536262b ? I =J
Mar 22 10:13:28 rps-ppcdb02 mysqld: ( last28Daysgaw6984979110131870167811653564712273030b I -cp q -\ $ Z , Q 9 #G + - @ dJ " 0 last28Daysgaw6984979110131870167811653564712273030b ? I -cp q ." $ Z , Q 9 #G + - @ dJ " 8 last28Daysgaw6984979110131870167811653564714301982e I -cp q . @ last28Daysgaw6984979110131870167811653564714301982e ? I -cp q /
Mar 22 10:13:28 rps-ppcdb02 mysqld: H last28Daysgaw69849791101318701678116535647155229313b I -cp q 0t U P last28Daysgaw69849791101318701678116535527288759138e ? I -cp q X last28Daysgaw698497911013187016781165355273167429721b I -cp q ! ` last28Daysgaw69
Mar 22 10:13:28 rps-ppcdb02 mysqld: 8497911013187016781165355273167429721b ? I -cp q ! ! h last28Daysgaw69849791101318701678116535527331901175p I -cp q "h ! p last28Daysgaw69849791101318701678116535527331901175p ? I -cp q #0 ! x last28Daysgaw69849791101318701678116535527332063535b I -cp q # -
Mar 22 10:13:28 rps-ppcdb02 mysqld: , 3 last28Daysgaw69849791101318701678116535527332063535b ? I -cp q $ - , 3 last28Daysgaw69849791101318701678116535527354040457e I -cp q % last28Daysgaw69849791101318701678116535527354040457e ? I -cp q &P last28Daysgaw69849791101318701678116535527362993901
Mar 22 10:13:28 rps-ppcdb02 mysqld: 2b I -cp q ' ` , last28Daysgaw698497911013187016781165355273629939012b ? I -cp q ' ` , last28Daysgaw69849791101318701678116535527545438488p I -cp q ( last28Daysgaw69849791101318701678116535527545438488p ? I -cp q )t
Mar 22 10:13:28 rps-ppcdb02 mysqld: last28Daysgaw698497911013187016781165355279003012893e I -cp q *< last28Daysgaw698497911013187016781165355279003012893e ? I -cp q + 4last28Daysgaw6984979110131870167811653552798990962p I -cp q + R '
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld: p T c A4 I f;
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: End of page dump
Mar 22 10:13:28 rps-ppcdb02 mysqld: 140322 10:13:28 InnoDB: Page checksum 2261674940 (32bit_calc: 1016644945), prior-to-4.0.14-form checksum 2218865919
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: stored checksum 1441467806, prior-to-4.0.14-form stored checksum 2218865919
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: Page lsn 1241 3091796070, low 4 bytes of lsn at page end 3091796070
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: Page number (if stored to page already) 3036,
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 1897433
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: Page may be an index page where index id is 1848415
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: (index "PRIMARY" of table "igrp_422"."ppcFixedPeriodKeywordAggregate")
Mar 22 10:13:28 rps-ppcdb02 mysqld: 140322 10:13:28 InnoDB: Apparent corruption of an index page n:o 3036 in space 1897433
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: to be written to data file. We intentionally crash server
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: to prevent corrupt data from ending up in data
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: files.
Mar 22 10:13:28 rps-ppcdb02 mysqld: 140322 10:13:28 InnoDB: Assertion failure in thread 140077629941504 in file buf0flu.c line 806
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: We intentionally generate a memory trap.
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: If you get repeated assertion failures or crashes, even
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: immediately after the mysqld startup, there may be
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Mar 22 10:13:28 rps-ppcdb02 mysqld: InnoDB: about forcing recovery.
Mar 22 10:13:28 rps-ppcdb02 mysqld: 140322 10:13:28 [ERROR] mysqld got signal 6 ;
Mar 22 10:13:28 rps-ppcdb02 mysqld: This could be because you hit a bug. It is also possible that this binary
Mar 22 10:13:28 rps-ppcdb02 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Mar 22 10:13:28 rps-ppcdb02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld: We will try our best to scrape up some info that will hopefully help
Mar 22 10:13:28 rps-ppcdb02 mysqld: diagnose the problem, but since we have already crashed,
Mar 22 10:13:28 rps-ppcdb02 mysqld: something is definitely wrong and this may fail.
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld: Server version: 5.5.36-MariaDB
Mar 22 10:13:28 rps-ppcdb02 mysqld: key_buffer_size=134217728
Mar 22 10:13:28 rps-ppcdb02 mysqld: read_buffer_size=131072
Mar 22 10:13:28 rps-ppcdb02 mysqld: max_used_connections=97
Mar 22 10:13:28 rps-ppcdb02 mysqld: max_threads=1002
Mar 22 10:13:28 rps-ppcdb02 mysqld: thread_count=28
Mar 22 10:13:28 rps-ppcdb02 mysqld: It is possible that mysqld could use up to
Mar 22 10:13:28 rps-ppcdb02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2329316 K bytes of memory
Mar 22 10:13:28 rps-ppcdb02 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Mar 22 10:13:28 rps-ppcdb02 mysqld:
Mar 22 10:13:28 rps-ppcdb02 mysqld: Thread pointer: 0x0x0
Mar 22 10:13:28 rps-ppcdb02 mysqld: Attempting backtrace. You can use the following information to find out
Mar 22 10:13:28 rps-ppcdb02 mysqld: where mysqld died. If you see no messages after this, something went
Mar 22 10:13:28 rps-ppcdb02 mysqld: terribly wrong...
Mar 22 10:13:28 rps-ppcdb02 mysqld: stack_bottom = 0x0 thread_stack 0x48000
Mar 22 10:13:28 rps-ppcdb02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa6c18b]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6cdb98]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /lib64/libpthread.so.0(+0xf710)[0x7f6a49cd1710]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /lib64/libc.so.6(gsignal+0x35)[0x7f6a48532925]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /lib64/libc.so.6(abort+0x175)[0x7f6a48534105]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /usr/sbin/mysqld[0x89829c]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /usr/sbin/mysqld[0x89bf48]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /usr/sbin/mysqld[0x8436f7]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /lib64/libpthread.so.0(+0x79d1)[0x7f6a49cc99d1]
Mar 22 10:13:28 rps-ppcdb02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f6a485e8b6d]
Mar 22 10:13:28 rps-ppcdb02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Mar 22 10:13:28 rps-ppcdb02 mysqld: information that should help you find out what is causing the crash.
Mar 22 10:13:30 rps-ppcdb02 mysqld_safe: Number of processes running now: 0
Mar 22 10:13:30 rps-ppcdb02 mysqld_safe: mysqld restarted
Mar 22 10:13:30 rps-ppcdb02 mysqld: 140322 10:13:30 InnoDB: The InnoDB memory heap is disabled
Mar 22 10:13:30 rps-ppcdb02 mysqld: 140322 10:13:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Mar 22 10:13:30 rps-ppcdb02 mysqld: 140322 10:13:30 InnoDB: Compressed tables use zlib 1.2.3
Mar 22 10:13:30 rps-ppcdb02 mysqld: 140322 10:13:30 InnoDB: Using Linux native AIO
Mar 22 10:13:30 rps-ppcdb02 mysqld: 140322 10:13:30 InnoDB: Initializing buffer pool, size = 14.0G
Mar 22 10:13:31 rps-ppcdb02 mysqld: 140322 10:13:31 InnoDB: Completed initialization of buffer pool
Mar 22 10:13:31 rps-ppcdb02 mysqld: 140322 10:13:31 InnoDB: highest supported file format is Barracuda.
Mar 22 10:13:31 rps-ppcdb02 mysqld: InnoDB: Log scan progressed past the checkpoint lsn 5333142933456
Mar 22 10:13:31 rps-ppcdb02 mysqld: 140322 10:13:31 InnoDB: Database was not shut down normally!
Mar 22 10:13:31 rps-ppcdb02 mysqld: InnoDB: Starting crash recovery.
Mar 22 10:13:31 rps-ppcdb02 mysqld: InnoDB: Reading tablespace information from the .ibd files...
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Restoring possible half-written data pages from the doublewrite
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: buffer...
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333148175872
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333153418752
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333158661632
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333163904512
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333169147392
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333174390272
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333179633152
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333184876032
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333190118912
Mar 22 10:14:07 rps-ppcdb02 mysqld: InnoDB: Doing recovery: scanned up to log sequence number 5333193700145
Mar 22 10:14:08 rps-ppcdb02 mysqld: InnoDB: 3 transaction(s) which must be rolled back or cleaned up
Mar 22 10:14:08 rps-ppcdb02 mysqld: InnoDB: in total 119080 row operations to undo
Mar 22 10:14:08 rps-ppcdb02 mysqld: InnoDB: Trx id counter is 49E02F00
Mar 22 10:14:08 rps-ppcdb02 mysqld: 140322 10:14:08 InnoDB: Starting an apply batch of log records to the database...
Mar 22 10:14:09 rps-ppcdb02 mysqld: InnoDB: Progress in percents: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
Mar 22 10:14:09 rps-ppcdb02 mysqld: InnoDB: Apply batch completed
Mar 22 10:14:10 rps-ppcdb02 mysqld: InnoDB: Starting in background the rollback of uncommitted transactions
Mar 22 10:14:10 rps-ppcdb02 mysqld: 140322 10:14:10 InnoDB: Rolling back trx with id 49E02D63, 102278 rows to undo
Mar 22 10:14:10 rps-ppcdb02 mysqld:
Mar 22 10:14:10 rps-ppcdb02 mysqld: InnoDB: Progress in percents: 1140322 10:14:10 InnoDB: Waiting for the background threads to start
Mar 22 10:14:11 rps-ppcdb02 mysqld: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34140322 10:14:11 Percona XtraDB (http://www.percona.com) 5.5.36-MariaDB-33.0 started; log sequence number 5333193700145
Mar 22 10:14:11 rps-ppcdb02 mysqld: 35 36 37140322 10:14:11 [Note] Plugin 'FEEDBACK' is disabled.
Mar 22 10:14:11 rps-ppcdb02 mysqld: 38 39140322 10:14:11 [Note] Server socket created on IP: '::'.
Mar 22 10:14:11 rps-ppcdb02 mysqld: 40 41 42 43 44 45140322 10:14:11 [Note] Event Scheduler: Loaded 0 events
Mar 22 10:14:11 rps-ppcdb02 mysqld: 46140322 10:14:11 [Note] /usr/sbin/mysqld: ready for connections.
Mar 22 10:14:11 rps-ppcdb02 mysqld: Version: '5.5.36-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
Mar 22 10:14:13 rps-ppcdb02 mysqld: 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
Mar 22 10:14:13 rps-ppcdb02 mysqld: InnoDB: Rolling back of trx id 49E02D63 completed
Mar 22 10:14:13 rps-ppcdb02 mysqld: 140322 10:14:13 InnoDB: Rolling back trx with id 49E02D62, 16800 rows to undo
Mar 22 10:14:13 rps-ppcdb02 mysqld:
Mar 22 10:14:13 rps-ppcdb02 mysqld: InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
Mar 22 10:14:13 rps-ppcdb02 mysqld: InnoDB: Rolling back of trx id 49E02D62 completed
Mar 22 10:14:13 rps-ppcdb02 mysqld: 140322 10:14:13 InnoDB: Rolling back trx with id 49DE716A, 2 rows to undo
Mar 22 10:14:13 rps-ppcdb02 mysqld:
Mar 22 10:14:13 rps-ppcdb02 mysqld: InnoDB: Rolling back of trx id 49DE716A completed
Mar 22 10:14:13 rps-ppcdb02 mysqld: 140322 10:14:13 InnoDB: Rollback of non-prepared transactions completed

Hi,

Do you have any idea what is causing this issue? We do not experience on the Oracle distribution of the same version number.

Thanks

Hi,

You are not getting the failure on Oracle distribution apparently because it is XtraDB-specific. This is common, both Percona XtraDB and Oracle InnoDB have their share of sporadic assertion failures, but they are usually different, so you were unlucky to hit those of XtraDB kind.

As an experiment and possibly a workaround, you could try to switch your MariaDB distribution from using XtraDB (which is default) to InnoDB plugin, for this you would need to add ignore-builtin-innodb and plugin-load=ha_innodb.so.

Please create a backup first, just in case.

Hi Jan,

This is another addition to your collection of InnoDB/XtraDB assertion failures, maybe you will be able to make sense out of it by the stack traces alone, or maybe you have already seen/solved similar ones.

Thanks for the tip Elena, I have tried that and will let you know the results.

Although the immediate issue I can see is that we have started getting these warning messages in the logs (although no crashes yet):

Mar 24 20:39:29 rps-ppcdb02 mysqld: 140324 20:39:29 [ERROR] Found index PRIMARY whose column info does not match that of MySQL.
Mar 24 20:39:29 rps-ppcdb02 mysqld: 140324 20:39:29 [ERROR] Build InnoDB index translation table for Table ./igrp_405/ppcDailyAccountAggregate#P#p2014@002d04@002d01 failed

Hi Tom,

Could you please provide SHOW CREATE TABLE for this table?

Sure, here you go:

Error for table below:

Mar 24 20:51:58 rps-ppcdb02 mysqld: 140324 20:51:58 [ERROR] Found index PRIMARY whose column info does not match that of MySQL.
Mar 24 20:51:58 rps-ppcdb02 mysqld: 140324 20:51:58 [ERROR] Build InnoDB index translation table for Table ./igrp_238/ppcDailyCampaignAggregate#P#p2014@002d04@002d01 failed

CREATE TABLE `ppcDailyCampaignAggregate` (
`metricDatehour` datetime NOT NULL,
`metricDate` date NOT NULL,
`networkCode` char(3) NOT NULL,
`accountRef` varchar(20) NOT NULL,
`campaignRef` varchar(20) NOT NULL,
`matchType` char(1) NOT NULL,
`attributionModelId` int(10) unsigned NOT NULL,
`goal` int(10) unsigned NOT NULL,
`txc` varchar(3) NOT NULL,
`goalCurrency` varchar(3) NOT NULL,
`adNetwork` enum('','Content','Google Display Network','Google Search','Search','Search Partners') NOT NULL,
`device` enum('','Computers','Mobiles','Mobile devices with full browsers','Tablets','Tablets with full browsers','Other') NOT NULL,
`clickType` enum('','Breadcrumbs','Driving direction','Email subscription','Get location details','Headline','Image Ad image click','Location expansion','Manually dialed phone calls','Mobile app extension','Mobile phone calls','Offer','Other','Phone calls','Print offer','Product plusbox offer','Sitelink') NOT NULL,
`avgPosition` decimal(4,2) NOT NULL,
`dupImpressions` tinyint(3) unsigned NOT NULL,
`impressions` int(10) unsigned NOT NULL,
`clicks` int(10) unsigned NOT NULL,
`cost` decimal(11,2) NOT NULL,
`nwConversions` int(10) unsigned NOT NULL,
`nwConversionValue` decimal(11,2) NOT NULL,
`txvTotal` decimal(15,6) NOT NULL,
`callCount` decimal(15,6) unsigned NOT NULL,
`goalCount` decimal(15,6) unsigned NOT NULL,
`goalValueTotal` decimal(15,6) NOT NULL,
`infCallGoalValue` decimal(15,6) NOT NULL,
PRIMARY KEY (`metricDate`,`attributionModelId`,`networkCode`,`accountRef`,`campaignRef`,`matchType`,`metricDatehour`,`goal`,`txc`,`goalCurrency`,`adNetwork`,`device`,`clickType`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
/*!50100 PARTITION BY RANGE (to_days(metricDatehour))
(PARTITION p0 VALUES LESS THAN (0) ENGINE = InnoDB,
PARTITION `p2011-11-01` VALUES LESS THAN (734807) ENGINE = InnoDB,
PARTITION `p2011-12-01` VALUES LESS THAN (734837) ENGINE = InnoDB,
PARTITION `p2012-01-01` VALUES LESS THAN (734868) ENGINE = InnoDB,
PARTITION `p2012-02-01` VALUES LESS THAN (734899) ENGINE = InnoDB,
PARTITION `p2012-03-01` VALUES LESS THAN (734928) ENGINE = InnoDB,
PARTITION `p2012-04-01` VALUES LESS THAN (734959) ENGINE = InnoDB,
PARTITION `p2012-05-01` VALUES LESS THAN (734989) ENGINE = InnoDB,
PARTITION `p2012-06-01` VALUES LESS THAN (735020) ENGINE = InnoDB,
PARTITION `p2012-07-01` VALUES LESS THAN (735050) ENGINE = InnoDB,
PARTITION `p2012-08-01` VALUES LESS THAN (735081) ENGINE = InnoDB,
PARTITION `p2012-09-01` VALUES LESS THAN (735112) ENGINE = InnoDB,
PARTITION `p2012-10-01` VALUES LESS THAN (735142) ENGINE = InnoDB,
PARTITION `p2012-11-01` VALUES LESS THAN (735173) ENGINE = InnoDB,
PARTITION `p2012-12-01` VALUES LESS THAN (735203) ENGINE = InnoDB,
PARTITION `p2013-01-01` VALUES LESS THAN (735234) ENGINE = InnoDB,
PARTITION `p2013-02-01` VALUES LESS THAN (735265) ENGINE = InnoDB,
PARTITION `p2013-03-01` VALUES LESS THAN (735293) ENGINE = InnoDB,
PARTITION `p2013-04-01` VALUES LESS THAN (735324) ENGINE = InnoDB,
PARTITION `p2013-05-01` VALUES LESS THAN (735354) ENGINE = InnoDB,
PARTITION `p2013-06-01` VALUES LESS THAN (735385) ENGINE = InnoDB,
PARTITION `p2013-07-01` VALUES LESS THAN (735415) ENGINE = InnoDB,
PARTITION `p2013-08-01` VALUES LESS THAN (735446) ENGINE = InnoDB,
PARTITION `p2013-09-01` VALUES LESS THAN (735477) ENGINE = InnoDB,
PARTITION `p2013-10-01` VALUES LESS THAN (735507) ENGINE = InnoDB,
PARTITION `p2013-11-01` VALUES LESS THAN (735538) ENGINE = InnoDB,
PARTITION `p2013-12-01` VALUES LESS THAN (735568) ENGINE = InnoDB,
PARTITION `p2014-01-01` VALUES LESS THAN (735599) ENGINE = InnoDB,
PARTITION `p2014-02-01` VALUES LESS THAN (735630) ENGINE = InnoDB,
PARTITION `p2014-03-01` VALUES LESS THAN (735658) ENGINE = InnoDB,
PARTITION `p2014-04-01` VALUES LESS THAN (735689) ENGINE = InnoDB)

Tom, thanks, I can reproduce the error message. It is most likely bug MDEV-4024 / MDEV-5476 which was fixed in 10.0.
It is not expected to cause data damage though.
I was able to get rid of the nasty error message by forcing table rebuild, e.g. ALTER TABLE ppcDailyCampaignAggregate ENGINE=InnoDB;
Assuming you have a backup and you can afford experimenting with the instance in question, you could try the same.

Hi Tom,

Can I see the show create table from table that uses ROW_FORMAT=COMPRESSED, based on original crash you must have at least one of those. I have seen several different crashes especially on compressed tables and XtraDB.

Thanks Jan, we do use compressed tables quite a bit.

The solution Elena proposed of switching to the Oracle Innodb plugin seems to be working well.

We've had no crashes in 2 days.

We used the following command to convert xtradb back to innodb to avoid the warnings we were getting:

mysqlcheck --optimize --all-databases

This does a re-create on innodb tables so achieves the same effect as ALTER TABLE ppcDailyCampaignAggregate ENGINE=InnoDB without all the typing

I think we'll steer clear of XtraDB for a while though, this way we get to use MariaDB with the stability of the Oracle Innodb engine.

Please reopen if reproducable with 5.5.39.