140212 16:07:20 mysqld_safe Starting mysqld daemon with databases from /base/mysql

140212 16:07:20 InnoDB: The InnoDB memory heap is disabled

140212 16:07:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins

140212 16:07:20 InnoDB: Compressed tables use zlib 1.2.3

140212 16:07:20 InnoDB: Using Linux native AIO

140212 16:07:20 InnoDB: Initializing buffer pool, size = 16.0G

140212 16:07:21 InnoDB: Completed initialization of buffer pool

140212 16:07:21 InnoDB: highest supported file format is Barracuda.

140212 16:07:23  InnoDB: Waiting for the background threads to start

140212 16:07:24 Percona XtraDB (http://www.percona.com) 5.5.35-MariaDB-33.0 started; log sequence number 20323202681282

140212 16:07:24 [Note] Plugin 'FEEDBACK' is disabled.

140212 16:07:26 [Note] Server socket created on IP: '10.10.1.26'.

InnoDB: Error: page 1575499 1581852 not found in the hash table

140213  1:24:45  InnoDB: Assertion failure in thread 1099708736 in file buf0lru.c line 2355

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

140213  1:24:45 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed, 

something is definitely wrong and this may fail.

Server version: 5.5.35-MariaDB-log

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=12

max_threads=502

thread_count=8

It is possible that mysqld could use up to 

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2260475 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x2aaef902f780

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x418c30a0 thread_stack 0x48000

(my_addr_resolve failure: fork)

/usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x2aed35b449ee]

/usr/libexec/mysqld(handle_fatal_signal+0x4d1) [0x2aed35774b01]

/lib64/libpthread.so.0 [0x2aed36c7eb10]

/lib64/libc.so.6(gsignal+0x35) [0x2aed38237265]

/lib64/libc.so.6(abort+0x110) [0x2aed38238d10]

/usr/libexec/mysqld [0x2aed35a4d055]

/usr/libexec/mysqld [0x2aed35a4f2ad]

/usr/libexec/mysqld [0x2aed35a5060d]

/usr/libexec/mysqld [0x2aed35a50b4f]

/usr/libexec/mysqld [0x2aed35a3e5df]

/usr/libexec/mysqld [0x2aed35a31e56]

/usr/libexec/mysqld [0x2aed359e4ebc]

/usr/libexec/mysqld [0x2aed359b23d7]

/usr/libexec/mysqld(rr_sequential(READ_RECORD*)+0xba) [0x2aed3588601a]

/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x137) [0x2aed35667067]

/usr/libexec/mysqld [0x2aed356674f2]

/usr/libexec/mysqld(JOIN::exec()+0xa81) [0x2aed35676611]

/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1a9) [0x2aed35678b19]

/usr/libexec/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x2b6) [0x2aed35679616]

/usr/libexec/mysqld [0x2aed35625d96]

/usr/libexec/mysqld(mysql_execute_command(THD*)+0x423f) [0x2aed3562b1cf]

/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x1cc) [0x2aed3562dbac]

/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1829) [0x2aed3562f3e9]

/usr/libexec/mysqld(do_command(THD*)+0xd2) [0x2aed3562fc82]

/usr/libexec/mysqld(do_handle_one_connection(THD*)+0x12f) [0x2aed356e559f]

/usr/libexec/mysqld(handle_one_connection+0x54) [0x2aed356e56b4]

/lib64/libpthread.so.0 [0x2aed36c7673d]

/lib64/libc.so.6(clone+0x6d) [0x2aed382dad1d]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x2aaf185facf8): SELECT /*!40001 SQL_NO_CACHE */ * FROM `received_resumes_events`

Connection ID (thread ID): 3306

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file

140213 01:25:51 mysqld_safe Number of processes running now: 0

140213 01:25:51 mysqld_safe mysqld restarted

# gdb /usr/libexec/mysqld  core.13507 

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.1)

Copyright (C) 2009 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /usr/libexec/mysqld...done.

[New Thread 16788]

[New Thread 16210]

[New Thread 16209]

[New Thread 15910]

[New Thread 15909]

[New Thread 15908]

[New Thread 15906]

[New Thread 14661]

[New Thread 14105]

[New Thread 14104]

[New Thread 14103]

[New Thread 13539]

[New Thread 13532]

[New Thread 13531]

[New Thread 13530]

[New Thread 13529]

[New Thread 13528]

[New Thread 13527]

[New Thread 13521]

[New Thread 13520]

[New Thread 13519]

[New Thread 13518]

[New Thread 13517]

[New Thread 13516]

[New Thread 13515]

[New Thread 13514]

[New Thread 13513]

[New Thread 13512]

[New Thread 13509]

[New Thread 13507]

Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

Loaded symbols for /lib64/libpthread.so.0

Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libz.so.1

Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/librt.so.1

Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypt.so.1

Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libdl.so.2

Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libssl.so.6

Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcrypto.so.6

Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libstdc++.so.6

Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libm.so.6

Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

Loaded symbols for /lib64/libc.so.6

Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2

Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libgssapi_krb5.so.2

Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libkrb5.so.3

Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libcom_err.so.2

Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libk5crypto.so.3

Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libgcc_s.so.1

Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/libkrb5support.so.0

Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libkeyutils.so.1

Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libresolv.so.2

Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libselinux.so.1

Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.

Loaded symbols for /lib64/libsepol.so.1

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_dns.so.2

Core was generated by `/usr/libexec/mysqld --basedir=/usr --datadir=/base/mysql --plugin-dir=/usr/lib6'.

Program terminated with signal 6, Aborted.

#0  0x00002aed36c7bd02 in pthread_kill () from /lib64/libpthread.so.0

(gdb) bt

#0  0x00002aed36c7bd02 in pthread_kill () from /lib64/libpthread.so.0

#1  0x00002aed35774a8b in handle_fatal_signal (sig=6) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/signal_handler.cc:262

#2  <signal handler called>

#3  0x00002aed38237265 in raise () from /lib64/libc.so.6

#4  0x00002aed38238d10 in abort () from /lib64/libc.so.6

#5  0x00002aed35a4d055 in buf_LRU_block_remove_hashed_page (bpage=0x2aaef9714d80, zip=<value optimized out>)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:2355

#6  0x00002aed35a4f2ad in buf_LRU_free_block (bpage=0x2aaef9714d80, zip=1, have_LRU_mutex=0x418bff60) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1951

#7  0x00002aed35a5060d in buf_LRU_free_from_common_LRU_list (buf_pool=0x2aed44bbbfd8, n_iterations=1048575)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1055

#8  buf_LRU_search_and_free_block (buf_pool=0x2aed44bbbfd8, n_iterations=1048575) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1104

#9  0x00002aed35a50b4f in buf_LRU_get_free_block (buf_pool=0x2aed44bbbfd8) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0lru.c:1343

#10 0x00002aed35a3e5df in buf_page_get_gen (space=1575442, zip_size=8192, offset=32585, rw_latch=1, guess=0x0, mode=10, 

    file=0x2aed35c12458 "/usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/btr/btr0pcur.c", line=429, mtr=0x418c0380)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/buf/buf0buf.c:2692

#11 0x00002aed35a31e56 in btr_block_get_func (cursor=0x2aaefc49d8e8, mtr=0x418c0380) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/btr0btr.ic:59

#12 btr_pcur_move_to_next_page (cursor=0x2aaefc49d8e8, mtr=0x418c0380) at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/btr/btr0pcur.c:427

#13 0x00002aed359e4ebc in btr_pcur_move_to_next (buf=0x2aaefc125f78 "\377\336\065\001", mode=1, prebuilt=0x2aaefc49d878, match_mode=0, direction=1)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/include/btr0pcur.ic:360

#14 row_search_for_mysql (buf=0x2aaefc125f78 "\377\336\065\001", mode=1, prebuilt=0x2aaefc49d878, match_mode=0, direction=1)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/row/row0sel.c:4887

#15 0x00002aed359b23d7 in ha_innobase::general_fetch (this=0x2aaefd894798, buf=0x2aaefc125f78 "\377\336\065\001", direction=1, match_mode=0)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/storage/xtradb/handler/ha_innodb.cc:7326

#16 0x00002aed3588601a in ha_rnd_next (info=0x2aaefd890a98) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_class.h:4313

#17 rr_sequential (info=0x2aaefd890a98) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/records.cc:467

#18 0x00002aed35667067 in sub_select (join=0x2aaf185fb600, join_tab=0x2aaefd8909e8, end_of_records=<value optimized out>)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:16808

#19 0x00002aed356674f2 in do_select (join=0x2aaf185fb600, fields=0x2aaef9033108, table=0x0, procedure=0x0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:16451

#20 0x00002aed35676611 in JOIN::exec (this=0x2aaf185fb600) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:2859

#21 0x00002aed35678b19 in mysql_select (thd=0x2aaef902f780, rref_pointer_array=0x2aaef9033260, tables=0x2aaf185faf60, wild_num=1, fields=<value optimized out>, conds=0x0, 

    og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147781376, result=0x2aaf185fb5e0, unit=0x2aaef9032918, select_lex=0x2aaef9032ff0)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:3079

#22 0x00002aed35679616 in handle_select (thd=0x2aaef902f780, lex=0x2aaef9032868, result=0x2aaf185fb5e0, setup_tables_done_option=0)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_select.cc:319

#23 0x00002aed35625d96 in execute_sqlcom_select (thd=0x2aaef902f780, all_tables=0x2aaf185faf60) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:4688

#24 0x00002aed3562b1cf in mysql_execute_command (thd=0x2aaef902f780) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:2232

#25 0x00002aed3562dbac in mysql_parse (thd=0x2aaef902f780, rawbuf=<value optimized out>, length=<value optimized out>, parser_state=0x418c2e80)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:5799

#26 0x00002aed3562f3e9 in dispatch_command (command=COM_QUERY, thd=0x2aaef902f780, packet=0x2aaef8ad39d1 "\260зинов)L", packet_length=64)

    at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:1078

#27 0x00002aed3562fc82 in do_command (thd=0x2aaef902f780) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_parse.cc:793

#28 0x00002aed356e559f in do_handle_one_connection (thd_arg=0x2aaf0ec87cd0) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_connect.cc:1266

#29 0x00002aed356e56b4 in handle_one_connection (arg=<value optimized out>) at /usr/src/redhat/BUILD/mariadb-5.5.35/sql/sql_connect.cc:1181

#30 0x00002aed36c7673d in start_thread () from /lib64/libpthread.so.0

#31 0x00002aed382dad1d in clone () from /lib64/libc.so.6

(gdb) q