Server crash / memory corruption with MariaDB-Galera

130218 7:14:08 [Note] Slave I/O thread: connected to master 'replication@10.240.170.40:53306',replication started in log 'mysql-bin.001384' at position 796660420
130218 7:14:08 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.5.28a-MariaDB-log
key_buffer_size=32768
read_buffer_size=32768
max_used_connections=0
max_threads=102
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2511512 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x56ceb010
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x2b4098605080 thread_stack 0x48000
130218 7:14:08 InnoDB: Starting shutdown...
??:0(my_print_stacktrace)[0xab166e]
??:0(handle_fatal_signal)[0x6eb91c]
:0()[0x37e080eca0]
??:0(my_real_read(st_net*, unsigned long*))[0x50609e]
??:0(my_net_read)[0x5065ce]
??:0(cli_safe_read)[0x6c4416]
??:0(read_event(st_mysql*, Master_info*, bool*))[0x5125ee]
??:0(handle_slave_io)[0x51eb11]
??:0(pfs_spawn_thread)[0x85e608]
:0()[0x37e080683d]
:0()[0x37e00d503d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

[root@WPDB03 ~]# *** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption (fast): 0x0000000058a58cd0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x37e0072c3b]
/lib64/libc.so.6(__libc_malloc+0x6e)[0x37e0073f6e]
/usr/sbin/mysqld(my_malloc+0x42)[0xaada92]
/usr/sbin/mysqld(_Z17flush_master_infoP11Master_infobb+0x10b)[0x64269b]
/usr/sbin/mysqld(handle_slave_io+0x12f9)[0x51ebd9]
/usr/sbin/mysqld[0x85e608]
/lib64/libpthread.so.0[0x37e080683d]
/lib64/libc.so.6(clone+0x6d)[0x37e00d503d]
======= Memory map: ========
00400000-00f4c000 r-xp 00000000 08:03 11798179 /usr/sbin/mysqld
0114b000-01197000 rw-p 00b4b000 08:03 11798179 /usr/sbin/mysqld
01197000-019e0000 rw-p 01197000 00:00 0
01b96000-01c14000 rw-p 00b96000 08:03 11798179 /usr/sbin/mysqld
1cf5e000-58d51000 rw-p 1cf5e000 00:00 0 [heap]
37dfc00000-37dfc1c000 r-xp 00000000 08:03 134611002 /lib64/ld-2.5.so
37dfe1c000-37dfe1d000 r--p 0001c000 08:03 134611002 /lib64/ld-2.5.so
37dfe1d000-37dfe1e000 rw-p 0001d000 08:03 134611002 /lib64/ld-2.5.so
37e0000000-37e014f000 r-xp 00000000 08:03 134611021 /lib64/libc-2.5.so
37e014f000-37e034f000 ---p 0014f000 08:03 134611021 /lib64/libc-2.5.so
37e034f000-37e0353000 r--p 0014f000 08:03 134611021 /lib64/libc-2.5.so
37e0353000-37e0354000 rw-p 00153000 08:03 134611021 /lib64/libc-2.5.so
37e0354000-37e0359000 rw-p 37e0354000 00:00 0
37e0400000-37e0402000 r-xp 00000000 08:03 134611025 /lib64/libdl-2.5.so
37e0402000-37e0602000 ---p 00002000 08:03 134611025 /lib64/libdl-2.5.so
37e0602000-37e0603000 r--p 00002000 08:03 134611025 /lib64/libdl-2.5.so
37e0603000-37e0604000 rw-p 00003000 08:03 134611025 /lib64/libdl-2.5.so
37e0800000-37e0816000 r-xp 00000000 08:03 134611023 /lib64/libpthread-2.5.so
37e0816000-37e0a16000 ---p 00016000 08:03 134611023 /lib64/libpthread-2.5.so
37e0a16000-37e0a17000 r--p 00016000 08:03 134611023 /lib64/libpthread-2.5.so
37e0a17000-37e0a18000 rw-p 00017000 08:03 134611023 /lib64/libpthread-2.5.so
37e0a18000-37e0a1c000 rw-p 37e0a18000 00:00 0
37e0c00000-37e0c3b000 r-xp 00000000 08:03 134611028 /lib64/libsepol.so.1
37e0c3b000-37e0e3b000 ---p 0003b000 08:03 134611028 /lib64/libsepol.so.1
37e0e3b000-37e0e3c000 rw-p 0003b000 08:03 134611028 /lib64/libsepol.so.1
37e0e3c000-37e0e46000 rw-p 37e0e3c000 00:00 0
37e1000000-37e1015000 r-xp 00000000 08:03 134611029 /lib64/libselinux.so.1
37e1015000-37e1215000 ---p 00015000 08:03 134611029 /lib64/libselinux.so.1
37e1215000-37e1217000 rw-p 00015000 08:03 134611029 /lib64/libselinux.so.1
37e1217000-37e1218000 rw-p 37e1217000 00:00 0
37e1400000-37e1482000 r-xp 00000000 08:03 134611040 /lib64/libm-2.5.so
37e1482000-37e1681000 ---p 00082000 08:03 134611040 /lib64/libm-2.5.so
37e1681000-37e1682000 r--p 00081000 08:03 134611040 /lib64/libm-2.5.so
37e1682000-37e1683000 rw-p 00082000 08:03 134611040 /lib64/libm-2.5.so
37e1800000-37e1814000 r-xp 00000000 08:03 134611031 /lib64/libz.so.1.2.3
37e1814000-37e1a13000 ---p 00014000 08:03 134611031 /lib64/libz.so.1.2.3
37e1a13000-37e1a14000 rw-p 00013000 08:03 134611031 /lib64/libz.so.1.2.3
37e1c00000-37e1c07000 r-xp 00000000 08:03 134611024 /lib64/librt-2.5.so
37e1c07000-37e1e07000 ---p 00007000 08:03 134611024 /lib64/librt-2.5.so
37e1e07000-37e1e08000 r--p 00007000 08:03 134611024 /lib64/librt-2.5.so
37e1e08000-37e1e09000 rw-p 00008000 08:03 134611024 /lib64/librt-2.5.so
37e2000000-37e200d000 r-xp 00000000 08:03 134611041 /lib64/libgcc_s-4.1.2-20080825.so.1
37e200d000-37e220d000 ---p 0000d000 08:03 134611041 /lib64/libgcc_s-4.1.2-20080825.so.1
37e220d000-37e220e000 rw-p 0000d000 08:03 134611041 /lib64/libgcc_s-4.1.2-20080825.so.1
37e2400000-37e2409000 r-xp 00000000 08:03 134611034 /lib64/libcrypt-2.5.so
37e2409000-37e2608000 ---p 00009000 08:03 134611034 /lib64/libcrypt-2.5.so
37e2608000-37e2609000 r--p 00008000 08:03 134611034 /lib64/libcrypt-2.5.so
37e2609000-37e260a000 rw-p 00009000 08:03 134611034 /lib64/libcrypt-2.5.so
37e260a000-37e2638000 rw-p 37e260a000 00:00 0
37e2800000-37e28e6000 r-xp 00000000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
37e28e6000-37e2ae5000 ---p 000e6000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
37e2ae5000-37e2aeb000 r--p 000e5000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
37e2aeb000-37e2aee000 rw-p 000eb000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
37e2aee000-37e2b00000 rw-p 37e2aee000 00:00 0
37e2c00000-37e2c11000 r-xp 00000000 08:03 134611027 /lib64/libresolv-2.5.so
37e2c11000-37e2e11000 ---p 00011000 08:03 134611027 /lib64/libresolv-2.5.so
37e2e11000-37e2e12000 r--p 00011000 08:03 134611027 /lib64/libresolv-2.5.so
37e2e12000-37e2e13000 rw-p 00012000 08:03 134611027 /lib64/libresolv-2.5.so
37e2e13000-37e2e15000 rw-p 37e2e13000 00:00 0
37e3000000-37e3002000 r-xp 00000000 08:03 134611026 /lib64/libkeyutils-1.2.so
37e3002000-37e3201000 ---p 00002000 08:03 134611026 /lib64/libkeyutils-1.2.so
37e3201000-37e3202000 rw-p 00001000 08:03 134611026 /lib64/libkeyutils-1.2.so
37e3400000-37e3402000 r-xp 00000000 08:03 134611030 /lib64/libcom_err.so.2.1
37e3402000-37e3601000 ---p 00002000 08:03 134611030 /lib64/libcom_err.so.2.1
37e3601000-37e3602000 rw-p 00001000 08:03 134611030 /lib64/libcom_err.so.2.1
37e3800000-37e3940000 r-xp 00000000 08:03 134611032 /lib64/libcrypto.so.0.9.8x
37e3940000-37e3b40000 ---p 00140000 08:03 134611032 /lib64/libcrypto.so.0.9.8x
37e3b40000-37e3b62000 rw-p 00140000 08:03 134611032 /lib64/libcrypto.so.0.9.8x
37e3b62000-37e3b66000 rw-p 37e3b62000 00:00 0
37e4000000-37e4024000 r-xp 00000000 08:03 11798306 /usr/lib64/libk5crypto.so.3.1
37e4024000-37e4223000 ---p 00024000 08:03 11798306 /usr/lib64/libk5crypto.so.3.1
37e4223000-37e4225000 rw-p 00023000 08:03 11798306 /usr/lib64/libk5crypto.so.3.1
37e4400000-37e442c000 r-xp 00000000 08:03 11798308 /usr/lib64/libgssapi_krb5.so.2.2
37e442c000-37e462c000 ---p 0002c000 08:03 11798308 /usr/lib64/libgssapi_krb5.so.2.2
37e462c000-37e462e000 rw-p 0002c000 08:03 11798308 /usr/lib64/libgssapi_krb5.so.2.2
37e4800000-37e4891000 r-xp 00000000 08:03 11798307 /usr/lib64/libkrb5.so.3.3
37e4891000-37e4a91000 ---p 00091000 08:03 11798307 /usr/lib64/libkrb5.so.3.3
37e4a91000-37e4a95000 rw-p 00091000 08:03 11798307 /usr/lib64/libkrb5.so.3.3
37e4c00000-37e4c08000 r-xp 00000000 08:03 11798305 /usr/lib64/libkrb5support.so.0.1
37e4c08000-37e4e07000 ---p 00008000 08:03 11798305 /usr/lib64/libkrb5support.so.0.1
37e4e07000-37e4e08000 rw-p 00007000 08:03 11798305 /usr/lib64/libkrb5support.so.0.1
37e5800000-37e584f000 r-xp 00000000 08:03 134611033 /lib64/libssl.so.0.9.8x
37e584f000-37e5a4f000 ---p 0004f000 08:03 134611033 /lib64/libssl.so.0.9.8x
37e5a4f000-37e5a55000 rw-p 0004f000 08:03 134611033 /lib64/libssl.so.0.9.8x
2b3572849000-2b357284b000 rw-p 2b3572849000 00:00 0
2b3572853000-2b357285b000 rw-p 2b3572853000 00:00 0
2b357285b000-2b357285c000 ---p 2b357285b000 00:00 0
2b357285c000-2b357325c000 rwxp 2b357285c000 00:00 0
2b357325c000-2b358abef000 rw-p 2b357325c000 00:00 0
2b358abf6000-2b358ac00000 r-xp 00000000 08:03 134610979 /lib64/libnss_files-2.5.so
2b358ac00000-2b358adff000 ---p 0000a000 08:03 134610979 /lib64/libnss_files-2.5.so
2b358adff000-2b358ae00000 r--p 00009000 08:03 134610979 /lib64/libnss_files-2.5.so
2b358ae00000-2b358ae01000 rw-p 0000a000 08:03 134610979 /lib64/libnss_files-2.5.so
2b358ae01000-2b3ce9cdc000 rw-p 2b358ae01000 00:00 0
2b3ce9cdc000-2b3ce9cdd000 ---p 2b3ce9cdc000 00:00 0
2b3ce9cdd000-2b3cea6dd000 rwxp 2b3ce9cdd000 00:00 0
2b3cea6dd000-2b3cea6de000 ---p 2b3cea6dd000 00:00 0
2b3cea6de000-2b3ceb0de000 rwxp 2b3cea6de000 00:00 0
2b3ceb0de000-2b3ceb0df000 ---p 2b3ceb0de000 00:00 0
2b3ceb0df000-2b3cebadf000 rwxp 2b3ceb0df000 00:00 0
2b3cebadf000-2b3cebae0000 ---p 2b3cebadf000 00:00 0
2b3cebae0000-2b3cec4e0000 rwxp 2b3cebae0000 00:00 0
2b3cec4e0000-2b3cec4e1000 ---p 2b3cec4e0000 00:00 0
2b3cec4e1000-2b3cecee1000 rwxp 2b3cec4e1000 00:00 0
2b3cecee1000-2b3cecee2000 ---p 2b3cecee1000 00:00 0
2b3cecee2000-2b3ced8e2000 rwxp 2b3cecee2000 00:00 0
2b3ced8e2000-2b3ced8e3000 ---p 2b3ced8e2000 00:00 0
2b3ced8e3000-2b3cee2e3000 rwxp 2b3ced8e3000 00:00 0
2b3cee2e3000-2b3cee2e4000 ---p 2b3cee2e3000 00:00 0
2b3cee2e4000-2b3ceece4000 rwxp 2b3cee2e4000 00:00 0
2b3ceece4000-2b3ceece5000 ---p 2b3ceece4000 00:00 0
2b3ceece5000-2b3cef6e5000 rwxp 2b3ceece5000 00:00 0
2b3cef6e5000-2b3cef6e6000 ---p 2b3cef6e5000 00:00 0
2b3cef6e6000-2b3cf00e6000 rwxp 2b3cef6e6000 00:00 0
2b3cf00e6000-2b3cf00e7000 ---p 2b3cf00e6000 00:00 0
2b3cf00e7000-2b3cf0ae7000 rwxp 2b3cf00e7000 00:00 0
2b3cf0ae7000-2b3cf0ae8000 ---p 2b3cf0ae7000 00:00 0
2b3cf0ae8000-2b3cf14e8000 rwxp 2b3cf0ae8000 00:00 0
2b3cf14e8000-2b3cf14e9000 ---p 2b3cf14e8000 00:00 0
2b3cf14e9000-2b3cf1ee9000 rwxp 2b3cf14e9000 00:00 0
2b3cf1ee9000-2b3cf1eea000 ---p 2b3cf1ee9000 00:00 0
2b3cf1eea000-2b3cf28ea000 rwxp 2b3cf1eea000 00:00 0
2b3cf28ea000-2b3cf28eb000 ---p 2b3cf28ea000 00:00 0
2b3cf28eb000-2b3cf32eb000 rwxp 2b3cf28eb000 00:00 0
2b3cf32eb000-2b3cf32ec000 ---p 2b3cf32eb000 00:00 0
2b3cf32ec000-2b3cf3cec000 rwxp 2b3cf32ec000 00:00 0
2b3cf3cec000-2b3cf3ced000 ---p 2b3cf3cec000 00:00 0
2b3cf3ced000-2b3cf46ed000 rwxp 2b3cf3ced000 00:00 0
2b3cf46ed000-2b3cf46ee000 ---p 2b3cf46ed000 00:00 0
2b3cf46ee000-2b3cf4736000 rwxp 2b3cf46ee000 00:00 0
2b3cf4736000-2b3cf4737000 ---p 2b3cf4736000 00:00 0
2b3cf4737000-2b3cf477f000 rwxp 2b3cf4737000 00:00 0
2b3cf477f000-2b3cf4780000 ---p 2b3cf477f000 00:00 0
2b3cf4780000-2b3cf47c8000 rwxp 2b3cf4780000 00:00 0
2b3cf8000000-2b3cf8034000 rw-p 2b3cf8000000 00:00 0
2b3cf8034000-2b3cfc000000 ---p 2b3cf8034000 00:00 0
2b3d0c6d8000-2b3d13a5c000 rw-p 2b3d0c6d8000 00:00 0
7fff16415000-7fff1642a000 rwxp 7ffffffe9000 00:00 0 [stack]
7fff16436000-7fff16439000 r-xp 7fff16436000 00:00 0 [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]