[MDEV-16302] MariaDB server crashed while dumping a big database Created: 2018-05-27  Updated: 2018-07-31  Resolved: 2018-07-31

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.0.30-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Artur Čuvašov Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

CentOS 7.4.1708



 Description   

DB size 36392.8 MB.

Running

mysql -uroot -hlocalhost ${DBNAME} -B -e 'show tables' | while read table; do echo -e "$table dumping"; mysqldump -uroot -hlocalhost --no-create-info --max_allowed_packet=512M --complete-insert --skip-triggers --lock-tables=false ${DBNAME} $table > /backup/removed/${DBNAME}/$table.sql; echo -e "$table done.\n"; done

caused a server crash:

2018-05-27 21:42:56 139791266872064 [ERROR] InnoDB: Tried to read 16384 bytes at offset 4154556416. Was only able to read 0.
2018-05-27 21:42:56 7f23b0cdfb00  InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
 InnoDB: Operation read to file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.1.30/storage/xtradb/os/os0file.cc and at line 3199
2018-05-27 21:42:56 139791266872064 [ERROR] InnoDB: File (unknown): 'read' returned OS error 205. Cannot continue operation
180527 21:42:56 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.30-MariaDB
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=9
max_threads=1026
thread_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2286668 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f05b29f5008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f23b0cdf080 thread_stack 0x48400
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f23b377bd87]
/lib64/libc.so.6(+0x10df40)[0x7f23b3779f40]
/lib64/libc.so.6(+0x10fcf7)[0x7f23b377bcf7]
/usr/sbin/mysqld(my_addr_resolve+0x48)[0x55e030260838]
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55e03024cea2]
/usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55e02fd6f925]
/lib64/libpthread.so.0(+0xf5e0)[0x7f23b53c85e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f23b36a11f7]
/lib64/libc.so.6(abort+0x148)[0x7f23b36a28e8]
/usr/sbin/mysqld(+0x872219)[0x55e030020219]
/usr/sbin/mysqld(+0x87691c)[0x55e03002491c]
/usr/sbin/mysqld(+0x87960b)[0x55e03002760b]
/usr/sbin/mysqld(+0x9bfc9b)[0x55e03016dc9b]
/usr/sbin/mysqld(+0x9808ef)[0x55e03012e8ef]
/usr/sbin/mysqld(+0x95f67b)[0x55e03010d67b]
/usr/sbin/mysqld(+0x949e11)[0x55e0300f7e11]
/usr/sbin/mysqld(+0x37f970)[0x55e02fb2d970]
/usr/sbin/mysqld(+0x8d9905)[0x55e030087905]
/usr/sbin/mysqld(+0x80ee93)[0x55e02ffbce93]
/usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x79)[0x55e02fd74169]
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x20)[0x55e02fe7acd0]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1fb)[0x55e02fc1df1b]
/usr/sbin/mysqld(+0x47efcd)[0x55e02fc2cfcd]
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xb4c)[0x55e02fc3f9dc]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x54)[0x55e02fc419e4]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x112)[0x55e02fc3e062]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x255)[0x55e02fc3eb65]
/usr/sbin/mysqld(+0x3769c6)[0x55e02fb249c6]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6e09)[0x55e02fbe96c9]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x342)[0x55e02fbec122]
/usr/sbin/mysqld(+0x43ea41)[0x55e02fbeca41]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2111)[0x55e02fbef391]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x136)[0x55e02fbf03e6]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55e02fcbc11a]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x55e02fcbc2c0]
/usr/sbin/mysqld(+0x74f57d)[0x55e02fefd57d]
/lib64/libpthread.so.0(+0x7e25)[0x7f23b53c0e25]
/lib64/libc.so.6(clone+0x6d)[0x7f23b376434d]
======= Memory map: ========
55e02f7ae000-55e03073f000 r-xp 00000000 fd:00 2952817670                 /usr/sbin/mysqld
55e03093e000-55e0309ef000 r--p 00f90000 fd:00 2952817670                 /usr/sbin/mysqld
55e0309ef000-55e030aa1000 rw-p 01041000 fd:00 2952817670                 /usr/sbin/mysqld
55e030aa1000-55e031321000 rw-p 00000000 00:00 0 
7f0551400000-7f057a000000 rw-p 00000000 00:00 0 
7f057a3f6000-7f057a3f7000 ---p 00000000 00:00 0 
7f057a3f7000-7f057abf7000 rw-p 00000000 00:00 0 
7f057abf7000-7f057abf8000 ---p 00000000 00:00 0 
7f057abf8000-7f057b3f8000 rw-p 00000000 00:00 0 
7f057b3f8000-7f057b3f9000 ---p 00000000 00:00 0 
7f057b3f9000-7f057bbf9000 rw-p 00000000 00:00 0 
7f057bbf9000-7f057bbfa000 ---p 00000000 00:00 0 
7f057bbfa000-7f057c3fa000 rw-p 00000000 00:00 0 
7f057c3fa000-7f057c3fb000 ---p 00000000 00:00 0 
7f057c3fb000-7f057cbfb000 rw-p 00000000 00:00 0 
7f057cbfb000-7f057cbfc000 ---p 00000000 00:00 0 
7f057cbfc000-7f057d3fc000 rw-p 00000000 00:00 0 
7f057d3fc000-7f057d3fd000 ---p 00000000 00:00 0 
7f057d3fd000-7f057dbfd000 rw-p 00000000 00:00 0 
7f057dbfd000-7f057dbfe000 ---p 00000000 00:00 0 
7f057dbfe000-7f057e3fe000 rw-p 00000000 00:00 0 
7f057e3fe000-7f057e3ff000 ---p 00000000 00:00 0 
7f057e3ff000-7f057ebff000 rw-p 00000000 00:00 0 
7f057ebff000-7f057ec00000 ---p 00000000 00:00 0 
7f057ec00000-7f0593c00000 rw-p 00000000 00:00 0 
7f0593fff000-7f0594000000 ---p 00000000 00:00 0 
7f0594000000-7f05e6400000 rw-p 00000000 00:00 0 
7f05e67ea000-7f05e67eb000 ---p 00000000 00:00 0 
7f05e67eb000-7f05e6feb000 rw-p 00000000 00:00 0 
7f05e6feb000-7f05e6fec000 ---p 00000000 00:00 0 
7f05e6fec000-7f05e77ec000 rw-p 00000000 00:00 0 
7f05e77ec000-7f05e77ed000 ---p 00000000 00:00 0 
7f05e77ed000-7f05e7fed000 rw-p 00000000 00:00 0 
7f05e7fed000-7f05e7fee000 ---p 00000000 00:00 0 
7f05e7fee000-7f05e87ee000 rw-p 00000000 00:00 0 
7f05e87ee000-7f05e87ef000 ---p 00000000 00:00 0 
7f05e87ef000-7f05e8fef000 rw-p 00000000 00:00 0 
7f05e8fef000-7f05e8ff0000 ---p 00000000 00:00 0 
7f05e8ff0000-7f05e97f0000 rw-p 00000000 00:00 0 
7f05e97f0000-7f05e97f1000 ---p 00000000 00:00 0 
7f05e97f1000-7f05e9ff1000 rw-p 00000000 00:00 0 
7f05e9ff1000-7f05e9ff2000 ---p 00000000 00:00 0 
7f05e9ff2000-7f05ea7f2000 rw-p 00000000 00:00 0 
7f05ea7f2000-7f05ea7f3000 ---p 00000000 00:00 0 
7f05ea7f3000-7f05eaff3000 rw-p 00000000 00:00 0 
7f05eaff3000-7f05eaff4000 ---p 00000000 00:00 0 
7f05eaff4000-7f05eb7f4000 rw-p 00000000 00:00 0 
7f05eb7f4000-7f05eb7f5000 ---p 00000000 00:00 0 
7f05eb7f5000-7f05ebff5000 rw-p 00000000 00:00 0 
7f05ebff5000-7f05ebff6000 ---p 00000000 00:00 0 
7f05ebff6000-7f05ec7f6000 rw-p 00000000 00:00 0 
7f05ec7f6000-7f05ec7f7000 ---p 00000000 00:00 0 
7f05ec7f7000-7f05ecff7000 rw-p 00000000 00:00 0 
7f05ecff7000-7f05ecff8000 ---p 00000000 00:00 0 
7f05ecff8000-7f05ed7f8000 rw-p 00000000 00:00 0 
7f05ed7f8000-7f05ed7f9000 ---p 00000000 00:00 0 
7f05ed7f9000-7f05edff9000 rw-p 00000000 00:00 0 
7f05edff9000-7f05edffa000 ---p 00000000 00:00 0 
7f05edffa000-7f05ee7fa000 rw-p 00000000 00:00 0 
7f05ee7fa000-7f05ee7fb000 ---p 00000000 00:00 0 
7f05ee7fb000-7f05eeffb000 rw-p 00000000 00:00 0 
7f05eeffb000-7f05eeffc000 ---p 00000000 00:00 0 
7f05eeffc000-7f05ef7fc000 rw-p 00000000 00:00 0 
7f05ef7fc000-7f05ef7fd000 ---p 00000000 00:00 0 
7f05ef7fd000-7f05efffd000 rw-p 00000000 00:00 0 
7f05efffd000-7f05efffe000 ---p 00000000 00:00 0 
7f05efffe000-7f05f07fe000 rw-p 00000000 00:00 0 
7f05f07fe000-7f05f07ff000 ---p 00000000 00:00 0 
7f05f07ff000-7f05f0fff000 rw-p 00000000 00:00 0 
7f05f0fff000-7f05f1000000 ---p 00000000 00:00 0 
7f05f1000000-7f0679800000 rw-p 00000000 00:00 0 
7f0679b00000-7f0919800000 rw-p 00000000 00:00 0 
7f0919b00000-7f0bb9800000 rw-p 00000000 00:00 0 
7f0bb9b00000-7f0e59800000 rw-p 00000000 00:00 0 
7f0e59b00000-7f10f9800000 rw-p 00000000 00:00 0 
7f10f9b00000-7f1399800000 rw-p 00000000 00:00 0 
7f1399b00000-7f1639800000 rw-p 00000000 00:00 0 
7f1639b00000-7f18d9c00000 rw-p 00000000 00:00 0 
7f18d9f00000-7f1b83400000 rw-p 00000000 00:00 0 
7f1b837ff000-7f1b83800000 ---p 00000000 00:00 0 
7f1b83800000-7f1b84800000 rw-p 00000000 00:00 0 
7f1b84bff000-7f1b84c00000 ---p 00000000 00:00 0 
7f1b84c00000-7f1b86400000 rw-p 00000000 00:00 0 
7f1b867fe000-7f1b867ff000 ---p 00000000 00:00 0 
7f1b867ff000-7f1b86fff000 rw-p 00000000 00:00 0 
7f1b86fff000-7f1b87000000 ---p 00000000 00:00 0 
7f1b87000000-7f1b89c00000 rw-p 00000000 00:00 0 
7f1b89f98000-7f1b89f99000 ---p 00000000 00:00 0 
7f1b89f99000-7f1b8a799000 rw-p 00000000 00:00 0 
7f1b8a799000-7f238a79a000 rw-s 00000000 fd:00 347967580                  /var/lib/mysql/galera.cache
7f238a79a000-7f238a9f3000 r-xp 00000000 fd:00 2684401710                 /usr/lib64/galera/libgalera_smm.so
7f238a9f3000-7f238abf3000 ---p 00259000 fd:00 2684401710                 /usr/lib64/galera/libgalera_smm.so
7f238abf3000-7f238abfa000 r--p 00259000 fd:00 2684401710                 /usr/lib64/galera/libgalera_smm.so
7f238abfa000-7f238abfe000 rw-p 00260000 fd:00 2684401710                 /usr/lib64/galera/libgalera_smm.so
7f238abfe000-7f238b000000 rw-p 00000000 00:00 0 
7f238b0c4000-7f238b0c5000 ---p 00000000 00:00 0 
7f238b0c5000-7f238b10e000 rw-p 00000000 00:00 0 
7f238b10e000-7f238b10f000 ---p 00000000 00:00 0 
7f238b10f000-7f238b158000 rw-p 00000000 00:00 0 
7f238b158000-7f238b159000 ---p 00000000 00:00 0 
7f238b159000-7f238b1a2000 rw-p 00000000 00:00 0 
7f238b1a2000-7f238b1a3000 ---p 00000000 00:00 0 
7f238b1a3000-7f238b1ec000 rw-p 00000000 00:00 0 
7f238b1ec000-7f238b1ed000 ---p 00000000 00:00 0 
7f238b1ed000-7f238b9ed000 rw-p 00000000 00:00 0 
7f238b9ed000-7f238b9f9000 r-xp 00000000 fd:00 4030028849                 /usr/lib64/libnss_files-2.17.so
7f238b9f9000-7f238bbf8000 ---p 0000c000 fd:00 4030028849                 /usr/lib64/libnss_files-2.17.so
7f238bbf8000-7f238bbf9000 r--p 0000b000 fd:00 4030028849                 /usr/lib64/libnss_files-2.17.so
7f238bbf9000-7f238bbfa000 rw-p 0000c000 fd:00 4030028849                 /usr/lib64/libnss_files-2.17.so
7f238bbfa000-7f23b0c00000 rw-p 00000000 00:00 0 
7f23b0c03000-7f23b0c04000 ---p 00000000 00:00 0 
7f23b0c04000-7f23b0c4d000 rw-p 00000000 00:00 0 
7f23b0c4d000-7f23b0c4e000 ---p 00000000 00:00 0 
7f23b0c4e000-7f23b0c97000 rw-p 00000000 00:00 0 
7f23b0c97000-7f23b0c98000 ---p 00000000 00:00 0 
7f23b0c98000-7f23b0ce1000 rw-p 00000000 00:00 0 
7f23b0ce1000-7f23b0ce2000 ---p 00000000 00:00 0 
7f23b0ce2000-7f23b0d2b000 rw-p 00000000 00:00 0 
7f23b0d2b000-7f23b0d3a000 r-xp 00000000 fd:00 4026534945                 /usr/lib64/libbz2.so.1.0.6
7f23b0d3a000-7f23b0f39000 ---p 0000f000 fd:00 4026534945                 /usr/lib64/libbz2.so.1.0.6
7f23b0f39000-7f23b0f3a000 r--p 0000e000 fd:00 4026534945                 /usr/lib64/libbz2.so.1.0.6
7f23b0f3a000-7f23b0f3b000 rw-p 0000f000 fd:00 4026534945                 /usr/lib64/libbz2.so.1.0.6
7f23b0f3b000-7f23b0f52000 r-xp 00000000 fd:00 4026559543                 /usr/lib64/libelf-0.168.so
7f23b0f52000-7f23b1151000 ---p 00017000 fd:00 4026559543                 /usr/lib64/libelf-0.168.so
7f23b1151000-7f23b1152000 r--p 00016000 fd:00 4026559543                 /usr/lib64/libelf-0.168.so
7f23b1152000-7f23b1153000 rw-p 00017000 fd:00 4026559543                 /usr/lib64/libelf-0.168.so
7f23b1153000-7f23b11b3000 r-xp 00000000 fd:00 4029147190                 /usr/lib64/libpcre.so.1.2.0
7f23b11b3000-7f23b13b3000 ---p 00060000 fd:00 4029147190                 /usr/lib64/libpcre.so.1.2.0
7f23b13b3000-7f23b13b4000 r--p 00060000 fd:00 4029147190                 /usr/lib64/libpcre.so.1.2.0
7f23b13b4000-7f23b13b5000 rw-p 00061000 fd:00 4029147190                 /usr/lib64/libpcre.so.1.2.0
7f23b13b5000-7f23b13b9000 r-xp 00000000 fd:00 4026534959                 /usr/lib64/libattr.so.1.1.0
7f23b13b9000-7f23b15b8000 ---p 00004000 fd:00 4026534959                 /usr/lib64/libattr.so.1.1.0
7f23b15b8000-7f23b15b9000 r--p 00003000 fd:00 4026534959                 /usr/lib64/libattr.so.1.1.0
7f23b15b9000-7f23b15ba000 rw-p 00004000 fd:00 4026534959                 /usr/lib64/libattr.so.1.1.0
7f23b15ba000-7f23b15bd000 r-xp 00000000 fd:00 4026539011                 /usr/lib64/libkeyutils.so.1.5
7f23b15bd000-7f23b17bc000 ---p 00003000 fd:00 4026539011                 /usr/lib64/libkeyutils.so.1.5
7f23b17bc000-7f23b17bd000 r--p 00002000 fd:00 4026539011                 /usr/lib64/libkeyutils.so.1.5
7f23b17bd000-7f23b17be000 rw-p 00003000 fd:00 4026539011                 /usr/lib64/libkeyutils.so.1.5
7f23b17be000-7f23b17cb000 r-xp 00000000 fd:00 4026704902                 /usr/lib64/libkrb5support.so.0.1
7f23b17cb000-7f23b19ca000 ---p 0000d000 fd:00 4026704902                 /usr/lib64/libkrb5support.so.0.1
7f23b19ca000-7f23b19cb000 r--p 0000c000 fd:00 4026704902                 /usr/lib64/libkrb5support.so.0.1
7f23b19cb000-7f23b19cc000 rw-p 0000d000 fd:00 4026704902                 /usr/lib64/libkrb5support.so.0.1
7f23b19cc000-7f23b19e1000 r-xp 00000000 fd:00 4028735515                 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f23b19e1000-7f23b1be0000 ---p 00015000 fd:00 4028735515                 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f23b1be0000-7f23b1be1000 r--p 00014000 fd:00 4028735515                 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f23b1be1000-7f23b1be2000 rw-p 00015000 fd:00 4028735515                 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f23b1be2000-7f23b1c26000 r-xp 00000000 fd:00 4026613774                 /usr/lib64/libdw-0.168.so
7f23b1c26000-7f23b1e26000 ---p 00044000 fd:00 4026613774                 /usr/lib64/libdw-0.168.so
7f23b1e26000-7f23b1e28000 r--p 00044000 fd:00 4026613774                 /usr/lib64/libdw-0.168.so
7f23b1e28000-7f23b1e29000 rw-p 00046000 fd:00 4026613774                 /usr/lib64/libdw-0.168.so
7f23b1e29000-7f23b1e3f000 r-xp 00000000 fd:00 4030041102                 /usr/lib64/libresolv-2.17.so
7f23b1e3f000-7f23b203f000 ---p 00016000 fd:00 4030041102                 /usr/lib64/libresolv-2.17.so
7f23b203f000-7f23b2040000 r--p 00016000 fd:00 4030041102                 /usr/lib64/libresolv-2.17.so
7f23b2040000-7f23b2041000 rw-p 00017000 fd:00 4030041102                 /usr/lib64/libresolv-2.17.so
7f23b2041000-7f23b2043000 rw-p 00000000 00:00 0 
7f23b2043000-7f23b2047000 r-xp 00000000 fd:00 4026534955                 /usr/lib64/libgpg-error.so.0.10.0
7f23b2047000-7f23b2246000 ---p 00004000 fd:00 4026534955                 /usr/lib64/libgpg-error.so.0.10.0
7f23b2246000-7f23b2247000 r--p 00003000 fd:00 4026534955                 /usr/lib64/libgpg-error.so.0.10.0
7f23b2247000-7f23b2248000 rw-p 00004000 fd:00 4026534955                 /usr/lib64/libgpg-error.so.0.10.0
7f23b2248000-7f23b22c5000 r-xp 00000000 fd:00 4026559547                 /usr/lib64/libgcrypt.so.11.8.2
7f23b22c5000-7f23b24c4000 ---p 0007d000 fd:00 4026559547                 /usr/lib64/libgcrypt.so.11.8.2
7f23b24c4000-7f23b24c5000 r--p 0007c000 fd:00 4026559547                 /usr/lib64/libgcrypt.so.11.8.2
7f23b24c5000-7f23b24c8000 rw-p 0007d000 fd:00 4026559547                 /usr/lib64/libgcrypt.so.11.8.2
7f23b24c8000-7f23b24c9000 rw-p 00000000 00:00 0 
7f23b24c9000-7f23b24ed000 r-xp 00000000 fd:00 4026539023                 /usr/lib64/libselinux.so.1
7f23b24ed000-7f23b26ec000 ---p 00024000 fd:00 4026539023                 /usr/lib64/libselinux.so.1
7f23b26ec000-7f23b26ed000 r--p 00023000 fd:00 4026539023                 /usr/lib64/libselinux.so.1
7f23b26ed000-7f23b26ee000 rw-p 00024000 fd:00 4026539023                 /usr/lib64/libselinux.so.1
7f23b26ee000-7f23b26f0000 rw-p 00000000 00:00 0 
7f23b26f0000-7f23b26f7000 r-xp 00000000 fd:00 4030041104                 /usr/lib64/librt-2.17.so
7f23b26f7000-7f23b28f6000 ---p 00007000 fd:00 4030041104                 /usr/lib64/librt-2.17.so
7f23b28f6000-7f23b28f7000 r--p 00006000 fd:00 4030041104                 /usr/lib64/librt-2.17.so
7f23b28f7000-7f23b28f8000 rw-p 00007000 fd:00 4030041104                 /usr/lib64/librt-2.17.so
7f23b28f8000-7f23b28fc000 r-xp 00000000 fd:00 4026559531                 /usr/lib64/libcap.so.2.22
7f23b28fc000-7f23b2afb000 ---p 00004000 fd:00 4026559531                 /usr/lib64/libcap.so.2.22
7f23b2afb000-7f23b2afc000 r--p 00003000 fd:00 4026559531                 /usr/lib64/libcap.so.2.22
7f23b2afc000-7f23b2afd000 rw-p 00004000 fd:00 4026559531                 /usr/lib64/libcap.so.2.22
7f23b2afd000-7f23b2b2e000 r-xp 00000000 fd:00 4026554408                 /usr/lib64/libk5crypto.so.3.1
7f23b2b2e000-7f23b2d2d000 ---p 00031000 fd:00 4026554408                 /usr/lib64/libk5crypto.so.3.1
7f23b2d2d000-7f23b2d2f000 r--p 00030000 fd:00 4026554408                 /usr/lib64/libk5crypto.so.3.1
7f23b2d2f000-7f23b2d30000 rw-p 00032000 fd:00 4026554408                 /usr/lib64/libk5crypto.so.3.1
7f23b2d30000-7f23b2d33000 r-xp 00000000 fd:00 4026535941                 /usr/lib64/libcom_err.so.2.1
7f23b2d33000-7f23b2f32000 ---p 00003000 fd:00 4026535941                 /usr/lib64/libcom_err.so.2.1
7f23b2f32000-7f23b2f33000 r--p 00002000 fd:00 4026535941                 /usr/lib64/libcom_err.so.2.1
7f23b2f33000-7f23b2f34000 rw-p 00003000 fd:00 4026535941                 /usr/lib64/libcom_err.so.2.1
7f23b2f34000-7f23b300c000 r-xp 00000000 fd:00 4026565675                 /usr/lib64/libkrb5.so.3.3
7f23b300c000-7f23b320b000 ---p 000d8000 fd:00 4026565675                 /usr/lib64/libkrb5.so.3.3
7f23b320b000-7f23b3219000 r--p 000d7000 fd:00 4026565675                 /usr/lib64/libkrb5.so.3.3
7f23b3219000-7f23b321c000 rw-p 000e5000 fd:00 4026565675                 /usr/lib64/libkrb5.so.3.3
7f23b321c000-7f23b3266000 r-xp 00000000 fd:00 4026554369                 /usr/lib64/libgssapi_krb5.so.2.2
7f23b3266000-7f23b3466000 ---p 0004a000 fd:00 4026554369                 /usr/lib64/libgssapi_krb5.so.2.2
7f23b3466000-7f23b3467000 r--p 0004a000 fd:00 4026554369                 /usr/lib64/libgssapi_krb5.so.2.2
7f23b3467000-7f23b3469000 rw-p 0004b000 fd:00 4026554369                 /usr/lib64/libgssapi_krb5.so.2.2
7f23b3469000-7f23b346b000 r-xp 00000000 fd:00 4026533936                 /usr/lib64/libfreebl3.so
7f23b346b000-7f23b366a000 ---p 00002000 fd:00 4026533936                 /usr/lib64/libfreebl3.so
7f23b366a000-7f23b366b000 r--p 00001000 fd:00 4026533936                 /usr/lib64/libfreebl3.so
7f23b366b000-7f23b366c000 rw-p 00002000 fd:00 4026533936                 /usr/lib64/libfreebl3.so
7f23b366c000-7f23b3824000 r-xp 00000000 fd:00 4026819817                 /usr/lib64/libc-2.17.so
7f23b3824000-7f23b3a24000 ---p 001b8000 fd:00 4026819817                 /usr/lib64/libc-2.17.so
7f23b3a24000-7f23b3a28000 r--p 001b8000 fd:00 4026819817                 /usr/lib64/libc-2.17.so
7f23b3a28000-7f23b3a2a000 rw-p 001bc000 fd:00 4026819817                 /usr/lib64/libc-2.17.so
7f23b3a2a000-7f23b3a2f000 rw-p 00000000 00:00 0 
7f23b3a2f000-7f23b3b30000 r-xp 00000000 fd:00 4030028833                 /usr/lib64/libm-2.17.so
7f23b3b30000-7f23b3d2f000 ---p 00101000 fd:00 4030028833                 /usr/lib64/libm-2.17.so
7f23b3d2f000-7f23b3d30000 r--p 00100000 fd:00 4030028833                 /usr/lib64/libm-2.17.so
7f23b3d30000-7f23b3d31000 rw-p 00101000 fd:00 4030028833                 /usr/lib64/libm-2.17.so
7f23b3d31000-7f23b3e1a000 r-xp 00000000 fd:00 4030041113                 /usr/lib64/libstdc++.so.6.0.19
7f23b3e1a000-7f23b401a000 ---p 000e9000 fd:00 4030041113                 /usr/lib64/libstdc++.so.6.0.19
7f23b401a000-7f23b4022000 r--p 000e9000 fd:00 4030041113                 /usr/lib64/libstdc++.so.6.0.19
7f23b4022000-7f23b4024000 rw-p 000f1000 fd:00 4030041113                 /usr/lib64/libstdc++.so.6.0.19
7f23b4024000-7f23b4039000 rw-p 00000000 00:00 0 
7f23b4039000-7f23b403b000 r-xp 00000000 fd:00 4030028828                 /usr/lib64/libdl-2.17.so
7f23b403b000-7f23b423b000 ---p 00002000 fd:00 4030028828                 /usr/lib64/libdl-2.17.so
7f23b423b000-7f23b423c000 r--p 00002000 fd:00 4030028828                 /usr/lib64/libdl-2.17.so
7f23b423c000-7f23b423d000 rw-p 00003000 fd:00 4030028828                 /usr/lib64/libdl-2.17.so
7f23b423d000-7f23b4471000 r-xp 00000000 fd:00 4026538026                 /usr/lib64/libcrypto.so.1.0.2k
7f23b4471000-7f23b4671000 ---p 00234000 fd:00 4026538026                 /usr/lib64/libcrypto.so.1.0.2k
7f23b4671000-7f23b468d000 r--p 00234000 fd:00 4026538026                 /usr/lib64/libcrypto.so.1.0.2k
7f23b468d000-7f23b469a000 rw-p 00250000 fd:00 4026538026                 /usr/lib64/libcrypto.so.1.0.2k
7f23b469a000-7f23b469e000 rw-p 00000000 00:00 0 
7f23b469e000-7f23b4705000 r-xp 00000000 fd:00 4026539036                 /usr/lib64/libssl.so.1.0.2k
7f23b4705000-7f23b4905000 ---p 00067000 fd:00 4026539036                 /usr/lib64/libssl.so.1.0.2k
7f23b4905000-7f23b4909000 r--p 00067000 fd:00 4026539036                 /usr/lib64/libssl.so.1.0.2k
7f23b4909000-7f23b4910000 rw-p 0006b000 fd:00 4026539036                 /usr/lib64/libssl.so.1.0.2k
7f23b4910000-7f23b4918000 r-xp 00000000 fd:00 4026819821                 /usr/lib64/libcrypt-2.17.so
7f23b4918000-7f23b4b17000 ---p 00008000 fd:00 4026819821                 /usr/lib64/libcrypt-2.17.so
7f23b4b17000-7f23b4b18000 r--p 00007000 fd:00 4026819821                 /usr/lib64/libcrypt-2.17.so
7f23b4b18000-7f23b4b19000 rw-p 00008000 fd:00 4026819821                 /usr/lib64/libcrypt-2.17.so
7f23b4b19000-7f23b4b47000 rw-p 00000000 00:00 0 
7f23b4b47000-7f23b4b78000 r-xp 00000000 fd:00 4026533950                 /usr/lib64/libjemalloc.so.1
7f23b4b78000-7f23b4d77000 ---p 00031000 fd:00 4026533950                 /usr/lib64/libjemalloc.so.1
7f23b4d77000-7f23b4d79000 r--p 00030000 fd:00 4026533950                 /usr/lib64/libjemalloc.so.1
7f23b4d79000-7f23b4d7a000 rw-p 00032000 fd:00 4026533950                 /usr/lib64/libjemalloc.so.1
7f23b4d7a000-7f23b4d7b000 rw-p 00000000 00:00 0 
7f23b4d7b000-7f23b4d90000 r-xp 00000000 fd:00 4029147194                 /usr/lib64/libz.so.1.2.7
7f23b4d90000-7f23b4f8f000 ---p 00015000 fd:00 4029147194                 /usr/lib64/libz.so.1.2.7
7f23b4f8f000-7f23b4f90000 r--p 00014000 fd:00 4029147194                 /usr/lib64/libz.so.1.2.7
7f23b4f90000-7f23b4f91000 rw-p 00015000 fd:00 4029147194                 /usr/lib64/libz.so.1.2.7
7f23b4f91000-7f23b4f92000 r-xp 00000000 fd:00 4026744879                 /usr/lib64/libaio.so.1.0.1
7f23b4f92000-7f23b5191000 ---p 00001000 fd:00 4026744879                 /usr/lib64/libaio.so.1.0.1
7f23b5191000-7f23b5192000 r--p 00000000 fd:00 4026744879                 /usr/lib64/libaio.so.1.0.1
7f23b5192000-7f23b5193000 rw-p 00001000 fd:00 4026744879                 /usr/lib64/libaio.so.1.0.1
7f23b5193000-7f23b51b8000 r-xp 00000000 fd:00 4029147198                 /usr/lib64/liblzma.so.5.2.2
7f23b51b8000-7f23b53b7000 ---p 00025000 fd:00 4029147198                 /usr/lib64/liblzma.so.5.2.2
7f23b53b7000-7f23b53b8000 r--p 00024000 fd:00 4029147198                 /usr/lib64/liblzma.so.5.2.2
7f23b53b8000-7f23b53b9000 rw-p 00025000 fd:00 4029147198                 /usr/lib64/liblzma.so.5.2.2
7f23b53b9000-7f23b53d0000 r-xp 00000000 fd:00 4030041096                 /usr/lib64/libpthread-2.17.so
7f23b53d0000-7f23b55cf000 ---p 00017000 fd:00 4030041096                 /usr/lib64/libpthread-2.17.so
7f23b55cf000-7f23b55d0000 r--p 00016000 fd:00 4030041096                 /usr/lib64/libpthread-2.17.so
7f23b55d0000-7f23b55d1000 rw-p 00017000 fd:00 4030041096                 /usr/lib64/libpthread-2.17.so
7f23b55d1000-7f23b55d5000 rw-p 00000000 00:00 0 
7f23b55d5000-7f23b55f6000 r-xp 00000000 fd:00 4029311046                 /usr/lib64/ld-2.17.so
7f23b55fb000-7f23b55fc000 rw-p 00000000 00:00 0 
7f23b55fc000-7f23b55fd000 ---p 00000000 00:00 0 
7f23b55fd000-7f23b5646000 rw-p 00000000 00:00 0 
7f23b5646000-7f23b5647000 ---p 00000000 00:00 0 
7f23b5647000-7f23b5690000 rw-p 00000000 00:00 0 
7f23b5690000-7f23b5691000 ---p 00000000 00:00 0 
7f23b5691000-7f23b56da000 rw-p 00000000 00:00 0 
7f23b56da000-7f23b56e0000 rw-s 00000000 fd:00 347967582                  /var/lib/mysql/tc.log
7f23b56e0000-7f23b56e3000 rw-s 00000000 00:0a 10765430                   /[aio] (deleted)
7f23b56e3000-7f23b56e6000 rw-s 00000000 00:0a 10765429                   /[aio] (deleted)
7f23b56e6000-7f23b56e9000 rw-s 00000000 00:0a 10765428                   /[aio] (deleted)
7f23b56e9000-7f23b56ec000 rw-s 00000000 00:0a 10765427                   /[aio] (deleted)
7f23b56ec000-7f23b56ef000 rw-s 00000000 00:0a 10765426                   /[aio] (deleted)
7f23b56ef000-7f23b56f2000 rw-s 00000000 00:0a 10765425                   /[aio] (deleted)
7f23b56f2000-7f23b56f5000 rw-s 00000000 00:0a 10765424                   /[aio] (deleted)
7f23b56f5000-7f23b56f8000 rw-s 00000000 00:0a 10765423                   /[aio] (deleted)
7f23b56f8000-7f23b56fb000 rw-s 00000000 00:0a 10765422                   /[aio] (deleted)
7f23b56fb000-7f23b56fe000 rw-s 00000000 00:0a 10765421                   /[aio] (deleted)
7f23b56fe000-7f23b5701000 rw-s 00000000 00:0a 10765420                   /[aio] (deleted)
7f23b5701000-7f23b5704000 rw-s 00000000 00:0a 10765419                   /[aio] (deleted)
7f23b5704000-7f23b5707000 rw-s 00000000 00:0a 10765418                   /[aio] (deleted)
7f23b5707000-7f23b570a000 rw-s 00000000 00:0a 10765417                   /[aio] (deleted)
7f23b570a000-7f23b570d000 rw-s 00000000 00:0a 10765416                   /[aio] (deleted)
7f23b570d000-7f23b5710000 rw-s 00000000 00:0a 10765415                   /[aio] (deleted)
7f23b5710000-7f23b5713000 rw-s 00000000 00:0a 10765414                   /[aio] (deleted)
7f23b5713000-7f23b5716000 rw-s 00000000 00:0a 10765413                   /[aio] (deleted)
7f23b5716000-7f23b5719000 rw-s 00000000 00:0a 10765412                   /[aio] (deleted)
7f23b5719000-7f23b571c000 rw-s 00000000 00:0a 10765411                   /[aio] (deleted)
7f23b571c000-7f23b571f000 rw-s 00000000 00:0a 10765410                   /[aio] (deleted)
7f23b571f000-7f23b5720000 ---p 00000000 00:00 0 
7f23b5720000-7f23b5769000 rw-p 00000000 00:00 0 
7f23b5769000-7f23b576a000 ---p 00000000 00:00 0 
7f23b576a000-7f23b57c3000 rw-p 00000000 00:00 0 
7f23b57c3000-7f23b57e9000 r-xp 00000000 fd:00 4026613782                 /usr/lib64/libsystemd.so.0.6.0
7f23b57e9000-7f23b57ea000 r--p 00025000 fd:00 4026613782                 /usr/lib64/libsystemd.so.0.6.0
7f23b57ea000-7f23b57eb000 rw-p 00026000 fd:00 4026613782                 /usr/lib64/libsystemd.so.0.6.0
7f23b57eb000-7f23b57ef000 rw-p 00000000 00:00 0 
7f23b57ef000-7f23b57f0000 rw-s 00000000 00:0a 10765431                   /[aio] (deleted)
7f23b57f0000-7f23b57f3000 rw-s 00000000 00:0a 10765409                   /[aio] (deleted)
7f23b57f3000-7f23b57f4000 rw-s 00000000 00:0a 10765408                   /[aio] (deleted)
7f23b57f4000-7f23b57f6000 rw-p 00000000 00:00 0 
7f23b57f6000-7f23b57f7000 r--p 00021000 fd:00 4029311046                 /usr/lib64/ld-2.17.so
7f23b57f7000-7f23b57f8000 rw-p 00022000 fd:00 4029311046                 /usr/lib64/ld-2.17.so
7f23b57f8000-7f23b57f9000 rw-p 00000000 00:00 0 
7ffce12b0000-7ffce12d1000 rw-p 00000000 00:00 0                          [stack]
7ffce13ad000-7ffce13af000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]



 Comments   
Comment by Artur Čuvašov [ 2018-05-27 ]

[mysql]
 
# CLIENT #
port                           = 3306
socket                         = /var/lib/mysql/mysql.sock
default-character-set=utf8
 
[galera]
 
# GALERA #
wsrep_on=ON
 
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
wsrep_log_conflicts=1
 
## Galera Provider Configuration
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options="cert.log_conflicts=1; gcache.size=32G"
 
## Galera Cluster Configuration
wsrep_cluster_name="galera"
#wsrep_cluster_address="gcomm://"
wsrep_cluster_address="gcomm://1.2.3.4,1.2.3.5,1.2.3.6"
#wsrep_cluster_address="gcomm://db1:3307,db2:3307"
 
## Galera Synchronization Congifuration
wsrep_sst_method=rsync
#wsrep_sst_auth=user:pass
 
## Galera Node Configuration
wsrep_node_address="1.2.3.6"
wsrep_node_name="db3"
 
[mysqld]
 
# GENERAL #
user                           = mysql
default-storage-engine         = InnoDB
socket                         = /var/lib/mysql/mysql.sock
pid-file                       = /var/lib/mysql/mysql.pid
 
performance_schema              = ON
 
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
 
# MyISAM #
key-buffer-size                = 32M
myisam-recover                 = FORCE,BACKUP
 
# SAFETY #
max-allowed-packet             = 512M
max-connect-errors             = 1000000
skip-name-resolve
sql-mode                       = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE
# ONLY_FULL_GROUP_BY
sysdate-is-now                 = 1
innodb                         = FORCE
innodb-strict-mode             = 1
log-bin-trust-function-creators = 1
 
# DATA STORAGE #
datadir                        = /var/lib/mysql/
 
# BINARY LOGGING #
skip-log-bin
#log-bin                        = /var/lib/mysql/mysql-bin
#expire-logs-days               = 14
#sync-binlog                    = 1
 
# CACHES AND LIMITS #
tmp-table-size                 = 32M
max-heap-table-size            = 32M
query-cache-type               = 0
query-cache-size               = 0
max-connections                = 1024
thread-cache-size              = 50
#open-files-limit               = 65535
open-files-limit                = 1024000
table-definition-cache         = 4096
table-open-cache               = 102400
 
# INNODB #
innodb-flush-method            = O_DIRECT
innodb-log-files-in-group      = 2
innodb-log-file-size           = 512M
innodb-log-buffer-size         = 536870912
innodb-write-io-threads        = 16
innodb-flush-log-at-trx-commit = 2
innodb-file-per-table          = 1
innodb-buffer-pool-size        = 80G
 
# LOGGING #
log-error                      = /var/log/mysql/mysql-error.log
#log-queries-not-using-indexes  = 1
#slow-query-log                 = 1
#slow-query-log-file            = /var/log/mysql/mysql-slow.log
 
 
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
#[client-server]
 
#
# include all files from the config directory
#
#!includedir /etc/my.cnf.d

Comment by Elena Stepanova [ 2018-05-28 ]

I think there was some investigation around similar error messages some time ago, maybe marko will remember or otherwise have suggestions on how to proceed.

Comment by Marko Mäkelä [ 2018-05-30 ]

Unfortunately, InnoDB is notorious for bad error messages regarding files, and for its unforgiveness to read errors. With MDEV-12266 in MariaDB 10.3 we could more easily display the file name in this case (I am not sure if we already do that there).

Based on the message, the file was shorter than expected. Usually this is a sign of corruption. Maybe an error occurred but was ignored earlier during a snapshot transfer?

Comment by Artur Čuvašov [ 2018-05-31 ]

This was executed on a working healthy node.

Comment by Marko Mäkelä [ 2018-07-02 ]

mtxd, can you please provide more information? Does it crash if you try to repeat the dump? Can you determine on which table the crash occurred? Was the table being modified at that time? Could the .ibd file have been extended around that time?

Can you try with a newer MariaDB? 10.0.35 is currently the newest in the 10.0 series. That should improve the stack traces in the crash output.

Generated at Thu Feb 08 08:27:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.