[MDEV-12814] Got stacktrace with mariabackup 10.1.23 Created: 2017-05-16  Updated: 2017-05-18  Resolved: 2017-05-18

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.1.23
Fix Version/s: 10.1.24

Type: Bug Priority: Major
Reporter: Oli Sennhauser Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None
Environment:

mariabackup, 10.1.23

Sprint: 10.1.24

 Description   

mariabackup --user=root --copy-back --datadir=/home/mysql/data/mariadb-101 --target-dir=/tmp/backup/full
mariabackup based on MariaDB server 10.1.23-MariaDB Linux (x86_64)
170516 15:25:55 [01] Copying ib_logfile0 to /home/mysql/data/mariadb-101/ib_logfile0
2017-05-16 15:25:55 7f631e8a67c0 InnoDB: Assertion failure in thread 140063690876864 in file os0sync.cc line 377
InnoDB: Failing assertion: event
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170516 15:25:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.23-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5297 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
mariabackup(my_print_stacktrace+0x2b)[0x7f631e14235b]
mariabackup(handle_fatal_signal+0x4d5)[0x7f631dca1d35]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f631d26e330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f631c07ec37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f631c082028]
mariabackup(+0x88f5f4)[0x7f631df305f4]
mariabackup(_Z24xtrabackup_io_throttlingv+0x40)[0x7f631da75c10]
mariabackup(_Z9copy_fileP7ds_ctxtPKcS2_j+0x212)[0x7f631da872f2]
mariabackup(+0x3e9fbc)[0x7f631da8afbc]
mariabackup(_Z9copy_backv+0x257)[0x7f631da8b667]
addr2line: 'mariabackup': No such file
mariabackup(main+0x8dd)[0x7f631da54c4d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f631c069f45]
mariabackup(+0x3ccc79)[0x7f631da6dc79]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Aborted (core dumped)

 Comments   
Comment by Vladislav Vaintroub [ 2017-05-16 ]

Since you got a core dump, maybe could you provide the gdb stacktrace of the dump?

Comment by Oli Sennhauser [ 2017-05-16 ]

If you tell me how to do it and where I find the dump file I can try.

Comment by Vladislav Vaintroub [ 2017-05-16 ]

"Aborted (core dumped)" message from the shell, so this should be a usual place where your Linux saves core files. maybe current directory. Maybe executable directory. I'm not sure where

Comment by Oli Sennhauser [ 2017-05-16 ]

Found a solution: http://fromdual.com/hunting-the-core

gdb create_core core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
create_core: No such file or directory.
[New LWP 3790]
Core was generated by `mariabackup --user=root --copy-back --datadir=/home/mysql/data/mariadb-101 --ta'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd88c587f07 in ?? ()
(gdb) print p
No symbol table is loaded. Use the "file" command.
(gdb) backtrace
#0 0x00007fd88c587f07 in ?? ()
#1 0x00007fd88e1aacc4 in ?? ()
#2 0x000000000000002b in ?? ()
#3 0x00007ffc00000000 in ?? ()
#4 0x0000000000000028 in ?? ()
#5 0x0000000600000000 in ?? ()
#6 0x0000000000000000 in ?? ()
(gdb)

Comment by Oli Sennhauser [ 2017-05-16 ]

Let me know Vlad, if you wish something else.

Comment by Oli Sennhauser [ 2017-05-16 ]

gdb /home/mysql/product/mariadb-10.1/bin/mariabackup core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mysql/product/mariadb-10.1/bin/mariabackup...done.
[New LWP 3790]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `mariabackup --user=root --copy-back --datadir=/home/mysql/data/mariadb-101 --ta'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd88c587f07 in kill () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) print p
No symbol "p" in current context.
(gdb) backtrace
#0 0x00007fd88c587f07 in kill () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fd88e1aacc4 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:308
#2 <signal handler called>
#3 0x00007fd88c587c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#4 0x00007fd88c58b028 in __GI_abort () at abort.c:89
#5 0x00007fd88e4395f4 in os_event_reset (event=0x0) at /home/buildbot/buildbot/build/storage/xtradb/os/os0sync.cc:377
#6 0x00007fd88df7ec10 in xtrabackup_io_throttling () at /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc:2173
#7 0x00007fd88df902f2 in datafile_read (cursor=0x7ffc75390cb0)
at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:534
#8 copy_file (datasink=0x7fd88ac00000, datasink@entry=0x7fd88b84ca98,
src_file_path=0x7ffc00000003 <error: Cannot access memory at address 0x7ffc00000003>,
src_file_path@entry=0x7ffc75391b30 "ib_logfile0", dst_file_path=dst_file_path@entry=0x7ffc75391b30 "ib_logfile0",
thread_n=thread_n@entry=1) at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:999
#9 0x00007fd88df93fbc in copy_or_move_file (src_file_path=src_file_path@entry=0x7ffc75391b30 "ib_logfile0",
dst_file_path=dst_file_path@entry=0x7ffc75391b30 "ib_logfile0",
dst_dir=dst_dir@entry=0x7fd88ef590c0 <mysql_real_data_home> "/home/mysql/data/mariadb-101", thread_n=thread_n@entry=1)
at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:1202
#10 0x00007fd88df94667 in copy_back () at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:1738
#11 0x00007fd88df5dc4d in main (argc=<optimized out>, argv=<optimized out>)
at /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc:7474
(gdb)

Comment by Oli Sennhauser [ 2017-05-16 ]

(gdb) frame 6
#6 0x00007fd88df7ec10 in xtrabackup_io_throttling () at /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc:2173
2173 /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc: No such file or directory.
(gdb) print xtrabackup_throttling
No symbol "xtrabackup_throttling" in current context.

Comment by Oli Sennhauser [ 2017-05-16 ]

gdb /home/mysql/product/mariadb-10.1/bin/mariabackup core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mysql/product/mariadb-10.1/bin/mariabackup...done.
/home/mysql/product/mariadb-10.1.23-linux-glibc_214-x86_64/bin/core: No such file or directory.
(gdb) quit
mysql@laptop4:~/product/mariadb-10.1/bin [mariadb-101, 3306]> cd /tmp/
mysql@laptop4:/tmp [mariadb-101, 3306]> gdb /home/mysql/product/mariadb-10.1/bin/mariabackup core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mysql/product/mariadb-10.1/bin/mariabackup...done.
[New LWP 3790]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `mariabackup --user=root --copy-back --datadir=/home/mysql/data/mariadb-101 --ta'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd88c587f07 in kill () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) print p
No symbol "p" in current context.
(gdb) backtrace
#0 0x00007fd88c587f07 in kill () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fd88e1aacc4 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:308
#2 <signal handler called>
#3 0x00007fd88c587c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#4 0x00007fd88c58b028 in __GI_abort () at abort.c:89
#5 0x00007fd88e4395f4 in os_event_reset (event=0x0) at /home/buildbot/buildbot/build/storage/xtradb/os/os0sync.cc:377
#6 0x00007fd88df7ec10 in xtrabackup_io_throttling () at /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc:2173
#7 0x00007fd88df902f2 in datafile_read (cursor=0x7ffc75390cb0)
at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:534
#8 copy_file (datasink=0x7fd88ac00000, datasink@entry=0x7fd88b84ca98,
src_file_path=0x7ffc00000003 <error: Cannot access memory at address 0x7ffc00000003>,
src_file_path@entry=0x7ffc75391b30 "ib_logfile0", dst_file_path=dst_file_path@entry=0x7ffc75391b30 "ib_logfile0",
thread_n=thread_n@entry=1) at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:999
#9 0x00007fd88df93fbc in copy_or_move_file (src_file_path=src_file_path@entry=0x7ffc75391b30 "ib_logfile0",
dst_file_path=dst_file_path@entry=0x7ffc75391b30 "ib_logfile0",
dst_dir=dst_dir@entry=0x7fd88ef590c0 <mysql_real_data_home> "/home/mysql/data/mariadb-101", thread_n=thread_n@entry=1)
at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:1202
#10 0x00007fd88df94667 in copy_back () at /home/buildbot/buildbot/build/extra/mariabackup/backup_copy.cc:1738
#11 0x00007fd88df5dc4d in main (argc=<optimized out>, argv=<optimized out>)
at /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc:7474
(gdb) frame 6
#6 0x00007fd88df7ec10 in xtrabackup_io_throttling () at /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc:2173
2173 /home/buildbot/buildbot/build/extra/mariabackup/xtrabackup.cc: No such file or directory.
(gdb) print xtrabackup_throttl
No symbol "xtrabackup_throttl" in current context.
(gdb) print xtrabackup_throttle
$1 = 2

Generated at Thu Feb 08 08:00:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.