[MDEV-6590] InnoDB recover ends in signal 11 Created: 2014-08-15  Updated: 2014-08-26  Resolved: 2014-08-25

Status: Closed
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.0.13
Fix Version/s: 10.0.14

Type: Bug Priority: Major
Reporter: Tristan Lins Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: innodb
Environment:

Linux exa 3.12.21-gentoo-r1 #1 SMP Mon Aug 11 23:38:40 CEST 2014 x86_64 Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz GenuineIntel GNU/Linux

LSB Version: n/a
Distributor ID: Gentoo
Description: Gentoo Base System release 2.2
Release: 2.2
Codename: n/a

Portage 2.2.8-r1 (default/linux/amd64/13.0/desktop/kde, gcc-4.8.3, glibc-2.19-r1, 3.12.21-gentoo-r1 x86_64)
=================================================================
System uname: Linux-3.12.21-gentoo-r1-x86_64-Intel-R-Core-TM-_i7-2600K_CPU@_3.40GHz-with-gentoo-2.2
KiB Mem: 16397384 total, 1930272 free
KiB Swap: 16776188 total, 16773536 free
Timestamp of tree: Fri, 15 Aug 2014 10:15:01 +0000
ld GNU ld (Gentoo 2.23.2 p1.0) 2.23.2
app-shells/bash: 4.2_p45
dev-java/java-config: 2.2.0
dev-lang/python: 2.7.7, 3.3.5-r1
dev-util/cmake: 2.8.12.2-r1
dev-util/pkgconfig: 0.28-r1
sys-apps/baselayout: 2.2
sys-apps/openrc: 0.12.4
sys-apps/sandbox: 2.6-r1
sys-devel/autoconf: 2.13, 2.69
sys-devel/automake: 1.11.6, 1.12.6, 1.13.4
sys-devel/binutils: 2.23.2
sys-devel/gcc: 4.7.3-r1, 4.8.3
sys-devel/gcc-config: 1.7.3
sys-devel/libtool: 2.4.2-r1
sys-devel/make: 3.82-r4
sys-kernel/linux-headers: 3.13 (virtual/os-headers)
sys-libs/glibc: 2.19-r1
Repositories: gentoo javer x-portage
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=corei7-avx -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=corei7-avx -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8 --ask"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="ftp://de-mirror.org/gentoo/ http://de-mirror.org/gentoo/ ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo http://gentoo.mneisen.org/ http://mirror.netcologne.de/gentoo/ ftp://mirror.netcologne.de/gentoo/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ ftp://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.halifax.rwth-aachen.de/gentoo/ ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo ftp://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/ http://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/ ftp://ftp.uni-erlangen.de/pub/mirrors/gentoo http://ftp.uni-erlangen.de/pub/mirrors/gentoo ftp://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/"
LANG="de_DE.UTF-8"
LDFLAGS="Wl,-O1 -Wl,-as-needed"
MAKEOPTS="-j9 -l8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/javer /usr/local/portage"
SYNC="rsync://rsync.de.gentoo.org/gentoo-portage"
USE="X a52 aac aacplus acl acpi aim alsa amd64 ao audiofile avahi avx bash-completion berkdb bluetooth branding bzip2 cairo cdda cddb cdr cli consolekit cracklib crypt css cups cvs cxx dbus declarative djvu dri dts dv dvb dvd dvdr emboss encode exif fam ffmpeg firefox flac fontconfig fortran ftp gd gdbm gif gimp git gmp gpm graphviz gsm gstreamer gtk gzip iconv icq icu idn imagemagick imap inotify ipv6 jabber joystick jpeg jpeg2k kde kipi lame latex lcms ldap libass libnotify libwww lm_sensors lzma lzo mad matroska mime mmx mng modules mp3 mp4 mpeg mpi msn multilib mysql mysqli ncurses networkmanager nfs nls nntp nptl nsplugin ogg openal openexr opengl openmp openssl osc oscar pam pango pcntl pcre pdf phonon plasma png policykit ppds pulseaudio qt3support qt4 quicktime rar raw rdesktop readline recode rss samba sasl scanner sdl semantic-desktop session soap sound speex spell sse sse2 ssl startup-notification static-libs subversion svg szip tcpd theora threads tiff timidity truetype udev udisks unicode upower usb v4l vaapi vcd videos vim-syntax vnc vorbis wmf wxwidgets x264 xcb xcomposite xine xinerama xml xmlrpc xmp xscreensaver xsl xv xvid yahoo zeroconf zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_ftp proxy_http rewrite setenvif speling status unique_id usertrack vhost_alias" APACHE2_MPMS="event" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc efi-32 efi-64" INPUT_DEVICES="evdev keyboard mouse wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="de" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5 php5-4 php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="dummy fbdev intel nouveau vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

Attachments: File my.cnf    

 Description   

Some days ago, I need to reset my local system (press the reset button) because it was frozen.
After reboot, mariadb won't start up, here is the output from the log file:

140815 15:09:42 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used;  This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=exa' or '--log-bin=mysqld-bin' to avoid this problem.
 
140815 15:09:42 [Note] CONNECT: Version 1.03.0002 Aug 15 2014 15:05:37
 
140815 15:09:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
140815 15:09:42 [Note] InnoDB: The InnoDB memory heap is disabled
 
140815 15:09:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
140815 15:09:42 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
140815 15:09:42 [Note] InnoDB: Using Linux native AIO
 
140815 15:09:42 [Note] InnoDB: Using CPU crc32 instructions
 
140815 15:09:42 [Note] InnoDB: Initializing buffer pool, size = 64.0M
 
140815 15:09:42 [Note] InnoDB: Completed initialization of buffer pool
 
140815 15:09:42 [Note] InnoDB: Highest supported file format is Barracuda.
 
140815 15:09:42 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3359224610
 
140815 15:09:42 [Note] InnoDB: Database was not shutdown normally!
 
140815 15:09:42 [Note] InnoDB: Starting crash recovery.
 
140815 15:09:42 [Note] InnoDB: Reading tablespace information from the .ibd files...
 
140815 15:09:42 [Note] InnoDB: Restoring possible half-written data pages 
140815 15:09:42 [Note] InnoDB: from the doublewrite buffer...
 
InnoDB: Doing recovery: scanned up to log sequence number 3359225074
 
140815 15:09:42 [Note] InnoDB: Starting an apply batch of log records to the database...
 
InnoDB: Progress in percent: 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
 
InnoDB: Last MySQL binlog file position 0 203636560, file name ./mysqld-bin.000039
 
140815 15:09:42 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.13-MariaDB-log
 
key_buffer_size=67108864
 
read_buffer_size=1048576
 
max_used_connections=0
 
max_threads=153
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 538697 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x48000
 
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0xb30f79]
 
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6f2658]
 
/lib64/libpthread.so.0(+0x11260)[0x7f3c9564a260]
 
:0(dict_get_first_path(unsigned long, char const*))[0x9d4a08]
 
:0(dict_check_tablespaces_and_store_max_id(dict_check_t))[0x9d59a4]
 
:0(innobase_start_or_create_for_mysql())[0x9533f1]
 
ha_innodb.cc:0(innobase_init(void*))[0x8ad168]
 
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x57)[0x6f4797]
 
sql_plugin.cc:0(plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool))[0x5ce8ca]
 
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x4ab)[0x5cf4ab]
 
mysqld.cc:0(init_server_components())[0x5492a7]
 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3e0)[0x54d290]
 
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3c94255db5]
 
/usr/sbin/mysqld[0x5430e5]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

The startup fails every time I try.
Moving /var/lib/mysql away and reinstall the tables works, so it seems to be a problem with the existing tables.

 Comments   
Comment by Jan Lindström (Inactive) [ 2014-08-25 ]

Hi,

Do you still have the broken database somewhere, can you download it here ? Code tries to find filepath for a spaceid from SYS_DATAFILES and checks it against the contents of a link file, but something goes wrong.

Comment by Tristan Lins [ 2014-08-25 ]

Yes I still have the broken database files.
I will send you an E-Mail with a download link later (the database contains some sensitive data). I'm still packing the archive and upload it to my server.

Comment by Jan Lindström (Inactive) [ 2014-08-25 ]

Hi,

Have you migrated database from some older version, for some reason InnoDB data dictionary is missing table SYS_DATAFILES.

Comment by Tristan Lins [ 2014-08-25 ]

Mh, yes I have transfered the database from my old laptop. If I remember correctly I have currently installed MariaDB 5.6 on it. It was already migrated from an MySQL 5.5 as I remember correctly.

Comment by Jan Lindström (Inactive) [ 2014-08-25 ]

revno: 4362
committer: Jan Lindström <jplindst@mariadb.org>
branch nick: 10.0-innodb
timestamp: Mon 2014-08-25 13:35:33 +0300
message:
MDEV-6590: InnoDB recover ends in signal 11

Analysis: When database is migrated from 5.5 or earlier and
database needs crash recovery, there is possibility that
SYS_DATAFILES system table does not exists, but
crash recovery in function dict_check_tablespaces_and_store_max_id()
assumes that SYS_DATAFILES exists.

Fix: If SYS_DATAFILES does not exists, create it before
we end up to function dict_check_tablespaces_and_store_max_id()
on crash recovery.

Comment by Tristan Lins [ 2014-08-26 ]

Yeah, my database is running, great work!

Generated at Thu Feb 08 07:13:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.