IBM x3650 M3 Centos 6.7 X64

Version: '5.5.45-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

InnoDB: Error: trying to load index PRIMARY for table <dbname/tablename>

InnoDB: but the index tree has been freed!

150813 10:52:54  InnoDB: Assertion failure in thread 140667958499072 in file row0mysql.c line 3686

InnoDB: Failing assertion: table

/lib64/libpthread.so.0(+0xf790)[0x7fefcf70f790]

/lib64/libc.so.6(gsignal+0x35)[0x7fefce5bb625]

/lib64/libc.so.6(abort+0x175)[0x7fefce5bce05]

/usr/local/mysql/lib/plugin/ha_innodb.so(+0xd796f)[0x7fefb925096f]

/usr/local/mysql/lib/plugin/ha_innodb.so(+0xb9c3a)[0x7fefb9232c3a]

sql/handler.cc:641(dropdb_handlerton)[0x69938f]

sql/sql_plugin.cc:2344(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x584e3d]

sql/sql_db.cc:856(mysql_rm_db(THD*, char*, bool, bool))[0x558b85]

sql/sql_parse.cc:3390(mysql_execute_command(THD*))[0x579232]

sql/sql_parse.cc:5911(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x57d5dc]

sql/sql_parse.cc:1081(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x57eb68]

sql/sql_connect.cc:1269(do_handle_one_connection(THD*))[0x624774]

sql/sql_connect.cc:1187(handle_one_connection)[0x62480a]

/lib64/libpthread.so.0(+0x7a51)[0x7fefcf707a51]

/lib64/libc.so.6(clone+0x6d)[0x7fefce6719ad]

As I understand, it most likely indicates tablespace corruption (jplindst, is that right?)

bzwitt, do you happen to know how the problem started? Did you have some kind of an outage, disk problems, maybe a crash upon altering the table, anything like that?

Hi,

Since this is a test database I did 'DROP/CREATE DATABASE
arnes_20150701' several times. Recreation of the data was done with
command like 'zcat /data/mysql-arnes-dump-20150701-09:13:01.gz | mysql
-S /tmp/mysql.sock arnes_20150701'. It happened once or twice that
during the recreation of the data server went away:

1. zcat /data/mysql-arnes-dump-20150701-09:13:01.gz | mysql -S
   /tmp/mysql.sock arnes_20150701
   ERROR 2013 (HY000) at line 6745: Lost connection to MySQL server during
   query

It had helped to recreate the database in question until drop caused
database crash.

The server crashed once at the beginning of Aug. It could be power
outage but I don't know since I was on vacation. After the crash I was
able to DROP/CREATE the database in question. innodb files reside on 3TB
FC disk (EMC VNX5300 array made of NL-SAS disks).

As I remember innodb files were created with mysql-5.x (not mariadb).
I have found attached messages in error log which dates in Jun 2015 and
could be related to the problem we have.

Best regards,

Benjamin

Thank you.

The error log shows that the server was started on an old datadir (MySQL datadir, I assume), and few hours later the server got stuck dropping the table.
First question is – do you know whether mysql_upgrade was run between these two events?
Not running it does not explain the problem right away, but it might be related.

Please also attach your cnf file(s).

Unfortunately I run mysql_upgrade after I had got problems dropping the
database.

Best regards,

Benjamin

bzwitt, do you still have the database which causes the crash?
I'm afraid without the actual binary data there isn't much we can do to investigate the issue. Similar cases that I can find online suggest ibdata corruption, e.g. this one.
I suppose if we had the ibdata* files and the schema directory, jplindst could at the very least confirm that the reason is the same here, but otherwise only guessing is possible.

Unfortunately 'corrupted' ibdata* files were deleted and all databases
were restored from dumps. If the problem recurred I would keep all the
data for analysis.

Best regards,

Benjamin

Closing it as 'Incomplete' for now. Please comment to re-open the issue if you have more data.