Elena Stepanova added a comment - 2016-01-17 01:00 GeoffMontee , Can you describe the setup, please? Apparently this server is a slave, what does it replicate from? Better still if you can identify the actual event it breaks on – should not be difficult as it should write the binlog position into the error log after restart, – and describe involved table(s). Please also attach cnf file(s). Thanks.

Geoff Montee (Inactive) added a comment - 2016-01-19 23:03 - edited Apparently this server is a slave, what does it replicate from? The slave is replicating from a server running MySQL Enterprise 5.5 on Windows. The master was running 5.5.30 a couple months ago. I'm trying to find out from the DBA if that is still the version that the master is running. Better still if you can identify the actual event it breaks on – should not be difficult as it should write the binlog position into the error log after restart, – and describe involved table(s). The slave crash seemed to result in corruption, so the slave wouldn't even start up without setting innodb_force_recovery . The first time they tried to restart it, the slave logged errors like this: 160114 19:22:00 [Note] InnoDB: Database was not shutdown normally! 160114 19:22:00 [Note] InnoDB: Starting crash recovery. 160114 19:22:00 [Note] InnoDB: Reading tablespace information from the .ibd files... 160114 19:22:01 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace db1/tab1#P#p20151231 uses space ID: 10979234 at filepath: ./db1/tab1#P#p20151231.ibd. Cannot open tablespace db2/tab2#p#d1536300000 which uses space ID: 10979234 at filepath: ./db2/tab2/wt_run#p#d1536300000.ibd 2016-01-14 19:22:01 7f230f9b1820 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: Error: could not open single-table tablespace file ./db2/tab2#p#d1536300000.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here. After that, the DBA set innodb_force_recovery=1 , and the slave was able to start, but then the slave failed because of missing tables: 160119 12:05:51 [Warning] Partition table db2/tab3#P#D1533500000 opened after converting to lower case. The table may have been moved from a case in-sensitive file system. Please recreate table in the current file system   160119 12:05:51 [ERROR] InnoDB: Failed to find tablespace for table '"db2"."tab3" /* Partition "d1533600000" */' in the cache. Attempting to load the tablespace with space id 10864620. 160119 12:05:51 [ERROR] InnoDB: Trying to add tablespace 'db2/tab3#p#d1533600000' with id 10864620 to the tablespace memory cache, but tablespace 'db2/tab3#P#D1533600000' with id 10864620 already exists in the cache! 160119 12:05:51 [Warning] Partition table db2/tab3#P#D1533600000 opened after converting to lower case. The table may have been moved from a case in-sensitive file system. Please recreate table in the current file system   2016-01-19 12:05:51 7f463c879700 InnoDB: cannot calculate statistics for table "db2"."tab3" /* Partition "d1533600000" */ because the .ibd file is missing. For help, please refer to http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html 160119 12:05:51 [ERROR] Slave SQL: Error executing row event: 'Table 'db2.tab3' doesn't exist in engine', Internal MariaDB error code: 1932 160119 12:05:51 [Warning] Slave: Tablespace is missing for table 'db2/tab3#P#D1533600000' Error_code: 1812 160119 12:05:51 [Warning] Slave: Table 'db2.tab3' doesn't exist in engine Error_code: 1932 160119 12:05:51 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000004' position 664980700 I will ask the DBA if they can provide binlog mysql-bin.000004 from the master. Please also attach cnf file(s). The configuration files are attached. The DBA says that these configuration files are slightly old, and the master now has binlog_format set to ROW.

Elena Stepanova added a comment - 2016-01-19 23:56 While it is likely that the chain of events was "try to execute an event on tab3 => crash => corrupt the table => restart => attempt to restart replication from the guilty event => find out that the table is corrupted => abort", still, it's not necessarily the case. It would be more reliable to see from which event the slave started replication after restart/recovery – that is, to see the whole log of the server restart, or at least a bigger part of it.

Geoff Montee (Inactive) added a comment - 2016-01-20 21:59 Here's what I found out so far. Here's where the slave started after recovery: 160119 12:05:51 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000004' at position 664980700, relay log './slave-relay-bin.000002' position: 664980987 The event at this binlog position is a large transaction of inserts into the db2.tab3 table mentioned above. I also noticed that there appears to be multiple .ibd files for some of the partitions in this table: one file that is owned by root:root and that has all lower case characters in the file name, so probably came from Windows, and one file that is owned by mysql:mysql and that has a case-sensitive file name, so probably came from Linux. For partitions that have two data files, the data files are the same size. e.g.: -rwxrwxrwx 1 root root 4148166656 Jun 9 2015 tab3#p#d1515900000.ibd -rwxrwxrwx 1 root root 3653238784 Jun 10 2015 tab3#p#d1516000000.ibd -rwxrwxrwx 1 root root 3376414720 Jun 11 2015 tab3#p#d1516100000.ibd -rwxrwxrwx 1 root root 4651483136 Jun 12 2015 tab3#p#d1516200000.ibd -rwxrwxrwx 1 root root 8359247872 Dec 2 23:31 tab3#p#d1533600000.ibd -rw-rw---- 1 mysql mysql 8359247872 Dec 4 08:30 tab3#P#D1533600000.ibd -rwxrwxrwx 1 root root 6874464256 Dec 4 00:07 tab3#p#d1533700000.ibd -rw-rw---- 1 mysql mysql 6878658560 Dec 4 08:38 tab3#P#D1533700000.ibd Is MariaDB creating these duplicate data files? Judging by this error message, I think these duplicate file names may be part of the problem: 160119 12:05:51 [ERROR] InnoDB: Failed to find tablespace for table '"db2"."tab3" /* Partition "d1533600000" */' in the cache. Attempting to load the tablespace with space id 10864620. 160119 12:05:51 [ERROR] InnoDB: Trying to add tablespace 'db2/tab3#p#d1533600000' with id 10864620 to the tablespace memory cache, but tablespace 'db2/tab3#P#D1533600000' with id 10864620 already exists in the cache!

Elena Stepanova added a comment - 2016-01-26 20:56 GeoffMontee , can you get SHOW CREATE TABLE for this db2.tab3 table? I haven't been able to reproduce this particular crash yet, I got another one ( MDEV-9471 ), but it's not the same.

Geoff Montee (Inactive) added a comment - 2016-01-26 22:49 Here's the table definition: CREATE TABLE `tab3` ( `col1` int(10) unsigned NOT NULL DEFAULT '0', `col2` smallint(5) unsigned NOT NULL DEFAULT '0', `col3` smallint(5) unsigned NOT NULL DEFAULT '0', `col4` smallint(5) unsigned NOT NULL DEFAULT '0', `col5` binary(1) NOT NULL DEFAULT '\0', `col6` float DEFAULT NULL, PRIMARY KEY (`col1`,`col2`,`col3`,`col4`), KEY `tab3_index` (`col1`,`col2`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 PACK_KEYS=1 /*!50100 PARTITION BY RANGE (col1) (PARTITION P1209 VALUES LESS THAN (1210000000) ENGINE = InnoDB, PARTITION P1210 VALUES LESS THAN (1211000000) ENGINE = InnoDB, PARTITION P1211 VALUES LESS THAN (1212000000) ENGINE = InnoDB, PARTITION P1212 VALUES LESS THAN (1301000000) ENGINE = InnoDB, PARTITION P1301 VALUES LESS THAN (1302000000) ENGINE = InnoDB, PARTITION P1302 VALUES LESS THAN (1303000000) ENGINE = InnoDB, PARTITION M1303200000 VALUES LESS THAN (1305999999) ENGINE = InnoDB, PARTITION M1306000000 VALUES LESS THAN (1309099999) ENGINE = InnoDB, PARTITION M1309100000 VALUES LESS THAN (1312099999) ENGINE = InnoDB, PARTITION M1312100000 VALUES LESS THAN (1315199999) ENGINE = InnoDB, PARTITION M1315200000 VALUES LESS THAN (1318199999) ENGINE = InnoDB, PARTITION M1318200000 VALUES LESS THAN (1321299999) ENGINE = InnoDB, PARTITION M1321300000 VALUES LESS THAN (1324399999) ENGINE = InnoDB, PARTITION M1324400000 VALUES LESS THAN (1327399999) ENGINE = InnoDB, PARTITION M1327400000 VALUES LESS THAN (1330499999) ENGINE = InnoDB, PARTITION M1330500000 VALUES LESS THAN (1333499999) ENGINE = InnoDB, PARTITION M1333500000 VALUES LESS THAN (1400099999) ENGINE = InnoDB, PARTITION M1400100000 VALUES LESS THAN (1403199999) ENGINE = InnoDB, PARTITION M1403200000 VALUES LESS THAN (1405999999) ENGINE = InnoDB, PARTITION M1406000000 VALUES LESS THAN (1409099999) ENGINE = InnoDB, PARTITION M1409100000 VALUES LESS THAN (1412099999) ENGINE = InnoDB, PARTITION M1412100000 VALUES LESS THAN (1415199999) ENGINE = InnoDB, PARTITION M1415200000 VALUES LESS THAN (1418199999) ENGINE = InnoDB, PARTITION M1418200000 VALUES LESS THAN (1421299999) ENGINE = InnoDB, PARTITION M1421300000 VALUES LESS THAN (1424399999) ENGINE = InnoDB, PARTITION M1424400000 VALUES LESS THAN (1427399999) ENGINE = InnoDB, PARTITION M1427400000 VALUES LESS THAN (1430499999) ENGINE = InnoDB, PARTITION W1430500000 VALUES LESS THAN (1430699999) ENGINE = InnoDB, PARTITION W1430700000 VALUES LESS THAN (1431399999) ENGINE = InnoDB, PARTITION W1431400000 VALUES LESS THAN (1432099999) ENGINE = InnoDB, PARTITION W1432100000 VALUES LESS THAN (1432799999) ENGINE = InnoDB, PARTITION W1432800000 VALUES LESS THAN (1433499999) ENGINE = InnoDB, PARTITION W1433500000 VALUES LESS THAN (1434199999) ENGINE = InnoDB, PARTITION W1434200000 VALUES LESS THAN (1434899999) ENGINE = InnoDB, PARTITION W1434900000 VALUES LESS THAN (1435599999) ENGINE = InnoDB, PARTITION W1435600000 VALUES LESS THAN (1436299999) ENGINE = InnoDB, PARTITION W1436300000 VALUES LESS THAN (1500499999) ENGINE = InnoDB, PARTITION W1500500000 VALUES LESS THAN (1501199999) ENGINE = InnoDB, PARTITION W1501200000 VALUES LESS THAN (1501899999) ENGINE = InnoDB, PARTITION W1501900000 VALUES LESS THAN (1502599999) ENGINE = InnoDB, PARTITION W1502600000 VALUES LESS THAN (1503299999) ENGINE = InnoDB, PARTITION W1503300000 VALUES LESS THAN (1503999999) ENGINE = InnoDB, PARTITION W1504000000 VALUES LESS THAN (1504699999) ENGINE = InnoDB, PARTITION W1504700000 VALUES LESS THAN (1505399999) ENGINE = InnoDB, PARTITION W1505400000 VALUES LESS THAN (1506099999) ENGINE = InnoDB, PARTITION W1506100000 VALUES LESS THAN (1506799999) ENGINE = InnoDB, PARTITION W1506800000 VALUES LESS THAN (1507499999) ENGINE = InnoDB, PARTITION W1507500000 VALUES LESS THAN (1508199999) ENGINE = InnoDB, PARTITION W1508200000 VALUES LESS THAN (1508899999) ENGINE = InnoDB, PARTITION W1508900000 VALUES LESS THAN (1509599999) ENGINE = InnoDB, PARTITION W1509600000 VALUES LESS THAN (1510299999) ENGINE = InnoDB, PARTITION W1510300000 VALUES LESS THAN (1510999999) ENGINE = InnoDB, PARTITION W1511000000 VALUES LESS THAN (1511699999) ENGINE = InnoDB, PARTITION W1511700000 VALUES LESS THAN (1512399999) ENGINE = InnoDB, PARTITION W1512400000 VALUES LESS THAN (1513099999) ENGINE = InnoDB, PARTITION W1513100000 VALUES LESS THAN (1513799999) ENGINE = InnoDB, PARTITION W1513800000 VALUES LESS THAN (1514499999) ENGINE = InnoDB, PARTITION W1514500000 VALUES LESS THAN (1515199999) ENGINE = InnoDB, PARTITION W1515200000 VALUES LESS THAN (1515899999) ENGINE = InnoDB, PARTITION D1515900000 VALUES LESS THAN (1515999999) ENGINE = InnoDB, PARTITION D1516000000 VALUES LESS THAN (1516099999) ENGINE = InnoDB, PARTITION D1516100000 VALUES LESS THAN (1516199999) ENGINE = InnoDB, PARTITION D1516200000 VALUES LESS THAN (1516299999) ENGINE = InnoDB, PARTITION D1516300000 VALUES LESS THAN (1516399999) ENGINE = InnoDB, PARTITION D1516400000 VALUES LESS THAN (1516499999) ENGINE = InnoDB, PARTITION D1516500000 VALUES LESS THAN (1516599999) ENGINE = InnoDB, PARTITION D1516600000 VALUES LESS THAN (1516699999) ENGINE = InnoDB, PARTITION D1516700000 VALUES LESS THAN (1516799999) ENGINE = InnoDB, PARTITION D1516800000 VALUES LESS THAN (1516899999) ENGINE = InnoDB, PARTITION D1516900000 VALUES LESS THAN (1516999999) ENGINE = InnoDB, PARTITION D1517000000 VALUES LESS THAN (1517099999) ENGINE = InnoDB, PARTITION D1517100000 VALUES LESS THAN (1517199999) ENGINE = InnoDB, PARTITION D1517200000 VALUES LESS THAN (1517299999) ENGINE = InnoDB, PARTITION D1517300000 VALUES LESS THAN (1517399999) ENGINE = InnoDB, PARTITION D1517400000 VALUES LESS THAN (1517499999) ENGINE = InnoDB, PARTITION D1517500000 VALUES LESS THAN (1517599999) ENGINE = InnoDB, PARTITION D1517600000 VALUES LESS THAN (1517699999) ENGINE = InnoDB, PARTITION D1517700000 VALUES LESS THAN (1517799999) ENGINE = InnoDB, PARTITION D1517800000 VALUES LESS THAN (1517899999) ENGINE = InnoDB, PARTITION D1517900000 VALUES LESS THAN (1517999999) ENGINE = InnoDB, PARTITION D1518000000 VALUES LESS THAN (1518099999) ENGINE = InnoDB, PARTITION D1518100000 VALUES LESS THAN (1518199999) ENGINE = InnoDB, PARTITION D1518200000 VALUES LESS THAN (1518299999) ENGINE = InnoDB, PARTITION D1518300000 VALUES LESS THAN (1518399999) ENGINE = InnoDB, PARTITION D1518400000 VALUES LESS THAN (1518499999) ENGINE = InnoDB, PARTITION D1518500000 VALUES LESS THAN (1518599999) ENGINE = InnoDB, PARTITION D1518600000 VALUES LESS THAN (1518699999) ENGINE = InnoDB, PARTITION D1518700000 VALUES LESS THAN (1518799999) ENGINE = InnoDB, PARTITION D1518800000 VALUES LESS THAN (1518899999) ENGINE = InnoDB, PARTITION D1518900000 VALUES LESS THAN (1518999999) ENGINE = InnoDB, PARTITION D1519000000 VALUES LESS THAN (1519099999) ENGINE = InnoDB, PARTITION D1519100000 VALUES LESS THAN (1519199999) ENGINE = InnoDB, PARTITION D1519200000 VALUES LESS THAN (1519299999) ENGINE = InnoDB, PARTITION D1519300000 VALUES LESS THAN (1519399999) ENGINE = InnoDB, PARTITION D1519400000 VALUES LESS THAN (1519499999) ENGINE = InnoDB, PARTITION D1519500000 VALUES LESS THAN (1519599999) ENGINE = InnoDB, PARTITION D1519600000 VALUES LESS THAN (1519699999) ENGINE = InnoDB, PARTITION D1519700000 VALUES LESS THAN (1519799999) ENGINE = InnoDB, PARTITION D1519800000 VALUES LESS THAN (1519899999) ENGINE = InnoDB, PARTITION D1519900000 VALUES LESS THAN (1519999999) ENGINE = InnoDB, PARTITION D1520000000 VALUES LESS THAN (1520099999) ENGINE = InnoDB, PARTITION D1520100000 VALUES LESS THAN (1520199999) ENGINE = InnoDB, PARTITION D1520200000 VALUES LESS THAN (1520299999) ENGINE = InnoDB, PARTITION D1520300000 VALUES LESS THAN (1520399999) ENGINE = InnoDB, PARTITION D1520400000 VALUES LESS THAN (1520499999) ENGINE = InnoDB, PARTITION D1520500000 VALUES LESS THAN (1520599999) ENGINE = InnoDB, PARTITION D1520600000 VALUES LESS THAN (1520699999) ENGINE = InnoDB, PARTITION D1520700000 VALUES LESS THAN (1520799999) ENGINE = InnoDB, PARTITION D1521000000 VALUES LESS THAN (1521099999) ENGINE = InnoDB, PARTITION D1521100000 VALUES LESS THAN (1521199999) ENGINE = InnoDB, PARTITION D1521200000 VALUES LESS THAN (1521299999) ENGINE = InnoDB, PARTITION D1521300000 VALUES LESS THAN (1521399999) ENGINE = InnoDB, PARTITION D1521400000 VALUES LESS THAN (1521499999) ENGINE = InnoDB, PARTITION D1521500000 VALUES LESS THAN (1521599999) ENGINE = InnoDB, PARTITION D1521600000 VALUES LESS THAN (1521699999) ENGINE = InnoDB, PARTITION D1521700000 VALUES LESS THAN (1521799999) ENGINE = InnoDB, PARTITION D1521800000 VALUES LESS THAN (1521899999) ENGINE = InnoDB, PARTITION D1521900000 VALUES LESS THAN (1521999999) ENGINE = InnoDB, PARTITION D1522000000 VALUES LESS THAN (1522099999) ENGINE = InnoDB, PARTITION D1522100000 VALUES LESS THAN (1522199999) ENGINE = InnoDB, PARTITION D1522200000 VALUES LESS THAN (1522299999) ENGINE = InnoDB, PARTITION D1522300000 VALUES LESS THAN (1522399999) ENGINE = InnoDB, PARTITION D1522400000 VALUES LESS THAN (1522499999) ENGINE = InnoDB, PARTITION D1522500000 VALUES LESS THAN (1522599999) ENGINE = InnoDB, PARTITION D1522600000 VALUES LESS THAN (1522699999) ENGINE = InnoDB, PARTITION D1522700000 VALUES LESS THAN (1522799999) ENGINE = InnoDB, PARTITION D1522800000 VALUES LESS THAN (1522899999) ENGINE = InnoDB, PARTITION D1522900000 VALUES LESS THAN (1522999999) ENGINE = InnoDB, PARTITION D1523000000 VALUES LESS THAN (1523099999) ENGINE = InnoDB, PARTITION D1523100000 VALUES LESS THAN (1523199999) ENGINE = InnoDB, PARTITION D1523200000 VALUES LESS THAN (1523299999) ENGINE = InnoDB, PARTITION D1523300000 VALUES LESS THAN (1523399999) ENGINE = InnoDB, PARTITION D1523400000 VALUES LESS THAN (1523499999) ENGINE = InnoDB, PARTITION D1523500000 VALUES LESS THAN (1523599999) ENGINE = InnoDB, PARTITION D1523600000 VALUES LESS THAN (1523699999) ENGINE = InnoDB, PARTITION D1523700000 VALUES LESS THAN (1523799999) ENGINE = InnoDB, PARTITION D1523800000 VALUES LESS THAN (1523899999) ENGINE = InnoDB, PARTITION D1523900000 VALUES LESS THAN (1523999999) ENGINE = InnoDB, PARTITION D1524000000 VALUES LESS THAN (1524099999) ENGINE = InnoDB, PARTITION D1524100000 VALUES LESS THAN (1524199999) ENGINE = InnoDB, PARTITION D1524200000 VALUES LESS THAN (1524299999) ENGINE = InnoDB, PARTITION D1524300000 VALUES LESS THAN (1524399999) ENGINE = InnoDB, PARTITION D1524400000 VALUES LESS THAN (1524499999) ENGINE = InnoDB, PARTITION D1524500000 VALUES LESS THAN (1524599999) ENGINE = InnoDB, PARTITION D1524600000 VALUES LESS THAN (1524699999) ENGINE = InnoDB, PARTITION D1524700000 VALUES LESS THAN (1524799999) ENGINE = InnoDB, PARTITION D1524800000 VALUES LESS THAN (1524899999) ENGINE = InnoDB, PARTITION D1524900000 VALUES LESS THAN (1524999999) ENGINE = InnoDB, PARTITION D1525000000 VALUES LESS THAN (1525099999) ENGINE = InnoDB, PARTITION D1525100000 VALUES LESS THAN (1525199999) ENGINE = InnoDB, PARTITION D1525200000 VALUES LESS THAN (1525299999) ENGINE = InnoDB, PARTITION D1525300000 VALUES LESS THAN (1525399999) ENGINE = InnoDB, PARTITION D1525400000 VALUES LESS THAN (1525499999) ENGINE = InnoDB, PARTITION D1525500000 VALUES LESS THAN (1525599999) ENGINE = InnoDB, PARTITION D1525600000 VALUES LESS THAN (1525699999) ENGINE = InnoDB, PARTITION D1525700000 VALUES LESS THAN (1525799999) ENGINE = InnoDB, PARTITION D1525800000 VALUES LESS THAN (1525899999) ENGINE = InnoDB, PARTITION D1525900000 VALUES LESS THAN (1525999999) ENGINE = InnoDB, PARTITION D1526000000 VALUES LESS THAN (1526099999) ENGINE = InnoDB, PARTITION D1526100000 VALUES LESS THAN (1526199999) ENGINE = InnoDB, PARTITION D1526200000 VALUES LESS THAN (1526299999) ENGINE = InnoDB, PARTITION D1526300000 VALUES LESS THAN (1526399999) ENGINE = InnoDB, PARTITION D1526400000 VALUES LESS THAN (1526499999) ENGINE = InnoDB, PARTITION D1526500000 VALUES LESS THAN (1526599999) ENGINE = InnoDB, PARTITION D1526600000 VALUES LESS THAN (1526699999) ENGINE = InnoDB, PARTITION D1526700000 VALUES LESS THAN (1526799999) ENGINE = InnoDB, PARTITION D1526800000 VALUES LESS THAN (1526899999) ENGINE = InnoDB, PARTITION D1526900000 VALUES LESS THAN (1526999999) ENGINE = InnoDB, PARTITION D1527000000 VALUES LESS THAN (1527099999) ENGINE = InnoDB, PARTITION D1527100000 VALUES LESS THAN (1527199999) ENGINE = InnoDB, PARTITION D1527200000 VALUES LESS THAN (1527299999) ENGINE = InnoDB, PARTITION D1527300000 VALUES LESS THAN (1527399999) ENGINE = InnoDB, PARTITION D1527400000 VALUES LESS THAN (1527499999) ENGINE = InnoDB, PARTITION D1527500000 VALUES LESS THAN (1527599999) ENGINE = InnoDB, PARTITION D1527600000 VALUES LESS THAN (1527699999) ENGINE = InnoDB, PARTITION D1527700000 VALUES LESS THAN (1527799999) ENGINE = InnoDB, PARTITION D1527800000 VALUES LESS THAN (1527899999) ENGINE = InnoDB, PARTITION D1527900000 VALUES LESS THAN (1527999999) ENGINE = InnoDB, PARTITION D1528000000 VALUES LESS THAN (1528099999) ENGINE = InnoDB, PARTITION D1528100000 VALUES LESS THAN (1528199999) ENGINE = InnoDB, PARTITION D1528200000 VALUES LESS THAN (1528299999) ENGINE = InnoDB, PARTITION D1528300000 VALUES LESS THAN (1528399999) ENGINE = InnoDB, PARTITION D1528400000 VALUES LESS THAN (1528499999) ENGINE = InnoDB, PARTITION D1528500000 VALUES LESS THAN (1528599999) ENGINE = InnoDB, PARTITION D1528600000 VALUES LESS THAN (1528699999) ENGINE = InnoDB, PARTITION D1528700000 VALUES LESS THAN (1528799999) ENGINE = InnoDB, PARTITION D1528800000 VALUES LESS THAN (1528899999) ENGINE = InnoDB, PARTITION D1528900000 VALUES LESS THAN (1528999999) ENGINE = InnoDB, PARTITION D1529000000 VALUES LESS THAN (1529099999) ENGINE = InnoDB, PARTITION D1529100000 VALUES LESS THAN (1529199999) ENGINE = InnoDB, PARTITION D1529200000 VALUES LESS THAN (1529299999) ENGINE = InnoDB, PARTITION D1529300000 VALUES LESS THAN (1529399999) ENGINE = InnoDB, PARTITION D1529400000 VALUES LESS THAN (1529499999) ENGINE = InnoDB, PARTITION D1529500000 VALUES LESS THAN (1529599999) ENGINE = InnoDB, PARTITION D1529600000 VALUES LESS THAN (1529699999) ENGINE = InnoDB, PARTITION D1529700000 VALUES LESS THAN (1529799999) ENGINE = InnoDB, PARTITION D1529800000 VALUES LESS THAN (1529899999) ENGINE = InnoDB, PARTITION D1529900000 VALUES LESS THAN (1529999999) ENGINE = InnoDB, PARTITION D1530000000 VALUES LESS THAN (1530099999) ENGINE = InnoDB, PARTITION D1530100000 VALUES LESS THAN (1530199999) ENGINE = InnoDB, PARTITION D1530200000 VALUES LESS THAN (1530299999) ENGINE = InnoDB, PARTITION D1530600000 VALUES LESS THAN (1530699999) ENGINE = InnoDB, PARTITION D1530700000 VALUES LESS THAN (1530799999) ENGINE = InnoDB, PARTITION D1530800000 VALUES LESS THAN (1530899999) ENGINE = InnoDB, PARTITION D1530900000 VALUES LESS THAN (1530999999) ENGINE = InnoDB, PARTITION D1531000000 VALUES LESS THAN (1531099999) ENGINE = InnoDB, PARTITION D1531100000 VALUES LESS THAN (1531199999) ENGINE = InnoDB, PARTITION D1531200000 VALUES LESS THAN (1531299999) ENGINE = InnoDB, PARTITION D1531300000 VALUES LESS THAN (1531399999) ENGINE = InnoDB, PARTITION D1531400000 VALUES LESS THAN (1531499999) ENGINE = InnoDB, PARTITION D1531500000 VALUES LESS THAN (1531599999) ENGINE = InnoDB, PARTITION D1531600000 VALUES LESS THAN (1531699999) ENGINE = InnoDB, PARTITION D1531700000 VALUES LESS THAN (1531799999) ENGINE = InnoDB, PARTITION D1531800000 VALUES LESS THAN (1531899999) ENGINE = InnoDB, PARTITION D1531900000 VALUES LESS THAN (1531999999) ENGINE = InnoDB, PARTITION D1532000000 VALUES LESS THAN (1532099999) ENGINE = InnoDB, PARTITION D1532100000 VALUES LESS THAN (1532199999) ENGINE = InnoDB, PARTITION D1532200000 VALUES LESS THAN (1532299999) ENGINE = InnoDB, PARTITION D1532300000 VALUES LESS THAN (1532399999) ENGINE = InnoDB, PARTITION D1532400000 VALUES LESS THAN (1532499999) ENGINE = InnoDB, PARTITION D1532600000 VALUES LESS THAN (1532699999) ENGINE = InnoDB, PARTITION D1532700000 VALUES LESS THAN (1532799999) ENGINE = InnoDB, PARTITION D1532800000 VALUES LESS THAN (1532899999) ENGINE = InnoDB, PARTITION D1532900000 VALUES LESS THAN (1532999999) ENGINE = InnoDB, PARTITION D1533000000 VALUES LESS THAN (1533099999) ENGINE = InnoDB, PARTITION D1533100000 VALUES LESS THAN (1533199999) ENGINE = InnoDB, PARTITION D1533200000 VALUES LESS THAN (1533299999) ENGINE = InnoDB, PARTITION D1533300000 VALUES LESS THAN (1533399999) ENGINE = InnoDB, PARTITION D1533400000 VALUES LESS THAN (1533499999) ENGINE = InnoDB, PARTITION D1533500000 VALUES LESS THAN (1533599999) ENGINE = InnoDB, PARTITION D1533600000 VALUES LESS THAN (1533699999) ENGINE = InnoDB, PARTITION D1533700000 VALUES LESS THAN (1533799999) ENGINE = InnoDB, PARTITION D1533800000 VALUES LESS THAN (1533899999) ENGINE = InnoDB, PARTITION D1533900000 VALUES LESS THAN (1533999999) ENGINE = InnoDB, PARTITION D1534000000 VALUES LESS THAN (1534099999) ENGINE = InnoDB, PARTITION D1534100000 VALUES LESS THAN (1534199999) ENGINE = InnoDB, PARTITION D1534200000 VALUES LESS THAN (1534299999) ENGINE = InnoDB, PARTITION D1534300000 VALUES LESS THAN (1534399999) ENGINE = InnoDB, PARTITION D1534400000 VALUES LESS THAN (1534499999) ENGINE = InnoDB, PARTITION D1534500000 VALUES LESS THAN (1534599999) ENGINE = InnoDB, PARTITION D1534600000 VALUES LESS THAN (1534699999) ENGINE = InnoDB, PARTITION D1534700000 VALUES LESS THAN (1534799999) ENGINE = InnoDB, PARTITION D1534800000 VALUES LESS THAN (1534899999) ENGINE = InnoDB, PARTITION D1534900000 VALUES LESS THAN (1534999999) ENGINE = InnoDB, PARTITION D1535000000 VALUES LESS THAN (1535099999) ENGINE = InnoDB, PARTITION D1535100000 VALUES LESS THAN (1535199999) ENGINE = InnoDB, PARTITION D1535200000 VALUES LESS THAN (1535299999) ENGINE = InnoDB, PARTITION D1535300000 VALUES LESS THAN (1535399999) ENGINE = InnoDB, PARTITION D1535400000 VALUES LESS THAN (1535499999) ENGINE = InnoDB, PARTITION D1535500000 VALUES LESS THAN (1535599999) ENGINE = InnoDB, PARTITION D1535600000 VALUES LESS THAN (1535699999) ENGINE = InnoDB, PARTITION D1535700000 VALUES LESS THAN (1535799999) ENGINE = InnoDB, PARTITION D1535800000 VALUES LESS THAN (1535899999) ENGINE = InnoDB, PARTITION D1535900000 VALUES LESS THAN (1535999999) ENGINE = InnoDB, PARTITION D1536000000 VALUES LESS THAN (1536099999) ENGINE = InnoDB, PARTITION D1536100000 VALUES LESS THAN (1536199999) ENGINE = InnoDB, PARTITION D1536200000 VALUES LESS THAN (1536299999) ENGINE = InnoDB, PARTITION D1536300000 VALUES LESS THAN (1536399999) ENGINE = InnoDB, PARTITION D1536400000 VALUES LESS THAN (1536499999) ENGINE = InnoDB, PARTITION D1600300000 VALUES LESS THAN (1600399999) ENGINE = InnoDB, PARTITION D1600400000 VALUES LESS THAN (1600499999) ENGINE = InnoDB, PARTITION D1600500000 VALUES LESS THAN (1600599999) ENGINE = InnoDB, PARTITION D1600600000 VALUES LESS THAN (1600699999) ENGINE = InnoDB, PARTITION D1600700000 VALUES LESS THAN (1600799999) ENGINE = InnoDB, PARTITION D1600800000 VALUES LESS THAN (1600899999) ENGINE = InnoDB, PARTITION D1600900000 VALUES LESS THAN (1600999999) ENGINE = InnoDB, PARTITION D1601000000 VALUES LESS THAN (1601099999) ENGINE = InnoDB, PARTITION D1601100000 VALUES LESS THAN (1601199999) ENGINE = InnoDB, PARTITION D1601200000 VALUES LESS THAN (1601299999) ENGINE = InnoDB, PARTITION D1601300000 VALUES LESS THAN (1601399999) ENGINE = InnoDB, PARTITION D1601400000 VALUES LESS THAN (1601499999) ENGINE = InnoDB, PARTITION D1601500000 VALUES LESS THAN (1601599999) ENGINE = InnoDB, PARTITION D1601600000 VALUES LESS THAN (1601699999) ENGINE = InnoDB, PARTITION D1601700000 VALUES LESS THAN (1601799999) ENGINE = InnoDB, PARTITION D1601800000 VALUES LESS THAN (1601899999) ENGINE = InnoDB, PARTITION D1601900000 VALUES LESS THAN (1601999999) ENGINE = InnoDB, PARTITION D1602000000 VALUES LESS THAN (1602099999) ENGINE = InnoDB, PARTITION D1602100000 VALUES LESS THAN (1602199999) ENGINE = InnoDB, PARTITION D1602200000 VALUES LESS THAN (1602299999) ENGINE = InnoDB, PARTITION D1602300000 VALUES LESS THAN (1602399999) ENGINE = InnoDB, PARTITION D1602400000 VALUES LESS THAN (1602499999) ENGINE = InnoDB, PARTITION D1602500000 VALUES LESS THAN (1602599999) ENGINE = InnoDB, PARTITION LASTPART VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ Thanks!

Vladislav Vaintroub added a comment - 2016-02-15 16:49 Posting a little better stacktrace of the crash , from the core file #0 __pthread_kill (threadid=<value optimized out>, signo=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63 #1 0x0000000000737622 in handle_fatal_signal () #2 <signal handler called> #3 0x000000000088fe1d in ha_innobase::write_row(unsigned char*) () #4 0x00000000007408ff in handler::ha_write_row(unsigned char*) () #5 0x0000000000b64f33 in ha_partition::write_row(unsigned char*) () #6 0x00000000007408ff in handler::ha_write_row(unsigned char*) () #7 0x00000000007fadd3 in Rows_log_event::write_row(rpl_group_info*, bool) () #8 0x00000000007fb072 in Write_rows_log_event::do_exec_row(rpl_group_info*) () #9 0x00000000007ef1e1 in Rows_log_event::do_apply_event(rpl_group_info*) () #10 0x000000000056fb6d in apply_event_and_update_pos(Log_event*, THD*, rpl_group_info*, rpl_parallel_thread*) () #11 0x000000000057211f in handle_slave_sql () #12 0x000000332d007a51 in start_thread (arg=0x7f58633ff700) at pthread_create.c:301 #13 0x000000332cce893d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Vladislav Vaintroub added a comment - 2016-02-15 16:53 After looking function that crashed in disassembly(attached disassemble.txt) , I believe to have identified place that crashed This place in disassembly (crash on offset +973) 0x000000000088fe0f <+959>: mov 0x480(%rbx),%rax 0x000000000088fe16 <+966>: mov 0x110(%rax),%rax => 0x000000000088fe1d <+973>: cmpq $0x0,0x1d0(%rax) corresponds to https://github.com/MariaDB/server/blob/b63bf7368b4881789bef931f90d4719b358cf507/storage/xtradb/handler/ha_innodb.cc#L7804 > if (UNIV_UNLIKELY(share >ib_table->is_corrupt)) Here $rax(i.e share->ib_table) is 0, thus trying to access is_corrupt dereferences NULL pointer, and crashes .

Jan Lindström (Inactive) added a comment - 2016-02-15 18:41 With below commit the problem should be easier to find. commit a70b8961f4b8b4862677b60b925282ca1d3063d9 Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Mon Feb 15 18:38:15 2016 +0200 MDEV-9424 : Server crashes when slave works with partitioned tables copied from Windows to Linux Add check to avoid NULL-pointer access and added warning if share->ib_table is not what expected.