[MDEV-8906] Cannot start with corrupted ibdata1 file Created: 2015-10-06 Updated: 2015-10-29 Resolved: 2015-10-29 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 5.5.45, 10.0.21 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | azurit | Assignee: | Jan Lindström (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | corruption, crash, innodb | ||
| Environment: |
Debian Wheezy 64bit |
||
| Attachments: |
|
| Description |
|
Our RAID controller goes crazy while replacing the faulty drive, which results into corrupted ibdata1 file. MariaDB server was unable to start and was crashing all the time. For version 5.5, i was only able to start it with 'innodb=OFF', for 10.0 (i desided to upgrade to see, if newer version can do better in this situation so i can get the data) also this helps 'innodb_force_recovery=4' (must be 4 or above). Unfortunately, either way won't help me much. I was trying this: In the logs (attaching) you can see one database, which i thought was the one corrupted and causing problems. I was trying really hard to remove it from tablespace and let MariaDB server to start and give me access to all other databases but i failed. I think it would be great to allow MariaDB to just discard some (corrupted) data and allow access to remaining data. I believe that only few databases were corrupted but i had to restore all InnoDB tables from backup and loosing lots of data. Btw, I'm using 'innodb_file_per_table = 1'. |
| Comments |
| Comment by Elena Stepanova [ 2015-10-12 ] |
|
According to the description of innodb_force_recovery, with 4 you should (only) be able to make a fresh backup, which you can later use to re-initialize your database (which is still better than using the old backup). Were you able to do it? |
| Comment by azurit [ 2015-10-12 ] |
|
No, every attempt of selecting any data results into error something like 'unknown storage engine' (don't remember the exact message, the same message as with innodb=OFF). |
| Comment by Elena Stepanova [ 2015-10-12 ] |
|
This one is different from MDEV-8787. Here the disk failure has already been confirmed, but the question is if there is any way to recover from it? |
| Comment by Jan Lindström (Inactive) [ 2015-10-14 ] |
|
Can you provide us full all the database files I will have a look what we can do to get server up and running (does not mean that all the data is recoverable). |
| Comment by azurit [ 2015-10-14 ] |
|
Thank you but the data are already useless for us, everything was restored from backup (except the lost data from one day, which are useless now). My question was different - can you do something so MariaDB server can better recover from such ibdata1 corruption so restoring from backup won't be needed in the future? As far as i know, there are NO real data in ibdata1 file, only some information about tablespaces. The real data are in ibd files and these were NOT corrupted in this case. |
| Comment by Jan Lindström (Inactive) [ 2015-10-14 ] |
|
There is real data on ibdata1 in the doublewrite buffer. When a page is written, it is first written to the doublewrite buffer + flushed. After this has completed it is written to correct position to actual table (tablespase). Lets assume you crash after a page is written to doublewrite buffer and after page is flushed. If this page is corrupted the actual tablespace page might not be corrupted but does not contain up to date data. At the crash recovery based on redo-log we see that ok we have new version of the page but it is not found from tablespace, so we try to read it from doublewrite buffer. If that is corrupted we stop. In your case, maybe, there is possibility to use innodb_force_recovery to ignore page corruption on doublewrite buffer and use the page in tablespace to get at least some of the data back but this all must be handled by the user. If you do not really need your data, ok. |
| Comment by azurit [ 2015-10-14 ] |
|
I needed them at the time i was trying to recover them. Anyway, you still didn't answer my question. I don't care about corrupted data, let's assume, i don't want to recover them. But there was LOTS of non-corrupted data which i wasn't able to access because MariaDB was crashing. Do you understand whan i mean? Why MariaDB cannot just discard corrupted data and get me access to non-corrupted data? I had to recover 500 databases from backup because (maybe) 1% of it was corrupted - this doesn't make sense. |
| Comment by Jan Lindström (Inactive) [ 2015-10-14 ] |
|
I understand your point. From my point database is in consistent state if and only if all data is accessible and not corrupted. MariaDB could offer you a read-only access to all non-corrupted data. It does not make sense to allow write access, because that would mean that you modify database from non consistent state to ?. In your case MariaDB is missing something that would have allowed the read-only access and recover most of the data using e.g. mysqldump, but with information I have, I can't fix the issue. |
| Comment by azurit [ 2015-10-14 ] |
|
I still have that corrupted ibdata1 file. Will that help? |
| Comment by azurit [ 2015-10-14 ] |
|
Also, i have ONE database. Maybe you can try to run MariaDB with this one database and ibdata1 and maybe it will start up and you will see problems which i experienced. If you run it with 'innodb_force_recovery=4', MariaDB should ignore all other missing databases/tables and start up. |
| Comment by Jan Lindström (Inactive) [ 2015-10-14 ] |
|
Yes, if you have ibdata1 file and one database, they will help to fix the issue. |
| Comment by azurit [ 2015-10-14 ] |
|
How and where can i upload it? It's about 400 MB (uncompressed). |
| Comment by Jan Lindström (Inactive) [ 2015-10-14 ] |
|
ftp://ftp.askmonty.org/ is fine for me (use compression). |
| Comment by azurit [ 2015-10-14 ] |
|
I uploaded it to directory 'private', the file has string ' |
| Comment by Jan Lindström (Inactive) [ 2015-10-14 ] |
|
Yes, now it will take some time before I have time to work on this. |
| Comment by Jan Lindström (Inactive) [ 2015-10-29 ] |
|
Hi, I tried to fix this but could not, too many tables/files missing and problem is also related to |