[MDEV-14099] IBD file extension is not suitable for Windows system restore Created: 2017-10-21  Updated: 2020-07-06  Resolved: 2017-10-21

Status: Closed
Project: MariaDB Server
Component/s: Platform Windows, Server, Storage Engine - InnoDB
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Tobia Scapin Assignee: Unassigned
Resolution: Not a Bug Votes: 1
Labels: innodb, windows


 Description   

Dear all,
I found that IBD extension of mariaDB InnoDB data files is included in "monitored files" of Windows System Restore (https://msdn.microsoft.com/en-us/library/windows/desktop/aa378870%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396).
This means that if the data folder is inside a disk volume with System Restore enabled there is the serious hazard of silent data loss in case of System Restore and there is no way to avoid this.

If System Restore starts (consider that this can starts automatically from a simple power loss) all "monitored" files are rolledback included IBD file of MariaDB databases. After the Windows restore MariaDB service starts normally but all its data is rolledback at restore point due to IBD file restore!

I tried to find a solution configuring Windows Restore, if I exclude MariaDB folder from the restore using FilesNotToSnapshot (https://msdn.microsoft.com/en-us/library/windows/desktop/bb891959(v=vs.85).aspx) the restore process simply delete IBD files instead of restoring them. *Seems that there is not way to do not touch IBD files during System Restore. *The only solution is to change the file extension.

I think this is a serious problem and I spent a lot of time to understand what was the cause of some huge (many days!) rollback of my users after power loss.
The only solution for me now is to disable the Windows System restore.

I wonder if MariaDB can optionally use a different extension for IBD files.



 Comments   
Comment by Vladislav Vaintroub [ 2017-10-21 ]

It does not help if MariaDB is using a different extension, because for all supported version of Windows that MariaDB supports, Windows system restore is not using file extensions anymore (see e.g Wikipedia on that topic https://en.wikipedia.org/wiki/System_Restore). Apart from that, why would one want to exclude database files. It is a backup-restore on system level, and as with any backup-restore it rolls back to the state when the backup was taken. I'd say it is the expected behavior. As for silent "system restore" after power loss, I am not aware of that, but even if that happens, mariadb can't do much.

Comment by Vladislav Vaintroub [ 2020-07-06 ]

So, somebody else asked about that today, and I gave it a try myself. Have to confess that 3 years ago I did not even really understood what it was about, that system restore thing is very obscure, and I did not do this ever, nor I was affected by it, for the 2.5 decades of using Windows.

So here is the question
https://mariadb.com/kb/en/the-database-mariadb-is-affected-by-the-restoration-of-windows/

I checked, the system restore seems to be a combination of 2 volume snapshots , the taking one at the time of "system restore", for the "Undo" functionality, and another one of the past point in time, to which "system restore" would rollback , and I had another one, taken by MSI for "MariaDB installation",

C:\Windows\System32>Vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
 
Contents of shadow copy set ID: {9c74015c-e39d-492c-a3a9-8b1db10a8cb6}
   Contained 1 shadow copies at creation time: 06/07/2020 15:24:58
      Shadow Copy ID: {40629602-54f5-48a2-a39a-1ce6954f6355}
         Original Volume: (C:)\\?\Volume{9de9f1b3-96ad-4b74-8976-d2e1c2a48402}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
         Originating Machine: workpc
         Service Machine: workpc
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessibleWriters
         Attributes: Persistent, Client-accessible, No auto release, Differential, Auto recovered
 
Contents of shadow copy set ID: {3ee3d99e-a8a9-4e7d-9813-0c857068acb4}
   Contained 1 shadow copies at creation time: 06/07/2020 15:27:57
      Shadow Copy ID: {60e43468-ea57-475b-8aea-477cf12b93d3}
         Original Volume: (C:)\\?\Volume{9de9f1b3-96ad-4b74-8976-d2e1c2a48402}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2
         Originating Machine: workpc
         Service Machine: workpc
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessibleWriters
         Attributes: Persistent, Client-accessible, No auto release, Differential, Auto recovered
 
Contents of shadow copy set ID: {66e7fcec-72d3-4dbb-8c8e-eb393bc18d4a}
   Contained 1 shadow copies at creation time: 06/07/2020 15:30:35
      Shadow Copy ID: {026f3e38-773b-4b6c-828d-f167e02c8ca2}
         Original Volume: (C:)\\?\Volume{9de9f1b3-96ad-4b74-8976-d2e1c2a48402}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3
         Originating Machine: workpc
         Service Machine: workpc
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: ClientAccessibleWriters
         Attributes: Persistent, Client-accessible, No auto release, Differential, Auto recovered

I had created "06/07/2020 15:27:57" restore point myself, added some rows to innodb, then in 3 minutes told "system restore" to restore to the said point, and , after much huffing and puffing (not sure how much it took, 1 hour, more maybe, I was away ), it reverted innodb to the state 3 minutes before.

Yes, this seems like a bad thing would have happened, but the backup is there, and it is accessible

I do now

mklink /d C:\last_snapshot \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3\
net stop mariadb
ROBOCOPY /MIR "C:\last_snapshot\Program Files\MariaDB 10.5\data" "C:\Program Files\MariaDB 10.5\data"
net start mariadb
rmdir C:\last_snapshot

to restore to the "pre-recovery" time. That's it.
Someone stated, that Windows would not always create the "undo" snapshot just prior to restoring old version. I do not know, I have no intention to try that thing again, as it is very time consuming thing.

Yet so far, it seems like "system restore" is slightly unpleasant, but not the fatal-data-loss scenario.

Comment by Tobia Scapin [ 2020-07-06 ]

Thank you for sharing. I added the system restore disabling in my installation procedure.
I had too much troubles related. Many times the user did not noticed the automatic system restore and the MariaDb rollback, and he started a new dataset branch. When he discovered the data loss I had to provide a manual data merge between the last db dump and the current branch.

Generated at Thu Feb 08 08:10:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.