I'm SO confused as to Macrium Differential / Incremental Backups vs Full Backups

Ghot · Dec 20, 2023

@Quandary

If worse comes to worst, you can just install the game or program twice.
All I had to do was create a shortcut for the program on the "other" Windows version.
If you're only running one Windows... then you won't have that problem.

cereberus · Dec 29, 2023

wiganken said:
As I understand it the problem with incremental backups is that it results in many small images and all of them are needed for a full restore. If any one of these images is corrupt then the restore cannot work. I know this can apply to Differential images but there is more risk with Incrementals due to the larger number of images created and needed for restore.

This is only partly true. You only need the incrementals up to date you intend to restore from. So if you made say 5 incrementals, one per day and wanted to revert to status as of end of day 3, you only need incrementals for days 1, 2, 3 but you do not need days 4 and 5.

cereberus · Dec 29, 2023

CSharpDev said:
Do you trust Macrium's auto-verification? I always have it enabled.

This is personal choice but I never bother as it slows things down.

Auto verification was important back in days of e.g. dvd backups as write errors were common but modern hard drives are far more reliable and write errors are rare.

Of course, verifications may help to identify if drives are failing, but I never use hdds anymore, and modern ssds/nvmes are extremely write resilient.

I asked a while back if anybody has experienced a verification error in recent years (say 5 years) and got loads of largely irrelevant replies by people saying they used verification as a precaution (which is fair enough), but not one person could say they had ever had issues.

To me, a better way of testing the backup is to load the image using viboot, or restore image to a native (dual boot) vhdx file if you are really concerned about quality of backup.

hsehestedt · Dec 29, 2023

cereberus,

I have had a little bit of a change of heart on the topic of performing verifications. I had two separate instances within a period of about 2 or 3 months where my data was getting silently corrupted.

The first one was difficult to track down, but I finally determined it to be a bad Ethernet cable. I was replicating data from one system to another using FreeFileSync. On the destination system I tried to install some program but the .EXE install file was corrupted. I checked my source, and the file was fine there. That prompted a CRC comparison of every single file, and I found several bad files. After extensive testing I found out that the issue was a bad network cable. Really odd that no errors were ever reported when data was in flight, but that was 100% the problem because I was getting repeated occasional corruption but after replacing the Ethernet cable never a single instance after that and it has been at least six months now.

On another system, a few months later, I was also getting silent corruption but it was of data at rest. I could do a CRC comparison and all would be okay. A week later, the CRC on two or three files would be different and the files corrupt. But I never changed those files in any way. This would happen repeatedly, every week another file or possibly up to three files would be bad. That turned out to be a defective SSD. The SSD was under a year old and was still at something like 99% health.

Years ago (several decades ago?) I had another similar issue but back then it was a bad IDE HD cable. Agin, silent corruption with no warnings. I actually lost a few files back on that one, but thankfully it was minor stuff of zero importance.

Bottom line: I'm a bit paranoid now

.

Stigg · Dec 29, 2023

This is a very short and easy to understand video of Incrementals Forever / Synthetic Full Backups

But I remember seeing a similar one explaining the difference between full, differential and incremental backups. But I can't find it right now.

Bree · Dec 29, 2023

cereberus said:
I asked a while back if anybody has experienced a verification error in recent years (say 5 years) and got loads of largely irrelevant replies by people saying they used verification as a precaution (which is fair enough), but not one person could say they had ever had issues.

For those past 5 years I have made Full images of all my machines once a month, auto-verifying every time. Never less than half a dozen machines, sometimes as many as 9, so that's something like 400-500 images. This month for the first time ever I saw a verification error.

Bree said:
All my patching/cleaning up went smoothly. But one of my backups threw up a verification error while backing up to it.

I have 3 usb hdd backup drives to put my backups on, one of them appeared to be faulty. After many hours of testing (Chkdsk /R alone takes 4.9 hours) I finally traced it to not the drive itself, but its usb cable. Replaced that and left Chhkdsk running again while I (finally) slept. All checks out as good this morning.

So a faulty cable can cause a data error, fortunately in this case it was a read error during verification, but it could just as easily have been a write error during imaging.

cereberus · Dec 29, 2023

hsehestedt said:
c

Bottom line: I'm a bit paranoid now .

Yep - safest assumption is "THEY" are out to get you.

cereberus · Dec 29, 2023

Bree said:
For those past 5 years I have made Full images of all my machines once a month, auto-verifying every time. Never less than half a dozen machines, sometimes as many as 9, so that's something like 400-500 images. This month for the first time ever I saw a verification error.

So a faulty cable can cause a data error, fortunately in this case it was a read error during verification, but it could just as easily have been a write error during imaging.

I mostly backup to second nvme, and occasionally copy backup to a usb drive and now onedrive.
I agree a faulty cable can give an issue which is why I tend to buy braided cables which seem more reliable.

In the end, I think the risk is low and all my critical data is not in my image backups anyway. In the end, I can recover back to a reasonable position.

I would have to have several simultaneous failures before I was really in deep dung.

glasskuter · Dec 29, 2023

Throwing my ideas into the mix here since the OP's question was regarding the necessity of incrementals and/or differentials. IMO it all depends on how an individual system is used, what data he wants to protect, where that data is stored, how often that data changes, and how critical that data is. Whether one's personal files are synced to Onedrive or not comes into the mix, too, when it comes to deciding the need for incrementals or differentials.

The purpose of imaging/backups is to be able to recover in case of catostrophe. For most people, I believe using incrementals or differentials is overkill.

Let's say a computer is used by a business (or an individual who works from home) where including each and every file stored on that computer is critical to the operation of that business. In this case, yes, an incremental or differential backup plan would be 100% appropriate and necessary. If that user has not made major changes to the OS and those files are stored in the cloud, incremental or differential backups would not make much sense.

Then let's say we have an individual user (like me) whose drive(s) data is rarely changed except for Windows update or me installing a new app or updating a driver, neither of which can not be done again if a catastrophe happens in the time between my full images. I make these full images if I make major changes to my system or once a month before update Tuesday.

My data is stored on a separate drive. I do not sync to Onedrive. This data rarely changes and nothing on it is something that can't easily be reproduced or I can't live without. In case of catastrophe (which rarely happens) I do not consider losing the few pictures of my dog I took in between images as critical data.(I don't have grandchildren, but if I did losing a few images of them would not be considered critical to me either since I probably would have 1000 more already included in my full images)
So in my case, incremental/differental backups make no sense. If I synced my photos to onedrive, it would make even less sense as my dog would still be there. I image this data drive once every 1-2 months.

I'm a KISS kind of gal. Why complicate something simple any more than is necessary.

jimbo45 · Jan 11, 2024

glasskuter said:
Throwing my ideas into the mix here since the OP's question was regarding the necessity of incrementals and/or differentials. IMO it all depends on how an individual system is used, what data he wants to protect, where that data is stored, how often that data changes, and how critical that data is. Whether one's personal files are synced to Onedrive or not comes into the mix, too, when it comes to deciding the need for incrementals or differentials.

The purpose of imaging/backups is to be able to recover in case of catostrophe. For most people, I believe using incrementals or differentials is overkill.

Let's say a computer is used by a business (or an individual who works from home) where including each and every file stored on that computer is critical to the operation of that business. In this case, yes, an incremental or differential backup plan would be 100% appropriate and necessary. If that user has not made major changes to the OS and those files are stored in the cloud, incremental or differential backups would not make much sense.

Then let's say we have an individual user (like me) whose drive(s) data is rarely changed except for Windows update or me installing a new app or updating a driver, neither of which can not be done again if a catastrophe happens in the time between my full images. I make these full images if I make major changes to my system or once a month before update Tuesday.

My data is stored on a separate drive. I do not sync to Onedrive. This data rarely changes and nothing on it is something that can't easily be reproduced or I can't live without. In case of catastrophe (which rarely happens) I do not consider losing the few pictures of my dog I took in between images as critical data.(I don't have grandchildren, but if I did losing a few images of them would not be considered critical to me either since I probably would have 1000 more already included in my full images)
So in my case, incremental/differental backups make no sense. If I synced my photos to onedrive, it would make even less sense as my dog would still be there. I image this data drive once every 1-2 months.

I'm a KISS kind of gal. Why complicate something simple any more than is necessary.

Love gals who like KISS !!!!.
The days where individuals needed a load of this stuff like differentials / incrementals is ad you have said hugely overkill.

The easiest way for maintaining a system IMO these days is to install Windows in a small vhdx partition (either on a real machine or a VM if you use those) and have any data you want on another vhdx partition. So the disk geometry of your primary hard disk would look something like this -- EFI / Ms partition, big partition. In the big partition vhdx1 (windows OS), vhdx2 data etc.

Then just backup the vhdx files to NAS, external local storage, cloud etc etc.

If you need to do a bare metal restore just boot a winpe / windows install disk (use repair system->command prompt) etc. Copy the appropriate vhdx files to the target drive (bog standard windows copy command), attach the appropriate vdisks via disk part, mount those plus the efi boot partition and re-install the boot loader via bcdboot.exe.

All 100 % built in windows --no 3rd party software required and 100% reliable. These days with decent hardware fast large NvMe's (and even spinners) if your basic Windows system is say 50 - 70 GB copying and restoring vhdx files shouldn't take too long.

I'm really surprised many more don't use vhdx files more -- it's fairly straightforward and is also a great way of being able to have several different Windows systems e,g a normal "daily use one" and say Canary and Dev systems -- with the easy way to boot at will via Windows boot and not have to use the Bios menu in the computer. E.g :

cheers
jimbo

cereberus · Jan 11, 2024

jimbo45 said:
Love gals who like KISS !!!!.
The days where individuals needed a load of this stuff like differentials / incrementals is ad you have said hugely overkill.

The easiest way for maintaining a system IMO these days is to install Windows in a small vhdx partition (either on a real machine or a VM if you use those) and have any data you want on another vhdx partition. So the disk geometry of your primary hard disk would look something like this -- EFI / Ms partition, big partition. In the big partition vhdx1 (windows OS), vhdx2 data etc.

Then just backup the vhdx files to NAS, external local storage, cloud etc etc.

If you need to do a bare metal restore just boot a winpe / windows install disk (use repair system->command prompt) etc. Copy the appropriate vhdx files to the target drive (bog standard windows copy command), attach the appropriate vdisks via disk part, mount those plus the efi boot partition and re-install the boot loader via bcdboot.exe.

All 100 % built in windows --no 3rd party software required and 100% reliable. These days with decent hardware fast large NvMe's (and even spinners) if your basic Windows system is say 50 - 70 GB copying and restoring vhdx files shouldn't take too long.

I'm really surprised many more don't use vhdx files more -- it's fairly straightforward and is also a great way of being able to have several different Windows systems e,g a normal "daily use one" and say Canary and Dev systems -- with the easy way to boot at will via Windows boot and not have to use the Bios menu in the computer. E.g :

View attachment 83785
cheers
jimbo

VHDX files are a good way to native multiboot but there are restrictions e.g. you cannot use hibernate. Trimming etc. is a bit of a mystery.

Also, you do get a few % performance reduction using vhdx files.

Also backing up a vhdx can lead to unexpected issues if OS inside vhdx was not properly shutdown.

Also, vhdxs can be very space inefficient as a backup.

You can have dynamic vhdx sizing, but these grow in size due to inability to reclaim deleted file space (without going through hoops and loops).

So say you have a dynamic 100 GB vhdx with nominal 40 GB used. An image backup will be say 20 GB. A dynamic VHDX backup will be 40GB but could grow to 100 GB even though nominal storage is 40 GB.

I do not bother backing up my Insider vhdx files. If I get an issue, I just clone my RTM Host OS to a new vhd, and rejoin Insider so my Insider version is inline with Host OS.

VHDX files have their place but not as a reliable backup system.

jimbo45 · Jan 11, 2024

cereberus said:
VHDX files are a good way to native multiboot but there are restrictions e.g. you cannot use hibernate. Trimming etc. is a bit of a mystery.

Also, you do get a few % performance reduction using vhdx files.

Also backing up a vhdx can lead to unexpected issues if OS inside vhdx was not properly shutdown.

Also, vhdxs can be very space inefficient as a backup.

You can have dynamic vhdx sizing, but these grow in size due to inability to reclaim deleted file space (without going through hoops and loops).

So say you have a dynamic 100 GB vhdx with nominal 40 GB used. An image backup will be say 20 GB. A dynamic VHDX backup will be 40GB but could grow to 100 GB even though nominal storage is 40 GB.

I do not bother backing up my Insider vhdx files. If I get an issue, I just clone my RTM Host OS to a new vhd, and rejoin Insider so my Insider version is inline with Host OS.

VHDX files have their place but not as a reliable backup system.

Agreed but it's still OK, all Windows 100% free etc. I tend to use fixed vhdx files - cloning the RTM host is a good idea too.

I do backup data to NAS via rsync and grsync -- the GUI version of rsync (but that's a non windows solution). E.g backing up some classical music I'd got from BBC R3. The Windows disk is mounted on /mnt/DV1 -- most Linux based NAS systems can read / write NTFS files -- ensure package ntfs-3g is installed -- usually is by default anyway these days.

I haven't had problems so far with VHDX files and there's very little data I have on Windows that changes -- any important stuff I have on cloud / NAS / external drives. Things like multi-media for me rarely change so it's not necessary to back that up regularly -- rsync does a good job of selecting whats changed or modified (but again a non windows solution).

The % performance reduction in using vhdx files is negligible unless you've got a really ancient machine with an old fashioned processor in it . Browsing the web, using Ms office apps etc won't be noticeably slower unless you are using Excel to solve a long complex 5th order differential equation or plotting the heating distribution characteristics of an Airbus 320 wing being heat tested at 90 deg.

Cheers
jimbo

ThrashZone · Jan 11, 2024

CSharpDev said:
Do you trust Macrium's auto-verification? I always have it enabled.

Hi,
Yeah I've been using it seems fine
I also use shutdown after.

I only do full images usually about two week interval

ThrashZone · Jan 11, 2024

Might add I do not trust reflect in the os to do system images this is more doubt on the os to cooperate than reflect bug.
I only trust and use the winpe recovery media for system images os disruption is out of the equation

hsehestedt · Jan 11, 2024

I like using a VHDX for a test installation on two main occasions:

1) I need access to some physical hardware that a Hyper-V VM will not give me.

2) If I need to do some testing with items related to personalization since that requires activation. Installation to a VHDX will activate automatically whereas it will not on Hyper-V.

Bree · Jan 11, 2024

cereberus said:
VHDX files are a good way to native multiboot but there are restrictions e.g. you cannot use hibernate. Trimming etc. is a bit of a mystery.

Also, you do get a few % performance reduction using vhdx files.

The performance hit is real, but in my experience it's minimal....

cereberus said:
....vhdxs can be very space inefficient as a backup.

You can have dynamic vhdx sizing, but these grow in size due to inability to reclaim deleted file space (without going through hoops and loops)....

The hoops are not that difficult to jump through. My System Four (in 'Other Info') runs four native boot vhdx in addition to its conventionally installed Windows. With just a 256GB SSD space is at a premium, even with a 64GB virtual drive and dynamic vhdx for each native boot I need to keep at least 40GB free space available. To reclaim deleted space in the vhdx the steps are first to clean up the vhdx install, then boot to conventional Windows and....

mount the vhdx
from the command line (admin) defrag <mounted drive letter>: /v /u
detach mounted vhdx
in Diskpart select vdisk file=<full path to vhdx> then compact vdisk

I can get each vhdx down to less that 20GB.

cereberus said:
.....I do not bother backing up my Insider vhdx files. If I get an issue, I just clone my RTM Host OS to a new vhd, and rejoin Insider so my Insider version is inline with Host OS.

I do.

I used to use 7-Zip to archive a vhdx, but that takes hours (something to leave running overnight). I have now removed a Reflect Home licence from a lesser used machine and installed it on this one. I have set up a separate Files and Folders backup definition file for each vhdx. For Dev and Canary which is a full reinstall for each new update that's a Full backup each time, but for Beta and RTM which are just CU updates, I have found that Incrementals work just as well on a vhdx file as they would on imaging a conventionally installed OS.

cereberus · Jan 11, 2024

ThrashZone said:
Might add I do not trust reflect in the os to do system images this is more doubt on the os to cooperate than reflect bug.
I only trust and use the winpe recovery media for system images os disruption is out of the equation

This does not reflect (pun intended) the experience of vast the Macrium Reflect user base.

If you are worried about VSS interaction, you can always use Reflect in WinPE mode.

hsehestedt · Jan 11, 2024

ThrashZone said:
Might add I do not trust reflect in the os to do system images this is more doubt on the os to cooperate than reflect bug.
I only trust and use the winpe recovery media for system images os disruption is out of the equation

But bear in mind that the application is using VSS, a Windows OS component. That's what is so wonderful about it. You are not reliant upon the application handling open files in the OS in order to perform the backup. That part is handled by the OS so I would consider it stone cold reliable.

VSS has been around many years now and it's pretty much a known quantity. Are there rare instances where a problem can occur? Sure, but you will be informed if there is any issue.