A very difficult lesson in why good backups are important

I was setting up Macrium Reflect on a laptop. I was planning on doing my backups over WiFi. I use WiFi 6 so it's quick enough to do the backups. I elected to perform a verification after the backup.

I ran my backup, and it failed on the verification saying that a block hash failed. I repeated this several times and every single time it failed, but always at a different point in the verification.

I was sure that I had a Reflect problem. Wrong!

As I test, I simply copied about 50GB of files from the laptop to the Windows system that I was baking up to simply using File Explorer. Then I ran checksums on all the files. Sure enough, one file showed a different checksum. The files look absolutely identical, with the exact same size, but clearly, they are different.

So, this falls into the category of a "silent failure". If I were simply copying files using File Explorer, I would have never been the wiser. It would have looked like everything was fine.

So, this just shows how important it is to have good quality backups. The irony in this whole thing was that merely the act of setting up my backups for the first time showed me that I have a major problem.

Well, tomorrow (okay, later today) will be interesting as I figure out where the problem is and whether I have bad data elsewhere.

For backing up important data...




On a side note: I always verify after making a backup, but I do it manually.
I started doing it manually, in version 7, and have been doing manual verifications ever since.




I seem to remember that I had difficulties when I just check marked the box, for the auto-verification.

Ghot, I agree with you. I was only planning on using WiFi on one laptop, and due to my distrust of WiFi I enabled the verification.

Unfortunately, the problem has gotten even worse now.

I'm having trouble with the system being used as the destination for the backups over Ethernet as well. Even going to a different HD fails so it's not the drive that is the problem.

I did a whole bunch of testing overnight. I tried 3 laptops and one other desktop system. I get that silent corruption of data both over WiFi and Ethernet to that system. The only combination with which I have seen no corruption is going Ethernet from the other desktop.

The ironic part is that I just transferred 7 TB (!) from that desktop to this problem system about a week ago. Fortunately, that connection is the only one that I am not seeing the silent corruption on. If all that data had been corrupted, I would have really been up the creek without a paddle.

So, here I sit, trying to figure out the cause of this weird issue but so far I just cannot make sense of the results I'm getting. I get the feeling that it's going to be a long day and night.

Is it the same file and only that file you mentioned in your OP that has the problem?

barreleye said:

Is it the same file and only that file you mentioned in your OP that has the problem?

Click to expand...

It was multiple different files, as well as every backup. However, I have an update. A couple of hours ago I finally figured out what the problem was, and I just finally finished my testing that proved where the issue was.

I got it narrowed down to one of these:

1) A bad port on my switch
2) A bad Ethernet adapter
3) A bad Ethernet cable

After more testing, it turned out to be a bad cable.

I'm running all 2.5Gb networking here and the cables I am using are supposedly Cat 7, but they are flat cables which means no shielding. I just ordered all new cables, I've lost confidence in these.

My really big worry is that because the symptom was silent data corruption, I'm worried that the 7 TB+ of data I moved over to that machine may have some bad spots. Interesting this though is that in all my testing, the only connection that seemed to never show failures is the one that I used to transmit that data, so hopefully I dodged a bullet.

Wow, this was a frightening experience.

Well that's some bad luck. Amidst your multitude of silent errors, I would think you got some that were detected and silently fixed, because error checking and retransmission are part of the transfer protocol. If it were a suspect HD cable, you could look at SMART, but I don't know where to look for networking.