Best way to copy 20TB of data from a number of drives and consolidate them on one

I have 5 USB 3 WD Elements external HDD's and I want to consolidate them all on a WD 22TB Elements external.
This is obviously a massive data transfer and will probably run for a week or more.
I want to do it with maximum efficiency and minimal wear and tear on the pc (laptop).
Maybe done from a WinRE boot media to eliminate all the Windows fluff that would run otherwise?
I believe that all transfers go through the OS drive - C: , and SSD in my case, so I don't want to wear the poor old SSD out prematurely either.
So any ideas on the best way? Preferably a way you've actually done yourself for a similar reason.

Well, frankly my advice is don't.

Bssically, if you put all data on one drive, you are putting all your eggs in one basket. If that drive fails, you lose the lot.

If you do need to go down such a route, you need a raid system with two backup drives at least.

As an aside, what on earth do you need 22TB data backup anyway?

There are a few ways that I could think of.

First, I would suggest that you are better off doing it with Windows. Windows will likely do a better more efficient job than doing this in something like Win PE. Also, Win PE will auto reboot after a period of time. If memory is correct, I think that may be 72 hours.

Next, no, these copies have no need to go through the C: drive so no worry there.

Okay, idea number one:

Just set up a small batch file that does robocopy operations. Do disk 1, then disk 2, etc. If you do decide to go that route and need help, let me know. I can whip up a batch file in a matter of minutes.

Another idea: One of my favorite apps for moving data around is FreeFileSync. It is free and for a big job like this I like that it will give you an estimate on how long the copy will talke so that you can always see progress.





Once again, if you need help setting this up, I would be happy to assist.

cereberus said:

As an aside, what on earth do you need 22TB data backup anyway?

Click to expand...

Never question people reasons for a lot of data. Some may get along fine with a few GB while I have about 100TB. A pal of mine has about a PB. Never did I question his backups either. He has his reasons and I'm actually quite confident I know what most of it is. And no it's not BR movies or similar...

I recommend using FastCopy for the job.

altae said:

I recommend using FastCopy for the job.

Click to expand...

Yes.

SlicEnDicE said:

Never question people reasons for a lot of data. Some may get along fine with a few GB while I have about 100TB. A pal of mine has about a PB. Never did I question his backups either. He has his reasons and I'm actually quite confident I know what most of it is. And no it's not BR movies or similar...

Click to expand...

My reaaon for asking is more related to optimising backup. By understanding what OP backs up, we can perhaps advise a good backup strategy.

If I had such a large quantity, I would categorise my data into (say) 3 levels

1) critical data that would be a disaster if lost. Such data should be backed up to at least two/three separate drives (cloud is also good as an option).

Example - legal documents.

2) important data where it would also be a pain if lost. Such data should be backed up to at least two places.

Example - wedding photos

3) routine data which if lost would be a nuisance. Such data should be backed up to at least one drive. Bear in mind if the backup drive fails, data is lost.

example - videos downloaded that can be redownloaded.

My point really is a single backup drive risks losing all data, so make sure critical data at least is backed up elsewhere.

Of course if OP feels all data is critical/important, then as a minimum OP should have 2 backup drives.

@SlicEnDicE

Hi

plenty of decent reasons to have that type / amount of storage -- I've got a load of storage on a Hyper-V linux VM - runs loads of backups from Windows clients, has a huge amount of multi-media files, loads of archived TV stuff, DVD and CD rips, etc etc. These don't change - so you only need to back those up once. You will as I emphasize later on in the post of course have to back these HDD's themselves up from time to time too.



@Morph000

Modern decent drives are very reliable -- it's OK to consolidate to larger drives - but again always remember THESE need to be backed up themselves (Not necessarily the whole lot in one go but files that change should be backed up regularly).

The easiest way IMO is to use a Linux VM - on Hyper-V is good enough if you don't want to mess around with a Linux Host, the Linux WSL or unpredictable things like VmWare workstation or Oracle's Virtual Box.

Then you can use a GUI front end to rsync which can take care of creating / populating all the relevant directories on the drives without running out of space etc etc. Linux can read / write NTFS files - has been able to for years.

The other thing to be aware of if you use a strictly Windows solution is that although Windows internally can handle path / file name etc lengths longer than 266 chars many of the apps Windows uses for directory browse etc still use the old API which limits the length to 266 chars after which you see some random name in 8.3 format. With large disks you could run into the 266 char limit. Linux has no problems.

There's loads of options - I know the GUI isn't the most exciting one around but I haven't found anything better that does the job in anything like the same manner.



One word of advice though -- If using these large capacity HDD's - I'd forget any sort of RAID whether hardware or software - just adds a whole lot of complexity to the whole kybosh -- IMHO unless running a 24/7 commercial web site / data centre - just not worth it.


Cheers
jimbo