Syncing backups from multiple external drives


ludditefornow

Member
Local time
9:56 PM
Posts
3
OS
Windows 11 Home
Hi all,
This is not a strictly W11 issue but it is a backup issue. Over time, I have backed up the same or similar data to different external devices through necessity. I now have the task of syncing all this data. I am happy for the newest version of duplicates to be kept and the older ones deleted. But I am not really sure how to go about it.

I do have an old NAS with about 1.5 terrabytes available that would allow me to copy all data there. But is there any recommendation of the process I should take and software to utilise to set up and walk away from it after it starts?
 
Windows Build/Version
21H2

My Computers

System One System Two

  • OS
    Windows 11 Home
    Computer type
    Laptop
    Manufacturer/Model
    HP Paviliion X360
    CPU
    i5-10210U
    Memory
    2x8gb
    Graphics Card(s)
    intel UHD630
    Screen Resolution
    1920x1080
  • Operating System
    Windows 11 Home
    Computer type
    Laptop
    Manufacturer/Model
    Lenovo Legion 5
    CPU
    Ryzen 5800H
    Memory
    2x8gb 3200
    Screen Resolution
    2560x1440

FreeBooter

Well-known member
Power User
VIP
Local time
4:56 AM
Posts
733
OS
Windows 11
I use a batch script to manually backup files and folders.
 

Attachments

  • Incremental_Backup.bat
    2.6 KB · Views: 1

My Computer

System One

  • OS
    Windows 11
    Computer type
    PC/Desktop
    Manufacturer/Model
    HP Pavilion
    CPU
    AMD Ryzen 7 5700G
    Motherboard
    Erica6
    Memory
    Micron Technology DDR4-3200 16GB
    Graphics Card(s)
    NVIDIA GeForce RTX 3060
    Sound Card
    Realtek ALC671
    Monitor(s) Displays
    Samsung SyncMaster U28E590
    Screen Resolution
    3840 x 2160
    Hard Drives
    SAMSUNG MZVLQ1T0HALB-000H1

ludditefornow

Member
Thread Starter
Local time
9:56 PM
Posts
3
OS
Windows 11 Home
thank you. How does this deal with files of identical name with possible different sizes or dates?
 

My Computers

System One System Two

  • OS
    Windows 11 Home
    Computer type
    Laptop
    Manufacturer/Model
    HP Paviliion X360
    CPU
    i5-10210U
    Memory
    2x8gb
    Graphics Card(s)
    intel UHD630
    Screen Resolution
    1920x1080
  • Operating System
    Windows 11 Home
    Computer type
    Laptop
    Manufacturer/Model
    Lenovo Legion 5
    CPU
    Ryzen 5800H
    Memory
    2x8gb 3200
    Screen Resolution
    2560x1440

hsehestedt

Well-known member
Power User
VIP
Local time
8:56 PM
Posts
705
Location
Texas, USA
OS
Windows 11 21H2
@ludditefornow,

welcome to ElevenForum!

There are a number of different ways that you can handle syncing of data, but here just a couple of methods that I use with great success:

1) You can use the Robocopy command line utility to sync data between locations. If you are not really familiar with Robocopy the syntax can be a little daunting at first and there can definitely be a bit of a learning curve. If you decide to look at Robocopy the best tip that I can provide is to make use of the "/L" switch while you are learning / experimenting with Robocopy. The "/L" tells Robcopy to not actually perform any copy / move / delete operations, but to simply report what it would do if you had run the command without the "/L". It's a great way to ensure that you don't get yourself into trouble as you experiment.

2) If you are okay with using a third-party utility, my favorite is "FreeFileSync" which is available for free at www.freefilesync.org. This utility has a lot of great features. For example, you could tell it to sync a source to a destination and only keep the newest version of a file or folder. In your example, you note that you may have duplicate files but from different dates or times. FreeFileSync can sync the source and destination and keep whatever version of the file is newest. You can also do 2-way syncs.

Example: I have a laptop and a desktop. Sometimes I update files on my desktop computer, but at other times I may do work on my laptop that updates certain files. With a 2-way sync, FreeFileSync will sync the laptop and desktop, being smart enough to sync whatever the newest version of a file is whether that means copying from the laptop to the desktop or from the desktop to the laptop. If any file has changed in both locations, it's smart enough to know this and warn you of that situation it won't blindly delete something with changes just because it is older than the file in the other location.

There are some other cool tricks it can do, but I won't go into those now unless you are interested in knowing more.

If you need any more details, please say so and I would be happy to answer any questions that you have.
 

My Computers

System One System Two

  • OS
    Windows 11 21H2
    Computer type
    PC/Desktop
    Manufacturer/Model
    Home Built
    CPU
    Intel i7-11700K
    Motherboard
    ASUS Prime Z590-A
    Memory
    128GB Crucial Ballistix 3200MHz DRAM
    Graphics Card(s)
    No GPU - CPU graphics only (for now)
    Sound Card
    Realtek (on motherboard)
    Monitor(s) Displays
    HP Envy 32
    Screen Resolution
    2560 x 1440
    Hard Drives
    1 x 1TB NVMe Gen 4 x 4 SSD
    1 x 2TB NVMe Gen 3 x 4 SSD
    2 x 512GB 2.5" SSDs
    2 x 8TB HD
    PSU
    Corsair HX850i
    Case
    Corsair iCue 5000X RGB
    Cooling
    Noctua NH-D15 chromax.black cooler + 10 case fans
    Keyboard
    CODE backlit mechanical keyboard
    Mouse
    Logitech MX Master 3
    Internet Speed
    300Mb down / 20Mb up
    Browser
    Chromium Edge
    Antivirus
    Windows Defender
    Other Info
    Additional options installed:
    WiFi 6E PCIe adapter
    ASUS ThunderboltEX 4 PCIe adapter
  • Operating System
    Windows 11 21H2
    Computer type
    Laptop
    Manufacturer/Model
    HP Spectre x360 15-BL012DX
    CPU
    Intel i7-7500U
    Memory
    32GB
    Graphics card(s)
    Dual Intel HD 620 and Nvidia GeForce 940MX
    Sound Card
    Built-in Realtek HD Audio
    Monitor(s) Displays
    4k 15-inch
    Screen Resolution
    4k (3840 x 2160)
    Hard Drives
    1TB Seagate FireCuda 510 NVMe SSD
    Internet Speed
    300Mb down / 20Mb up
    Browser
    Chromium Edge
    Antivirus
    Windows Defender
    Other Info
    RAM Upgraded from 16GB to 32GB WiFi Upgraded from WiFi 5 to WiFi 6 SSD upgraded from 512GB NVMe SSD to 1TB Seagate FireCuda 510 NVMe SSD

jimbo45

Well-known member
Power User
VIP
Local time
1:56 AM
Posts
1,631
Location
Hafnarfjörður IS
OS
Windows XP,7,10,11 Linux Arch Linux
Hi all,
This is not a strictly W11 issue but it is a backup issue. Over time, I have backed up the same or similar data to different external devices through necessity. I now have the task of syncing all this data. I am happy for the newest version of duplicates to be kept and the older ones deleted. But I am not really sure how to go about it.

I do have an old NAS with about 1.5 terrabytes available that would allow me to copy all data there. But is there any recommendation of the process I should take and software to utilise to set up and walk away from it after it starts?
Hi there
If you can use Linux (not sure what your OS is on the NAS - but you could even boot an external device with any Linux disto on it) then the simplest method is :

1) Mount all the external drives to /mnt/disk1, /mnt/disk2 /mnt/disk3 etc where the drives are /dev/sda. /dev/sdb etc
2) set up a crontab job for the backup using rsync.
3) if actually there's no duplicate data on the different external devices then you could run these jobs in parallel.

(probably safer toi use rsync becuase you can prevent an older file overwriting a newer one).

There's zillions of parameters to rsync so test carefully. A good way to start is to use the GUI (GRSYNC) as you can see the commands it generates for the rsync command for various options.

Note also once the program starts running there's a popup log screen which gives you real time info on backup progress, speed etc etc - you can optionally supress it.

E.g :

Screenshot_20220622_170112.png

I use Macrium on Windows for OS image backups and rsync for data backups -- note that Linux can read / write windows ntfs (and fat 32) disks so no prob mounting Windows data disks. (as root / sudo ) -- mount /dev/sdx -t auto -o rw /mnt/dvy for the windows disks -- the -t auto tells linux to determine the file system automatically --- easy - peasy.

Cheers
jimbo
 

My Computer

System One

  • OS
    Windows XP,7,10,11 Linux Arch Linux
    Computer type
    PC/Desktop
    CPU
    2 X Intel i7

ludditefornow

Member
Thread Starter
Local time
9:56 PM
Posts
3
OS
Windows 11 Home
thanks for this info all. I will take a look over the weekend. I will look at robocopy but these days a GUI based solution like Freefilesync might just be the answer.
 

My Computers

System One System Two

  • OS
    Windows 11 Home
    Computer type
    Laptop
    Manufacturer/Model
    HP Paviliion X360
    CPU
    i5-10210U
    Memory
    2x8gb
    Graphics Card(s)
    intel UHD630
    Screen Resolution
    1920x1080
  • Operating System
    Windows 11 Home
    Computer type
    Laptop
    Manufacturer/Model
    Lenovo Legion 5
    CPU
    Ryzen 5800H
    Memory
    2x8gb 3200
    Screen Resolution
    2560x1440
Top Bottom