Solved What is the Robocopy syntax?

Haydon · Mar 11, 2022

Thanks, I added the switches /XJ /XJD /XJF then verified that the script below works on a W10 and a W11 machine :):)

On the W11 machine, I tried to mirror the AppData folder with the script (including the 3 switches) but robocopy still hangs


:: Robocopy script by Haydon and others on elevenforum.com version 0.02
:: IMPORTANT: Use the script at your own risk, folks!
:: This robocopy script mirrors the contents of the Users folder Haydon to the Haydon folder on the external D: drive.
:: Change the name of the external drive to D: if need be.
:: Mirror means copy and delete, run the script for the latest mirror.
:: The Haydon folder on the external drive holds a full mirror along with file security info like file creation date, etc.
:: IMPORTANT: Check the log file ROBOLOG on the D: drive after running the script for any failures.
:: /B :: copy files in Backup mode
:: /MIR :: MIRror a directory tree
:: /XJ :: eXcludes Junction points, which are normally included by default
:: /XJD :: eXcludes Junction points for Directories
:: /XJF :: eXcludes Junction points for Files
:: /R:n :: number of Retries
:: /W:n :: Wait time between retries in seconds
:: /LOG: :: Output specified LOG file at specified location, the first command line has no + , subsequent command lines have the + , to append and consolidate the logs in one big log file.
:: pause :: keeps the CLI window open after running the script, useful for debugging purposes
robocopy.exe C:\Users\Haydon\Documents D:\Haydon\Documents /B /MIR /XJ /XJD /XJF /R:2 /W:20 /LOG:D:\ROBOLOG.txt
robocopy.exe C:\Users\Haydon\Pictures D:\Haydon\Pictures /B /MIR /XJ /XJD /XJF /R:2 /W:20 /LOG+:D:\ROBOLOG.txt
:: add command lines for additional folders to mirror. Does NOT work for the AppData folder!
:: pause

Haydon · Mar 12, 2022

It looks like I interrupted my W11 machine prematurely in the previous posting. I let my W10 machine run overnight, and surprise, surprise, it did mirror the AppData folder. I don't know exactly how long it took, but it took surprisingly long to mirror a 2 GB folder. The contents of the AppData folder is not interesting to me (although interesting to app developers and others) I was merely probing the boundaries/capabilities of the script. The script mirrors all non-hidden user folders expediently, no problem.

The comments thus needs changing from no to yes, the script can mirror the AppData folder too, although it takes a long time to do.


:: Robocopy script by Haydon and others on elevenforum.com version 0.03
:: IMPORTANT: Use the script at your own risk, folks!
:: This robocopy script mirrors the contents of the Users folder Haydon to the Haydon folder on the external D: drive.
:: Change the name of the external drive to D: if need be.
:: Mirror means copy and delete, run the script for the latest mirror.
:: The Haydon folder on the external drive holds a full mirror along with file security info like file creation date, etc.
:: IMPORTANT: Check the log file ROBOLOG on the D: drive after running the script for any failures.
:: /B :: copy files in Backup mode
:: /MIR :: MIRror a directory tree
:: /XJ :: eXcludes Junction points, which are normally included by default
:: /XJD :: eXcludes Junction points for Directories
:: /XJF :: eXcludes Junction points for Files
:: /R:n :: number of Retries
:: /W:n :: Wait time between retries in seconds
:: /LOG: :: Output specified LOG file at specified location, the first command line has no + , subsequent command lines have the + , to append and consolidate the logs in one big log file.
:: pause :: keeps the CLI window open after running the script, useful for debugging purposes
robocopy.exe C:\Users\Haydon\Documents D:\Haydon\Documents /B /MIR /XJ /XJD /XJF /R:2 /W:20 /LOG:D:\ROBOLOG.txt
robocopy.exe C:\Users\Haydon\Pictures D:\Haydon\Pictures /B /MIR /XJ /XJD /XJF /R:2 /W:20 /LOG+:D:\ROBOLOG.txt
:: add command lines for additional folders to mirror
:: AppData folder takes a long time to mirror
:: pause

Try3 · Mar 12, 2022

But I thought you were robocopying C:\Users\Haydon\Documents etc not C:\Users\Haydon itself?

Haydon said:
I tried to mirror the AppData folder with the script ... but robocopy still hangs

If you are certain that AppData contains no references to external drives or network drives that are not currently attached then the hang is unexplained.
If you are not certain that AppData contains no references to external drives or network drives that are not currently attached then do not use RoboCopy because it will hang.

Denis

ddelo · Mar 12, 2022

As far as I know %AppData% has a lot of locked files.
So, with /R:2 and /W:20. means that it takes 40 sec to copy each locked file. Maybe that's the reason for such a long time.
Out of curiosity though, why would you want to backup the AppData folder? I would understand it for a system image, but for file/folder backup?

Update: Just run a PS script which checks for locked files and in AppData it found 404 locked directories!!!! That will take at least 4.5 hours to complete with your /R and /W switches.

ddelo · Mar 12, 2022

Just for reference some of my backup lines (I have ~20 of them in a batch file), which have been working flawlessly since the W7 days!

Code:

RoboCopy "%USERPROFILE%\Documents" "D:\Backup\Documents" /MIR /DCOPY:DAT /XJ /MT:16 /R:0 /W:0 /UNILOG+:"D:\Backup\backup.log" /TEE /NP /FP /NDL /BYTES
RoboCopy "%USERPROFILE%\Music" "D:\Backup\Music" /MIR /DCOPY:DAT /XJ /MT:16 /R:0 /W:0 /UNILOG+:"D:\Backup\backup.log" /TEE /NP /FP /NDL /BYTES
RoboCopy "%USERPROFILE%\Pictures" "D:\Backup\Pictures" /MIR /DCOPY:DAT /XJ /MT:16 /R:0 /W:0 /UNILOG+:"D:\Backup\backup.log" /TEE /NP /FP /NDL /BYTES

Try3 · Mar 12, 2022

Haydon,

Please notice that, as in the sample RoboCopy script I posted earlier, Dimitri thinks the /DCOPY:DAT switch is useful.
As you'll see in the RoboCopy Help file & the online references I posted for you, /DCOPY:DAT preserves folder dates just as RoboCopy does by default for files.

Note also that you can stop the forum software interpreting things, like the colonD you mentioned earlier, by wrapping them in the appropriate "BB code"
[PLAIN]/DCOPY:DAT[/PLAIN]
BB codes - ElevenForum
Many are the same as BB Codes - TenForums but not this one.
- In TenForums you'd wrap them in [NoParse]/DCOPY:DAT[/NoParse] or select the words then click on the TenForums editing bar icon that looks like a yellow warning triangle on top of a spreadsheet

All the best,
Denis

Haydon · Mar 12, 2022

@Try3 What took so long was C:\Users\Haydon\AppData

@ddelo And there is the explanation for the overnight run!

And no, as I wrote in an earlier post, I am not interested in the contents of the AppData folder, I was just probing the limits of the script!

@Try3 @ddelo It is very reassuring that robocopy has been working since ages

Haydon · Mar 12, 2022

Denis, we posted at the same time, I will try the /DCOPY:DAT switch, stay tuned!

Try3 · Mar 12, 2022

Haydon,

"Probing the limits" then posting about the results - Well done, that's how communal knowledge progresses.

There are other undocumented limits. Many are just discovered by people posting about problems they've encountered.
- Some people found problems with paths-filenames containing special characters [or it might have been me, my notes are dated 2011 & I don't remember the incident]. One example is Robocopy Syntax, Command Line Switches and Examples » My Digital Life.lnk and another is Something containing &, an ampersand. Wrapping paths-filenames in " protects you against many of these things but I just never use them instead.
- RoboCopy can also hang if a file is on use but it does not do so for all file types. I know it is happy with open Excel files. My notes are dated 2012.
- I never use RoboCopy with MSOffice Outlook pst files whilst Outlook is open. RoboCopy can often cope but Outlook might not; the act of attempting to copy them has been found to cause pst file corruption. I saw evidence of this posted by another user and have avoided doing so ever since. I never saw the problem on my own computer; I took his word for it.
- Perhaps the limits I know of have changed over the years. I'd only find out if I started doing something different & saw it behave in an unexpected manner. Witness my earlier comments about how to exclude AppData. I've been using my multi-line scripts for each of my individual high-level folders since before AppData was even a maniacal gleam in the eye of an MS programmer.

All the best,
Denis

Haydon · Mar 12, 2022

Incorporating the latest feedback, I can confirm that the updated script works on a W10 machine, the original time stamps are now on the folders, the log shows 0 failures.

I ran it as a subsequent run (not an original run) and I was again amazed at how fast robocopy works. I read on the microsoft docs site that the default value for multi-threading is 8, my goodness, who needs more, I don't.

I now have all I need which is much, much more than I originally expected from robocopy

And yeah, in the same vein as trying to give something back, others may take the following script as their own starting points

Code:

:: Robocopy script by Haydon and others on elevenforum.com version 0.04
:: IMPORTANT: Use the script at your own risk, folks!
:: This robocopy script mirrors the contents of the Users folder Haydon to the Haydon folder on the external D: drive.
:: Change the name of the external drive to D: if need be.
:: Mirror means copy and delete, run the script for the latest mirror.
:: The Haydon folder on the external drive holds a full mirror along with file security info like file creation date, etc.
:: IMPORTANT: Check the log file ROBOLOG on the D: drive after running the script for any failures.
:: /B :: copy files in Backup mode
:: /MIR :: MIRror a directory tree
:: /DCOPY: Specifies what to copy in directories. The valid values for this option are:
:: D - Data
:: A - Attributes
:: T - Time stamps
:: /XJ :: eXcludes Junction points, which are normally included by default
:: /XJD :: eXcludes Junction points for Directories
:: /XJF :: eXcludes Junction points for Files
:: /R:n :: number of Retries
:: /W:n :: Wait time between retries in seconds
:: /LOG: :: Output specified LOG file at specified location, the first command line has no + , subsequent command lines have the + , to append and consolidate the logs in one big log file.
:: pause :: keeps the CLI window open after running the script, useful for debugging purposes
robocopy.exe C:\Users\Haydon\Documents D:\Haydon\Documents /B /MIR /DCOPY:DAT /XJ /XJD /XJF /R:2 /W:2 /LOG:D:\ROBOLOG.txt
robocopy.exe C:\Users\Haydon\Pictures D:\Haydon\Pictures /B /MIR /DCOPY:DAT /XJ /XJD /XJF /R:2 /W:2 /LOG+:D:\ROBOLOG.txt
:: add command lines for additional folders to mirror
:: pause

Bree · Mar 12, 2022

Haydon said:
as I wrote in an earlier post, I am not interested in the contents of the AppData folder, I was just probing the limits of the script!

My own backup script is mainly used to perform either a full or incremental backup of my user folder. I do not want to mirror the folder, rather I am making a straight copy, with each backup being to a new folder of its own rather than modifying an existing backup.

To determine which files are new or modified for the incremental I use the /M switch (a switch which exists and behaves the same in Xcopy that I used previously).

This only copies files with the Archive attribute set, then clears it. To make a full backup the script first runs the command ATTRIB +A *.* /s so that every file has its Archive attribute set and will thus be backed up. Only files that are new or modified since my last full backup will have their Archive attribute set, so only those files will be copied in a subsequent incremental run of the script.

I do include the AppData folder in the full backup, but exclude it from subsequent incrementals. While most of AppData may appear to be of little use there are a few gems that can usefully be restored, such as my Firefox profile. This is where it is absolutely essential to use the /XJD switch, without it RoboCopy will follow the 'Application Data' Junction in an infinite loop - the junction points to the folder that contains it!

All these junctions are there for a good reason. They are the names of the real folders that were in use up until XP, but were moved and renamed from Vista onwards. Older software will expect these folders to exist, the junctions allow old software to use the new locations as if they were still the old ones.

Try3 · Mar 12, 2022

Haydon said:
I ran it as a subsequent run (not an original run) and I was again amazed at how fast robocopy works. I read on the microsoft docs site that the default value for multi-threading is 8, my goodness, who needs more, I don't.

I once ran some comparative tests.

The best /MT:n setting is not necessarily the highest number. Here are some sample test results
that demonstrate this fact -
1 thread 16 mins
2 threads 9 mins
3 threads 6 mins
4 threads 5 mins
5 threads 7 mins
6 threads 5 mins
16 threads 4 mins
32 threads 2 mins
64 threads 3 mins
128 threads 5 mins
[the tests involved copying a large number of files across a WiFi network]

I would hazard the guess that the most appropriate MT value varies with the computers & disks in use as well as, in my example, the network connection involved.

All the best,
Denis

Haydon · Mar 12, 2022

I wonder whether I should start a new thread on Backup Strategies or just continue here. Let me try here, now that there is a very good discussion going on.

I use File History, because I like the hourly backups. But exactly that is my concern too. There are 2,000 work hours in a year, that's up to 2,000 incremental backups that have to work together if you need a restore

I resorted to deleting the File History folder every now and then to force a full backup every now and then. To be sure, File History has never failed me, but that's a brute force method, and I am looking for a better solution.

Enter Macrium (or Aomei, etc.) I could do weekly Full Backups and File History would have to do 40 to 50 interpolations (the number of work hours in a work week)

Enter RoboCopy, I could update the RoboCopy mirror at the end of the work day and File History would have to do 8 to 10 interpolations (the number of work hours in a work day)

The File History interpolations work because File History features an 'All but the latest version' deletion and it will continue therefrom. I believe that File History creates a Synthetic Full Backup at that point to continue with the hourly backups. I would not rely on only 1 backup method though (no matter what it is) 2 is better, 3 is very good, and 4 may be beyond the point of diminishing returns.

What does your Backup Strategy look like?
Let's compare notes!
-----------------------------
Macrium backs up the AppData folder (and the NTUSER.DAT file) so I won't let Robocopy do that.

ddelo · Mar 12, 2022

Daily more than once (depending on the work done), Robocopy backup of my files,
Marcium Reflect Full image once a week.
Macrium Reflect Incremental image, whenever I wanna update a driver or make a system change or before a Microsoft update.

Haydon · Mar 13, 2022

Back to robocopy syntax. The switches /NS /NC /NFL /NDL /NP causes the log file to contain only Header and Summary. This is handy to quickly check if the run was error free

Thus, I have now 2 scripts, one with those switches for routine runs and another without those switches to produce the full log file if I need to track back some errors

Edit: so the pseudocode for the first command line with those switches is as follows, use /LOG+: for subsequent command lines

robocopy.exe <source> <destination> <switches> /LOG:D:\logfile.txt /NS /NC /NFL /NDL /NP

Haydon · Mar 14, 2022

Interesting readings to 'broaden your view and find your place in the world'

List of file copying software - Wikipedia

en.wikipedia.org

List of backup software - Wikipedia

en.wikipedia.org

Comparison of file synchronization software - Wikipedia

en.wikipedia.org

workablob · Sep 10, 2022

I got turned on to Robocopy years ago by a colleague in I.T.

It didn't take me long to realize I hate creating batch files so I did a google search for 'robocopy gui' and voila.

There are these:

Robogui
copyrite xp
rich copy

Some are free but the pay versions are reasonably priced.

Rich Copy is free and does not use Robocopy but it is pretty decent and was developed by a man from Microsoft.

CopyRite XP is fantastic but it's almost $20.00

I sprung for it and am happy with it) YMMV.

A quick Google search will yield results so I won't post any URLs here.

I am a gui guy so I love these options.