RoboCopy not 'seeing' the target files.

BaJohn · Nov 16, 2022

I have been using RoboCopy for many years, without too much trouble.
I copy files from my win 11 PC to (1) another disk on the same machine, (2) a USB attached disk and (3) a NAS system on my local network.
Sometime between 13/10/22 and 16/10/22 the actions changed for the (2) USB process ONLY.
Every single file is copied every single time, rather than just changes and new files.
I have checked and there were no apps or programs updated during that period.
Has anybody any idea why this is occurring?

OAT · Nov 16, 2022

What is the command used? You give us very little here.

To copy only changed files with Robocopy, you have two options, namely using maxage: n or /XO /maxage:
n. Some users say that they don't want to create all the subfolders on the target disk if there are no new files in them.
In this case, you need to add extra switch /S to exclude the empty folder.

BaJohn · Nov 16, 2022

The guts of it
---------------------------------------
for %%F in ( Downloads
"End User") do (
echo *
echo * Folder %DSK%\%%~F

robocopy "%DSK%:\%%~F" "%NEW%:\TP-Robocopy\Disk %DSK%\%%~F" ^
/e /mir /np /a-:sh /copy

ATS /dcopy:T ^
/z /r:5 /w:2 /mt:8 ^
/xd "$RECYCLE.BIN" "System Volume Information" "RECYCLER" "Temporary Files" "Config.Msi" ^
/xf "pagefile.sys" ^
/log+:%DSK%2%NEW%_RoboCopy.log
)
--------------------------------------

BaJohn · Nov 16, 2022

The happy face

is actually ':' + 'D' - i.e. 2 characters - tried to change it and failed.

OAT · Nov 16, 2022

You are using /mir which is a mirror image of a complete directory tree.
So everything will always be copied.

Replace /mir with /XO /S (XO stands for Older file(s) - S stands for Copy subs but not empty ones)

rick99 · Nov 16, 2022

OAT said:
You are using /mir which is a mirror image of a complete directory tree.
So everything will always be copied.

Replace /mir with /XO /S (XO stands for Older file(s) - S stands for Copy subs but not empty ones)

/MIR will mirror the source and it will ignore files that have not been changed if there is a /XO. I do this frequently with both /MIR and /XO. However, the OP stated it has been working correctly and it still does on another hard drive and his NAS. It's just the USB drive that it is not working correctly.

BaJohn, can you try it with a different USB drive?

OAT · Nov 16, 2022

Good point Rick. Thanks.

BaJohn · Nov 16, 2022

Do you mean a different USB port or actually a different USB 'portable disc drive', as I have the former but not the later?
Remember the current USB drive was working happily for over 2 years.
The problem has only occurred about 5 weeks ago.
Something must have changed - I know things update in the background - but what?

hsehestedt · Nov 16, 2022

Quick question as I take a glance at this:

Are the following 2 lines meant to really be 2 separate lines or should that be a single line of text?

for %%F in ( Downloads
"End User") do (

Should that line look line this?

for %%F in ( Downloads "End User") do (

I'm not trying to say that this is part of the problem, just want to be sure I understand what this is really supposed to look like before a dive in.

BTW, to get better formatting and avoid stuff like that smily face drop a code tag in using this:

Then drop the code between the tags. In this case, when it asked what kind of code it was, I also selected batch rather than just general.

The end result:

Batch:

for %%F in ( Downloads
"End User") do (
echo *
echo * Folder %DSK%\%%~F

robocopy "%DSK%:\%%~F" "%NEW%:\TP-Robocopy\Disk %DSK%\%%~F" ^
/e /mir /np /a-:sh /copy:DATS /dcopy:T ^
/z /r:5 /w:2 /mt:8 ^
/xd "$RECYCLE.BIN" "System Volume Information" "RECYCLER" "Temporary Files" "Config.Msi" ^
/xf "pagefile.sys" ^
/log+:%DSK%2%NEW%_RoboCopy.log
)

PerpetualCycle · Nov 16, 2022

BaJohn said:
The happy face is actually ':' + 'D' - i.e. 2 characters - tried to change it and failed.

Code:

Just a note for posting code. If you select the textr and format it as a code section like this, it wouldn't happen :D.

hsehestedt · Nov 16, 2022

One more idea:

I'm rather slow with some of these weird elements of batch file syntax, but rather good with robocopy syntax. I'd like to see the actual command that is about to be run with robocopy. If you try this, it might help:

Place the word echo before robocopy. Then, just prior to the last closing paren, place a pause. This will cause the program to echo the command that would be run rather than to actually run it. Then it will pause. If you could simply grab the result displayed by the echo command and paste it here I would appreciate it.

This is what I mean:

Batch:

for %%F in ( Downloads
"End User") do (
echo *
echo * Folder %DSK%\%%~F

echo robocopy "%DSK%:\%%~F" "%NEW%:\TP-Robocopy\Disk %DSK%\%%~F" ^
/e /mir /np /a-:sh /copy:DATS /dcopy:T ^
/z /r:5 /w:2 /mt:8 ^
/xd "$RECYCLE.BIN" "System Volume Information" "RECYCLER" "Temporary Files" "Config.Msi" ^
/xf "pagefile.sys" ^
/log+:%DSK%2%NEW%_RoboCopy.log
pause
)

PerpetualCycle · Nov 16, 2022

I haven't used a USB drive as source for robocopy myself.

When you insert a USB drive, is its policy "quick removal" or "better performance". I can imagine this happening with the former. AAnd often I have found with updates etc., that the policy can revert to "quick removal". Might what to check this out in Computer Management -> disk drives.

hsehestedt · Nov 16, 2022

One idea already.

NOTE: I'm a long way off from complete analysis but try doing this one thing. This is going to sound really weird, but trust me on this. If it pans out, I'll explain later:

One the initial portion of the robocopy command that looks like this:

robocopy "%DSK%:\%%~F"

Please add a single space just before the closing double quote. It should then look like this:

robocopy "%DSK%:\%%~F "

I know this sounds wacko, but please try it!

EDIT: If that does not help, please try the same for the destination folder (add a space before the closing double quote).

hsehestedt · Nov 16, 2022

geneo said:
I haven't used a USB drive as source for robocopy myself.

When you insert a USB drive, is its policy "quick removal" or "better performance". I can imagine this happening with the former. AAnd often I have found with updates etc., that the policy can revert to "quick removal". Might what to check this out in Computer Management -> disk drives.

Unfortunately, that won't make a difference. That affects how data to the device is cached and flushed, but it makes no functional difference in what data is copied to the drive or not. I've had to experiment with this extensively as I copy hundreds of GB a day to flash drives. Not every day, but at least once or twice a week.

hsehestedt · Nov 16, 2022

I didn't want to take the time before to explain my odd request with the space before the closing double quote, but now that there is a break in the action, let me do so:

If you hard code a path in the batch file for either the source or destination, so long as you don't terminate the path with a backslash (\), the command should work fine.

However, when working with replaceable variables, I don't know what you may get. The problem occurs when the path ends with a backslash like this:

robocopy "D:\My path\" ....

In this instance, the backslash is used to escape the double quote mark, which you do not want here! As a result, a space is needed between the backslash and double quote mark. Note that even if the path does NOT end with a backslash, this will have no negative impact so it's fine to make this a habit

.

rick99 · Nov 17, 2022

BaJohn said:
Do you mean a different USB port or actually a different USB 'portable disc drive', as I have the former but not the later?
Remember the current USB drive was working happily for over 2 years.
The problem has only occurred about 5 weeks ago.
Something must have changed - I know things update in the background - but what?

I'm thinking a different USB Drive. If I understand this correctly, when robocopy copies to the USB drive, every file is copied to this drive instead of just new or changed files... correct? Something me be changing the timestamp on the USB files in which case they would be new files to robocopy. Perhaps antivirus scanning this device or some other issue with the drive.

Also, why are you wrapping the robocopy command in this batch script? Describe what it's doing as opposed to a standalone command?

hsehestedt · Nov 17, 2022

@BaJohn,

There have been a number of posts here since you last responded. I'm particularly interested in see if you could post the info requested in posting #11 of this thread. That will allow us to see the actual robocopy command in text with real values plugged in rather than all the variables and replaceable parameters.

I do, however, have one additional question: Is it EVERY robocopy command run in that loop that is failing now or just some of them? If it is only some, let me know and I will modify the batch file as I have posted it so that we can capture all of the robocopy commands rather than just a single instance.

Try3 · Nov 17, 2022

BaJohn said:
Every single file is copied every single time, rather than just changes and new files.

Just to avoid misunderstandings, please would you post a screenshot showing that.

All the best,
Denis

hsehestedt · Nov 17, 2022

Okay,

I have a new revision of some changes I would like to see made tothe batch file. This is temporary, for testing only.

Below is your original code with the following changes:

1) The block in the middle simple echos the Robocopy command that is about to be run to a text file called Robolog.txt.
2) Before running the FOR loop, it deletes the Robolog.txt file if there was already one so that we have a fresh copy of the file each time we run it.

What this does:

This does not change the behavior of the batch file at all. It merely add the logging of each command being run to a text file. This way we can see the actual Robocopy command in context with the variables replaces by the actual names of files, paths, etc.

This does not log the output from the robocopy commnad. I saw no need to do that since the robocopy command itself is creating a log file.

Batch:

if exist Robolog.txt del Robolog.txt

for %%F in ( Downloads
"End User") do (
echo *
echo * Folder %DSK%\%%~F

echo ------------------ >> Robolog.txt
echo Command: robocopy "%DSK%:\%%~F" "%NEW%:\TP-Robocopy\Disk %DSK%\%%~F" ^
/e /mir /np /a-:sh /copy:DATS /dcopy:T ^
/z /r:5 /w:2 /mt:8 ^
/xd "$RECYCLE.BIN" "System Volume Information" "RECYCLER" "Temporary Files" "Config.Msi" ^
/xf "pagefile.sys" ^
/log+:%DSK%2%NEW%_RoboCopy.log >> Robolog.txt
echo ------------------ >> Robolog.txt
echo. >> Robolog.txt

robocopy "%DSK%:\%%~F" "%NEW%:\TP-Robocopy\Disk %DSK%\%%~F" ^
/e /mir /np /a-:sh /copy:DATS /dcopy:T ^
/z /r:5 /w:2 /mt:8 ^
/xd "$RECYCLE.BIN" "System Volume Information" "RECYCLER" "Temporary Files" "Config.Msi" ^
/xf "pagefile.sys" ^
/log+:%DSK%2%NEW%_RoboCopy.log
)