Recent Wildly Fluctuating Copy Speeds of Large Files (125+ GB) to External Hard Drives

garioch7 · Nov 4, 2023

I conducted some further testing, copying the Macrium Reflect backup image file from the new 4 TB WD Black internal HDD drive to the existing 2 TB Seagate Barracuda internal HDD. Copy speeds fluctuated between 140 MB/s to 210 MB/s. I would not have expected that much copy speed fluctuation after the first 10 GB of the file was copied because the image file is a single file (134 GB).

Perhaps this is a new Windows "feature." If so, it is unwelcome, in my view.

It looks like I might just have to live with this copy speed fluctuation. Overall transfer speed is about the same to my external HDD: 100 MB/s, plus or minus 10 MB/s, as it was before the copy speed fluctuations started a couple of months ago. Then the copy speeds only fluctuated about 20 MB/s during the copy operation.

Have a great day.

Regards,
Phil

hsehestedt · Nov 5, 2023

I think that your methodology is flawed. Rather than watching the variation from moment to moment, take a nice large file(s) to avoid the overhead associated with small files, and time the whole transfer. Then calculate what that works out to per second. As a completely fake set of numbers for an example, if you transfer 100GB in 100 Seconds then 100 GB divided by 100 Seconds gives you 1 GB per second.

Berton · Nov 5, 2023

And I always wonder if b/bits is involved or B/Bytes is involved. 8 bits = 1 Byte. When I got my first computer 30 years ago it took a couple of chastisements to get me to remember the difference. Most times speed references are for bits.

BamaInArk · Nov 5, 2023

hsehestedt said:
I think that your methodology is flawed. Rather than watching the variation from moment to moment, take a nice large file(s) to avoid the overhead associated with small files, and time the whole transfer. Then calculate what that works out to per second. As a completely fake set of numbers for an example, if you transfer 100GB in 100 Seconds then 100 GB divided by 100 Seconds gives you 1 GB per second.

If you read the OPs topic post he stated even with all the variations in copy speed indications the overall time to copy was basically the same. So in reality this appears to be some kind of varied displayed speed indication issue not a total time to copy.

hdmi · Nov 5, 2023

All my HDDs are Conventional Magnetic Recording (CMR), but AFAIK it is completely normal for Shingled Magnetic Recording (SMR) HDDs to tend to show astronomical fluctuations in write performance so, if previously these fluctuations were invisible and all of a sudden something magical happened that caused them to be unhidden from you, you should actually be thankful IMO.

For HDDs I always use HD Tune Pro to get an accurate idea of how fast they can perform.

As for copying large amounts of data, I find that TeraCopy is still a bit slow when compared to FastCopy so I never use TeraCopy even though I know that some will prefer the GUI of TeraCopy (version 2.3). FastCopy has other benefits like not flaking out when copying very large amounts of data with it (I've witnessed TeraCopy flake out on multiple occasions in this regard...), really perfect logging capabilities, a robust verify option and the option to calculate and write file hash codes to the file log that it creates.

hsehestedt · Nov 6, 2023

On this topic of copying files, there is one thing that I really wish they would implement in Windows:

Have you ever started a large copy operation, then started another copy operation between the same source and destination drives? On spinning disk drives that causes a huge amount of thrashing and makes the operations SO much slower.

As an example, suppose you have two different copy jobs that each would take 10 minutes on their own. If you run one job, and then run the other job right after that would take 20 minutes. But if you start both jobs simultaneously, rather than taking 20 minutes it will take multiple times that amount of time as the hard drives thrash between non-contiguous data.

So, my great wish is that Microsoft would let you queue the copy jobs in File Explorer so that they are queued rather than running at the same time.

The great irony is that Microsoft is already 95% of the way there. When they added the ability to pause an operation in File Explorer, how difficult would it have been to add an option to automatically unpause that job as soon as the previous job completed? I keep hoping for that one tiny enhancement, but version after version it just never happens.

I know that for some persons this may be a complete non-issue, but I move large amounts of data around so it would be a great option for me. To work around I just have to script everything rather than use the GUI.

Berton · Nov 6, 2023

That would be an idea to put in the Feedback Hub, Start, All apps.

jimbo45 · Nov 6, 2023

hsehestedt said:
On this topic of copying files, there is one thing that I really wish they would implement in Windows:

Have you ever started a large copy operation, then started another copy operation between the same source and destination drives? On spinning disk drives that causes a huge amount of thrashing and makes the operations SO much slower.

As an example, suppose you have two different copy jobs that each would take 10 minutes on their own. If you run one job, and then run the other job right after that would take 20 minutes. But if you start both jobs simultaneously, rather than taking 20 minutes it will take multiple times that amount of time as the hard drives thrash between non-contiguous data.

So, my great wish is that Microsoft would let you queue the copy jobs in File Explorer so that they are queued rather than running at the same time.

The great irony is that Microsoft is already 95% of the way there. When they added the ability to pause an operation in File Explorer, how difficult would it have been to add an option to automatically unpause that job as soon as the previous job completed? I keep hoping for that one tiny enhancement, but version after version it just never happens.

I know that for some persons this may be a complete non-issue, but I move large amounts of data around so it would be a great option for me. To work around I just have to script everything rather than use the GUI.

One of the problems is the poor or very small cache memory in current disc controller chips and even in the disk hardware itself. There might be some optimisation parameters somewhere to minimise disk I/O overhead - again though it's "risk to reward" as some options might provide minimal tolerance against failure.

If you DO though move regularly large amounts of data then provided you have reliable disks and backup regularly try using RAID 0 -- can be very fast - but remember if you lose the array all the data is lost. RAID 0 works like JBOD (just a bunch of disks) where data is aggregated from the disks in the array to "one big disk" with disk I/O balanced across the devices. Modern RAID systems also means you can aggregate disks of any size -- although whether Windows allows that now --not sure. Windows used to require all disks to be of the same data capacity -- that of the smallest one with space over not used. Sensible RAID controllers should allow different capacity disks. Linux has software raid too (MDADM) which works a treat if you are using a Linux NAS type device,

cheers
jimbo

garioch7 · Nov 6, 2023

Thank you all for your comments. The overall time to copy is the same. On average, over the entire copy operation of one large single image file, I am averaging between 100 MB/s to 110 MB/s to external WD hard drives. Before this problem surfaced a couple of months ago, my transfer speeds with both Explorer and Teracopy were relatively stable, between 100 MB/s and 110 MBs. Now the speeds are up and down over the duration of the copy from 3.4 MB/s to 240 MBs.

Source and destination drives have all passed extended testing. SFC and DISM scans are clean.

So, I remain baffled as to what is causing that.

Thanks again, and have a great day.

Regards,
Phil

johnlgalt · Nov 7, 2023

I'm going out on a limb here, but I'm betting that the system has somehow started giving more instantaneous data transfer rates, whereas before it was giving more 'average' DTRs - or, at the very least, was polling less often when doing so.

I mentioned this in one of my previous posts, but since the overall time to copy remains the same, the actual DTR is for all intents and purposes not changing, so I suspect it is simply a change in the way the Windows (Explorer) is reporting it, or measuring what it reports.

garioch7 · Nov 8, 2023

@johnlgalt

You could well be correct. I sure don't have a better hypothesis. I am out of ideas.

Thanks for your assistance. Have a great day.

Regards,
Phil

johnlgalt · Nov 8, 2023

It's been a fun ride, trying to figure this out - I wonder if @Brink could do some sleuthing with the MVPs to find out if my hypothesis is correct?

garioch7 · Nov 8, 2023

@johnlgalt

That would be great! As you know, I spent a lot of time and money trying to figure this out. Oh well, I now have a new 5 TB WD My Passport Ultra external drive and a new 4 TB WD Black internal drive. They are not going to waste. And I know all my drives are in great shape. All internal and external drives have been thoroughly tested, and I know that my SFC and DISM results are clean.

I like a mental challenge, but as I have said before, I prefer it when it is another user's computer!

I will keep tinkering and researching periodically to see if an answer surfaces.

Thanks again for your assistance. We walked quite a road . . .

Have a great day.

Regards,
Phil

johnlgalt · Nov 8, 2023

garioch7 said:
I like a mental challenge, but as I have said before, I prefer it when it is another user's computer!

One of the reasons I continue to help troubleshoot here and other places -it's not (usually) mine!

garioch7 said:
I will keep tinkering and researching periodically to see if an answer surfaces.

That would be fantabulous. I'm just as curious to see if my theory pans out or not.

Have a great day as well!

memo09 · Nov 8, 2023

I had a similar issue. For me, the cause was that the folder to which I copied the big file had set the 'compress map contents to reduce diskspace' bit set. Please click on the destination map where you experience slow copy speeds , select properties , advanced, and be sure to clear the option to let Windows compress the contents of the map where the file is to be copied.

garioch7 · Nov 8, 2023

@memo09

I just checked one of the WD My Passport Ultra drives, as you suggested. No compression is on. I never turn on compression or encryption because it just adds another obstacle to successful data recovery, if necessary, and my drives have ample space.

I will check the other drives on Friday when I do my backup copies, but I would be astonished to see compression turned on because I would never do that.

Thank you for your suggestion, though. I remain optimistic that, eventually, we will find out the cause of the copy speed variations.

Have a great day.

Regards,
Phil

garioch7 · Nov 10, 2023

@johnlgalt

I found one advantage to my internal 5 TB WD Black HDD. It reduced the Backup and Check Image times with Easeus Todo Backup Home by 15% in comparison to backing up the internal 2 TB Seagate Barracuda drive.

Have a great weekend.

Regards,
Phil

garioch7 · Nov 10, 2023

The same copy speed variations persist. When I got my new 4 TB internal WD Black drive, it was (and still is) making regular clicking noises, which baffled me. Turns out it is a known issue with these drives. HD Sentinel Pro reports the health of the drive at 100%. Here is just one link:

WD Black 4TB clicking noises.

Hello, I recently bought and installed a WD Black 4TB (WD4005FZBX ) and there's some "light" clicking noise coming from it in intervals of time ranging from 5 seconds up to 25 seconds. It is constant(never goes more than 25 seconds) and it doesn't t matters if I'm using the drive or if it's...

forums.anandtech.com

I have Hard Disk Sentinel Pro installed, and it is constantly pinging and monitoring all connected drives. Couple that with the fact that the new WD Black drive is mounted in a bare metal HDD cage (no plastic caddy) and screwed in would mean that any drive noise would be amplified.

I ran WD Dashboard short and long tests on it, and it passed with flying colours. If it had not, I would have sought an RMA from WD.

I know that some here are fans of HD Tune, for which I have an old license, but I find HD Sentinel Pro to be better (personal choice).

So, another weekend and the mystery remains . . .

Oh, and I did check: both source and destination folders and drives have no compression enabled.

Have a great weekend.

Regards,
Phil

garioch7 · Nov 23, 2023

I first posted about this issue on 2023-09-30. I have had lots of good suggestions that I really appreciated. I do realize that posting in multiple Forums about the same issue is a breach of Forum "netiquette," here and elsewhere. I am not abandoning this topic, but I do plan now to post in the Windows 11 Forum at Bleeping Computer (my cyber "home") to see if any of the members there have any ideas that have not been discussed or tried here. I would like to solve the mystery. If I ever do, I will update this topic with the solution. I will not be posting anywhere else about this issue other than at Bleeping Computer.

Thank you again for all of your expert assistance. I am truly grateful to all those members here who tried to assist me.

I will be continuing to participate at ElevenForum, as I have been since I joined. There is a wealth of great information here.

Have a great day, one and all.

Regards,
Phil

garioch7 · Nov 23, 2023

As a gesture of courtesy and respect for all of those who assisted me here, I am providing the link to my Bleeping Computer topic. It is currently in the Windows 11 Forum, but the Moderators there might move it to the External Hardware Forum since the target drives are external hard drives.

Wildly Fluctuating Copy Speeds When Copying Large Image Files - Windows 11

Thank you again. Have a great day.

Regards,
Phil