Recent Wildly Fluctuating Copy Speeds of Large Files (125+ GB) to External Hard Drives

garioch7 said:

I concur that it is might be an issue with the USB ports or drivers. It would very odd for them to suddenly start misbehaving after years of consistently fine performance, . . . unless I got sideswiped by a Windows update. If that were the case, I would expect to be seeing posts from many other users complaining of the issue.

Click to expand...

This is not accurate thinking.

A Windows update might only affect your USB component because of various software you have installed and use, not just because of the system manufacturer, chipset, and / or drivers used. So expecting there to be a large public outcry about an issue that you're experiencing is not a direct correlation.

Even something as random as Windows updating your UEFI firmware could make a difference, and the reason there may not be a public outcry is that others have already cried out about this in the past, and thus most people using systems similar to yours have blocked said updates.

Definitely a good idea to test internally with a drive to rule out entire system bogging down, and focus on USB, and as more than a couple of us has suggested, rule out USB hubs, the USB port in use, and more.

@johnlgalt

I am grateful for your explanation. Thank you. Unfortunately, I am quite busy until Friday afternoon, when I hope to run some more tests, as have been suggested here by you and other contributors: trying different USB ports, installing a SATA drive into the computer to test copy speeds, disabling my security software, etc. While overall copy speed is the same in terms of time to complete a large copy, the fluctuations are a mystery that I would love to solve. Indeed we must determine conclusively whether this is strictly a USB external drive phenomenon to be able to chart the direction of further testing.

I am grateful to all who are assisting me to identify the root cause of this issue. Thank you. I will be back with a report on Friday with the results of my testing.

Have a great day.

Regards,
Phil

JAM83 said:

Quoting doesn't really take up that much room on the server. It's KBs. Not MB or GB. . Just small KB. An image file is bigger than a quoted post

Click to expand...

Have you ever tried following a post using your cell phone? I don't know about you but I hate having to scroll down a long list to continue reading the thread. There is more involved than just the amount of space used on the server. Not only that but it's not needed if your only replying to a part of the post.

Winuser said:

Have you ever tried following a post using your cell phone? I don't know about you but I hate having to scroll down a long list to continue reading the thread. There is more involved than just the amount of space used on the server. Not only that but it's not needed if your only replying to a part of the post.

Click to expand...

Nope, i don't visit forums using a mobile device, because they aren't mobile friendly.

JAM83 said:

Nope, i don't visit forums using a mobile device, because they aren't mobile friendly.

Click to expand...

I find this forum to be very mobile friendly.

Tenforums, on the other hand, is mobile hostile.

In all fairness I'd edit my offending post and delete all the quoted text but I can't find anywhere to "edit" it. Maybe I'm missing something?

Edit button is right under the My Computer link in your posts.

No Edit button option showing up. There was an Edit option on my above post after I posted it but it's not there anymore either. Time limit to edit? Or something wonky with my profile?

No time limit that I know of.

I can still edit the first post I made in this thread on Saturday.



Try force refreshing the page, or else closing your browser and then opening it again. May be a temporary glitch.

Or, there might be a limit on minimum number of posts before you're allowed to edit, like 100, which you're not quite at.

Must be a minimum post level limit. My last post to you is still showing an "Edit" option however the one above is missing.

BamaInArk said:

Time limit to edit? Or something wonky with my profile?

Click to expand...

As I understand it there is a time limit on editing a post, with the limit would be shortest for the 'New Member' group, longer for those in higher groups like johnlgalt in the 'Pro User' group.

@johnlgalt Your VIP status allows editing without any time limit.

I just figured that out - I went to a post over 2 years old and I could edit it.

Sorry, @BamaInArk I had forgotten about that.

Thanks @Bree and @LesFerch for confirming.

garioch7 said:

I use Teracopy Pro as my main copy engine. For years, I have been achieving a steady copy speed of 105 MBs, plus or minus 15 MBs. About five weeks ago, my copy speeds, after the first 7 GB of the image file was copied, started fluctuating wildly from 220 MBs to 3.8 MBs, up and down repeatedly. The average copy time is the same: about 24 minutes, but the cause of the copy speed fluctuations is a mystery to me. I also tried using the native Windows copy command from File Explorer, and it, too, had the same wild copy speed fluctuations.

Click to expand...

This behavior starting out of the blue seems odd. With a single large file copy, I always get the behavior you originally posted about. I did some testing at home tonight. I have a Gen 4 NVMe drive with a bunch of virtual machines, and I have an 8TB WD Elements external mechanical drive connected via USB 3 on my PC.

The default behavior when I connect the drive is for it to be set at Quick removal. This says that it's NOT going to buffer the writes to RAM and then dump to the external hard drive, but rather it's going to just copy to the external drive directly as fast as the physical drive can take it. I copied a nearly 30GB VHDX file to my external, and the speeds started out around 135MB/sec, dropped a bit to around 95MB/sec a couple of times right in the first 10%, and then it stayed pretty much between 135MB/sec and 140MB/sec for the remainder of the file. It took about 3:30 to copy the file. Here is what it looked like from File explorer:



I then changed the policy on the external drive instead to Better Performance and enable write-caching. So now, my NVMe can dump to my RAM and then dump to the external as the external as the external is available. When I kicked this one off, it started out at 200MB/sec+, and then shot up to something like 1.4GB/sec (which OBVIOUSLY my USB 3 spinning hard drive could not possibly do), and then it settled back down to around 146MB/sec. Copying this same source file, it showed the file copy was done at 2:20. Sounds great, went from 3:30 to 2:20....much better performance. However, watching disk writes via Task Manager showed that after windows "said" the file copy was "done", it was still writing to the external drive in the background. It wrote for about 1:08 after that. So, 2:20 + 1:08, is 3:28.......or pretty much just about the same speed as the quick removal option. Here is what it looked like from File explorer:




Any chance that your USB based mechanical hard drive is badly fragmented, resulting in the disk head having to move around a lot to situate files?

Hi garioch7... when I'm transferring a lot of files, when the transfer hits small files the speeds drop a lot. Has something to do with disk access... I'm not the guy to explain the technical aspect, and not sure if this is what you are experiencing, but it seems pretty common. Also, can have to do with your disk cache filling up, from what I have read. Hope you are able to figure it out.

Drybonz said:

Hi garioch7... when I'm transferring a lot of files, when the transfer hits small files the speeds drop a lot. Has something to do with disk access... I'm not the guy to explain the technical aspect, and not sure if this is what you are experiencing, but it seems pretty common. Also, can have to do with your disk cache filling up, from what I have read. Hope you are able to figure it out.

Click to expand...

In the OP case, he's copying large backup files....not lots of little files. Otherwise that would have been my suspicion too, but he notes after copying 7GB of the source file, he starts to encounter these problems.

I have completed my testing. I tried the other USB-C port on the back of the Dell 8930 SE computer and another USB 3.1 port on the back of the computer, with a USB-C to USB 3.1 adapter. Same issue.

I disabled all of my security software and tried again. No change.

I thought that I had spare HDDs, but it turns out that they went into the garbage at the time of the divorce several years ago. I was brutal in discarding stuff that I thought I would never use again. The package that I had seen wrapped downstairs in the basement contained two full-size LG optical drives (a BD and a DVD burner), not HDDs, which is what I thought the package contained.

I then tested copying the backup file from its original folder on the D: drive to another folder on the D: drive. The speeds were slow (40 MBs, plus or minus 10) but consistent, which is to be expected when the HDD 7200 rpm drive is reading and writing to itself. I then tried copying the Macrium 134 GB backup file on the D: drive (Seagate 2 TB HDD) to my C: drive, which is an Intel NVMe SSD. I achieved consistent transfer speeds of 200 MBs, plus or minus 15, using it.

I think that I am safe in assuming that the problem lies with the USB ports/drivers on my computer, or that some software on my computer is interfering with large file transfers to USB external drives.

I am very open to further suggestions. I apologize that it has taken me so long to resurrect this topic, but I have a lot on my plate.

Thank you all for your previous suggestions. They were great. I can live with this issue, but I would really like to find a solution.

Have a great weekend, Thanksgiving weekend in Canada.

Regards,
Phil

garioch7 said:

I have completed my testing. I tried the other USB-C port on the back of the Dell 8930 SE computer and another USB 3.1 port on the back of the computer, with a USB-C to USB 3.1 adapter. Same issue.

I disabled all of my security software and tried again. No change.

I thought that I had spare HDDs, but it turns out that they went into the garbage at the time of the divorce several years ago. I was brutal in discarding stuff that I thought I would never use again. The package that I had seen wrapped downstairs in the basement contained two full-size LG optical drives (a BD and a DVD burner), not HDDs, which is what I thought the package contained.

I then tested copying the backup file from its original folder on the D: drive to another folder on the D: drive. The speeds were slow (40 MBs, plus or minus 10) but consistent, which is to be expected when the HDD 7200 rpm drive is reading and writing to itself. I then tried copying the Macrium 134 GB backup file on the D: drive (Seagate 2 TB HDD) to my C: drive, which is an Intel NVMe SSD. I achieved consistent transfer speeds of 200 MBs, plus or minus 15, using it.

I think that I am safe in assuming that the problem lies with the USB ports/drivers on my computer, or that some software on my computer is interfering with large file transfers to USB external drives.

I am very open to further suggestions. I apologize that it has taken me so long to resurrect this topic, but I have a lot on my plate.

Thank you all for your previous suggestions. They were great. I can live with this issue, but I would really like to find a solution.

Have a great weekend, Thanksgiving weekend in Canada.

Regards,
Phil

Click to expand...

Older drives might be USB2 - that would explain low transfer rate times. Some mobos also if you have any USB2 devices on the same port using a multi-hub usb connector or even the same mobo bus as a USB3 device e.g wireless mouse / keyboard then the speed will drop to the lower one e.g USB 2. Depends on the mobo etc -- modern hardware though will allow USB 2 and USB 3 devices to co-exist and operate at their native speeds. Just an idea - might not be your problem but it could be.


Cheers
jimbo

@jimbo45

Your insights are helpful, but the issue is not really the overall transfer speed. It is the wild fluctuations in transfer speed when copying large files to an external USB HDD. As my original post stated, I have had this computer since October 2019. Until six weeks ago, the transfer speed was a consistent 110 MBs, plus or minus 10 MBs, during those copies. That minor speed fluctuation is within reason. It never caused me any concern, but speeds that repeatedly fluctuate up and down from 3.0 MBs to 220 MBs are not reasonable.

I had really hoped it was my front USB 3.1 and USB-C ports that had gone flaky, but connecting to the rear ports yesterday produced the same results.

I am open to any further insights. Thank you, and have a great day.

Regards,
Phil

Just a few additional notes, I checked to determine the degree of fragmentation on the D: HDD, if any. That was suggested as a possible cause. The Windows Defragmeter and Optimizer app stated 0% fragmentation. I ran it anyway. It finished in under 5 minutes, so I am going to eliminate that as a possible cause. Going to run a Check Disk now on the drive and then run Seatools as well (it is a Seagate drive).

Thank you all for your interest. Have a great day.

Regards,
Phil