When copying large files to/from NVME drives, CPU spikes and copy performance plummets

lunadesign · Jul 12, 2024

I've got a fairly high-end 24 core Windows 11 workstation (see specs) currently on version 23H2 22631.3737.

When I use File Explorer to copy a large (6.7 GB) file from one NVMe drive to another (or to the same drive), the transfer initially starts around 3-4 GB/s then plummets to 150 MB/s and stays there for the rest of the transfer.

In Task Manager's "Performance" tab, the CPU goes to 50% during the 150 MB/s portion of the transfer. And in the "Processes" view, the top 10 or so processes will be far higher than usual even though the processes have nothing to do with the file transfer. But curiously, in Task Manager's "Details" tab sorted by CPU usage, "System Idle Process" stays around 98% during the slow portion.

After I do this test a handful of times, most of the transfer is stuck in the 150-200 MB/s range.

I've tried this with antivirus off and it still happens.

However, if I use CrystalDiskMark 8, the SEQ1M Q8T1 test on each drive reliably shows 7.4 GB/s read and 6.8 GB/s write on each of the NVMe drives.

FWIW, what I'm seeing seems very similar to what this user posted at In Windows 11, when copying files, CPU usage abnormally approaches 100%. - Microsoft Q&A.

Any ideas what's going on here? This is driving me crazy!

cereberus · Jul 12, 2024

This is quite normal - it just means the copy buffering cache is full up. It is all bit of an illusion really - copy cache fills up faster than the pc can write data to drive. Eventually, the cache fills up and the data can only write as fast as it released.

Overall, it is the average time that is important.

lunadesign · Jul 12, 2024

cereberus said:
This is quite normal - it just means the copy buffering cache is full up. It is all bit of an illusion really - copy cache fills up faster than the pc can write data to drive. Eventually, the cache fills up and the data can only write as fast as it released.

Overall, it is the average time that is important.

If I was copying between two hard drives, I think your statement would be accurate. The copy performance graph does look like that in those cases.

But I'm copying from one NVMe drive to another. They should be order of magnitude faster than 150 MB/s drive-to-drive without *any* cache assistance.

Also, your theory doesn't explain:
a) Why it would take 50% of a 24 core CPU to copy this data.
b) Why after a handful of tests, the copying performance is stuck around 150 MB/s. Hours later, I can't get that 3-4 GB/s performance (but I suspect I would for a while after a reboot).

lunadesign · Jul 12, 2024

cereberus said:
This is quite normal - it just means the copy buffering cache is full up. It is all bit of an illusion really - copy cache fills up faster than the pc can write data to drive. Eventually, the cache fills up and the data can only write as fast as it released.

Overall, it is the average time that is important.

With regards to averages, I just used a stopwatch and timed a nearly 6.8 GB file taking 49 seconds to copy. That's an average of 135 MB/s. So this isn't a case of the UI presenting misleading information. The copying is really that slow.

Berton · Jul 12, 2024

It's been that way since the days of the floppy drives then the Zip drives. What I get is that copying reads the source data, writes the source data into a temporary folder then writes that data to the destination followed by deleting what is put in that temporary folder. It's not as quick as desired. Moving can be a bit quicker but more dangerous as it's lost if the computer has a problem such as power failure.

lunadesign · Jul 12, 2024

Berton said:
It's been that way since the days of the floppy drives then the Zip drives. What I get is that copying reads the source data, writes the source data into a temporary folder then writes that data to the destination followed by deleting what is put in that temporary folder. It's not as quick as desired. Moving can be a bit quicker but more dangerous as it's lost if the computer has a problem such as power failure.

I'm pretty sure this is not the case on a drive-to-drive copy on the same system. But even if it were, this would not slow down drives capable of doing 6800 MB/s to 150 MB/s.

neves · Jul 12, 2024

Try this, see if it helps: Optimize SSD Performance with 4K Alignment Software and Speed Up PC

LE. Then again, that feature is not free. Try the next guide - at least, to see if your disks are 4k aligned:

How to Carry out SSD 4K Alignment to Make SSD Perform Better?

Maybe it’s time for SSD 4K alignment when the PC’s reading and wring speed is so slow. In this article, you’ll know what the alignment is and how to commit it.

www.diskpart.com

LE2: GParted can do that too automatically, but the trial version of MiniTool Partition Wizard Free 12.8 - should also work within Windows.

Wisewiz · Jul 12, 2024

Note: I get the same slow-down with Windows' native copy service with large files. But with FastCopy, I just copied an 18.5GB file from an internal NVMe to an EXTERNAL NVMe through a USB 3 port in 18 seconds. If I copy multiple large files with FastCopy, there's a slow-down after a while, but nothing at all like the Windows copy slow-down.

FastCopy

FastCopy Official site

fastcopy.jp

Give it a try. It's free and needs no installation, and it's Japanese.

Wisewiz · Jul 12, 2024

Just did a second, third, and fourth in 18, 20, and 18 seconds, respectively. Same drives, roughly the same size compressed Macrium Reflect images.

Catnip · Jul 12, 2024

I have the same problem as the OP. When I copy from NVMe to NVMe drive it goes fast for about three seconds and then slows down to about 150 MB/s. It's just the caching and I accept that, but it is kind of silly. You would think that with today's advanced hardware, the Windows copy/move function would have kept pace with advancements in technology. That doesn't see to be the case.

As to the OP's problem: There isn't much we can do about it, we just have to live with it. I am trying Fast Copy to see if it helps when I get home.

Wisewiz · Jul 12, 2024

Best tests for it are large files (10-20GB) and internal to external, not internal to internal, which is quite naturally faster.
You'll find the Japanese devs update it fairly frequently, and it's fast and automatic: just open FastCopy and after a couple of seconds, it'll disappear and reappear with an update shown in the title bar. That's fast and efficient, like FastCopy itself.

BamaInArk · Jul 12, 2024

I had to test my system for a comparison. I copied a 6 GB file between 2 Gen 4 NVMe drives. It copied so fast I wasn't even able to see the transfer speeds. So I copied a much larger 42 GB file and my copy speed was showing in the GB range(not MB range). With Task Manager open to CPU utilization only went up to around 30%.
I do find that the transfer speed graph seems to change everytime I do a large transfer. This one shows the most recent test. The first time I test this larger file it was completely different but still high.

Wisewiz · Jul 12, 2024

Bama, Not sure I understand you. Are you getting these results with native WinCopy or with third-party software?

cereberus · Jul 12, 2024

lunadesign said:
If I was copying between two hard drives, I think your statement would be accurate. The copy performance graph does look like that in those cases.

But I'm copying from one NVMe drive to another. They should be order of magnitude faster than 150 MB/s drive-to-drive without *any* cache assistance.

Also, your theory doesn't explain:
a) Why it would take 50% of a 24 core CPU to copy this data.
b) Why after a handful of tests, the copying performance is stuck around 150 MB/s. Hours later, I can't get that 3-4 GB/s performance (but I suspect I would for a while after a reboot).

It is the buffer that is the issue. I get the same copying between nvme drives.

BamaInArk · Jul 12, 2024

Wisewiz said:
Bama, Not sure I understand you. Are you getting these results with native WinCopy or with third-party software?

Yes using the standard Windows File Explorer Copy. Dragging and dropping.

volfy · Aug 30, 2024

Bama, were you able to figure out what's going on? I started experiencing the exact same issue you described (not sure if it started happening after a recent Windows update or a recent Asus bios / Intel microcode update).

Basically, when copying a large file (about 5GB) from one NVME SSD to another on my computer (both Samsung 990 Pros with plenty of empty space, so cache issues shouldn't be the cause), the transfer speed crawls (300-400mb per second, as opposed to > 4gb per second I should be getting), and the CPU usage spikes to 100%. I checked the task manager and it's the System process that's taking all the CPU.

I noticed that the problem goes away after rebooting the computer, only for it to come back soon (possibly after a few sleep/wake cycles, but I can't confirm if that's what triggers the issue).

cereberus · Aug 30, 2024

lunadesign said:
If I was copying between two hard drives, I think your statement would be accurate. The copy performance graph does look like that in those cases.

But I'm copying from one NVMe drive to another. They should be order of magnitude faster than 150 MB/s drive-to-drive without *any* cache assistance.

Also, your theory doesn't explain:
a) Why it would take 50% of a 24 core CPU to copy this data.
b) Why after a handful of tests, the copying performance is stuck around 150 MB/s. Hours later, I can't get that 3-4 GB/s performance (but I suspect I would for a while after a reboot).

The problem is MSs copy algorithms are ancient and helped back in day of slow HDDs. MS need to rewrite them to take account of modern drives, use multitasking etc.

Every gets same issue - nothing helps.

BamaInArk · Aug 30, 2024

volfy said:
Bama, were you able to figure out what's going on? I started experiencing the exact same issue you described (not sure if it started happening after a recent Windows update or a recent Asus bios / Intel microcode update).

I'm not the OP and not seeing the same issues as shown by the graph I posted. Also mine is an AMD(AM4) system so the Intel microcode update doesn't apply in this systems specifically. I do have a late gen Intel system I can test on however. But as has been pointed out there appears to be issues with the way Microsoft handles file copy operations that needs tweaking.

hsehestedt · Aug 30, 2024

I came across this thread and thought that I would test this on the laptop that I am currently in front of. This is a pretty ancient HP Spectre x360 laptop from roughly 2017. It has a dual core i7-7500U CPU. It's been upgraded from 16GB RAM to 32GB and the 512GB NVMe SSD has been replaced with a Seagate FireCuda 1TB NVMe SSD.

Even running on battery, I'm seeing way better performance:

Note that this was a larger data set of 33.5GB and I am using the same drive as both source and destination. As for my CPU, it gets pegged at 100% utilization for the bulk of the copy but drops down to somewhere around 45% during the last few seconds of the copy.

Oh yeah, I should also point out that this drive is BitLocker encrypted. I mention this only because I suspect that this introduces additional overhead.

Other than the hardware, the only big difference is that I am running on Win 11 26100.1591 and of course the BitLocker thing.

So, while I don't know the cause of the problem, this tells me that this is decidely not normal, especially since my ancient hardware is not having this issue even on a copy of several times the size of your test.

While I don't suspect that this is the issue, or not the entire issue at least, have you verified that your SSD has been optimized recently?

PerpetualCycle · Sep 1, 2024

What make and model is the destination drive and how is it cooled?

What is the drive temperature(s). When this happens. There may be 2 temperatures reported, one for the data storage NAND and one for the internal controller, which can get very hot on nvme drives especially on writes.. The controller will start slowing down the transfer speed in order to reduce the temperature when it crosses over a threshold.

Also nvme SSD usually have two speeds of data NAND chips, a smaller amount of faster NAND that is used as a buffer fior writes, and the rest slower for storage, designed such that the buffer can cover most transfer sizes at full speed. When you write a file that is bigger than the buffer, once it gets full, the speed will drop to that of the slower nand. One the write completes the data from the buffer gets copied to storage in the background.

The only type of nvme ssd that would drop to around 150 MB/s would be one with really slow QLC NAND.

When copying large files to/from NVME drives, CPU spikes and copy performance plummets

Member

My Computer

Well-known member

My Computer

Member

My Computer

Member

My Computer

Ret. ATC

My Computers

Member

My Computer

Well-known member

My Computer

𝐒𝐩𝐞𝐚𝐤𝐬 𝐀𝐧𝐜𝐢𝐞𝐧𝐭 𝐆𝐞𝐞𝐤

My Computers

𝐒𝐩𝐞𝐚𝐤𝐬 𝐀𝐧𝐜𝐢𝐞𝐧𝐭 𝐆𝐞𝐞𝐤

My Computers

Forum Nitwit

My Computer

𝐒𝐩𝐞𝐚𝐤𝐬 𝐀𝐧𝐜𝐢𝐞𝐧𝐭 𝐆𝐞𝐞𝐤

My Computers

Well-known member

Attachments

My Computer

𝐒𝐩𝐞𝐚𝐤𝐬 𝐀𝐧𝐜𝐢𝐞𝐧𝐭 𝐆𝐞𝐞𝐤

My Computers

Well-known member

My Computer

Well-known member

My Computer

New member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computers

Syzygy

My Computers