Solved Where are the speed bottlenecks?

Haydon · Feb 26, 2022

I copied a large folder containing many subfolders and files. Why is the speed going up and down? Is there something like a slowest component or a slowest process? Is there a way to determine what speed bottleneck it is that slows things down? That would provide a pointer as to what kind of upgrade(s) would speed up the machine, at least for the task at hand, would it not?

Malware scans may be subject to similar 'speed going up and down' and 'speed bottleneck' mechanisms. For example, when running a Defender full scan, the file names are flashing by with slow downs every now and then.

Other tasks may have different speed bottlenecks. Where/what are those speed bottlenecks?

pparks1 · Feb 27, 2022

Anytime you are dealing with lots of files, especially lots of small files, the transfer speed is going to bounce around like that. It's gotta commit lots of writes to the file allocation table and such.

When you right a large file, like an ISO or a ZIP, it should go substantially faster. Not really sure of any way to speed up that process, it's how it goes.

DigitalGoat · Feb 27, 2022

Try copying the same folders/ files with something like Robocopy with multi-thread switch and/ or zip them to see if there is an improvement.
In general though most media maintains higher throughput with large files rather than small files as there is less write overhead involved, when talking number of files involved.
Even fast media like NAND can suffer from lots of small writes, not because the actual writing to the cells takes time, but because of the overhead generated by a combination of the copy method (OS, application design) and device interface, from connection type through to the device controller and even it's firmware. Test 5 different manufacturers USB 3.0 flash drives on the same machine with the same test files and you would be amazed at the potential for difference in performance.
Spinner type hard disks show this best with sustained performance versus random due to rotational speed of the platters and the heads having to physically move to read/ write data.

wiganken · Feb 27, 2022

I agree. When I copy a single, large (30 to 50 GB) Macrium Reflect image file (mrimg extension) from my USB 3.1 connected external SSD to my PC desktop the copy speed is quite constant. See photo. If I copy mutiple smaller files then the copy speed fluctuates as others have said.

FreeBooter · Feb 27, 2022

Windows User Account Control (UAC) and Windows Defender can affect the copy speed as they are scanning copied files disabling them may help the speed.

x BlueRobot · Feb 27, 2022

As others have mentioned already, this is largely due to operating system design if were to consider the same operation running on the same hardware. However, one thing which might make some marginal difference is fragmentation, this isn't as prevalent as it was on older operating systems, although, it can affect the read performance and therefore may make the copy operation faster.

Haydon · Feb 27, 2022

wiganken said:
I agree. When I copy a single, large (30 to 50 GB) Macrium Reflect image file (mrimg extension) from my USB 3.1 connected external SSD to my PC desktop the copy speed is quite constant. See photo. If I copy mutiple smaller files then the copy speed fluctuates as others have said.
View attachment 23038

In the example given, what would it take to raise the speed from 300 MB/s to 400 MB/s ?

wiganken · Mar 1, 2022

Odd question but I suspect a faster CPU and a higher USB / USB bus spec would do it.

TheMystic · Mar 1, 2022

pparks1 said:
Anytime you are dealing with lots of files, especially lots of small files, the transfer speed is going to bounce around like that. It's gotta commit lots of writes to the file allocation table and such.

When you right a large file, like an ISO or a ZIP, it should go substantially faster. Not really sure of any way to speed up that process, it's how it goes.

In addition to what you have mentioned, I think the cluster size of the partition too plays a role. If the cluster size is higher, then it takes even longer to copy small files.

jimbo45 · Mar 1, 2022

@wiganken @x BlueRobot @pparks1
@Haydon and the rest

What one needs is a proper caching system for the Disk I/O subsystem. Most domestic I/O devices have a tiny cache so once it's full the computer essentially goes into a "Wait state" until the Disk I/O system has emptied its cache and more data transfer can proceed.

It's rarely a lack of CPU power that causes slow I/O. First the memory must be sufficient in the computer -- if it's not then a load more I/O's have to be done to the paging devices which slow down the system even more as those "pages" have to be retrieved and processed. However assuming you have enough RAM then you must have not only fast Disks but those must have decently large cache sizes too.

Finally the software must be sensibly written -- if it can "block" data in large chunks before writing to the output device then there will be a lot less I/O requests and transfer will go a lot quicker.

For example in the Linux dd (data / disk duplicator) comand you can specify a data transfer size (dd if=<file/partition/disk> of=<file/partition/disk> bs=xxxM status=progress where the bs parameter specifies the data chunk size.

"here endeth the 1st lesson " !!!!!

Cheers
jimbo

hdmi · Mar 1, 2022

First, try copying of files/folders with multithreaded read/write/verify, Overlapped I/O, Direct I/O, while avoiding the MFC bottleneck:

FastCopy

FastCopy Official site

fastcopy.jp

It's still faster than Robocopy, noticeably when you're dealing with very long tasks. (Other areas where Robocopy is lacking compared to FastCopy are things like, e.g., error logging of individual files/folders with proper support of special characters in the names of the files/folders that will appear in the log and the ability to let hash codes that can be calculated for file verification be saved, either in the log or in Alternate Data Streams.)

During the copy procedure, if other processes are accessing the same storage device(s), having a caching software solution can help to boost their performance additionally also, albeit there can be adverse effects, specifically in situations when e.g. the amount of RAM and/or SSD space that is consumed by the cache is what causes performane to degrade. These adverse effects can happen as a result from, e.g., other processes receiving a suboptimal amount of RAM space when the RAM cache is comparatively oversized (or, in the particular case of shared bandwidth with an SSD used for caching one or more slower drives such as HDDs and/or the types of SSDs that are comparatively slow e.g. due to their specs and/or due to their being held back, the SSD caching taking up too much bandwidth). It also is possible to use tiered caching and/or use a Ramdisk with or without Dynamic Memory Management (DMM Compacted) optional feature.

PrimoCache - Excellent Software Caching Solution to Accelerate Storage

Home page of PrimoCache product which is a supplementary software caching scheme to improve the system performance.

www.romexsoftware.com

Primo Ramdisk - Powerful Disk Emulator to Create Ultra-Fast RAM-Disks

Home page of Primo Ramdisk product which is a powerful disk emulator to solve hard disk bottleneck issue, speeding up your computer and improving the performance.

www.romexsoftware.com

This is all in addition to trying to use separate storage devices when possible so as to balance the load by dividing it across multiple hardware components, as part of a combined strategy. But the number of ports hardwired to a controller is limited, as is the total bandwidth capability of the controller, so bottlenecks and the various optimization techniques that exist to steer around them have a bit of a tendency to be multifactorial and complex─although a lot also depends on what the characteristics of the workload are like─so, here's another excellent example of a software product that effectively combines a whole number of different I/O optimization technologies:

DymaxIO

DymaxIO is fast data performance software that returns 40%+ of your throughput that is lost due to I/O inefficiencies. DymaxIO fixes I/O issues at source.

condusiv.com

pparks1 · Mar 1, 2022

Haydon said:
In the example given, what would it take to raise the speed from 300 MB/s to 400 MB/s ?

In that example, it was an SSD connected to a USB 3.1 port.

So, first things first....it's most likely a standard SATA based SSD...so likely going to be limited there to ~500MB/sec. (best case). If it's a less capable SSD, your speeds may be slower...like 240-400MB/sec. This could be the limiting factor.

Next, it's over USB 3.1. This likely means it's a USB 3.2 Gen 1 type device, which is limited to 5Gbps. In theory, math says that's about 640MB/sec as the max here. Generally speaking, about 85%-90% of max is what is achieveable as there is some overhead. So, think 540 to about 580MB/sec is what you can expect.

For myself, I bought a Mokin 10Gbps external NVMe enclosure (about $20), then I bought a Crucial P2 1TB NVMe (~$90) that is capable of about 2400MB/sec read and 1800MB/sec writes. My enclosure at 10Gbps allows about ~1280MB/sec max throughput. It plugs in via a USB 3.2 Gen 2 port, or USB-C port. I get about 1100MB/sec to this drive. That's my fastest external drive that I have.

Haydon · Mar 1, 2022

pparks1 said:
In that example, it was an SSD connected to a USB 3.1 port.

So, first things first....it's most likely a standard SATA based SSD...so likely going to be limited there to ~500MB/sec. (best case). If it's a less capable SSD, your speeds may be slower...like 240-400MB/sec. This could be the limiting factor.

Next, it's over USB 3.1. This likely means it's a USB 3.2 Gen 1 type device, which is limited to 5Gbps. In theory, math says that's about 640MB/sec as the max here. Generally speaking, about 85%-90% of max is what is achieveable as there is some overhead. So, think 540 to about 580MB/sec is what you can expect.

For myself, I bought a Mokin 10Gbps external NVMe enclosure (about $20), then I bought a Crucial P2 1TB NVMe (~$90) that is capable of about 2400MB/sec read and 1800MB/sec writes. My enclosure at 10Gbps allows about ~1280MB/sec max throughput. It plugs in via a USB 3.2 Gen 2 port, or USB-C port. I get about 1100MB/sec to this drive. That's my fastest external drive that I have.

That answers the issue of getting the fastest BackUp on an external medium

Now on to the Full Scans of Windows Defender, what causes it to be hours long (apart from the GBs) ? Can it be made minutes long (with a practical solution) ?

hdmi · Mar 1, 2022

Haydon said:
Now on to the Full Scans of Windows Defender, what causes it to be hours long (apart from the GBs) ? Can it be made minutes long (with a practical solution) ?

A lot of useful info about that subject can be pulled from this discussion thread:

Limit Windows Defender CPU Usage

I have the problem that our Clients use too much CPU during a FullScan. Actually, the usage is limited to 20%, but the setting seems to have no effect. Whether I set it via Configuration Manager or GPO, the result is the same. Does anyone have a similar problem or even better... a solution?

techcommunity.microsoft.com

pparks1 · Mar 2, 2022

Haydon said:
That answers the issue of getting the fastest BackUp on an external medium

Now on to the Full Scans of Windows Defender, what causes it to be hours long (apart from the GBs) ? Can it be made minutes long (with a practical solution) ?

So, i rarely run a manual scan with Windows defender, so I don't have a good baseline for understanding the timings. However, I did some testing today.

#1). It says right when you run the scan that it could take 30 minutes or longer. So unsure you are going to be able to drop it to just a few minutes.

I have a testbox, (a beelink mini pc), with a Core i5-8279u CPU that has 4 cores and 8 logical processors. It has a middle of the road NVMe drive running the OS (WIndows 11). The box is used for testing and playing, so it doesn't have a lot on it...only about 60GB used on the drive. Watching it (with Task Manager) while full scanning shows that my CPU is pretty much pegged at 100%. The usage on the NVMe is very low....it's bouncing around normally between 1% and 8%. About 10 minutes into my scan, it's at 400k worth of files scanned, the slider is about 1/4 of the way across and it shows about 22 minutes remaining. When I got to about 75-85% of the way across the slider on this system, I noticed that the CPU dropped off to around 20% on average, and the disk utilization dropped to nothing. (assume we are scanning memory resident files and processes at this point) The estimates for scanning time completion was way off at this point. The slider bar barely moved, and the time just kept going up. Started at 3 minutes then 4, then 6 minutes....and it's been running for 15+ minutes showing that.

On my main desktop, i'm running a Ryzen 9 5900x with 12 cores and 24 logical processors. It has a super fast NVMe (WD SN850 PCIe Gen 4). I have about 1.3TB of files on this box, so a lot more to scan. About 450GB worth is on a standard SATA SSD. Watching this one, my CPU shot up near 100% for about 30 seconds, and then I watched my SSD go up to about 100% usage for a around 30 seconds as well. Now my CPU has stablized out to around 25% and my NVMe is bouncing around between 15% and 80%. It's usually in the 25%-50% range. Occasionally seeing my CPU spike up to 80-90% and when that happens the SSD is also close to getting maxed out as well. When I got to about 75-85% of the way across the slider on this system, I noticed that the CPU dropped off to around 6% on average, and the disk utilization dropped to nothing. (assume we are scanning memory resident files and processes at this point)

On the Beelink mini-pc, the scan lasted 52 minutes and 5 seconds, and scanned 1,676,687 files scanned.

On my desktop, the scan lasted 2 hours and 26 minutes and scanned 4,173,733 files.

From what I can tell, the speed of the CPU and the # of cores seems to make the most difference in the full scan.

Haydon · Mar 2, 2022

pparks1 said:
So, i rarely run a manual scan with Windows defender, so I don't have a good baseline for understanding the timings. However, I did some testing today.

#1). It says right when you run the scan that it could take 30 minutes or longer. So unsure you are going to be able to drop it to just a few minutes.

I have a testbox, (a beelink mini pc), with a Core i5-8279u CPU that has 4 cores and 8 logical processors. It has a middle of the road NVMe drive running the OS (WIndows 11). The box is used for testing and playing, so it doesn't have a lot on it...only about 60GB used on the drive. Watching it (with Task Manager) while full scanning shows that my CPU is pretty much pegged at 100%. The usage on the NVMe is very low....it's bouncing around normally between 1% and 8%. About 10 minutes into my scan, it's at 400k worth of files scanned, the slider is about 1/4 of the way across and it shows about 22 minutes remaining. When I got to about 75-85% of the way across the slider on this system, I noticed that the CPU dropped off to around 20% on average, and the disk utilization dropped to nothing. (assume we are scanning memory resident files and processes at this point) The estimates for scanning time completion was way off at this point. The slider bar barely moved, and the time just kept going up. Started at 3 minutes then 4, then 6 minutes....and it's been running for 15+ minutes showing that.

On my main desktop, i'm running a Ryzen 9 5900x with 12 cores and 24 logical processors. It has a super fast NVMe (WD SN850 PCIe Gen 4). I have about 1.3TB of files on this box, so a lot more to scan. About 450GB worth is on a standard SATA SSD. Watching this one, my CPU shot up near 100% for about 30 seconds, and then I watched my SSD go up to about 100% usage for a around 30 seconds as well. Now my CPU has stablized out to around 25% and my NVMe is bouncing around between 15% and 80%. It's usually in the 25%-50% range. Occasionally seeing my CPU spike up to 80-90% and when that happens the SSD is also close to getting maxed out as well. When I got to about 75-85% of the way across the slider on this system, I noticed that the CPU dropped off to around 6% on average, and the disk utilization dropped to nothing. (assume we are scanning memory resident files and processes at this point)

On the Beelink mini-pc, the scan lasted 52 minutes and 5 seconds, and scanned 1,676,687 files scanned.

On my desktop, the scan lasted 2 hours and 26 minutes and scanned 4,173,733 files.

From what I can tell, the speed of the CPU and the # of cores seems to make the most difference in the full scan.

Considering that Windows Defender is cloud-assisted, while testing, were you connected to the Internet?

pparks1 · Mar 2, 2022

Haydon said:
Considering that Windows Defender is cloud-assisted, while testing, were you connected to the Internet?

Yes, always connected to the Internet

Haydon · Mar 3, 2022

I did a few tests myself and now believe that Windows Defender Full Scan needs the 30 minutes for the OS itself, plus extra time for apps and data.

I will mark this thread 'Solved', thanks to all who responded

Edit: Windows Defender Full Scan is very resource intensive, your apps barely work in spite of MS' "Feel free to keep working". IMHO, you can't run Windows Defender Full Scan in the middle of a busy work day.

hdmi · Mar 4, 2022

Haydon said:
Edit: Windows Defender Full Scan is very resource intensive, your apps barely work in spite of MS' "Feel free to keep working". IMHO, you can't run Windows Defender Full Scan in the middle of a busy work day.

If the Full Scan in question is a Manual Scan, then it translates to "Feel free to keep looking at how your high CPU load makes working impossible". Whereas if it is a Scheduled Scan, then depending on specs and the specific type/characteristics of the workload that defines a busy work day, you still might be able to pull it off somehow, i.e. by toying with the AvgCPULoadFactor. kek

Solved Where are the speed bottlenecks?

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computers

Well-known member

My Computer

Active member

My Computer

Well-known member

My Computer

Well-known member

My Computers

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Well-known member

My Computer

Similar threads