What are "Sparse files"

jimbo45 · Mar 23, 2022

Hi folks
bit confused as to the meaning of Sparse files. Google explanations not clear.
Anybody !!!

Thanks

Cheers
jimbo

Bree · Mar 23, 2022

jimbo45 said:
bit confused as to the meaning of Sparse files. Google explanations not clear.

Wikipedia seems clear enough. A sparse file is one where empty space is not stored in the file itself. A typical example of a sparse file would be an expanding .vhdx file for a Hyper-V VM.

The advantage of sparse files is that storage space is only allocated when actually needed: Storage capacity is conserved, and large files can occasionally be created even if insufficient free space for the original file is available on the storage media....

For example, a virtual machine image with max size of 100 GB that has 2 GB of files actually written would require the full 100 GB when backed by pre-allocated storage, yet only 2 GB on a sparse file.

Sparse file - Wikipedia

en.wikipedia.org

glasskuter · Mar 23, 2022

I didn't know either, but I think MS's explanation says it best.

"A file in which much of the data is zeros is said to contain a sparse data set. Files like these are typically very large for example, a file that contains image data to be processed or a matrix within a high-speed database. The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space.

The file compression in the NTFS file system is a partial solution to the problem. All data in the file that is not explicitly written is explicitly set to zero. File compression compacts these ranges of zeros. However, a drawback of file compression is that access time may increase due to data compression and decompression.

Support for sparse files is introduced in the NTFS file system as another way to make disk space usage more efficient. When sparse file functionality is enabled, the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data."

Edwin · Mar 23, 2022

Is not Sparse Files a method many Disk Cleaners use to perform a Drive Wipe?

Ghot · Mar 23, 2022

Sparse file

A file with all the "few and far between" sucked out of it.

Winuser · Mar 24, 2022

glasskuter said:
A file in which much of the data is zeros is said to contain a sparse data set. Files like these are typically very large for example, a file that contains image data to be processed or a matrix within a high-speed database. The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space.

Unless I'm missing something, it sounds like they are only good for wasting drive space.

Bree · Mar 24, 2022

Winuser said:
Unless I'm missing something, it sounds like they are only good for wasting drive space.

What you are missing is that a sparse file with a lot of empty space in it doesn't actually use any physical storage for the unused space.

For example, I can create a .vhdx file for a 10TB virtual drive. But as a sparse file its physical size is just 230MB until I start to write to it. It looks and behaves as a full 10TB file, and as I write real data to it then it will expand. But at around 180GB written I will not be able to add any more data, as it would have expanded to occupy all the available space of the C: partition where the file is located.

Winuser · Mar 24, 2022

Bree said:
What you are missing is that a sparse file with a lot of empty space in it doesn't actually use any physical storage for the unused space.

For example, I can create a .vhdx file for a 10TB virtual drive. But as a sparse file its physical size is just 230MB until I start to write to it. It looks and behaves as a full 10TB file, and as I write real data to it then it will expand. But at around 180GB written I will not be able to add any more data, as it would have expanded to occupy all the available space of the C: partition where the file is located.

View attachment 25253

I'm only going by what MS said. because of this, they are an inefficient use of disk space.

Haydon · Mar 24, 2022

sparse file = balloon file, it expands/shrinks with content/air you let in/out, it has been invented to be most efficient with physical resources

OP, please mark this thread 'Solved'

PerpetualCycle · Mar 24, 2022

Winuser said:
I'm only going by what MS said. because of this, they are an inefficient use of disk space.

You misunderstood what Microsoft is saying. They were talking about sparse data (unused segments of the file) in a non-sparse file being inefficient because disk blocks for the unused space are reserved and unavailable for use. Sparse files do not reserve disk blocks for unused file segments, so are efficient when the whole file is not populated. Sparse files have nothing to do with compression, so Microsoft's use of sparse data to include zeros (compressible data) confuses their explanation. Sparse files can contain a series of consecutive zeros or other data and hence can be compressed to a smaller size. Sparse files have nothing to do with file compression.

Sparse files are efficient and most useful for use cases where segments of the file are not filled, are deleted, or written to after creation. So a file that contains database information is continually updated like this, or a system mapped to a virtual image as mentioned already. And another use is encrypted folders that may be implemented as an encrypted sparse file, since you can add and remove files from an encrypted folder, and you want it to take as little disk space as needed - an example is Apple's sparse bundle.

Winuser · Mar 24, 2022

@Bree and @geneo, both of you are probably correct but this is the complete sentence I'm going by. "The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space." I don't see how I'm missunderstanding that sentence.

PerpetualCycle · Mar 24, 2022

Winuser said:
@Bree and @geneo, both of you are probably correct but this is the complete sentence I'm going by. "The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space." I don't see how I'm missunderstanding that sentence.

As I mentioned, "sparse data" is not the same as a "sparse file". They are trying to say that sparse data in a regular file is inefficient, while sparse data in a sparse file is not.

Winuser · Mar 24, 2022

geneo said:
As I mentioned, "sparse data" is not the same as a "sparse file". They are trying to say that sparse data in a regular file is inefficient, while sparse data in a sparse file is not.

Like I said, I'm only going on that one sentence.

Haydon · Mar 24, 2022

Winuser said:
Like I said, I'm only going on that one sentence.

That is the problem. Context plays a role in what that one sentence means. The meaning is in the message as a whole.

Bree · Mar 24, 2022

Winuser said:
this is the complete sentence I'm going by. "The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space." I don't see how I'm missunderstanding that sentence.

Your not, but MS were not describing a sparse file in that sentence. They were setting out the scenario for which a sparse file is the solution.

They then went on to say why a sparse file is the solution to that problem.

Microsoft said:
When sparse file functionality is enabled, the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data.... maintenance of the sparse file is transparent to all processes that access it, and is more efficient than compression for this particular scenario.

Full document here:

Sparse Files - Win32 apps

File compression of files that contain mostly zeros makes efficient use of disk space.

docs.microsoft.com

What are "Sparse files"

Well-known member

My Computer My Computer

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

aka Mama Glass

My Computers My Computers

At a glance

At a glance

♦ ♦

My Computers My Computers

At a glance

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Well-known member

My Computer My Computer

At a glance

Syzygy

My Computers My Computers

At a glance

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Syzygy

My Computers My Computers

At a glance

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Well-known member

My Computer My Computer

At a glance

Well-known member

My Computers My Computers

At a glance

At a glance

Similar threads

My Computer

My Computers

My Computers

My Computers

My Computers

My Computers

My Computers

My Computers

My Computer

My Computers

My Computers

My Computers

My Computers

My Computer

My Computers