What are "Sparse files"

Hi folks
bit confused as to the meaning of Sparse files. Google explanations not clear.
Anybody !!!

Thanks

Cheers
jimbo

jimbo45 said:

bit confused as to the meaning of Sparse files. Google explanations not clear.

Click to expand...

Wikipedia seems clear enough. A sparse file is one where empty space is not stored in the file itself. A typical example of a sparse file would be an expanding .vhdx file for a Hyper-V VM.

The advantage of sparse files is that storage space is only allocated when actually needed: Storage capacity is conserved, and large files can occasionally be created even if insufficient free space for the original file is available on the storage media....

For example, a virtual machine image with max size of 100 GB that has 2 GB of files actually written would require the full 100 GB when backed by pre-allocated storage, yet only 2 GB on a sparse file.

Click to expand...

I didn't know either, but I think MS's explanation says it best.

"A file in which much of the data is zeros is said to contain a sparse data set. Files like these are typically very large for example, a file that contains image data to be processed or a matrix within a high-speed database. The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space.

The file compression in the NTFS file system is a partial solution to the problem. All data in the file that is not explicitly written is explicitly set to zero. File compression compacts these ranges of zeros. However, a drawback of file compression is that access time may increase due to data compression and decompression.

Support for sparse files is introduced in the NTFS file system as another way to make disk space usage more efficient. When sparse file functionality is enabled, the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data."

Is not Sparse Files a method many Disk Cleaners use to perform a Drive Wipe?

Sparse file

A file with all the "few and far between" sucked out of it.

glasskuter said:

A file in which much of the data is zeros is said to contain a sparse data set. Files like these are typically very large for example, a file that contains image data to be processed or a matrix within a high-speed database. The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space.

Click to expand...

Unless I'm missing something, it sounds like they are only good for wasting drive space.

Winuser said:

Unless I'm missing something, it sounds like they are only good for wasting drive space.

Click to expand...

What you are missing is that a sparse file with a lot of empty space in it doesn't actually use any physical storage for the unused space.

For example, I can create a .vhdx file for a 10TB virtual drive. But as a sparse file its physical size is just 230MB until I start to write to it. It looks and behaves as a full 10TB file, and as I write real data to it then it will expand. But at around 180GB written I will not be able to add any more data, as it would have expanded to occupy all the available space of the C: partition where the file is located.

Bree said:

What you are missing is that a sparse file with a lot of empty space in it doesn't actually use any physical storage for the unused space.

For example, I can create a .vhdx file for a 10TB virtual drive. But as a sparse file its physical size is just 230MB until I start to write to it. It looks and behaves as a full 10TB file, and as I write real data to it then it will expand. But at around 180GB written I will not be able to add any more data, as it would have expanded to occupy all the available space of the C: partition where the file is located.

View attachment 25253

Click to expand...

I'm only going by what MS said. because of this, they are an inefficient use of disk space.

sparse file = balloon file, it expands/shrinks with content/air you let in/out, it has been invented to be most efficient with physical resources

OP, please mark this thread 'Solved'

Winuser said:

I'm only going by what MS said. because of this, they are an inefficient use of disk space.

Click to expand...

You misunderstood what Microsoft is saying. They were talking about sparse data (unused segments of the file) in a non-sparse file being inefficient because disk blocks for the unused space are reserved and unavailable for use. Sparse files do not reserve disk blocks for unused file segments, so are efficient when the whole file is not populated. Sparse files have nothing to do with compression, so Microsoft's use of sparse data to include zeros (compressible data) confuses their explanation. Sparse files can contain a series of consecutive zeros or other data and hence can be compressed to a smaller size. Sparse files have nothing to do with file compression.

Sparse files are efficient and most useful for use cases where segments of the file are not filled, are deleted, or written to after creation. So a file that contains database information is continually updated like this, or a system mapped to a virtual image as mentioned already. And another use is encrypted folders that may be implemented as an encrypted sparse file, since you can add and remove files from an encrypted folder, and you want it to take as little disk space as needed - an example is Apple's sparse bundle.

@Bree and @geneo, both of you are probably correct but this is the complete sentence I'm going by. "The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space." I don't see how I'm missunderstanding that sentence.

Winuser said:

@Bree and @geneo, both of you are probably correct but this is the complete sentence I'm going by. "The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space." I don't see how I'm missunderstanding that sentence.

Click to expand...

As I mentioned, "sparse data" is not the same as a "sparse file". They are trying to say that sparse data in a regular file is inefficient, while sparse data in a sparse file is not.

geneo said:

As I mentioned, "sparse data" is not the same as a "sparse file". They are trying to say that sparse data in a regular file is inefficient, while sparse data in a sparse file is not.

Click to expand...

Like I said, I'm only going on that one sentence.

Winuser said:

Like I said, I'm only going on that one sentence.

Click to expand...

That is the problem. Context plays a role in what that one sentence means. The meaning is in the message as a whole.

Winuser said:

this is the complete sentence I'm going by. "The problem with files that contain sparse data sets is that the majority of the file does not contain useful data and, because of this, they are an inefficient use of disk space." I don't see how I'm missunderstanding that sentence.

Click to expand...

Your not, but MS were not describing a sparse file in that sentence. They were setting out the scenario for which a sparse file is the solution.

They then went on to say why a sparse file is the solution to that problem.

Microsoft said:

When sparse file functionality is enabled, the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data.... maintenance of the sparse file is transparent to all processes that access it, and is more efficient than compression for this particular scenario.

Click to expand...

Full document here: