Reliability of hash tags in file comparison

I am aware that generally if two files have the same hash tag they will be identical, but collisions do occur so why the question?

I have been monitoring village weather for the parish council using the Davis weather station for many years. It stores the data in .wlk files. Needless to say I have backups - too many! If I use a duplicate checking app (Duplicate Cleaner Pro) I find variable number of duplicates for each file. The hashtags are the same (about 48 digits in each case).

I'd like to settle simplify my backups to just a couple and delete the rest.

There are several other reasons why I am seeking reassurance but they are irrelevant to the query as they relate to some issues with the Davis weather app..

Hi, Ian.

Are the backup extensions the same. Some programs create two files (don’t ask me why, I am not that clever)
I assume you would have to know where they are stored to find out.

Name can vary slightly (eg 2008-01.wlk and 2008-01 (2).wlk)
File created and file modified are different. Extension always .wlk
Hash always the same for each group of duplicates.
Folders can vary eg 2008-01.wlk and 2008-01 (2).wlk can be in the same folder or different folder

Frankly I think I am good to go - keep two and zap the rest. I need to hand the data to someone else and it's a bit of a mess at present. I understand my mess; the person taking oven over might not! An oven wouldn't help!

TLDR: I think it's perfectly fine to use the hashes in question to remove duplicates.

Since you said "about 48" characters, I'm going to assume you were using SHA-1, which produces hashes of 40 characters. Duplicate Cleaner Pro also supports SHA-256, but that produces 64-character strings.

There are certainly collisions with SHA-1. It's been theoretically broken for almost 20 years now, and definitely broken for over a decade. That being said, the real danger in hashing collisions lies with encrypted communications; think certificates, HTTPS and what-not.

I think verifying the contents of private, local backups, today, with SHA-1 is perfectly fine. Not as reliable as SHA-256, but certainly faster.

A quick Google search finds a collision attack in 2020, which took about 107 GPU years and cost about $45,000 (USD). Attacks would certainly be faster and cheaper now, but is someone going to spend the money to alter your files in such a way to trick you into thinking you have a certain data backup when you really don't? Probably not.

Thanks. A knowledgeable confirmation of a gut feeling.

Sigurd said:

Name can vary slightly (eg 2008-01.wlk and 2008-01 (2).wlk)
File created and file modified are different. Extension always .wlk
Hash always the same for each group of duplicates.
Folders can vary eg 2008-01.wlk and 2008-01 (2).wlk can be in the same folder or different folder

Click to expand...

Yeah, anything with the same extension, hash code, file name with the only difference being that they are (numbered) would be a duplicate.
I wonder why it’s making duplicate backups?

I have too much respect and affection for my computer to falsely accuse it of such a crime.
It's not, I the guilty party!

Sigurd said:

I have too much respect and affection for my computer to falsely accuse it of such a crime.
It's not, am I the guilty party!

Click to expand...

Ha! If it was you accidentally copy/pasting into the same folder, they would be renamed “filename -copy.extension”
Unless you’ve been running two consecutive backups, I don’t know, Ian