Saint-Archives.com Forum

Thanks, Venizelos.

I very much like the new format. 

The thumbnail size is much more useful and the information banner now stands out very nicely.

Have you or Adam had any thoughts on the possibilty of providing MD5 hashes of your collections?

Spike

Adam,

Sorry if it was not clear what we were discussing about hashes and tags.

You obviously understand how tags are used to do "fuzzy searches" for sets in a collection.

On the other hand, MD5 hashes have a very different purpose.  They provide a unique signature for a specific file by doing a sort of checksum calculation on its content.  Very similar files, even if they differ only by a single byte, will have vastly different signatures. So if two files have identical signatures, you are guaranteed they are identical in every way, irrespective of their file names.

Consequently, if each of us has a collection of photos and their MD5 hashes, we can easily identify which files we both have (same MD5 hash) and which only one of us has (unique MD5 hash).  Thus it is a straightforward process to confirm which photos we share in common and which may be new to each other.

If you and Venizelos could provide MD5 hashes for your collections, and  Bernd and I do the same, we would be able to identify possible new photos in our collections that we could then share with each other with relatively little effort.

Spike

Many thanks for the MD5 hashes, Venizelos.

My preliminary comparison of our two collections based on these are quite interesting. I filtered out all the "thumbnail" and "vidcap" entries from your list and then compared the remaining hashes against my own.

We appear to share 85308 identical photos, but you have 23090 photos that I don't have, and I have 93181 that you don't have!  These are much larger numbers than I was anticipating and indicates we could have a very large number of photos we could exchange.

To verify the accuracy of the comparison, I have picked one image at random with MD5 hash 4cfeedd74021234564f7e3ed637d9734:

You should be able to verify that this photo from Set383 corresponds to the photo at the following location on your system:

F:\SILVIA\SILVIA SAINT ORGANIZACION\set383_2001h-SSaint&Sean-Michaels_SUZE\set383_048.jpg

To show how the hashes can be used to identify new photos, I picked Set043 at random and found the following image is unique to my collection with MD5 b34d61f6aa3f08aa7af5b604656fce89.  This is NOT in your collection and is NOT shown in the catalogue, although it is cut-out of set043_020 so could be considered as being in the catalogue.

There are also two nearly identical photos that I have, but you only have one of them.  The one on the left is "F:\SILVIA\SILVIA SAINT ORGANIZACION\set043_legworld-9711_HOTLEGSHOW.COM\set043_029.jpg" on your system and is set043_077 in the catalogue.  You don't have the one on the right and this precise version is not in the catalogue.

Although it is debateable whether these are different photos for catalogue purposes, it does show that we can identify new files relatively easily using MD5 hashes.

Looking at set043 in general, we share 240 identical photos. Excluding the above photos, none of my other 111 unique photos appear to be new to the catalogue.  Presumably the 13 unique photos you have are also in the catalogue.  However, in both cases we have different versions of the photos, so they may be worth sharing.

While I delve more deeply into the comparisons, here is a copy of my MD5 hashes from a couple of months ago for your reference:

https://mega.nz/file/MDxCzA5B#0JfzRvjKBFmI02ZD0gwge14c6WCUrguGKodOLuGj1aw

The zip password is currently shown in my profile whiteboard.  These hashes don't incorporate the recent catalogue updates and scans, as I have yet to collate these.

Spike

Adam,

Good question.  I couldn't find any tool myself, so I wrote my own command line app in Python to do this, viz.:

>zipsum --help
usage: zipsum.py [-h] [--debug-level DEBUG_LEVEL]
                 {find,scan,compare,check,zip,unzip,organise,unmerge,renumber}
                 ...

Utilities to manage zipped collections of images and identify/remove
duplicates.

positional arguments:
  {find,scan,compare,check,zip,unzip,organise,unmerge,renumber}
    find                  Find directories or zip containing files whose names
                            match the specified pattern.
    scan                 Scan a directory or zip file to generate an MD5
                            signature file.
    compare           Compare MD5 signatures to identify duplicates.
    check               Check the MD5 signatures in the specified file are
                            duplicated in the master MD5 files.
    zip                   Zip directories and optionally remove the original
                           files after validating zipped content.
    unzip               Unzip zip files in named directory to individual sub-
                           directories.
    organise          Organise numbered top-level sub-directories in a
                           directory tree with name of form 'NAME[nn]'
    unmerge          Extract files from a numbered directory that match
                           those in the specified master directory and place them
                           in a new numbered directory.
    renumber        Renumber files in a directory to ensure they are
                           numerically and alphabetically consistent.

optional arguments:
  -h, --help           show this help message and exit
  --debug-level DEBUG_LEVEL
                        set the debug level

If you can install a Python 3.7 interpreter and get to a command prompt, you could use this too.

Spike

OK I found one app : I said 100 000 but it's 200 000 !

k2s.cc/file/14e0343ba3e6b/Files.xlsx