Reproducable MKV Creation

[thetoad]

An interesting feature (to me at least), would be for MakeMKV to have what I want to refer to as "Reproducable MKVs". Basically running it multiple times should result in a bit for bit MKV file.

Today that's not possible for one obvious reason (metadata includes date and versioning in the mkv metadata). While it might not be really doable to get rid of the version (/not worth trying to make different versions produce compatible results, as that could prevent bug fixes), it would seemingly be valuable to paying customers (less to non paying customers as older versions aren't usable) to be able to get bit for bit copies.

I see 2 possible ways to achieve this (perhaps more, just 2 that stick out)

1) a reproducable flag that removes metadata and the like that would prevent things from being reproducable. This would require users to set it up in the exact same way (perhaps not the most straight forward)

2) output a command line that can be run manually that would result in the same output (as effectively the same "command" that is being used for this execution.

[Ezatoka]

The Matroska format itself is not deterministic by default when it comes for header and metadata.

But the streams should already be always the same if there wasn't any error on dumping.

[flojo]

mkvextract has a "raw" switch, I tried it and it seemingly worked. Read here:

viewtopic.php?p=177372#p177372

Data other than the streams is junk and you really don't want to validate junk. If you're thinking along the lines of something like "TorrentZip", well that was kind of a bad idea being all files in the ROM could be validated anyways (it would just take longer).

[Coopervid]

Please elaborate: What's the benefit?

[flojo]

Please elaborate: What's the benefit?

I think they're thinking to ultimately identify content, say S07E05 of Game of thrones with just a hash value. Of course, who would keep the database of these hashes is the real question since running things like http://gnudb.org or http://db.cuetools.net isn't free.

[Coopervid]

That's kind of lazy. Why wouldn't you name your files in a manner to identify them? OK. Some guy struggeld with my GoT abbreviation for Game of thrones. Otherwise WTF.

[flojo]

Lazy? Almost nobody names files when ripping music CDs. There's also a lot of people that use things like Plex/Jellyfin that scrape data based on a filename they don't care about. There's also people who want the TV episodes they rip auto-identified. There's also the case that if your hard drive fails and you have to recover files you very well might not have logical filenames after recovery. That's just file naming, then there's also using hashes for dupe checks and corruption detection.

Why have less options?

[Coopervid]

You want to live in "ripping paradise"? Dream on.

[Toad King]

If you want to verify the content was ripped correctly you can use the hash tables in an encrypted dump. Those have checksums for all the content streams on a disc. For everything else you can compare them to their copies in the BACKUP folder for BDMV files, or DUPLICATE for AACS files.

A tool for checking hashes on encrypted dumps can be found here.

[flojo]

If you want to verify the content was ripped correctly...

While hashing is faster, if you were going to keep the files you can just create a small par2 set which will not only check the files but repair them (possibly). I can't argue that checking the files on disc to the files you've ripped is nice, unpess makemkvcon incorporates a hashing feature as it rips, it still requires rereading the entire disc.

Of course none of this really helps identify the streams. Ripping audio CDs is kind of the reference to strive for, I'm just not sure that ripping video discs has the same repeatability. For instance are the streams written by makemkvcon the same as DVDFab, AnyDVD and other rippers identical? Although if makemkccon incorporated hashing as it ripped the streams it would open some options up.

[flojo]

If you want to verify the content was ripped correctly...

While hashing is faster, if you were going to keep the files you can just create a small par2 set which will not only check the files but repair them (possibly). I can't argue that checking the files on disc to the files you've ripped is nice, but unless makemkvcon incorporates a hashing feature as it rips, it still requires rereading the entire disc.

Of course none of this really helps identify the streams. Ripping audio CDs is kind of the reference to strive for, I'm just not sure that ripping video discs has the same repeatability. For instance are the streams written by makemkvcon identical to the streams ripped by DVDFab, AnyDVD and other rippers? Although if makemkccon incorporated hashing as it ripped, it would open some options up.

[keydb_helper]

Of course none of this really helps identify the streams. Ripping audio CDs is kind of the reference to strive for, I'm just not sure that ripping video discs has the same repeatability.

Content fingerprinting or hashing would be cool for identification purposes. Verifying the accuracy of ripped content after it has been decrypted is a bit trickier. If you forego the creation of an MKV file and stick to a Blu-ray folder structure or ISO file you might be able to use the AACS content hashes.

If you do remux your disc into an MKV file you'd need to be mindful about stream modifications such as the TrueHD seamless branching improvements in v1.15.4 or the re-packaging of Dolby Vision content to STDL. The implementation of this type of functionality has evolved over time and it could complicate content hashing.

For instance are the streams written by makemkvcon the same as DVDFab, AnyDVD and other rippers identical?

No, there can be certain differences to the binary data even when the decoded data will be an exact match. For example, copy control flags in the M2TS files have not consistently been cleared by all the different ripping tools.

Certain protections like BD+ and AACS 2.1 also cause the decoded video output to be visually distinct as they are intended to reveal information about the decryption process and this varies between each rip tool.

[flojo]

No...

So to create something similar to what audio CDs isn't really possible or restrictions would have to be applied. I know audio CDs had restrictions created for disc submission to CDDB and other programs followed them, although it's obviously easier with CDs as the stream you want is already "demuxed" so O(1) is the complexity to identify. From what you're stating, to achieve something usable with video discs you'd have to hash the data on disc as-is, then associate the hash to a filename (.ifo/.pls) _AND_ a 1-liner command to use with a demuxer/filter (ie. mkvmerge)... ? Clearly I don't really know how the streams are filtered out of the decrypted .m2ts files to put forth any solution to this.

[thetoad]

a reason for reproducible would be in some decentralized ecosystems where a file is created and associated with it's hash. The file then disappears, but the hash reference remains. Either one can create a new file with a new hash (and all old hash references will never be able to be resolved again) or one can recreate the file as it was with the same hash value and thereby make all those existing references valid.