Error Handling ideas for damaged media

[bugdude]

Good day,

I occasionally try to back up DVDs of children's films that have some degree of damage on them. Even after the cleaning, polishing and other manual techniques are exhausted I wondered if it would be possible to have some software recovery options in MakeMKV. Currently after MakeMKV fails a read and all its retries it fails the conversion process. In the same situation I think most DVD player firmware actually reads the subsequent sector and re-syncs allowing a jump, pop or click rather than failing to play the media. I don't know if Bluray players do this, but the I-Frames in DVDs do allow a resync going forward from a bad block (I think).

From a software perspective I think adding the possibility of either replacing the sector with a null sector, omitting it, or even replacing it with a black cell (if that is possible) might allow the data that is available to be recovered.

Is it possible to add suck an option to MakeMKV ??
-- I think it would make a great addition

Also, I have noticed a lot of foreign film DVDs have bad UDF filesystem data but valid ISO filesystem data (often the directory points at different start sectors). It could be useful to have an option to ignore the UDF filesystem and use the older ISO filesystem (I think the DVD players may do this by default). Assuming that MakeMKV doesn't already handle this problem silently of course.

Dave

[pneumatic]

Assuming MakeMKV can arbitrarily try to read whatever block it wants to, I can imagine a scheme like eg. n retries on unreadable, then skip ahead q blocks and if that can't be read either, keep skipping ahead until you get a readable block and then continue reading as usual. And then replace all the unreadable blocks with dummy data.

But then the resulting mkv might not be playable. It might need some advanced knowledge of the video codec in order to fill in the dummy blocks with something that won't crash the decoder during playback. Or maybe the codecs are already tolerant enough of data corruption -- when I get file corruption due to signal interference on h.264 PVR recordings it just artefacts and skips ahead over the corrupt data. But maybe that's because the chipset's decoder is designed to handle corruption. Maybe not all decoders can deal with corrupt data so elegantly -- I've been getting data corruption recently on my mkv's (due to USB interface corruption it seems) and LAV video decoder sometimes freezes unrecoverably on the corrupt parts, while on others I can seek forward over the corrupt section and resume playback there.

To get a nice clean solution you probably would need some advanced knowledge of the video codecs, something that is probably beyond the scope of MakeMKV developer(s). Maybe that could be handed off to ffmpeg to repair the file afterwards.

[bugdude]

It's exactly the player resync behavior that gives me the idea that simple omission may prove to be the best choice.. M2T@ and VOB are essentially stream files and the stream formats (I think) already define resync mechanics based on I-Frames or full frames of some kind.. In theory if you have a damaged DVD/BR stream that was say 4.00 G and you simply omitted the bad sectors it would 'shrink' in size, but if the resync mechanics for the streams could handle it you would (I think) get a 'frame gap' on the video stream and a corresponding audio gap on the audio stream.. the exact alignment of the two would actually depend on the multiplexer of the encoder because that decides how to interleave the actual stream blocks within the package file, but it 'seems' like it could work. Substituting a dummy block would as you mentioned require more knowledge of the current stream frame position/muxing and data requirements and be significantly more complex. Replacing it with a null block is also possible, but you would need to test the decoder/demux/players to see if that was at all useful to do.

The reason I suggested looking at it is implementing and testing is actually relatively easy in the command line binary, and after a testing the outcome with a handful of players it should be easy to tell which (if any) of these options actually works to any degree. At that point the one(s) that are potentially useful could be surfaced in the UI. I suggested the generic options based on blocks rather than any stream specific magic methods because those are much more complex to even try..

[peter0302]

I also want to vote strongly for this. Seems like a no-brainer.

On the assumption that MakeMKV uses LibAV (aka ffmpeg) under the hood (and what else could it possibly use, honestly?) - which is a library with which I have extensive experience - this is totally possible. Maybe a tiny bit tricky to avoid things getting out of sync, but still possible.

If your player can skip a bad part of a disk with you barely noticing, so can the ripper.

The part that I don't understand is if Windows can copy the files directly off the disk without error (albeit in encrypted form), how can they possibly be corrupted? Is Windows really reading incorrect bytes without knowing it? I've never heard of this when it comes to disk I/O. Either the read fails or the bytes are correct. How is 4K blu ray different?

[Woodstock]

On the assumption that MakeMKV uses LibAV (aka ffmpeg) under the hood (and what else could it possibly use, honestly?)

You assume a lot, there.

Why would MakeMKV use ffmpeg? In general, it is not displaying video, but ripping data from disks. ffmpeg isn't ripping video from disks, so they aren't trying to do the same thing.

[peter0302]

Well, based on some of the symbols in the .DLLs, such as this one in libffm.dll:

Code: Select all

 s->avctx->codec_id != AV_CODEC_ID_H264

or this error message:

decoding to AV_PIX_FMT_NONE is not supported

it most clearly is utilizing libav.

LibAV is the library underlying ffmpeg. It has 100s of different use cases, only one of which is to make a video player, but another is multiplexing. That's exactly what MakeMKV (or some other library it depends on) is doing when it takes dozens of M2TS files and muxes them to a single MKV file.

So it is indeed using LibAV, and therefore it's quite possible to optionally skip over corrupted packets (fill them with empty packets of the same duration for example) and then continue.

Sorry I'm a bit new here. Who actually develops the underlying libraries?

[thetoad]

you want ddrescue to image the DVD (perhaps with needing libdvdcss to authenticate drive to disc), one can then generally brute force decrupt the CSS protected DVD (dvd decrypter can do this with proper settings). (generally, as small VOBs can't be decrypted, but they are generally not needed to be decrypted). If one has 2 or more copies of the same DVD (say from library), ddrescue can fill in the blanks between them (assuming identical discs).

For blurays same process (though might need libredrive active to remove bus encryption for ddrescue). In this case, makemkv can store the decryption keys if you started then quickly stopped an encrypted backup (and then just copy your recovered data into the folder makemkv was creating). Or, one can mount it using an image mounter and anydvd can decrypt to folder or iso (sometimes anydvd doesn't have the keys, as you might be the first person to try and decrypt it), but then its still easy enough, teach anydvd about the disc by inserting it and have it generate the VUK for the disc, and then once generated can go to the iso mounting method.

sometimes though, even with ddrescue, you will never get a complete copy (for DVDs this can be because of intentional corruption when the disc was pressed, but that doesn't matter much, but can also be because no matter how many times one rereads, the sector is just bad)

[jd1337]

i would love an option within makemkv where we could read the errored sectors from a secondary medium, say i have 2 identical dvds / brs / 4k etc, but they have both been damaged in seperate ways, neither will rip on its own, however, the read errors occur on a different sector so if i can somehow combine the 2 i would be able to rescue my media.

I feel like this will occur more regularly as time goes on, media goes further out of print etc, picking up a cheap second copy would have a much lower risk of both existing and new copy not working.

As you mentioned I believe ddrescue CAN do this, but, it's a bit of fluffing around when makemkv already has all the retry / decryption logic there waiting

[dmdmmatt4]

"Revision history
MakeMKV v1.17.6 (20.1.2024)

Improved handling for discs with mastering errors"

.. dunno what this coding voodoo was, exactly, but a disk I've spent the last week trying to rip (due to a clear mastering fault, and on multiple drives) ripped first time after upgrading makemkv. Barely a squizz of a complaint from it, just a brief mention of:

"Encountered 7 errors of type 'Read Error' - see http://www.makemkv.com/errors/dvdread/
1 titles saved"

Nice one!

[thetoad]

i would love an option within makemkv where we could read the errored sectors from a secondary medium, say i have 2 identical dvds / brs / 4k etc, but they have both been damaged in seperate ways, neither will rip on its own, however, the read errors occur on a different sector so if i can somehow combine the 2 i would be able to rescue my media.

I feel like this will occur more regularly as time goes on, media goes further out of print etc, picking up a cheap second copy would have a much lower risk of both existing and new copy not working.

As you mentioned I believe ddrescue CAN do this, but, it's a bit of fluffing around when makemkv already has all the retry / decryption logic there waiting

yes, it be nice if ddrescue type functionality was built in, but its really a different program than makemkv is. What one might really want is a being able to wrap ddrescue with makemkv like functionality (i.e. when one opens the device, it does all the magical encryption authentication / generation of keys, and every read() does a decryption behind the scenes if neccessary (as I believe aacs is 1:1 in size, so would work).

I doubt makemkv would ever have this functionality native in it, its just very different than what its intended to do, it be nice though if makemkv had a wrapper mode (such as I described) that would enable others to write tools to leverage its decryption capabilities.