Lessons learned from backing up hundreds of discs

[kaelef]

I thought I'd share my recent experiences:

I started making backups of my Blu-rays and DVDs a few weeks ago. This was partly to make it easier to view multi-disc sets (a full season of TV shows, for example), but also to see if any of the discs had suffered from rot.

So far, I've backed up over 500 discs using a combination of MakeMKV, AnyDVD, and Linux's "dvdbackup". I've been using 3 drives for this:

• Pioneer BDR-209M (cross-flashed to BDR-211UBK firmware)
  LG WH16NS60
  LG BU40N (inside an Archgon MD-8107S-U3-UHD)

Drive performance
Usually (but not always), the WH16 is much faster at ripping Blu-rays than the Pioneer, and the Pioneer is much faster at ripping DVDs. The BU40N is consistently the slowest.

Unfortunately, I found that both LG drives will occasionally (but rarely) get in that "confused" mode and start making bad rips until they're powered down. See below.

DVD rips cannot be automatically verified (we need AccurateRip for DVDs)
Unlike Blu-rays, there are no hashes stored on DVDs to use in checking that the files are being correctly ripped. I realized my LG drive was silently providing bad data when I tried to turn a "VIDEO_TS" folder into an ISO and Linux's "genisoimage" reported invalid data in one of the IFO files.

A clean rip should result in all the BUP and IFO files matching exactly. I scripted a check of all the BUP/IFO pairs in my ISOs and found several cases where they didn't match. I then re-ripped those discs on my Pioneer drive and confirmed that the new rip had matching BUP/IFO pairs. (Re-ripping using the LG drives also worked after powering-down and restarting.)

I haven't found a practical way to check the actual video files (VOB) for corruption, so I just have to hope that catching the BUP/IFO mismatches is going to find most or all cases where the drive started silently misreading.

To be sure, I can rip each disc twice using two different drives and compare the results, but that's not practical when you're doing hundreds of discs.

I now do a comparison of all BUP/IFO pairs after every rip as a form of spot-checking. It'd be nice if MakeMKV had something like this built-in.

We need an AccurateRip database for DVDs. There's a thread on DBPowerAmp suggesting the same https://forum.dbpoweramp.com/showthread ... for-DVD-BR thing, but the replies are that it's not necessary. Clearly this is not correct.

LG drives getting confused is a real thing
One of the reasons I found I was getting some bad DVD copies is that MakeMKV started reporting bad M2TS hash values on Blu-ray copies using the WH16. Once it started, it kept doing it with other discs, and this is also when I noticed that the DVD copies I'd made recently with that drive were bad.

Turning off the machine and restarting it made the hash errors go away.

My Pioneer drive hasn't done that once yet.

AnyDVD positives and negatives
I like that AnyDVD can directly create an ISO image of a Blu-ray. It's a two-step process with MakeMKV (make Blu-ray backup with MakeMKV then create an ISO image using ImgBurn). It's weird that MakeMKV creates ISO files for DVD backups but not for Blu-rays.

On the other hand, AnyDVD doesn't report much of anything while it's working, so I don't trust that it's catching errors as well as MakeMKV does. I haven't been able to confirm whether or not it checks M2TS hashes on Blu-rays, but it didn't do it as of a couple of years ago.

Check your old discs
The majority of my discs were fine, but there were a small number that were unreadable even though they looked completely fine. No scratches or dust. When a disc failed to read, I would clean it and try it on a different drive, but some were just unreadable on any drive despite looking perfectly spotless.

This was most common with DVDR's. Adult Swim had this great build-to-order service where you could pick specific episodes of shows you wanted and they'd send you a custom DVD with just those items on it. These discs full of rare shows are all unreadable now.

I hope these notes help someone else out!

[Ezatoka]

We need an AccurateRip database for DVDs. There's a thread on DBPowerAmp suggesting the same https://forum.dbpoweramp.com/showthread ... for-DVD-BR thing, but the replies are that it's not necessary. Clearly this is not correct.

Too bad, http://redump.org/ only accepts dumps of game discs.

[kaelef]

Adding to my previous note:

I was just now using MakeMKV on my 3 drives simultaneously to back up the 5-disc Speed Racer Blu-ray set. Both the LG drives returned bad M2TS hashes part way into the backup. Turned them off and restarted. Completed the copies without issue. The Pioneer never generated an error.

[ChibiRobo]

The LG Drives get confused for 2 reasons I have seen.
1.) "Jamless play" technology: Instead of reporting error when reading bad blocks, LG firmware will simply return junk data and call it success. The junk data will always be random.

2.) Long operation uptime: After some long time, drive should be rebooted or it will return random junk.

For BD this not as big a deal because of hash verification in MakeMKV . But for DVD this is bad, can cause silent errors without you knowing. For DVD I prefer to rip to ISO twice and compare checksum. Since the junk data appears to always be random, doing a 2-pass rip will rule out the bad dump.

[TomB19]

Great thread and great info. Thank you.

I have a pair of WH16NS40 drives, flashed to look like WH16NS60. Oh, boy, I'm going to get roasted for this post...

My drives have gone into a state where they have required hard reset on a few occasions over the last year. It has happened 4 times that I am aware of. It's particularly annoying when I have 24+ hours of transcoding queued up, as it can knock me back to a single drive until I get a chance to cold boot.

So, installed a DPST switch (one for each drive) in a blank faceplate so I can shut off power to the individual drives. This clears the problem. Essentially all SATA is hot plug so the drives go away when off and come back when powered back on.

I also drilled a 10mm hole in the top of the drive housing, right above the laser pickup (immediately rearward of the spindle, and covered it with aluminum tape (speed tape). The drives are on rails so I can pull them out of the case (Super old Antec P182) and clean the laser with a q-tip and some alcohol in about 1 minute.

When I have read problems, it is almost never the LG drive losing it's mind or a dirty laser pickup but it's nice to be able to eliminate these things as possibilities without rebooting. My system runs for months between reboots and, for some reason, that is important to me.

[Billycar11]

pioneers i have never had to reset tested 500+ not even ever a read error on my test discs some of those hard to read discs fail on 50% of lg/asus where it would take multiple trys
did have to clean a laser on a pioneer an old one some one sent in it would see no UHD discs after flashing cleaned it and worked fine.
pretty much if you can get a pioneer you should while we still have them with old firmware. pioneer is alot better than lg at patching this stuff

[hoth80]

I haven't found a practical way to check the actual video files (VOB) for corruption, so I just have to hope that catching the BUP/IFO mismatches is going to find most or all cases where the drive started silently misreading.

To be sure, I can rip each disc twice using two different drives and compare the results, but that's not practical when you're doing hundreds of discs.

No. VOB file can be corrupt only. No way to check but twice rip and hash check. Not need different drive, just rip x2. If bad, hash not match.

[kaelef]

For BD this not as big a deal because of hash verification in MakeMKV . But for DVD this is bad, can cause silent errors without you knowing. For DVD I prefer to rip to ISO twice and compare checksum. Since the junk data appears to always be random, doing a 2-pass rip will rule out the bad dump.

That matches my experience. I've been mixing my DVD rips with Blu-ray rips in between as a way to 'catch' when one of the LG's starts returning bad data by seeing those bad M2TS hashes. Honestly, it hasn't happened that often: I think 3 or 4 times during ripping a few hundred discs, but I'm very wary of it now.

Having a Pioneer in there with the LG's is great to keep a check on things, and also helps when I have a problem reading a disc. Swapping from one to the other has helped a few times.

I have a pair of WH16NS40 drives, flashed to look like WH16NS60. Oh, boy, I'm going to get roasted for this post...

I don't even remember if my WH16NS60 is actually an NS40 that I flashed, and I don't feel like opening the case to check...

When I have read problems, it is almost never the LG drive losing it's mind or a dirty laser pickup but it's nice to be able to eliminate these things as possibilities without rebooting. My system runs for months between reboots and, for some reason, that is important to me.

That's a great idea! I imagine the amount of ripping I do will go down to very little once I've backed up my years of discs, but I might try the same thing if it becomes more of an issue.

did have to clean a laser on a pioneer an old one some one sent in it would see no UHD discs after flashing cleaned it and worked fine.
pretty much if you can get a pioneer you should while we still have them with old firmware. pioneer is alot better than lg at patching this stuff

My Pioneer is so old (209M) and not an official UHD reader, so I'm surprised it's been working so perfectly. It will be great if the Libre firmware starts working on them again -- I'll definitely buy another one soon.

No. VOB file can be corrupt only. No way to check but twice rip and hash check. Not need different drive, just rip x2. If bad, hash not match.

It should be possible to write a VOB analyzer that would tell you if the transport stream has lots of errors, but it would take more time and skill than I have. I'm sure commercial tools for that exist for professionals, but I haven't found anything for us amateurs.

I was hoping to use FFMPEG to try converting a VOB to something else, and then give you an alert if FFMPEG generated a lot of warnings/errors, but that doesn't seem like it'll work reliably enough.

So, yeah, it's ripping twice and comparing for now. That's why having an AccurateRip database for DVD's would be so great. It's the same problem people ripping audio CD's had 30 years ago. (I just made myself feel extra old...)

[TomB19]

I've stayed away from Pioneer drives because LibreDrive firmware has been so long in coming. If it were here, I would jump.

Does everyone run the slim, external, Pioneer or is there a good 5.25 internal Pioneer drive for old guys like me who like their giant, old school, tower cases?

[Billycar11]

I've stayed away from Pioneer drives because LibreDrive firmware has been so long in coming. If it were here, I would jump.

Does everyone run the slim, external, Pioneer or is there a good 5.25 internal Pioneer drive for old guys like me who like their giant, old school, tower cases?

yes all the pioneers i sell are great i have ones with old firmware
i have 212 for $158 with UBK firmware a UBK from pioneer is $160
also have the s13UBK and s13UX but they are alot more pioneer charges $300 for the UX plus tax and shipping but i have them all with old firmware

[TomB19]

I still use Pledge to clean discs. I've tried a couple of off brand clones and they seem just as good but a can goes a long way so cost is not a big deterrent. Does anyone have a better option for this?

I also use Meguiar's PlastX as scratch remover. I don't polish to make it look like new, just enough to read the disc. This stuff is the best scratch remover I've used, so far. Anyone have a better option?

Dollarama has microfiber cloths for something like $2.50 for 5. They are well adequate to the task. I keep 10 around and launder them after cleaning very few discs. A little dirt from a previous disc will scratch so I toss them in with my wife's fine washables. She may have noticed her situation is a bit more gritty lately but that might not be a bad thing.

It's surprising how nasty a DVD can be and still have a chance to read. It's a bit of a crap shoot but a visibly damaged BD has zero chance of reading while a DVD does.

For a while, I was loading up heavily scratched discs with Pledge. The thought was to fill in the scratches a bit. This just corrupted drive optics so I make sure to wipe it all off before inserting in the drives.

Please don't hesitate to post a better process. I'd like to improve my game, also.

[kaelef]

Does everyone run the slim, external, Pioneer or is there a good 5.25 internal Pioneer drive for old guys like me who like their giant, old school, tower cases?

I just got my hands on one of those slim external Pioneers from Adorama. Very handy since most PC cases these days don't have slots for 5.25" optical drives anymore. It works fine, but it's definitely slower than the two full size drives sitting in my giant, old school tower case. (4x vs. ~8x rips of BR)

It's surprising how nasty a DVD can be and still have a chance to read. It's a bit of a crap shoot but a visibly damaged BD has zero chance of reading while a DVD does.

As you say - you can't really tell from looking at the disc whether it's going to be readable or not. I have pristine-looking discs that are unreadable, and ones in bad shape that are fine.

[Ezatoka]

Also one has to consider the data density on such a disc.

The disc is always the same size, but for a DVD there's only 4.7/8.5 GB of data on it. But for a BluRay it's at least (up to) five times more. So one scratch has to be considered a much bigger problem for all data "below".

[outlier]

After doing over 1000 titles I have a few thoughts to add.

1) Some folks mention they favor Pioneers (vs LG) to get more accurate reads, but I’ll still not sure if this is a valid concern for blurays given the m2ts hashes.
2) Be careful with foreign language titles. If you soecify a default language in makemkv you’ll end up with only dubbed audio (if they exist) or no audio at all (if they don’t exist).
3) When in doubt over which movie track to select choose 800, or pick the option with English+(european languages) over English+Japanese. If still in doubt you’ll probably want the longer duration or larger file (but there’s exceptions, like movies that have a mode where actors or directors cut in with additional commentary).
4) Seems like 95%+ of tv shows have misordered episodes so you’ll need to check manually or use some other app to handle this. Some shows are properly ordered or completely misordered, and others might only have order swaps on 1-2 discs out of the 10+ or 20+ disc set.

Thoughts for makemkv:
1) Needing to click on the drive icon after inserting the disk is a tiresome hassle.
2) Sometimes when clicking the eject button in the GUI I accidentally click the button bext to it. If the system is busy with other saves then it can take awhile for the configuration dialog box to show up so I can dismiss it. I’d suggest shifting the eject button further from it’s neighbor, or keep cached whatever is needed to show the dialog.
3) The functionality to select a language and have only those audio tracks selected for saving generally works well for English in the USA. The main exception are titles that don’t have any English tracks (only native language with subtitles). Makemkv will save the movie with no audio tracks. I’d prefer a warning prompt, or fallback logic that selects all (vs no) audio tracks in the situation where all the audio tracks are for 1/same language that differs from the desired language.
4) More feedback in log on whether a disc read issue was resolved or not. Possibly also more info recorded on how far into output file the issue was found (for people who aren’t monitoring the save and want to know during a re-try when they’re reached/passed the problem point).
5) Ability to eject and clean disc and resume/retry a save from that point rather than have to start from scratch.
6) Option to pre-grow/allocate mkv files. Akaict Makemkv likes to grow files as it writes, which can lead to more file fragmentation and filesystem metadata operations. Alternative is to grow the file to maximum anticipated size at creation and then later truncate to final size at the end.

[460cidpower]

I have a Pioneer BDR-209M on 1.54 firmware. Is it possible to cross-flash it?