Strange corruption issue - MakeMKV's fault or bad HDD?

[Billycar11]

usb has error checking
https://www.usb.org/sites/default/files/crcdes.pdf
i see your chasing your tail not sure what your exact issue is but i recently had a bad sick of ram really through me off with hash errors in makemkv but i figured it out in 2 days i even guessed the right ram stick to pull out on the first well really i asked my fiancée right or left apparently she knows.

random fact for people pioneers dont get hash errors if they do you have a different problem on your pc
they can get read errors but not hash lg/asus get both

[pneumatic]

usb has error checking
https://www.usb.org/sites/default/files/crcdes.pdf

Yep, and my first thought was that it couldn't be USB corruption due to the built-in error correction, but then I saw this article talking about firmware/driver bugs with USB devices that can potentially cause corruption. The article also mentions a separate issue where the receiver validates the CRC by sending ACK to the transmitter, but then the ACK gets corrupted on the wire. But it says all that would happen is the transmitter would re-transmit and the receiver would just ignore it cause their toggle bit hasn't changed, so it shouldn't get corrupted, all it would do is slow the data rate due to all the re-transmissions.

But I have read and experienced anecdotes of data corruption on front USB ports due to poor cable quality so I'm pretty sure corruption is still possible somehow, perhaps more to do with incorrect implementation in the firmware/driver than the USB spec itself. Also recently had an SD card corrupted via a USB card reader but I don't know if it was USB or SD interface at fault.

i see your chasing your tail not sure what your exact issue is but i recently had a bad sick of ram really through me off with hash errors in makemkv but i figured it out in 2 days i even guessed the right ram stick to pull out on the first well really i asked my fiancée right or left apparently she knows.

Thanks for reminding me to do this, I did recently get new RAM and did memtest on it on the first day and got no errors but I'll have to do it again. In OP I mentioned I also did an overnight write-read-validate every block test on my Seagate USB HDD which I initially thought was the culprit and that got no errors either. If the corruption is intermittent then perhaps there was no corruption at the time of the test.

[pneumatic]

Got corruption on my rear USB ports, so that theory is bust too.

[Billycar11]

Got corruption on my rear USB ports, so that theory is bust too.

Try. Prime 95 blend test with full ram usage they finds bad ram way faster then memtest

[pneumatic]

Try. Prime 95 blend test with full ram usage they finds bad ram way faster then memtest

Ok will do, thanks for the tip.

In the meantime I'm going to try some different BIOS USB settings, different cable, and different ports (half of my port controllers are Intel, other half are Asmedia). So far so good on the Asmedia controller but haven't done enough discs to be sure.

[pneumatic]

Try. Prime 95 blend test with full ram usage they finds bad ram way faster then memtest

Just did 2 hours of this, everything loaded up 100% CPU usage & 16GB RAM usage. Got no errors - is 2hrs long enough?

If it's a USB issue I think it has something to do with synchronous read+write. i.e loading up the USB bus with max read and max write simultaneously. Because I'm extracting mkv's from the ISO on the same USB drive so it's reading data from the ISO at max rate and writing to mkv files on the same drive at max rate too. This is unusual because I think when most people use USB drives they are only either reading or writing at max speed but not in both lanes simultaneously.

I've set the USB mode in the BIOS to make it so it doesn't use this "auto smart" XHCI thing where it starts as 2.0 mode and then switches automatically to 3.0 mode when the OS loads. Maybe the switch is mucking up the initialisation somehow?

[Billycar11]

Try. Prime 95 blend test with full ram usage they finds bad ram way faster then memtest

Just did 2 hours of this, everything loaded up 100% CPU usage & 16GB RAM usage. Got no errors - is 2hrs long enough?

If it's a USB issue I think it has something to do with synchronous read+write. i.e loading up the USB bus with max read and max write simultaneously. Because I'm extracting mkv's from the ISO on the same USB drive so it's reading data from the ISO at max rate and writing to mkv files on the same drive at max rate too. This is unusual because I think when most people use USB drives they are only either reading or writing at max speed but not in both lanes simultaneously.

I've set the USB mode in the BIOS to make it so it doesn't use this "auto smart" XHCI thing where it starts as 2.0 mode and then switches automatically to 3.0 mode when the OS loads. Maybe the switch is mucking up the initialisation somehow?

2 hours is usually enough mine would show up in about 15-30 mins or like 2 hours in memtest

[dcoke22]

Have you tried reading from one drive and writing to another on different USB ports?

[pneumatic]

Have you tried reading from one drive and writing to another on different USB ports?

Not yet, but will do when I get the time (too busy the last few days).

First I want to see whether these AsMedia USB ports can produce corruption.

I'm suspicious of the JMicron controller on the SATA-to-USB adapter as well. It's a very VERY cheap piece of kit and it wouldn't surprise me if it wasn't tested under "edge case" loads, heck even MX500's internal SATA controller had some corruption recently fixed with that.

[pneumatic]

Don't seem to be getting any corruption on my Z97X's Asmedia USB ports after doing the last 33 discs. The speed of these ports is slightly slower - 90MB/sec in each lane so read+write total 180MB/sec vs 240MB/sec on the Intel USB ports. Not sure if that has anything to do with it - maybe Intel uses a different error correction threshold to allow higher speeds and it's letting it go too fast and into potential corruption?

Another difference is I've been leaving the USB cable connected so when mobo is powered on it goes USB 2.0 -> 3.0 at the BIOS screen instead of plugging it in at the Windows desktop and going straight to 3.0 mode.

I'm still not totally convinced after 33 discs, will have to do some more discs.

[pneumatic]

Still haven't reproduced any corruption on the AsMedia USB ports so I'm guessing that's it, but I did switch to another USB cable at the same time as switching to AsMedia so it could be that as well. Would be very surprised if it was the cable as the USB spec and hardware/driver implementation should be completely tolerant of that.

[pneumatic]

Had this issue rear its ugly head again, except this time on a different motherboard - Asus H87. This board is from the same generation of Intel chipset as my other Asrock Z97X board - 2014 Haswell socket.

Here is a sample of the errors which occured today according to ffmpeg...

Code: Select all

[matroska,webm @ 000000000046f340] Element at 0x1000523b ending at 0x10005253 exceeds containing master element ending at 0x10005241
[matroska,webm @ 000000000046f340] Element at 0x11000d31 ending at 0x1109c6cd exceeds containing master element ending at 0x1101cf24
[ac3 @ 0000000002f58ac0] expacc 125 is out-of-range
[ac3 @ 0000000002f58ac0] error decoding the audio block
[aist#0:1/ac3 @ 000000000049bd80] [dec:ac3 @ 00000000004afc40] Error submitting packet to decoder: Invalid data found when processing input
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19372032
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19373568
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19375104
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19376640
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19378176
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19379712
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19381248
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19382784
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19384320
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19385856
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19387392
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19388928
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19390464
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19392000
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 38481408 >= 19393536
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 20040 >= 10089
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 20040 >= 10090
[null @ 0000000002f228c0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 20040 >= 10091

The file is definitely broken as it freezes at certain parts during playback.

I then repeated the process of creating the .mkvs from the .ISO image following the exact same process that produced the corrupted file above, and the resulting file is fine.

I'm thinking it's probably an issue with the 2014 Intel chipsets in terms of how they've implemented the USB hardware or possibly the USB driver. Asus and Asrock are kind of the same company so maybe that's another correlation.

The H87 has no third party USB 3.0 ports so I can't avoid the issue that way unlike the Z97.

I've improved the ffmpeg corruption checking Autohotkey script here so that it ignores false positives (benign errors) which saves a lot of time having to comb through many files full of false positives after scanning is complete.

[Billycar11]

urf well you could try on usb 2 ports makemkv is unlikely to exceed the bandwidth unless you are using the speed unlock on the lg/asus drives

[pneumatic]

urf well you could try on usb 2 ports

I had considered that but 20MB/sec is painful when you're doing hundreds of gigabytes. Instead I got this 3.5" SATA hotswap bay so that I can avoid USB altogether.

makemkv is unlikely to exceed the bandwidth unless you are using the speed unlock on the lg/asus drives

Yeah it's not MakeMKV's fault, it's something in Intel 2014 mobo/driver as I've had the same issue on across 2 different brands of drives, 3 different cables and 2 different (but same gen Intel) motherboards. The only common denominator is the Intel 2014 chipset, specifically the Asus and Asrock implementation of it.

What's really weird is the corrupted data sometimes comes from another unrelated DVD. This time I was ripping Lois and Clark season 3, and in one episode at 4:40 a random scene will appear from this other sitcom that I ripped weeks ago, which I even deleted from the drive. I guess when you delete it all it does is flags the blocks as free, and when the USB glitched it must have pointed to those blocks and copied them instead. Still that is spooky that it happened to select blocks that happened to also contain MPEG2 video otherwise the media player would just reach invalid data and not play anything or freeze. Here's the corrupted file if you want to see for yourself that I'm not making this up, and it is not the first time this has happened either. Seek to 4:35 and let it play til 4:40 when it reaches the corrupt data. https://drive.google.com/uc?export=down ... 5W-05eCJfG

I'm still trying to figure out if the corruption only occurs when both USB lanes are reading and writing simultaneously, or if a one-directional read/write is still safe. In any case I've just about had it with USB - too many issues over the years. To me it feels like USB 3.0 hasn't been designed properly and only works under optimal conditions with short, high quality cables and connectors.

I left my AHK script scanning my whole DVD archive overnight and the results came back good which was a huge relief, only some benign errors on a small handful of mkvs according to ffmpeg. Many of the mkvs originated from a one-way USB copy which I thought was safe, and this result seems to imply that it is. I am considering doing a whole new PC build to be finally free of this issue but I'm so invested in the issue now it's probably more of a sunk cost fallacy tbh.

So the problem seems to only occur when both read and write USB lanes are loaded up at max speed. Maybe this is considered an "edge case" and Intel/Asus/Asrock didn't really do enough testing under this kind of load scenario. We're talking 2014 implementation here but USB 3.0 came out in 2008 so 6 years should be enough to figure out the bugs. Although it wasn't until 2012 they seemed to realise the EMI issues so anything's possible.

At the end of the day the USB spec is responsible for error correction and it really shouldn't be allowing it to occur this often, so that's why I'm really disillusioned with USB right now. I read an anecdote on reddit that says 1 corruption every 10TB is normal for consumer devices which if true is still pretty crap cause what if that corruption was some important instruction in the kernel or a driver (I tried reinstalling the drivers btw) then it's game over BSOD and your confidence in your system drops to zero. It's like all the data is meaningless and untrustworthy, a really horrible feeling in the pit of my stomach. Thanks for reading my blog.

[Billycar11]

I had considered that but 20MB/sec is painful when you're doing hundreds of gigabytes. Instead I got this 3.5" SATA hotswap bay so that I can avoid USB altogether.


Yeah it's not MakeMKV's fault, it's something in Intel 2014 mobo/driver as I've had the same issue on across 2 different brands of drives, 3 different cables and 2 different (but same gen Intel) motherboards. The only common denominator is the Intel 2014 chipset, specifically the Asus and Asrock implementation of it.

What's really weird is the corrupted data sometimes comes from another unrelated DVD. This time I was ripping Lois and Clark season 3, and in one episode at 4:40 a random scene will appear from this other sitcom that I ripped weeks ago, which I even deleted from the drive. I guess when you delete it all it does is flags the blocks as free, and when the USB glitched it must have pointed to those blocks and copied them instead. Still that is spooky that it happened to select blocks that happened to also contain MPEG2 video otherwise the media player would just reach invalid data and not play anything or freeze. Here's the corrupted file if you want to see for yourself that I'm not making this up, and it is not the first time this has happened either. Seek to 4:35 and let it play til 4:40 when it reaches the corrupt data. https://drive.google.com/uc?export=down ... 5W-05eCJfG

I'm still trying to figure out if the corruption only occurs when both USB lanes are reading and writing simultaneously, or if a one-directional read/write is still safe. In any case I've just about had it with USB - too many issues over the years. To me it feels like USB 3.0 hasn't been designed properly and only works under optimal conditions with short, high quality cables and connectors.

I left my AHK script scanning my whole DVD archive overnight and the results came back good which was a huge relief, only some benign errors on a small handful of mkvs according to ffmpeg. Many of the mkvs originated from a one-way USB copy which I thought was safe, and this result seems to imply that it is. I am considering doing a whole new PC build to be finally free of this issue but I'm so invested in the issue now it's probably more of a sunk cost fallacy tbh.

So the problem seems to only occur when both read and write USB lanes are loaded up at max speed. Maybe this is considered an "edge case" and Intel/Asus/Asrock didn't really do enough testing under this kind of load scenario. We're talking 2014 implementation here but USB 3.0 came out in 2008 so 6 years should be enough to figure out the bugs. Although it wasn't until 2012 they seemed to realise the EMI issues so anything's possible.

At the end of the day the USB spec is responsible for error correction and it really shouldn't be allowing it to occur this often, so that's why I'm really disillusioned with USB right now. I read an anecdote on reddit that says 1 corruption every 10TB is normal for consumer devices which if true is still pretty crap cause what if that corruption was some important instruction in the kernel or a driver (I tried reinstalling the drivers btw) then it's game over BSOD and your confidence in your system drops to zero. It's like all the data is meaningless and untrustworthy, a really horrible feeling in the pit of my stomach. Thanks for reading my blog.

ah i thought you were just doing 1 copy at a time usb 2 should go to 48MB but with over head the most i have ever seen is 42MB.

thats really weird for it to randomly combine it with a another video

i have had pretty good luck with usb i used to connect all the drives sata but i can do more with USB i write 1.2TB over usb a day but its only using about 106MB/s with 6 pioneers doing UHD at once on a usb hub that should be ok till 500ish
this poor ssd is less than 1 1/2 years old and i did this to it over 100% endurance used lol 400TB


i think this unreliability is making it seem worse that it really is like LG/asus did with me i think a whole new platform will solve your issues and restore yor lost confidence in your data being safe over usb.