Strange corruption issue - MakeMKV's fault or bad HDD?

[pneumatic]

i write 1.2TB over usb a day

Hmm interesting. That is a lot of video and there is no way you could be watching 1.2TB a day so maybe you should scan the files for errors with ffmpeg afterwards.

It's a similar situation where you buy a DVD box set and think you've got it safely but until you actual play every disc you could have a bad disc in there and not know it until a year later when you finally got around to watching/ripping that episode.

but its only using about 106MB/s

Hmm well my third party AsMedia ports which seem to be immune to corruption will only do 90MB/sec (180MB/sec total read+write) whereas the Intel ports does about 130MB/sec (260MB/sec total read+write).

So just to clarify, I've got my DVD .ISO file on the USB drive, and I'm extracting it to mkv files with the destination being the same USB drive. I figured if MakeMKV is reporting 130MB/sec writing to mkv files, that must mean it's also reading at 130MB/sec from the ISO file, making the total 130+130=260 is that correct?

Back in 2014 when this motherboard was new, were people even connecting SSD's over USB 3.0 and getting these kinds of fast speeds like 260MB/sec? Maybe that's why it wasn't detected in testing or reported by many people until recently where more people are actually trying to utilise USB 3.0 to its full potential and seeing that it actually breaks when you stress it too hard.

[Billycar11]

i write 1.2TB over usb a day

Hmm interesting. That is a lot of video and there is no way you could be watching 1.2TB a day so maybe you should scan the files for errors with ffmpeg afterwards.

It's a similar situation where you buy a DVD box set and think you've got it safely but until you actual play every disc you could have a bad disc in there and not know it until a year later when you finally got around to watching/ripping that episode.

but its only using about 106MB/s

Hmm well my third party AsMedia ports which seem to be immune to corruption will only do 90MB/sec (180MB/sec total read+write) whereas the Intel ports does about 130MB/sec (260MB/sec total read+write).

So just to clarify, I've got my DVD .ISO file on the USB drive, and I'm extracting it to mkv files with the destination being the same USB drive. I figured if MakeMKV is reporting 130MB/sec writing to mkv files, that must mean it's also reading at 130MB/sec from the ISO file, making the total 130+130=260 is that correct?

Back in 2014 when this motherboard was new, were people even connecting SSD's over USB 3.0 and getting these kinds of fast speeds like 260MB/sec? Maybe that's why it wasn't detected in testing or reported by many people until recently where more people are actually trying to utilise USB 3.0 to its full potential and seeing that it actually breaks when you stress it too hard.

its all drive testing im that guy selling flashed and tested UHD drives

[pneumatic]

Also I checked S.M.A.R.T attributes in CrystalDiskInfo many times and it never says anything is wrong. So the SSD is blissfully unaware that it's writing garbage to the cells, more reason to think it's happening downstream at the USB stage.

I had seriously considered whether it's the USB to SATA controller's fault but I've had the same corruption on 2 different drives that use different USB to SATA controllers so I figured it's unlikely (albeit not impossible - maybe they both use some dodgy write caching strategies)

[dcoke22]

Have you thought about a USB PCIe card?

[pneumatic]

its all drive testing im that guy selling flashed and tested UHD drives

Oh I see. Can I ask what software you use for testing? I was looking at this one called FreeFileSync to verify the actual contents of files are the same. I think first it checks to see if the number of bytes are the same, and if they are then it will start comparing contents but I couldn't find a clear answer on that. My corrupt files have the same number of bytes but different checksums.

[pneumatic]

Have you thought about a USB PCIe card?

Yes, that may be a good option but I'm going to try the SATA rack bay thing first as well.

[Billycar11]

Oh I see. Can I ask what software you use for testing? I was looking at this one called FreeFileSync to verify the actual contents of files are the same between 2 discs. I think first it checks to see if the number of bytes are the same, and if they are then it will start comparing contents but I couldn't find a clear answer on that. My corrupt files have the same number of bytes but different checksums.

just makemkv i do full backup of UHD blurays on blurays it has a hash table and checks the files are correct as it copies. not sure if it checks before or after it writes to disk i asked once but dont think ill get an answer.
not sure if it does that on dvds but you saw makemkv said something when it saw weird junk

[pneumatic]

not sure if it does that on dvds but you saw makemkv said something when it saw weird junk

Makemkv never gives me any error messages. Anyway screw these USB ports I'm done wasting my life on them I'll get a PCI-E card with hopefully 8 working USB 3.2 ports which might be even faster than 3.0, now that I think about it I should have done this ages ago.

[Radiocomms237]

~ on blurays it has a hash table and checks the files are correct as it copies. not sure if it checks before or after it writes to disk ~

The hash table can only verify the integrity of the files on the optical disc itself. It cannot be used to check for file copy errors once the content has been decrypted and muxed into an .mkv file.

~ not sure if it does that on dvds ~

AFAIK there is no way to detect errors from DVDs unless it finds a sector of the disc is unreadable.

[Billycar11]

~ on blurays it has a hash table and checks the files are correct as it copies. not sure if it checks before or after it writes to disk ~

The hash table can only verify the integrity of the files on the optical disc itself. It cannot be used to check for file copy errors once the content has been decrypted and muxed into an .mkv file.

just adding some real world experience still dont know exactly where and when it happens but when a stick of ram went bad in my test pc i was getting hash errors on pioneers and well thats not possible they dont have that jamless play junk lg has so makemkv was catching it somewhere between it decrypting it in ram and after writing it to disk/hdd it may be catching it in ram or it may catch it after it writes it and then compares not sure.

mike has eluded to adding checking after the fact eventually.
dcoke22 linked it a few days ago
viewtopic.php?p=119692#p119692

[pneumatic]

It seems a lot of these third party PCIe USB expansion cards could be cheaply made junk prone to failure and corruption as well. Not sure I want to risk 2 years of archiving to some $30 PCB made to a price. Maybe I just have to get used to the idea of scanning everything with ffmpeg. Maybe that is just the reality if you want data integrity.

[dcoke22]

It seems a lot of these third party PCIe USB expansion cards could be cheaply made junk prone to failure and corruption as well. Not sure I want to risk 2 years of archiving to some $30 PCB made to a price. Maybe I just have to get used to the idea of scanning everything with ffmpeg. Maybe that is just the reality if you want data integrity.

I don't really know all that much about USB chipsets, but something from Sonnet (https://www.sonnettech.com/product/allegro-pro-usb31-pcie.html) or another company serving the video editor market seems like a reasonable bet. That market generally prioritizes reliability above all else since time is money for those folks.

[pneumatic]

It seems a lot of these third party PCIe USB expansion cards could be cheaply made junk prone to failure and corruption as well. Not sure I want to risk 2 years of archiving to some $30 PCB made to a price. Maybe I just have to get used to the idea of scanning everything with ffmpeg. Maybe that is just the reality if you want data integrity.

I don't really know all that much about USB chipsets, but something from Sonnet (https://www.sonnettech.com/product/allegro-pro-usb31-pcie.html) or another company serving the video editor market seems like a reasonable bet. That market generally prioritizes reliability above all else since time is money for those folks.

That looks better but costs more than a new motherboard.

Saw this Inateck brand one on Amazon that seems to have good reviews but 2 of them costs more than a new motherboard.

There's another generic popular one using a Renesas UPD720201 controller that only costs $30 so I'll try one of those and see if I can reproduce any corruption on it. But one of the amazon reviews is saying it uses an Asmedia controller so maybe there's some chip lottery going on.

I was thinking about how the corruption manifests as copying wrong blocks of memory and what could be deduced from that. I think that could mean the data itself isn't getting corrupted because the corrupted data is MPEG2 video from another DVD but still in tact because the media player can still decode it into video and audio which seems to imply it's simply reading from the wrong address rather than the data getting scrambled.

[pneumatic]

Got the unbranded PCI USB card off ebay which looks like this


https://www.amazon.com.au/Expansion-Exp ... B0B5M1J3LY

They've scrubbed out the chip numbers for whatever reason but they are Renesas UPD720201 and UDP720210.

There is no driver support for Windows 7, only 8+ so I had to hunt down the drivers from the support page of another branded version of it (Startech - https://www.startech.com/en-au/cards-adapters/pexusb3s7).

The drivers are ancient from 2012 but the release notes have some interesting things to say about various USB bugs they found which were causing BSODs and such.

Speed is on par with my good mobo ports (Asmedia) - 90MB/sec (180MB/sec total read+write). Intel ports which suffer corruption do 260MB/sec so maybe that is a good sign they don't runf aster.

No corruption yet but we shall see.

In the meantime I have uninstalled the Intel Rapid Storage Technology driver in case that was causing the issue, as it is another common denominator in all the corruption scenarios. USB speed not affect after uninstall.

Spotted another issue that was giving me false *negatives* for corruption - Windows seems to cache files in RAM so when I run ffmpeg.exe to check the mkvs for errors, the USB drive light was not flashing. I take this to mean the result is invalid as it must be getting the mkv data from system RAM and not actually checking the bytes on disk. This is consistent with a scenario where a mkv file had no corruption according to ffmpeg.exe but then had corruption after copying it elsewhere.