2000+ rips on my 8 drive NAS

[newbob]

I am wondering how y'all store your rips. For my part, I have 2000+ blu-ray rips stored on a NAS with 8 drives of 20 TB each. I use TrueNAS, with the drives in a dual parity array. I do daily offsite backups with rsync to a server in my vacation home, a few hundreds miles away. The whole setup has worked very well for me, for many years. I had 3 drive failures over the last decade or so, but never once lost data thanks to raidz2.

You do need to set up monitoring to be alerted as soon as possible when your array is unhealthy. Raid protects you from drive failures, but you still need to eventually replace failing drives.

I rip a few discs per week. MakeMKV has never let me down. Great software. Happy paying user!

---

Edit: I am, however, not going to continue to be able to rip myself. Between launching a startup, travelling, family life, I can't find the time to sit down and rip. A few months ago I hired a full-time Personal Assistant who I delegate a lot of things in my life, but she'd be the wrong person to teach her MakeMKV and delegate that to. So I'm instead looking to hire a remote assistant who can do ripping for me. I pay 30 USD per rip. I will request around 20 movies/month, so your income can be 600 USD/month. Email me for details: movies5draeh@proton.me

[Woodstock]

Mine go to a pair of Synology units. The main one's half-full now, with around 100,000 files of various titles. The Synology software keeps track of failures for me.

[Radiocomms237]

I started-out digitizing my optical media library with an 8TB WD NAS, and I thought "I'll never fill that up".

Then, once that was full, I bought a Synology 4-bay NAS and loaded it with 16TB drives (the largest drive on the compatibility list), and I thought "I'll never fill that up".

Then, once that was full, I bought the 5-bay expansion unit and loaded it with 16TB drives, and I thought "I'll never fill that up".

Then, once that was full, I started buying spare drive trays and playing 'musical hard drives', which I've been doing ever since.

I'm currently at 248TB of media (not counting any PC storage) and I'm adding a 16TB drive roughly every three months.

I've also learned to stop thinking "I'll never fill that up" !!!

I'm now saving-up for a new 36-bay Synology rack-mount NAS so I can have access to all my media without swapping-out drives all the time. Once that's full, I'll be well on my way to having a Petabyte of storage!

I'm maybe a little over halfway through my optical media collection at the moment (although it's still growing almost as fast as I'm working on it), I still have boxes and boxes of DVDs in storage, and I haven't even started on the UltraHD content yet. I expect that will suck-up HDD space even quicker.

I don't use RAID at all, I just don't have the space to keep two copies of everything (and I'd hate to think how expensive that would be in terms of buying twice as many HDDs) so I regard the physical disc (DVD/BRD/UHD/etc.) as the backup copy of my data.

Yes, it would be a huge PITA to re-rip an entire HDD worth of data (although a total loss of all 16TB is unlikely), but weighing the risk against the cost, I consider that risk acceptable (touch wood). Plus, I believe that they way I'm filling a drive, then removing it from the NAS for safe storage, alleviates some of the risk of drive failure.

[newbob]

Woah, Radiocomms237, you have the largest collection I have heard about.

I recommend you to use RAID though. You wouldn't need twice as many drives. If you use dual-parity RAID, for example, the overhead is only 2 drives. So if you have 10 drives' worth of data, you only need 12 drives total, a 20% overhead.

[Ezatoka]

I don't use RAID at all, I just don't have the space to keep two copies of everything (and I'd hate to think how expensive that would be in terms of buying twice as many HDDs)

This is only true for RAID 1 aka Mirroring. if you use like RAID 5 or 6, it'll be only a few disks which will be used for parity information, so let's say you have 10 discs in a RAID 6, then up to two can fail without losing any data. After replacing the damaged discs the RAID will rebuild itself.

[Radiocomms237]

Not sure what's going on with this thread, the OP disappeared yesterday and now it's back, but heavily edited?

~ I'm instead looking to hire a remote assistant who can do ripping for me. ~

I'm not looking for the work (Lord knows I have enough of my own to go on with), but I'm interested to know how you plan to get the data to a "remote assistant" in the first place?

Do you plan to mail individual discs back 'n forth, or back them up locally and transfer TBs of data each way over the internet? Or would you trust them enough to give them, say, several months' worth of work out of your collection at a time (assuming you would meet-up every so often)?

This is only true for RAID 1 aka Mirroring. if you use like RAID 5 or 6, it'll be only a few disks which will be used for parity information, so let's say you have 10 discs in a RAID 6, then up to two can fail without losing any data. After replacing the damaged discs the RAID will rebuild itself.

So how would this work with me swapping-out full HDDs for new HDDs every so often? Wouldn't RAID try to automatically re-copy the data from the full HDD to the fresh one?

[awdspyder]

So how would this work with me swapping-out full HDDs for new HDDs every so often? Wouldn't RAID try to automatically re-copy the data from the full HDD to the fresh one?

I've been managing all manner of RAID arrays for decades in my professional life (Linux mdadm RAID, Dell Perc, HP Smart Array, LSI/Avago hardware RAIDs, FreeNAS/TrueNAS RAIDz, etc.)

You definitely don't want to intentionally swap disks out in a parity RAID-5 scenario. If you pull out a disk, the array will be "degraded." When you put a new disk in to repair the array, the data that was previously on the missing disk will be copied to the new disk to make it whole. Furthermore, in a degraded state, you're risking data loss during the rebuild period, as a (perhaps statistically unlikely) 2nd failure will result in total data loss. RAID-6 addresses this to some degree, but at the detriment of write performance. In a professional setting, we use all RAID-10, all the time, because we can.

Furthermore, RAID arrays are typically "expanded" or "grown" by adding additional disks, which you'd need additional disk slots for. Some proprietary systems do allow you add larger drives, but you generally wouldn't benefit from the additional space until all disks are replaced.

If you want to dabble with something a bit in between, I strongly recommend StableBit's Drivepool (https://stablebit.com/DrivePool), which allows you create multiple copies of files across disks (or not - your choice), aggregate disks in a single large pool, evacuate files from a disk for removal, balance space utilization, etc. It's Windows only, however, but not sure if that matters.

Finally, remember that RAID is not a backup; RAID is for uptime. Nothing replaces a good backup.

[Radiocomms237]

~ I strongly recommend StableBit's Drivepool (https://stablebit.com/DrivePool) ~ It's Windows only, however, but not sure if that matters. ~

Thanks for your reply.

When I talk about swapping full drives for fresh ones, I'm doing this on a Synology NAS, which I believe runs some flavor of Linux (?), so "Windows only" software is of no use to me.

I think I can already do most of the functions you mention within the Synology OS, it can certainly do all manner of RAID and allows for the creation of "storage pools" anyway.

Unfortunately, I'm maxed-out at nine drive bays on my current NAS (that's including the expansion unit), so sacrificing two or three bays for a RAID 5/6 setup would be not inconsequential. And going by the advice above, I think that utilizing RAID would put an end to my current 'musical hard drives' methodology.

As I said previously, and I know it's not a great solution, but I consider my optical disc collection to be my 'if-all-else-fails' backup. I am also keeping the full backup created by MakeMKV, as well as the ripped & prepped .mkv files, so if we're only talking bad sectors and not a total drive failure, then there's a reasonable chance I'd still be able to recover most of the data.

When I (eventually) get my planned 36 bay rack-mount NAS I will reconsider a RAID array of some description, assuming it's still only two or three drives required. Or does that RAID 5/6 requirement scale upward with more bays?

[awdspyder]

I manage about 70 Synology NASes currently, in various environments, and you're exactly correct. The DSM software is Linux and won't support DrivePool, but you can have all the RAID options at your disposal or just JBOD as you're doing now. Really, I think the answer is as you state with a larger NAS.

My setup is a little more complicated (Linux SAN running mdadm/LVM/LIO, with Mellanox fiber channel block devices exported to clustered Hyper-V hypervisors) that lets me add 4 disks at at time, will online scale to 512 disks total (limits of 2 x LSI SAS HBAs), and uses commodity hardware. Basically, if get low on disk slots (unlikely with 512 drives, lol), I can migrate data to a new RAID group, and "retire" the old disks, freeing up those slots. This of course requires another OS (Linux or Windows) to serve up the data. I personally love the solution, but I don't think I'd recommend it for someone that didn't live in that IT world.

Truthfully, you could achieve something similar with Synology NASes, by utilizing the iSCSI functionality on the NAS, connecting the LUNs to a Windows box running DrivePool. This would allow you to add Synology NASes at your leisure, DrivePool would present the storage as a single volume on Windows (presumably running Plex/Emby) and would incur zero to minimal downtime during expansion. You could even add a new NAS, migrate the data to that device, and retire the old if you desired. Expansion would be virtually unlimited. Of course, not everyone wants the level of complexity that iSCSI and a separate server adds. I'd think you'd be quite happy with a 36-bay NAS.

Another thing I'll mention when scoping this out: If you go with the 36-bay NAS and decide to utilize RAID-5/RAID-6 or even RAID-10, I'd be cautions about the number of disks in a single RAID group. Keep in mind that the larger the RAID group the longer the rebuild times, and the greater potential risk of data loss during a rebuild. That said, RAID groups can be as big as your risk tolerance (i.e. 35+1 RAID-5 or 34+2 RAID-6 if you're a madman). However, IMHO, going with smaller RAID groups (4+1/5+1 RAID-5 or 6+2/8+2 RAID-6), and using disk pooling in the NAS or otherwise is a more sound long-term solution, especially with large consumer SATA disks.

Which brings me to a final point - I'd spend the extra coin and purchase enterprise 7.2k drives. We currently use Seagate EXOS drives (albeit SAS) in the data center for large WORM backup storage and have yet to experience a failure in hundreds of TB written. I've also used plenty of WD/Hitachi He enterprise 7.2k drives with similar results.

[Radiocomms237]

All of my NAS drives (without exception) are Western Digital WD161KFGX 16TB Red PRO 3.5" 7200RPM SATA3's.

I don't like to mix 'n match brands etc. and I know there are better drives available but I feel the 'reds' are more than good enough (and more than expensive enough) for my purposes.

(Touch wood) I've not had a drive failure yet.

[awdspyder]

Nah, you're doing it right. I like the WD Red Pros just fine. You get the 7.2k spindle speed and the 5-yr warranty. Red Pros are all we use in our client Synology on-prem backup solutions. Rarely ever have failures (can't remember the last one) with nearly 150 Reds in various sizes in production use (mostly RAID-1 in DS224+ NASes).

Let us know when you get that 36 bay monster up and running!

[Radiocomms237]

When I say 36 bay, I'm actually looking at a Synology RS3621RPxs 12 bay NAS with two RX1217RP 12 bay expansion units attached.

I mostly like that it's scalable, so I can buy the NAS first and add the expansion units as needed, not to mention the options of 16GB RAM, an M.2 adapter card and dual SFP28 fiber ports!

[dcoke22]

StableBit's Drivepool looks interesting. Do I correctly interpret that you could turn an old Windows computer and some old hard drives into something more useful than things to collect dust in the closet? (Within whatever your fault tolerance for failure is, of course.)

[awdspyder]

Correct - I use it in two ways:

1. In my storage server to aggregate multiple RAID volumes into a larger single drive pool for ease of presentation as a single volume/file share
2. In my backup server which is essentially just JBOD to provide a single backup target without having to worry about tracking individual drive utilization

For the backup server, I don't really care about having redundancy since it's the backup (hence no RAID) except for really critical stuff like family photos, financial documents, etc. For those folders, I direct DrivePool to keep some number of copies spread across the JBOD (2x, 3x, whatever). You can also tell it what level of drive filling you want (equal across drives, fill one to 90% then move to other drives, etc.); it's quite flexible.

Another thing I like about it is that you can still get to all of your data if something goes catastrophically wrong. It's just their filesystem layered on top of NTFS. You can pull a single drive out, stick it in another machine, and still access your data.

[newbob]

I'm not looking for the work (Lord knows I have enough of my own to go on with), but I'm interested to know how you plan to get the data to a "remote assistant" in the first place?

Do you plan to mail individual discs back 'n forth, or back them up locally and transfer TBs of data each way over the internet? Or would you trust them enough to give them, say, several months' worth of work out of your collection at a time (assuming you would meet-up every so often)?

(As a new forum member, when I edited my original post, it was temporarily hidden until a moderator could approve the edit. That's what happened.)

The remote assistant would simply buy blu-rays on my behalf. They can keep them. I don't care to have the discs. A good chunk of them degrade anyway after 5-10 years to the point they become unreadable. Blu-ray is just not a stable media. As to the data, transferring 20 movies a month is only 1 TB/month. It's doable with most high-end cable & fiber residential plans in the US.