audiomuze

Meeting the storage needs of a digital home

Recommended Posts

Hi

I'm hoping I can get some advice as to storage options to cater for storing large volumes of data on a home network for entertainment and archival purposes. I currently have approximately 20 GB of data stored on two separate PCs running Linux with RAID 5 arrays. The data is accessed from a number of locations in the house and is principally streaming audio and video content.

The RAID arrays were built using 3TB NAS drives and whilst everything is working just fine, however, I'm faced with the knowledge that in time I will need to expand the arrays/ storage capacity of each. Adding additional drives in a RAID 5 config will almost certainly result in hardware failure during array expansion, and is thus not a viable option. Similarly, should a current drive fail, another may fail whilst integrating a replacement disk.

On the face of it I don't really need to have RAID in the mix as I could simply go with individual drives, however, from a media access perspective:

  • it's great to be able to store things logically, e.g. all artists whose first name starts with the letter A are filed under A/
  • being able to point to a single volume is pretty convenient
  • there's a performance gain provided by accessing the array rather than an individual drive

Looking at options and having regard to drive sizes it would seem that RAID is simply not a viable alternative going forward, and with my limited knowledge the only solution I can think of is to deploy individual drives by creating a single folder on an OS disk and including symbolic links to each drive or mounting them to the single folder, along the lines of the following:
/mediahome
/mediahome/d1
/mediahome/d2
/mediahome/d3
etc.

where d1...3 are symbolic links pointing to the mount point of the individual drives, or otherwise used as the actual mount points for each of the drives. Media software can then be pointed to /mediahome and they'll find all the content on the individual drives.

What my proposed solution doesn't address at all is being able to store things logically, e.g. all artists whose first name starts with the letter A are filed under A/ (when a drive is filled one is forced to add new albums that should be filed under A/ to another physical drive, presumably under another folder, also A/ ... you can see the mess that will ensue).

So, my question is how are others dealing with storage requirements in the home in instances where 20-40TB of storage is required? Are there any affordable enterprise/ SOHO solutions e.g. DAS/ Unified Storage that may be viable and affordable?

Edited by audiomuze

Share this post


Link to post
Share on other sites

Why not look at a NAS? You sound like the perfect use case and then you won't need to manage both machines and worry about their RAID volumes quite so much. An 8-bay Synology or QNAP would get you up to 48TB raw, with the ability to expand via JBOD if needed. If you see needing a ton more space, one of their rack units would do the trick as well. You have several options...

Share this post


Link to post
Share on other sites

^^^ using a 3rd party NAS is no different to running mdadm on any Linux box - even the developer of mdadm does not recommend running raid arrays of the size I'm already running because drive failure stats are such that building/expanding or repairing an array of such size will give rise to a number of r/w operations that makes one or more drive failures highly probable.

Share this post


Link to post
Share on other sites

+1

Ive been doing 6x3tb arrays in raid6, and moving onto the next array.... Sync checks are in a 1 week frame. It's usable. And bitmaps makes things take seconds instead of days.

Mdadm rocks. 4/2core amd @4ghz and $30 sata cards can make a screaming system.....

Sent from my rooted HTC Supersonic using Tapatalk 2 Pro

Share this post


Link to post
Share on other sites

I guess it just depends what you're after. You could opt for several smaller volumes. My suggestion was more about manageability and expandability though you're right, rebuilding for days at a time isn't fun should something go bad and that article is a good reference on the viability of RAID6. I think that's why what the guys at X-IO are doing is so cool, aggregating storage and a much finer level, so even when a platter goes bad on a drive they can live without it. You pay for it some with over-provisioning loss more or less, but that type of management is pretty cool.

Share this post


Link to post
Share on other sites

IMO, (non-RAID) parity protection schemes such as SnapRAID or FlexRAID makes a lot of sense for home media storage. You don't need real-time RAID for data that seldom changes. If you're concerned about the limitations of single or dual parity, just employ multiple smaller arrays.

And... back up your most important data. RAID/parity is fine for protecting agaginst disk failures, to a degree, but the really important files get backed to at least two different places. In most situations, the bulk of a 3 or 30 or 300TB media collection will be films and TV. I can live with losing my rips of Frozen and Seinfeld, but I will do everything in my power to protect family photos, writing, financial information, etc.

Share this post


Link to post
Share on other sites

Thanks for the suggestions. Snapraid looks pretty compelling. If Btrfs were ready for prime time I'd just go with that and replicate to a 2nd machine with the same setup. For the moment I'll replicate to a Snapraid setup on a 2nd machine which I'll store on other premises.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now