akhtis

Designing a home storage solution

Recommended Posts

Hello everyone!

first, excuse me for my poor English. I am Belgian and my mother tongue is French.

I stumbled across this wonderful forum a few days ago while I was searching for informations about RAID enclosures and home servers. As I was reading posts, I realized that some users here have a really impressive amount of knowledge and experience in this area. So, if I may take a bit of your time, I would like to present you what I'd like to achieve in order to collect some advices. In fact, let's be honest: I really need help on this one! :huh:

I currently have roughly 2Tb of data that reside on 2 WD MyBook 3.0 1Tb and 1 Hitashi internal 500Gb drive and I have the exact same configuration stored in a safe place for backup. Those data are exclusively music files, ebooks, images, movies (mkv) and archives (7z). I store all executables/OS/Configuration Files on another location so there is not much stressed imposed on those drives. But note however that being a paranoiac weirdo I encrypt all my drives using Truecrypt Full-Disk-Encryption (Serpent-Whirlpool).

However, my problem is that this amount is rapidly growing and I am getting tired to add a consumer-grade external drive whenever I need more space.

So right now, I'm looking for a future-proof, convenient and reliable solution to store ALL of my data. Performance falls in second place: I don't care if I will have a maximum read of 25Mbit/s as long as my data are secure. Secured against unauthorized manipulations and secured against corruption. I want my data to be literally carved in stone as they are not subject to much changes.

I want a capacity of at least 6Tb for start, but it has to be expandable (to at least 12Tb). It also has to be a single volume and to be fault-tolerant so I'm thinking RAID6 (or more exactly RAIDZ2, see below).

As I mentioned above, I am kind of paranoiac regarding security and data integrity, so I want to implement my future setup as a zpool with ZFS. So I will probably go for openIndiana for OS and here comes the tricky part: I want to be able to access my files from windows AND linux and also to be able to maintain the same level of security I have now, which means encryption (but not necessarily FDE). As I mentioned above, due to the kind of data I intend to store on my new system, the drives will not be put under much stress so no need for redundant PSU, battery card etc... nor for extensive server-like possibilities. I will be perfectly happy if the only way to access my data was to directly connect the system to my desktop by eSATA or USB3 for example. Plus, I think the system will only be powered on between 4 and 8 hours a day.

So here I am: completely lost before the astronomical number of hardware and/or software solutions possible. I should also mention that I am on a budget of roughly 1K$ and the less it costs me the better (without losing reliability).

Imagine you were me, what would you do?

any solution/hardware/software recommendation is wholeheartedly welcome.

Thanks in advance

Edited by akhtis

Share this post


Link to post
Share on other sites

This open might not be as cool as others, but when it comes to what you are talking about needing (security, ease of use, reliability, compatibility) I would seriously consider one of the embedded NAS platform like a Synology or the like. Super easy to expand as time goes on, good encryption if you need it (through external software like you use now), very reliable, and cross compatible with any platform. Another cool thing is if the NAS failed in some matter, at least I know the Synology just uses mdadm for its RAID profile, so you could recover it in any linux system.

The only problem you might run into is future expansion. Being a fixed unit, it has 4 bays. 8TB max using 2TB drives, 12TB max with 3TB drives. Compared to a desktop where you might be able to just throw in up to 6 drives on internal storage, and pop in addon cards, you are a tad more limited. The tradeoff though is a server that is dead reliable, uses about 40W of power when running, and is super small.

For drives, any of the current Eco-friendly models would be fine. I am a big fan of the Caviar Green models, although they don't run friendly with systems like the QNAP models because of the 4K sectors. Synology doesn't care and they work as well as any other drive installed.

Share this post


Link to post
Share on other sites

Thank you very much for your opinion! I'll look into that later this day.

Since you were speaking of drives, I would like to add that I am myself a big fan of WD but due to my severe budget at the moment I thought of going with samsung drives. I can have 6 Spinpoint F4 Eco-green 2To for just 450$ which seems like a good bargain. However as reliability is my biggest concern, I want rock-solid drives. So, please share your experiences, if any, with samsung drives.

Share this post


Link to post
Share on other sites

I've looked up the range of product TSullivan proposed and I found two major flaws: The inability to use ZFS as volume management and new standards like SATAIII and USB3 are generally not supported. This is why I will probably need to go with a full PC like setup in order to be able to use any hardware/software I want. However I'm still completely clueless regarding what hardware I need...

Share this post


Link to post
Share on other sites

I've looked up the range of product TSullivan proposed and I found two major flaws: The inability to use ZFS as volume management and new standards like SATAIII and USB3 are generally not supported. This is why I will probably need to go with a full PC like setup in order to be able to use any hardware/software I want. However I'm still completely clueless regarding what hardware I need...

Ahh yea well some of the more exotic RAID implementations might not be available on some of the off-the-shelf NAS units.

If you are dead set on desktop hardware, give it a couple of months until Intel's new stuff comes out which should support all the fancy new standards. Right now Intel stuff doesnt support USB3 or SATAIII natively... its all thought external controllers which in my opinion just plain suck right now.

Depending on the deals currently being offered, I have seen the 2TB WD and Samsung drives both hovering in the same range.

Share this post


Link to post
Share on other sites

Just scanning through your post...

What about an HP x510 Windows Home Server - get a 1TB one, then add 3 2TB drives (3TB should work too, but they are pricey)

It has 3USB ports and 1 eSata port, you can up to 5 more drives via the eSata port in an external enclosure as it has a port multiplier (ICH9 on that note ;):))

Alternatively, you could build your own Home Server from a desktop case an accommodate more than 4 drives in the same "case".

-> If you want to read up on the home server:

http://www.microsoft.com/windows/products/winfamily/windowshomeserver/default.mspx

(PS: You want version 1 of it, not Vail which is still in Beta and might not have the drive extender)

PPS: Yes, I like the Windows Home Server platform :D

Edit:

PS: My Home Server + 3 2TB drives, Seagate was about 800€ - but the Seagate drives have become cheaper since.

Edited by DetlevCM

Share this post


Link to post
Share on other sites

Hey guys,

After extensive search, I found exactly what I was looking for on the software side: FreeNAS

http://en.wikipedia.org/wiki/FreeNAS

It is based on FreeBSD and as such is Rock-Solid and fully supports ZFS and disk encryption.

If any of you have experiences with this piece of software, please let me know!

For the hardware side, as TSullivan said, I think it would be nice to wait until SATAIII and USB3 are supported in chipsets. I'm currently looking on the AMD side to see if it's better than Intel's at the moment.

One question remains though, how much computing power do I need to run a system like that (considering encryption of course)? I was thinking of going for 2gb DDR3 and Atom/I3, will it be enough?

Share this post


Link to post
Share on other sites

Hey guys,

After extensive search, I found exactly what I was looking for on the software side: FreeNAS

http://en.wikipedia.org/wiki/FreeNAS

It is based on FreeBSD and as such is Rock-Solid and fully supports ZFS and disk encryption.

If any of you have experiences with this piece of software, please let me know!

For the hardware side, as TSullivan said, I think it would be nice to wait until SATAIII and USB3 are supported in chipsets. I'm currently looking on the AMD side to see if it's better than Intel's at the moment.

One question remains though, how much computing power do I need to run a system like that (considering encryption of course)? I was thinking of going for 2gb DDR3 and Atom/I3, will it be enough?

For pure file management an Atom will be enough - if you want to encrypt stuff and have fast access to it you'll need something more powerful.

Hardware encryption might work better in that case.

Also - how strongly will you encrypt etc.

Share this post


Link to post
Share on other sites

I personally like Serpent which was one of the finalist algorithm for the AES Contest back in 1998 because few weaknesses were discovered. But the winner, Rijndael (now known as AES) is fine too.

I tend to prefer Full disk encryption over file container or other methods, so it needs a little more operations. Plus, I like to encrypt the system drive too... So I suppose an Atom will not be enough...

You mentioned another area where I am a complete noob: Hardware encryption. I always used open source software encryption for fear of reverse engineering and back-doors but for this particular case I might consider it. What hardware would you recommend?

Share this post


Link to post
Share on other sites

I personally like Serpent which was one of the finalist algorithm for the AES Contest back in 1998 because few weaknesses were discovered. But the winner, Rijndael (now known as AES) is fine too.

I tend to prefer Full disk encryption over file container or other methods, so it needs a little more operations. Plus, I like to encrypt the system drive too... So I suppose an Atom will not be enough...

You mentioned another area where I am a complete noob: Hardware encryption. I always used open source software encryption for fear of reverse engineering and back-doors but for this particular case I might consider it. What hardware would you recommend?

I'm no expert on encryption in drives - I think most implement some form of AES though.

If security is a key concern I think you should look at industrial drives.

Share this post


Link to post
Share on other sites

Yes, security IS a major concern but my major restraint still is budget. I don't have the choice, I have to go with low-cost consumer drives.

I'm looking at the moment for the hardware I'll go for.

I've decided to choose an AMD platform for its native support of SATAIII. I'm looking at motherboards for now but have found none so far that math all of my criteria (except one that costs 900$) which are:

1- AM3 Socket

2- DDR3 support

3- Native SATAIII support

4- 8 or more SATA connectors. 6 is definately not enough.

(5- Onboard graphics)

(6- eSATAIII Support)

If you find by chance something like that, please let me know :rolleyes:

Edited by akhtis

Share this post


Link to post
Share on other sites

Hold tight for CES next week, there will be a slew of new gear at least announced at the show; you'll probably have much beter guidance as to what's coming then so you can make a better decision.

Share this post


Link to post
Share on other sites

Hold tight for CES next week, there will be a slew of new gear at least announced at the show; you'll probably have much beter guidance as to what's coming then so you can make a better decision.

The only problem there is that the fact that stuff is announced doesn't mean it's available.

Canon announces stuff... and it becomes available 3 to 4 months afterwards... except maybe the rebels...

On the plus side - it will possibly result in a price drop of existing technologies.

Share this post


Link to post
Share on other sites

Oh true enough - my point was more about giving it one more week to see if the "perfect" solution will emerge.

Share this post


Link to post
Share on other sites

That's seems like a good idea.

Plus, I first heard about AMD new platform codenamed Bulldozer which will use new socket AM3+ and chipsets 9XX yesterday (Yes I'm a Intel guy). So if I want an evolutionary system, I guess now is definitely not the time to buy...

Crap. There is always something better ahead :lol:

Edited by akhtis

Share this post


Link to post
Share on other sites

Well, even though I have ~50 TB of storage so far, I went a different route than most of you folks, I just run Windows 7x64, EVGA x58 board, in a Coolermaster HAF case . I have two five 3 1/2 drive 3 bay hot swap cages in the case, hooked up to the onboard intel and jmicron ports. The boot drive is a Revodrive. If I ever need a dvd, I hook up an external. I have a low end ATI/AMD 4550 hooked to my monitor, plus my receiver, going to a 46"TV, I am able to watch 1080p, while downloading 15 threads from usenet at the same time, no problem.

Of course, I just don't have redundancy, since I don't use use raid, I just swap the 65 hard drives in various sizes in or out as needed - about like an old 8-track tape player from the 1970's. I have some drives goign back to 2001 still in use. To keep up with where everything is, I use the Whereisit database program. The drives are sitting in a bookcase, numbered on the end with a silver Sharpie pen. I figure these drives will last a LONG time, since the only time they get power is when they are in use.

I did lose a drive a couple of years ago, a WD400. None of my data is irreplaceable though, and I can probably find it all on usenet again if needed, since Giganews has an ~850 day binary retention now. Even two years ago, I was able to eventually find everything I lost on that drive that went bad.

I thought about a rack, a Chenbro or Norco case, and Home Server - but even with the largest Chenbro, I could not keep all my drives in it, so there would be no point. I also don't have to worry if MS has really killed the usefulness of Home Server.

I can't really afford to add a 4U case plus a few 2 TB drives every few months either. Of course, I envy those of you who can ;)

Share this post


Link to post
Share on other sites

Well, even though I have ~50 TB of storage so far, I went a different route than most of you folks, I just run Windows 7x64, EVGA x58 board, in a Coolermaster HAF case . I have two five 3 1/2 drive 3 bay hot swap cages in the case, hooked up to the onboard intel and jmicron ports. The boot drive is a Revodrive. If I ever need a dvd, I hook up an external. I have a low end ATI/AMD 4550 hooked to my monitor, plus my receiver, going to a 46"TV, I am able to watch 1080p, while downloading 15 threads from usenet at the same time, no problem.

Of course, I just don't have redundancy, since I don't use use raid, I just swap the 65 hard drives in various sizes in or out as needed - about like an old 8-track tape player from the 1970's. I have some drives goign back to 2001 still in use. To keep up with where everything is, I use the Whereisit database program. The drives are sitting in a bookcase, numbered on the end with a silver Sharpie pen. I figure these drives will last a LONG time, since the only time they get power is when they are in use.

I did lose a drive a couple of years ago, a WD400. None of my data is irreplaceable though, and I can probably find it all on usenet again if needed, since Giganews has an ~850 day binary retention now. Even two years ago, I was able to eventually find everything I lost on that drive that went bad.

I thought about a rack, a Chenbro or Norco case, and Home Server - but even with the largest Chenbro, I could not keep all my drives in it, so there would be no point. I also don't have to worry if MS has really killed the usefulness of Home Server.

I can't really afford to add a 4U case plus a few 2 TB drives every few months either. Of course, I envy those of you who can ;)

There was a link on here to the "backblaze blog" about how they built their storage units... I'm not sure if you could afford it, but with 50TB of drives, I think you could.

-> 67TB at the time in 1 case or something like that, you could easily make it 90 with 1TB drives...

If I had the money that would be great to have... well, but I haven't, on the other hand, right now a home server is enough :) (900GB or Photo files :D)

Share this post


Link to post
Share on other sites

There was a link on here to the "backblaze blog" about how they built their storage units... I'm not sure if you could afford it, but with 50TB of drives, I think you could.

-> 67TB at the time in 1 case or something like that, you could easily make it 90 with 1TB drives...

If I had the money that would be great to have... well, but I haven't, on the other hand, right now a home server is enough :) (900GB or Photo files :D)

Thanks, I even signed up for a trial there, since I have a 5 Mbps upload. Unfortunately, one that they do not come close to maxing - backing up 2 TB would have taken something like 37 days.

I have a drawer full of raid cards - 3 arecas, a bunch of highpoints (NEVER trust date to a Highpoint), and an LSI. I just don't have data important enough to bother with raid, I guess.

Edited by a_b704

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now