Jump to content


Photo

Best solution for large backup? (> 10 TB)


  • You cannot start a new topic
  • Please log in to reply
9 replies to this topic

#1 alpha754293

alpha754293

    Member

  • Member
  • 2,015 posts

Posted 29 July 2013 - 03:35 PM

What's the most cost effective, cost efficient way of backing up ~ 10 TB of data?

The probability that the data is going to change is very small.

Would it be better for me to just build another live hard disk array and then power it up only when I want to run rsync or would it be better for me to use some kind of optical medial like BD-DL or BD-XL or would it be better for me to go with something like a LTO-3/LTO-4 solution?

Backup speed isn't too critical/too important (as I can likely run the backup job overnight).

I'm looking for something that has a relatively low cost of entry (initial capital expenditure) and also a reasonably low cost of maintaining the backup solution (as the volume of data increases over time).

Advice/suggestions/comments is GREATLY appreciated.

Thanks.
All 'round übergeek.

#2 jtsn

jtsn

    Member

  • Member
  • 94 posts

Posted 29 July 2013 - 08:46 PM

Optical discs and tape are plain dead, they just don't know it yet. It is most likely, that you will not be able to buy a new optical drive for data storage or a tape drive by 2020. So don't invest money into these dead horses.

If you already have archive data on optical discs, then start migrating it NOW.

Use hard disks (Disk2Disk) until SSD for bulk storage become available and affordable. Then migrate to all solid storage.

#3 [ETA]MrSpadge

[ETA]MrSpadge

    Member

  • Member
  • 721 posts

Posted 01 August 2013 - 12:49 PM

Use hard disks (Disk2Disk) until SSD for bulk storage become available and affordable. Then migrate to all solid storage.

Then you either have not much to backup or a looong time to wait. Unless the HDD companies go crazy, i.e. charge flood level prices so that blu ray becomes cheaper in terms of GB/€ again, flash can't beat HDDs in terms of value. And flash cells hardly hold their data for a year (3 months guaranteed on consumer grade stuff), so it would have to be some other technology (e.g. memristors).. which is not even on the market yet. And HDDs have some ace left: they can scale rather easily, 5.25" drives would be sufficient for backups and massively improve manufactoring cost per TB.

I agree, though, that optical disks are totally out of the question for large scale backups. They can still be fine for smaller single-time backups, though, like movies.

MrS

#4 alpha754293

alpha754293

    Member

  • Member
  • 2,015 posts

Posted 01 August 2013 - 12:52 PM


Then you either have not much to backup or a looong time to wait. Unless the HDD companies go crazy, i.e. charge flood level prices so that blu ray becomes cheaper in terms of GB/€ again, flash can't beat HDDs in terms of value. And flash cells hardly hold their data for a year (3 months guaranteed on consumer grade stuff), so it would have to be some other technology (e.g. memristors).. which is not even on the market yet. And HDDs have some ace left: they can scale rather easily, 5.25" drives would be sufficient for backups and massively improve manufactoring cost per TB.

I agree, though, that optical disks are totally out of the question for large scale backups. They can still be fine for smaller single-time backups, though, like movies.

MrS


So...mechanical hard drives is still the way for me to go then?
All 'round übergeek.

#5 [ETA]MrSpadge

[ETA]MrSpadge

    Member

  • Member
  • 721 posts

Posted 02 August 2013 - 03:23 PM

Optical disks: just imagine shuffling the amount of disks around needed to backup 10 TB.. that should be enough to rule it out, without considering pricing at all.

Tape: I'm no expert here, but prices per TB should be lower. However, an automatic system which changes tapes itself is usually expensive to begin with. You could probably save money by going with a manual one and maybe by buying something off Ebay. But then the backup would require several manual tape switches, meaning it will take a long time and/or not be done that often.

HDDs: a few 3/4 TB Green HDDs don't cost all that much and are comparably simple to handle. Depending on how critical your data is and how tight your budget is you might not even need redundancy here - it's "just" a backup of the original data anyway.

MrS

#6 kane_southgate

kane_southgate

    Member

  • Member
  • 3 posts

Posted 03 August 2013 - 04:26 AM

So...mechanical hard drives is still the way for me to go then?




Alpha754293,

Backup solutions are often debatable as to what medium is best for you so I want to throw a couple key points in the mix to help identify your needs:

Your subject line indicates > 10TB. Does that infer you have 10TB now? Do you have any projections as to how much your data might grow by over the next 6-12-18 months?

TCO (total cost of ownership) is definitely important when considering all variables involved with your data's longevity. I would agree that an optical disk based solution is somewhat out of the question based on the breakdown of media available (200 discs min is not efficient). A tape based solution would work great and the media itself is cost friendly (eg. LTO4 ~$60 USD per tape which gets you 800GB native or 1600GB compressed) but the cost of infrastructure is much higher (both capital and maintenance). Imagine you employed an LTO4 or LTO5 solution, tape drives alone are around $1500 per unit and that is aside from any system you plan to link it to. So whist tape can be efficient (I work with tape full time) it really seeks to capture the audience of an SME up to corporate level environments.

Hard disks are likely the best option for your plans of backup but not just a bunch of disks slapped together to qualify a 10TB + volume. Consider these few options before proceeding:

a) how do I want to constitute my 10 - >10TB solution? Do I use 2TB, 3TB or 4TB disks (Think scalability and expandability).
--> Do *NOT* use any old drives in your solution. As someone who has been affiliated with storage for 3+ years now I highly discourage getting the absolute cheapest drives available. Why? Because I see them failed all the time. Hitachi Desktar drives are a great place to start.

B) How much redundancy is enough to safeguard your data?
--> This is aimed mostly at whether you invest in HDD's, throw them in your PC as an integrated backup and once day an electrical fault occurs. *Single point of failure*
--> I would recommend looking at a NAS enclosure that is self managed (has its own O/S) so that you have the ability to keep 24x7 availability to your data even when your PC is offline. this also means your data could be kept offsite (added security in natural disasters should they ever happen). Look at the level of risk management needed and weigh up as desired.

I would lean heavily towards a NAS unit (like Synology etc). Keep it simple yet robust, buy the enclosure and add disks later, be sure to have your backup volume configured in a RAID (min RAID 5 for the distributed parity i.e. redundancy, you need min 3 disks to perform that function).

I could go into an entire topic about RAID functions but RAID 5 in your prospective NAS would be ideal, keep in mind the capacity after formatting:

Example, a RAID 5 stripe would leave a participating 3TB drive with 2.7TB (2TB = 1.8TB formatted, 4TB = 3.6TB formatted, like 9 times tables) available for use so 4 drives x 2.7TB availability = 10.8TB total. if you use a mirror (RAID 1 or any participating RAID with this function) you will halve your capacity. use these drive calculations to identify how many disks you need and the capacity. a 4 x 4TB setup would leave you with 14.4TB so that may just house your needs for now and into the future at the best possible price. I have seen this kind of solution with a NAS box between a range of $900-1200 all up. Good luck!

#7 alpha754293

alpha754293

    Member

  • Member
  • 2,015 posts

Posted 03 August 2013 - 08:36 AM



Alpha754293,

Backup solutions are often debatable as to what medium is best for you so I want to throw a couple key points in the mix to help identify your needs:

Your subject line indicates > 10TB. Does that infer you have 10TB now? Do you have any projections as to how much your data might grow by over the next 6-12-18 months?

TCO (total cost of ownership) is definitely important when considering all variables involved with your data's longevity. I would agree that an optical disk based solution is somewhat out of the question based on the breakdown of media available (200 discs min is not efficient). A tape based solution would work great and the media itself is cost friendly (eg. LTO4 ~$60 USD per tape which gets you 800GB native or 1600GB compressed) but the cost of infrastructure is much higher (both capital and maintenance). Imagine you employed an LTO4 or LTO5 solution, tape drives alone are around $1500 per unit and that is aside from any system you plan to link it to. So whist tape can be efficient (I work with tape full time) it really seeks to capture the audience of an SME up to corporate level environments.

Hard disks are likely the best option for your plans of backup but not just a bunch of disks slapped together to qualify a 10TB + volume. Consider these few options before proceeding:

a) how do I want to constitute my 10 - >10TB solution? Do I use 2TB, 3TB or 4TB disks (Think scalability and expandability).
--> Do *NOT* use any old drives in your solution. As someone who has been affiliated with storage for 3+ years now I highly discourage getting the absolute cheapest drives available. Why? Because I see them failed all the time. Hitachi Desktar drives are a great place to start.

B) How much redundancy is enough to safeguard your data?
--> This is aimed mostly at whether you invest in HDD's, throw them in your PC as an integrated backup and once day an electrical fault occurs. *Single point of failure*
--> I would recommend looking at a NAS enclosure that is self managed (has its own O/S) so that you have the ability to keep 24x7 availability to your data even when your PC is offline. this also means your data could be kept offsite (added security in natural disasters should they ever happen). Look at the level of risk management needed and weigh up as desired.

I would lean heavily towards a NAS unit (like Synology etc). Keep it simple yet robust, buy the enclosure and add disks later, be sure to have your backup volume configured in a RAID (min RAID 5 for the distributed parity i.e. redundancy, you need min 3 disks to perform that function).

I could go into an entire topic about RAID functions but RAID 5 in your prospective NAS would be ideal, keep in mind the capacity after formatting:

Example, a RAID 5 stripe would leave a participating 3TB drive with 2.7TB (2TB = 1.8TB formatted, 4TB = 3.6TB formatted, like 9 times tables) available for use so 4 drives x 2.7TB availability = 10.8TB total. if you use a mirror (RAID 1 or any participating RAID with this function) you will halve your capacity. use these drive calculations to identify how many disks you need and the capacity. a 4 x 4TB setup would leave you with 14.4TB so that may just house your needs for now and into the future at the best possible price. I have seen this kind of solution with a NAS box between a range of $900-1200 all up. Good luck!


Thank you all for your input.

I actually purposely left out a few of the details because I didn't want to contaminate or influence the opinions of the people who have been providing them.

So, I'm going to fill in the back story now.

My current RAID array is 10 * 3 TB drives on RAID5. They're Hitachi 3 TB drives that I bought a couple of months before the floods in Thailand, not knowing that there was going to be a flood, but because they worked out to be like $114 each per drive, so it was a really good deal. Yayyy eBay.

My original intent was to have the first 10 drives (out of a box of 20) to be deployed in my current server and then the second group of 10 to be deployed as the backup; BUT now you might be asking "so why did you ask?" well...I want to make sure that that was still the better way to go and that LTO and BD-DL or BD-XL hasn't replaced it in terms of cost and viability and reliability.

I haven't built the second backup server yet because I'm still debating as to whether I want to keep it as NTFS or whether I want to migrate back over to ZFS as the backup server. (Aside from one or two other experimental file systems, those are the only ones that will support a single volume that size.)

I actually already have the drives, I'm just waiting to get the enclosure and the rest of the server hardware (motherboard, CPU, PSU) etc. before I can launch and deploy the backup server, but I just wanted to make sure that the initial plan still held its own when tested by other people.

And given the feedback, it seems like that hard drives is the way to go, although trying to expand the capacity of either systems would be a little harder, unless I just get bigger drives, but the same number of them.

Thank you all for your feedback/input.
All 'round übergeek.

#8 kane_southgate

kane_southgate

    Member

  • Member
  • 3 posts

Posted 04 August 2013 - 05:02 AM

Thank you all for your input.

I actually purposely left out a few of the details because I didn't want to contaminate or influence the opinions of the people who have been providing them.

So, I'm going to fill in the back story now.

My current RAID array is 10 * 3 TB drives on RAID5. They're Hitachi 3 TB drives that I bought a couple of months before the floods in Thailand, not knowing that there was going to be a flood, but because they worked out to be like $114 each per drive, so it was a really good deal. Yayyy eBay.

My original intent was to have the first 10 drives (out of a box of 20) to be deployed in my current server and then the second group of 10 to be deployed as the backup; BUT now you might be asking "so why did you ask?" well...I want to make sure that that was still the better way to go and that LTO and BD-DL or BD-XL hasn't replaced it in terms of cost and viability and reliability.

I haven't built the second backup server yet because I'm still debating as to whether I want to keep it as NTFS or whether I want to migrate back over to ZFS as the backup server. (Aside from one or two other experimental file systems, those are the only ones that will support a single volume that size.)

I actually already have the drives, I'm just waiting to get the enclosure and the rest of the server hardware (motherboard, CPU, PSU) etc. before I can launch and deploy the backup server, but I just wanted to make sure that the initial plan still held its own when tested by other people.

And given the feedback, it seems like that hard drives is the way to go, although trying to expand the capacity of either systems would be a little harder, unless I just get bigger drives, but the same number of them.

Thank you all for your feedback/input.


All sounds good. I would keep NTFS there for the purpose of an accurate replication (any efforts made to restore can be guaranteed to be identical without any FS issues). Good luck!

#9 alpha754293

alpha754293

    Member

  • Member
  • 2,015 posts

Posted 04 August 2013 - 09:54 AM

All sounds good. I would keep NTFS there for the purpose of an accurate replication (any efforts made to restore can be guaranteed to be identical without any FS issues). Good luck!


Thanks.

I thought about using ZFS because then I can enable compression on it and also turn on deduplication as well, so it would save a bit of physical storage space.

That was the logic/reasoning behind it.

But I do agree with the NTFS though. Such a shame that it doesn't have de-dup natively (although I was reading that I think it was in Windows Server 2012 that it has it or something very similiar to it, but I've not read ALL of the details in terms of how it works/how it does the dedup), so...

Thanks for your feedback.
All 'round übergeek.

#10 [ETA]MrSpadge

[ETA]MrSpadge

    Member

  • Member
  • 721 posts

Posted 06 August 2013 - 03:19 PM

NTFS also has compression, although it only works on 64kb chunks at a time and has to be quick, i.e. can't compress as much as Winrar or alternatives.

MrS



2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users