Sy808

What OS and filesystem can support a 3+ TB array?

Recommended Posts

Building a VOD server that will have 12 300gig SATA drives in a RAID 5 array that should net me +3tb. I was wondering what operating systems and file systems will allow me to have 1 drive > 3tb in size.

~Sy

Share this post


Link to post
Share on other sites

Thanks for the links... I figure this topic came up before but when I tried searching for raid and 2tb it came back with a bazillion posts.

Question. if I build a dynamic disk in windows and then I expand the array by adding drives will windows automatically recognize this and automatically expand the dynamic drive or will it require a reformat?

~MK

Share this post


Link to post
Share on other sites

To have volumes larger than 2TB requires you to use at least 2 arrays per system! (Each array must be limited to 2TB).

The first volume, which will hold the OS, MUST be smaller than 2TB, as it's volume information is held in both the MBR and within the DD structures, (and plus it has be a simple volume, which is limted to 2TB in size).

The second volume (which will hold all the data) can then be any size up to 64TB.

To do this, however requires a bit of thought. The install sequence would be like:

1. Install OS to small partition (4-16GB in size, depending on server role).

2. Install all drivers + patches.

3. Convert array(s) from basic disk to dynamic disk.

4. Create a spanned volume to take up the rest of the space.

5. Format spanned volume with NTFS.

...

6. Expand array by adding disks.

7. In Disk Management, perform a dynamic expansion to fill up the rest of the space (no reformat needed).

The catch, each single dynamic disk is limited to 2TB. To get to 64TB requires using spanned volumes spread across multiple dynamic disks, as opposed to a simple volume on 1 dynamic disk. So for each 2TB of storage, you need to create a separate hardware based array. (This can be on the small controller, but each presented array just needs to be limited to 2TB, which you then basically create a JBOD of those 2TB arrays using Windows software RAID, to get to your 64TB).

Yes it is messy...

I would sincerely hope that Blackcomb (next Windows Server), doesn't have this limit.

But getting away from Windows, Solaris (and I would assume both AIX and HP-UX) would support very *large* arrays and filesystems.

Edited by Chewy509

Share this post


Link to post
Share on other sites

There is an web page at Microsoft documenting how they implement support for drives over 2TB using SCSI's SBC-2's READ (10) in Windows 2003 SP1. This is for external SCSI RAID.

A RAID adapter's driver may use other methods to bypass this limitation.

Question.  if I build a dynamic disk in windows and then I expand the array by adding drives will windows automatically recognize this and automatically expand the dynamic drive or will it require a reformat?

199445[/snapback]

Basic disks will show new capacity as free space. Dynamic disks cannot because the LDM 1MB database is at the end. RAID utilities should have a work-around.

Share this post


Link to post
Share on other sites
Building a VOD server that will have 12 300gig SATA drives in a RAID 5 array that should net me +3tb.  I was wondering what operating systems and file systems will allow me to have 1 drive > 3tb in size.

~Sy

199408[/snapback]

Will you be using a hardware RAID 5 controller, or Windows soft RAID? AFAIK you cannot expand a Windows soft RAID 5 array. And the RAID controller may limit you to 2TB arrays, something to double check.

Share this post


Link to post
Share on other sites
Building a VOD server that will have 12 300gig SATA drives in a RAID 5 array that should net me +3tb.  I was wondering what operating systems and file systems will allow me to have 1 drive > 3tb in size.

~Sy

199408[/snapback]

Will you be using a hardware RAID 5 controller, or Windows soft RAID? AFAIK you cannot expand a Windows soft RAID 5 array. And the RAID controller may limit you to 2TB arrays, something to double check.

199477[/snapback]

Using the 3ware 9500s-12. They say they support 64-bit lba... We'll soon see :blink: !!!

~Sy

Share this post


Link to post
Share on other sites
Thanks for all your help.  I was searching through M$'s website and came across this article:

http://www.microsoft.com/whdc/device/storage/LUN_SP1.mspx

Based on that I decided to go ahead and order Microsoft Server 2003.  Hopefully I recieve the software by friday (cmon newegg.. you can do it!).  I'll keep you guys posted on my progress.  Wish me luck!

~Sy

199467[/snapback]

You realise SP1-RTM hasn't been released yet? It's still in beta. :unsure:

I would also reread the article very carefully. Only disks that have a GDT can be larger than 2TB... (Disks with MBR's cannot support disks larger than 2TB due to design limitations within the MBR itself).

Share this post


Link to post
Share on other sites

I know the focus has switched to Windows, but I have two linux servers at work with 11TBs on a single volume on each - using reiser4 on one and jfs on the other. Each volume is coming off a single LUN from an IBM DS4500 SAN. Neither system had any problems recognizing the volume under Red Hat Enterprise Server 4 or Gentoo linux (under Gentoo we made sure to enable >2TB block device support). Since it's a single LUN it looks to the system like a single SCSI drive. They show up as /dev/sdc on these machines (sda and sdb are the internal boot disks). If I find my misplaced win2k3 server volume license discs I will install windows on one of our test blades and will see if windows can even see a single "drive" that is this large.

Michael

Share this post


Link to post
Share on other sites

Reiser4 eh? Gonadial... Testing or production? I know some people running it at home, with careful backups. Have you had any issues lately? I have heard problems are becoming pretty rare.

XFS is the king, as far as I know, at 9 EB (yes, exabytes, one million terabytes...). I think you need a system with 64-bit memory addressability to actually use that though.

EXT3 is 32TB with the largest cluster size.

ReiserFS is 16TB, and I don't know what the limit is on Reiser4. It might not have one...

XFS is very nice. I use it on my laptop. You shouldn't use it on a system without a UPS though. It can cache writes forever before committing them. In general it caches things as much as possible. (This is nice on my laptop. It's lengthened battery life noticeably, by ~15 minutes when using the larger battery, by reducing HD spinups. I do have 1.5GB of RAM which helps it a lot.)

Share this post


Link to post
Share on other sites

Reiser4 is for Testing only at this point. It has become surprisingly mature over the past few weeks. We use JFS for most of our prod systems. We run linux on Power4 based blades, so we have lots of 64 bit goodness. On my intel servers I run XFS except for my boot partitions - there i run ext2 out of old habits

Share this post


Link to post
Share on other sites
Ho do u plan to back up this monster array.  Cuz to backup that much data gets hard?

SCSA

P.S. it aint cheap either

199540[/snapback]

I dont :D . I only did it to test out if it was possible or not.

But our company does have a 6TB backup server used for staging our off site backups. But that array is pretty redundant, as it consists of 3 x 2TB raid5s striped together.

Share this post


Link to post
Share on other sites
[You realise SP1-RTM hasn't been released yet? It's still in beta.  :unsure:

I would also reread the article very carefully. Only disks that have a GDT can be larger than 2TB... (Disks with MBR's cannot support disks larger than 2TB due to design limitations within the MBR itself).

199482[/snapback]

Yeah I know SP1 is in Beta... I'll run with that for a while until they release the final SP1, which I read should be soon. Also the MBR will be going on 2 37gig raptor drives set up in RAID 1 on the built in controller on the motherboard.

Ho do u plan to back up this monster array.  Cuz to backup that much data gets hard?

SCSA

P.S. it aint cheap either

199540[/snapback]

Well this array is actually a backup of my DVD collection so If I lose the array i won't really lose any data... just a lot of time...

~Sy

Edited by Sy808

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now