abild

My fileserver needs an upgrade

Recommended Posts

Hi

My fileserver is in serious need for an upgrade.

Currently i have 4 160gb Hitachi 7K250 on a Promise SX4000 controller with 128mb cache. The disks are set up in RAID 5. The disks and the controller have now endured more than 2 1/2 year of 24/7 operation. So I'm getting increasingly concerned about how much longer they will live. I would like to replace all of the disks before the first one dies so i wont have to buy an outdated replacement disk. I'm also starting to run out of disk space.

Also I'm currently stuck with windows on my file server because I've formatted the RAID array as NTFS. All my other computers are now running Linux. I was also forced to set up a second server for web, FTP etc with Linux. I want to get rid of that one as well now so i will only have one server with Linux.

Reliability is my top requirement, price is important to. Performance isn't that important since most data will be transfered over a 100mbit network.

Currently I'm thinking of buying 4 320gb Hitachi T7K500 and a Promise SATA300 TX4 controller and set them up in RAID 5 with Linux software RAID. I don't know how reliable Hitachi drives are but I currently have 9 Hitachi drives in use (all form the 7K250 series) with the oldest being about 3 years old. None of them have crashed on me yet. I'm considering software raid over Hardware raid because it's cheaper and because it can be expanded beyond the number of disks supported by one controller. It's also not bound to a specific RAID controller

In my search for a SATA controller I also found a Stlab 4-port SATA150 controller. It doesn't support SATA300, NCQ, TCQ or any other fancy stuff but it's about 1/5 of the price of the Promise controller. After all a HD controller is not a very complex peace of hardware. Stlabs controllers consists of a Silicon Image chip put on top of a pretty much standardized circuit board. What are you thoughts about this. Any reason why i should go with the more expensive Promise controller?

The alternative solution is, of course, that i buy 4 P-ATA hd's and keep using my current Promise controller but I'm feeling quite bad about investing in P-ATA drives today. Would that in your opinion be preferable?

I wanted to ask you guys what you think about the above. Would you have done anything different?

I've also got some miscellaneous questions

1) The two servers i mentioned earlier have the following specs: Compaq deskpro <something> 1,7ghz p4 (the old Willamette), 256 MB good old SD RAM, 17gb SCSI disks and the other is a Dell Dimension 8100 with 1,5ghz Pentium 4 (also Willamette), 256mb RD-RAM. The biggest difference is RD vs SD ram (I can always swap the cpu's). Does RD provide any significant performance increase compared to SD?

I can't decide on which one to choose. Which one would you have chosen? The chosen one will undergo a ram upgrade.

2) Does anyone know what sort of performance i can expect from software raid 5 and how much cpu power does the parity calculations require?

3) A Linux specific question: Which file system is best suited for servers? ext3, ReiserFS or something else? I'm using ReiserFS on all my desktops and I'm quite happy with it, but i don't know whats best for servers.

Thanks in advance

Share this post


Link to post
Share on other sites

If you have to run with Linux, then I suggest that you have the OS on a separate drive if you don't already do so.

Believe me, it'll save you a lot of hassle.

Secondly, Hitachi doesn't make a 320 GB drive. They have 160, 250, 400, and 500 GB. In terms of their reliability I think that they're top notch.

They're not as fast compared to others, but they're no slouch either. The oldest I have is about 5.25 years old and the newest I have is about a month old. (And I have a total of 50+ drives, all except one by Hitachi, and a few that are IBM branded Seagates.)

I run most of them 24/7/365, except for when my power goes out which can happen quite frequently during the summer storms. Of them, I've only had two 18 GB SCSI drives die on me and I think that it was because they were sitting right in front of the heater vent and thus died due to thermal.

Third - if performance isn't necessarily a concern to you, then perhaps you might want to consider building a system that I can run Solaris and ZFS. Granted, it's not perfect (yet), but with the testing that I have done with them, a few 500 GB Hitachi drives, using a native SATA2 controller can get immense performance out of them.

(For example, 3 drives stripped, max write speeds of ~112 MB/s. 3 drive raidz (which is akin to enhanced raid5) max write speeds: 78 MB/s.)

On a Duron 1.8 GHz, I was at about 14% CPU utilization for stripped (across 5 drives, 4 SCSI, 1 PATA). On my new dual Opteron 2210 (1.8 GHz), average utilization was < 1%.

It seems to me that the latest "standard" is to use Ext3 for everything. I don't know if that is really a true statement or not, but that what it looks like (at least for the Linux front). UNIX (such as Solaris) is still a mix between UFS, and NFS; although I do like the new ZFS.

Since you won't see much of a performance with a 100Mbps network, therefore; you're pretty much open to all the hardware and software choices available out there. However, should you plan for an EVENTUAL upgrade to GbE, that's when you'd really notice the difference.

Share this post


Link to post
Share on other sites

Thanks a lot for your reply.

I don't know if i want to run the OS from a non-raid drive. The risk of a single drive failure bringing down the entire system is simply to big. If running OS from software RAID turn out to be too troublesome i do have a cheap controller which supports RAID 1 i can set up with a couple of drives.

Hitachi do actually make a 320gb hd. It's availible in Denmark and we are always among the last to get new electronics. http://www.edbpriser.dk/Products/Listprices.asp?ID=233960

Raidz sounds really interesting. I've been wanting to try out Solaris for quite some time. Now, i might actually do it. How different is Solaris from Linux and how long do you think it'll take a user who is used to Linux to adapt? Do Solaris have binary compatibility wit Linux and will Linux source code compile on Solaris? And how does RAID in Solaris compare to Linux raid feature wise? will i eg be able to expand an array with an extra hard drive without having to reformat?

Are the CPU utilization examples you mention with RAID 0 or RAID 5? I ask because i have found this: http://www.beowulf.org/archive/2002-March/006575.html which gives an entirely different picture of CPU utilization.

Once I'm able to rebuild a stockpile of money my next big upgrade will be from 100mbit to gbit LAN. My biggest concern is that the PCI bus isn't able to support both the RAID array and the gigabit ethernet card. I simply don't have the economy to get a motherboard with PCI64.

Share this post


Link to post
Share on other sites

My advice is to not run a RAID5 array at all. Disk capacity increases at a rate that makes it much more economical to add capacity as you need it. Always buy the capacity with the best GB/$ ratio, stick 'em in a Linux box. Use LVM, keep backups, and be done with it. You save hundreds of dollars this way.

RAID5 is the current silly storage fad du-jour IMHO.

Share this post


Link to post
Share on other sites
My advice is to not run a RAID5 array at all. Disk capacity increases at a rate that makes it much more economical to add capacity as you need it. Always buy the capacity with the best GB/$ ratio, stick 'em in a Linux box. Use LVM, keep backups, and be done with it. You save hundreds of dollars this way.

RAID5 is the current silly storage fad du-jour IMHO.

You are right. In my case RAID is not most cost efficient solution. But for me RAID is all about data safety and not price. By running raid i know that a single drive failure won't make me loose all my data. I know it's not an ultimate solution and that there are lots of other things that can knock my data out but it adds an extra layer of security compared to simply running separate disks.

Edited by abild

Share this post


Link to post
Share on other sites

Very rough figures, can go up or down depending on choices:

Budget MB with 4-6 SATA on-board:~ $100

Budget CPU:~ $75

Budget 512 MB RAM:~ $75

Total: ~$250

Advantages:

On-board SATA should avoid PCI and out-perform it

On-board GbE should avoid PCI and out-perform it, and far out-perform 10/100

Modern build should be easier to upgrade down the road

Newer hardware could be more stable and better supported.

Disadvantages:

Price

Need to check compatibility with other parts & software, etc.

Newer hardware could be flaky and have unstable drives and support

Budget pricing on the MB and RAM could be detrimental.

I'd probably do this instead of going the legacy route, perpetuating an obsolete platform, with obsolete RAM, a PCI storage controller + eventual PCI GbE NIC, etc., which gave worse performance.

BTW, I and others have used XFS in such roles for its performance and availability / support.

Your results, needs, preferences, prices and budget may be different of course.

Share this post


Link to post
Share on other sites
Thanks a lot for your reply.

I don't know if i want to run the OS from a non-raid drive. The risk of a single drive failure bringing down the entire system is simply to big. If running OS from software RAID turn out to be too troublesome i do have a cheap controller which supports RAID 1 i can set up with a couple of drives.

Hitachi do actually make a 320gb hd. It's availible in Denmark and we are always among the last to get new electronics. http://www.edbpriser.dk/Products/Listprices.asp?ID=233960

Raidz sounds really interesting. I've been wanting to try out Solaris for quite some time. Now, i might actually do it. How different is Solaris from Linux and how long do you think it'll take a user who is used to Linux to adapt? Do Solaris have binary compatibility wit Linux and will Linux source code compile on Solaris? And how does RAID in Solaris compare to Linux raid feature wise? will i eg be able to expand an array with an extra hard drive without having to reformat?

Are the CPU utilization examples you mention with RAID 0 or RAID 5? I ask because i have found this: http://www.beowulf.org/archive/2002-March/006575.html which gives an entirely different picture of CPU utilization.

Once I'm able to rebuild a stockpile of money my next big upgrade will be from 100mbit to gbit LAN. My biggest concern is that the PCI bus isn't able to support both the RAID array and the gigabit ethernet card. I simply don't have the economy to get a motherboard with PCI64.

I actually had one of my servers running Redhat9 with a Promise SX6000 6-channel PATA RAID HBA and I had 4x 250GB drives and I installed the OS on the RAID array with a similar mindset.

Over the summer, we had some power glitches (a common occurance where I live) during the summer storms (can't use UPS because I've actually killed a lot of the consumer ones running 8 systems off of it) and commercial ones start at around $3Gs.

As a result, the GRUB bootloader got corrupted and since it can't load the driver to initialize the array, so far; I haven't been able to do anything with the system in terms of data recovery. As such, once I have a second server that I can offload the entire contents of the array to; I am thinking of using a single 80 GB drive, install the OS on there, and also hope that it would be able to pick up the array without doing anything to it, so that I can mount it as read-only long enough to evacuate the data. That's a stint I've never attempted before for I am a little weary about doing it.

In talking to some of my friends that does Linux administration for their day-job; after they stopped laughing and wanting to hit me for pulling a bonehead move like that, they suggested that I can try and do a linux rescue with the install disk (also weary of that too).

In suggesting that you should NEVER run your boot drive on a RAID array - it is aimed to prevent you from having to go through the same or a similar experience.

Most Linuxes, I'm told nowadays would be able to detect that you have an existing SW RAID array if your OS drive was independent. That way, it SHOULD (I haven't tested this yet) just pick up the array and move on with it.

- * - * - * -

Hitachi 320 GB drives aren't available in North America. What a shock eh?

- * - * - * -

In terms of time that it would take you to adapt from Linux to Solaris, I don't think that it would be that hard. Some of the commands would have a different syntax, but if you go to the Solaris 10 Command Reference Manual, it goes through it in excrutiating detail and it is VERY well written, and easy to read.

There are online tutorials that explain some of the basic commands and operations for the ZFS administration and in my opinion, I think that it is one of the easiest.

If you are going with a stripped array (i.e. raid0), a command might look like the following:

# zpool create myfiles c0d0 c0d1 c1d0 c1d1

for a four drive, PATA array.

If you want to create a RAID10 array, it would be something like

# zpool create myfiles2 c0d0 c0d1 mirror c1d0 c1d1

That was one of the big reasons why I went with it because it was so simple to administer it that even I can do it (and I'm not a sysadmin by a long shot).

The biggest thing to it is to make sure that your hardware that you're planning on running/using is listed on the Solaris HCL. The lack of a LOT of drivers is still one of the big problems that Solaris is facing for widespread adoption. However, it is getting better, but the market is definitely outpacing it's driver development group,

Having said that, if you were to build a system designed to be able to run Solaris - this is an example of what I am currently using right now:

MSI KT4AV-L (SocketA)

AMD Duron 1.8 GHz

2x 256 MB PC2100

Powercolor ATi Radeon 9250 64 MB

Netgear GA311 GbE NIC (based on the realtek 8169 chipset)

(hard drives)

My base system at the time of build, including case, power supply, optical drive, etc.. came up to around $500 US.

The only downside to it is that ZFS has a max CPU utilization of 14% on that system with a 4 SCSI + 1 PATA drive RAIDZ array.

SOME native Linux stuff is supposed to be able to run entirely within a Solaris Container that segregates the core OS from the Linux. I don't know enough about it to be able to say whether it works one way or another because I haven't been able to find very many good "how-to" guides to set that up and get it working/running.

It's supposed to fully support VMWare ESX server and that's probably the route that I would be going with if I were to try and do any virtualization stuff.

Note: Solaris REALLY works well with AMD Opterons, both the old ones using DDR and the new ones using DDR2 memory. Yes, they're more expensive, but it's performance is like an exponential increase compared to my "lowly" Duron 1.8 GHz.

It is my understanding that when you want to add capacity to the array, that depends on the type of array you have set up.

I think that for a stripped array (raid0), you should be able to do a straight add. (I don't know, haven't tried it).

For a raidz; the configuration has to be either identical or better than the array that you started out with. So if you have a 3-drive raidz (capacity of n-1); that your minimum add-on should also be a 3-drive raidz.

http://en.wikipedia.org/wiki/ZFS

Read up on ZFS to get a feel for it.

http://www.sun.com/software/solaris/zfs_learning_center.jsp

- * - * - * -

The CPU utilization figures were for a raidz array (so it is technically neither a raid0 or a raid5).

- * - * - * -

Realistically, you're either going run out of users pulling data from the storage server, or you're going to run out of network bandwidth (even if you're on a GbE) or you're hardware limited (i.e. hard drives, memory, etc.)

If performance is really that much of a concern, then going GbE is the first thing you should also consider (at minimum) along with the server upgrade.

If you're just expanding to a GbE - chances are - you're still quite a ways before you're going to hit your limit. And when you do, then you can consider alternative options such as SCSI, FC, SAS, and fibre switches.

If you're the only one using the network bandwidth primary and manipulating and moving the data around; you're rarely ever going to see the limit.

Share this post


Link to post
Share on other sites
My advice is to not run a RAID5 array at all. Disk capacity increases at a rate that makes it much more economical to add capacity as you need it. Always buy the capacity with the best GB/$ ratio, stick 'em in a Linux box. Use LVM, keep backups, and be done with it. You save hundreds of dollars this way.

RAID5 is the current silly storage fad du-jour IMHO.

Depends on what you're doing. I like the ability to simply add a disk to an array, expand the array and have the extra space. That's also why I would NEVER use software RAID except for mirroring. I'd rather trust a decent hardware RAID solution like an IBM ServeRAID controller or something similar.

Share this post


Link to post
Share on other sites
Over the summer, we had some power glitches (a common occurance where I live) during the summer storms (can't use UPS because I've actually killed a lot of the consumer ones running 8 systems off of it) and commercial ones start at around $3Gs.

Going off on a tangent for a moment...

Considering the grief you're going through with the corrupted file system, I sincerely believe you need to re-think your priorities regarding a UPS. It will save you tons of trouble, and it does not have to cost you a fortune.

My current UPS - an APC SmartUPS 2200XL - I bought through eBay in the winter of 2003 after searching for just a couple of weeks. I intentionally limited my search to local auctions so I would not have to worry about paying for freight on a 110lb box. I paid US$107.50 for it including tax and drove just twenty two miles round trip to pick it up. I was the ONLY bidder on this auction.

While I fully expected the UPS batteries to be knackered on delivery, they lasted until literally just three weeks ago (nearly three years) when the semi-weekly self-test failed with a battery replacement indication. It cost me just over $200 to refresh them with four Panasonic 20AH SLA batteries (better than the originals and better than APC's current OEM replacements), including shipping, and the new set ought to be good for at least another four years. I saved nearly $100 by not buying APC's replacements. I'm sure this UPS should last at least another six years, so my "power insurance" works out to be about $60/year including the extra bit of electricity to keep the UPS juiced up. Damn good deal in my opinion.

This unit supports eight systems, just barely. The use of an eight-way KVM switch keeps me within the UPS's design specs. If your power requirements are greater you could just buy two consumer-grade 1500-2000KVA UPSs rather than blowing big bucks for commercial grade equipment and still save substantially. Keep laser printers offa' these things (they often draw spikes of ten to twenty amps while keeping the drum warm) and life will be good.

Good luck,

-Brad

Share this post


Link to post
Share on other sites

Over the summer, we had some power glitches (a common occurance where I live) during the summer storms (can't use UPS because I've actually killed a lot of the consumer ones running 8 systems off of it) and commercial ones start at around $3Gs.

As a result, the GRUB bootloader got corrupted and since it can't load the driver to initialize the array, so far; I haven't been able to do anything with the system in terms of data recovery.

You shouldn't mount the boot partition on a server normally. It's a very simple way to prevent this sort of thing from happening.

Most Linuxes, I'm told nowadays would be able to detect that you have an existing SW RAID array if your OS drive was independent. That way, it SHOULD (I haven't tested this yet) just pick up the array and move on with it.

It depends on whether or not you setup the RAID array to include superblocks. If you do, any kernel that supports RAID or can load the modules will recognize it and can mount it just fine. Any RAID array setup using mdadm, by default, writes superblocks. If you use the older raidtab then the you won't have superblocks. Anyone using Linux software RAID should use mdadm these days. It's a far better tool.

My advice is to not run a RAID5 array at all. Disk capacity increases at a rate that makes it much more economical to add capacity as you need it. Always buy the capacity with the best GB/$ ratio, stick 'em in a Linux box. Use LVM, keep backups, and be done with it. You save hundreds of dollars this way.

RAID5 is the current silly storage fad du-jour IMHO.

Depends on what you're doing. I like the ability to simply add a disk to an array, expand the array and have the extra space. That's also why I would NEVER use software RAID except for mirroring. I'd rather trust a decent hardware RAID solution like an IBM ServeRAID controller or something similar.

Linux software RAID 5 can be expanded now-a-days. I haven't had to do it myself, but it's possible.

B)-->

QUOTE(Brad.B @ Nov 4 2006, 09:59 PM) 235797[/snapback]

Considering the grief you're going through with the corrupted file system, I sincerely believe you need to re-think your priorities regarding a UPS. It will save you tons of trouble, and it does not have to cost you a fortune.

While a UPS is a good idea in a lot of cases, if uptime during a power outage isn't crucial not mounting the boot filesystem, and using journalling filesystems are both entirely sufficient and much more economical options. If data absolutely cannot be lost ext3 should be used with the data=journal option. This is massive overkill generally --in most situations consistency is sufficient and any journalling filesystem will be adequate.

Share this post


Link to post
Share on other sites
While a UPS is a good idea in a lot of cases, if uptime during a power outage isn't crucial not mounting the boot filesystem, and using journalling filesystems are both entirely sufficient and much more economical options. If data absolutely cannot be lost ext3 should be used with the data=journal option. This is massive overkill generally --in most situations consistency is sufficient and any journalling filesystem will be adequate.

A generator is for maintaining uptime during substantial outages. A UPS is for being able to seamlessly ride out brief outages, dips, brownouts, etc. and to serve is a stage of isolation to protect equipment from the usual problems like spikes and lightning strike effects.

When judging the relative ecomony of a journalling FS compared to the investment of a UPS, I would consider the performance impact of the journalling overhead as well as the need to disable any OS and drive write caching (lest ye journal be corrupt as well), or the cost of the BBWC RAID controllers that would be required to overcome those performance and reliability obstacles.

Share this post


Link to post
Share on other sites

B)-->

QUOTE(Brad.B @ Nov 5 2006, 12:58 PM) 235825[/snapback]

I would consider ... the need to disable any OS and drive write caching (lest ye journal be corrupt as well)

Not necessary. Journal operations are atomic. That's the whole point of the journal, to guarantee consistency.

Share this post


Link to post
Share on other sites
Not necessary. Journal operations are atomic. That's the whole point of the journal, to guarantee consistency.

Hmmm... educate me... is each journal write accompanied by a flush operation?

-Brad

Share this post


Link to post
Share on other sites

B)-->

QUOTE(Brad.B @ Nov 5 2006, 10:58 AM) 235825[/snapback]
While a UPS is a good idea in a lot of cases, if uptime during a power outage isn't crucial not mounting the boot filesystem, and using journalling filesystems are both entirely sufficient and much more economical options. If data absolutely cannot be lost ext3 should be used with the data=journal option. This is massive overkill generally --in most situations consistency is sufficient and any journalling filesystem will be adequate.

A generator is for maintaining uptime during substantial outages. A UPS is for being able to seamlessly ride out brief outages, dips, brownouts, etc. and to serve is a stage of isolation to protect equipment from the usual problems like spikes and lightning strike effects.

When judging the relative ecomony of a journalling FS compared to the investment of a UPS, I would consider the performance impact of the journalling overhead as well as the need to disable any OS and drive write caching (lest ye journal be corrupt as well), or the cost of the BBWC RAID controllers that would be required to overcome those performance and reliability obstacles.

I can't go with consumer-grade UPSes because with 8 systems, about 20 processors, and 52 hard drives, I am able to trip my 15A breaker on a hot summer day with the A/C going and such. I am presuming that to mean that it is an overcurrent project, which also means that I am pulling at least 2.7 kW.

If I were to go with consumer-grade UPS, I would have to get one per system almost, which takes up a fair amount of space.

That's one of the big reasons why I do not have it. (Other than cost).

Share this post


Link to post
Share on other sites
I can't go with consumer-grade UPSes because with 8 systems, about 20 processors, and 52 hard drives, I am able to trip my 15A breaker on a hot summer day with the A/C going and such. I am presuming that to mean that it is an overcurrent project, which also means that I am pulling at least 2.7 kW.

If I were to go with consumer-grade UPS, I would have to get one per system almost, which takes up a fair amount of space.

Even with pretty heavily-loaded machines, you should be able to get about 4 of them on a 2200VA UPS. Fully loaded, that will only buy you maybe 10 minutes of runtime but at least where I am that covers about 90-95% of outages, and is expandable on APC's XL models. Granted, 2-3 of those suckers is neither small nor light, but it's hardly unreasonable given the investment you've made in space, weight, and money for the machines themselves.

Not necessary. Journal operations are atomic. That's the whole point of the journal, to guarantee consistency.

Journal ops may be flushed, but that doesn't actually guarantee atomicity. Many drives lie about having flushed data to the platters, returning success when the data has been received rather than when the write cycle is complete. It's certainly an improvement in reliability but it's not a mathematical guarantee of correctness. Turning off write caching on the drives probably solves this problem. There's still some amount of buffering being done, even with the write cache off, but it's probably a fifo so the data portion of a write could not occur before the journal update, even if the journal update has yet not been committed to the platters when the drive reports the flush to be complete.

Share this post


Link to post
Share on other sites
I can't go with consumer-grade UPSes because with 8 systems, about 20 processors, and 52 hard drives, I am able to trip my 15A breaker on a hot summer day with the A/C going and such. I am presuming that to mean that it is an overcurrent project, which also means that I am pulling at least 2.7 kW.

If I were to go with consumer-grade UPS, I would have to get one per system almost, which takes up a fair amount of space.

That's one of the big reasons why I do not have it. (Other than cost).

Haversian already replied perfectly to your post but I feel the need to add an expression of general shock and surprise...

I can't believe how many seriously brilliant people here (a) never thought of spreading the load across a couple of cheaper UPSs and (B) have justified spending many thousands of dollars on rooms full of computer equipment but can't make sense of another few hundred dollars to protect not just the equipment but the data on it.

My final word on this is to consider how much your time is worth. Then consider how long it would take you to recover or repair your data or make up for lost processing time. If your systems have clients, consider what their time is worth and the impact on your reputation if you're down for a day. Unless you're a fast order cook, I'm pretty sure the UPS is a sensible investment.

-Brad

Share this post


Link to post
Share on other sites

I'm sorry I've been a little slow replying. I haven't had much time lately.

Very rough figures, can go up or down depending on choices:

Budget MB with 4-6 SATA on-board:~ $100

Budget CPU:~ $75

Budget 512 MB RAM:~ $75

Total: ~$250

Advantages:

On-board SATA should avoid PCI and out-perform it

On-board GbE should avoid PCI and out-perform it, and far out-perform 10/100

Modern build should be easier to upgrade down the road

Newer hardware could be more stable and better supported.

Disadvantages:

Price

Need to check compatibility with other parts & software, etc.

Newer hardware could be flaky and have unstable drives and support

Budget pricing on the MB and RAM could be detrimental.

I'd probably do this instead of going the legacy route, perpetuating an obsolete platform, with obsolete RAM, a PCI storage controller + eventual PCI GbE NIC, etc., which gave worse performance.

BTW, I and others have used XFS in such roles for its performance and availability / support.

Your results, needs, preferences, prices and budget may be different of course.

You're right that upgrading the old hardware i have now would be a rather idea. I've just always assumed that doing a major upgrade would be way too expensive. After doing some actual checks on hardware prices I've realized that a major upgrade isn't as expensive as i once thought. Thanks for opening my eyes :)

Unfortunately i can't reuse any hardware from my old servers. The Dell uses some weird proprietary design on it's PSU connectors and their motherboard backplate isn't standard ATX either. The PSU in the Compaq is not by far powerful enough to be able to support a new system and it's case is not very expandable.

Finding a cheap case which provides lots of 3,5" device bays and provides proper cooling for the disks is not easy. A reasonable quality PSU which is not too expensive is not easy to find either.

I've compiled a list with 3 possible hardware suggestion together with their pros and cons (as I see them).

I was only able to find motherboards with 4 sata connectors within a reasonable price range. Does anyone know any cheap boards with 6 sata's ? ,

It might be useful if i said something about what the server is being used for. It works as a central fileserver/backup for two families (8 people in total). It also runs a ftp server and a web server. The web server is mostly being used to test the web pages i develop and it's also going to host my personal site. I also run a picture gallery for some friends. This is probably the task requiring the most performance. Resizing pictures and converting videos is really slow on my current server.

Suggestion one

Asus A8N-VM CSM 640kr ($110,4)

AMD Athlon 64 3400+ 2,2ghz 579kr ($99,87)

2x256mb ram Corsair XMS CL2 PC3200 642kr ($110,74)

Total: 1861kr ($331)

Pros

Will receive memory and CPU upgrades from my current computer when I'm replacing it.

Cons

Socket 939 and DDR (1) ram are both soon to be obsolete

Suggestion two

Asus P5L-VM 1394 623kr ($107.46)

Intel Pentium D 820 2.8ghz. 719kr ($124.02)

Kingston HyperX 2x256mb CL3 PC2-4300 596kr ($102.81)

Total: 1938kr ($334.29)

Pros

Dual Core cpu

Uses future proof hardware (socket 775 and DDR 2)

Cons

Most expensive

Have a slightly higher power consumption than the other suggestions (Higher electrical bill and harder to keep cool)

Slower single threaded processing

I don't know if I'll even need Dual Core

Suggestion three

ASUS M2N-MX 460kr ($79.35)

Athlon 64 3500+ 2,2 ghz 665kr ($114.72)

Kingston HyperX 2x256mb CL3 PC2-4300 596kr ($102.81)

Total: 1721kr ($296.86)

Pros

Other than DDR2 and the AM2 socket being future proof i don't know

Cons

More expensive

To my knowledge the AM2 athlons are architecturally identical to the s939 athlons and the high latency DDR2 ram running at low speeds will actually run slower than low latency running at equivalent speeds .

Does anyone know how the nForce Sata controllers performs compared to the Intel ones and what the performance difference is between Realtek, nVidia, Marvell, Attansic and Intel gigabit lan chips?

I would appreciate to get some feedback on above suggestions. Completely different suggestions are welcomed as well.

I actually had one of my servers running Redhat9 with a Promise SX6000 6-channel PATA RAID HBA and I had 4x 250GB drives and I installed the OS on the RAID array with a similar mindset.

Over the summer, we had some power glitches (a common occurance where I live) during the summer storms (can't use UPS because I've actually killed a lot of the consumer ones running 8 systems off of it) and commercial ones start at around $3Gs.

As a result, the GRUB bootloader got corrupted and since it can't load the driver to initialize the array, so far; I haven't been able to do anything with the system in terms of data recovery. As such, once I have a second server that I can offload the entire contents of the array to; I am thinking of using a single 80 GB drive, install the OS on there, and also hope that it would be able to pick up the array without doing anything to it, so that I can mount it as read-only long enough to evacuate the data. That's a stint I've never attempted before for I am a little weary about doing it.

In talking to some of my friends that does Linux administration for their day-job; after they stopped laughing and wanting to hit me for pulling a bonehead move like that, they suggested that I can try and do a linux rescue with the install disk (also weary of that too).

In suggesting that you should NEVER run your boot drive on a RAID array - it is aimed to prevent you from having to go through the same or a similar experience.

Most Linuxes, I'm told nowadays would be able to detect that you have an existing SW RAID array if your OS drive was independent. That way, it SHOULD (I haven't tested this yet) just pick up the array and move on with it.

Ok, You have convinced me not to boot from my RAID array. Running into such problems would be a nightmare for me. Power outages are a really rare occurrence here though (i can remember having one in the past 6 years) so it probably wouldn't be a big problem for me anyway.

I'm currently considering using the 17gb SCSI drive which currently resides in one of my servers) as the boot/os partition, at least as a temperate solution. I still plan to provide some kind of redundancy for the boot drive in the future. (Most likely in the form of some cheap hardware RAID 1 controller). Would that be a usable solution?

Hitachi 320 GB drives aren't available in North America. What a shock eh?

yea... Thats kinda odd.

In terms of time that it would take you to adapt from Linux to Solaris, I don't think that it would be that hard. Some of the commands would have a different syntax, but if you go to the Solaris 10 Command Reference Manual, it goes through it in excrutiating detail and it is VERY well written, and easy to read.

There are online tutorials that explain some of the basic commands and operations for the ZFS administration and in my opinion, I think that it is one of the easiest.

If you are going with a stripped array (i.e. raid0), a command might look like the following:

# zpool create myfiles c0d0 c0d1 c1d0 c1d1

for a four drive, PATA array.

If you want to create a RAID10 array, it would be something like

# zpool create myfiles2 c0d0 c0d1 mirror c1d0 c1d1

That was one of the big reasons why I went with it because it was so simple to administer it that even I can do it (and I'm not a sysadmin by a long shot).

[/qoue]

I've been doing a lot of reading on Solaris and ZFS and i must say it seems absolutely amazing. I love the concept of storage pools, prevention of silent data corruption

I have also been playing around with Solaris in vmware and it doesn't seems all that hard getting used to. I'm still feel a little lost from time to time but i had the same feeling when i started out with Linux years back so it'll probably go away. I do, however, really miss a proper package manager and centralized repositories, like dpkg in Debian.

The biggest thing to it is to make sure that your hardware that you're planning on running/using is listed on the Solaris HCL. The lack of a LOT of drivers is still one of the big problems that Solaris is facing for widespread adoption. However, it is getting better, but the market is definitely outpacing it's driver development group,

Having said that, if you were to build a system designed to be able to run Solaris - this is an example of what I am currently using right now:

MSI KT4AV-L (SocketA)

AMD Duron 1.8 GHz

2x 256 MB PC2100

Powercolor ATi Radeon 9250 64 MB

Netgear GA311 GbE NIC (based on the realtek 8169 chipset)

(hard drives)

My base system at the time of build, including case, power supply, optical drive, etc.. came up to around $500 US.

The only downside to it is that ZFS has a max CPU utilization of 14% on that system with a 4 SCSI + 1 PATA drive RAIDZ array.

The sun HCL only seems to feature a very limited selection of hardware but according to what I've read all the hardware I've picked above should work.

The only issue I've read about is that solaris apparently don't have any drivers for the nForce SATA controllers. The controller will work but it'll run in non-native mode which means that it will operate with the same featureset and speed as good old PATA drives. That means: no SATA300, no NCQ and no TCQ. I don't know if that has any real-life impact on performance though.

I don't know if the Intel SATA controllers are better supported.

SOME native Linux stuff is supposed to be able to run entirely within a Solaris Container that segregates the core OS from the Linux. I don't know enough about it to be able to say whether it works one way or another because I haven't been able to find very many good "how-to" guides to set that up and get it working/running.

It's supposed to fully support VMWare ESX server and that's probably the route that I would be going with if I were to try and do any virtualization stuff.

I've been doing some searching and apparently there is an utility called lxrun that aims to provide binary compatibility with linux. It seems that it'll fulfill my needs if i should ever need to run some binary-only Linux stuff (Like a game server)

Note: Solaris REALLY works well with AMD Opterons, both the old ones using DDR and the new ones using DDR2 memory. Yes, they're more expensive, but it's performance is like an exponential increase compared to my "lowly" Duron 1.8 GHz.

Opertons and Athlon 64's are essentially the same thing right?

It is my understanding that when you want to add capacity to the array, that depends on the type of array you have set up.

I think that for a stripped array (raid0), you should be able to do a straight add. (I don't know, haven't tried it).

For a raidz; the configuration has to be either identical or better than the array that you started out with. So if you have a 3-drive raidz (capacity of n-1); that your minimum add-on should also be a 3-drive raidz.

That's a major drawback for me.

I personally think that expanding raidz will be added as an option in future versions zfs. It's a pretty important feature IMO. As far as i know it's not mentioned in any current roadmap so i don't know how long it will take.

The CPU utilization figures were for a raidz array (so it is technically neither a raid0 or a raid5).

ok

Realistically, you're either going run out of users pulling data from the storage server, or you're going to run out of network bandwidth (even if you're on a GbE) or you're hardware limited (i.e. hard drives, memory, etc.)

If performance is really that much of a concern, then going GbE is the first thing you should also consider (at minimum) along with the server upgrade.

If you're just expanding to a GbE - chances are - you're still quite a ways before you're going to hit your limit. And when you do, then you can consider alternative options such as SCSI, FC, SAS, and fibre switches.

If you're the only one using the network bandwidth primary and manipulating and moving the data around; you're rarely ever going to see the limit.

Performance isn't that much of a concern. When i wrote the original post i weren't even going to upgrade to gigabit LAN, but in the past few days I've realized how nice it would be to have higher transferring speeds(and also because the current 100mbit network gets easily overloaded). And when I'm buying completely new hardware anyway i might just as well go for the best performance possible within my economical limits. SCSI etc. will never be an option for me. My needs are simply not big enough to justify the price. (And they'll never be)

I'm still really unsure whether i should go with Linux or Solaris. I have lots of expirience with Linux and i know how to use it and administer it. I know it will support everything i need now and in the future. Even though Solaris in many ways are similar to Linux it's still different in many areas so i will spend some time learning it (and time is not something i have lots of). The features and easy administration of zfs would be nice to have but i don't desperately need it. Linux storage solutions would fulfill my needs just fine. I must admit I'm currently heading towards the Linux side. btw. Is it normal to run a Solaris server with X? Most of the administration tools seems rather gui-centric.

What would you have chosen in my situation?

Share this post


Link to post
Share on other sites

I don't know what went wrong with that post. Why weren't the

tags made into quotes? I cant seem to be able to edit the post either? Is only one edit allowed on the sr forums?

Share this post


Link to post
Share on other sites

I'm a newbie at storage but I'm building a fileserver as well at the moment.

You want low power and then you can get at cheaper UPS (well it won't hurt) I love my APC RS1500VA btw.

Get a sempron AM2 (do you really need dual core?) There are some 35w that are a bit harder to find and are a bit more expensive. But if you need dual core get the X2 65w.

Mobo: The Gigabyte GA-M55plus-S3G which has undervolting in bios (can't use Crystalcpuid on Linux) becuase you want to undervolt that chip. All Semprons undervolt well and the EE X2 does also.

It has nv raid 5 (for what it's worth).

I would (will) install Linux on a Flash card via USB or ide adapter.

Share this post


Link to post
Share on other sites
Finding a cheap case which provides lots of 3,5" device bays and provides proper cooling for the disks is not easy. A reasonable quality PSU which is not too expensive is not easy to find either.

I've compiled a list with 3 possible hardware suggestion together with their pros and cons (as I see them).

I was only able to find motherboards with 4 sata connectors within a reasonable price range. Does anyone know any cheap boards with 6 sata's ? ,

It might be useful if i said something about what the server is being used for. It works as a central fileserver/backup for two families (8 people in total). It also runs a ftp server and a web server. The web server is mostly being used to test the web pages i develop and it's also going to host my personal site. I also run a picture gallery for some friends. This is probably the task requiring the most performance. Resizing pictures and converting videos is really slow on my current server.

[...]

Suggestion three

ASUS M2N-MX 460kr ($79.35)

Athlon 64 3500+ 2,2 ghz 665kr ($114.72)

Kingston HyperX 2x256mb CL3 PC2-4300 596kr ($102.81)

Total: 1721kr ($296.86)

Pros

Other than DDR2 and the AM2 socket being future proof i don't know

Cons

More expensive

To my knowledge the AM2 athlons are architecturally identical to the s939 athlons and the high latency DDR2 ram running at low speeds will actually run slower than low latency running at equivalent speeds .

Does anyone know how the nForce Sata controllers performs compared to the Intel ones and what the performance difference is between Realtek, nVidia, Marvell, Attansic and Intel gigabit lan chips?

I would appreciate to get some feedback on above suggestions. Completely different suggestions are welcomed as well.

I'll start with Intel chipsets here, because it's likely to be better supported under Linux/Solaris, and also because you're likely to find the cheapest dual core options in that line. I believe that for encoding / etc., a dual core could be beneficial (depending on software). Of course you could go down to a single inexpensive core should you so decide, and even to a faster dual core down the road, as they become cheaper.

E.g. Pentium 9-series dual core. E.g. 915 ~$120.

http://www.newegg.com/Product/Product.asp?...N82E16819116253

E.g. Intel Desktop Board DQ965GFEKR for $122 USD

http://www.directron.com/dq965gfekr.html

Here's a more general query -- >= 6 SATA ports with Intel P/Q965 Express Chipset:

http://www.newegg.com/Product/ProductList....%2C2968%3A22403

There are a few hits. Intel boards should be nice for their SATA implementation and nice on-board Intel GbE. Intel GbE should perform about as well or better than most others, and also has good driver support.

You need to be careful with memory compatibility -- check support notes; probably stick to 1.8V.

For case / PSU, there are also lots of options. I suggest the Antec SLK3000B or variant for the case. With an optional 120mm intake fan, it'll hold 5 drives cooled. With an optional drive cage such as a Thermaltake iCage, Lian Li, or some other, you can add 3 or more cooled drives. The SLK3000B is not expensive and nicely designed.

On one system here, running an Enermax NoiseTaker 425 PSU, dual core AMD, a storage controller and 7 drives (no staggered spin-up), I measured ~300W peak during start-up, and ~145W run-time. The Enermax is a bit more expensive than some other decent PSUs. A Fortron 400W or so should be fine.

All this boosts the price much beyond the original $250 or so that I guessed, and I don't really like doing that, so they're just options for you to consider if you like. You could go down to a Celeron or Sempron and not lose much in file server performance.

Share this post


Link to post
Share on other sites

I'd consider looking at the XFX Revo 64. It matches my needs perfectly and sounds like it may be good for you also. It's a 5-port SATA hardware RAID 3 card (much like RAID 5). The big advantage for me is that it's dirt cheap (relatively speaking) - mine was £30 (~$55) compared to around £50 for most 4-port cards, or £75 for an 8-port.

It's also nicely upgradable - add an extra disk and it just sorts it out automatically. I'm currently using 3x400GB discs in RAID3, which gives 800 GB array. Just plug in a fourth drive, and it appears as a 1.2TB array, and a fifth to get 1.6TB. I'm currently ripping DVDs onto a server for my media center PC so this simple expandibility is perfect and means I get to save money now without causing hassle down the line.

It also has a useful feature of just appearing as an IDE disk to the rest of the system. I haven't tried using it as a boot disk as I have a WD Raptor for that, but I believe it should just work.

Share this post


Link to post
Share on other sites

I know this post is getting really old but I thought I'd drop by to tell what my final choice fell on.

I have just ordered the following.

Intel DQ965GF Motherboard

Intel Core 2 Duo E6300 1.8GHz (It performs a lot better than the Pentium D and uses less power. Worth the price IMO)

2x512MB PC2-4200 CL4 Kingston Value (You don't get far with 512mb ram these days so i reached the conclusion that it was better to get 1gb at once)

4x Hitachi T7K500 16MB 320GB

Antec New Solution NSK6500 with 430W psu (I really hope it will be powerful enough)

2x 92mm Antec TriCool fans (for the hd's)

Some cheap Sony DVD-ROM drive

I am going to use my 17GB SCSI drive as system drive with ext3/ReiserFS. I hope it's not going to fail anytime soon. The data disks are going to be set up as RAID 5 and formatted with the XFS filesystem.

So i ended up spending a lot more money than I originally intended to. I was somehow able to convince myself of how great it would be to have a separate mythTV backend server (My current Athlon 2500+ mythtv box is getting a bit slow) so i had an excuse to get this expensive server :)

My choice of OS fell on Linux. Mainly for it's wide Hardware/Software support and the fact that even though ZFS is nice i don't actually need all the features it provides.

And finally a big thank you to all of you who have contributed to this post. I really appreciated your very useful advice and information.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now