Eugene

Tcq, Raid, Scsi, And Sata

Recommended Posts

Okay, lots of stuff to respond to.

The driver only "comes with" the SATA adapter if you are trapped in the Windows prison. On Linux, FreeBSD, et al the driver authors and the hardware vendors are orthogonal. So it's possible to have command queueing even if the Windows driver does not support same.

Secondly, I think it is too bad that software RAID wasn't tested. Quiz: what makes a better RAID processor, 1) an Opteron 244, or 2) an Intel i960? If you picked 1, you picked correctly. And the Mylex 170, what a dog. It does not exactly have a reputation for speed. Software RAIDs will blow its proverbial doors off.

Also it is easy to disable or limit command queueing on some SCSI HBAs. On Adaptec HBAs the depth can be clamped anywhere between 0 and 255, on a per-target basis.

I do take some minor issue with how the article is presented, especially its conclusions. I think the definition of single-user or desktop user is vague. Many people do absurd operations on their personal computers that would make some servers blush. I personally have a PostgreSQL database that exceeds a terabyte, and I like to compile very large trees of software. I use a RAID5, because I dislike losing my data, and there is a significant benefit to compile time from the RAID setup. The RAID5 setup also improves performance on metadata-intensive workloads, like tar/untar or cvs update.

Finally, thanks for a nice article with some good hard data.

Share this post


Link to post
Share on other sites
I was wondering if you have done any analysis on the actual performance gains of multiple non-raid disk configurations in desktops?

Unfortunately, the SR drivemarks can only be run on a single volume due to the manner in which they were captured. However, there is a great deal of information on using independent disks in Storagereview's Computing forum.

The incendiarily titled thread, "The Death of RAID" is the most recent thread on the front page of the computing forum which involves discussion of the merits of independent disks. It's quite large so I will shamelessly direct you to a detailed post I made explaining why independent disks are always better than striping for multitasking situations. It also details some basic theory that explains the consistently verified observation that striped sets rarely perform better than single disks in single-user scenarios.

I regularly involve myself in threads related to topics involving the most effective use of multiple disks so my handle may be of use if you search the archives. I suggest combinations of +multiple +spindle* +independent +dependent +striping +configuration and whatever else your imagination can come up with. There's lots in there.

my question is: does the same hold true for the single user scenario under heavy load? Under low memory conditions? Working with lots of monstrously large files that can’t fit into RAM? This would seem to be a better approach to leveraging multiple hard drives in the same desktop. Would you agree?

Ys, yes indeed. Independent disks can offer dramatically superior performance gains relative to striped sets (which generally perform just like single disks). While you cite the ubiquitous appreciation of this fact in the enterprise server realm, many professional single-user workloads have been known to benefit from this configuration as well:

1. The example most recognizable to most everyday computer users would be Adobe's Photoshop software. It performs significantly better if its scratchdisk is isolated to its own spindle. Many video editing programs also use a scratchdisk. While enthusiasts often recommend RAID 0 for video editing, they actually have no idea what they are talking about. RAID should not be considered until the source file, destination file, and scratchfile are located on seperate spindles. Performance critical production environments have recognized the necessity of this for years and years.

2. It has also been known for years that placing the Windows pagefile on an independent spindle will also improve Windows virtual memory performance which is very important during multitasking or application loading, as you suggested it might. This is one of the single, most significant performance enhancements any multitasker can make. You see, Windows doesn't wait to page applications or other data out of memory until you run out of RAM. It will page things out even if you're only using half your memory. Some enthusiasts freak out over this, but it is a very intelligent design optimization. It does this to optimize responsiveness by devoting more RAM to the disk cache which greatly improves system responsiveness. It also allows the forground application to run as smoothly as possible.

Windows is particularly likely to simultaneously page memory to disk as it loads an application. The sudden memory changes induced by a starting up application provoke this, so the pagefile should also be on a different disk than one's applications if one is concerned about app loading speed.

To these examples we should add any sort of multitasking that involves concurrent IOs. Such workloads will function more efficiently on independent disks because an independent configuration maximizes the effects of caching by the disks. Also importantly, the independent configuration dramatically reduces the time the disks spend seeking. Disks do not read data while seeking, so seeking has a debilitating effect on storage performance. Minimizing seeks is an important technique for optimizing IO. In servers this can be impossible, on the desktop it is as simple as using multiple independent disks.

Share this post


Link to post
Share on other sites
Limitations of the PCI Bus

The 133 MB/sec limit of the standard 32-bit, 33 MHz PCI bus may be of concern to some, especially those seeking for various reasons to maximize sequential transfer rates. The practical real-world limit remains slightly below that threshold- STR tests associated with the results below top out at 126 MB/sec.

Slight correction:

Standard 32-bit, 33 MHz PCI bus has a limitation of 127MB/s (32 bit * 33.33MHz = 133,320,000 bytes = 127.14 MB/s). Given that threshold, the real world results (126MB/s) show efficient use of the bandwidth that is available (the rest being consumed by command overhead?).

Share this post


Link to post
Share on other sites

Thanks for a good article, however it does not help me :(

How did you define the single-user/desktop user, is it my girlfriend browsing the internet and running only Internet Explorer and ZoneAlarm. Or was it defined for someone like me running :

* 8 Internet Explorer sessions

* ZoneAlarm

* Antivirus

* Downloading a 500 Mb iso image

* Downloading 4 small files simultaneously

* Lotus Notes 6.5 (mail/database application)

* copying 1000's small files from disk to disk or from disk to server

* copying 500 Mb photo images CD ROM to disk

* watching a DVD from a DVD drive

all at the same time of course.

It has worked better for me when I used SCSI instead of PATA. The system seems more "crisp" :D but I cannot substantiate this with any test.

I have never seen a test that has really focused on power users like me - all the users in this forum are power users, but I find it hard to believe that the tests are designed for me, rather they are designed for my girlfriend, however she is not participating in this forum - this is a forum for technical users.

Do you have any poweruser tests ?

Thanks

Carsten

Share this post


Link to post
Share on other sites
How did you define the single-user/desktop user

clille, that information is outlined in the TB3 article, you may which to note that a link to it was prominently placed at the top of the Tcq/Raid article (i.e. see the header that reads "How we test Drives"). The page that is of most interest to you can be found here.

I have never seen a test that has really focused on power users like me - all the users in this forum are power users, but I find it hard to believe that the tests are designed for me, rather they are designed for my girlfriend, however she is not participating in this forum - this is a forum for technical users.
Sorry, but I really don't see anything in your task list that really dictates "power" usage. Although you may be "multitasking" more than an average computer user (i.e. your girlfriend), the access patterns and performance requirements seem to fall within the realm of single user desktop useage. For example:

* 8 Internet Explorer sessions - not disk intensive

* ZoneAlarm - not disk intensive

* Antivirus - if just residing in the background, not disk intensive

* Downloading a 500 Mb iso image - nothing a HDD is going to break a sweet over. Your bottlenecked by the network connection

* Downloading 4 small files simultaneously - ditto

* Lotus Notes 6.5 (mail/database application) - I have no idea what your database activity may be, but I highly suspect that unless you have multiple users connceted to it, it would probably fall squarely into the single user desktop category. Email is definitely not disk intensive.

* copying 1000's small files from disk to disk or from disk to server - are you constantly copying thousands of files?

* copying 500 Mb photo images CD ROM to disk - are you constantly copying image files?

* watching a DVD from a DVD drive - not disk intensive (no activity at all in fact)

As some food for thought, consider the following statement Eugene once made:

Its interesting to see many readers say "but I multitask so single-user patterns don't apply to me!" The term "single-user" means exactly that. It doesn't matter if you're running multiple video streams while compiling your kernel and running Quake and Unreal at the same time. Such usage is still single-user and features localized patterns much closer to traditional desktop usage than server patterns.

Share this post


Link to post
Share on other sites

In the same spirit as this excellent article, would it be useless to buy an Adaptec 2120S instead a 29320R (or 39320R) scsi controller to control two (at first) and later four Seagate 15k.3 (or 15k.4 now....) ?

The first one has a processor and 64MB RAM, the second one has the software integrated "Hostraid" function. What is this hostraid function ? Is it TCQ ? Does it work fine ? Doesn't it take too much CPU usage ?

It's for a future AMD bi-opteron CAD/Rendering/Video station, with a small server usage.

Thanks for answering, i really don't know what to buy and i don't finc articles/comparaison between these two adaptec product families.

Share this post


Link to post
Share on other sites

Considering you wouldn't want to use the RAID functionality, you should simply buy the least expensive SCSI card that supports your OS and has a bus that is compatible with your drives.

LSI Logic makes good, inexpensive SCSI HBAs that have driver support for all major OSes.

Share this post


Link to post
Share on other sites

Hello, the question is between two raid controllers :

Adaptec 2120S and the 2(3)9320R

To use them in a raid configuration, with two or four drives.

Thanks

Share this post


Link to post
Share on other sites

In a recent article from PC World, testing SATA and SATA with NCQ (single drive/single controller), their results were opposite of yours.

PC World

They tested 2 PCs - a PC using SATA and a PC using SATA with NCQ (HDD and controller). The results showed a performance increase when using the SATA with NCQ. There were caveats to their test, which they point out, but compared to your test results, you would think their results would show the same performance, if not a degraded performance, in the PC using SATA and NCQ.

My decision is whether to use the latest SCSI, SATA with NCQ or just SATA in my next single drive/single controller (non-RAID) desktop PC.

My guess is we will start seeing more and more SATA HDDs and controllers using NCQ. If your tests are correct, then this will actually degrade performace in PCs.

Perhaps it's too early to tell which is better... Different testing software on different hardware can yield different results. For now, the jury is out on which is better (for me, at least). I will have to see more test results before making a decision.

Share this post


Link to post
Share on other sites
They tested 2 PCs - a PC using SATA and a PC using SATA with NCQ (HDD and controller).  The results showed a performance increase when using the SATA with NCQ.  There were caveats to their test, which they point out, but compared to your test results, you would think their results would show the same performance, if not a degraded performance, in the PC using SATA and NCQ.

You left out the best part:

"We couldn't turn off the NCQ capabilities of the DiamondMax 10, so for comparison purposes we also tested Maxtor's next-closest drive--a shipping 250GB DiamondMax Plus 9 with half as much (8MB) cache ($200)."

That said, our own (yet to be published) take on NCQ indicates that it may be more efficient than the Raptor's TCQ. I would, however, caution you from drawing conclusions based on PC World's apples-to-oranges comparison.

Share this post


Link to post
Share on other sites

Hi Eugene,

I just read the article that has initiated this thread.

Your conclusion states:

"Non-server use, even in heavy multitasking situations, generates lower-depth, highly-localized access patterns where read-ahead and write-back strategies dominate. Theory has told those willing to listen that striping does not yield significant performance benefits."

and

"Save your time, money and data- leave RAID for the servers!"

I am wondering if this conclusion applies equally to my non-server single user scenario: statistical analysis of data sets that are hundreds of GB in size using SAS statistical software.

I am about to begin a project that will analyze an entire state's set of health care claims data (for readers who may know of such things). The subset of these data that just had one particular disease (a previous project) was 5 GB in size. We will be purchasing new hardware for this work, and had thought that a RAID 0 workstation would be a sensible solution.

The folks at SAS indicate that I/O is the biggest issue for large data set analysis:

I asked "Is maximizing the RAM always step one? That would be the 4 GB. Even if that means cutting back the processor or the RAID?"

SAs Answer: "Absolutely no. I/O is the biggest performance problem with

most of our customers who are using PC's."

I asked: "When you say that I/O is the biggest performance issue for SAS on PC's,

I assume that is because of reads and writes to virtual memory during processing?"

SAS Answer: "No. It's the amount of data that is being accessed from the hard drive. If you don't have enough memory to do the work SAS is asking as it ccessed the data then the system will page to the hard drive which can add dditional I/O over and beyond what SAS is already using. This would slow you down even more."

So it appears that for SAS analyis of large data sets, just getting the data off the hard drive is time consuming, even without virtual memory access issues.

(Since I am no computer expert, does striping affect virtural memory performance?)

So would you agree that for this type of work, RAID may make sense in a single workstation? We do want to make the right hardware choice.

StatsDoc

Share this post


Link to post
Share on other sites

You should probably have posted that question in the computing forum, but never mind. This definitely doesn't count as typical single user/desktop usage, and needs considering on its own merits.

Given what they've said, it sounds like you'll need to address the database storage and RAM issues separately.

It's obviously important to have enough memory for the task, but if you can't afford that much RAM, then having a dedicated disk for virtual memory is a high priority.

This will have to be balanced against your budget for the database storage.

Ideally, you'd want a drive for OS & Apps, a drive for pagefile, and a drive or array for the database. All depending on budget. Perhaps one drive for OS, Apps and Pagefile while another drive/array handles the database would be a reasonable compromise.

I guess you'll be doing mostly reads. RAID might be useful for this - how important is uptime? Will the database be written to, or will it be fairly static apart from importing new data? Do you need the extra reliability of RAID 1 or 5, or can you afford to lose your copy of the database and restore it from backup if a drive fails?

We'll need a bit more info about the nature of database accesses - highly random? High queue depths? Access time or STR limited? SAS may know, if you don't. They probably have their own recommendations for hardware set ups - ask them. Let us know what they say.

Share this post


Link to post
Share on other sites

Hi Spod,

Thanks for the reply. The article (TCQ, RAID, SCSI, and SATA) had a link directly to this thread, so that is how I ended up here.

The "Single Drive vs. RAID 0" article on the side said:

"Again, RAID 0 does have its advantages in a handful of key applications and uses where data files are huge and/or data requests are highly sequential in nature."

I assume that huge SAS data sets would have highly sequential data requests.

What we would do is take the original data base and run a SAS program to convert it into a "SAS data set". That is a data array where each row is an observation, and each column is a variable. Statewide claims data would have hundreds of millions of observations and perhaps several hundred variables.

My understanding is that when SAS processes a procedure (a frequency count or a mean, for example), it goes through the data array line by line (row by row). So it ought to be sequential reading, I would think.

The SAS data set itself is not altered by running the analysis procedures. So I suspect that analysis is lots of read an little write. We can modify the data set using the "data step" in programming, which would generate a new SAS data set. That would be a less common occurrence than the analysis procedures. So I think that the answer to one of your questions is that we will be doing mostly reads.

The data are fixed once all of a calendar year's claims are compiled. We are always working with the data from years past. The 2002 data base does not change after it is compiled. So the database is static.

We can always re-create the SAS data set from the original source data and the SAS programs. We are not acquiring new data in real time on this system. So we thought that RAID 0 with a nightly external backup would be perhaps the fastest performance (vs. RAID 5) without the expense of the capacity redundancy (vs. RAID 1).

The real key may be to periodically save the output files to removable media, since it is the relatively small output files (e.g. a cross-tab of disease or not by age categories, in strata of race, where sex is female: huge processing job to count it up, but the output fits on 4 pages of Word) that are the important work product.

Lots I don't know about how SAS accesses data. I can ask SAS about the random, queue, access time vs. STR issues. They were not willing to commit to a specific hardware suggestion when I asked, other than what I already posted about their impression that I/O is more important than RAM (but maybe that was assuming adequate RAM? how much is adequate RAM for a 100 GB data set?) and a couple of other comments:

I asked: "If I have $383 more to spend, should I upgrade the processor or the

RAM?" (383 at Dell site upgrades 3.0 Xeon to 3.4, or 2GB RAM to 3GB)

SAS Answer: "I would spend it on Ram since this could impact sorting."

I asked: "If I have $1200 more to spend, should upgrade the RAM or the RAID (850 at Dell would upgrade 2GB RAM to 4 GB RAM, 1200 would upgrade 2 disk RAID 0 to 4 disk RAID 0)

SAS Answer: "I would upgrade the Raid. And if data redundancy is not

a problem as you have indicated, I would not use Raid 5 as it is very slow."

My comments: "Our network folks are fond of RAID 5, but they use it for servers. I can see that running SAS is a different beast, and that RAID 0 could be better for SAS performance."

SAS Answer: "This is what our experience is and that of our customers

providing you have provisions for backing your data up of course."

SAS actually suggested that I look at StorageReview for advice.

What I don't know is whether SAS is smart enough to read the data in small units off the disks as it processes, or if it wants to try to put the whole SAS data set in RAM before it starts. That would seem to be key to whether virtual memory is a processing issue. Maybe just the SAS software and my program that needs to fit in RAM, not the data.

You suggest a dedicated virtual memory disk. My original thought was that virtual memory would be the logjam and that RAID 0 would be a way to speed up virtual memory processing. But that assumes that virtual memory can be striped: maybe it can't? (you see how non-expert I am) Is virtual memory better done by a single disk or a RAID 0 array?

Also, during a SAS computation, would access to the OS files be likely to be a frequent event (probably a question for SAS)? If not, then maybe a separate disk for OS and apps is less critical?

Given the likely highly sequential nature of data access for any computational SAS procedure, I assume a small block size would make sense in a striped array? (If I am recalling the terms right, block refers to the amount written to one disk before writing to other disks in the array)

Dell offers 73 GB SCSI drives at 15,000 rpm and 146 GB SCSI drives at 10,000 rpm. (I suspect we are limited to the state's designated vendor) Do you think the extra RPM is worth doubling the number of drives (I have read that more drives increases array failure rate in RAID 0)?

Share this post


Link to post
Share on other sites
When released this August, the Promise TX4200's price will be on par with that of the S150 TX4 ...

I have been waiting since mid-July for the release of the TX4200 from Promise. I have tried searching the internet for any details on the card's release date, but I have not found anything mentioned on Promise's website or any other site. The only thing I have found is a listing on Western Digital's online store, though the details only say that the card is not in stock.

Does anyone here have any clue when this card will become available, or know where I could find out?

Thank you.

Share this post


Link to post
Share on other sites

I called Promise Technology's California Sales department that they are expecting their first shipment of TX4200's at the end of this month. It will then take a week or more for the cards to make their way to retailers.

Share this post


Link to post
Share on other sites
I called Promise Technology's California Sales department that they are expecting their first shipment of TX4200's at the end of this month.  It will then take a week or more for the cards to make their way to retailers.

SO.. Any news about the release date now ???

TanGU.

Share this post


Link to post
Share on other sites
I called Promise Technology's California Sales department that they are expecting their first shipment of TX4200's at the end of this month.  It will then take a week or more for the cards to make their way to retailers.

Card's out !

it's directly on the cover page of Promise.com, here is the link for those who was waiting for it:

http://www.promise.com/product/product_det...=127&familyId=2

Hey check their logo :

for%20OII%20RAID-small.jpg

mouahaha :)

FastTrak TX4200 Highlights:

* 4 Enhanced Serial ATA 1.5G Ports for support of up to 4 drives

* RAID 0/1/10 and JBOD

* PCI - 32-Bit / 66 MHz

* Offers sustained data transfers up to 266MB/sec

* Native Command Queuing

* SATA Tagged Command Queuing

* Large LBA support for drives above 137GB

* Disk Activity LED Headers

* Combine with the SuperSwap 1100 or SuperSwap 4100 for an advanced internal storage system

Who will buy that card :)

TanGU.

Share this post


Link to post
Share on other sites
First up will be a revised look at today's 7200 RPM ATA drives. Players such as Seagate, Hitachi, and Western Digital have all updated previously released models with newer firmware and, in some cases, newer capacity.

The articles mention virtually nothing about this in reference to Seagate drives. I couldn't care less about Western Digital or Hitachi -- bombs away.

I'm particularly interested in hearing if these firmwares can be obtained and applied to previously-released drives. In my case, I'm the owner of numerous SATA 7200.7 units (ST3120026AS running firmware 3.05), and am interested in hearing if these units can be upgraded to support TCQ, or have any other improvements.

Advice to SR: "I can't maneuver!" "Stay on target!" "We're too close!" "Stay on target!" ;-)

Share this post


Link to post
Share on other sites

few things

1. the controllers in this review all kind of suck-- try including the escalade 9xxx or RAIDcore controllers, as suggested.

2. the benchmarks all represent a productivity user on crack scenario-- real power users move happy fellowtons of data around, and I didn't see a single large file transfer benchmark, something that is either going to represent a *very* deep queue environment (gather of fragmented sectors in P2P downloads) or a linear read/write (something an optimized RAID-0 implemention should slaughter).

3. real world benchmarks. these synthetic drivemark scores are all fine in practice, but they don't represent real-world patterns at all, and IO/sec is a pretty vague figure- real file systems aren't perfectly sector aligned, and the clock you're staring at waiting for UT2K4 to load isn't demarcated in I/Os.

while i agree that random access time is not going to improve a lot through the use of RAID-0 in the average single user environment, any respectable implementation of RAID-0 will yield much higher average real-world throughput, and there are in fact single user tasks that display random throughput benefits in RAID0. (small file throughput in reiserFS comes to mind- single 30GB disk took ~9 minutes to prune undesired content from an archived web site, 3x 9gb (5 year old scsi disks with absolutely miserable STR) array took ~2 minutes, array-to-array versus disk-to-disk times for tar extraction and file copy are orders of magnitude greater as well)

biased data against RAID doesn't do anyone any more good than biased data for it, and dismissing dissenting opinions *certainly* doesn't do any good.

Share this post


Link to post
Share on other sites

i hate to triple reply but i just noticed the comment about software RAID up top-- you are exactly right! the RAIDcore is actually a highly optimized software implementation, and you can tell- it doesn't require obscenely deep queues to break even on performance, and in certain benchmarks that traditionally rub very poorly with hardware-accelerated RAID controllers, the RAIDcore delivers performance shockingly superior to not only comparable SATA controllers, but high end SCSI implementations.

if you are going to dismiss RAID out of hand, at least catch the right culprit: processing overhead. an optimal implementation of RAID would yield performance benefits in any disk-limited application, but realistically processing overhead and serialization of requests limits the realization of this theoretical advantage.

Share this post


Link to post
Share on other sites

I've read this article a few times now, and each time I find it more and more informative. Thanks.

And as others have said, I too would be interested in seeing some comparison of the new 9xxx series from 3ware.

Share this post


Link to post
Share on other sites

I'm a bit of a graybeard, so please bear with me if I'm asking a stupid question. Back when 10K SCSI drives came out, I had (still have) a MegaRAID controller. One of the first lessons that I learned was that if there was an error on either drive in RAID 0, everything was toast. So, I quickly bought a third Cheetah and went to RAID 5. Now most good MBs have RAID 0 or 1 on the MB. Since RAID 0 is striping and is NOT redundant, it's a bit of an oxymoron. Your review bypassed RAID 5 and went to RAID 10.

1) Is RAID 10 striping two pairs of mirrored drives or mirroring two pairs of striped drives? Just curious on this one. I don't think it would make a lot of difference.

2) Have prices changed so much that four spindles in RAID 10 makes more economic sense than three spindles in RAID 5? Mathematically, it's hard to believe that argument would scale, since it would require 10 spindles in RAID 10 to duplicate a six spindle RAID 5 array's capacity.

As I said, I've been away from the RAID game for a while, so I really don't know the current basics. It seems to fly in the face of what I learned before, but that's not that unusual, as hardware evolves (this from a guy who punched cards to program an IBM 360 in Fortran IV, in high school in 1969).

chartguy

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now