Sign in to follow this  
Followers 0
T-Minus10

leaderboard not effective

25 posts in this topic

I've been reading Storage Review articles for years now and the site persists in providing the best hard drive data anywhere. Information obtained through your articles has influenced almost every one of my hard drive purchases. However, the leaderboard, as a tool to guide hard drive purchases, has a few flaws.

As a simple critique of the way the leaderboard is updated (and please, correct me if I'm wrong) -- when a new drive is released that is superior to previous leaders but inferior to the current champion, it gets no recognition on the leaderboard. Although the leaderboard appears to be a list of current top drives, really it's a history of champions -- and as a tool to purchase hard drives, a list of current top-drives is all that concerns me.

My second critique is of the performance-focus of Storage Review. Really, this critique is an appeal for a new category. The fact is, hard drives, especially external hard drives are one of the easiest, highest-capacity, highest-performance forms of portable media available on the market today. While Storage Review does a good job testing and categorizing hard drives on the basis of performance, the reliability database seems like an afterthought. New product reviews conclude, in the future, the database will complete this article." Old drives -- and by those, I mean those that do not occupy a seat on the leaderboard, are only notable when they blow up sooner than they should.

My proposal is for a new category ranking hard drives on proven reliability, durability, and cost-effectiveness. Of the last six hard drives I bought, only three of them are being used in performance-critical applications. The rest are all in external enclosures that spend a large chunk of their lives both in transit, and turned off. They get bumped around and power cycled more than regular hard drives. And most importantly, when buying them, performance was not a feature that I took into consideration. Cost-per-gigabyte (including a cost of enclosure in the cost of a drive), heat production, and reliability were all features that I looked for. Articles and a leaderboard category covering the use of hard drive as removable media would be wonderful. Even better -- also ranking enclosures for different applications.

Other forms of media are inherently more reliable than hard drives. Moreover, DVDs and tape are each significantly cheaper per-gigabyte forms of media. However all around, hard drives have better storage densities and are easier to use than other medias, and the enclosures offered for external hard drives often provide extensive functionality unavailable with DVD and tape drives: What kind of DVD-RW drive, when detatched from your computer, will accept images from a digital camera, or play or display digital media, or will hook up to a network and form a fileserver? What kind of tape drive is 1.8 inches wide and can fit in your wallet?

Understandably, with the scope of applications that hard drives as media provide, it would be difficult to cover everything and maintain the attention to detail that Storage Review is known for. I admit, I get a little excited when I think about my ipod as more than a mp3 player. However, ignoring the applications, as noted earlier, the hard drive as a storage media has a number of features that can be weighted, and one of the things that often doesn't matter is performance.

So there you go. Keep on keeping on, Storage Review, and hopefully my next archival drive will come straight from the SR leaderboard instead of from relatively extensive research.

-Mike Glass

SR long time fan and general hard drive enthusiast.

Share this post


Link to post
Share on other sites

i realize that that post was monstrous and verbose.

SUMMARY:

1: NEW DRIVES SUPERIOR TO OLD LEADERS BUT WORSE THAN THE NEWEST LEADERS GET NO RECOGNITION

2: PERFORMANCE IS ALL THAT IS CONSIDERED IN GRADING BEST DRIVES. ALTERNATE USES OF HARD DRIVES = INTERESTING, OFTEN NOT PERFORMANCE-HUNGRY

2.1: HARD DRIVES' USE AS (REMOVABLE) MEDIA BEGS A LEADERBOARD CATEGORY FOR CHEAP RELIABLE DRIVES (PERHAPS UPDATED DYNAMICALLY GIVEN CONSTANTLY UPDATING PRICE/RELIABILITY DATA)

Share this post


Link to post
Share on other sites
My proposal is for a new category ranking hard drives on proven reliability, durability, and cost-effectiveness.
Proven by whom, exactly? And how?
The rest are all in external enclosures that spend a large chunk of their lives both in transit, and turned off. They get bumped around and power cycled more than regular hard drives. And most importantly, when buying them, performance was not a feature that I took into consideration. Cost-per-gigabyte (including a cost of enclosure in the cost of a drive), heat production, and reliability were all features that I looked for. Articles and a leaderboard category covering the use of hard drive as removable media would be wonderful.
I don't see how this benefits anyone. You want drives in external enclosures looked at too? Is that right? So now we have enclosure chipsets & mainboard chipsets to factor in? Is this what you're asking?

And cost per gigabyte? For what market? Albania? Ireland? Tanzania? Can't you take a look at a disk at your local shop & figure out what it costs per gigabyte, then compare it to what ever other disk you're interested in? You typed a thousand words to ask for cost-per-gig & external-enclosure numbers? Are you mad?

Share this post


Link to post
Share on other sites
1: NEW DRIVES SUPERIOR TO OLD LEADERS BUT WORSE THAN THE NEWEST LEADERS GET NO RECOGNITION
But how do we know that a "new drive superior to old leader" is "worse than the newest leader (not yet reviewed)"? Even Eugene isn't THAT allmighty, AFAIK.
2: PERFORMANCE IS ALL THAT IS CONSIDERED IN GRADING BEST DRIVES. ALTERNATE USES OF HARD DRIVES = INTERESTING, OFTEN NOT PERFORMANCE-HUNGRY
Alternative uses doesn't probably require their Leaderboard entry. One can always access Performance Database and select to sort by Idle power consumption, and find out which drives are best fit for external HDD solutions, etc.
2.1: HARD DRIVES' USE AS (REMOVABLE) MEDIA BEGS A LEADERBOARD CATEGORY FOR CHEAP RELIABLE DRIVES (PERHAPS UPDATED DYNAMICALLY GIVEN CONSTANTLY UPDATING PRICE/RELIABILITY DATA)
Leaderboard based on price = useless. Do you assume the prices are the same in Australia and in Austria? People should always find the cheapest deal where they live, not where StorageReview.com is operated.

Doing performance benchmarks to multiple drives inside a USB enclosure from Brand X is probably quite useless. Some 0.1ms to 1ms overhead per random seek added to the one measured for a drive mounted internally and STR limited to around 25...35 MB/s, depending on the system configuration. And like you mentioned, the slight differences probably doesn't even matter, so why benchmark? Removable storage is mostly used for transferring big files and the access pattern is usually quite sequential. Noticing USB interface limits sequential transfers to a constant that doesn't depend on the drive itself, benchmarking different drives in one enclosure isn't worth the effort.

Benchmarking different enclosures might be more useful as there are actually differences. But there is only a handful of IDE-USB and IDE-FW bridge chip suppliers. Performance isn't probably the most important criteria. It's more important to have a bug free chip that doesn't cause data corruption, drive lock-ups, etc.

Three most important features of an enclosure are probably:

- cooling characteristics

- the chip

- appeareance.

Three most important features of a HDD to be used in external, fanless enclosure:

- power consumption

- power consumption

- power consumption. :)

Share this post


Link to post
Share on other sites

Totally off-topic but:

T-Minus10 is member No.: 4,936 but has only 2 posts (the ones in this topic), joined 13-November 02. I, for example, am member No.: 36,473.

Has T-Minus10 been idle that long? I have a recollection he has posted something here on SR before... I just wonder why the counter shows "2".

Edited by whiic

Share this post


Link to post
Share on other sites
Proven by whom, exactly? And how?

Well, inferred from the reliability database. Any better ideas?

I don't see how this benefits anyone. You want drives in external enclosures looked at too? Is that right? So now we have enclosure chipsets & mainboard chipsets to factor in? Is this what you're asking?

It's not like SR's never reviews controllers before. Drives in enclosures aren't so much of interest to me as much as the specific performance differences between different controller chipsets. However, point taken, perhaps this is too broad.

And cost per gigabyte? For what market? Albania? Ireland? Tanzania? Can't you take a look at a disk at your local shop & figure out what it costs per gigabyte, then compare it to what ever other disk you're interested in?

SR, when providing a review, doesn't do any magic that any of us couldn't do on our own. If you wanted to, you could go borrow any of the drives reviewed here (those that are released at least), test it to your heart's content, and return it when you were satisfied. Effectively, by providing me with a review, SR is helping me out and saving me time.

Although I layed down some guidelines for how a new category could be graded, more importantly I presented a new hard drive usage completely neglected at SR. Perhaps when it comes to hard drives used as semi-removable media, we can treat all drives equally and pick the cheapest one. My proposal, is to have SR, with its reliability survey and hard drive wherewithal to help, first define which features I might look for in a drive to be used a medea, and then with those guidelines, narrow down which drives fulfill those features the best.

But how do we know that a "new drive superior to old leader" is "worse than the newest leader (not yet reviewed)"? Even Eugene isn't THAT allmighty, AFAIK.

Let me make myself clear: Whenever a leader is retired on the leaderboard to become a "past leader," the board makes it seem like the top drives are the drives on the leaderboard in order of release date.

An example today: The first past leader in the desktop/single user category is the Maxtor MaXLine III. The Seagate Barracuda 7200.9 is arguably a better series than the MaXLine III, yet it gets no mention.

Has T-Minus10 been idle that long? I have a recollection he has posted something here on SR before... I just wonder why the counter shows "2".

I registered for the reliability survey years ago. '2' refers to the first two posts in this thread -- or -- did. It should say '3' now, I think.

You typed a thousand words to ask for cost-per-gig & external-enclosure numbers? Are you mad?

That's not all I asked for, but mad? perhaps.

Edited by T-Minus10

Share this post


Link to post
Share on other sites
I registered for the reliability survey years ago. '2' refers to the first two posts in this thread -- or -- did. It should say '3' now, I think.
OK. Then welcome to the forums. I must have remembered incorrectly that I had seen your nick previously on the forums. Must have been some kind of a déja vu experience - a short circuit in my brain, that is. :)
Let me make myself clear: Whenever a leader is retired on the leaderboard to become a "past leader," the board makes it seem like the top drives are the drives on the leaderboard in order of release date.
Now that you mention it, it does create a possibility of such misinterpretation, but the leaderboard history is probably just kept for "nice to know" purposes. It trivial knowledge.
An example today: The first past leader in the desktop/single user category is the Maxtor MaXLine III. The Seagate Barracuda 7200.9 is arguably a better series than the MaXLine III, yet it gets no mention.
7200.9 is better in normal desktop performance but MaXLine III still beats it in server performance. Because 7200rpm drives are not typically in any but the most value-oriented servers, it's kinda irrelevant, and in most common applications 7200.9 beats MaXLine III. It might also be due to the larger capacity of Seagate as the difference isn't that great.

http://www.storagereview.com/php/benchmark..._3=265&devCnt=4

MaXLine III has superior NCQ implementation over 7200.9's.

Well, inferred from the reliability database. Any better ideas?
While Reliability Survey is much more reliable source of information than rumours on this forum, it's still far from accurate. Even with a couple thousand participants, there's no way near enough information. Also these participants are not randomly selected but they tend to be enthusiasts.

Normal (non-enthusiastic) people tend to forget the presence of a HDD until it fails, so they don't include drives into the survey before one dies.

An example: Raptors are solely used by enthusiasts. They have a high reliability percentile, probably because not only dead drives get reported by the ones alive as well.

Compare the users of Raptors to the users of 80GB DiamondMax Plus 9. Any difference? Would you assume that users of DM+9 report their drives as "alive and kicking"? I sure hope they did, but I don't trust that's going to happen.

While Reliability Survey is one of the best sources of information available to everyone (providing one is registered and participates in it), it is still far from accurate.

When choosing a drive to be mounted in external enclosure, I'd choose a drive that uses less power (and idle consumption being more important than active consumption, considering the typical way most people use their external drives). Temperature is still an enemy for reliability. Shock tolerance is another feature worth checking. Most manufacturers rate it at 350Gs, but I'd probably prefer load/unload technology (only used by Hitachi in desktop sized drives). There is two drives currently available on the market I'd consider for external use: Seagate 7200.9 160GB and Hitachi T7K250 250GB. If 250GB isn't enough, there isn't an abvious winner anymore. Probably 320GB WD is an option. Seagate NL35 seems to have a low power consumption but I wonder if it's sample variance as it should be indentical to 7200.8 400GB. WD is probably safer (and much cheaper choice). Other 400GB and 500GB are all around 8.5 watts when idle. http://www.storagereview.com/php/benchmark..._5=295&devCnt=6

Share this post


Link to post
Share on other sites

I disagree with your opinion on the leaderboard. If you are choosing what drive to buy based on its "past leader" status, then you are misusing the leaderboard. It shows the fastest drive in each category at this time. The past leaders thing is simply giving props to drives that once held that position, its not saying they still perform well. Heck, there are some drives on there from 2002. Do you know how many drives would perform better than that one from 2002? Dozens! Should we add them all to the leaderboard? No.

Past leaders isn't anything other than an honorable mention. For what you're looking for, the performance database is the correct tool.

Share this post


Link to post
Share on other sites

The reliability database is a useful tool, but it's still questionable whether it should be the sole considseration in a purchasing decision. Half the drives in there aren't available now, but still contribute to the statistics. You could compare drives that are currently available, and rank them against each other, but I'm not sure that I'd feel confident to put it in a "Most reliable drive" category on the leaderboard.

Regarding past leaders, as hddmaster just mentioned, if you want to see current drives ranked against each other, the performance database is where you should look. If a particular drive comes bvery close to the current leader, it will often get an honourable mention in the leaderboard, but the point of the leaderboard is that it's a quick and easy reference, and it's not intended as a tool for finding the second and third best drives of the time.

Performance in an enclosure is almost irrelevant. It will be dictated by the enclosure, and then by the existing performance metrics of the drive, which are best measured with the drive directly connected to the controller. The main important difference is the increased weighting of power consumption in the decision, which you can already compare in the performance database.

While current enclosures differ, I hope that external SATA will level the playing field considerably, making heat dissipation the main distinguishing factor between enclosures. You can still search and ask in the forums for experiences with a particular enclosure or chipset in the meantime.

What this comes to is an argument for doing practically nothing about your suggestions, but don't be discouraged, this is definitely a topic worth discussing, and it's good to see you taking an interest and making a contribution! Besides, we're generally an open minded lot, and you may yet convince us!

Share this post


Link to post
Share on other sites

The reliability database is not statistically significant in its measurement of harddisk reliability. It is not random data, nor is it of sufficient sample size. Actually trying to use this to justify purchasing decisions is not something I would do except perhaps as a very, very small component.

SR getting into testing shock and temperature tolerance testing, particularly for external drives, is a whole 'nother fount of possible trouble that I don't think SR is properly equipped to handle. Even fairly large corporations tend to outsource shock and vibration testing to 3rd party companies unless they specifically need to invest in such equipment. Consequently I do not believe this is a worthwhile area for SR to spend assets.

Share this post


Link to post
Share on other sites

While what you are asking for is challenging, I do agree with the spirit of trying to expand beyond performance. I have been visiting this site for many years (since well before the winter-2001 site loss) and nowadays, I pretty much only visit to check out some I/O numbers for a particular drive. Reliability, satisfaction, etc. are difficult things to measure, and that's where people have to get creative. Industries have been measuring these things quite quantitatively for decades so I don't understand why the notion seems like rocket science to the many experts in the peanut gallery. Moreover, all sorts of sites from epinions.com to makeupalley.com (my girlfriend lives on the latter) to amazon.com to audioreview.com find ways to collect "noisy" commentary and present it in a fairly quick-to-absorb format.

No reason we can't do the same. Things shouldn't reach 75GXP-like status before someone has the guts to say, "this brand/model as a reputation for poor reliability."

My favorite "review" website - dpreview.com. Why? The reviewer(s) have the guts to simply say, "I like and recommend this camera for these sorts of uses" or "I don't recommend this camera." Very subjective assessments, based on objective analysis but still open to flaming, and yet they take stands anyway. Would love to see some subjective recommendations posted on this site that do the same....

Share this post


Link to post
Share on other sites

I agree with the advocates of change.

I have used this site a very long time, but find myself

seeking other reviews to make my final decisions.

Share this post


Link to post
Share on other sites
No reason we can't do the same.
True.

As someone in the manufacturing area, though, I don't think a statistically reliable reliability survey is within SR's means or any major hardware review website. Shock/vibration testing is more realistic, but that is also hugely expensive. The testing budget for one product line for one of our customers is over $500,000 a year, just for shock/vibration/thermal, and they are just able to get conclusive reliability data within that budget.

Share this post


Link to post
Share on other sites

I haven't visited SR in a while because -- I haven't bought any cutting edge hard drives since I realized it was a waste of money. However, returning and seeing a budget spot on the leaderboard makes me want to say:

i rock.

:) I have no soul.

(completely useless bump)

Share this post


Link to post
Share on other sites

No reason we can't do the same.

True.

As someone in the manufacturing area, though, I don't think a statistically reliable reliability survey is within SR's means or any major hardware review website. Shock/vibration testing is more realistic, but that is also hugely expensive. The testing budget for one product line for one of our customers is over $500,000 a year, just for shock/vibration/thermal, and they are just able to get conclusive reliability data within that budget.

Agreed about the reliability testing. It takes hundreds of drives and hundreds of hours to get meaningful results. And then you need engineers to accurately diagnose the failures to see if they were the fault of the drive, the test-bed hardware, the test software, the user, etc.

But, maybe SR could add something beyond performance, such as a compatibility test? Get maybe 20 motherboards and/or drive controllers with different chipsets, and see if each drive+controller combination passes a basic functional test of an hour or so.

Good compatibility might be indicative of how hard the HDD manufacturer works to iron out problems...

Share this post


Link to post
Share on other sites

Feasible, if time consuming. Also, firmware/BIOS versions come into it, though testing with the latest firmware available for the controller/chipset would seem sensible.

Do you really think that compatability issues are that common, though? Is it likely that a given drive would prove incompatible in one or more chipsets, in a way that reveals itself within a short period of testing?

It wouldn't be worth the extra work unless there was a significant number of incompatabilities. It would be difficult to determine whether the problems were the fault of the drive or the chipset failing to adhere to standards. Still, if it happens, no-one else but SR is likely to do it.

I honestly don't know whether it's worth it or not. It's a lot of extra work, for a test that I'd expect most drives to pass easily, but if I'm wrong about that, and incompatibilities are actually rife, then that should be exposed.

Opinions, anyone?

Share this post


Link to post
Share on other sites
I disagree with your opinion on the leaderboard. If you are choosing what drive to buy based on its "past leader" status, then you are misusing the leaderboard. It shows the fastest drive in each category at this time. The past leaders thing is simply giving props to drives that once held that position, its not saying they still perform well. Heck, there are some drives on there from 2002. Do you know how many drives would perform better than that one from 2002? Dozens! Should we add them all to the leaderboard? No.

Past leaders isn't anything other than an honorable mention. For what you're looking for, the performance database is the correct tool.

The leaderboard is still showing the WD4000YR as the fastest "Light-Duty Server / Nearline" HDD and I know the WD5000YS it...

So what use is the leaderboard if it is not updated regularly IMMLO?

Share this post


Link to post
Share on other sites
Feasible, if time consuming. Also, firmware/BIOS versions come into it, though testing with the latest firmware available for the controller/chipset would seem sensible.

Do you really think that compatability issues are that common, though? Is it likely that a given drive would prove incompatible in one or more chipsets, in a way that reveals itself within a short period of testing?

It wouldn't be worth the extra work unless there was a significant number of incompatabilities. It would be difficult to determine whether the problems were the fault of the drive or the chipset failing to adhere to standards. Still, if it happens, no-one else but SR is likely to do it.

I honestly don't know whether it's worth it or not. It's a lot of extra work, for a test that I'd expect most drives to pass easily, but if I'm wrong about that, and incompatibilities are actually rife, then that should be exposed.

Opinions, anyone?

Maybe you're right, most manufacturers should be able to pass a compatibility test. Especially since SATA is not new anymore, most of the kinks should be worked out. A few years ago it might have been a better measure of the strength of engineering teams.

Just trying to think of some quality measures beyond performance that SR could do.

Perhaps some fault injection?

- Hack open a SATA cable and short the differential pins while the drive is operating. While doing this, the host could be running a read/write/compare test. That way SR could test the robustness of CRC checking. Might need to make some home-brew hardware to make this deterministic.

- Using write long and read commands, test the drive to see how many bytes/sector the drive can correct.

- Test performance vs. vibration (varying frequency and amplitude) on a single drive to test mechanical and servo systems. SR would need to invest in some test hardware, for example http://www.labworks-inc.com/ (not sure on pricing for that)

Having a large sample size is maybe not so critical for fault injection, since you are testing the design of error recovery mechanisms, rather than evaluating overall reliability of the system.

Share this post


Link to post
Share on other sites

I demand that Eugene also physically open each HDD and check the platters for imperfections under a electron microscope... I demand this immediate implementation! :-D (In addition, it would be preferred if Eugene wore a pink ballerina tutu while performing future HDD tests... LOL)

Share this post


Link to post
Share on other sites
I demand that Eugene also physically open each HDD and check the platters for imperfections under a electron microscope... I demand this immediate implementation! :-D (In addition, it would be preferred if Eugene wore a pink ballerina tutu while performing future HDD tests... LOL)

Performance testing can be done by a trained monkey. Why is it unreasonable to ask for more?

Share this post


Link to post
Share on other sites
That is review enough for me...
You miss the point... it's in the database, but the review isn't up yet. SR policy seems to be to not update the Leaderboard until the review has been offically completed and released. At least that's my personal observation.

Patience, grasshopper! :P

Hack open a SATA cable and short the differential pins
Given the data rate, I don't think this would be really practical, unless you can control the short at a rate measured in megahertz...
Test performance vs. vibration (varying frequency and amplitude) on a single drive to test mechanical and servo systems.
Sounds like a pretty good way to risk killing a drive, and if SR is getting demos from manufacturers, returning them dead might not be the best idea ;) . SR's goal isn't to test drive durability-- at least not from what I can see. Single sample data doesn't tell you much more than just going from the manufacturer specifications on vibration tolerance.

I wonder what testing error recovery would tell you, at least through those testing means. Error recovery in a large part is defined by the ATA working group, you'd really just want to test the internal error recovery mechanism... I wonder how you'd isolate that...

Why is it unreasonable to ask for more?
Time and benefits. IIRC, testing cycles for a lot of hardware at Tech Report is 20-24 hours per configuration being tested. Say they test five things, that's 100-120 hours for a single published review. Adding another condition such as vibration (rather than just static) would add considerably to testing time-- doubling it in many situations.

I understand that just testing error recovery is a much more limited application than trying to generate any sort of reliability data, but I'm just wondering what the actual benefit is and how useful it is for others.

Share this post


Link to post
Share on other sites

I demand that Eugene also physically open each HDD and check the platters for imperfections under a electron microscope... I demand this immediate implementation! :-D (In addition, it would be preferred if Eugene wore a pink ballerina tutu while performing future HDD tests... LOL)

Performance testing can be done by a trained monkey. Why is it unreasonable to ask for more?

It's not unreasonable at all to ask for more...I just want Eugene wearing a pink ballerina tutu when conducting HDD tests.

I really think what is already here at SR is excellent...I just wished it were maintained a bit better...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0