Davin

How StorageReview.com Tests Hard Drives

Recommended Posts

Last November, StorageReview.com debuted its third-generation testbed. This comprehensive overhaul brought changes to the hardware, operating systems, benchmarks, and methodology behind SR. Consistent test platforms featuring a minimum of change have been the hallmark of SR over the last three years. This is as close as it gets to required reading for both site regulars as well as the occasional visitor. How are drives judged? What motives exist behind the methodology revision? What do the benchmarks measure? All this and more is answered in this sweeping article. You can't afford to miss this one!

http://www.storagereview.com/articles/2001...aissance_1.html

Share this post


Link to post
Share on other sites

I've just read this again (to see if it offered any insight into the "correct" way to work out seek time from access time). One thing I noticed was the apparent importance of burst rate, due to the likelyhood of cache hits:

buffer-to-host transfer rates, otherwise known as burst transfer rates, are significant.

In the light of this, it would seem possible the Maxtor ATA drives may benefit from using UDMA-133 as opposed to UDMA-100. Since TB3 only uses UDMA-100 it would seem that these drives are not being allowed to work at their fullest performance.

Therefore it seems to me that to make fair comparisons, a UDMA-133 compatible controller should be used. At the least, it would seem a good idea to test a current Maxtor drive (eg. DM+9) again using a UDMA-133 controller, testing both in 133 and 100 modes and seeing if there is a significant difference in any tests. If not... then the current tests are valid, but if the difference is significant then I think the published tests should reflect this.

I'm not saying that using UDMA-133 will make much difference; I really don't know.

I'm using UDMA-66 for drive :wink:

Share this post


Link to post
Share on other sites
Guest Eugene
Therefore it seems to me that to make fair comparisons, a UDMA-133 compatible controller should be used. At the least, it would seem a good idea to test a current Maxtor drive (eg. DM+9) again using a UDMA-133 controller, testing both in 133 and 100 modes and seeing if there is a significant difference in any tests. If not... then the current tests are valid, but if the difference is significant then I think the published tests should reflect this.

The Promise Ultra133 was used in out DM+9 tests as well as the older 160 DB 540DX review.

Regards,

Eugene

Share this post


Link to post
Share on other sites

I've noticed a "mistake" in your test configurations.

Every time you test a new drive >147GB, you're using an external controller which forbids you from doing write tests and it's limited by the PCI max troughtput of 80MB/s~.

What you didn't noticed, is that upgrading the motherboard firmware of the Intel D850 series enables 48bit LBA(along with several system changes that are noted in the release notes) hence, it would allow you to go back to test with the onboard controller, which is much powerfull than the promise external.

The firmware upgrade doesn't change the storage performance, so it wouldn't invalidate any of your previous tests with the 82801BA

think about it, if the new firmware(P15 i think) isn't satisfactory(because maybe it changed the previous results too much) then you can fall back to the obsolete BIOS you currently use

BTW:I have tried to email both of you about this, but every time my mail is returned like "spam filtered" or something like that......

Share this post


Link to post
Share on other sites

I am new to the site but I did not see any information on SATA controllers used. You also refer to Athalon performance. Where would this data be found?

Thanks and congratulations on your dedicated effort to provide real world performance results.

westcott

Share this post


Link to post
Share on other sites

I've searched, including scouring the Testbed3 description pages, but I can't seem to find it:

Does StorageReview test with AAM consistently disabled, or does it use whatever mode to which the drives are set by default upon acquisition?

Share this post


Link to post
Share on other sites
Guest Eugene

AAM is tested disabled. There has been the rare case where "disabled" was actually the equivalent to a setting less than "enabled @ 254" but it should be assumed that we take all pains to ensure the drive is in optimum seek mode.

Share this post


Link to post
Share on other sites

Users have growing concerns about PC noise pollution. Do you have any intentions to increase your Noise testing to cover active noise? I would view this as highly beneficial if it can be done in a manner than produces consistent and reliable results.

Love,

Fairy

Share this post


Link to post
Share on other sites
Users have growing concerns about PC noise pollution.  Do you have any intentions to increase your Noise testing to cover active noise?  I would view this as highly beneficial if it can be done in a manner than produces consistent and reliable results.

192730[/snapback]

You have visited SilentPCReview, haven't You? :)

There are few issues when doing noise measurements.

- You need a very good (=sensitive) SLM (sound level meter). SR measures the noise from a distance of 18 mm, which is done because their meter is a basic one (which most likely cannot register lower than 25-30 dBA). The meter used at SPCR is actually borrowed from University of British Columbia, and is rather expensive.

- The environment & rest of the computer need to be very quiet. If You have read the articles here, You've seen comments like "While one may argue that differences below, say, 43 dB/A or so get lost in the noise generated by other parts of the system (or even general ambient noise, for that matter)" (from SP1614C review) basically mean that the reviewer's system was masking the noise created by the drive.

Improving (or even changing) the testing methods would also mean that the comparison between existing results and new results would be difficult, if not impossible.

Well, keeping in mind that Eugene asked about suitable motherboards for Testbed IV a while ago, testing methologies could also be changed in the process...

Cheers,

Jan

Share this post


Link to post
Share on other sites
You have visited SilentPCReview, haven't You? :)

193176[/snapback]

I have now :-) However when I posted this it was prior to my visit and just based on personal concerns. My own PC can't be considered silent, however I have been making purchases for some time with a mixture of performance and noise goals. Noise reduction didn't figure in my HDD purchases in the past and as a consequence I suspect that my two HDDs are the noisiest components in my PC.

I was unaware of the equipment issues prior to you posting however I can now see that significant investment would be needed in order to improve this situation.

Finally, I have pondered before whether it is better to have old yet consistent testing methods, or newer methods but loose continuity with the past. At times it will be necessary to change methods however clearly these changes are best done only when the benefits are great and probably best done at the start of a new generation of drives.

Thanks for your kind reply,

Fairy

Share this post


Link to post
Share on other sites
Guest Eugene
Well, keeping in mind that Eugene asked about suitable motherboards for Testbed IV a while ago, testing methologies could also be changed in the process...

It is indeed true that Testbed3 has been retired for nearly a month and that testing under TB4 is well underway. Expect to hear more very soon...

Share this post


Link to post
Share on other sites
Well, keeping in mind that Eugene asked about suitable motherboards for Testbed IV a while ago, testing methologies could also be changed in the process...

It is indeed true that Testbed3 has been retired for nearly a month and that testing under TB4 is well underway. Expect to hear more very soon...

193280[/snapback]

How about a sneak preview? What motherboard did you end up finding?

Share this post


Link to post
Share on other sites

In relation to testbed3 and possible changes to testing methologies for testbed4.

* I think the ServerMarks should have the IO coefficients to add to 1. Currently it creates a score that is not IO/sec. This would make it that coefficients would represent percentage of time.

For example of File server.

1=0.339423888

4=0.277149167

16=0.205666264

64=0.177760681

Total = 1.00

The other idea is to have a workload in MB per IO depth setting to give a better representation of performance. The final score could still be represented as IO/s or MB/s or as time (for the specified MB workload).

* You may like to create your own workload for fileserver and webserver usage similar to your desktop drivemarks.

* Other things to note is to focus on significant usage that involves restrictions that limit performance. Some short/rare tasks can be ignore but there are other frequent/time critical tasks that should be represented in the suite of tests.

I assume you are updating and expanding your drivemark usage data and tests.

Any possibility of adding more tests based on industry/job/app/task for a finer grain look at performance potential in real usage ? Eg. data manipulation/analysis/mining using SAS, take the IO trace and use it to make your own model.

* Look at how some of the tests can be used to evaluate other system aspects such as use of RAID, OS/mem cache and defragers.

* You must test/admit that different capacity drives in the same family can have different performance due to platter design and actuator parameters. The larger SCSI drives often have slower seeks but are overall better with given data stored on smaller drive.

Share this post


Link to post
Share on other sites
You have visited SilentPCReview, haven't You? :)

193176[/snapback]

[...]

I was unaware of the equipment issues prior to you posting however I can now see that significant investment would be needed in order to improve this situation.

193264[/snapback]

I haven't suggested this over at SPCR yet (I'll point them to this post/thread), but how about SR sending drives to the SPCR labs for acoustic testing? So we'd have the best available performance data combined with the best available acoustic data, on the widest possible range of hard drives.

SR would benefit from SPCR's more sensitive and calibrated testing and recording facilities, and would have acoustic information in the database of a comparably high quality to the performance measurements SR makes. It also saves SR having to invest in expensive acoustic measurement facilities of its own - the savings would more than pay for the shipping costs.

SPCR, on the other hand, would benefit from being able to test the acoustics on a much wider selection of drives than they normally encounter, and their readers would be very interested in the results. Hopefully they will feel this worth the testing time involved.

I was about to start a thread in the Computing forum, asking how people thought SR should test acoustics, but after re-reading this thread and the relevant part of the article it stems from (see here), I thought it worth asking if, rather than trying to emulate the quality of SPCR's equipment and methodology, SR could actually use it directly instead.

Opinions, anyone (especially Eugene, Davin and the SPCR crew)? I can see this could be unpopular with both sites, but it would be great if we could make this work.

Share this post


Link to post
Share on other sites

Thread started at SPCR on same topic. Running two threads on the same topic can get confusing, but there will always be people who won't visit or post on another site's forums.

Opinions welcome from all (though it's up to Eugene, MikeC and the other admins to make the final decision, of course).

Share this post


Link to post
Share on other sites

Can you please release more information about your testing methodologies for Testbed4 and a few beta results before making concrete. That way we can peer-review and suggest changes/clarification before getting in too deep.

Have you fixed the problem of ServerMarks etc. not giving a true IOps result as it has in the past. You had created a rating (no units, the result is not IOps) and not a true statistically/mathematically correct measure.

You should probably start a new thread for Testbed4 (with a link to the older ones if someone wants to find the earlier content).

Can you please answer, or are you too busy making the same mistakes with testbed4 ?

Share this post


Link to post
Share on other sites

Hi,

I tried to track down Intel's Ipeak storage benchmark without luck. It appears to

have been discontinued (?), and mirrors of the evaluation version have also

evaporated.

Can you say what you've replaced it with in your upcoming version 4 suite?

(or can you suggest where to obtain (not necessarily free) the Ipeak storage benchmark, if it's still available)?

Thanks,

Dave Holt

Share this post


Link to post
Share on other sites
Hi,

I tried to track down Intel's Ipeak storage benchmark without luck.  It appears to

have been discontinued (?), and mirrors of the evaluation version have also

evaporated.

Can you say what you've replaced it with in your upcoming version 4 suite?

(or can you suggest where to obtain (not necessarily free) the Ipeak storage benchmark, if it's still available)?

Thanks,

Dave Holt

200605[/snapback]

You can download the demo here:

http://arethusa.tweakers.net/~femme/spt/spt-v30.zip

Share this post


Link to post
Share on other sites

Question about the Testbed 3 Legacy results. WinBench 99 reports results in "Thousands Bytes/sec". StorageReview reports in "MB/s". I know the storage industry takes some liberty in defining a MB as either 1,000,000 bytes or 1,048,576 bytes. Which "definition" of MB is StorageReview using when converting from WinBench scores into the MB score?

Thank you,

Mark Nelson

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now