Benchmarking: SPC-1 & Nürburgring

By on 03/07/2015.

Recently, NetApp released all-new SPC-1 results for our latest all-flash platform, the EF560, with amazing $/IOPS numbers, blazing fast speeds, and some of the lowest latency ever benchmarked. This was met with a lot of what I call “Yea, but…” reactions from pundits and citizen journalists, citing reasons why it wasn’t the best, it was missing XYZ feature, or any other myriad of reasons to shun such an amazing accomplishment. This is not something that is special or unique to the storage industry.

Also recently, Honda announced the release of the all-new Civic Type-R hatchback. This is the first to come turbocharged from the factory, and it sounds like we’re going to get them in the US!  And the peasants rejoiced!

When I shared some of the articles around the web within my circles, it was met with similar reactions.Nürburgring

“Yea but it’s front-wheel drive…”
“Ew, that thing is ugly…”
“That is for ricers…”

When the light bulb went off for me to write this article, it wasn’t about the EF560 or the new Honda Civic Type-R.  I was watching the recent posting of the Honda lapping the Nürburgring in 7 minutes and 50 seconds.

7:50.63, to be completely accurate.

That’s faster than what some people consider to be the fanciest fastest most expensive cars in the world…

Lamborghini Murcielago
Porsche 997 Carrera S
BMW M4 (new for 2015)

…to name a few.  It is even only a mere 2 seconds slower than the fully built BMW M3 GTS touring car.

Yet people shun it for reasons that aren’t favorable to them.

Where are you going with this, Nick?

Benchmarks exist for a reason.  We need a standardized measuring stick for which to measure things against. For the automotive world, the lap time at the Nürburgring has become this benchmark for which to strive. For the storage sector of the IT industry, that benchmark is SPC-1.

People will always pick it apart, just like they do in the YouTube comments of that lap video, but it doesn’t take anything away from the raw performance on display.

“SPC-1 is meaningless because companies build their testing rigs with ridiculous unrealistic hardware to game the results.”

“That Civic was wearing soft compound slicks so that time isn’t right.”

Benchmarking is a necessity and a way of life in many industries, not just storage.  Just because you don’t believe in the results and the showcase of what a rig is truly capable of, doesn’t mean it isn’t a true result on a standardized benchmark. Having gone through several of the rigorous SPC-1 benchmarking processes now, I can attest to the attention-to-detail those guys provide, and in-depth testing they perform. I also hear there is a new SPC-1 2014 test that is coming soon to update from the 2008 test we’ve all been running the past few years, and I cannot wait to see how well we all do!

Why do we have to live in a society where we have to troll and pick everything apart as opposed to celebrating accomplishments that are for the greater good of the industry?

A genuine hat-tip to Hitachi for being the first to surmount the 2M IOPS barrier. That’s impressive. Everybody loves a dragster. And to the EF-series engineering team, keep doing what you do.  You’re accomplishing some amazing results around low latency while keeping costs low!

It’s a shame EMC won’t put VMAX up, or Pure Storage won’t put up FlashArray.  It would be nice to have everyone visible on what I consider to be the one true universal benchmark of this industry…

The Nürburgring for storage systems.

Let the race to 0ms latency commence!

Nick Howell
Tech Evangelist at DatacenterDude, LLC
One of the top technology evangelists in the IT Industry, Nick Howell has spent 15 years driving multi-million dollar solutions across multiple Fortune 500 companies and implementing new systems from an engineering, architectural, and consultant capacity across many industry verticals including finance, defense, retail & more.

2 Comments

  • Kevin D.

    Great article!

  • Hans De Leenheer

    The danger of car analogies and datacenter technologies is that you can take them as far as you want to the point that you take it out of context. In this case the car analogy you use makes perfect sense to why those dragster benchmarks are ridiculous; no-one in their right mind would use Nürburgring timelaps as a reason to buy a car. That is practically a completely empty car stripped from all it’s possibilities to be the prinses of the ball for just one night.

    Speaking of EMC & Pure and their absence on this carnival; EMC acknowledged how F-ed up these benchmarks are by breaking one with a ratio of 2:1 to the previous leader two years ago. They did need 4VNX-s with over 450 SSDs for it. A couple of million $ down the drain for something FusionIO did in a single box just 2 months later. And PureStorage shares ACTUAL aggregated analytics of ALL their arrays in the field. What do you think I would want to see as a customer?