![[H]ardOCP](/images/blank.gif){kind=small}
- Date:
- Thursday , July 20, 2006
- Author:
- Kyle Bennett
- Google +1
Firing Squid Wastes their Ink
Our reply to an attack on real world testing. Some will have you believe that a canned benchmark or gaming timedemo is a crystal ball into your gaming future.
We have jokingly referred to the Firing Squad as the "Firing Squid" for years now. You know, the website that will post pages and pages of canned benchmarks without any analysis at all? Here is an example of them going on for 7 pages without any interpretation for the readers. They do however follow up with a 6 sentence conclusion and an award. Wow, more pages of canned benchmarks than conclusion sentences? No wonder they are threatened by HardOCP showing you what actually happens during real gameplay and describing our experience to you.
Anyway, today they have a nice rant about HardOCP that I encourage you to read. Because as I read it, it became clear to me how many people do not understand our philosophies surrounding real world gameplay evaluation. Here are a couple of sentences that made me realize how misunderstood some of what we do actually is.
What’s a “real-world” benchmark anyway?
According to [H]ardOCP, "real-world" CPU performance testing means using a variety of AA, AF, resolution and in-game settings to try to average 40fps... or 35fps, or 73fps.
We have moved away from “benchmarks” fully, real-world or not. Firing squad fails to even understand that we are not "benchmarking." We are using real applications in our testing, both in gaming and other desktop programs. Firing Squad refers to “real world CPU performance testing” and what it means to HardOCP and how we try to do it without ever asking us about our methods or philosophies. Certainly the self proclaimed “hardware journalists” would not publish an article without contacting the persons they are writing about with specific questions would they? Their assumptions about some of our testing and thoughts are ludicrous at best. At least they should get their facts and nomenclature correct.
Testing at HardOCP has evolved over the years. We were the first to bear the expense of real world gameplay for video card evaluations years ago. Brent Justice, Senior Video Card Editor, and I were flying back from a meeting in Toronto with ATI when the conversation turned to what is HardOCP really trying to deliver to our readers. We looked at the canned benchmarks we were claiming to our readers to be a good evaluation tool and decided that the industry was moving beyond these simple tools. Were canned benchmarks really telling our readers what we thought they needed to know? No, they were not, so we decided to change the ways we were doing things.
I had much the same revelation in the passing months as it pertains to CPU testing. I saw Intel and other websites telling people just how great Core 2 was and how it gave these huge gaming performance increase numbers...in canned benchmarks. I wanted to know what Core 2 meant to many of our readers that already own high end Athlon 64 gaming machines. As our Core 2 Gaming Performance spells out, there is not much of a reason for many current gamers to upgrade. (We also did an article on focused on other real world desktop applications as well.)
So what exactly is HardOCP doing? We test using real gaming scenarios like you would find yourself in at home (when we are testing games) and give our readers feedback on our experiences with the hardware. We tell our readers what their expectation of their computer experience should be based on our own experiences. Firing Squad gives you pages of canned benchmarks (Although they are following suit on encoding) without any or very little analysis, then declares a winner.
Firing Squad also goes on to tell you only “the downsides to real-world testing.” Given they promised to discuss the “pros” of real-world benchmarking, they must've accidentally not posted the "pros of real world testing" page. Maybe it didn't fit with agenda of their editor? After much more thought and explanation is given, more than goes into their usual conclusion statement, they come to this statement.
With traditional benchmarks, you'll still be able to see its relative performance to other CPUs with no bottlenecks, while "real-world" benchmarks will uselessly indicate that all CPUs are created equal and there is no point trying to discern between them.
Obviously Firing Squad ignores statements that do not prop up their flimsy argument like this one in our “Intel Core 2 Gaming Performance” article conclusion.
If I had an older system and had to put my foot down and choose a system with the future in mind, I would probably lean toward the Intel Core 2 Duo E6700 platform for “future proofing” if Oblivion were any indication of future games. If you have a higher-end AMD Athlon 64 system platform right now though, there really isn’t any need to go scrambling to Intel Core 2 at this particular time for gaming. I’d wait it out and see what the future brings.
Another interesting point is that Firing Squad’s entire editorial fails to point out this real world testing article entitled Intel Core 2 Music, Images, & Movie Performance published on the same day as the gaming performance article. Using the exact same philosophies as our gaming article we showed the gains you could enjoy today using an Intel Core 2 processor. Here is a quote from the conclusion.
There is no doubt that when it comes to editing video, manipulating images, or encoding music, the Intel Core 2 Duo and Extreme processors at 2.66GHz and above currently enjoy a healthy performance advantage over AMD’s Athlon FX and Athlon 64 line of processors. Moreover, I would be remiss if I did not mention the Core 2 Duo and Extreme also do a great job of making the Pentium 4 look like a dinosaur.
Sounds to me that real world testing is working very well there too.
The Firing Squad author goes on to explain why canned benchmarks and time demos are great tools for gaming and not broken. He is somewhat right, in that they have their place, but I find their usefulness waning every day as the desktop computer diversifies and advances.
The author then cites my words and editorial as being “unfair to all other websites with Core 2 reviews.” And closes with, “If this is what hardware journalism has resorted to, we want no part of it.” Firing Squad’s Core 2 article covered 21 pages; much of it canned benchmark after canned benchmark with little or no analysis. Pages and pages of graphs that did nothing to tell you what you should expect when buying the hardware to play games on. How exactly is that journalism to begin with? That my friends is nothing more than a vehicle to show you 20 pages of advertisements.
The Bottom Line
Firing Squad is threatened by what they see going on around them with real world benchmarking catching on. There are more sites than just HardOCP doing real world evaluations and speaking out about it. Firing Squad have continually proven their hardware reviews are little more than pages full of canned demo frames per second graphs with little or no analysis. They survive by the canned benchmark. A well trained monkey can pull off that sort of “hardware journalism.” So why do they continue to test like this and now find it convenient to attack real world testing? That is simple to answer and it all comes down to money. Real world testing is very expensive and time consuming. Canned benchmarks cost nothing in comparison. Real world testing is starting to develop a wider fan base as more of you are seeing the true value in it. Canned benchmarking is showing its age and its value is waning as gaming and applications are changing and becoming more complicated. The more people that slip away from wanting to view their pages of canned benchmark graphs means less money for Firing Squad.
It all comes down to some very simple question. Do you want advice and feedback on computer hardware from people that actually use that hardware much like you are going to do at home, or do you want some “hardware journalist” that knows how to run a canned benchmark telling you what you need to buy?
Discussion
Please join this thread in our forums to discuss this topic.
You might also enjoy these past editorials on benchmarking that we have written.