Futuremark & NVIDIA...

[Julio Franco]

I just refused to pay more attention to the whole drivers 'cheating' thing, that's why you didn't hear from us on the matter yesterday although we were also contacted by Futuremark.

Like I stated since my first post, cheating or not, I can't blame NVIDIA for cutting corners, if a lesson should be learned would be not to rely completely on synthetic benchmarks but rather on real world applications/games... here's something nice to check out (thanks Carter).

 

[Per Hansson]

It's really a shame that a product that already was so flawed had to prove this via such a low move by Futuremark. This just makes the point so much more clear why a benchmark program is inappropriate for judging graphicscard performance. I.e. a third party benchmark program receives income due to it's participants. An actual computer game receives income if it sells good or not. It's all about $$$

They first say Nvidia cheats, and at that giving out undisputable evidence to back that up. Well; now they believe everything is all of a sudden ok... Did I just hear someone saying "bend over"?

 

[Julio Franco]

I believe this happens all the time up to a point that I've got used to it... I mean, if you didn't know it was all about the money then you haven't been paying enough attention.

In the end, if you were fooled too bad for you, I still believe benchmarks work and will continue working as long as you get the results from a trusted source... and I must say there are a few 'sensationalistic' sources out there, at least for some things.

 

[Per Hansson]

Originally posted by TS | Julio
if you didn't know it was all about the money then you haven't been paying enough attention.

Of course it's all about money, though with the "beta program" I think Futuremark took it one step too far...
"If you pay us money we will let you optimize your product for our benchmark, if not; screw you" (And mind you we are talking lots of money here...)

It's not just that either, because having a company spend time and money on optimizing their product for a benchmark when that time and money could have been spent on optimizing for a game instead is just plain wrong IMHO... (Nvidia has even said that themselves)

 

[Per Hansson]

Originally posted by TS | Julio
I believe this happens all the time up to a point that I've got used to it...

Program optimizations happen all the time yes, however whenever the final picture does not look as the developer intended the image quality has been reduced and I consider that a "driver cheat"
Any optimization is fine for me, the graphicscard manufacturers may change the programs code as much as they like as long as the final picture looks as the developer intended.

 

[Per Hansson]

Originally posted by TS | Julio
In the end, if you were fooled too bad for you, I still believe benchmarks work and will continue working as long as you get the results from a trusted source... and I must say there are a few 'sensationalistic' sources out there, at least for some things.

For the averge user there was no way of telling what was happening, for that you needed the beta programs developer tool, which costs 5000$
I do not quite think I follow you on the trusted source thing, see above...
Yes, certain sites do whatever they can to attract visitors.

 

[Phantasm66]

I am sure that this sort of thing goes on A LOT MORE than any of us really hear about.

I've never been enough into your area of computing, Per, to ever be able to form an informed opinion of my own as regards the usefulness or legitimacy of benchmarks, so I couldn't really comment with too much gusto myself, as I don't really have an informed opinion. I think you do have an informed opinion.

I certainly don't bother benchmarking my equipment, and I have been a computer technician for a while. I guess maybe once in a blue moon, but it means so little to me. Is there an established standard for benchmarking 3D because every time I download and install one of these pieces of software, it never seems that there is.

I am much more concerned about how a piece of equipment gets on actually fitted in my machine. That includes stability, compatibility and many other issues as well as performance, which overall go towards making a good product.

 

[Julio Franco]

Originally posted by Per Hansson
I do not quite think I follow you on the trusted source thing, see above...

Forgot to add, either you can trust one source of your choice or test yourself if you have a way to do so, fortunately we (people involved in the medium) generally do .

 

[TS | Thomas]

Yeah, way to back down Futuremark. So replacing "your" shaders with ones that NVIDIA decided to use is an "optimization" now. Hmm ok. Well, glad I try to avoid using 3D Mark now.