ATi & NVIDIA on Futuremark Report

Status
Not open for further replies.

TS | Thomas

Posts: 1,318   +2
Regarding the Catalyst performance gains that were discovered in 3D Mark 2003 ATi has responded with the following:
The 1.9% performance gain comes from optimization of the 2 DX9 shaders (water & sky) in Game Test 4. We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically & functionally identical to Futuremark's & there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality & as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, & so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.

Meanwhile NVIDIA says the following regarding Futuremarks findings:
Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer.

Draw your own conclusions to these reponses.
 
I think ATi's reponse there is milking it just a bit there (Can't blame em though). Gotta love NVIDIA's "real applications developer" bit though.
 
Well when Nvidia complained to Futuremark about methodology, I believe they were told to get lost.

Kind of funny, but when you tell that to a company with 70% marketshare, something like this happens.

Given the initial response from Futuremark and the fact that major games these days ARE optimized, I've no sympathy for Futuremark. I'm not loving how Nvidia went about it, but I think this was a quite humourous reply. Don't want to talk about what your testing program does right or wrong? Fine, don't expect us to make your life easy.

People so easily forget the history and just expect Nvidia to quietly live with whatever someone wants to say about its products. Were I them this is the very least I would have done. I'd not expect Futuremark to be around very long unless it changes its tune.

Agree or not, we'll see whether I'm right.
 
funny how people and nvida try to downplay futuremarks importance, if it wasnt, why did they cheat on the test. Cause it sells there cards, but they dont want to pay futuremark so they can optimize there drivers. I just think nvidia should realize that the are doing things the wrong way.
 
the right thing would be not to rate the performance of hardware with just a little benchmark and assume lots of things depending of how much points it gets
 
Replacing shaders with your own version is not "optimizing".

When I get Doom 3, or any other game for that matter, I don't expect my Graphics card's Driver to replace anything in the game with something else.

NVIDIA didn't optimize anything, beyond REPLACING shaders with their own "similar" versions they also screwed up rendering in several places for artifical performance improvements, which anyone with the Developer version of 3D Mark could see.

NVIDIA are effectively fraudulently misrepresenting the performance of their cards with these Drivers. ATi got in similar trouble for early Radeon Drivers in Quake 3. Do you consider what they did to be optimisation also? Afer all it improved performance & only reduced visual quality.
 
First off, congratulations to FutureMark for their handling of this. While I'm more in the camp of gaming benchmarks over synthetic benchmarks, each have their own signifigance.

Secondly, has Nvidia has lost their collective minds in this matter? They cover up lie with even more lies, and accusations. They insinuate FutureMark and ATI are out to get them. I wonder what the other Beta members would say about this? In fact, I think nVidiais about the ONLY gpu maker that is NOT a Beta member. Also, if Futuremark is out to get them, (supposably since ATI is a Beta member) does that mean we must ignore EVERY OTHER BENCHMARK that shows how poorly the FX5800 and FX5900 handles shader routines? Virtually every single shader benchmark shows the same thing, that the Nvdia shaders are not performing nearly as well as the R9700/9800's.

nVidia also claimed the cost of the Beta membership as costing *hundreds of thousands* of dollars. Well, consider that the cost is actually $5,000, along with the fact that Nvidia spent over $50,000,000 on R&D with the NV30 core, and that just reeks of mud-slinging.

Of course, the nVidia response also fails to acknowledge that Nvidia was indeed a beta member, for approx 16 out of 20 months during the design of 3DM03, and only pulled out at the last second.

Of course, Nvidia also lied about their drivers, and how they are dsigned, "We don't think that 3DMark2003 is a good benchmark. Do you really want our engineers to spend X amount of time optimizing for a synthetic benchmark that we don't feel is representative of real games, or spend it optimizing our drivers for real/actual/shipping games?"

And they went on to say something like...

"nVidia doesn't optimize for specific titles, rather, general pipeline optimizations..."

I know these quotes aren't word-for-word, ... they're very close...

I was only mildly disspointed w/ Nvidia with the cheating in 3dM03, but this response is infuriating, insulting, and appalling.
 
More mudslinging. At X-Bit Labs.
According to the representative, since NVIDIA was not part of the Futuremark Beta Program (a program which costs of hundreds of thousands of dollars to participate in), NVIDIA did not get a chance to work with Futuremark on writing the shaders like they would with a real applications developer. NVIDIA did not know what Futuremark had done, but they think the benchmark developer had intentionally tried to create a scenario that makes NVIDIA’s products look bad.
I keep reading it only costs $5000 to join FutureMarks beta program. IF this is true, surely they could afford that.

And here is some more. Is nVidia cheating with the Splinter Cell Demo? Read it at Driver Heaven. Looks like it's getting deeper every day.
 
From Beyond 3D comes a short read about the nVidia cheating problem. And, Patrick Ojala, 3D Mark producer, is already looking ahead to the next release of 3D Mark. I really had no idea how small 3D Mark is, only 26 emplyees. They must be some brave souls to take on a company like nVidia the way the have. They have my respect.
I have read some threads in our forum and here too, where people speculate that we are out to do some damage to Nvidia. Let me ensure you that is not the case. If it was up to me, I would only patch 3DMark03 when some critical bug is found, or the hardware detection needs to be updated. Otherwise I would like to concentrate on making the next 3DMark, once again a better benchmark than ever before. I'm doing my best to keep the next 3DMark on track despite this PR mess, but it's not easy. We are a small company (26 employees tot.) in Finland and a 3 person sales office in the US. We have no desire to confront a huge >1500 employee market leader of high end PC graphics. Some of our documents may be seen as harmful to Nvidia, and I find that regrettable. We only want to defend the validity of our benchmark products, since our business depends on it.
I wish we could somehow leave behind us our differences in opinions, again work together on a new 3DMark version, and again do our best to make it reflect future 3D game performance. Even though Nvidia left our beta program, we will make sure the next 3DMark is an impartial benchmark that genuinely measures 3D hardware performance. We will test it also on any Nvidia hardware available, but if Nvidia are out of the beta program, they will not get any pre-releases. You should therefore not judge them even if the next 3DMark doesn't run that well on their hw right after we launch. Give them some time to get possible driver issues fixed, and start comparing scores only when the rendering is correct.
_________________
Patric Ojala - 3DMark Producer
 
I keep reading it only costs $5000 to join FutureMarks beta program. IF this is true, surely they could afford that.

Futuremark requires a 'bte aprogram fee' proportional to your position on the graphics chip market, therefore expects nvidia to pay sunstantially more than anyone else. 5000 is the price for a NON-MANUFACTURER to enter.

Being a 'beta tester' gives you the ability to submit 'suggestions' to futuremark as to how to structure and optimise the benchmark - ie: gives you the right to special consideration.

Basically, the money is a direct bribe to have the benchmark written in a way that suits your hardware, Nvidia were not happy with this and withdrew from the system, and they are being suitably punished for that, I guess the bribe looks like a sensible business decision now, which is a great shame.
 
If you read here, you'll see that nvidia only have to pay $5000...
That will give them a place on the beta program...

But that will most likely only give them "Beta Member" status, and not "Strategic Beta Member"...

You can read about the difference between the member status' here...

Futuremark requires a 'bte aprogram fee' proportional to your position on the graphics chip market, therefore expects nvidia to pay sunstantially more than anyone else. 5000 is the price for a NON-MANUFACTURER to enter.

Where do you have this from?

From FM:
Participating companies may choose from three different service levels; BETA Member, Active BETA Member, and Strategic BETA Member. Service levels have fixed annual membership fees for each calendar year.

How can you get that to mean "proportional to you position in the graphcis chip market"?

As far as the bribe goes...
Originally by Patric Ojala:
And second, Nvidia was a member of our beta program, got pre-release builds, shader source code, and participated actively in the development of 3DMark03. After they left our beta program, there was mostly just bug hunting and product quality assurance left.
(My emphasis...)

Do you have any proof what-so-ever to come with? Yes, you are entitled to your opinion, but in light of the information I've posted here you might want to rethink yours...
Or at least give us more details as to why you think it's still the most likely reason.....

Maybe even register :)
Even though we (for the moment) disagrees on this subject, it shouldn't be a reason not to join... (Or at least I hope it isn't!)
 
i found this statemant in guru3d.com
and i found it very funny, it really
makes me wonder some things

For the bigger part of it you should not blame FutureMark for this though, but blame the parties that started cheating. These are both nVIDIA and ATI and I don't care wether its a 2% or a 25% difference, cheating is cheating. I actually applaud nVIDIA for the way they did it, if you do it then have the b@lls to do it well.
 
I can understand you last statement, but I feel they went about it the wrong way...

IMO, they shouldn't have published the drivers as regular drivers, but should instead have called them "3dmark killers" or something like that, as that would have told people what to expect from them...

But to cheat like this, and then deny it hoping it would blow over doesn't really make me want to applaud them...

I don't like what ATI did either (yes, the only re-arranged the shaders, but they still had to recognize the program for it to work), but they at least said, ok, we'll remove it in the next drivers.... No talk about bugs or anything....
 
Most of us here will say, we don't look at benchmarks, that the way the games play is what counts. But, the fact of the matter is, most of us, when we get a new card can't wait to benchmark it and compare it to others. And this is what sells cards. I might be stepping out on a limb here, but I imagine, other than brand loyalty, whatever the scores of the top end cards for each manufacturer is, becomes what most of us use as a basis for buying cards. It's probably percieved, that if the top end card is higher scoring, the rest of the line is better too. That's what sells cards and makes big companies cheat. To bad that most of us believe that big corporations would never cheat, or maybe I should say, it's getting to be common place.

I'm sure that nVidia and ATi will come out of this just fine in the end. But who's going to be next, Intel and AMD?
 
Status
Not open for further replies.
Back