TechSpot

ATi & NVIDIA on Futuremark Report

By TS | Thomas
May 24, 2003
  1. Regarding the Catalyst performance gains that were discovered in 3D Mark 2003 ATi has responded with the following:
    The 1.9% performance gain comes from optimization of the 2 DX9 shaders (water & sky) in Game Test 4. We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically & functionally identical to Futuremark's & there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality & as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, & so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.

    Meanwhile NVIDIA says the following regarding Futuremarks findings:
    Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer.

    Draw your own conclusions to these reponses.
     
  2. MrGaribaldi

    MrGaribaldi TechSpot Ambassador Posts: 2,512

    ROTFL!!!

    One has the decency to appologize, the other talks FUD...
     
  3. TS | Thomas

    TS | Thomas TS Rookie Topic Starter Posts: 1,319

    I think ATi's reponse there is milking it just a bit there (Can't blame em though). Gotta love NVIDIA's "real applications developer" bit though.
     
  4. Well when Nvidia complained to Futuremark about methodology, I believe they were told to get lost.

    Kind of funny, but when you tell that to a company with 70% marketshare, something like this happens.

    Given the initial response from Futuremark and the fact that major games these days ARE optimized, I've no sympathy for Futuremark. I'm not loving how Nvidia went about it, but I think this was a quite humourous reply. Don't want to talk about what your testing program does right or wrong? Fine, don't expect us to make your life easy.

    People so easily forget the history and just expect Nvidia to quietly live with whatever someone wants to say about its products. Were I them this is the very least I would have done. I'd not expect Futuremark to be around very long unless it changes its tune.

    Agree or not, we'll see whether I'm right.
     
  5. forciano

    forciano TS Rookie Posts: 24

    funny how people and nvida try to downplay futuremarks importance, if it wasnt, why did they cheat on the test. Cause it sells there cards, but they dont want to pay futuremark so they can optimize there drivers. I just think nvidia should realize that the are doing things the wrong way.
     
  6. DigitAlex

    DigitAlex TechSpot Paladin Posts: 536

    the right thing would be not to rate the performance of hardware with just a little benchmark and assume lots of things depending of how much points it gets
     
  7. TS | Thomas

    TS | Thomas TS Rookie Topic Starter Posts: 1,319

    Replacing shaders with your own version is not "optimizing".

    When I get Doom 3, or any other game for that matter, I don't expect my Graphics card's Driver to replace anything in the game with something else.

    NVIDIA didn't optimize anything, beyond REPLACING shaders with their own "similar" versions they also screwed up rendering in several places for artifical performance improvements, which anyone with the Developer version of 3D Mark could see.

    NVIDIA are effectively fraudulently misrepresenting the performance of their cards with these Drivers. ATi got in similar trouble for early Radeon Drivers in Quake 3. Do you consider what they did to be optimisation also? Afer all it improved performance & only reduced visual quality.
     
  8. PreservedSwine

    PreservedSwine TS Rookie Posts: 325

    First off, congratulations to FutureMark for their handling of this. While I'm more in the camp of gaming benchmarks over synthetic benchmarks, each have their own signifigance.

    Secondly, has Nvidia has lost their collective minds in this matter? They cover up lie with even more lies, and accusations. They insinuate FutureMark and ATI are out to get them. I wonder what the other Beta members would say about this? In fact, I think nVidiais about the ONLY gpu maker that is NOT a Beta member. Also, if Futuremark is out to get them, (supposably since ATI is a Beta member) does that mean we must ignore EVERY OTHER BENCHMARK that shows how poorly the FX5800 and FX5900 handles shader routines? Virtually every single shader benchmark shows the same thing, that the Nvdia shaders are not performing nearly as well as the R9700/9800's.

    nVidia also claimed the cost of the Beta membership as costing *hundreds of thousands* of dollars. Well, consider that the cost is actually $5,000, along with the fact that Nvidia spent over $50,000,000 on R&D with the NV30 core, and that just reeks of mud-slinging.

    Of course, the nVidia response also fails to acknowledge that Nvidia was indeed a beta member, for approx 16 out of 20 months during the design of 3DM03, and only pulled out at the last second.

    Of course, Nvidia also lied about their drivers, and how they are dsigned, "We don't think that 3DMark2003 is a good benchmark. Do you really want our engineers to spend X amount of time optimizing for a synthetic benchmark that we don't feel is representative of real games, or spend it optimizing our drivers for real/actual/shipping games?"

    And they went on to say something like...

    "nVidia doesn't optimize for specific titles, rather, general pipeline optimizations..."

    I know these quotes aren't word-for-word, ... they're very close...

    I was only mildly disspointed w/ Nvidia with the cheating in 3dM03, but this response is infuriating, insulting, and appalling.
     
  9. olefarte

    olefarte TechSpot Ambassador Posts: 1,343   +6

    More mudslinging. At X-Bit Labs.
    I keep reading it only costs $5000 to join FutureMarks beta program. IF this is true, surely they could afford that.

    And here is some more. Is nVidia cheating with the Splinter Cell Demo? Read it at Driver Heaven. Looks like it's getting deeper every day.
     
  10. olefarte

    olefarte TechSpot Ambassador Posts: 1,343   +6

    From Beyond 3D comes a short read about the nVidia cheating problem. And, Patrick Ojala, 3D Mark producer, is already looking ahead to the next release of 3D Mark. I really had no idea how small 3D Mark is, only 26 emplyees. They must be some brave souls to take on a company like nVidia the way the have. They have my respect.
     
  11. I keep reading it only costs $5000 to join FutureMarks beta program. IF this is true, surely they could afford that.

    Futuremark requires a 'bte aprogram fee' proportional to your position on the graphics chip market, therefore expects nvidia to pay sunstantially more than anyone else. 5000 is the price for a NON-MANUFACTURER to enter.

    Being a 'beta tester' gives you the ability to submit 'suggestions' to futuremark as to how to structure and optimise the benchmark - ie: gives you the right to special consideration.

    Basically, the money is a direct bribe to have the benchmark written in a way that suits your hardware, Nvidia were not happy with this and withdrew from the system, and they are being suitably punished for that, I guess the bribe looks like a sensible business decision now, which is a great shame.
     
  12. MrGaribaldi

    MrGaribaldi TechSpot Ambassador Posts: 2,512

    If you read here, you'll see that nvidia only have to pay $5000...
    That will give them a place on the beta program...

    But that will most likely only give them "Beta Member" status, and not "Strategic Beta Member"...

    You can read about the difference between the member status' here...

    Where do you have this from?

    How can you get that to mean "proportional to you position in the graphcis chip market"?

    As far as the bribe goes...
    (My emphasis...)

    Do you have any proof what-so-ever to come with? Yes, you are entitled to your opinion, but in light of the information I've posted here you might want to rethink yours...
    Or at least give us more details as to why you think it's still the most likely reason.....

    Maybe even register :)
    Even though we (for the moment) disagrees on this subject, it shouldn't be a reason not to join... (Or at least I hope it isn't!)
     
  13. forciano

    forciano TS Rookie Posts: 24

    i found this statemant in guru3d.com
    and i found it very funny, it really
    makes me wonder some things

    For the bigger part of it you should not blame FutureMark for this though, but blame the parties that started cheating. These are both nVIDIA and ATI and I don't care wether its a 2% or a 25% difference, cheating is cheating. I actually applaud nVIDIA for the way they did it, if you do it then have the b@lls to do it well.
     
  14. MrGaribaldi

    MrGaribaldi TechSpot Ambassador Posts: 2,512

    I can understand you last statement, but I feel they went about it the wrong way...

    IMO, they shouldn't have published the drivers as regular drivers, but should instead have called them "3dmark killers" or something like that, as that would have told people what to expect from them...

    But to cheat like this, and then deny it hoping it would blow over doesn't really make me want to applaud them...

    I don't like what ATI did either (yes, the only re-arranged the shaders, but they still had to recognize the program for it to work), but they at least said, ok, we'll remove it in the next drivers.... No talk about bugs or anything....
     
  15. olefarte

    olefarte TechSpot Ambassador Posts: 1,343   +6

    Most of us here will say, we don't look at benchmarks, that the way the games play is what counts. But, the fact of the matter is, most of us, when we get a new card can't wait to benchmark it and compare it to others. And this is what sells cards. I might be stepping out on a limb here, but I imagine, other than brand loyalty, whatever the scores of the top end cards for each manufacturer is, becomes what most of us use as a basis for buying cards. It's probably percieved, that if the top end card is higher scoring, the rest of the line is better too. That's what sells cards and makes big companies cheat. To bad that most of us believe that big corporations would never cheat, or maybe I should say, it's getting to be common place.

    I'm sure that nVidia and ATi will come out of this just fine in the end. But who's going to be next, Intel and AMD?
     
Topic Status:
Not open for further replies.

Similar Topics

Add New Comment

You need to be a member to leave a comment. Join thousands of tech enthusiasts and participate.
TechSpot Account You may also...