NVIDIA caught cheating?

By on May 15, 2003, 1:36 PM
ExtremeTech has posted some information regarding NVIDIA's latest Detonator FX drivers that supposedly could be inflating 3D Mark 2003 scores a tad by cutting corners on the "game based" tests. NVIDIA has said they are looking into the problem and it may be just a 'driver bug'.

Seriously, cheating or not, I don't blame NVIDIA for trying to cut corners, if a lesson should be learned from this is not to rely on standard benchmark utilities completely but rather on real world applications/games... all other benchmarks we have seen and read based on games such as UT2003, Serious Sam 2 and even Doom 3 remain valid.




User Comments: 19

Got something to say? Post a comment
acidosmosis said:
This "controversy" has been all over the net. To be honest, about ATI vs. Nvidia. It' just an issue I really dont give a flip about. Companies are going to cheat, lie and be deceitful. That is why I rely on my own instincts and knowledge instead of others opinions, and benchmarks that already don't show true ratings for hardware. The rest of you should do the same. We will let computer illiterate's rely on benchmarks. Otherwise they can pay people like me or you for consultation charges to help them buy the right products.
TS | Thomas said:
It'll be most interesting of course to see how long it takes for this bug to be fixed.HardOCP has a good post on this matter though, just a few bits here of interest;"Two days after Extremetech was not given the opportunity to benchmark DOOM3, they come out swinging heavy charges of NVIDIA intentionally inflating benchmark scores in 3DMark03. What is interesting here is that Extremetech uses tools not at NVIDIA's disposal to uncover the reason behind the score inflations. These tools are not "given" to NVIDIA anymore as the will not pay the tens of thousands of dollars required to be on the "beta program" for 3DMark "membership".Conversely, our own Brent Justice found a NVIDIA driver bug last week using our UT2K3 benchmark that slanted the scores heavily towards ATI. Are we to conclude that NVIDIA was unfairly increasing the workload to decrease its UT2K3 score? I have a feeling that Et has some motives of their own that might make a good story."
forciano said:
perhaps you guys should do a comparison when cat 3.5 come out, many people (including me) want to see the performance of both cards with more mature drivers.
TS | Julio said:
[quote][i]Originally posted by TS | Thomas [/i][b]HardOCP has a good post on this matter though, just a few bits here of interest[/b][/quote] Ironically Hardocp is posting that, the same guys that in the old days made up controversy for everything only for the extra site exposure, not a bad site just that bit of it always bothered me.
MrGaribaldi said:
One of the reasons many are up in arms about this isn't because it's [i]just another syntethic benchmark[/i], but because a lot of OEM deals center around the perfomance of a card in these benchmarks... So when a company cheats in 3dmark (which is a very well known benchmark for OEM's) it can sway deals worth millions, based on a [b]fake[/b] result...I, personally, think Nvidia is cheating in 3dmark03 with their latest drivers based on the fact that the "clipping" trick used [i]doesn't just insert itself like this [b]by itself[/b][/i]...Had it been happening where everyone could see it, yes then it would most likely be a bug, but as it only happens when you "go off the rail" the chances are [i]extremly[/i] slim...As far as [H] goes, yes his post is valid... Kyle does state the facts known to him, [i]but the way it's stated removes all claims of objectivity![/i] Nor has he tried to find out more about this himself, or substantiated the "ET is pissed since they didn't get to benchmark D3" remark...But let's split it up:1. Yes, it was released 2 days after the D3 benchmarks... [b]But[/b] it was also released two days after the DetFX was released... And if you take a look at B3D, you'll find Wavey and Reverend talking about this article [b]one week ago[/b]... Wavey sent pictures of what was happening to Nvidia asking them for an explanation, [i]but has yet to receive any official explanations[/i].2. Nvidia doesn't have access to the beta/debug version of 3dmark03. Quite correct. [b]But[/b] [i]they were a beta member for (over) 16 months[/i] before they left... So it was their own decision not to have access to the build..."They don't want to pay the tens of thousands of dollars to be a beta member"....Isn't this the company which has around a billion of cash lying around?!? To optimize for 3dmark03 the way they've done also costs them quite a lot of money....3. Kyle can post anything he wants on [H], as he owns the site... Salvator (sp?) is an employee of [i]Ziff-Davis[/i] and [b]can not[/b] just post whatever he fancies... ZD is a [b]large[/b] IT company, and stand to loose quite a lot if these are baseless charges, but still they decided to publish them.4. "Nvidia is doing this do discredit 3dmark/futuremark"This could very well be the case, [b]but why don't Nvidia say that openly?!?[/b]There is no mention of this in the readme, and the drivers are published like regular drivers!Thus it does not seem to me like they're doing this to discredit 3dmark, but just to boost their score...5. "Only fanATIcs are shouting abuse at this"... Errr, where did he get this idea... Take a look at B3D (as the discussion is running very hot there), strip away the obvious fanboys (at both sides), and you'll see that most (if not all) of the "in betweens" are outraged as well... This is because it this could be further reaching than most people think about... Yes, it's only been noticed in 3dmark so far. But who's to say that Nvidia aren't doing this in any other benchmark which "runs on rails"? They could very well be doing the same thing for Q3's Timedemo 1,2,3... Yes, it's an actual game, [b]but[/b] with the timedemos [i]it is [b]only[/b] a benchmark[/i]...There is [b]no way[/b] to "jump into" the demo and look around... Or take the flyby in UT2k3... the FX scores way above the 9800pro, but not in the botmatch... Could this be because of similar culling/clipping?And before you tell you me you can fly around by yourself, it's not that hard to do a "listen" for a key being pressed, and stop doing the optimization....Or how about RightMark, AquaMark etc? [b][i]NOTE: I'm not saying Nvidia are cheating in all the benchmarks, but that it is a possibility based on what ET discovered![/b][/i]IF Nvidia [i]really[/i] wanted to stop people using 3dmark as a benchmark, [i]why not add some code omt the drivesr which makes it impossible to run 3dmark[/i]? That would've been a fair way to tell the world about Nvidias dislike for 3dmark, since "it doesn't test the way we think it should do"... Why all this optimizing/cheating, [i]if you don't want people to use the benchmark[/i]?According to B3D and ET there has been a noticeable increase in bugs with nvidia drivers, that seem to correspond with increased scores...I think I'll stop now before I start raving...I just got pissed of at Kyle's "fact post" (which i'm posting at the end of this post, so you can see the entire one for yourself...[i]One last thing before I shut up (for now), read ET's article for yourself, [b]and think about it[/b][/i]...Kyle never had a proper rebuttal of what they found, he only talked about the circumstances they posted it....02$ (and thank you for reading so far)[quote][i]Originally posted by [email=kyle@hardocp.com]Kyle[/email] on [url=www.hardocp.com][H]ard|OCP[/url]:[/i]Two days after Extremetech was not given the opportunity to benchmark DOOM3, they come out swinging heavy charges of NVIDIA intentionally inflating benchmark scores in 3DMark03. What is interesting here is that Extremetech uses tools not at NVIDIA's disposal to uncover the reason behind the score inflations. These tools are not "given" to NVIDIA anymore as the will not pay the tens of thousands of dollars required to be on the "beta program" for 3DMark "membership".nVidia believes that the GeForceFX 5900 Ultra is trying to do intelligent culling and clipping to reduce its renderingworkload, but that the code may be performing some incorrect operations. Because nVidia is not currently a member of FutureMark's beta program, it does not have access to the developer version of 3DMark2003 that we used to uncover these issues.I am pretty sure you will see many uninformed sites jumping on the news reporting bandwagon today with "NVIDIA Cheating" headlines. Give me a moment to hit this from a different angle.First off it is heavily rumored that Extremetech is very upset with NVIDIA at the moment as they were excluded from the DOOM3 benchmarks on Monday and that a bit of angst might have precipitated the article at ET, as I was told about their research a while ago. They have made this statement:We believe nVidia may be unfairly reducing the benchmark workload to increase its score on 3DMark2003.nVidia, as we've stated above, is attributing what we found to a bug in their driver.Finding a driver bug is one thing, but concluding motive is another. Conversely, our own Brent Justice found a NVIDIA driver bug last week using our UT2K3 benchmark that slanted the scores heavily towards ATI. Are we to conclude that NVIDIA was unfairly increasing the workload to decrease its UT2K3 score? I have a feeling that Et has some motives of their own that might make a good story. Please don't misunderstand me. Et has done some good work here. I am not in a position to conclude motive in their actions, but one thing is for sure. 3DMark03 scores generated by the game demos are far from valid in our opinion. Our reviewers have now been instructed to not use any of the 3DMark03 game demos in card evaluations, as those are the section of the test that would be focused on for optimizations. I think this just goes a bit further showing how worthless the 3DMark bulk score really is. The first thing that came to mind when I heard about this, was to wonder if NVIDIA was not doing it on purpose to invalidate the 3DMark03 scores by showing how the it could be easily manipulated.Thanks for reading our thoughts; I wanted to share with you a bit different angle than all those guys that will be sharing with you their in-depth "NVIDIA CHEATING" posts. While our thoughts on this will surely upset some of you, especially the fanATIics, I hope that it will at least let you possibly look at a clouded issue through from a different perspective.[/quote]
DigitAlex said:
great analysis, MrGThere is only one problem :)I don't have any money to buy the GeforceFX and 3DMark2003 and I still would like to see some screenshots that show what's really happening ...I've coded some 3D apps so my developer side would like to find out a little bit more, at least some screenshots of these "corners" everybody's talking about ...
DigitAlex said:
oh ohsilly me, sorry guys, here they are[url]http://www.extremetech.com/article2/0,3973,1086853,0
.asp[/url]
MrGaribaldi said:
Hehe, thanks :)Nice to see that someone enjoyed it!
MrGaribaldi said:
Hmm... Either tech news is very slow at the moment, or I managed to scare off even the staff from this part of the site... :D
DigitAlex said:
i dunno, maybe they all had a buffer overflow while reading :)hey guys, please upgrade your drivers hehehehe :D :D
Mictlantecuhtli said:
By the way, has anyone noticed Nvidia is not the only one doing things like this?[url=http://www.spodesabode.com/content/article/trident
Trident Cheating Benchmarks?[/url]
TS | Thomas said:
Nope, I've just been away a few days :) Yes, I now see Beyond3D's bit at [url]http://www.beyond3d.com/forum/viewtopic.php?topic=5856&
orum=9[/url] as well. I dunno to be honest, it's a pretty crappy thing to do if it is true (i.e. it's a purposeful "bug"), though they would have to be aware that it would be noticed if anyone used 3D Mark's freeview mode.From NVIDIAs point of view "We improved performance & did not sacrifice image quality" - & to all *but* those beta testers they have indeed done that.More importantly though is that there's no real penalty for "cheating" like this. It annoys people like us, same as it did when ATi & Trident (Cough) do it, but is there really any long term harm done to the companies for this fraudulent misrepresentation? Nope.Well, this is why I prefer doing Soundcard reviews. There's not much that can be done underhandedly by companies about them :)
timmoore said:
If NVIDIA is cheating, very well, I accept this - after seeing valid evidence of course. But those who think that ATi or other companies have never done anything like this are fools to think so. The only difference is, they didn't get caught, or they did and nobody made a big deal about it - either way, I stand beside my point.
DigitAlex said:
tim, you speak as if you were NVIDIA's official lawyer or something hehheeh ;)
Unregistered said:
Tim if you recall ATI was hammered when it was said that Radeon 8500 drivers were quake optimized.And Nvidia was one of the first to cry foul.It seems to me that Nvidia knew its 5900 could only compete with the 9800 pro and not out perform it.Nvidia is desperate to show it can regain perfoermance interest from the consumer.This will all be mute at the 5900's launch as ATI has already Taped it's R360 out and is said to be releasing on Nvidia's 5900 launch date.Seems Nvidia is a day late and a dollar short again.Oh BTW whats up with all these artifacts in my latest detonator drivers (seems Nvidia is now the one with the crap drivers).
timmoore said:
Well, let's notice that ATi was also caught cheating by Futuremark, not as badly mind you, but it is there. It is in the margin of error so everyone says, but I will believe that when I see/hear it!Futuremarks PDF: [URL=http://www.futuremark.com/companyin...udit_report.pdf]*
*[/URL]
DigitAlex said:
not cheatingn optimizing for real, according to what they explained :)see the other threads where the difference between optimization and cheating is explained :)[url]http://www.techspot.com/vb/showthread.php?s=&threadid
5590[/url][url]http://www.techspot.com/vb/showthread.php?s
&threadid=5589[/url]
timmoore said:
Let me apologize for my invalid accusations against ATi, they were all unfounded and incorrect, apparently. I hope that I didn't offend anyone with my constant ranting about NVIDIA and ATi; my criticisms were purely comparisons between the two excellent chip manufacturing companies. Once again, I apologize, long live ATi and NVIDIA ;) !!!
DigitAlex said:
As you said, the two companies are not that bad :)If you remember some events in the past, the competition always results in better achievements.USA vs USSR / Russia for space, at the beginning of the space explorations.Many archeologist hunting and competing for dinosaur bones ...As long as the competition is FAIR, it always leads to positive results !ugh
Load all comments...

Add New Comment

TechSpot Members
Login or sign up for free,
it takes about 30 seconds.
You may also...
Get complete access to the TechSpot community. Join thousands of technology enthusiasts that contribute and share knowledge in our forum. Get a private inbox, upload your own photo gallery and more.