ATi and nVidia both cheating?

[simwiz2]

http://www.warp2search.net/article.p...thread&order=0

A new version of 3DMark 2003 fixed cheats by making detection of the tests more difficult to the drivers, and BOTH ATi and NVidia lost points.

nVidia lost over 20% of their score.

ATi lost just 1.9%. They shouldn't even have bothered. Their cheat, if it was indeed a cheat (still being investigated it seems) gave an 8.2 performance lead, but only in one of the tests.

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Although nVidia's cheats were more "effective", cheating is cheating. I'll probably be ignoring 3DMark results now, since it probably won't be long before both companies figure out how to cheat in the new version.

[pam123]

This a direct quote from the folks a FutureMark:In our testing, all identified detection mechanisms stopped working when we altered the benchmark code just trivially and without changing any of the actual benchmark workload. With this altered benchmark, NVIDIA’s certain products had a performance drop of as much as 24.1% while competition’s products performance drop stayed within the margin of error of 3%. To our knowledge, all drivers with these detection mechanisms were published only after the launch of 3DMark03. According to industry’s terminology, this type of driver design is defined as ‘driver cheats’.
We are publishing this document to report to our customers in detail about our findings. Main reason behind publishing this document is to answer the criticism presented against synthetic benchmarks and their reliability when testing hardware performance. The document follows a question/answer format.

All competing products stayed within the marging of error of 3%, at 1.9% ATI is well within that margin.

[whargoul]

There is also the reply issued by ATi at the link you posted:

"The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST."

I don't know if it is indeed a direct quote from ATi. But if it is, it's alot better than nVidia's reply - which basically stated everything is a lie meant to give them a bad name, for whatever reason. nVidia just keeps getting lower and lower.

[simwiz2]

An 8.2% drop in one test is quite large though.

[pam123]

Futuremark is very clear about optimizations that improve test scores and degrade gameplay and optimizations that increase FPS and maintain game quality.
They say the first is the cheat and the last is doing your job.
The quote is from ATI and it was a good move on their part but they were legit in what they did.

[glc]

See? I have no use for synthetic benchmarks. The only test results I will even look at are actual games and applications, using "off the shelf" hardware and publically available drivers/bioses. Same reason the only automotive reviews I place any credence in are ones done by Consumer Reports who buys the cars, not use media test fleet ones.

[pam123]

I'm thinking of writing in and asking them to test video cards.

[simwiz2]

http://www.extremetech.com/article2/...1105265,00.asp

The fact that ATi's "optimization" was stopped by the new version (which prevents driver detection of the test) suggests that ATi's "optimization" was not a general driver optimization, but a specific optimization for that test. If specific optimizations for a benchmark tests are legitimate, then everything NVidia did was perfectly fine. After all, both companies' optimizations show the scene in the release version of the final product correctly.

[Mac Medic]

ATI were caught doing the same thing once before, but i think it was a quake benchmark they were cheating with.

[simwiz2]

I saw some screenshots of Quake with those drivers, it looked really awful. I can't believe ATi thought no one would notice that big of a difference.

[sleepypost]

Lets be honest. nVidia is cheating like mad. They are blatantly cheating (20 %!!!). You cannot claim ATi is cheating however because 1.9 % is marginal. Benchmarks vary day by day at greater percentages than that. nVidia *AND* ATi cheating, please.

[Spyda]

any performance gain is seen as cheating, be it 0.1% or 10%, thats what the futuremark site says.

ATi arent gods to be worshipped, they can makes mistakes just like everyone else

[sleepypost]

Futuremark is being rediculous then. They of all people should know that benchmarks vary day by day, test by test, hour by hour by gains and losses greater than 1 percent. They are being rediculous and stupid.

***I AM defending ATi from people who falsely accuse them (that is ILLEGAL by the way), however I am not worshipping them. If I owned large amounts of stock in ATi, I would probably have this thread investigated... no joke.

[simwiz2]

An 8.2% loss in a test is just normal variation? If that is the case then 3DMark03 is a crappy benchmark anyway and no one should bother with the results. Imagine if such a drop occurred in all the tests, that is an 8.2% margin of error for the benchmark!

I will admit ATi's cheats were worthless and hardly helped their overall score, but cheating is cheating. Imagine how many people would cheat on SATs and other school tests if they could get off the hook by claiming they would have done almost as well without the cheating...

[Blakhart]

How about a card that plays dx7 thru dx9 games flawlessly?
How about a card that is faster than what we now have, instead of the same speed, but is faster at af and aa?
I do not care for either af or aa. My guess is that 99 percent of users never mess with af or aa. Want a better image? Nothing more powerfull than higher resolution.
About time we got cards that are faster than a gf3.
No need for benches, let the gamers decide.

[Alienware_Dude]

I, for one, chose the 9700 Pro over any nVidia card because of it's superior performance with AA enabled; I guess I'm in the 1%.

[sleepypost]

well said Blakhart, however the speculations (and intent???) of this thread kind of irritated me a bit.

[simwiz2]

The intent of this thread is to find the truth.

Remember that when the nVidia thread was first posted, it was also speculation based off the findings of one site. I wouldn't call this either thread speculation any more, since ExtremeTech now says both cards were probably cheating, and many online sources seem to agree.

[Spyda]

sleepy, no offence, i think you love ATi and AMD too much

[sleepypost]

Spyda, i think you're right. but its not like i have their logos engraved in my avatar for all to see or anything like that...

but seriously, i was (and still am) in a bad mood anyway today because of my damn HDD acting up, so i guess that is what set me off, not this thread. however, i stand by everything ive said thus far... and 20% = cheating. 2% = not cheating. (PERIOD). just leave it at that...

[Spyda]

http://www.xbitlabs.com/news/video/d...526040035.html

they admit it

[sleepypost]

ATI states that these were exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, were a realistic approach to a benchmark intended to measure in-game performance.

I will admit to being wrong about ATi. however, this article seems to take ATi way out of context. no where does the ATi rep say anything about cheating. he does admit to one thing. they optimized two settings. yes, that is unfair, but it is not altering code. did ATi cheat? yes. (i was wrong, okay...) but, i will still defend that ATi didn't cheat nearly as bad as nVidia, thus articles like these are rather obnoxious. the title of the article is borderlining that of a National Enquirer title: "Elvis is pregnant!"

I wonder if ATI actually optimizes anything for real games with pixel shaders.

more speculation by the author...