In this explainer we support with data why testing CPU performance with a GPU bottleneck is a bad idea, so you can have a full understanding of our testing methods when reviewing CPUs.
https://www.techspot.com/article/2618-cpu-benchmarks-explained/
How We Test: CPU Benchmarks, Misconceptions Explained
- Thread starter Steve
- Start date
- Status
Burty117
Posts: 4,781 +3,218
Impeccable article, even going out of your way to test and show the useless data. Expect this article to be linked in most comment sections of hardware reviews.
So can I expect any performance gains upgrading my 5820k to a 7800X3D in games? 
Theinsanegamer
Posts: 4,153 +7,457
The sad thing is, no matter how many times you explain it, the knuckle dragging m0r0ns will still reply with "why would you test low end CPUs with a 4090" and "who uses a 4090 at 1080p". Steve's got one hell o an uphill fight to make these people see the light.
gamerk2
Posts: 820 +792
I've been making the exact argument this article highlights for years. And I'm speaking as someone still rocking his 8070k paired with a 3080Ti (previously a 1070Ti, so close to exactly the case this article highlights).
You isolate the performance of the specific part you are trying to test, and for CPUs in gaming the best way to do that is low resolution with a powerful GPU, to see what the real difference is when the GPU is removed as a bottleneck. Not realistic for today, but gives you an idea how well parts will last tomorrow.
I'm also surprised users are doing three GPU upgrades per CPU; I guess most here are upgrading every GPU generation or so. I tend to do two generation GPU upgrade cycles (EG: 1070TI to 3080Ti), then junk the build once the second CPU/GPU combo becomes functionally obsolete, usually 4-5 years into the build. I then build around the best CPU on the market and go again.
You isolate the performance of the specific part you are trying to test, and for CPUs in gaming the best way to do that is low resolution with a powerful GPU, to see what the real difference is when the GPU is removed as a bottleneck. Not realistic for today, but gives you an idea how well parts will last tomorrow.
I'm also surprised users are doing three GPU upgrades per CPU; I guess most here are upgrading every GPU generation or so. I tend to do two generation GPU upgrade cycles (EG: 1070TI to 3080Ti), then junk the build once the second CPU/GPU combo becomes functionally obsolete, usually 4-5 years into the build. I then build around the best CPU on the market and go again.
poshflamingos
Posts: 57 +143
While everything said in this article is undeniably correct, I do have one question.
In the "i3-13100 vs Ryzen 5600" article, you guys showed how Nvidia drivers have more CPU overhead than AMD drivers, and as consequence the 13100/5600 perform better in Watch Dogs Legion at 1080p with a RX 6950 than they do with a RTX 4090, a completely counter-intuitive result.
Doesn't this driver CPU overhead issue disproportionally affect lower-end CPUs? Wouldn't using an AMD GPU instead of the 4090 allow lower-end CPUs to fare a little better relative to higher-end ones?
Since the goal is to eliminate other possible bottlenecks to evaluate CPU performance, wouldn't a 7900 XTX be a better choice for those tests, minimizing the driver overhead factor?
In the "i3-13100 vs Ryzen 5600" article, you guys showed how Nvidia drivers have more CPU overhead than AMD drivers, and as consequence the 13100/5600 perform better in Watch Dogs Legion at 1080p with a RX 6950 than they do with a RTX 4090, a completely counter-intuitive result.
Doesn't this driver CPU overhead issue disproportionally affect lower-end CPUs? Wouldn't using an AMD GPU instead of the 4090 allow lower-end CPUs to fare a little better relative to higher-end ones?
Since the goal is to eliminate other possible bottlenecks to evaluate CPU performance, wouldn't a 7900 XTX be a better choice for those tests, minimizing the driver overhead factor?
m3tavision
Posts: 1,256 +1,014
Thank you. I've always understood Techspot/HU tests. Each gamer has needs, but sometimes even they do not understand what piece of hardware will give them best results.
Over the last few month I have heard people dish on TECHSPOT and it's just redirect and pouting. TechSpot's graphics and charts are too hard to argue with and many simps just want to complain... bcz results don't align with their zealotry. Steve does a good job of stepping into the naysayers shadows forcing them back under their bridges.
I really like this simple article, & will be able to link it to all those who have "theories...."
As a follow-up...
I would really like to see TECHSPOT/Hardware Unboxed do this test with AMD's AM4 entire generation... it will be much easier, because you can use ONE motherboard to test 5 generations of CPUs. (1800x, 2800x, 3800x & 5800x, & 5800X 3D).
All baselined against the AM5 7800x 3D. Using generational GPU architecture from NVidia & AMD. (ie 2080 3080 4080 & RDNA1 ~3)
Over the last few month I have heard people dish on TECHSPOT and it's just redirect and pouting. TechSpot's graphics and charts are too hard to argue with and many simps just want to complain... bcz results don't align with their zealotry. Steve does a good job of stepping into the naysayers shadows forcing them back under their bridges.
I really like this simple article, & will be able to link it to all those who have "theories...."
As a follow-up...
I would really like to see TECHSPOT/Hardware Unboxed do this test with AMD's AM4 entire generation... it will be much easier, because you can use ONE motherboard to test 5 generations of CPUs. (1800x, 2800x, 3800x & 5800x, & 5800X 3D).
All baselined against the AM5 7800x 3D. Using generational GPU architecture from NVidia & AMD. (ie 2080 3080 4080 & RDNA1 ~3)
Depends on if you change monitor/resolution as well.
I run a 5820k, which has been paired with a GTX670, Radeon HD7990, GTX 1070 and now a RX 6800 XT (I'm just waiting for the 7800X3D to be released to finally put my 5820k on pension)
When I started, I had a 1650x1080@60Hz monitor, then a 1440p@60Hz and now 5120x1440p@240Hz
Granted, I don't play the newest games, but in most cases I still believe that the GPU is the limiting factor for my games.
What a great article! Congratulation Steve, thank you for the hard work and clear presentation and keep posting.
And kudos for Techspot to constantly giving us so good quality content.
And kudos for Techspot to constantly giving us so good quality content.
Last edited:
The dangers of the internet: No matter how solid your reasoning is, based on logic and science, you got morons questioning your validity based on their gut feelings.
dirtyferret
Posts: 844 +1,158
"In this case, we typically focus on low resolution testing (1080p), as this helps to alleviate the potential for GPU bottlenecks, allowing us to more closely look at actual CPU performance."
TPU tested at 720p for years with people too stupid to understand that concept and always complained. The techspot community is even worse for understanding that basic principle.
It basically comes down to "is it a positive review of the brand in my PC or a negative review"
Don't get me started on the "more cores equal better performance" crowd that Steve has clearly and factually disputed several times.
TPU tested at 720p for years with people too stupid to understand that concept and always complained. The techspot community is even worse for understanding that basic principle.
It basically comes down to "is it a positive review of the brand in my PC or a negative review"
Don't get me started on the "more cores equal better performance" crowd that Steve has clearly and factually disputed several times.
This is a good explanation, but I am much more curious why we see so much variation when comparing testing results from different review sites.
For example I'll use the Average FPS across a broad spectrum of games done by Techspot and Tom's Hardware.
The 13600K bests the 7600X 183 to 172 (6% more average frames) on Tom's
The 7600X bests the 13600K 232 to 225 (3% more average frame) on testing here. And this is testing Intel using DDR5-6400. The DDR4 results give AMD an over 9% margin of victory.
Are we seeing other influences or can choosing what games are tabulated totally distort results?
For example I'll use the Average FPS across a broad spectrum of games done by Techspot and Tom's Hardware.
The 13600K bests the 7600X 183 to 172 (6% more average frames) on Tom's
The 7600X bests the 13600K 232 to 225 (3% more average frame) on testing here. And this is testing Intel using DDR5-6400. The DDR4 results give AMD an over 9% margin of victory.
Are we seeing other influences or can choosing what games are tabulated totally distort results?
Avro Arrow
Posts: 3,714 +4,802
The formula for CPU gaming tests is both simple and time-consuming. Start with a suite of at least 20 games of various types and be sure to include ones that are known to be CPU-heavy like Shadow of the Tomb Raider and especially turn-based games like Civilization.
Using the lowest resolution that all of the games in the suite are capable of (usually 1080p these days), run the games to demonstrate the maximum number of draw-calls per second that the CPU is capable of. Next, switch to 1440p to demonstrate a more realistic case-use (because very few people actually game at 4K). Then include 4K if you really want to.
There is one point that I'd like to make and that is about turn-based strategy games like Civilization. This is because turn-based strategy games are some of the best games to use to test a CPU's capabilities because they're rarely, if ever, graphics-intensive. This property also makes them completely unsuitable for GPU testing and they should never be used as such.
Turn-based games should be treated differently however. In fact, they should exist as a separate category altogether and should never be used in a gaming suite to test GPUs. There's nothing more annoying than a reviewer that has a game like Civilization in their test suite and shows the performance in frames-per-second! Any fool knows that FPS is a completely irrelevant metric for a 2D turn-based strategy game.
Then, to make the situation even worse, the result gets included in the average FPS which affects the overall score of the CPU. It makes me wonder what drugs the reviewer was on when they did the test. I won't name names but I've seen it done by otherwise top-grade benchmarkers who definitely should know better and it drives me up the wall (if you couldn't tell). I've also seen these same reviewers use Civilization VI in GPU reviews! How the hell could anyone think that 2D game like Civilization VI, a game that could be played on a Radeon HD 5450, is even remotely suitable for testing a GPU?
Yes, I realise that I'm ranting, but can you blame me?
What should be shown is "average number of seconds between turns" because when playing Civilization, the most annoying part of the game is waiting for all of your computer-based opponents to complete their turns. For those games, the best CPU is the one that can perform those turns in the least amount of time. PLEASE, do this!
As for the productivity suite, I have very little expertise on that except that they often seem to include CineBench, 7-Zip, Blender, Photoshop and Adobe Premiere. I don't know enough about that side to comment further.
Using the lowest resolution that all of the games in the suite are capable of (usually 1080p these days), run the games to demonstrate the maximum number of draw-calls per second that the CPU is capable of. Next, switch to 1440p to demonstrate a more realistic case-use (because very few people actually game at 4K). Then include 4K if you really want to.
There is one point that I'd like to make and that is about turn-based strategy games like Civilization. This is because turn-based strategy games are some of the best games to use to test a CPU's capabilities because they're rarely, if ever, graphics-intensive. This property also makes them completely unsuitable for GPU testing and they should never be used as such.
Turn-based games should be treated differently however. In fact, they should exist as a separate category altogether and should never be used in a gaming suite to test GPUs. There's nothing more annoying than a reviewer that has a game like Civilization in their test suite and shows the performance in frames-per-second! Any fool knows that FPS is a completely irrelevant metric for a 2D turn-based strategy game.
Then, to make the situation even worse, the result gets included in the average FPS which affects the overall score of the CPU. It makes me wonder what drugs the reviewer was on when they did the test. I won't name names but I've seen it done by otherwise top-grade benchmarkers who definitely should know better and it drives me up the wall (if you couldn't tell). I've also seen these same reviewers use Civilization VI in GPU reviews! How the hell could anyone think that 2D game like Civilization VI, a game that could be played on a Radeon HD 5450, is even remotely suitable for testing a GPU?
Yes, I realise that I'm ranting, but can you blame me?
What should be shown is "average number of seconds between turns" because when playing Civilization, the most annoying part of the game is waiting for all of your computer-based opponents to complete their turns. For those games, the best CPU is the one that can perform those turns in the least amount of time. PLEASE, do this!
As for the productivity suite, I have very little expertise on that except that they often seem to include CineBench, 7-Zip, Blender, Photoshop and Adobe Premiere. I don't know enough about that side to comment further.
Last edited:
hahahanoobs
Posts: 4,840 +2,782
As a car reviewer, are you only going to review how well a Jeep does offroad? Only test how well sports cars go in a straight line?
You want to remove bottlenecks. Cool. Do it. 1080p. But....
I think what initiated this is/was with 1080p ONLY testing results with "overkill" hardware. One comment in the first image in this article is complaining about exactly that, and that's what I had a big problem with. Those are all the people that have no use for 1080p ONLY results. Those are the keywords here: 1080p ONLY.
It's a review, right? Test the more unlikely configuration to show best performance, but also test the more likely scenario especially for that type of CPU and GPU combo by upping the resolution my man! There is no need for guesswork. Tim doesn't even touch 1080p monitors in his MUB reviews. We're getting excited about higher refresh rates almost weekly. OLED. HDR. 4K. 8K. 1440p+ is just where the industry is going. Omitting it just doesn't make sense.
Aside from that, I also had the idea of a tech site eventually throwing a popular mainstream build in the mix (included in review/separate video series) with say an i3/i5/R5 + 16GB 3200MHz CL16 / DDR5 6000MHz paired with new low to high end GPU's, and a vice versa with low to high end GPU's? Rigs closer to what the majority actually have. I feel it would help a lot of people with upgrading in addition to standard reviews.
You want to remove bottlenecks. Cool. Do it. 1080p. But....
I think what initiated this is/was with 1080p ONLY testing results with "overkill" hardware. One comment in the first image in this article is complaining about exactly that, and that's what I had a big problem with. Those are all the people that have no use for 1080p ONLY results. Those are the keywords here: 1080p ONLY.
It's a review, right? Test the more unlikely configuration to show best performance, but also test the more likely scenario especially for that type of CPU and GPU combo by upping the resolution my man! There is no need for guesswork. Tim doesn't even touch 1080p monitors in his MUB reviews. We're getting excited about higher refresh rates almost weekly. OLED. HDR. 4K. 8K. 1440p+ is just where the industry is going. Omitting it just doesn't make sense.
Aside from that, I also had the idea of a tech site eventually throwing a popular mainstream build in the mix (included in review/separate video series) with say an i3/i5/R5 + 16GB 3200MHz CL16 / DDR5 6000MHz paired with new low to high end GPU's, and a vice versa with low to high end GPU's? Rigs closer to what the majority actually have. I feel it would help a lot of people with upgrading in addition to standard reviews.
takaozo
Posts: 676 +1,077
"Recently we've started to see a more overwhelming number of comments and complaints about how we test hardware in our reviews -- namely, CPU reviews that look at gaming performance. In this case, we typically focus on low resolution testing (1080p), as this helps to alleviate the potential for GPU bottlenecks, allowing us to more closely look at actual CPU performance."
Why don't you go for 640x480 or 320x240, that is a low resolution to me.
Why don't you go for 640x480 or 320x240, that is a low resolution to me.
Burty117
Posts: 4,781 +3,218
It's a CPU I repeat CPU, I'll repeat again, CPU test. As per this article you're commenting on, which has data to prove it, increasing the resolution makes the GPU a bottleneck, not the CPU, so you're no longer benchmarking a CPU.
In a GPU review? Makes sense to test the different resolutions, it does not in a CPU benchmark.
cldmstrsn
Posts: 229 +225
At this point in time of the internet with everyone being an "expert" I wouldn't even look at comments on a site. I would look at emails that people sent and screen those but past that you don't need to explain to people why you test the way you do cause anyone with half a brain would realize why you do what you do.
Theinsanegamer
Posts: 4,153 +7,457
As a car reviewer, do you go to offroad testing videos and complain that there is no sports car, or that they didnt do a highway test? After all those are really important and lord knows NOBODY has EVER dont a highway test of a jeep.
Or would that be a really ignorant thing to do?
Because that is what you are doing right now, asking "but why only 1080p" while drooling because you cant read the title of the video. IF you want higher rez results they are provided on every single GPU test. If you bothered to Read The ****ing Article you would know this. Reviewers wont include a midrange test bench because then people like you would bleat out "ok but why are you not including the high end CPU tests that would be able to show the best performance and be a more likely scenario", just like you are doing here.
BadThad
Posts: 1,328 +1,602
What a shame you have to waste your time explaining this to the ignorant. Best to simply ignore the noise and don't worry about appeasement. No matter what you write, there will ALWAYS be whiners. In this day and age of "wokeness" the people complaining are the least educated the most likely to not understand most of what you tell them. They complain because they can type...pathetic in my eyes.
hahahanoobs
Posts: 4,840 +2,782
Everything I suggested was in addition what is already being done and I stand by all of it.
Most reviewers were already doing 1440p in CPU tests, because it made sense. Why are people against having more data? Are you implying anything beyond 1080p in a CPU test is useless when most wouldn't be using flagship hardware at such a low res? wtf am I supposed to do with data that is the furthest from my own setup which is closer to the majority? Praise X company for their tremendous achievement at a level I have no interest in? Reviewers can do what they want, but asking to add one more resolution shouldn't be treated as something a noob would say.
Last edited:
Well couple things from my foxhole:
1. If you actually maxed the settings out completely rather than High/Very High the gap would narrow further.
2. The average pc gamer isn't chasing frames; and in the current economy probably cares more about bang for buck. Chasing frames is great for the minority who do so; but for the mass of computer gaming personnel; they aren't doing that.
3. A 2008 i7 920 can still deliver scaling at max settings @3440x1440 even with a 3080Ti; and can even hover around 60FPS average with a 1080Ti. So that means that someone still using an i7 920 can stick to a 2018 GPU and be fine.
4. Demonstrating the difference between cpus is fine in the manner you guys do; but there certainly is value to the more common and budgeted mass of pc gamers to consider how pointless upgrading could be for many as their older CPUs can still scale fine with newer GPUs (or in many cases the need to upgrade either might be unecessary).
Obviously there is a limit once you get so low; Core2Quads are still capable; for example; but you're going to be stuck at 30FPS for most newer games regardless of the GPU. And for all older cpu the implementation of DRM for AC Origins makes it unplayable on certain older CPUs.
1. If you actually maxed the settings out completely rather than High/Very High the gap would narrow further.
2. The average pc gamer isn't chasing frames; and in the current economy probably cares more about bang for buck. Chasing frames is great for the minority who do so; but for the mass of computer gaming personnel; they aren't doing that.
3. A 2008 i7 920 can still deliver scaling at max settings @3440x1440 even with a 3080Ti; and can even hover around 60FPS average with a 1080Ti. So that means that someone still using an i7 920 can stick to a 2018 GPU and be fine.
4. Demonstrating the difference between cpus is fine in the manner you guys do; but there certainly is value to the more common and budgeted mass of pc gamers to consider how pointless upgrading could be for many as their older CPUs can still scale fine with newer GPUs (or in many cases the need to upgrade either might be unecessary).
Obviously there is a limit once you get so low; Core2Quads are still capable; for example; but you're going to be stuck at 30FPS for most newer games regardless of the GPU. And for all older cpu the implementation of DRM for AC Origins makes it unplayable on certain older CPUs.
Last edited:
Burty117
Posts: 4,781 +3,218
Did you actually read the article? I feel like you didn't read it.
Upping the resolution in a CPU test is pointless, as per the difference between the 2700X and 8700K, by increasing the resolution on the best GPU of its day (1080TI but represented by the 3060) all you've done is become GPU bottlenecked and now it looks like both CPU's are near identical.
At 1080p, the 8700K is considerably better. Why is that important? Well if you read the article, the 8700K plays games much better today with today's GPU's and much better in modern games that are more CPU bound. By using 1080p back in the day, you could easily see the 8700k was better, by using a higher resolution, you simply wouldn't know it was the better CPU.
Well couple things from my foxhole:
1. If you actually maxed the settings out completely rather than High/Very High the gap would narrow further.
2. The average pc gamer isn't chasing frames; and in the current economy probably cares more about bang for buck. Chasing frames is great for the minority who do so; but for the mass of computer gaming personnel; they aren't doing that.
3. A 2008 i7 920 can still deliver scaling at max settings @3440x1440 even with a 3080Ti; and can even hover around 60FPS average with a 1080Ti. So that means that someone still using an i7 920 can stick to a 2018 GPU and be fine.
37% of the games made it to 60FPS average on a heavily overclocked i7 920. And all of 13% made it to 75FPS average, suggesting they might keep 1% lows close to 60FPS, which affect the game-immersion-breaking judder and lag spikes.
Even with the 3080 Ti, that OC CPU only exceeds 75FPS in 1/3 the games and in many of those, the 1% lows are very likely to be well below 60FPS simply due to the weak CPU. The problem with that chart is it's missing 1% lows, which are more heavily dependent on CPU performance than average FPS.
Sure, you can pick 'n choose and don't play modern/CPU-demanding games, but time has passed this rig by in almost all cases.
3 GPU upgrades per CPU upgrade seems a bit above normal. It is easier to add a new GPU whereas a CPU upgrade could entail new Mobo and memory in some cases. But, still. I upgrade a GPU about every 3-4 years, system upgrades around the 5-6 yr mark. And now with GPU prices being so ridiculous, I'll bet a lot of people are going to hold cards longer. So 2 upgrades, yeah I could see that. To be fair, I know some people upgrade parts every year but I think we know those are not the majority of users.
Thanks for the article.
Just to point out a flaw in the argument:
Back in the day the differences were small between the i7 8700k and R5 2600x with the GTX 1070. You then measured a 3% difference in 1080p Ultra in AC:O.
You pointed out this was half of what the difference with a 1080Ti was, a card Nvidia 2017 claimed to be the fastest gaming graphics card money can buy.
I understand your point: What then was a narrow difference in gaming performance between two cpus, could become a widening gap with modern GPUs over years to come. So lower resolutions are chosen to inform the reader.
The flaw: How could one calculate the degree of future gap between those two CPUs back then without todays modern GPUs and without todayd modern game engines? It's not possible. The gap could stay narrow, or it could evolve into a serious difference. There are lot's of variables there. This imho disables the whole 8700k 2600x argument in the article.
Another point: Testing in 1080p Ultra does not help your cause (the 3% difference). 1080p medium or low or even 720p would have shown a greater gap between those two CPUs.
Last point about the writing style: I feel it's not the smartest thing to talk about 'Misconceptions' while refering to a portion of your reader base. Obviously you are fighting 'misconceptions' in your own reader base, otherwise it would be pointless to write that emotionally. You don't get that reaction while browsing the comments section of quora. Yes, maybe some of your readers got things wrong. Maybe, as a writer I guess, one can get very upset when digging through some of the comments like in this recent Techspot article:
https://www.techspot.com/review/2615-ryzen-7600-vs-ryzen-5600/
And yes, people sometimes lack experience. Sometimes they are rude. Both can happen simultaneously. But as a professional tech blog your writers should stay above that. Don't start to mansplain benchmarks.
For instance please do not write: "When People Don't Understand CPU Benchmarks, Point Them Here". That's the actual Subtitle of this article. Just think about this sentence for a minute.
I liked Techspots writing style in general up to this point, but this article was not up to your standards imho. It felt like reading a tweet with graphs.
Just pointing these points out to reconsider, of course this is not meant as a harm and is my personal view. Other views may and will differ.
Just to point out a flaw in the argument:
Back in the day the differences were small between the i7 8700k and R5 2600x with the GTX 1070. You then measured a 3% difference in 1080p Ultra in AC:O.
You pointed out this was half of what the difference with a 1080Ti was, a card Nvidia 2017 claimed to be the fastest gaming graphics card money can buy.
I understand your point: What then was a narrow difference in gaming performance between two cpus, could become a widening gap with modern GPUs over years to come. So lower resolutions are chosen to inform the reader.
The flaw: How could one calculate the degree of future gap between those two CPUs back then without todays modern GPUs and without todayd modern game engines? It's not possible. The gap could stay narrow, or it could evolve into a serious difference. There are lot's of variables there. This imho disables the whole 8700k 2600x argument in the article.
Another point: Testing in 1080p Ultra does not help your cause (the 3% difference). 1080p medium or low or even 720p would have shown a greater gap between those two CPUs.
Last point about the writing style: I feel it's not the smartest thing to talk about 'Misconceptions' while refering to a portion of your reader base. Obviously you are fighting 'misconceptions' in your own reader base, otherwise it would be pointless to write that emotionally. You don't get that reaction while browsing the comments section of quora. Yes, maybe some of your readers got things wrong. Maybe, as a writer I guess, one can get very upset when digging through some of the comments like in this recent Techspot article:
https://www.techspot.com/review/2615-ryzen-7600-vs-ryzen-5600/
And yes, people sometimes lack experience. Sometimes they are rude. Both can happen simultaneously. But as a professional tech blog your writers should stay above that. Don't start to mansplain benchmarks.
For instance please do not write: "When People Don't Understand CPU Benchmarks, Point Them Here". That's the actual Subtitle of this article. Just think about this sentence for a minute.
I liked Techspots writing style in general up to this point, but this article was not up to your standards imho. It felt like reading a tweet with graphs.
Just pointing these points out to reconsider, of course this is not meant as a harm and is my personal view. Other views may and will differ.
Last edited:
- Status
Similar threads
Latest posts
-
AMD Ryzen 7 7800X3D vs 5800X3D: Zen 3 or Zen 4 3D V-Cache- godrilla replied
