NOT ENOUGH SAMPLES - Benchmark of CPU Sample Size ft. 68 CPUs, HUB, & Der8auer
video description
Date: 2023-08-31
Related videos
Comments and reviews: 20
Reda-Ou
I hate to be a pedant but as a scientist I will say this isn't quite convincing... I know it's not really feasible at all, so I don't at all criticise you for using single samples. It's just too expensive and time consuming. So this isn't a criticism of your reviews or your testing process, just of your conclusion in this particular video.
With that said... the stdev in the hardware performance across a particular product will likely only be valid for the particular product (or potentially only a particular production run for a particular product, although its possible your current 20 samples already sample over multiple production runs). Even if we imagine an extreme example where 50% of the field is low stddev and 50% of the field is high stddev, it's not unlikely that the 4 CPUs you've chosen will fall into the low stddev field, and therefore the data you've collected is not strong evidence that the entire market can be well characterised by a single sample.
In a situation where the distribution of standard deviations of product performance isn't as clear cut as that, you'd need to test far far more products to get a good sense of what the entire field looks like. To put it another way, there is the standard deviation within a particular product (or a particular production run) but then there is also the meta standard deviation of standard deviations among all products (and that will often be a complex function of a lot of variables including time, as a manufacturer's processes either degrade or improve).
I want to be clear here that I'm not saying you need to do large sample sizes all the time, or that your conclusion is definitely wrong -- it could be right! In fact, I think you're totally fine to do single samples purely from a feasibility point of view. I'm just saying this isn't convincing for the particular conclusion you reach in this video. What you have convinced me of is that for the 4 CPU models you showed in this test, only one sample is needed to give a good sense (at a consumer level) of what the performance is.
And not to make things more complex... but all of your sample runs were quite 'light' on the CPUs (in the sense that the FPS was very high, rather than say 30-60fps) and the stddev will also be a function of the loading (as to how strong the dependence will be, no idea, it could be quite minimal).
That being said, you may want to tighten up your language a bit about some of the explanations you gave, such as using the phrase system noise . What does that mean? I'm not a computer hardware person, I'm a physicist, so maybe you mean something completely different to what I'm understanding you're saying. But noise should generally be an unbiased error (i.e. symmetric about zero deviation) and noise-like effects should disappear through your sampling of multiple passes (and your standard error should reflect this, so measurements should generally be within error bars of each other). It's hard to tell since your charts showing the consistency don't include error bars, but it seems likely to me that the couple% deviations you're showing are likely to be real hardware deviations and not system noise .
reply
I hate to be a pedant but as a scientist I will say this isn't quite convincing... I know it's not really feasible at all, so I don't at all criticise you for using single samples. It's just too expensive and time consuming. So this isn't a criticism of your reviews or your testing process, just of your conclusion in this particular video.
With that said... the stdev in the hardware performance across a particular product will likely only be valid for the particular product (or potentially only a particular production run for a particular product, although its possible your current 20 samples already sample over multiple production runs). Even if we imagine an extreme example where 50% of the field is low stddev and 50% of the field is high stddev, it's not unlikely that the 4 CPUs you've chosen will fall into the low stddev field, and therefore the data you've collected is not strong evidence that the entire market can be well characterised by a single sample.
In a situation where the distribution of standard deviations of product performance isn't as clear cut as that, you'd need to test far far more products to get a good sense of what the entire field looks like. To put it another way, there is the standard deviation within a particular product (or a particular production run) but then there is also the meta standard deviation of standard deviations among all products (and that will often be a complex function of a lot of variables including time, as a manufacturer's processes either degrade or improve).
I want to be clear here that I'm not saying you need to do large sample sizes all the time, or that your conclusion is definitely wrong -- it could be right! In fact, I think you're totally fine to do single samples purely from a feasibility point of view. I'm just saying this isn't convincing for the particular conclusion you reach in this video. What you have convinced me of is that for the 4 CPU models you showed in this test, only one sample is needed to give a good sense (at a consumer level) of what the performance is.
And not to make things more complex... but all of your sample runs were quite 'light' on the CPUs (in the sense that the FPS was very high, rather than say 30-60fps) and the stddev will also be a function of the loading (as to how strong the dependence will be, no idea, it could be quite minimal).
That being said, you may want to tighten up your language a bit about some of the explanations you gave, such as using the phrase system noise . What does that mean? I'm not a computer hardware person, I'm a physicist, so maybe you mean something completely different to what I'm understanding you're saying. But noise should generally be an unbiased error (i.e. symmetric about zero deviation) and noise-like effects should disappear through your sampling of multiple passes (and your standard error should reflect this, so measurements should generally be within error bars of each other). It's hard to tell since your charts showing the consistency don't include error bars, but it seems likely to me that the couple% deviations you're showing are likely to be real hardware deviations and not system noise .
reply
blingleblorb
Sample size is a weird subject in statistics. While ideally you should have as many samples as possible, most of the time it s not viable. Even if you had the money, you have to run through multiple tests for all of them and it just takes forever
There s a lot of statistical techniques to get around this. Mathematics can t get past the needs of the real world. Factoring in time for tests is very important, and building your testing and analysis around n=1 can still give good results. I ve run into this problem with chemical analysis. 5-10 hours per sample means relying on tests and analyses designed with n=1 (technically it s just multiple samples of tests for each sample when it comes to math)
Basically if your main issue with tech analyses on the internet is sample size you probably have never actually done statistics with real examples
reply
Sample size is a weird subject in statistics. While ideally you should have as many samples as possible, most of the time it s not viable. Even if you had the money, you have to run through multiple tests for all of them and it just takes forever
There s a lot of statistical techniques to get around this. Mathematics can t get past the needs of the real world. Factoring in time for tests is very important, and building your testing and analysis around n=1 can still give good results. I ve run into this problem with chemical analysis. 5-10 hours per sample means relying on tests and analyses designed with n=1 (technically it s just multiple samples of tests for each sample when it comes to math)
Basically if your main issue with tech analyses on the internet is sample size you probably have never actually done statistics with real examples
reply
itech
Here's the one thing additional I wanted to see in the video. On a per game basis take the highest and lowest ranked cpu for that game and then instead of 4 more runs do like 20 runs in order to get the 20+ run variance specifically for each of those 2 specific cpus. Then you can show here's the variance across this batch of cpu's we tested and then we took the extremes (which weren't actually particularly extreme) and now here's the variance for those specific 2 cpus across 5x the number of runs. Which presumably would show that even taking your edge results and pounding on them still doesn't reveal anything of note (which is presumably where you would find something if there was anything there to find).
reply
Here's the one thing additional I wanted to see in the video. On a per game basis take the highest and lowest ranked cpu for that game and then instead of 4 more runs do like 20 runs in order to get the 20+ run variance specifically for each of those 2 specific cpus. Then you can show here's the variance across this batch of cpu's we tested and then we took the extremes (which weren't actually particularly extreme) and now here's the variance for those specific 2 cpus across 5x the number of runs. Which presumably would show that even taking your edge results and pounding on them still doesn't reveal anything of note (which is presumably where you would find something if there was anything there to find).
reply
Nelson
This discussion is moot, I am sorry you got stuck on that comment because I know that ate your brain for a while because no numbers , I have the same issue. Still, even with the same build, case, graphics, drivers, all a 3% to 6% difference is expected, there are too many variables, power flutuations on site, airflow, pressure, locations, loads. picking a unit and float it 5% (high balling it) is expected. I made 3 exactly equal systems, all 3 are different. Same performance approximatly but they deviate. this is only common sense. #Excelthoughtitwasadate
Still this was fun :P
reply
This discussion is moot, I am sorry you got stuck on that comment because I know that ate your brain for a while because no numbers , I have the same issue. Still, even with the same build, case, graphics, drivers, all a 3% to 6% difference is expected, there are too many variables, power flutuations on site, airflow, pressure, locations, loads. picking a unit and float it 5% (high balling it) is expected. I made 3 exactly equal systems, all 3 are different. Same performance approximatly but they deviate. this is only common sense. #Excelthoughtitwasadate
Still this was fun :P
reply
Cybernetic
I think you're right that one sample is sufficient to test the performance of a CPU within 1%, but if you had two products very close in performance, that 4% total swing of variance between two models added together is impactful.
It really only matters in close races, but it's still kind of relevant to know as a consumer that the delta can be altered by up to 4% depending on how each side is binned in a single-unit benchmark comparison.
reply
I think you're right that one sample is sufficient to test the performance of a CPU within 1%, but if you had two products very close in performance, that 4% total swing of variance between two models added together is impactful.
It really only matters in close races, but it's still kind of relevant to know as a consumer that the delta can be altered by up to 4% depending on how each side is binned in a single-unit benchmark comparison.
reply
Andy
Not to nitpick, but it would've been interesting to distill each run in each game to a percentage of the average for that game, then average all the percentages together, and only then rank the CPUs and try to determine whet the deviation was across all games. I doubt it'd show anything other than the 1-2 anomalously good or bad CPUs, but it'd be a better visualization than just straight average of rank.
reply
Not to nitpick, but it would've been interesting to distill each run in each game to a percentage of the average for that game, then average all the percentages together, and only then rank the CPUs and try to determine whet the deviation was across all games. I doubt it'd show anything other than the 1-2 anomalously good or bad CPUs, but it'd be a better visualization than just straight average of rank.
reply
JoN
As always, thank you and amazing work from you guys! That must have been a lot of work and sweat!! Thanks to Tim and Roman too for their participation too!
Ps: You guys seem to always have a good laugh when together! That would actually be fun to see you all doing something completely different together! (like an outdoor activity or anything else, it doesn't have to be about IT at all).
reply
As always, thank you and amazing work from you guys! That must have been a lot of work and sweat!! Thanks to Tim and Roman too for their participation too!
Ps: You guys seem to always have a good laugh when together! That would actually be fun to see you all doing something completely different together! (like an outdoor activity or anything else, it doesn't have to be about IT at all).
reply
Kay
Guys we all know what ppl like greg will say now: 68 cpus isnt a representative group size. You need a way bigger sample size to be statistically significant
And i dont really know who's supposed to actually test numbers in the 1000s or something until these guys will be happy and not make it their whole business model like the manufacturers or (rip) silicon lottery...
reply
Guys we all know what ppl like greg will say now: 68 cpus isnt a representative group size. You need a way bigger sample size to be statistically significant
And i dont really know who's supposed to actually test numbers in the 1000s or something until these guys will be happy and not make it their whole business model like the manufacturers or (rip) silicon lottery...
reply
Vincent
The reverse psychology Greg and Challenge accepted Steve content. For pointing out for Greg's the particular over the general view of the product, so he (as well as us) could have better bang for your buck. I think Greg forgot it was silicon lottery for every consumer cpu. And just want to properly justify his purchase because he is on a tight budget due to inflation.
reply
The reverse psychology Greg and Challenge accepted Steve content. For pointing out for Greg's the particular over the general view of the product, so he (as well as us) could have better bang for your buck. I think Greg forgot it was silicon lottery for every consumer cpu. And just want to properly justify his purchase because he is on a tight budget due to inflation.
reply
devilmikey00
Do people really think that companies are selling CPU's with massively huge variance from unit to unit out of the box? Do they think we wouldn't notice if that was the case? Do they not think the redditors wouldn't notice if massive swaths of people were all reporting hugely different numbers from the same product? That would be a massive controversy.
reply
Do people really think that companies are selling CPU's with massively huge variance from unit to unit out of the box? Do they think we wouldn't notice if that was the case? Do they not think the redditors wouldn't notice if massive swaths of people were all reporting hugely different numbers from the same product? That would be a massive controversy.
reply
itech
Benchmarks, drivers and OS are way more unscientific than CPUs.
If you're not running a real-time OS (hint: you're NOT), IT'S UNSCIENTIFIC.
Whoever wrote that comment about CPU sample size has no understanding of how to evaluate variable space.
Meaningful variables were never the hardware (with historical exceptions), it was always the software.
reply
Benchmarks, drivers and OS are way more unscientific than CPUs.
If you're not running a real-time OS (hint: you're NOT), IT'S UNSCIENTIFIC.
Whoever wrote that comment about CPU sample size has no understanding of how to evaluate variable space.
Meaningful variables were never the hardware (with historical exceptions), it was always the software.
reply
Karkarov
I legit work at a billion dollar company where we develop our own hardware. We would never even consider going this deep with our testing, it would be a monumental waste of time and money. Hell we have a cpu stress test we just run for like 1 hour and we dont even care if there are instances of failure as long as the hour averages pass.
reply
I legit work at a billion dollar company where we develop our own hardware. We would never even consider going this deep with our testing, it would be a monumental waste of time and money. Hell we have a cpu stress test we just run for like 1 hour and we dont even care if there are instances of failure as long as the hour averages pass.
reply
Vash
I find it weird to be focused on sample size when I've always considered reviews to be akin to the average end user experience where they get that one shot at a sample and stick with what they get. The only caveat would be if the reviewers are getting binned chips for the review, but that runs down conspiracy road, and we don't drive there.
reply
I find it weird to be focused on sample size when I've always considered reviews to be akin to the average end user experience where they get that one shot at a sample and stick with what they get. The only caveat would be if the reviewers are getting binned chips for the review, but that runs down conspiracy road, and we don't drive there.
reply
xO-Will
As a stats grad, I love to see this. Yes. Lol one person learns about sample size and then everyone starts flipping some shit over it thinking it applies to all scenarios, when actually having an appropriate sample size is debatable depending on your use case. I love to see this in action!
reply
As a stats grad, I love to see this. Yes. Lol one person learns about sample size and then everyone starts flipping some shit over it thinking it applies to all scenarios, when actually having an appropriate sample size is debatable depending on your use case. I love to see this in action!
reply
misterringer
Silicon Lottery closing up shop indirectly predicts these results. There just wasn't enough variance in CPU's anymore for their model to be profitable.
Perhaps on a brand new node, that variance might be a little wider (pure speculation) but still probably not significant anymore.
reply
Silicon Lottery closing up shop indirectly predicts these results. There just wasn't enough variance in CPU's anymore for their model to be profitable.
Perhaps on a brand new node, that variance might be a little wider (pure speculation) but still probably not significant anymore.
reply
Annagram
Am I bad if I saw an opportunity to use the Q-test at 16:05? Too geeky? Probably. (The outlier can be rejected at 95% confidence, BTW. :) ). The larger story is people generally don't understand statistics, but it's good to see you're doing your homework behind the scenes. Great work.
reply
Am I bad if I saw an opportunity to use the Q-test at 16:05? Too geeky? Probably. (The outlier can be rejected at 95% confidence, BTW. :) ). The larger story is people generally don't understand statistics, but it's good to see you're doing your homework behind the scenes. Great work.
reply
djstr0b3
Do you guys normally disable windows services that can impact the performance? E.g. Scheduled tasks, Windows update, background intelligent transfer service, and other services that are there to update stuff like Adobe components, Chrome etc?
Great video guys! Keep it up
reply
Do you guys normally disable windows services that can impact the performance? E.g. Scheduled tasks, Windows update, background intelligent transfer service, and other services that are there to update stuff like Adobe components, Chrome etc?
Great video guys! Keep it up
reply
Adnel
That question is MOOT for default behavior because, the hardware is guaranteed to run at such specs, but I can see the value of the question if you are talking overclocking, undervolting and any other optimization you can throw at them to squeeze all the performance you can.
reply
That question is MOOT for default behavior because, the hardware is guaranteed to run at such specs, but I can see the value of the question if you are talking overclocking, undervolting and any other optimization you can throw at them to squeeze all the performance you can.
reply
mina86
Seems to me that if one wants to complain about sample size, it should to suggest that reviewer should test multiple products and then show the worst performing one in the review (rather than trying to complain that one sample doesn t give meaningful mean).
reply
Seems to me that if one wants to complain about sample size, it should to suggest that reviewer should test multiple products and then show the worst performing one in the review (rather than trying to complain that one sample doesn t give meaningful mean).
reply
Blissy1175
Never really expected a test like this to ever happen, mostly because the variable difference in possible silicon quality isn't big enough to create a +15% delta in results that would skew a review in any given direction, but this is still really cool.
reply
Never really expected a test like this to ever happen, mostly because the variable difference in possible silicon quality isn't big enough to create a +15% delta in results that would skew a review in any given direction, but this is still really cool.
reply
Add a review, comment
Other channel videos
