13K of GPU Test Methodology: Airflow Photography, RTX Games, Pressure Tests, Power, & More
video description
Date: 2020-09-14
Related videos
Comments and reviews: 10
7thNoodle
I'm a game developer and I really do feel there's not much benchmarking being done within Game engines.
I work with both Unity and Unreal engine and my purchasing decisions would be SO much more clear if there was any testing done for my field,
most tests are done in-house in big studios and I kinda have to ask around or guess.
Game engines are interesting, especially in Unreal engine, there's multiple passes to be done in order to Build Static lighting for a scene and there's alot of nuance in trying to get Ray-Tracing features working. So far the closest thing I could draw a comparison to are there blender tests, but it's not exactly the same.
And this is not necessarily only about GPUs though, baking lights or building a project puts strain on the whole system, so methodology would probably be closer to the way you test game performance.
Here are some scenarios I can think of to get the ball rolling:
Get you a very nice, high fidelity level. Unreal engine has a bunch of free content for developers on its marketplace and you can build a pretty complex scene that can really strain a system.
You could approach this like people approached Crysis for benchmarks all these years, get you a chungus level and more fps=more better.
But with some isolation of engine functionality can get some more interesting results going.
1. Baking static lighting for a level, this puts a strain mostly on memory and CPU. GPU is secondary here. But getting some useful metrics about how a Ryzen with infinity fabric OC 1:1 would perform versus an Intel would be amazing and I've yet to see something like that anywhere. Note: My last bake at maximum quality for my level took 24 hours with my shitty rig, I don't imagine that's what most studios deal with though if they have current gear, there's been major architecture changes the past decade or even 2-3 years, putting those changes into numbers through bake-time would be awesome to see and very informative for me!
2. Build time. Building a project into a packaged build is a deterministic process and can be measured in seconds as long as you can make sure that the hard drive you're packaging to is not affecting results. This is a very useful metric for QA engineers and Build engineers. Packaging and re-packaging the project thousands of times is guaranteed and just like in video production, optimizing a render-machine to not waste your staff's time in useless coffe-breaks because it's still building is something many studios would be interested in. Afaik, this is CPU dependent mostly.
3. Build / In - Editor FPS. This is a pretty simple game-like benchmark. Get in that level and test. How well does it run? What's the frame timings like? Note that there's always a performance difference between the in-editor level (which is where developers spend their time) and the packaged/build level. This will be what players spend their time in and much closer to a game benchmark. Still, there's interesting comparisons to be drawn when considering the delta between in-editor and packaged build for different components. Luckily unreal also supplies a bunch of command-line performance tools and visualizers, so you can really go into detail for a scene's performance.
4. Ray-Tracing calculations. UE4 has built-in functionality and libraries to handle Ray-Traced Lighting, using the same level as before you could very easily get a useful metric for GPU performance with FPS. Note that there's also a difference here for in-editor and packaged builds.
5+ There's also so many more things you can test in game-engines: Baking materials, Simulation, mass drawing of actors, spawning and destroying large volumes of actors, complex animation blueprint performance impact, blueprint performance impact. The list goes on!
I would LOVE it if you guys could find a way to include UE4 or any type of game engine into your benchmarking. It's not really been done from what I've seen and with more and more people fiddling around with all these free tools, I'm sure there's an audience there to be interested in it. If not, I'm still pretty sure you'll be getting a useful metric for game performance eitherway.
If anyone from GN wants to reach out for a chat I'd be honored to help get this rolling. Steve also mentioned that he has some game devleoper contacts in the industry, so I'm pretty sure if you could talk abit with a technical director from a large studio, they could really fill you in and find a way to introduce game engine testing as a useful metric.
Snowflake if you're reading this, I love you, don't fire Steve yet, he's doing pretty good!
reply
I'm a game developer and I really do feel there's not much benchmarking being done within Game engines.
I work with both Unity and Unreal engine and my purchasing decisions would be SO much more clear if there was any testing done for my field,
most tests are done in-house in big studios and I kinda have to ask around or guess.
Game engines are interesting, especially in Unreal engine, there's multiple passes to be done in order to Build Static lighting for a scene and there's alot of nuance in trying to get Ray-Tracing features working. So far the closest thing I could draw a comparison to are there blender tests, but it's not exactly the same.
And this is not necessarily only about GPUs though, baking lights or building a project puts strain on the whole system, so methodology would probably be closer to the way you test game performance.
Here are some scenarios I can think of to get the ball rolling:
Get you a very nice, high fidelity level. Unreal engine has a bunch of free content for developers on its marketplace and you can build a pretty complex scene that can really strain a system.
You could approach this like people approached Crysis for benchmarks all these years, get you a chungus level and more fps=more better.
But with some isolation of engine functionality can get some more interesting results going.
1. Baking static lighting for a level, this puts a strain mostly on memory and CPU. GPU is secondary here. But getting some useful metrics about how a Ryzen with infinity fabric OC 1:1 would perform versus an Intel would be amazing and I've yet to see something like that anywhere. Note: My last bake at maximum quality for my level took 24 hours with my shitty rig, I don't imagine that's what most studios deal with though if they have current gear, there's been major architecture changes the past decade or even 2-3 years, putting those changes into numbers through bake-time would be awesome to see and very informative for me!
2. Build time. Building a project into a packaged build is a deterministic process and can be measured in seconds as long as you can make sure that the hard drive you're packaging to is not affecting results. This is a very useful metric for QA engineers and Build engineers. Packaging and re-packaging the project thousands of times is guaranteed and just like in video production, optimizing a render-machine to not waste your staff's time in useless coffe-breaks because it's still building is something many studios would be interested in. Afaik, this is CPU dependent mostly.
3. Build / In - Editor FPS. This is a pretty simple game-like benchmark. Get in that level and test. How well does it run? What's the frame timings like? Note that there's always a performance difference between the in-editor level (which is where developers spend their time) and the packaged/build level. This will be what players spend their time in and much closer to a game benchmark. Still, there's interesting comparisons to be drawn when considering the delta between in-editor and packaged build for different components. Luckily unreal also supplies a bunch of command-line performance tools and visualizers, so you can really go into detail for a scene's performance.
4. Ray-Tracing calculations. UE4 has built-in functionality and libraries to handle Ray-Traced Lighting, using the same level as before you could very easily get a useful metric for GPU performance with FPS. Note that there's also a difference here for in-editor and packaged builds.
5+ There's also so many more things you can test in game-engines: Baking materials, Simulation, mass drawing of actors, spawning and destroying large volumes of actors, complex animation blueprint performance impact, blueprint performance impact. The list goes on!
I would LOVE it if you guys could find a way to include UE4 or any type of game engine into your benchmarking. It's not really been done from what I've seen and with more and more people fiddling around with all these free tools, I'm sure there's an audience there to be interested in it. If not, I'm still pretty sure you'll be getting a useful metric for game performance eitherway.
If anyone from GN wants to reach out for a chat I'd be honored to help get this rolling. Steve also mentioned that he has some game devleoper contacts in the industry, so I'm pretty sure if you could talk abit with a technical director from a large studio, they could really fill you in and find a way to introduce game engine testing as a useful metric.
Snowflake if you're reading this, I love you, don't fire Steve yet, he's doing pretty good!
reply
Radcaster
I have a workload testing idea, but unfortunately not for GPUs.
I'm an audio engineer/sound designer/composer and I regularly run into a lot of very bad advice online regarding what hardware is necessary/useful when it comes to home studio set-ups. This kind of set-up has become increasingly popular over the last 10 years or so, as the evolution of DAWs and digital plugins makes creating and producing music from home very accessible. Unfortunately, general computer illiteracy is rampant in the community, and regularly people will make claims that you MUST HAVE the newest, most powerful Intel processor available or live in high-latency, low stability land.
I have found this to be very incorrect, or at least inconsistent, as CPUs have gotten so much more powerful recently but the workloads have remained the same. My personal testing with an old i5-4590 vs a newer i5-8600k has resulted in very minor differences in performance, but I do not have the means to produce a proper experiment and results. There are a few things that would be incredibly useful here - the big ones being latency and workload efficiency. With latency for audio recording we're looking at an optimal 0.125ms for USB 2.0, which most audio interfaces run on, but DAWs report a latency of around 10-25ms roundtrip (from input to headphones/monitors). There is contention as to what causes this, and ideally getting as close to no latency is preferred, though anything under around 10ms becomes practically imperceptible. For the workload itself, we're looking at the maximum number of tracks/plugins that can run simultaneously in one instance whilst recording or bounding/rendering - for this I think we would need to figure out the breaking point and work backwards to create a stable workload template that could be tested on multiple platforms.
I would love to see how different CPUs, RAM speeds, storage types, etc. affect the performance of audio workloads in these ways and more. My suspicion is that the newer Ryzen CPUs may well compete in latency, but Intel's relatively better single-thread performance may be more beneficial to this kind of workload. I would also expect to see lower-end CPUs compete to a much better degree than the community might think. The biggest challenge in testing would be coming to any conclusion that didn't involve a lot of depends on your workload results.
I realise this is kind of a long-shot, especially as it is unlikely you guys have the time to be messing with this kind of thing, but if anyone could provide concrete results to put an end to years of speculation and dodgy logic, it's GN.
reply
I have a workload testing idea, but unfortunately not for GPUs.
I'm an audio engineer/sound designer/composer and I regularly run into a lot of very bad advice online regarding what hardware is necessary/useful when it comes to home studio set-ups. This kind of set-up has become increasingly popular over the last 10 years or so, as the evolution of DAWs and digital plugins makes creating and producing music from home very accessible. Unfortunately, general computer illiteracy is rampant in the community, and regularly people will make claims that you MUST HAVE the newest, most powerful Intel processor available or live in high-latency, low stability land.
I have found this to be very incorrect, or at least inconsistent, as CPUs have gotten so much more powerful recently but the workloads have remained the same. My personal testing with an old i5-4590 vs a newer i5-8600k has resulted in very minor differences in performance, but I do not have the means to produce a proper experiment and results. There are a few things that would be incredibly useful here - the big ones being latency and workload efficiency. With latency for audio recording we're looking at an optimal 0.125ms for USB 2.0, which most audio interfaces run on, but DAWs report a latency of around 10-25ms roundtrip (from input to headphones/monitors). There is contention as to what causes this, and ideally getting as close to no latency is preferred, though anything under around 10ms becomes practically imperceptible. For the workload itself, we're looking at the maximum number of tracks/plugins that can run simultaneously in one instance whilst recording or bounding/rendering - for this I think we would need to figure out the breaking point and work backwards to create a stable workload template that could be tested on multiple platforms.
I would love to see how different CPUs, RAM speeds, storage types, etc. affect the performance of audio workloads in these ways and more. My suspicion is that the newer Ryzen CPUs may well compete in latency, but Intel's relatively better single-thread performance may be more beneficial to this kind of workload. I would also expect to see lower-end CPUs compete to a much better degree than the community might think. The biggest challenge in testing would be coming to any conclusion that didn't involve a lot of depends on your workload results.
I realise this is kind of a long-shot, especially as it is unlikely you guys have the time to be messing with this kind of thing, but if anyone could provide concrete results to put an end to years of speculation and dodgy logic, it's GN.
reply
Chris
In answer to the workstation tests question it would be nice to see how a GPU handles viewport performance for high-poly models. It's all well and good having a snappy interface if the model you're working with is only 100k polys; but how about CAD models with 60+ million polys? You could also try rendering with different engines like Octane or Vray GPU to get a full picture of commonly used engines.
Another area to explore would be Unreal Engine; tests could be as simple as frame rates for densely populated scenes, particle effects with Niagra or exported render speeds; all of which could be tested with RTX on and off (thinking about it you could build a really interesting set of custom tests in UE4).
There's also fluid simulation with some of the GPU accelerated tools like Realflow or Houdini (although that's not really an area I know much about, just know that those ones utilise the GPU).
reply
In answer to the workstation tests question it would be nice to see how a GPU handles viewport performance for high-poly models. It's all well and good having a snappy interface if the model you're working with is only 100k polys; but how about CAD models with 60+ million polys? You could also try rendering with different engines like Octane or Vray GPU to get a full picture of commonly used engines.
Another area to explore would be Unreal Engine; tests could be as simple as frame rates for densely populated scenes, particle effects with Niagra or exported render speeds; all of which could be tested with RTX on and off (thinking about it you could build a really interesting set of custom tests in UE4).
There's also fluid simulation with some of the GPU accelerated tools like Realflow or Houdini (although that's not really an area I know much about, just know that those ones utilise the GPU).
reply
Sebastian
If you guys weren't already slammed with work I'd ask that you look into Forza Motorsport 7. This game has some weird stuttering behavior that gets worse over time.
E.G.: at the start of a race I'm running at 160fps. 10 minutes in I'm somewhere around 100 and have noticeable frame time spikes. After 20 minutes my average is down to 60-70fps and it's basically unplayable, due to frame time spikes that are north of 150ms. Only a complete reload of the track and race resets this behavior. Just restarting the race itself does not change anything.
My temps are fine and cpu & ram load stay the same during the whole race. It just degrades in performance. (game only uses 8 threads and gpu load drops with time, due to the cpu not keeping up. might be a timing issue of internal processes, but I have no clue how to solve or validate this)
(I'm running a 3900x, 32GB ram and an rtx2070)
reply
If you guys weren't already slammed with work I'd ask that you look into Forza Motorsport 7. This game has some weird stuttering behavior that gets worse over time.
E.G.: at the start of a race I'm running at 160fps. 10 minutes in I'm somewhere around 100 and have noticeable frame time spikes. After 20 minutes my average is down to 60-70fps and it's basically unplayable, due to frame time spikes that are north of 150ms. Only a complete reload of the track and race resets this behavior. Just restarting the race itself does not change anything.
My temps are fine and cpu & ram load stay the same during the whole race. It just degrades in performance. (game only uses 8 threads and gpu load drops with time, due to the cpu not keeping up. might be a timing issue of internal processes, but I have no clue how to solve or validate this)
(I'm running a 3900x, 32GB ram and an rtx2070)
reply
TheAnoniemo
That's a lot of interesting testing to look forward too! I would like to share my concern that adding more and more data and graphs to reviews, while adding possibly useful information, can have a negative effect on the viewing of of video content. I personally get disconnected pretty quickly when there's graphs after graphs of game fps data all showing a very similar performance picture. Perhaps you can look into better ways to consolidate data, for example how Hardware Unboxed has the composite graphs showing performance for a whole list of games in one graph, offering a quick overview of relative performance.
I don't want to be negative about all your testing, it's amazing and really appreciated, I just think it could reach more people if it were presented better sometimes.
reply
That's a lot of interesting testing to look forward too! I would like to share my concern that adding more and more data and graphs to reviews, while adding possibly useful information, can have a negative effect on the viewing of of video content. I personally get disconnected pretty quickly when there's graphs after graphs of game fps data all showing a very similar performance picture. Perhaps you can look into better ways to consolidate data, for example how Hardware Unboxed has the composite graphs showing performance for a whole list of games in one graph, offering a quick overview of relative performance.
I don't want to be negative about all your testing, it's amazing and really appreciated, I just think it could reach more people if it were presented better sometimes.
reply
Guber
Hello! I am a professional CAD drafter! I would love to see 2 programs benchmarked: Autodesk Revit, a modeling and BIM program, as well as AutoCAD. I personally use Autodesk AutoCAD for an interior glass company. The company, and me personally, have used it to do shop drawings for the interior glass for Smithsonian museums, cancer research centers, hotels, etc. I would love to see a benchmark of its performance, both 2D and 3D.
AutoCAD is a bit outdated in terms of CPU usage, but i never see reviews of its performance in regards to GPUs. Revit is not as outdated. Many people in the construction industry use these programs, and i never really see benchmarks of them. I know quite a few people who would really appreciate seeing GPU impact on rendering performance.
reply
Hello! I am a professional CAD drafter! I would love to see 2 programs benchmarked: Autodesk Revit, a modeling and BIM program, as well as AutoCAD. I personally use Autodesk AutoCAD for an interior glass company. The company, and me personally, have used it to do shop drawings for the interior glass for Smithsonian museums, cancer research centers, hotels, etc. I would love to see a benchmark of its performance, both 2D and 3D.
AutoCAD is a bit outdated in terms of CPU usage, but i never see reviews of its performance in regards to GPUs. Revit is not as outdated. Many people in the construction industry use these programs, and i never really see benchmarks of them. I know quite a few people who would really appreciate seeing GPU impact on rendering performance.
reply
walentaz
Honestly - I find it totally pointless (from consumer perspective). Either GPU has good cooling or not. When dealing with bad cooling GPU - whatever reason it would be for doesn't make it any less shit, because if it's shit - it's shit and it's not worth buying regardless of why it's shit.
Also Rainbow Six Siege has VULKAN and it's better!
Ditch that unoptimized shit of a Hitman 2 with abysmal 1% lows on any h/w.
Also ditch Division 2 for Death Stranding.
reply
Honestly - I find it totally pointless (from consumer perspective). Either GPU has good cooling or not. When dealing with bad cooling GPU - whatever reason it would be for doesn't make it any less shit, because if it's shit - it's shit and it's not worth buying regardless of why it's shit.
Also Rainbow Six Siege has VULKAN and it's better!
Ditch that unoptimized shit of a Hitman 2 with abysmal 1% lows on any h/w.
Also ditch Division 2 for Death Stranding.
reply
Ethan
My old roommate is a physicist. His excuse for building a gaming desktop is driven by machine learning model compilation (and then also gaming). He has access to a super computer at school, but it's very timebox for students and requires special permissions. Basically it's not possible to build and test against it. A nice test would be to install Python and set up pip and then set up the GPU installation of Tensorflow and try to compile some of their base projects.
reply
My old roommate is a physicist. His excuse for building a gaming desktop is driven by machine learning model compilation (and then also gaming). He has access to a super computer at school, but it's very timebox for students and requires special permissions. Basically it's not possible to build and test against it. A nice test would be to install Python and set up pip and then set up the GPU installation of Tensorflow and try to compile some of their base projects.
reply
EVPointMaster
Will you add Cyberpunk to the list of games you test (with or without RTX&DLSS)?
I think it would also be interesting to see CPU tests with different APIs, especially games that support DX11 and DX12.
DLSS needs to be tested on a per game basis, because implementations vary wildly and also impacts image quality in different ways. I think Digital Foundry already does a great job testing that.
reply
Will you add Cyberpunk to the list of games you test (with or without RTX&DLSS)?
I think it would also be interesting to see CPU tests with different APIs, especially games that support DX11 and DX12.
DLSS needs to be tested on a per game basis, because implementations vary wildly and also impacts image quality in different ways. I think Digital Foundry already does a great job testing that.
reply
auwdioslave
Regarding under respresented benchmarks (CPU based, unfortunately not GPU based): I would love to see some CPU benchmarks with SPICE electronics simulation. I'm a power electronics (PSU) engineer and use the LTspice simulator on a daily basis. Simulations can easily take up to half an hour or longer. It would be great to see which architecture would work best with SPICE simulation!
reply
Regarding under respresented benchmarks (CPU based, unfortunately not GPU based): I would love to see some CPU benchmarks with SPICE electronics simulation. I'm a power electronics (PSU) engineer and use the LTspice simulator on a daily basis. Simulations can easily take up to half an hour or longer. It would be great to see which architecture would work best with SPICE simulation!
reply
Add a review, comment
Other channel videos
