AMD Cypress: A Driver Performance Review

As many of you know, drivers can play an important part in how well our computer components interact with each other. Changes in the code can have drastic effects. More often than not, the changes either make the product start working for you or make the product stop working for you. For video cards however, oftentimes features that were previously unavailable are added. For example, our review of the Catalyst 10.2 and 10.3 beta releases showed that AMD had enabled quad buffered stereoscopic display and the Ultra Low Power State for slave cards across the Radeon 5000 series. On rare occasions, the performance of the base functions of the device is increased.

This last idea is most noticeable in video card drivers which have been known to substantially impact performance. As we have seen in the RV870 story, the life of a GPU begins long before it is ever launched. But the launch of a chip is not the end of the road. Driver support must be maintained for the life of the chip. Furthermore, the support cannot just be improving compatibility issues. Video card purchasers demand that they get more out of their cards with new driver releases. Thus, video card manufacturers must vigilantly search out optimizations for their code in an effort to at least maintain parity with their competitors and appease their customers.

These vigilant searches sometimes lead to a focus on optimizations solely for synthetic benchmarks with no real world enhancement realized by any program but the benchmark. Many of us have seen video driver after video drive improve performance in 3DMark by a percentage or two, with no impact on real games as many synthetic benchmarks do not contain rendering code similar enough to actual video games, or rendering code several years ahead of games. However, because of the prevalence of said benchmarks and the importance still placed on them for bragging rights, video card manufacturers are stuck spending resources optimizing code for a program that will produce no real world benefit. Thus, we constantly need to answer one basic question: to what degree are optimizations in driver code made purely to boost synthetic benchmark scores? We’ve seen a significant disparity between the two happen in the past, and chances are it may still be happening today.

The Candidates

For this driver review, we tests all eight of the major driver releases for the Radeon 5800 series (Cypress or RV870 chips if you prefer) from the series launch through early May. We are going to mention the most relevant features added in each new revision along with any performance enhancements for programs we chose to test. 

Launch Drivers
The launch drivers do not carry a Catalyst number as they were what were sent to AnandTech by AMD for review and released to buyers for the launch of the new chip. The driver number is 8.660 and there are no obtainable release notes for this driver. This driver set is the base against which all the later revisions are compared.

Catalyst 9.10
Released on 10/22/2009 the Catalyst 9.10 drivers were the first major release since the launch of the 5800 series can carry the driver number 8.661. According to the release notes, there were no performance enhancements made, but the Drag and Drop video converting application got GPU acceleration enabled and Super Sample Anti-Aliasing was enabled for all Radeon HD 5800 series cards.

Catalyst 9.11
This driver was released on 11/17/2009 less than a month after 9.10. The actual video driver is numbered 8.671. Again the release notes do not document any performance enhancements from this driver revision, but GPU acceleration of H.264 video using Adobe Flash Player 10.1 Beta was enabled.

Catalyst 9.12
Some of you may remember that Catalyst 9.12 garnered some attention for a hotfix posted by AMD shortly after the first version released on 12/17/2009, which turned out to be an early preview of the Catalyst 10.2 drivers. Catalyst 9.12 is now only available in the "non-hotfixed" and the hotfix version was not tested for this article. The actual driver number for Catalyst 9.12 is 8.681 for those of you keeping track. The release notes for 9.12 indicate there should be an increase in performance for 3DMark Vantage of up to 9% for the 5800 series. The hotfix release also indicated that they improved performance for all OpenGL games, though this could be referring to the OpenGL 3.2 extension support feature that was added.

Catalyst 10.1
Released 01/27/2010 and carrying a driver number of 8.69, the Catalyst 10.1 revision claimed up to a 4% increase in performance in Crysis for 5800 series products. No additional features were added.

Catalyst 10.2
As the name implies this driver revision was released in the second month of 2010, more specifically on the 17th. The actual driver revision is number 8.702. Most of the new features are centered around Catalyst application compatibility with CrossFireX configurations and adding support for DisplayPort audio. Notable performance enhancements included: up to 8% increase in DiRT 2 with 5700 series and higher and CrossFire 5870's received an up to 6% improvement in performance for BattleForge.

Catalyst 10.3
This was the big one. Catalyst 10.3 released on 03/24/2010. It included a slew of performance increases for a host of games. Officially, the 8.712 driver, the 10.3 Catalyst revision added enhancements to Eyefinity and support for 3D stereoscopic glasses.

Performance enhancements abound:
3DMark Vantage: Overall scores increase by up to 4% on Radeon HD 5800 series
BattleForge: Improves up to 8% on Radeon HD 5000 series
Crysis and Crysis Warehead: Improves up to 6% on Radeon HD 5000 series
DiRT 2: Improves up to 20% on Radeon HD 5800 and 5700 series
Wolfenstein: Improves up to 4% on Radeon HD 5000 series

There were, of course other games in the list, but we have only listed ones that we are including in the article.

Catalyst 10.4
Finally, last month, in what appears to be a crunch to get something out before the end of the month, the Catalyst 10.4 drivers (officially 8.723) released on the 28th. Catalyst 10.4 came with some video playback and transcoding enhancements but no performance updates for any of the games we tested.

If you are at all curious about the full list of games with performance enhancements, you can look at the release notes of previous drivers here.

The Setup and The Test

The Setup

For those of you who are detail oriented, here are the specifications of the test setup.

The Test

The test we designed included 6 games and 2 synthetic benchmarks as hopeful controls.

Games Tested:
Crysis: Warhead
BattleForge
Mass Effect 2
Battlefield Bad Company 2
Wolfenstein
DiRT 2

Synthetics Tested:
3DMark Vantage
Unigine Heaven 2.0 Benchmark

The games chosen are to give a good mix of all types of games and application programming interfaces (APIs). There are 3 DirectX 11 games, 2 DirectX10 games, and one OpenGL game.

Crysis: Warhead
(results graphs provided in CrysisWarhead.csv; numbers in .csv truncated to 2 decimal places)

What we see with Crysis: Warhead is that at both resolutions each subsequent driver release maintains relative parity with the previous release up to 9.12. At 9.12 we get roughly a 1% increase in performance, but since the previous two releases had made performance decreases relative to the launch driver the 9.12 increase just put the game back on track with the launch drivers. The 10.3 Catalyst drivers yielded and almost 3% increase over the previous driver and over the launch drivers at 1792x1344 (read 1920x1200 if you have a 16:10 monitor). At 1600x1200 (read 1680x1050 for 16:10), there was a 2.65% increase over the previous driver and a 4% improvement over the launch drivers. We did not get a 6% change that was indicated by the release notes, but we do not know what settings they were using. 10.4 Catalyst drivers yielded another 2% increase over the 10.3 drivers at both resolutions bringing the improvement relative to launch drivers to about 5% at both tested resolutions

BattleForge
(results graphs provided in BattleForge.csv; numbers in .csv truncated to 2 decimal places)

BattleForge also had a 1% increase from 9.11 to 9.12 that brought it back into parity with the launch drivers. The real break away from launch performance happened with the 10.3 Catalyst package. The 10.3 instruction set increased performance about 2.5% on both tested resolutions. This was the only major gain for BattleForge. While it is not the 8% purported by the 10.3 release notes, the 10.3 and 10.4 drivers do perform about 3.5% better than the launch drivers for BattleForge.

Mass Effect 2
(results graphs provided in MassEffect2.csv; numbers in .csv truncated to 2 decimal places)

Mass Effect 2 has an interesting story. Multi-sampling Anti-Aliasing (MSAA) was not enabled for this game until the 10.2 release; even then, it had to be forced at the driver level. Thus we start at Catalyst 10.2 for our reference. In the 10.3 release we see a substantial 15% increase in performance with the 10.4 release maintaining relatively the same performance. On a side note, with AA off, the frame rates were between 100 and 110 for all driver releases with no significant difference from one driver to the next.

Battlefield Bad Company 2
(results graphs provided in BattlefieldBadCompany2.csv; numbers in .csv truncated to 2 decimal places)

Except for releases 9.12 and 10.3 most of the driver releases mimicked the one previous to it with a very minor change in performance. Both 9.12 and 10.3 only saw about a 1% gain compared to there previous counterparts. However, those tiny increases added up to a 2.5% to 3.5% increase in performance when comparing the final result to the original launch drivers. And while our benchmarks don’t record it, 10.4 introduced the loading time fix for the game when using Direct3D 11.

Wolfenstein
(results graphs provided in Wolfenstein.csv; numbers in .csv truncated to 2 decimal places)

Wolfenstein is our token OpenGL game. Unfortunately, Wolfenstein does not support 1792x1344, although this is not surprising because it is an odd resolution. What we do not see is any serious deviation from the performance of the launch drivers, not from any OpenGL optimizations that may have been made for 9.12, not even from the 10.3 drivers that were supposed to give us “up to 4%" improvement. Driver revisions 10.1 and 10.3 swing 1% in the positive direction but they are offset by a 1% opposite swing by 10.2 and the cumulative effect of all the other drivers to bring us back to even performance when compared to the launch driver.

DiRT 2
(results graphs provided in DiRT2.csv; numbers in .csv truncated to 2 decimal places)

DiRT 2 is odd. We actually do experience some of the performance enhancements purported by the release notes, we just did not experience them with the same drivers AMD did. Also, with the exception of the launch drivers, there seems to be little difference between running the game at 1600x1200 and 1792x1344. While doing some research in to this matter, an acquaintance of ours working with the Unreal Engine pointed out that some engines are much less dependent on resolution. This is because once the size of the textures is set and the number of polygonal details is set, the same amount of calculation is required for whatever resolution you run it at. While DiRT 2 does not run on the Unreal engine, this seems to hold true for engines that rely on the CPU to do all of the work. DiRT 2 may be such a game. Or perhaps the game uses some sort of scaling to achieve 99% of the performance with 25% more pixels.

3DMark Vantage
(results graphs provided in 3DMarkVantage.csv)

Our first control program seems to fail as a control. Driver writers often try to craft drivers specifically to make 3DMark scores higher. This is a blessing and a curse for any program. As a game it is good to be popular enough to warrant video card manufactures writing code specifically optimized for your game. As an objective tester, such specialized drivers for a benchmark program muddle the conclusions that can be made from the results of said benchmark.

AMD claimed to get up to 9% better results from the 9.12 drivers. We experienced a 7% gain. The 10.3 drivers were to bring a 4% gain and we experienced a 2% gain. At least we experienced the expected gains. The problem that we run into is that people use 3DMark to test whether or not their computers can handle a certain game, or more likely, how well their computers can handle a certain game. While we did see improvements of several of the games for Catalyst 9.12, none of them sowed a consistent 7% gain or even close to that. The only one that was close was DiRT 2 but that was only at 1600x1200.

.

Unigine Heaven 2.0 Benchmark
(results graphs provided in UnigineHeaven2.csv; numbers in .csv truncated to 2 decimal places)

The Unigine results are all over the place depending on what API you are looking at. With DirectX 9, we have insignificant but steady steps downward until driver revision 10.4 corrects the problem. DirectX 10 sees small increases at 9.12 and 10.3 again. DirectX 11 does not change performance significantly at any point.

Then there is OpenGL. Now admittedly this is Unigine's implementation of OpenGL and not necessarily a reflection of OpenGL as a whole. The benchmark suffered from being unable to render anything that was in soft focus in using a depth of field effect until driver 9.12. Before then any objects in soft focus would just be black. After 9.12 the objects would be rendered but only in sharp focus and if the focus area changed, the field of changing focus would be rendered black. The results for OpenGL were very odd and we went back and ran the tests several times over again to make sure they were not anomalies. However, the results do not correspond in any way to the real world results we obtained with Wolfenstein so they may be disregarded. 

Conclusions

It has been almost eight months since the release of the new generation of AMD graphics cards. We tested all the major driver releases. What did we learn?

First we should mention that AMD had a particular interest in optimizing some games over others. Battlefield Bad Company 2, DiRT 2, and Mass Effect 2 are all AMD sponsored games and therefore it is not unreasonable to expect AMD to make specific enhancements to those games that do not affect any others.

In such cases we saw that a driver change could make a huge difference. DiRT 2 saw an increase of about 24% compared to the launch drivers. Mass Effect 2 did see anti-aliasing enabled with 10.2 Catalyst drivers and a substantial increase in performance from 10.2 to 10.3.  Battlefield Bad Company 2 has not yet seen any major improvements in performance. Most games, like BFBC2, did not experience such significant changes in performance. The improvements for these games seem to be exceptions rather than the rule.

As it turns out, there were only two major releases that affected any great number of games across the board. Catalyst 9.12 and 10.3 made the biggest general contributions to overall performance. In most cases however it was not much, generally 1 or 2 percent.  We saw that with the Catalyst 9.12 release that 3DMark Vantage posted an almost 7% gain over the previous driver while most of the other tests posted meager 1% gains. DiRT 2 did manage a similar gain of 6% but only at the 1600x1200 resolution. These results would indicate that the changes made to improve 3DMark Vantage scores for the December release were made specifically to improve performance in 3DMark and had the added benefit of improving performance for DiRT 2 at lower resolutions or vice versa. As we mentioned earlier, the Catalyst 10.3 release exhibited a 2% performance gain 3DMark Vantage. Every other game benchmark also saw this magnitude of performance gain. Mass Effect 2 and DiRT 2 saw even more significant performance improvements. The best conclusion drawn here is that serious changes were made in an instruction set that almost all games use and additionally there were further tweaks made to Mass Effect 2 and DiRT 2. The resulting performance gains seen in 3DMark Vantage were most likely a byproduct of this event.

Unigine turned out to be a particularly useless control. The program only exhibited the expected behavior with the DirectX 9 and DirectX 10 APIs. The DirectX 11 tests did not behave similarly compared to the other DirectX 11 games and the OpenGL results were very erratic, repeatable, but erratic. The program is a very useful demonstration of tessellation and how it can affect the look of a game, but the paths it uses for rendering seem to be different from that of most other games.

As far as picking a particular driver for one game over another, the latest drivers (10.4) seem to offer all of the same relative benefits as the 10.3 drivers with some added features. If you are playing Mass Effect 2 be sure to use Catalyst 10.2 or later so you can take advantage of the smooth edges on your geometry. With that being said, it is a good idea when playing a game newer than your hardware to use drivers released after the release of the game. This way any potential problems or glitches have the opportunity to be resolved.

If you are looking for new driver releases to enable new features on your card, the past 7 driver releases have accomplished a lot for you. Not much of that was covered in this article as were we looking at the set from a performance point of view, but the number of issues that were fixed and useful features added would be quite beneficial to you.

As for future driver releases, we expect AMD will continue to do what it has always been doing: enable new features and occasionally target a popular game with a tweak or two. With any luck, based on what we found with a major performance release every 3 months, Catalyst 10.6 may be a good one.