Comparing the reliability of ARM and Qualcomm Android graphics drivers, S8 to S8
The GraphicsFuzz ShaderTest GLES test suite reveals a variety of graphics driver issues on the Samsung Galaxy S8. The Qualcomm GPU model suffers from considerably more rendering issues and crashes than the ARM GPU model. Skip to our analysis of the results.
See also: follow-on posts evaluating:
Hugues Evrard, Paul Thomson and I have recently founded a company, GraphicsFuzz, specializing in automated testing of graphics drivers. The company is the fruit of research at Imperial College London that we described last year through a series of Medium stories.
Click above to try out our GraphicsFuzz Android benchmark web app.
Our initial product is ShaderTest GLES, a large suite of fragment shaders that GPU designers can license in order to thoroughly test the reliability of OpenGL ES drivers.
This is the first in a series of stories where we’re going to look at the reliability of Android OpenGL ES drivers, as judged by running them on a representative sample of the ShaderTest GLES shaders.
You can also try out a selection of these shaders on your Android devices via the GraphicsFuzz Android benchmark — try out the web app in your browser and let us know what you find!
ShaderTest GLES: testing shader compiler quality
We’ll start with some background on ShaderTest GLES, but if you are impatient you can skip to the results!
Traditional GPU test suites focus on rendering speed or API feature-completeness. ShaderTest GLES adds another dimension to the story by testing shader compiler quality. After all, a fast rendering result may not be particularly useful if it is wrong!
Talking with software engineers in industry, we have heard time and again that GPU driver developers prioritize testing their drivers against high-profile games and apps— e.g. the top 100 games in the Google Play Store. An unfortunate side-effect of this is that developers of less popular apps must work around driver bugs to make their apps work on consumer devices. Coupled with the slow speed at which Android updates are pushed to end users, this creates a self-perpetuating situation where GPU driver bugs are not seen as a serious issue because popular games run fine (possibly using workarounds), and GPU driver bugs are not worth reporting because it is necessary to work around the bugs anyway if you want consumers to be able to use your app. It also discourages developers from writing interesting shaders: until the driver quality situation improves, they are unlikely to work on a range of devices.
The ShaderTest GLES approach
(See this post for some background on shaders and shader compilers.)
ShaderTest GLES consists of many shader families. Each family contains one reference shader that is used to render a single frame, and several hundred variant shaders designed to render a visually identical frame to the reference, but to exercise the driver in radically different ways in the process.
The variant shaders are obtained from the reference shaders by applying semantics-preserving transformations — transformations to the source code of the reference shader that should not affect rendering results. For instance, wrapping a block of code in the reference shader in a loop of the form:
for (int i = 0; i < 1; i++) {// original code from reference shader}
is a simple example of a semantics-preserving transformation.
ShaderTest GLES finds bugs in graphics drivers by identifying rendering mismatches, errors and crashes, across families of equivalent shaders.
ShaderTest ES highlights driver bugs and issues when:
- a variant shader is rejected by the driver’s shader compiler — a compile failure;
- a variant shader causes some component of the driver to crash — a crash error;
- a variant shader leads to a noticeably different image being rendered compared with the reference shader — a rendering issue;
- a variant shader causes the driver to bail out due to time or memory constraints — a resource issue.
Compile failures and crash errors always indicate bugs. Rendering issues usually arise due to miscompilation errors, whereby the shader compiler generates wrong code; they sometimes also manifest due to floating-point sensitivity in the graphics systems — ShaderTest GLES can shed light on both issues. A big win here is that miscompilations that trigger rendering issues are very difficult to detect and thus often evade traditional testing techniques. Resource issues provide insights into the nonfunctional constraints of the graphics driver.
To the Samsungs!
Let’s kick off by comparing the results for some reference and variant shaders with respect to one of the most popular Android devices on the market: the Samsung Galaxy S8 phone. Depending on territory, the S8 ships with either an ARM or Qualcomm GPU, so let’s see how the drivers from these GPU designers fare.
A snapshot of our online results table, which compares the ARM and Qualcomm S8s head-to-head.
Comparing results between the ARM and Qualcomm GPU drivers — we’ll simply call them “ARM” and “Qualcomm” from now on — the ARM model comes off looking more reliable overall. We find the following issues:
Rendering issues
Qualcomm suffers from significantly more rendering issues: our tables show five rendering issues from Qualcomm compared with one from ARM.
For example, check out the results for shader family 001. Crashes aside, ARM universally renders the reference image:
The reference image for shader family 001. All variants should render a similar image.
while with Qualcomm we get the reference image for most variants, but a couple of erroneous images in the mix too:
On Qualcomm, we get the expected image (left) for most variants in shader family 001, but two erroneous images (middle and right), likely due to shader compiler bugs.
The reference shader for shader family 001 shader is not floating-point-sensitive, so these discrepancies very likely correspond to shader compiler bugs.
We also observe one ARM rendering issue, for variant 77 of shader family 007:
Reference image for shader family 007 (left) compared with bad image rendered by variant 77 (right) on the ARM S8.
Nondeterminism
Our testing revealed two issues on Qualcomm where the driver behaves nondeterministically: sometimes the shader compiler crashes during compilation; other times compilation succeeds and an incorrect image is rendered. This occurs for variant 123 of shader family 007, and variant 51 of shader family 008.
Each pair consists of an expected reference image (left image in pair), and a bad variant image rendered by Qualcomm (right image in pair). However, the Qualcomm driver fluctuates nondeterministically between producing this wrong image vs. crashing during compilation.
Crash errors
Both drivers exhibit a number of crashes on our sample: 28 for Qualcomm vs. 15 for ARM. All are compile-time segmentation faults — Fatal signal 11 (SIGSEGV). The crash data available on Android doesn’t give anything away related to the cause of these crashes.
Interestingly, neither driver can handle variant 34 of shader family 006 without crashing. However, as we’ll see in future posts, other graphics drivers can handle this variant.
Compile failures
The ARM driver exhibits one compiler failure, which occurs for variant 216 of shader family 003. The issue appears to relate to mishandling of the ternary operator, (?:). Qualcomm shows more compiler failures —nine in total (combining compiler and linker errors), though several of these errors appear to be due to a single underlying issue related to handling of switch statements.
Resource issues
Our results don’t highlight any Qualcomm resource issues, but do show a resource issue for ARM on variant 122 of shader family 003, which gives a GL_OUT_OF_MEMORY during rendering.
Check out the full S8 vs. S8 results table, and try out the GraphicsFuzz Android benchmark web app!
Next time…
…we’ll see how the Android drivers for NVIDIA’s Shield TV hold up.