UX performance testing with GameBench and eggPlant

Antony Edwards, CTO, TestPlantSharif Sakr, Director of Research and Content, GameBench

How game technology can improve your app

Thanks for tuning in. I’m going to run through a quick intro to GameBench and the metrics we reveal. Then we’re going to show a quick demo of how GB works with eggPlant. And finally I’m going to look at some examples of good and bad optimisation in retail apps, to show you the power of what GameBench reveals.

● Founded in 2013● CEO and CTO came from ARM● Tools that measure fluidity of mobile UX● Customers are leaders in h/ware and s/ware

Customers include major hardware players like Samsung and ARM, QA houses like Enzyme, game developers like SpaceApe.

● Sharif Sakr: Director of Research and Content● Ex-journo: BBC News, Forbes, Engadget● Used GameBench for product reviews● Now specialise in objective analysis of UX

I used to use GameBench to help me with product reviews, but it was even a product. When I realised how important this sort of performance testing is, I quit journalism in order to work for GameBench.

Games often go wrong

GameBench was created to solve a market need -- games too often fail to perform consistently across the vast range of mobile hardware.

Games often go wrong

Even games from high-end, well-off studios like Rovio, using mainstream middleware like Unity, fail to run consistently -- even on top-end devices. Introduce the concepts of median and minimum frame rate.

Users notice these inconsistencies. It affects their reviews and hence the game’s commercial success.

Big game studios invest in UX testing

Months of QA, soft launches, feedback, performance analysis, tweaking. Smaller studios can’t afford this, and therefore usually fail. The gap between the two is growing.

As genres get more crowded, the industry becomes ever more brutal. “Trusted” brands, known for quality, get discoverability in the stores. Indies often get swamped in a sea of similar-looking titles.

●$500 million business built on realistic animation●Hundreds of developers, scores of QA testers●A brand trusted for quality

And this is what quality looks like, when you’re talking about the smoothness of a game. FPS is solid at 60fps during races. Very strong peak at 60fps on the stability chart. Remember this shape for later when we look at apps.

But none of this applies to apps, right?

Well, that’s what we initially thought. That’s why we called ourselves GameBench.

Games vs. Apps

Games are limited by local processing power

Apps are only affected by remote latencies

Games are for a demanding niche Apps are for the less discerning mainstream

Games depend on a higher frame rate (e.g., 60fps)

App frame rates can be lower

We thought that games and apps were totally different when it comes to the importance of a visually smooth UX. But our assumptions were wrong.

Games vs. Apps

Games are limited by local processing power

Apps are only affected by remote latencies

Games are for a demanding niche Apps are for the less discerning mainstream

Games depend on a higher frame rate (e.g., 60fps)

App frame rates can be lower

] We thought that games and apps were totally different when it comes to the importance of a visually smooth UX. But our assumptions were wrong.

Both iOS and Android operating systems run at 60fps, which means apps must scroll and animate at this rate too -- otherwise they feel janky. 60fps came from gaming, but is now a universal target on mobile.

PhoneArena tested frame rates of Android -- these are the frame rates an Android user will experience right before and after using an app. E.g. Swiping the home screens or accessing the app drawer both animate at 58fps, both on Samsung’s TouchWiz and raw Google. (If anything TouchWiz is slightly smoother today, as this table is a year old.) This tells us that app frame rates really need to be at least 55fps, and preferably above 58fps, if they’re to match the underlying OS.

A clear target: 60fpsSo how do real apps fare?

We know that apps need to animate at or close to 60fps. So how do we test this for a specific app? The answer lies in a combination of GameBench, which measures frame rate and other key performance data, alongside eggPlant, which allows you repeatedly test user sessions automatically. We’ve now got a demo to show you what that looks like...

Server / Cloud

Solution

Test Laptop

At the end of the test the standard ePF reports can be viewed in eggPlant Manager, but now in addition we have full profiling and UX data available in GameBench.

Looking at the architecture of the solution. Here we have eggPlant Functional driving and verifying an iPad as normal; and then uploading the results to eggPlant Manager for review. But in addition we are now getting the UX and performance information from GameBench.

The GameBench agent runs on the same machine as eggPlant Functional; and eggPlant Functional controls it via the command line, i.e. starts and stops the profiling. So we have eggPlant Functional running as normal, but it’s also getting extra profiling information by controlling the GameBench agent.

Demonstration

Info: http://www.testplant.comFeedback: support@testplant.com

So that’s it – very simple – and now lets look at that in action.

ASOS: pretty smooth

Let’s start with an example of a good app - ASOS for Android. It’s worth just looking at the basic shape of the chart: two sharp peaks, one at 0fps (representing static time) and one much bigger peak at 60fps (representing time spent at the target frame rate). That chart isn’t perfect -- a bit too much activity in the 45fps region, which drags the median fps down, but still… pretty smooth.

More smooth apps...

Two more smooth apps, again showing the same basic shape. A big, sharp peak at (or close to) 60fps, and a smaller sharp peak at 0fps. The relationship between the size of the two peaks is interesting -- an ebook reading app would have the sizes reversed, whereas in retail we’d expect a health shopping trip to mostly be spent on the move, so we can define an expected ratio between these peaks and look for exceptions to highlight problems. But we won’t get into that too much now.

Mango: not so smooth

Mango throws up a rather different chart. Both peaks are much broader. The 60fps peak isn’t at 60 at all, but at around 53fps, which is below our target. The 0fps peak is broader, showing more time spent on a roughly static screen. The ratios are out of balance.

GameBench metrics directly relate to UX

...which means users tend to notice

We noticed the issues in the Mango app immediately -- the shaky, unresponsive scrolling, the slow load times. But so did other users, judging from reviews on the Play Store.

We can’t be certain that they’re describing frame rate issues, but these words -- “laggy” and “slow-motion” -- tend to be linked with this issue, rather than just with simple load times. It’s worth bearing in mind that Mango for iOS has good reviews, and shows a very n

Spot, diagnose, fix

FPS is just the starting point

Screenshots:

Screenshots are captured once per second and allow problems to be correlated to onscreen events. Here’s the cause of the excess time spent close to zero: dead screens while the user is waiting for things to load. Other apps showed a spinning circle during loads, to tell the user to be patient -- something is happening. But Mango didn’t do that.

Network traffic:

FPS is the starting point for any analysis of smoothness. You correlate to screenshots and then also to secondary metrics such as network traffic. Here, it looks like Mango is downloading big bursts of data at up to 30,000Kbps -- nearly 14MB in the space of 5 seconds, which might be the reason why images are slow to load.

Network traffic (iPhone):

By contrast, the iPhone version of the app has many smaller and slower bursts of traffic, which is something we tend to see on better optimised retail apps.

System resources:

Here we’re looking at usage of the graphics processor, or GPU. It turns out that this is probably the biggest factor, since the GPU is getting maxed out for no good reason.

By now, we have a pretty good idea that somehow the layout of the app is highly inefficient. This is further corroborated by Android’s own “overdraw” feature, which shows a mass of pink on the Mango app (left), versus much less pink on the better-optimised H&M app on the right. It’s up to the developer to take it further and fix the problem.

GameBench info: www.gamebench.netContact: sharif@gamebench.com

Info: www.testplant.comFeedback: support@testplant.comSamples: www.testplant.com/dlds/eggplant-functional/