Share this eBook!

INTRODUCTIONBuilding a million-download mobile game is not one-day easy job. Not only do you

need to outperform your direct competitors on sales and marketing, but also you

should provide an ACE quality mobile game to your frontier colleagues to win over the

gameplayers.

By understanding the differences between regular app testing and mobile game testing,

you have taken the first step. Once you gain the familiarity in the key areas in mobile game

testing, you are on the right track. But there is much more.

With several years providing the service and support to one of the most successful

mobile gaming developers, we have been gaining rich experience in this area. And we

would like to share with you our knowledge regarding the fundamentals of mobile gaming

development and testing.

With this ebook you will learn three critical fundamentals in mobile gaming development

- Architecture and Infrastructure, Agile Process in Development, and Value of Testing.

Share this eBook!

CHAPTER 1

Architecture and InfrastructureAspects of Mobile Game Testing

Share this eBook!

O ne of the most common problems in mobile game testing – similar to app testing

– is how to access all possible devices that your game users are using. As said,

despite thousands of unique models already out there for Android (and with all possible

iOS variants with different OS versions) only few hundred of them are really relevant

in the big picture. And naturally, this keeps changing all the time and new devices are

introduced by tens of major OEMs every quarter.

ARCHITECTURE AND INFRASTRUCTURE

Many of today’s mobile games are based on open source or commercial game engines,

such as Unreal, Unity3D, CryEngine, Construct, PlayCanvas, Cocos2D, and many others!

Furthermore, these game engines provide great tools and frameworks to get you game

quickly built, with all those nice graphical features.

Looking this from the traditional software point of view, this is like a “tool – app –

middleware” offering that these products will provide you, as well as compiling your game

to specific platform (many of these products are cross-platform and enable you instantly

to compile the game for Android, iOS and other platforms as well). For the platform part,

let’s use Android as an example: an “open” operating system with a stack of software

Share this eBook!

components which can be roughly divided into four layers: Application, Application

framework (manager layer for content, resource, package etc.), Libraries (e.g. OpenGL ES

software, Fonts, WebKit, SGL etc.) and the Linux kernel (drivers for graphics, audio and

keypad, power management etc.). In addition, platform here includes the actual hardware

running whatever chipset on it, with GPU and displays of different resolutions.

After that your mobile game needs to communicate with back-end servers using your

own or Google/Apple/other services via WiFi, radio or some other type of communication

channel. The need for testing the connectivity between your mobile game and back-end

services is nowadays very important. For example, ads are shown via connectivity and

without those running properly on your device, you are taking an extra risk in front of

your core audience – mobile gamers. Sure, they would probably like it better without ads,

but from the monetization point of view, you surely want to test this thoroughly before

submitting your app to app stores – and make sure you are not losing revenue because

of faulty implementation.

Share this eBook!

As said, testing connectivity should be one of the key things to do when building mobile

games. Test automation can bring you a lot benefits for automating the connectivity

testing, and scaling the infrastructure from one user to hundreds of thousands of users.

HERMETIC ENVIRONMENT AND TESTING

In most cases, today’s mobile games have a server-client interaction, some requiring a login,

uploading of data (e.g. game results/progress) and downloading of data (e.g. ads). When

adopting these kinds of services to be part of your games, developers need to ensure that all

their changes done in code, to either server side or client side do not break the functionality

of the service. Furthermore, this type of environment can be also set up on a local machine/

server where critical connectivity and interaction between user and server can be tested. In

fact, testing for some of these types of use cases could be done efficiently with a hermetic

environment – however, not jeopardizing the integrity of your game.

Hermetic testing can be seen and done as easily as switching your device to flight mode.

Nothing gets sent out and nothing gets in.

Share this eBook!

COMPATIBILITY AND PORTING – TESTING

When your mobile game is getting developed, you should always test it on the real platform

that end-users are using as well. This means that you must test your game on real devices.

Emulators won’t simply provide you as authentic environment as it would be required to

get full understanding of real behavior in hands of real users. What makes a difference

between using an emulator or real hardware? Typically, the lack of real hardware slows

down and forces developers to use emulators, but this can shrunk into 4 categories why

games should be only tested on real devices:

1) SOFTWARE. It’s just not only about vanilla platform software that makes the difference,

but nowadays OEMs customize their software a lot: UI layers, customized legacy software,

only certain widely used apps are pre-integrated with the device (e.g. Twitter, Facebook).

2) HARDWARE. There are LOTS of differences in chipsets, GPUs, even connectivity

hardware, displays, sensors, and memory. This is very hard to emulate appropriately –

and even it would get as close to real world scenario – all of that is executed on PC x86

hardware, and not ARM-based hardware.

3) USER EXPERIENCE. This is heavily related to performance. The graphics performance

of the system makes the user experience, either good or bad. And there is an absolute

must to get mobile game tested on real hardware.

4) NETWORK. The system can experience network issues and slow network is not possible

to be tested on emulation. In terms of network configuration, emulators run on the PC,

connect to the LAN and access the internet via your corporate firewall. Using real handsets,

the network is connected to the radio interface and from there to the internet.

Share this eBook!

Whatever platform (e.g. native mobile platforms, web etc.) you are building your mobile

game on, remember that only real devices give you real results. Furthermore, the depth of

testing (e.g. how many devices are used, how many different configuration those cover)

is important as you don’t want to minimize your target audience who have potential to

play your game – make you money!

SMOKE AND REFERENCE TESTING

When the mobile game is in the state that is testable in the context of 1) game itself, 2)

platform-compiled and 3) connects with back-end services (or just platform APIs) the

full-blown reference testing can give great results and understanding of how well game

does. But even much earlier than that, it is possible to start doing smoke testing and using

even hundreds of real Android/iOS devices for that. When bugs are revealed and fixed

earlier, it will save you a lot of time (and money) and gets you earlier to publish your game

Share this eBook!

for gamers. In addition to regular smoke and reference testing, it’s a good idea to include

real performance and battery drain tests as part of your testing pattern.

The reference testing gives you great understanding of how your game does, but it doesn’t

cover everything. For example, huge number of users can quickly take down the back-

end, slow down the game-play and even worse, force users to stop playing the game.

STRESS, LOAD, SPIKE, SOAK – PERFORMANCE TESTING

Good performance is very closely related to good user experience. User wants to see

constant progress with the game, do the smooth game-playing, graphics performance

needs to be up to snuff and work across all different mobile devices (phones, tablets etc.)

the very same. In addition, it is important to understand how well back-end(s) can handle

massive number of users and significant load.

Share this eBook!

There are lots of different ways to build back-end testing and scale up the performance

so that simulates the huge load for your service. One easy and very straightforward way

to do this is to use device farm on cloud to get all of those devices to do the same thing

for your service. It literally takes minutes to set it up and get results back instantly.

Share this eBook!

CHAPTER 2

The Agile Process for Mobile Game Development and Testing

Share this eBook!

T hings that get done manually are hardly agile. Manual testing is not an exception – but

it will significantly slow down the delivery if each and every change or new feature

add-on have to be manually tested. So, how do the most successful game development

teams do their game development and testing?

They do Agile – Developers always concentrate on the delivery of new features,

enhancements, and bug fixes. The use of test automation is in an epicenter of how

developers already build their software and especially how this process can be significantly

boosted, with automated testing included in the process and using real devices – the

same ones that gamers use.

Why Testing Must Be Included in Day-to-Day Process?

The common approach, even today, with mobile game testing is that all of that important QA

and testing work gets done at very late of the development process. Typically, companies

are even dedicating specific time for this when QA, testers and beta testers do their work

post-production. While testing is very important, nearly all of it can be included as part

Share this eBook!

of the agile development process and every aspect of the game can be tested during

the development – and during the production. Some certain foundations can be even

tested (e.g. frameworks, platform setup) during pre-production, but the most important

thing in order to improve the efficiency is to seamlessly integrate testing as part of the

production phase.

The basic definition of continuous quality and delivery would be simply delivering the

better games faster and do the majority of the testing while actual development is ongoing.

In any cases, continuous integration, delivery and testing can significantly improve the

outcome of your mobile games and result as a more robust end-product.

Continuous testing using real devices can save up to 60% of overall development and

testing time when testing is done as an integral part of the development. This has a direct

impact on staffing, costs and faster time-to-market. And when things get fixed earlier,

debugging time can go from months to hours.

Share this eBook!

When testing is not part of the actual development: In the first sprint you do the regular

things and get the development going. Before the testing of even smaller parts or standalone

features gets done, you simply don’t have visibility of the actual robustness of your mobile

game. At some point, these bugs will become obvious and then stirs up again your sprint

procedures and causes an halt of the overall development. In this example, developers

spend much more time on firefighting issue that was part of the code for weeks or even

months. Even the worse example would be that the bug gets revealed when the game out

for users – e.g. as an user feedback or by using some other post-release analytic systems.

Developers should always spend more time on new features, incremental improvements,

than fire fighting the old bugs and problems. Things with bugs escalate quickly when

noticed later on, which will have a direct impact on developers daily tasks.

HOW TO ADD “EXTRA DAYS” FOR AGILE DEVELOPMENT?

Effective mobile game testing derives from a well-structured and systematic approach, use of

test automation framework(s) and real mobile devices, and seamless integration with the agile

development process. By adding an ‘extra time’ for the process – e.g. nights, when typically

development work is not going on – you can easily dedicate this time for test automation.

Written test scripts and automatic pre-processing of test data (logs, screenshots, results) will

provide a report of how well the recent regression worked, what were the exact problems

and even suggestions on how to fix those issues will be part of the deal with test automation.

After the package (APK or IPA) is built, it can be automatically send for the first test – the smoke

test. The smoke test can be also seen as a compatibility test when it will be executed on all

possible device variants. Also, each regression can be tested with the existing test cases/scripts

for certain specific function. For example, if you mobile game relies on back-end service, this

connection can be after each build regression tested with automated scripts. In addition, many

of aspects can be instantly automated: localization (testing different languages), different types

Share this eBook!

of performance tests (load, spike, stress and others are typically tests when game contains

more ‘beef’ and are truly stressing the hardware), connectivity and hermetic testing (isolation

of game from network, other services etc.).

Instead of fixing bugs later in the following sprints, this approach focus on fixing bugs

instantly after the regression. Developers will be notified about their enhancements, new

source code/build problems during the next day, and will have a chance to fix those

problems even early in the ongoing sprint.

Example of tests that can be done during the development

Smoke testing with automatic test exerciser. Automatic test exercisers provide a great

way to smoke-test games. They need no specific tests but focus is in testing the user

interface logic (e.g. opening menus, clicking buttons, doing swipes, multi-gestures etc.).

Automatic test exercisers provide the least exact results, but those are very powerful in

terms of providing quick feedback on each iteration/regression of your mobile game.

Share this eBook!

Regression testing after every code change. Regression testing can be used to ensure

that the quality of mobile game is still good after any features, add-ons or changes in

game. Each feature of mobile game should be tested on all available platforms before

development of the next feature. Thus, regression testing is a part of testing life cycle and

very important element in effort to build more robust games.

Connectivity testing. E.g most of the mobile games today have a server-client interaction,

requiring a login, uploading of data (e.g. results) and downloading of data (e.g. data,

images). If you’re developing these kinds of services around your game, developers need

to ensure all their changes are done in code, to either server side or client side, do not

break the functionality of the service.

Performance testing and all variants of it are important for mobile game testing. Sluggish and

badly performing games can ensure that game won’t be successful, gets bad ratings and

won’t make its creator name. Test automation can help to understand the real performance

Share this eBook!

and many other aspects of game’s performance (e.g. battery consumption). For example,

developers can create patterns to see how graphics assets gets loaded and used – and based

on that information to optimize handling of those.

Localization and language settings testing becomes highly important when your game is

targeted for the global markets. The word ‘global’ means your game needs to be ‘local’ for

everyone. When your game titles, texts and content needs to be translated and tested with

devices in multiple languages, these types of tests can be

easily performed automatically (with help of cloud-based

device access and test automation) to change the language

of devices, and even do it again with the same test runs.

The problems layouts (e.g. fonts behaving bad) can be

easily picked from screenshots. Real mobile devices can

be also easily switch languages so all possible languages

are testable with the exact same game content.

Functional testing is probably the most common method in mobile game testing. Typically

functional testing is associated with manual testing and playing ‘game through’. However,

to be agile, all functional testing should be automated. Functional testing – with help of test

automation frameworks – requires some basic understanding of programming and setting

up environment for testing. Automated functional testing can reveal issues related to user

interface (and graphics), stability, game flow/mechanism, and integration of graphics assets.

MORE ADVANCED TEST METHODS FOR MOBILE GAMES

Today, one of the hottest method for mobile game testing with lots of graphics (e.g. OpenGL ES)

content is the use of image recognition. It can solve the well-known issue of instrumentation

not being able to recognize UI elements based on IDs, descriptions and other typical handles

that work with majority of test automation frameworks.

Share this eBook!

CHAPTER 3

The Value of Testing in Mobile Game Development Process

Share this eBook!

Crossing the Mobile Game Development Chasm

T he fact is that mobile game testing is very different compared to the regular app

testing. Much of the mobile game testing is still done manually and that doesn’t

provide an efficient approach to get everything properly covered. In fact, effective mobile

game testing should derive from a well-structured and systematic approach, use of test

automation framework(s) and seamless integration with your agile process.

Share this eBook!

Furthermore, many of test automation frameworks have been originally built for native

apps (or web apps/elements) and one of the biggest misconceptions among mobile game

developers has been that test automation frameworks do not comply to game testing

– where certain native UI elements, their IDs, characteristics etc. cannot be right away

identified. However, this isn’t really the case and many of those frameworks provide

excellent ways to testing mobile games, e.g.using image recognition.

When it comes to testing,

the simplistic view of it is

that bugs are identified

and documented so that

developers can remove

those bugs. Therefore some

metrics traditionally used

for software testing – such

as number of bugs found

and number of test cases

created – are great but do not show the value-add of testing and because of this, QA is

hardly seen as a productive and well-spend effort in organizations. However, due to the

capability of instant feedback from gamers through App Markets, concerns about user

retention have changed the way in which serious mobile game developers think about

testing. Bad ratings and feedback lead to low number of downloads and eventually causing

the game to be a rogue investment.

Does crash reporting help then? Absolutely yes, but that only provides you the information

that your mobile game doesn’t work, crashes or users experience something else bad

– and that’s already too late. Your game has been exposed to users, negative feedback

is publicly available and fixing an issue doesn’t always help. In some cases, bad ratings/

reputation can even prevent getting next titles approved, or for sure not getting that title

in front of hundreds of millions of gamers.

THE THREE FACTORS FOR IMPROVED VALUE

The value-add of testing brought into development process can be also seen through

effectiveness, speed up development (improved productivity) and possibility to release

your title sooner (time-to-market). All these ‘improvements’ can directly be measured

in the terms of value:

Effectiveness of your QA on finding and development on fixing the defects. This is the

no. #1 cost driver in today’s mobile app and game development. As said, direct costs of

defects have a direct effect on your bottom line, like the actual cost of fixing the defects

from testing process, cost of fixing and verifying crash reports coming from the market,

and lost customer acquisition costs (CPI) and lost lifetime value of the customers (LTV,

affecting top-line).

Share this eBook!

Effectiveness very oftenly is easy to be neglected as a cost driver, but when you put

your figures honestly on the table, you will find that late bug-fixing is consuming a lot of

developers’ time as well result in delaying the release, which then equals to lost revenue

and lost customers.

Time-to-market cost directly affecting your top-line. Time is a crucial attribute not only

against competition, but also to gain revenue as soon as possible. When you launch your

game, every day counts as you start generating revenue earlier. The important metrics are

monthly/daily acquired users (MAU/DAU) as well as average revenue per (daily active)

user (ARPU, ARPDAU). Remember that the delays with publishing aren’t coming because of

testing – if your development process accommodates true agile development and testing.

Many game developers do not calculate the impact of delays, but frankly, it is quite easy to

calculate impact of each day of delay to your top-line. Manual testing – even with mobile

games – will delay the launch even more. The solution is the agile, test automation driven

development with continuous integration, continuous testing and quality assurance on

real end user devices, all the time during development.

Productivity of your QA. This is the most obvious one affecting your bottom line. The easiest

way to further analyze the productivity is to compare the costs of manual testing efforts to

cost on test automation tooling & automation costs. While this gives you only the financial

cost estimates, it is also important to consider other aspects: the common approach with

mobile game testing, unfortunately even still today, is that all of that important QA and

testing work gets done at latest phases of the development process.

Share this eBook!

HOW TO ADD VALUE TO GAME DEVELOPMENT PROCESS?

Effectiveness. The easiest way to tackle the challenges related to effectiveness is to integrate

game development and testing process into cohesive agile development process. This

enables you to automate huge number of different types of tests, using real mobile devices

(all sorts of those) and check code after every build (daily/nightly/weekly builds). The

effectiveness has three following attributes and advantages:

1. Instant feedback on defects. This is widely documented benefit of agile practices

like continuous integration and testing. Mobile

development brings the additional dimension with

automating the process on real devices your end

users are using. From developers point of view,

the instant feedback on defects increases the

productivity dramatically, since you still have the

code you recently touched fresh in your mind. Who

remembers what you had for lunch 2 weeks ago vs.

today?

2. Effectiveness in defect finding prior to the

release will lead to fewer field failures. According

to Crittercism’s research, the gaming apps are the

least robust and have the highest crash rate of any

app types. Let’s take an example: a mobile game

with 50,000 monthly active users make 2,200 crash

reports every month. If you have 3 games in the market, that would be 6,600 crashes per

month. If the analysis-fix-verification takes around 1h, the cost is huge. Not because of

engineering costs, but cost to the reputation, given-up users and potentially lost revenue.

Share this eBook!

Proactive testing on real mobile devices before the release would have improved and

made the game more stable – and therefore the development effectiveness would have

been better by factor of 2-4. Post-launch monitoring is naturally still crucial, but you better

not be in reactive mode what come to defect finding, or it will cost you a lot.

3. Lost CPI and LTV. You know how much each install costs you, and you probably are

aware of the LTV of your users. If your game fails after a release it is highly likely that you

will lose both the CPI as well as LTV of that user/all users using the device model. It is

highly recommended to plug in your CPI + LTV for each crash coming from the field to

understand the impact of the failures.

Being proactive, and using true Agile development process

with continuous game testing approach on real

mobile devices, will lead to higher top-line

and highly improved bottom line as well as

happier developers: Every developer wants

to work on the new features rather than

fix existing features in fire-fighting mode

based on the crashes coming from the

market. Remember! Manual testing is

not agile, and it is not an answer for

those mentioned aspects. You do not

get instant feedback. If you want to

test your game manually on top 50-100

devices for example, the cost of doing

that bi-weekly or even monthly is high, and

practically impossible. You should do it daily

or nightly.

Share this eBook!

Time-to-Market. Shorter time-to-market drives up your top-line. The earlier you launch,

the earlier you start generating revenues. When you release your new game – or old game

with new monetized features – 2 weeks earlier than previously using agile combined with

automated mobile testing, that is 14 days shaved off from every release cycle. If you do

6 releases per year (every 2 months), it is 84 days saved annually.

The maths for lost revenue is really simple:

Suppose 100,000 daily average users (DAU)

Suppose $0.05 average revenue per daily active user (ARPDAU)

14 days (x) 6 release/year (x) 100 000 (x) $0.05 = $420 000/year/game more to your

top line.

If you have a portfolio of 5 games, total would be around $2M from faster time-to-market.

You can achieve all this by shaving off 2 weeks in every release cycle. Not all games are

directly revenue generating, so replace ARPDAU with your specific value for each mobile

user.

Again, time-to-market is improved with integrated development and testing process.

This is the case for all software development, but in mobile app dev it is the only way to

do development efficiently. There is a huge number of end-user platforms (e.g. different

devices with different hardware and software setups), that your games should support

and ever increasing complexity in games that need to be tested on those devices in every

release.

You can automate huge amount of testing, and do it on real devices after every new code

checking and/or nightly builds. This will shorten your development-testing cycle, improve

defect-fixing times and result in improved top line.

Faster time-to-market also keeps your competition at bay. Your competitors will not

get a chance to promote their offerings over yours, which is always a danger when your

Share this eBook!

release starts to slip. You can release more often with more confidence in the quality of

your releases.

Productivity. The quality assurance (QA) is a critical part of the process when creating

and maintaining successful mobile games for hundreds of millions of gamers. One of

the metrics in this game creation process is naturally QA’s productivity – the success to

efficiently find and filter out the problems in games before they land on hands of users.

This type of productivity of quality assurance can be measured by various metrics, for

example:

Bugs found rate – how efficiently the QA team finds bugs during the testing process per

time used on testing

Post launch issue rate – how many issues are found after the go-live release

Customer satisfaction rate – for example complaints to customer support, issues reported

to app stores, overall ratings of the games

Quality of deliverables and issue documentation – how easy it is for developers to

understand the issue and make a fix based on the documentation

The number of new test cases added per period of time

The challenges unique to mobile game development & QA are device diversity, different

operating system versions, hardware configurations and OEM/carrier customizations.

Those aspects increase the complexity of QA and result in longer development cycles

(time-to-market), higher fields crash rates (lower customer satisfaction and uninstalls

hurting your top line) and higher customer support costs (more customer complaints) and

inefficient development team (dealing with customer issues vs. developing new features).

In order to improve the QA productivity, teams need to have access to real devices – the

very same ones that end-users use.

Share this eBook!

Developers and QA teams need an access to those devices exactly when they face an

issue during development, they are fixing post-launch issues or get customer complaints.

The earlier issues get found, the lower the cost of fixing them. To maximize bug-finding

efficiency, you can automate testing, and do it on real devices after every new code

checking and/or nightly builds. This will shorten your development-testing cycle, improve

defect-fixing times and result in improved bottom line.

When using the cloud based service for mobile development and QA, the performance

of your team will improve a lot: The cost is fraction of the manual testing, developers are

happier (instant feedback on new code, no need to deal with purchasing when acquiring

new devices), product owner is happier (higher customer loyalty, higher LTV, faster time-

to-market), customer service is happier (less customer complaints) and most important of

all, your customers are happier as they do not get frustrated by “not working on my device”.

Share this eBook!

There are several practical ways to add value to existing mobile game development

process. If you focus on mentioned three items and follow the agile process with continuous

testing, you’ll very quickly gain benefits and be able to build more robust games. Here is

a check-list to move forward with better quality:

1. Consider automating most of the testing, choose any of the excellent open source

testing frameworks available for the most popular mobile platforms

2. Avoid totally unnecessary vendor lock-in by using standard languages (Java, dotnet,

perl, php, python, ruby), there is a pool of millions of developers using these languages

to tap into

3. Automate as much as possible of the testing to improve the organizational agility (even

make developers responsible for creating automated tests)

4. Use continuous integration and continuous testing, testing is a continuous process in

agile and will deliver you results 24/7

5. Integrate with your test management systems to improve the collaboration and

transparency throughout the organization. Tools must enable a free flow of collaboration

6. Instant accessibility. The development teams do not have time to engage with long

sales cycles, procurement processes for each tool and device. Choose a service with

rapid provisioning without waiting for hardware, procurement or internal processes.

7. Finally, the most important one: Tools matter, but how you use them matters more.

Tools make a difference, but how you use them will ultimately determine your level

of success.

Share this eBook!

ConclusionB uilding a successful mobile game

starts with understanding how

intuitive and great gameplay can be

enabled for gamers. Making sure that

users understand that they are fully

capable to use all great features (and

graphical stuff) you’ve built in your

mobile game makes a difference. The

usability, user interfaces, graphics assets,

OpenGL ES content and implementation

are actually those that form the playing

experience. And eventually, when

gameplay is great and gamers stick

with it, you’ll see that as an increased

potential to monetize that enthusiasm.

By implementing the essential infrastructure and architecture, you will be able to make

sure that you are testing for the real environment and real gameplay. Then adopting agile

process in your mobile game development and testing would bring the greatest value

to your daily tasks. In the end the value of agile testing is far more than making your life

easier - enabling your company to deliver the continuous quality of app and generate

continuous revenues.

Share this eBook!

Thank you for reading this

e-book!

To learn how agile process can help you boost your mobile game development and testing. Check out Testdroid Cloud.

Upload your app and run the test on thousands of devices -

You’ll know the agility we can bring to you.

Drop us a line about your thoughts at sales@bitbar.com and we will provide you with

the expertise and everything we know to help you make the right decision.