With over 300 million monthly active users, the challenge is set: make sure King’s games work for everyone! When you consider the wide variety of mobile device models, manufacturers, and operating systems out there, validating that each game works on all possible combinations of these can be quite painful.
At first we hired an army of testers to perform regression testing of our mobile games. Sitting at the very end of the pipeline, before a game gets released, this team’s sole mission was to find bugs and knock the games back to the studios so that the issues uncovered could be addressed.
In an industry in which the speed to market is undoubtedly an advantage, as you might have already figured by now, this solution couldn’t scale up forever.
As our studios grew and expanded, as we released more and more games, existing games became more complex, and the amount and scope of work rapidly became too much for a manual testing team to handle. Some workarounds were made to speed things up – we tried reducing test scope and decreasing the level of priority for component tests, such as video ads and game engine components – basically exchanging quality for time to market. Such compromises however, are not what we are about at King: studios wanted to know why can’t we have both. Too much is at stake here, not to mention that in an ideal world regression should be done after each iteration – but that’s out of question here; it just takes too long.
What could we do? Automate all regression tests? Hmm, we understand some methods of testing are better suited for manual testing, where they are executed by a human. (Johan Hoberg explained on his blog post how manual exploratory testing, for example, proved to be successful in the Pet Rescue Saga team.) Oriol Marti, on the other hand, provided a different perspective in his blog post, stating that “manual testing is not scalable for mobile apps, as there are too many devices and operating systems to test on.”
So that’s what we aimed for, the ‘RoboCop’ approach: part human, part machine! Leveraging both automated and manual testing for optimal results. (Cue RoboCop theme.)
Device Cloud: Part Human, Part Machine, All Tester
The Device Cloud is best described as a cluster of mobile phones, accessible through a REST API and made available to different teams across King to attend their testing needs. The idea is to allow different studios and technology teams to execute their automated tests on multiple devices. The advantages are pretty obvious:
- Tests can be executed by anyone at any time, 24/7
- Fine-grained results and automatic generation of reports
- Access to test execution logs and screenshots
- Performance (CPU, memory) and event tracking tests that go way beyond game play
To make this system work we had to play around with the mix of hardware and software components we were using. We knew we wanted to implement a distributed environment, and picked a tech stack with that in mind, mainly built around Akka.io which has the added advantage of providing fault tolerance out of the box.
How the Device Cloud Works
Users prepare their test automation executions (upload game file, upload test script, select devices, etc.) by interacting with an API. Non-techies can of course use the web interface. Once everything is set up, the user is then able to trigger a final ‘run’ operation to send the execution to the core system.
This final operation returns a unique identifier of the given run, which can then be used to monitor the status of the execution. Once it is finished, the system generates a report containing test execution results, including logs and screenshots, that is then made available through the API (or web interface).
Under the hood, communication between the API and the core system is done via a message bus. Incoming messages eventually land on a scheduling system responsible for assigning each test execution request to a physical device. Communication to the actual device is done via either the Android Debug Bridge or the Xcode Instruments.
Other components, such as monitoring dashboards and alerting tools, were also put in place to monitor the devices and signal abnormal conditions. This helped us scale our infrastructure to 100+ phones, understand patterns, and save time in debugging device-specific issues.
The Best of Both (Testing) Worlds
Different King studios and teams have already adopted the Device Cloud as part of their testing strategy. The Barcelona Phoenix Studio were the first ones, and we worked closely with Test Automation Engineer, Ricard Perez to get it up and running with all their five titles. Their experience was that the adoption of the Device Cloud allowed them to scale their infrastructure of devices used for test automation without having to worry about all the deep details and configuration. They tested it all the games they produce on several devices in parallel, both iOS and Android. It proved successful in finding a number of defects.
It’s not just about games, though: our shared technology teams also benefit from the Device Cloud versatility. Our AdTech team, for instance, has been regularly using the Device Cloud to test the integration of video ads in King games. Here is what Enrico Carniani, Senior Developer at the AdTech team, has to say:
“We developed an in-house solution, called Victor, that allows us to run automation tests locally on iOS or Android devices. In a short time, we recognised that we definitely needed to scale up and run our tests in a much wider selection of devices. Also, our software was not able, alone, to run in parallel with multiple devices at the same time, it was a frustrating manual process. We also needed some company-wide service in order to make automation tests a part of the standard production loop: we surely could not rely on our laptops and a remote Jenkins. Device Cloud helped us fulfil all these objectives. The whole system is simple, yet effective and valuable, allowing us to uncover critical bugs early on the development life-cycle”
Is the Device Cloud a solution to all our testing challenges? No, but it certainly helps. As Alan Page puts it quite nicely in The A Word, “Test automation is just one tool from our tester toolbox that we can use to solve a specific set of testing problems. Like most tools, it works extremely well in some situations, and horribly in others. Use your brain and make the right choices.”