When I started working at King, I randomly picked up a task called something in the line of “make test automation work.” The job had already been started by another coder who had left the company and my task was to pick up on his work and get something running. What I didn’t realise is that a year later I would still be working on it, now having more than 70 UI tests running across multiple platforms. Also, test automation was completely new territory and getting to learn things like Java, Gradle, Gherkin, Jenkins etc was a great development opportunity for a die-hard cpp programmer like me.
Back then we gathered the entire team every second Wednesday to play through a series of highly monotone test cases. Today most of them are automated, and we can focus on what’s important: ensuring our game is fun to play. Our Agile Test Lead (ATL), Johan Hoberg, has been transforming the manual testing into a more creative exploratory form. We still have our test Wednesdays, but now they are more focused on playing the game than checking off test scenarios. And our test ninjas, Jens Rosander and Ioana Popa, now have more time to find those rare issues like that time a web view was incorrectly rendered after switching language to Japanese on a specific device.
So if you’re considering adding test automation to your game, I believe it is well worth the effort. It provides a great sense of comfort when doing big refactorings or changing core parts of the engine. In addition, it forces programmers to write code in a more testable way, so you will most likely get a set of debug tools that can also be used during manual testing. However, you must be prepared to invest a lot of time in it. And to do so continuously since things need to be monitored and maintained. For us, the first 3-6 months didn’t give much value back, but as soon as things were stable and the test count grew beyond 15-20 we started to really see the benefits. Now we’re in a state where we can’t live without it and it has become natural for everyone to include tests when developing new features.
How does it work?
First of all, our test cases are written in Java/Gherkin, and like most other King games we’re using the internally developed Test Automation Framework (TAF) as a base. A typical test definition could look something like this:
Each line in the test definition is matched with a Java function using a regular expression. The Java framework is structured into different classes for each GUI screen or feature area. For example, the start screen class has functions matching the buttons on the screen. The playGame() function will wait for the start screen to be loaded, then ask the game for the coordinates of the play button, and finally execute a click at those coordinates. If it should fail to do any of those things, an exception is thrown. Another part of the code catches the exception, generates a screenshot, and fails the test.
On the game side, we’re running an http server which we send commands to from the test code. For testing on mobile devices, we’re using Appium to simulate clicks, while on Mac it’s just faked on the client executable. The test code is executed using a Gradle script on Jenkins.
On Jenkins, we have a job (Entry point) that schedules the different build and test jobs associated with the triggered test run. It can be started manually, from the nightly build or from a git hook. Build jobs are triggered for each platform in parallel, and the test jobs are then started as soon as each build completes. When all tests have finished, the report job assembles the final report and sends it by email.
- Manual ~ When working on bigger features, it’s nice to be able to run the full test suite before merging back to master. We can also use tags to test everything related to a specific part of the game.
- CI ~ 20 minutes. Runs for every commit to master. It includes 2-3 tests for each feature.
- Nightly ~ 2-3 hours. Includes all tests we have. This is automatically triggered every time we build a release candidate (which we do every night and sometimes manually during the day).
Key success factors
Buy-in from developers and product owner
Both developers and product owners have believed in test automation from the start. The fact that I never had to fight anyone to get time for working on it has played a big part in our success.
Quality before quantity
Adding new test scenarios is quick and easy but if they are unstable and half of the report is red most of the time, they are all useless. So starting slowly and fixing all those edge cases have been a key component in making automation work for us. The first month we had only 1-2 tests running on a single device just to make sure the setup was stable enough.
Don’t let the slaves idle
The best way to catch all those edge cases is to constantly run tests. So one of the first things we did was to set up the continuous integration job. Not only will it catch bad commits quickly, but it also helps you getting really stable tests.
Not letting it slide
When a problem shows up, it’s essential to fix it immediately because fresh bugs are usually easy to fix. But if you ignore the daily monitoring, eventually a large part of the scenarios will be red and it would require a lot of digging to find out when they broke and why.
The default jUnit report is good for feeding different Jenkins plugins etc, but when it comes to the final report I think it’s worth the effort to make something custom. You are going to stare at it a lot, so make sure it looks good, is easy to navigate, and includes all relevant information (call stacks, screenshots, logs). Our final nightly report now contains about 50Mb of data, but we also have a trimmed down email friendly version of it.
Dealing with false negatives
There are a lot of ways the tests can fail that are outside our control (unstable network, power failures etc). While we could have spent a lot of time trying to create a fully isolated test environment, we have focused more on making sure errors are clearly visible in the test report. So when something breaks, our QA ninja Jens would do a quick manual test. That said, the norm is to have everything green, so if something can be fixed, we’ll do it.
What’s next for us?
More tests! And in parallel
All our tests are currently running on a single Mac Mini next to my desk, and nowadays it’s always busy. Being able to hook up with the 100s of phones in the device lab in Barcelona would be great since we could then run more tests on the actual phones. But as a first step I’m looking into parallelising the Mac tests, so we can run multiple instances of the game on the same machine.
Moving towards Test-Driven Development (TDD)
We’re now involving testers earlier in the design process of new features. For example, when we have a somewhat final design on a feature, testers and coders work together to create a list of Given/When/Then scenarios. At this stage, parts of the feature are usually already implemented, so while we’re not exactly doing test driven development, we are taking some steps in that direction.
A shared responsibility
Now with everyone writing tests, the responsibility of debugging them when things break has also been spread. I’m still the go-to guy for test automation, but it’s great to see that other team members are now much more involved in the daily monitoring. We still have a bit to go in this area, but I think we’re getting there.
Our game code is in a much better state now than it was a year ago, and test automation is part of that success. Bugs are caught earlier, which in turn makes our release candidates more stable and we rarely need to do patches late in the release process.
It’s great to see that the other games we are working on are including test automation from day one, but I think we have also shown that it’s not too late to add it to a mature game like Pet Rescue Saga.