In this article, I will describe how we in the Pet Rescue Saga team moved from manual scripted regression testing to exploratory testing before each release, and what benefits we have seen from this.
But first a look back at how we previously handled our regression tests before releases, and what prompted us to move to exploratory testing. Our sprints last two weeks, and at the end of each sprint, we have what we call ‘Test Wednesday,’ when we run our regression tests. Each development team got assigned a number of test cases to run, and these were divided within the group. Each person sat alone and ran his/her tests in isolation, and often only a few in each teams did all the tests while the rest continued their normal work. Testing is not something that our artists and software developers particularly enjoy, so their initial motivation was low and our setup did not improve it.
So why did we initiate the change? Partly because we saw that running the same regression tests over and over did not yield many new bugs. We did test all the critical areas, so no new major bugs slipped into the live environment, but we didn’t find all those smaller bugs, and old bugs persisted within our game. Mainly, we wanted better test coverage, but also, partly, because it was not very motivating for our coders and artists to run the same tests over and over again.
We wanted everyone in our production team to be involved in the testing of our product – product owner, scrum masters, business people, community people, coders, artists, and of course our testers. What state the game is in before release should be important to everyone, and something that everyone cares about and takes ownership of.
Looking at Self-determination Theory, we concluded that we did not want people to sit in isolation, running mind-numbing tests that were being forced on them. So we decided to adapt our way of working to mitigate these factors, and to evolve our list of scripted test cases into something else. The next iteration of our ‘Test Wednesday.’
We had an idea of what we wanted to achieve, so we created a number of exploratory test missions broadly covering the same areas as the old test cases had, which looked something like this:
On the back of these cards are some guidelines to how to run the specific test mission, but these guidelines are completely optional and only there to support the person who selects the specific mission.
We also changed how we assigned the testing to the teams. Instead of giving a set of scripted test cases to each development team, we instead gathered the entire production team in the morning, and everyone picked a mission themselves, giving them more autonomy to decide what they want to do. During this meeting, we also discussed what was new in the release, and which risks we saw that should get extra focus during testing.
Previously, a few people from the development teams sat running the scripted tests for several hours, but by involving everyone, we could limit the testing to one hour for each person, which makes it more manageable for everyone. So instead of having the entire day dedicated to testing, it would now only be one hour before lunch.
A positive side effect of this setup was that people took their test missions, and then sat down together in couches and chill-out areas and did their testing there instead of sitting in isolation at their desks. Now everyone could share experiences and help each other on the fly, and bounce ideas between each other to improve the testing further. Not only does this help cooperation, but it is also a team-building exercise that brings the production team together.
Another positive side effect is that this approach encourages everyone in the production team to explore the game and all its features, allowing them to improve their understanding of the game, as well as increase their competence in testing.
After everyone has spent an hour testing their respective missions, everyone returns their mission cards and has a short debriefing with the test lead, and then the results are summarised and analysed.
If someone is finished with their test missions before the hour is up they can go to the test lead and pick an additional lower priority test mission and continue their testing until the hour is up.
We complement this exploratory testing by the entire production team with a set of automated system tests, and some more complex and advanced testing performed by one of our test experts.
So the end result was:
- Better test coverage
- Quicker turnaround time
- Higher motivation
- Better understanding of the game
- Increased skill in testing
- More autonomy for everyone to decide what to do
- Better cooperation and communication
All in all everyone seems happy with the change and the results so far have been very promising.