Candy Crush Soda Saga delivery pipeline

We release a new version of Candy Crush Soda Saga every other week on iOS, Android, and Facebook. It’s a complicated picture – two code bases (C++ for mobile and AS3 for Facebook), as well as dependencies on internal libraries and frameworks that are updated frequently. It’s the kind of juggling act that we at King are constantly working to simplify and improve. Here’s how we’re currently doing it.

Version source control

version source control

The Soda team uses the same Mercurial repository for both code bases, and while the game logic is completely separate, the two applications share graphical and UI assets. We use a mixture of trunk-based development, feature branches and feature toggles. Some of our features can take a couple of sprints to be completed, in which case they’re implemented in a feature branch. Some features are developed in default but not enabled, most likely strong candidates for A/B testing. Finally smaller increments of code that can be completed within a day or so are implemented in default right away. As we gear up for a release, we create a Release Candidate (RC) branch. The RC branches are our only official branches.

Continuous integration

continuous integration

Jenkins is our continuous integration (CI) tool. With every change to the repository, both code bases are built and unit tests are run. Every successful mobile CI build results in runnable apps, and we run automated system tests on those apps several times a day. We also have a lightweight server layer written in Java, packaged with the AS3 flash app. The server code and flash app are deployed on demand to our test servers. The deploys can be easily done by any member of the team via Jenkins. We also use Jenkins for automating tasks such as creating RC branches or importing data that’s generated externally into our repository.

Releasing

As mentioned above, we first create a release candidate – at this point, all new features have already been tested by the group that implemented them. team testingThe automated Jenkins tests are then run on our release candidate. Next, the app is tested manually by the whole team. Everybody participates in playing the new levels and checking that the game looks and feels OK. Finally, the build is sent to specialised device-testing teams, who load the app manually and test it on all kinds of devices. The whole process takes a couple of days, and we allow for time to fix any issues if they’re found. Three days after creating the release candidate we deploy the server, Flash app, and Android app, and we submit the Amazon and Apple apps.

Release retrospective

We end the entire cycle by holding a release retrospective, where we create action points for the items we’d like to improve during our next release.

release retrospective

Who does all the work?

We all do! Since our release process is not yet 100% automated, it requires a technical person to be involved, so the responsibility of pressing the buttons is rotated among the developers, with two assigned per release. To ensure a continuous transfer of knowledge, and to avoid reinventing the wheel with each release, the rotation requires that one of the assignees was also assigned to the previous release, and the other assignee will be on the following release.

The reality

This sounds good, but is this really the way it works? As with all projects and processes this is a rosy picture of the perfect release workflow, when everything goes smoothly. It doesn’t always go smoothly, of course. Sometimes, Jenkins breaks, and the builds don’t go through; sometimes we’re not so great at testing all of our features before we create the release candidate, and spend the next day fixing unexpected bugs; we test too many things manually; every once in a while we even get a live bug that forces us to patch.

That’s the messy reality, but we’re always improving our release process. We focus on shortening the time between creating the Release Candidate and the actual release, while striking a balance between keeping up with the latest releases of the internal dependencies and ensuring that the app is always working. With each release, repetitive manual tasks are automated, and most importantly, we stay aware of our shortcomings and work actively to address them! So our rosy picture isn’t really too far from our everyday reality. 🙂

release

Yassal Sundman

About Yassal Sundman

Yassal is a Crisp consultant working as a Scrum Master at King. She works with amazing teams delivering great games, with a focus on a quality player experience while at the same time ensuring a creative development environment for all team members. With a toolbox of Agile techniques and methodologies, extensive software development experience, and a positive can-do attitude no challenge is too daunting! When she's not working you can find Yassal making slime, mob programming with kids, or hacking away on a hobby project. She also enjoys long-distance running all year round, rain or shine, in Stockholm's beautiful suburbs.

3 thoughts on “Candy Crush Soda Saga delivery pipeline

Leave a Reply

Your email address will not be published. Required fields are marked *