Tratt workflow

The story of the Tratt

At King, everything that has to do with analytics starts with an event. An event is an atomic occurrence that take place either on the game client when the player does something in one of our games or on any of our backend systems; it can be that the player starts a level, clicks a button, or uses gold bars to buy an in-game item, such as a lollipop hammer. The events are used by all the analytics’ teams at King, and because our business model relies heavily on understanding how our players engage with features of our games, it cannot be stressed enough how important it is to capture the right events and make sure they have the right content. Without them we are basically flying blind.

tratt-goldbar-workflow

The tracking system is the infrastructure that records that an event has happened and makes sure it is stored on our backend so it can be transferred and used in our analytics systems. We have the same basic tracking system today as we did then, which is very simple but has proven to scale well. To capture a new event, a developer registers the event in our metadata, gives it a name, and describes what fields it should contain. To fire the event it is inserted at the right place in the code and all fields are filled in. Firing the event means calling the API on the servers where it is written to an event-log and then written to the Apache Kafka instance, the message transport system we use. If it is a server-side event no API-call is done; the event is just written to the log and to Kafka. Kafka feeds our real-time analytics systems, whereas the logs are regularly transferred to our data warehouse to be pushed through the data pipeline.

The QA for tracking project started around August 2013, but at that time we did not think of anything like a Tratt for automated tracking tests. King had around five games out on mobile, of which Candy Crush Saga was the biggest and was really picking up speed. This meant that the volumes of game data that came in were growing every day. We also had Pet Rescue Saga and Bubble Witch Saga, as well as some other smaller games on mobile, for example, Bunny Hop and Miner Speed. In total around 5 billion events entered our systems every day.

Tratt funnel

At that time, we tested all events manually, that is, if we tested them at all. Usually games were released without any tracking testing at all. If tests were made, they were very rudimentary and those performing the tests had no tech support and very limited knowledge about what was the purpose of the tracking. The main problem, however, was that the tests were very much a manual task. It comprised a written description of how to trigger the event, following which the tester had to use a very simple web-interface or tail the server logs to make sure that the event ended up in the log or on our Apache Kafka. The content of the event was basically never checked, not to mention the sequences of events. Very often you are interested in sequences of events, for example, you want to see that a player installs a game, starts it for the first time, plays the tutorial, and goes on to play the first level. In these cases, the events are ‘linked.’ There can be fields that should contain the same value through the whole sequence so analysts can reconstruct it later. Writing a test case and checking such a sequence under those conditions was almost impossible.

Since we updated our game clients at least every two weeks, and had several backend releases every day (totaling say 20-25 releases spread out every two weeks), the chance of a tracking bug slipping through was quite high. And tracking bugs did slip through. A lot of them. And they were serious bugs too, such as not knowing how much money we made on a game for a week. For a while it was almost like we could not release anything without broken tracking. Analysts were frustrated because they could not focus on analysis but instead had to spend hours on cleaning and repairing data, figuring out ways to patch and compensate broken data, so that any conclusions at all could be made on the performance of our games. Game producers and business managers shook their heads because it took a long time to see if the time invested in new features had paid off or if our players didn’t like them and they therefore should be closed down. The frustration spread across the company.

This led to that a small team was created with a simple mission: ‘Fix tracking.’ From the beginning it was just one person and the focus then was to make sure that new games were released with working tracking. That is, the bare minimum tracking should work. To solve this the ‘tracking-guy’ worked close with the game teams that was about to release and helped them do test tracking. It was a completely manual process, but at least time was set aside from the game team to focus on getting the tracking right. This process continued for a year, another person joined, so now two people could help new games to release with working tracking. But this only brought us so far. The tracking-guys could only support the teams on their first release, after that the teams were on their own. As we have a lot of releases every day, updates, bug-fixes, new features, etc., tracking bugs kept coming. There was no way for the small team to keep up with the cadence of releases, especially as King now had around eight games live, with say ten releases every day. The approach didn’t scale.

Some tracking team members thinking about next-gen tracking:Tratt team discussion

The turning point came in early 2015 when a professional QA person joined the team as Agile Testing Lead (ATL). At this point we were five in the team: three tracking guys (former data scientists) and two developers. Together with the data analytics team we had built a funnel-analysis tool. The reason was not primarily to get a tracking test-tool but rather a tool that could decouple funnel analysis from the game code. We wanted to give the data scientists an easy way to do funnel analysis and get away from keeping funnel-states on the game-clients and backend that bloated the code, and we did that. This tool had a simple way to define the event sequence and relations between the fields in the event and could be run on data streamed from Hadoop. We didn’t know it then, but as it turned out this tool was to become the core of today’s Tratt system (in Swedish “tratt” means funnel).

The ATL, who in contrast to the rest of us in the team actually understood how to get testing to work in a larger context, realized that the only way to get this to scale was to create tools and automation that the embedded QA people and QRT teams could use. The focus for the team became to get such a system up and running. The first step was to build a simple end-to-end system where events triggered in the game were fed from Kafka into the funnel-analysis tool, which in turn could report if the test failed or not. In a week or so the team got the prototype system up and it became clear that if we could produce a tool like this we could be a long way towards working tracking.

Tracking Web tool – Running GamePlay Tratt:tratt (4)

Getting a new system out on a broad front in a company that has many independent studios and dev-teams across the world is quite a task. We built a stable version of the tool and then the team went on a road-show around our studios to demo it and help teams start using it. Very soon, on the basis of feedback from teams using it, we were able to improve the tool gradually.

The Tratt-system we use today has basically reduced the number of tracking bugs to zero. Three years ago we could hardly release a game without serious tracking bugs, but today we haven’t seen a serious one for a year or so. And this is when we have around 30 billion events coming in every day and many more teams than before. Now we introduce the Tratt-system to every new game team and automation is built in at very early stages in a game’s life. It takes around two days to educate the team on the system and to do initial integration of test automation; after that they can use it on their own. It may sound like big investment for a small game team early in their process, but it pays back many, many times later because all key people, data scientists, devs, and QA, get involved early so knowledge and understanding on how to create, integrate, and test tracking is in the team from the beginning. Also, once the system is in place it is minimal effort to add or change tracking. Just update the tratt-file and the whole test-system is ready for the new tracking. We also have a CIA-KGB-extension to the Tratt-tool that allows the central tracking team to monitor and record every time a test-case is run on King. We can follow up the usage of the system and this makes it possible to help out if a game team runs into problems when implementing or testing tracking.

Tracking Web Tool – Live statistics showing Tratt executions in CI environment:Tratt comments

Technical Overview

The tratt-api is a regular Java library API and can, for example, be used with any test automation framework, (for example, JUnit or TestNG) or serve as a backend for a frontend GUI. The tratt-api has been open sourced and is available at our GitHub: https://github.com/king/tratt-api.

Sequences of expected events, aka CheckPoints, are specified using the Tratt Definition Language (TDL). A TDL is written using an easy-to-understand JSON format and serves as validation rules.

Example TDL file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
{
"comment": "Comment describing the purpose of this TDL",
"variables": ["userId=some-user"],
"sequences": [{
"type": "CONTAINER",
"name": "SequenceA",
"sequenceMaxTime": "PT60S",
"checkPoints": [{
"eventType": "GameStart",
"match": "userId==$userId",
"validate": "level==2 && timestamp>=0",
"set": ["GS_level=level", "GS_timestamp=timestamp"]
}, {
"eventType": "GameEnd",
"match": "userId==$userId",
"validate": "level==GS_level && timestamp>=GS_timestamp && score==123 && endReason=='some-reason'"
}]
}]
}

Code execution

Starting the tratt-api will asynchronously start monitoring the event source. Events are processed and validated according to the CheckPoints in the TDL. The expected events can either be triggered manually or from within an automatic test. The tests will be successful upon fulfillment of checkPoints’ criteria or fail if checkPoints are not fulfilled, or upon timeout. The below image and code illustrates this in practice.

Usage example:Tratt workflow

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import static java.util.concurrent.TimeUnit.SECONDS;
import org.junit.Test;
import com.king.tracking.api.StartedEventProcessor;
import com.king.tracking.tdl.Tdl;
public class ExampleTest {
@Test
public void canProcessEventsAccordingToTdl() throws Exception {
// Configure Tratt-api
StartedEventProcessor started = Tratt.newEventProcessorBuilder()
.setTimeout(5, SECONDS)
.addEventIterator(new MyEventIterator())
.setEventMetaDataFatory(new MyEventMetaDataFactory())
.setValueFactory(new MyValueFactory())
.addTdls(Tdl.fromPath("classpath:com/king/tratt/acceptance.tdl"))
.addVariable("userId", 12345)

// start monitoring the event source asynchronously
.start();
triggerYourEventsHere();
// then wait until fulfillment of checkPoints' criteria,
// or throws exception upon CheckPoint Failure, or timeout occurs.
started.awaitSuccess();
}
private void triggerYourEventsHere() {
// do things here that will trigger the expected events.
}
}

Conclusion

Lots of resources has also freed up in the central tracking team. Actually, not much of their time is spent on tracking nowadays so it’s not just a tracking team anymore, it’s more of an analytics tools team. Now they spend time on improving the tracking system as a whole instead of fire-fighting. And also building and improving our other analytics systems, like the experimentation engine and our segmentation tools. The Tratt-system is now used by more than 50 teams across King and now we have control over basically every release, about 25 of them every day. No one ever speaks about tracking anymore, a sure sign that something is just working. Looking back three years it feels unreal.

Automated tracking implemented in all franchise games celebration:Tratt team

Alexander Poulikakos

About Alexander Poulikakos

Alexander is a well-experienced software developer and has a background within telecommunications and data communication. He has a great competence in testing, developing test tools, creating test automation frameworks, and providing automatic test writers with simple-to-use APIs; an example of his latest work is the tratt-api. Alexander spends his free time collecting hip hop vinyl records, biking, and programming on hobby projects.

Leave a Reply

Your email address will not be published. Required fields are marked *