debugging-falling

Debugging 102 – fixing things faster

As developers, we probably spend more time debugging than any other task (maybe a contentious issue, but in any case debugging takes a big chunk of time). Yes, we should keep the code simple, have design reviews, write unit tests, automated tests, and code reviews. This post is not about these things, which is not to say these things aren’t important. Just like skydiving without a parachute just because you have life insurance, being good at debugging is not a substitute for good development practices!

I will explore three different ideas on how to leverage debugging tools to fix bugs faster. To fix bugs faster, I will focus on the part that takes the most time. That is, getting a 100% repro to happen (the majority of the time fixing a bug is trivial once you know why it happens and can make it happen on demand). The ideas here work best with King’s internal Fiction Factory game engine, targeting our Saga-type games, and it is assumed that the reader already has a working knowledge of debugging. If you are interested in Debugging 101, let me know, perhaps that is another post.

Breadcrumbs

The dreaded crash bugs. The story is simple, your game crashes, this is bad. Fixing it however is harder than it seems. If you’re lucky, it will happen to you while you have the debugger attached. If you’re unlucky, it will happen on some Android devices on the other side of the world with a build for which the symbols didn’t build properly (and no crashdump is available). The tricky part is to figure out a 100% repro for getting it to crash.

Breadcrumbs are a log of events that you add to a circular buffer. If your game crashes, that log is saved to a file and then sent to a server for aggregation. All this magic is done as a last heroic effort  through our engine and the crash analytics side before the game dies.

There is a limited payload, so breadcrumbing everything will quickly exhaust its usefulness. We have had success tracking button presses, screen transitions, out of memory system events, app loss/gain of focus, level starts, and suspend/resume calls. Now analysing crashes gives us a little more context of what the user did before things went wrong, and hopefully we can reproduce the bug easier.

breadcrumbs
Sample breadcrumbs file from crash analytics. It will probably be easier to reproduce this crash on level 57.

The example above is a breadcrumb log. The middle number is the number of seconds since launch. So we now know the user played for 18 minutes, and in level 57 on a device that triggered a low memory event before the crash during gameplay. It’s not a huge amount of information, but it can be correlated with other crashes to narrow the suspects, and it’s much better than “I was playing, and the game crashed….”

The console

One of the staples of video game debugging is the console. That is the ability to type instructions to the game and have it perform some actions (for example, enabling godMode). The console is really useful for setting global game options or some kind of state. This allows you to better set the environment for your bug to happen.

console
Sample console from Scrubby Dubby Saga, probably just what you imagined!

Imagine the bug is a fault with the ‘rate us’ pop-up. It will be much easier to fix this if the pop-up is shown on every entry into our Saga map or upon completing any level. Using the console, we can set the chance of showing it to 100% and voila! Of course, this requires that your game supports a way of setting config variables at runtime from the console. If it doesn’t, maybe it should?

Here are some useful console commands:

  • Setting progression/lives
  • Setting a specific transaction outcome (for example, purchase failed)
  • Capturing a dump file (memory/file access/state/and so on)
  • Resetting a random seed
  • Changing language
  • Setting the move counter to 1 (only happens on the last move)

The downside of the console is the typing. This issue is twofold: first you need to know what you can type in the console, second you have to be able to spell (you’ll soon understand when you try typing something like ‘set tweaker popup.dialog.Fun-Or-NotMinimumLevel 5’ on your mobile – oh yeah, it is case sensitive). If you’re implementing your own console, an autocomplete, history buffer (very handy for spelling mistakes), and setting up aliases (typing ‘sp’ is easier than ‘setprogression’) are essential.

Lastly, don’t forget typing on mobile is much harder than on a keyboard. Think about the layout of the mobile keyboard: having to mix letters, symbols, uppercase, and numbers means a lot of extra button presses… For example, ‘Fun-Or-Not’ takes  17 key presses on a phone, which is 70% more than typing ‘fun.or.not’.

Cheat panel

The console is great for things that can be expressed with a command. In our games, we have a board that contains different pieces, and often there are bugs with how these items interact. These bugs are very hard to reproduce by just playing.

Imagine a bug that causes the power-up not be generated when you complete a five-in-a-row. This would be fairly easy to recreate in a test level by playing. But what if it only happened if the match occurs in the top left corner? Reproducing this just by playing would be very time consuming. Enter the cheat panel.

The cheat panel is a pop-up that pauses the game state and enables a visual menu of available items. Once an item is chosen, the game switches to a paint mode where the board can be painted with the current item. This can be repeated until the right situation is created, and then gameplay can resume. Now setting up the exact situation where the bug happens is a simple task, and your bug can be fixed.

cheat_panel
Example Cheat Panel from Scrubby Dubby Saga. Just pick your piece and start painting.

With the cheat panel, you (and your QA team) can now easily reproduce any scenario to ensure all of the features are working well: for example, what happens when you mix the frog with a striped candy, what about a wrapped candy, and how do the pieces fall when two blockers are next to each other?

falling
Does falling work right? First, we set up a match below the blockers. When we resume the game, the match happens and an empty square is left below the blockers as expected. Without the cheat panel, setting this scenario up would be very difficult.

Conclusion

In review:  Breadcrumbs provide clues into how to make ‘that’ crash happen on demand, and are even more powerful when aggregating a large set of similar crashes. The console is very useful for scenarios that can be expressed as commands. If you can’t easily express the scenario as a command, add it to your cheat panel. Armed with these tools, you can put your time into fixing ‘that’ bug (and adding a test, so it never happens again) instead of spending that time trying to reproduce it.

Tomasz Niwinski

About Tomasz Niwinski

Tom has been making games for almost two decades and continues to have the privilege of working on some great games with some outstanding teams. Following the scent of adventure, he left AAA console games on the wet coast(*) of Canada for the sunny shores of Barcelona (the only change for the family was “everything”; though he continues to be a strict part time vegetarian). When not in front of the computer, he is failing at KOM’ing Strava segments around Catalunya. (*) While Vancouver also happens to be on the west coast, the rain is what gives it its name.

One thought on “Debugging 102 – fixing things faster

  1. These kind of tools are indispensable to create a great products, because allow the best results in all aspects of the game. In my company, usually, i insist in the importance to have a good testing tools. It’s the key of a excellent product. Nice work guys.

Leave a Reply

Your email address will not be published. Required fields are marked *