Designing a Gesture Input System – Part 1

A big focus in the game will be the player’s ability to cast spells. In order to do so, the player must draw a gesture into the air, the game will recognize the gesture and cast the appropriate spell. For gesture recognition we make use of a custom-coded vanilla neural network. This is a machine learning algorithm that allows the system to learn any number of things, in our case the shape of spell gestures.

This works by first training the network with lots of example shapes. Once the network is trained, it can then recognize player input correctly. The great thing about this is that the training phase happens during development. So the game ships with a pre-trained network that allows for spot-on gesture recognition. Also, with a system like this in place, we may allow players to define their own spell shapes later in the game.

On the technical side, there are a few considerations that had to be worked out. Some of the more obvious pitfalls are that players won’t all draw their shapes in the same size or in the same position. This is easy to fix through normalization.

But in order to allow for perfectly smooth, frustration-free gameplay, we also want players to be able to draw their shapes in any direction, clockwise or counter-clockwise, and to start drawing closed shapes (such as a square) from any point (i.e. not force the player to start drawing a square from one of the corners). While some of these things were a bit tricky to implement, in the end we pulled it off without too many issues.


Leave a comment