Background

Sketch World is a mobile AR experience that lets you tear holes in your environment and reveal a paper sketch of the world underneath. The experience is inspired by sketch animation and is implemented using the AR creation tool, Meta Spark.

Crafting the Sketch World

To create the "sketch world", we need to take the camera view and make it look like a drawing in a sketchbook. To start, we need to create the illusion of straight sketchbook lines in 3D space. We do this by surrounding the user's phone with a virtual striped sphere. We then render those lines on top of a yellow background to complete the blank sketchbook.

A striped sphere surrounds the virtual representation of the phone. This creates the illusion of sketchbook lines when we look around through the phone's back camera.

To get the actual drawing, we use an "edge detection" algorithm to extract outlines from the camera feed. Then, we distort the outline and give it a "jiggling" motion. This makes the outline look more like a hand-drawn animation. Finally, we overlay the distorted drawing on top of the blank sketchbook to achieve the final look.

Image processing for creating the sketchbook look

Tearing into Sketch World

Now, we need to give the user the ability to reveal the sketch world as they swipe across their screen. To do this, we surround the virtual phone with a black cube that the user can draw on using white "paint". Wherever the virtual camera sees black, we show the raw camera feed. Wherever the virtual camera sees white, we show the sketch world instead.

‍

Drawing on the cube (left) vs resulting visual effect (right)

‍

Sketch Image Masking

‍

Animated Birds

The first time a user tears into the sketch world, a flock of cartoon birds come flying out. We do this to make the "world" feel more alive. Additionally, we add a subtle camera shake effect to increase the perceived impact of the birds.

Each bird is a 3D object modelled and animated in Blender. For a simple animation like this, you could achieve the same result by using an animated sequence of 2D images instead. However, using multiple 2D images would use more storage space than our single 3D object. File size is an important consideration when publishing to mobile AR platforms as they have strict file size limits.

Wing flapping animation in Blender

After importing the 3D object into Meta Spark and placing 3 instances of the bird into our scene: we combine the wing flapping animation with a procedural flocking movement to get the desired effect when the user first tears into the sketch world.