Sophie's Night Sky Adventure

The adaptation keeps the book's central experience, recognising a pattern in the night sky, and builds it around gesture instead of the book's story. Players raise an index finger as a pointer, hold it over a star until a ring fills, and connect the stars into constellation lines.

Pointing at stars is a familiar gesture, so I used it directly as the mechanic. Asking for a hint uses one other gesture, an open palm.

Six constellations are curated as a sequence: Ursa Major, Ursa Minor, Cassiopeia, Draco, Cepheus and Andromeda.

A round

Each constellation is a round against the clock, with more time for the larger ones: 55 seconds for Cassiopeia, 95 for Draco. The player selects a star, then the star it connects to. A correct line stays in the sky. A wrong one flashes and takes five seconds off the clock.

After three wrong connections, an open palm brings up the outline of the constellation and a tip, such as starting from Dubhe and Merak, the two pointer stars of Ursa Major.

Hand tracking runs in your browser using the camera; nothing is recorded or transmitted. A mouse fallback is built in.

The game is played with a hand held up to a camera. Each frame from the camera goes to a hand-tracking model, MediaPipe Hands, which runs in the browser. It finds up to two hands and returns 21 points for each.

The game reads only a few of those points. The tip of the index finger becomes the cursor, flipped left to right so that it moves the way a person expects when facing the screen. To tell an open palm from a pointing hand, the game checks whether each fingertip sits above its knuckle.

From a frame to a star

The raw fingertip shakes a little from frame to frame, even when the hand is held still. The cursor follows a smoothed copy, moving 15 per cent of the remaining distance each frame, so it stays put when the hand does. With too little smoothing nobody can hold still on a star; with too much the cursor no longer feels attached to the hand.

There is no click. Within reach of a star, a ring fills over 0.9 seconds and the star is selected when it closes. Moving away resets the ring, and a star cannot be selected again until the cursor has left it. The length of the dwell was the most sensitive setting in the project: too short and the game selects stars the player only glanced at, too long and holding still takes over the experience.

Try it with a mouse

After finishing the curated sequence, a player can draw their own constellation with the same stars, name it and leave it in the game.

The drawings are saved to a shared gallery that builds up across sessions and visitors, so the sky shows the shapes drawn by everyone who played before.

The current version uses a screen and a camera. The next step is an adaptation for a VR headset.