Gesture control

Gesture control lets you pan, zoom, tilt and hover on the map with your hands in front of the webcam. Mantis tracks your hands with MediaPipe, shows a small camera preview with the detected skeleton drawn on it, and maps a handful of poses onto map actions. It is off by default and has to be switched on per session.

Shortcut

Mod+Shift+H

Available on

Map (Legacy)

Requirements

• A secure context. The page has to be served over HTTPS or run on localhost. Otherwise gesture mode reports that it needs HTTPS or localhost camera access and refuses to start.
• Camera permission. The browser prompts on first use; the preview stays empty until you allow it.
• WebAssembly support, which every current desktop browser has.
• Network access to the model assets. The MediaPipe runtime is fetched from the jsDelivr CDN and the hand and face landmark models from Google’s mediapipe-models storage. On a locked-down network the overlay will sit at “Loading MediaPipe models”.
• A map to control. Gesture mode targets one map panel. Until one is chosen the status reads “Select a map to control”.

Turning it on

1. Open a space with a Map (Legacy) panel and click inside the map you want to drive.
2. Toggle gesture mode. Press Ctrl+⇧+H, or click the hand icon in the map’s view settings controls.
3. Allow the camera when the browser asks.
4. Wait for the preview. A camera panel appears in the map with a colored status dot; the same shortcut or icon turns it back off.

Arming: passive, arming, armed

Gestures do not act on the map the moment a hand appears. The overlay carries an intent gate with three states, shown at the bottom of the preview:

• PASSIVE. Hands are tracked and drawn, but nothing moves the map.
• ARMING… n%. You are holding a hand in frame in a neutral pose, that is, in frame but not performing any of the action poses below. Roughly a quarter of a second of that arms the gate, and less if the gaze tracker has you steadily looking at the screen.
• ARMED. Gestures drive the map. After about three seconds with no recognized pose, the gate falls back to passive.

The gesture vocabulary

Pose	Effect
All five fingers open, hand moving	Pan the map
Five fingers bunched, then spread or bunch	Zoom
Two open hands, spread apart or pushed together	Zoom
Thumb and index pinched and held, then moved	Tilt (up and down) and rotate (left and right), 2.5D only
Index finger pointing	Move the hover cursor; hold steady on a point to select it

There are no mode buttons. You do not switch between zooming and panning; the pose you make decides which action runs, and the arbiter grants control to one of them at a time.

Pointing at a point starts a dwell timer of about two thirds of a second, drawn as a progress bar under the point’s title in the preview. The card reads Hovering while the bar fills and Selected once the point is committed.

Reading the overlay

The status line under the preview tells you what the system thinks is happening:

• Loading MediaPipe models and Requesting camera access during startup.
• Show your hand(s) with thumb and index visible when nothing is detected.
• Flat hand = pan, 5-finger pinch = zoom, 2 hands = zoom when it is ready and idle, replaced by pose-specific text such as Both hands open. Spread or push to zoom or Grab held. Move up/down to tilt, left/right to rotate (2.5D only) while a gesture is running.
• Hand ready. Face tracking unavailable for gaze debug. when hands work but the face signal does not.
• Paused while this tab is hidden when you switch tabs.
• Gesture mode unavailable in this browser or context when a requirement above is not met.

The hand skeleton is drawn in blue when idle, orange while panning, green while zooming. The i button opens a readout of the current clutch, gaze state, pointer position and calibration state, and reminds you of the shortcut to exit. A gaze badge on the right shows CALIBRATING, then GAZE, FIXATION or FOCUS as the eye tracker settles.

Limits

• Gesture control runs on Map (Legacy) only. The default WebGL Map has no camera overlay.
• Tilt and rotate need the map in 2.5D; in 2D the grab pose does nothing.
• Everything runs locally in the browser tab. Turning gesture mode off releases the camera.

Read nextMapThe panel gesture control drives.