GeminiTeleportal

Result and evidence

• 90%+ of demo desktop tasks, unaidedControlled evaluationA demo task set, not general use.
• ~2-5 s from voice to actionControlled evaluationPhone audio and screen video over WebRTC to the Gemini Live API.

The case

Problem

Driving a Windows desktop from a phone usually means pinching at a small mirror of the screen. The aim was to say the task instead, and have an agent that can see the screen carry it out.

Decision

Stream phone audio and the desktop's screen over WebRTC to the Gemini Live API, and have a ReAct agent act on exact UI element ids from the Windows UI Automation tree rather than on pixel guesses.

Constraint

Pairing had to need no setup: the phone and the desktop find each other through the same Google account, with no QR codes or session URLs.

Architecture

1. Phone web appNext.js on Cloud Run
2. Desktop executorPython, WebRTC
3. ReAct agentUI Automation tree

Beside the path

1. Firestoresignalling, from Phone web app
2. Gemini Live APIvoice and screen, from Desktop executor

Around it

1. GitHub Actions + Cloud Builddeploys the web app

What was built

1. A ReAct agent that clicks exact UI element ids from the Windows UI Automation tree and from annotated screenshots, with perceptual-hash verification. It completed 90%+ of demo desktop tasks unaided.
2. Phone audio and screen video are streamed over WebRTC to the Gemini Live API. The agent acts on spoken commands within about 2-5 seconds. Pairing needs no configuration, using Google OAuth and Firestore signalling.
3. A GitHub Actions and Cloud Build pipeline that auto-deploys every frontend change to Cloud Run.