ZhuoQi Devzhuoqidev.comRSSllms.txt

Project · Argus

How Argus watches a long video

Scroll to follow one search step by step; drag the scene to turn it.

Skip to the write-up ↓
A small clay scene: a film strip (the sample video) lies in a browser window; Argus, covered in eyes, stands beside it, with the frames it pulled standing above the strip; over both edges of the red span hang three rows of ever denser small frames; a lavender sub-agent watches the second half; a tray next to Argus holds the last three batches of frames and a slip of text, with a sticky note beside it; an answer card reading 10.0s – 25.0s is tied by red threads to two frames on the strip; a wire runs from Argus to a cloud outside the window that stands for the model you configured, carrying small frames and an instruction card.

  1. One video, one question

    Take the 3-minute sample video that ships with Argus: drop it in and ask, “When does something red show up?” It reads the video's details first: 179.6 seconds, 10 frames a second.

    get_video_info()

  2. A skim, handed to the model

    Argus is an agent covered in eyes. It pulls one frame every 10 seconds, shrinks them and sends them up this wire to the model you configured; the model looks and replies with an instruction: which span to pull next, and how densely. This round, red shows up in the frames at 10 s and 20 s.

    extract_frames(0–180 s, every 10 s) → 18 frames

  3. Closing in on both edges

    Red appears somewhere in 0–10 s and is gone somewhere in 20–30 s. Around both, it pulls a frame every second, then every 0.5 s, then every 0.1 s, until each edge sits on a single frame.

    extract_frames(0–12 s, 18–32 s, every 1 s) → every 0.5 s → every 0.1 s

  4. A helper, and a tidy context

    Is there more red after 30 s? It sends a sub-agent to watch 30–180 s. The sub-agent pulls 64 frames of its own and hands back a short written finding (no red); its frames never enter the main agent's context. The main agent keeps only its last 3 batches of frames too: when the fourth came in, the first batch (the skim) became a line of text, and those frames can be pulled again when needed.

    spawn_subagent(30–180 s) → “no red”

  5. A note, then an answer with evidence

    It writes the finding down as a note, so a follow-up question needs no second look, then answers: red appears at 10.0 s and is gone at 25.0 s. Both times are pinned to frames it pulled (10.0 s is the first frame with red, 24.9 s the last); click a timestamp in Argus and the player jumps there.

    remember(0–180 s, “red object”)

  6. All inside your browser

    The video file never leaves your browser, and there is no backend. What went up this wire in this run was 129 shrunken frames (300–640 px wide: 65 for the main agent, 64 for the sub-agent) plus text, to the model you configured yourself (OpenAI, Anthropic, Gemini or a compatible API); the API key stays on your machine.

    The agent loop and the 8 tools behind these steps ↓

The scene follows one real run with DeepSeek on 2026-10-08 (full call record); every run goes a little differently.An agent built the model in three.js code; the poster is path traced in the browser too. The code

~/projects/argus.md

Argus — A Browser-Local Long-Video Understanding Agent Harness

Argus (named after Argus Panoptes, the hundred-eyed giant of Greek myth) is an open-source, long-video understanding agent harness I designed and built solo — frontend-only, no backend. Videos are decoded entirely in the browser, and users bring their own API keys and models. A multimodal LLM agent calls tools to “watch” the video itself: extracting frames, zooming into regions, taking notes, and spawning sub-agents. Typical scenarios: counting people in surveillance footage, or finding a specific object.

🔗 Live Demo 🐙 GitHub

The Problem

The hard part of long-video understanding isn’t the model — it’s how you feed it the video. Uploading everything is prohibitively expensive; blind frame sampling misses the signal. Argus hands that decision to the agent: it inspects video metadata first, then decides which frames to extract, at what rate, and where to zoom in — and when the context is about to explode, it spawns sub-agents to handle segments. Meanwhile the video file never leaves your browser and there is no backend: all the model receives is the downscaled frames the agent picked, plus text.

Architecture Highlights

  • Agent Loop — Vercel AI SDK streamText runs the multi-step tool loop (capped with stopWhen); the two parts that make the product are ours: prepareStep injects freshly extracted frames as a user message so any vision model can see them (images inside tool results don’t carry across OpenAI-compatible gateways), and old images are pruned (only the last 3 batches stay; earlier ones become a line of text saying how to pull them back) so long runs keep their context bounded.
  • 8 Video-Understanding Tools — get_video_info / extract_frames (range + rate sampling with timestamps) / extract_frame_at / list_frames / inspect_region (zoom into a region) / remember / recall (state memory against long-context drift) / spawn_subagent (segment very long videos, preventing context blow-up).
  • Browser-Native Hardware-Decoded Frame Extraction — the main path is mediabunny (WebCodecs demux + decode, no <video> seeking); when it can’t decode a file, Argus falls back to <video> + <canvas> (with black-frame retries); containers like AVI / WMV / FLV are transcoded locally with ffmpeg.wasm, fetched from a CDN only the first time one shows up; mediainfo.js only fills in fps / codec / bitrate on the fallback path.
  • 211 Providers via a Runtime Catalog — integrates the models.dev catalog, normalized into provider presets with a searchable dropdown; vision models are auto-flagged and selected by default; OpenAI-compatible / Anthropic / Gemini transports are detected automatically, with CORS hints and a connection test.
  • Sub-Agent Context Isolation — a sub-agent is a nested agent run without the spawn_subagent tool, so the intermediate work of segment-level understanding never pollutes the main context.

Tech Stack

Frontend · Vite 8 + React 19 + TypeScript 7 (native tsc) + Tailwind 4 + zustand

LLM Access · Vercel AI SDK (ai + @ai-sdk/openai / @ai-sdk/anthropic / @ai-sdk/google, all with custom baseURL support) · models.dev provider catalog

Infra · GitHub Actions CI/CD · dual-track publishing (Alibaba Cloud CDN + OSS for China, Cloudflare Pages overseas) · Let’s Encrypt DNS-01 certificate auto-renewal

Engineering Practices

  • Zero Backend — a pure static site; the build artifact is 7 files published to both tracks. User API keys live only in the browser.
  • Same-Artifact Dual Publishing — China (Alibaba CDN → OSS) and overseas (Cloudflare Pages) are served from a single dist/ build, with DNS split-horizon routing guaranteeing identical content on both tracks.
  • Certificate Automation — a weekly job checks the China CDN certificate and renews via DNS-01 when under 30 days remain, with edge verification.

My Role

Solo build: product design · agent harness architecture · frontend implementation · multi-provider integration · CI/CD dual-track deployment — from zero to open-source release, end to end.

Open-source project — see GitHub.

Say hi on WeChat

Liu ZhuoQi's WeChat QR code

Scan it in WeChat, or long-press it on your phone

Mention why you're reaching out, or I might miss it.