open platform · agentic video intelligence
Turn long videos into a queryable knowledge base.
An open platform for agentic video intelligence applications.
VideoIndex indexes hours of video once, into speech, on-screen text, frames and scenes at a shared timebase, and gives your agents tools to search it, look at it and answer with the exact moment cited. Rust core, open source, and agnostic to the vision and language models you run it with.
The whole idea
Watch the demo 1:21
A narrated walkthrough: the SDK, one library-wide question over 36 hours of talks, and cost and latency against Gemini agentic video.
Built for agentic video applications
The demo is one application. The platform underneath is what you build on: a persistent index, agent tools over it, your choice of models, and every integration surface an agent framework expects.
-
Index once
A Rust pipeline decodes video, runs speech recognition, OCR, shot detection and frame embeddings, and stores everything with timestamps at one timebase. An optional fine pass adds scene descriptions and entity and event extraction. The index is a directory you keep.
-
Agents that cite
Search, view and describe tools let an agent read transcripts and on-screen text, look at frames when text does not settle it, and answer with hh:mm:ss citations. Retrieval-only and custom policies when you need cheaper or deterministic behaviour.
-
Any VLM, any LLM
Model roles, not model names: speech, OCR, embeddings, description, extraction and the agent are each bound to a provider you choose. OpenAI-compatible servers, Anthropic, Gemini or local ONNX models, swapped in one config file, priced and rate-limited per role.
-
Integrate anywhere
A library from Python or Node, an HTTP API with streaming answers, or an MCP server that Claude Code, LangGraph and other agent frameworks call as a tool. One binary, one index directory, no services to run for the local case.
Where it stands
Same questions, same videos, same scoring for every row. Stratified 25% samples of two long-video benchmarks, runs of 2026-09-17; per-question cost is the LVBench figure and excludes indexing for the VideoIndex rows.
| Configuration | LVBench (340 q) | MINERVA (310 q) | cost / q |
|---|---|---|---|
| VideoIndex agent (Gemini 3.8 Flash) | 77.1% | 66.8% | $0.03 |
| VideoIndex agent (Claude Sonnet 5) | 67.6% | 55.8% | $0.10 |
| Retrieval only, no agent | 50.3% | 37.1% | $0.03 |
| Gemini 3.8 Flash agentic video (reference) | 80.8% | 76.5% | $0.06 |
The gap to Gemini is in the visual question types; speech and on-screen text are covered well. The results pages carry the intervals, per-type tables and the reading of each run.
Where it is going
Real-time understanding of live video streams
The same index, built continuously from a live stream, so interactive agentic AI applications can ask about what is happening now and what happened a minute ago, with the same citations. Follow progress in the repository.
Demo
The reference app built on the platform: a chat over 36 hours of AI Engineer talks and Berkeley lectures. Sign in and click any timestamp.
videoindex.app →Documentation
Architecture, data model, indexing pipeline, agent tools, model providers, SDK, HTTP and MCP reference.
videoindex.github.io →Source
Rust core, Python and Node bindings, provider adapters, evaluation harness. Apache-2.0.
github.com/videoindex →Community
Questions, ideas, and what you are building with VideoIndex. The developers are on the Discord server.
discord.gg/eW7HVMwUC →