videoindex

open platform · agentic video intelligence

Turn long videos into a queryable knowledge base.

An open platform for agentic video intelligence applications.

VideoIndex indexes hours of video once, into speech, on-screen text, frames and scenes at a shared timebase, and gives your agents tools to search it, look at it and answer with the exact moment cited. Rust core, open source, and agnostic to the vision and language models you run it with.

22×real time indexing on one A100, speech, on-screen text and frame embeddings included
93%answer quality on library-wide questions, level with Gemini agentic video at 2.6× lower cost
$0.03per LVBench question with Gemini 3.8 Flash; indexing is paid once per video
[demo]

The whole idea

Demo video poster Watch the demo 1:21 A narrated walkthrough: the SDK, one library-wide question over 36 hours of talks, and cost and latency against Gemini agentic video.

1080p download

[platform]

Built for agentic video applications

The demo is one application. The platform underneath is what you build on: a persistent index, agent tools over it, your choice of models, and every integration surface an agent framework expects.

  1. Index once

    A Rust pipeline decodes video, runs speech recognition, OCR, shot detection and frame embeddings, and stores everything with timestamps at one timebase. An optional fine pass adds scene descriptions and entity and event extraction. The index is a directory you keep.

  2. Agents that cite

    Search, view and describe tools let an agent read transcripts and on-screen text, look at frames when text does not settle it, and answer with hh:mm:ss citations. Retrieval-only and custom policies when you need cheaper or deterministic behaviour.

  3. Any VLM, any LLM

    Model roles, not model names: speech, OCR, embeddings, description, extraction and the agent are each bound to a provider you choose. OpenAI-compatible servers, Anthropic, Gemini or local ONNX models, swapped in one config file, priced and rate-limited per role.

  4. Integrate anywhere

    A library from Python or Node, an HTTP API with streaming answers, or an MCP server that Claude Code, LangGraph and other agent frameworks call as a tool. One binary, one index directory, no services to run for the local case.

[results]

Where it stands

Same questions, same videos, same scoring for every row. Stratified 25% samples of two long-video benchmarks, runs of 2026-09-17; per-question cost is the LVBench figure and excludes indexing for the VideoIndex rows.

ConfigurationLVBench (340 q)MINERVA (310 q)cost / q
VideoIndex agent (Gemini 3.8 Flash)77.1%66.8%$0.03
VideoIndex agent (Claude Sonnet 5)67.6%55.8%$0.10
Retrieval only, no agent50.3%37.1%$0.03
Gemini 3.8 Flash agentic video (reference)80.8%76.5%$0.06
VideoIndex against Gemini agentic video, 28 library-wide questions over 30 talks Results over our eval dataset 28 questions over 30 talks (36.6 hours): which videos mention X, the moments about a topic, cross-video comparisons, library summaries. Same questions, same answer format, same judge for both systems. Cost per question lower is better VideoIndex $0.16 2.6× cheaper Gemini agentic $0.41 Median latency lower is better · p95 58 s against 188 s VideoIndex 28 s 1.6× faster Gemini agentic 45 s Videos found (F1) higher is better VideoIndex 98% Gemini agentic 97% Answer quality higher is better · judge-scored against ground truth VideoIndex 93% Gemini agentic 93% VideoIndex cost excludes indexing, paid once per video. Gemini agentic video takes ten videos per request, so each question is three parallel agentic requests plus a merge, and its cost is the whole cost. Provider list prices, September 2026. VideoIndex against Gemini agentic video, 28 library-wide questions over 30 talks Results over our eval dataset 28 questions over 30 talks (36.6 hours): which videos mention X, the moments about a topic, cross-video comparisons, library summaries. Same questions, same answer format, same judge for both systems. Cost per question lower is better VideoIndex $0.16 2.6× cheaper Gemini agentic $0.41 Median latency lower is better · p95 58 s against 188 s VideoIndex 28 s 1.6× faster Gemini agentic 45 s Videos found (F1) higher is better VideoIndex 98% Gemini agentic 97% Answer quality higher is better · judge-scored against ground truth VideoIndex 93% Gemini agentic 93% VideoIndex cost excludes indexing, paid once per video. Gemini agentic video takes ten videos per request, so each question is three parallel agentic requests plus a merge, and its cost is the whole cost. Provider list prices, September 2026.
Questions that need the whole library, 2026-09-17: 28 questions over the 30-talk demo library. VideoIndex matches Gemini's agentic mode on answer quality (93% each) and finds every expected video (recall 100% against 97%), at 2.6× lower cost and 1.6× lower median latency (under a third at the 95th percentile), because the index answers a library-wide scan with one query where Gemini reads most of the library on every question. The results page has the per-question table, the judge protocol and the run files.

The gap to Gemini is in the visual question types; speech and on-screen text are covered well. The results pages carry the intervals, per-type tables and the reading of each run.

[roadmap]

Where it is going

Real-time understanding of live video streams

The same index, built continuously from a live stream, so interactive agentic AI applications can ask about what is happening now and what happened a minute ago, with the same citations. Follow progress in the repository.