VideoArgusVideoArgus
Sample-specific rubric evaluation
University of Rochester
VideoArgus reads each input, writes weighted, checkable, hard-capped criteria before it ever sees a video, then gathers evidence with CV tools and scores each criterion with a judge VLM. One evaluator, five conditioning settings, and rankings that track human judgment more closely than the benchmarks' own metrics.
The pipeline factorizes into two independently swappable backbones: the LLM that writes the rubric, and the VLM that applies it. Neither is baked in.
A strong LLM (OpenAI / Anthropic / Gemini) reads the prompt and any conditioning image or video, then emits weighted, hard-capped criteria — before any output video exists.
Per criterion, an agent runs CV tools (SAM3, DINOv3, depth, OCR, flicker probes) and optional reference lookup to collect visual evidence from the generated video.
Your own local vLLM judge scores each criterion 0–10 against the evidence. Any OpenAI-compatible vision model works — no hosted-API keys on the eval path.
Importance-weighted mean with a soft hard-cap: final = base·(1−α) + min(base,cap)·α. All shipped reports use α=0.5.
Why sample-specific? A single global metric rewards the same thing on every prompt. A rubric written for this prompt can ask “does the mortarboard follow projectile physics?” on one input and “is the subject's identity preserved through the edit?” on the next, and cap the score when a hard requirement fails.
Each row links straight to that task's split in the HuggingFace dataset viewer, browse the prompts, conditioning media, and shipped rubrics in your browser.
| Task | Conditioning | Inputs | Browse cases |
|---|---|---|---|
| T2V | text | 202 | open in HF viewer |
| TI2V | text + first frame | 203 | open in HF viewer |
| TS2V | text + subject image(s) | 205 | open in HF viewer |
| TV2V | text + source video | 208 | open in HF viewer |
| TSV2V | text + subject image + source video | 208 | open in HF viewer |
huggingface-cli download zengziyun/VideoArgusBench \ --repo-type dataset --local-dir VideoArgusBench
Each task ranks the evaluated generators by their VideoArgus score. Pick a task to see its board. Within each task, closed API models are listed above open-source models. Best per task is bold, second is underlined; #P is parameter size.
Ranks are computed across all models in the task (API and open-source together); the API / open-source split is presentational.
A case is deliberately lightweight: a prompt, whatever conditioning the task provides, and the sample-specific rubric VideoArgus induced for it. You supply your model's video; the rubric does the rest.
Example rubric criteria (T2V, graduation prompt):
Each report shows the input, two competing generators side by side, the soft-capped final score, and the per-criterion breakdown with the CV tools each criterion used. Click any card to view full size.
If you find VideoArgus useful, please cite our paper.
@article{zeng2026videoargus,
title={VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing},
author={Zeng, Ziyun and Wang, Zixuan and Yu, Yongsheng and Hua, Hang and Luo, Jiebo},
journal={arXiv preprint arXiv:2608.05485},
year={2026}
}