视频理解Benchmark中的字幕评测：
从VideoMME到VideoMME-v2
聚焦 VideoMME / VideoMME-v2 与主流评测框架，拆清字幕如何被切分、对齐、注入 prompt，并最终影响视频理解模型的推理难度与得分。
Subtitle Evaluation in Video Understanding Benchmarks:
From VideoMME to VideoMME-v2
A concrete walkthrough of how VideoMME / VideoMME-v2 and mainstream evaluation stacks parse, align, and inject subtitles into prompts—and why that changes what video understanding scores actually measure.
1引言 — 为什么需要字幕评测
在多模态大型语言模型（MLLMs）如 GPT-4o、Gemini 1.5 Pro、Qwen-VL 和 InternVL 的高速迭代中，视频理解正逐渐从单纯的“看”（视觉帧序列处理）向更深层次的“听”和“读”演进。视频本身是高度多模态的媒介，而字幕（Subtitle）作为视频中文本信息的直接载体，不仅精确包含了对话内容，更是串联视频剧情、理解人物意图、甚至进行多跳逻辑推理的关键维度。
不同模型家族对字幕的处理能力差异巨大。例如，GPT-4o 展现了强大的原生视听融合能力，而多数开源模型目前仍依赖外部抽取的文本字幕进行 Prompt 拼接。这种差异直接催生了对字幕评测基准的迫切需求。
1. 盘点当前主流 Benchmark 如何设计字幕测试（VideoMME, MMOU 等）。
2. 深入解析最新一代 VideoMME-v2 的渐进式复杂推理机制。
3. 从底层代码（lmms_eval, VLMEvalKit）剖析框架是如何将字幕注入到视觉模型中的。
4. 探讨 Qwen3-VL 等前沿模型在数据集层面的字幕处理范式。
1Introduction — Why Subtitle Evaluation Matters
In the rapid iteration of Multimodal Large Language Models (MLLMs) such as GPT-4o, Gemini 1.5 Pro, Qwen-VL, and InternVL, video understanding is evolving from merely "watching" (visual frame processing) to deeper levels of "listening" and "reading". Video is inherently a highly multimodal medium, and subtitles, as the direct carrier of textual information, not only contain exact dialogues but also serve as a crucial dimension for connecting plots, understanding human intentions, and performing multi-hop logical reasoning.
Different model families exhibit vast differences in handling subtitles. For instance, GPT-4o demonstrates strong native audio-visual fusion, while most open-source models currently rely on externally extracted text subtitles concatenated via Prompts. This divergence has directly birthed an urgent need for subtitle evaluation benchmarks.
1. Review how mainstream Benchmarks design subtitle tests (VideoMME, MMOU, etc.).
2. Deep dive into the progressive complex reasoning mechanisms of the next-gen VideoMME-v2.
3. Analyze from the code level (lmms_eval, VLMEvalKit) how frameworks inject subtitles into vision models.
4. Discuss the dataset-level subtitle processing paradigms of cutting-edge models like Qwen3-VL.
2Benchmark字幕支持总览
不同的 Benchmark 对视频理解的侧重点不同，这也直接体现在它们对字幕的支持和处理方式上。从早期的纯视觉评测，到加入 SRT 文本，再到如今的词级别时间戳与原生音频理解，基准测试的难度在不断攀升。
2Overview of Benchmark Subtitle Support
Different benchmarks emphasize varying aspects of video understanding, which is directly reflected in their subtitle support. From early pure-visual evaluations to the inclusion of SRT texts, and now word-level timestamps and native audio understanding, the difficulty of benchmarks is constantly escalating.
趋势分析：从上表可以看出，评测体系正在经历从“文本附加物”向“原生模态”的转变。早期的评测集要么忽略字幕（如 MLVU），要么简单地将 SRT 作为文本补充（VideoMME）。而新一代评测集则开始追求极高的时间精度（VideoMME-v2 的词级时间戳）或完全抛弃文本转向真正的多模态端到端音频理解（MMOU）。
Trend Analysis: As seen in the table, the evaluation paradigm is shifting from treating subtitles as a "textual appendage" to treating them as a "native modality". Early benchmarks either ignored subtitles (MLVU) or simply used SRT as a text supplement (VideoMME). New generation benchmarks pursue extreme temporal precision (word-level timestamps in VideoMME-v2) or completely abandon text for true multimodal end-to-end audio understanding (MMOU).
3VideoMME — 首个系统性字幕评测
作为视频理解领域的标杆之一，VideoMME 开创性地将字幕作为评测的一个独立控制变量。它构建了一个包含 900 个视频 的庞大数据集，广泛覆盖了 6 大领域（知识、电影、体育、艺术、生活、科学）。
在时长分布上，数据集被精心划分为三个层级：短视频（< 2分钟）、中等视频（4-15分钟）和长视频（30-60分钟）。为了保证评测质量，VideoMME 提供了 744 个标准 SRT 格式字幕文件。特别是对于长视频，考虑到自动生成字幕（ASR）可能存在的幻觉和漂移，团队几乎全部采用了高质量的人工校对字幕。
3VideoMME — First Systematic Evaluation
As a flagship in video understanding, VideoMME pioneered the use of subtitles as an independent control variable for evaluation. It constructed a massive dataset of 900 videos, covering 6 major domains (Knowledge, Film, Sports, Artistic, Life, Science).
In terms of duration, the dataset is meticulously divided into three tiers: short (< 2min), medium (4-15min), and long (30-60min) videos. To ensure evaluation quality, VideoMME provides 744 standard SRT formatted subtitle files. Notably, for long videos, considering the hallucinations and drift of auto-generated subtitles (ASR), the team almost exclusively utilized high-quality human-proofread subtitles.
VideoMME 采用了严格的平行测试模式。在启用字幕时，使用的 Prompt 模板极为直接：
VideoMME employs a strict parallel testing mode. When subtitles are enabled, the Prompt template used is extremely direct:
核心发现：研究结果明确表明，对于对话密集型视频（如访谈、电影片段），引入高质量的人工字幕能使主流开源 MLLMs 的准确率平均提升 12% - 15%。然而，当使用低质量自动生成的字幕时，模型的表现甚至可能产生倒退。这揭示了大模型强烈的“文本依赖偏见”——当视觉信号与文本信号冲突时，模型往往盲信文本。
Key Finding: Research results clearly show that for dialogue-dense videos (e.g., interviews, movie clips), introducing high-quality human subtitles boosts the accuracy of mainstream open-source MLLMs by 12% - 15% on average. However, when using low-quality auto-generated subtitles, model performance can actually degrade. This reveals a strong "text dependency bias" in LLMs—when visual signals conflict with text signals, models tend to blindly trust the text.
4VideoMME-v2 — 下一代字幕评测 (2604.05015)
2026年4月，VideoMME 团队推出了重磅升级的 VideoMME-v2。这一版本彻底重构了评测体系，包含 800 个精选视频和 800 个精准字幕文件，构建了 3200 个精标注问答对，耗费了惊人的 3300 个人类标注工时。它带来了三大革命性创新：
A. 渐进式多层级系统 (Progressive Multi-Level)
VideoMME-v2 摒弃了扁平的提问方式，将视频理解拆解为三个严格递进的认知层级。只有当模型在底层理解无误时，才能真正证明其具备高层推理能力，从而严厉打击了模型的“猜题”行为：
- L1: Information Aggregation (信息聚合) - 考察模型对字幕基本事实、视频画面实体的基础识别与提取。
- L2: Temporal Understanding (时序理解) - 考察事件发生的先后顺序、因果关系，以及字幕台词与特定画面的时序对齐。
- L3: Temporal Complex Reasoning (时空复杂推理) - 需要跨越长视频的多个片段，结合前后文语境进行多跳逻辑推理。
B. 分组非线性评分机制 (Grouped Non-Linear Scoring)
为了进一步消除随机猜测带来的虚高分数，v2 版本将传统选择题的选项数从 4 个激增至 8 个（选项 A-H）。更绝的是其独特的评分公式：
4VideoMME-v2 — Next-Gen Eval (2604.05015)
In April 2026, the VideoMME team launched the massive upgrade: VideoMME-v2. This version completely restructures the evaluation system, containing 800 curated videos and 800 precise subtitle files, formulating 3,200 meticulously annotated QA pairs at the cost of a staggering 3,300 human-hours. It introduces three revolutionary innovations:
A. Progressive Multi-Level System
VideoMME-v2 abandons flat questioning, dissecting video understanding into three strictly progressive cognitive levels. Only when a model succeeds at the base level can it prove high-level reasoning, heavily penalizing "guessing":
- L1: Information Aggregation - Tests basic recognition and extraction of facts from subtitles and visual entities.
- L2: Temporal Understanding - Examines chronological order, causality, and temporal alignment between spoken lines and specific frames.
- L3: Temporal Complex Reasoning - Requires multi-hop logical inference across multiple segments of a long video, combining context.
B. Grouped Non-Linear Scoring
To further eliminate inflated scores from random guessing, v2 expands the number of multiple-choice options from 4 to 8 (A-H). More impressively, it introduces a unique scoring formula:
在这个机制下，同一个视频的多个相关问题被打包成一个“Group”。系统实行首错截断 (First-error truncation)：如果模型在 L1 级别的问题上回答错误，那么该组后续的 L2、L3 问题无论是否猜对，都记为错误。这种指数级的惩罚曲线 ($x^2$) 迫使模型必须展现出高度的内部一致性。
C. 词级别时间戳与双重注入模式
格式上，v2 彻底抛弃了粗粒度的 SRT，转而采用词级别（Word-level）时间戳的 JSONL 格式。在评测框架（如 VLMEvalKit）中，它提供了多种 Config 变体（如 Video-MME-v2_64frame_subs），并定义了两种字幕与视频帧的融合注入模式：
Under this mechanism, multiple related questions for a single video are bundled into a "Group". The system enforces First-error truncation: if a model fails an L1 question, subsequent L2/L3 questions in that group are marked wrong regardless of the guess. This exponential penalty curve ($x^2$) forces models to exhibit high internal consistency.
C. Word-level Timestamps & Dual Injection Modes
Format-wise, v2 abandons coarse SRT for Word-level timestamped JSONL. In evaluation frameworks (e.g., VLMEvalKit), it provides various Config variants (like Video-MME-v2_64frame_subs) and defines two distinct modes for fusing subtitles with video frames:
5MMOU — Omni-Modal的隐式路径 (2603.14145)
如果说 VideoMME 致力于将文本字幕做到极致，那么由 NVIDIA 发布的 MMOU 则代表了迈向 AGI 的另一种截然不同的路径：抛弃文本，拥抱原生音频。
MMOU 包含 9038 个视频和 15000 个问题，横跨 6 大能力维度。其最核心的特点是拒绝提供任何形式的字幕文件。模型必须像人类一样，直接处理视频帧和伴随的 Raw Audio 流。这意味着模型必须在内部同时完成语音识别（ASR）和多模态联合语义推理。
这种隐式理解范式带来了全新的提问结构。常见的测试题如：
"When the speaker says [specific audio segment], what object is shown on the right side of the screen?"
这里，声音信号成为了视觉帧序列的“时间锚点”。这不仅测试了模型是否“听懂了”，更测试了模型是否能在毫秒级别上将声音流与像素流进行对齐对位。
5MMOU — The Implicit Omni-Modal Path (2603.14145)
If VideoMME strives to perfect text subtitles, MMOU released by NVIDIA represents an entirely different path towards AGI: abandoning text to embrace raw audio.
MMOU encompasses 9,038 videos and 15,000 questions across 6 capability dimensions. Its defining feature is the refusal to provide any subtitle files. The model must process video frames and the accompanying Raw Audio stream directly, like a human. This means the model must internally perform simultaneous Speech Recognition (ASR) and multimodal joint semantic reasoning.
This implicit understanding paradigm introduces novel question structures. Common test formats include:
"When the speaker says [specific audio segment], what object is shown on the right side of the screen?"
Here, the audio signal becomes the "temporal anchor" for the visual frame sequence. This tests not just whether the model "heard it", but whether it can align the audio stream with the pixel stream at millisecond precision.
6技术解密：lmms_eval 的字幕管线
在工程层面，评测框架是如何将孤立的字幕文件与视频帧耦合在一起的？在开源评测集大成的 lmms_eval 中，这一切由精确的时间戳计算和 Prompt 模板注入完成。
通过对比其 YAML 配置文件（videomme.yaml 与 videomme_w_subtitle.yaml），我们可以清晰地看到字幕流的开关。当开启字幕模式时，系统会调用手写的正则解析器读取 SRT，并根据模型指定的 frame_num 参数（采样帧数）使用 np.linspace 进行帧对齐。
6Tech Decoded: Subtitle Pipeline in lmms_eval
At the engineering level, how do evaluation frameworks couple isolated subtitle files with video frames? In the widely used lmms_eval suite, this is accomplished via precise timestamp calculation and Prompt template injection.
By comparing its YAML configurations (videomme.yaml vs videomme_w_subtitle.yaml), we can clearly see the subtitle stream switch. When enabled, the system uses a hand-written regex parser to read the SRT and applies np.linspace for frame alignment based on the model's frame_num parameter.
实况演示：从 SRT 到 Prompt 的数据流管线
Live Demo: Data Pipeline from SRT to Prompt
✓ [1.0, 3.5]
✓ [1.0, 3.5]
✓ [8.5, 12.0]
- Today we discuss video understanding.
- Let's look at how models process subtitles.
7VLMEvalKit vs lmms_eval 架构深度对比
与 lmms_eval 的纯手工解析不同，OpenCompass 团队主导的 VLMEvalKit 采用了更工程化的设计模式。两者在字幕解析、时间轴对齐、配置管理和异常处理上存在根本性的架构分歧。接下来我们从代码级别逐层拆解。
7VLMEvalKit vs lmms_eval: Deep Architecture Comparison
Unlike the manual parsing in lmms_eval, the VLMEvalKit led by the OpenCompass team adopts a fundamentally different engineering approach. The two frameworks diverge at every layer — from subtitle parsing to timeline alignment, configuration management, and error handling. Let's dissect them at code level.
A. 技术路线全景对比
A. Full Technical Comparison
B. 字幕解析器：手工正则 vs pysubs2
这是两个框架最根本的分歧。lmms_eval 使用手写正则逐行切割 SRT 文件，手动将时间戳字符串 HH:MM:SS,mmm 转换为秒数；而 VLMEvalKit 直接调用 pysubs2 的高级 API，一行完成全部解析。
B. Subtitle Parser: Hand-written Regex vs pysubs2
This is the most fundamental divergence. lmms_eval splits SRT files line-by-line with hand-crafted regex, manually converting HH:MM:SS,mmm timestamp strings into seconds; VLMEvalKit calls the pysubs2 high-level API to parse everything in one line.
# load_subtitles() in lmms_eval/utils.py
def load_subtitles(srt_path):
with open(srt_path, 'r') as f:
content = f.read()
# Hand-crafted regex to split SRT blocks
pattern = r'(\d+)\n(\d{2}:\d{2}:\d{2},\d{3})'
blocks = re.findall(pattern, content)
# Manual time conversion
def to_seconds(ts):
h, m, rest = ts.split(':')
s, ms = rest.split(',')
return (int(h)*3600
+ int(m)*60
+ int(s)
+ int(ms)/1000)# video_dataset_config.py
import pysubs2
# Entire parsing: one line
subs = pysubs2.load(srt_path)
# Handles SRT, ASS, SSA, VTT auto
# Auto-strips HTML tags
# Auto-detects encoding
# Time alignment via absolute ms
frame_time = pysubs2.make_time(
fps=fps, frames=frame_idx)
# Match subtitle to frame
matched = [s.text for s in subs
if s.start <= frame_time
< s.end]C. 配置管理哲学
两个框架在“如何控制字幕开关”上也走了截然不同的路线。lmms_eval 采用声明式 YAML：为每个评测变体创建独立的 YAML 文件；VLMEvalKit 则采用编程式 Python：用 functools.partial() 在代码中动态绑定参数。
C. Configuration Philosophy
The two frameworks also differ in how they toggle subtitle mode. lmms_eval uses declarative YAML: creating standalone YAML files for each eval variant; VLMEvalKit uses programmatic Python: dynamically binding parameters via functools.partial().
# videomme_w_subtitle.yaml
include: videomme.yaml
metadata:
load_subtitle: true
subtitle_dir: /data/videomme/srt/
# Separate file per variant
# videomme.yaml → no subs
# videomme_w_subtitle.yaml → subs# Programmatic config
from functools import partial
DATASETS = {
'Video-MME': partial(
VideoMME,
use_subtitle=False,
nframe=128
),
'Video-MME_sub': partial(
VideoMME,
use_subtitle=True,
nframe=128
),
}D. 完整数据流对比动画
下面的动画展示了同一份 SRT 文件在两个框架中的完整处理流程。左侧是 lmms_eval 的手工流水线，右侧是 VLMEvalKit 的 pysubs2 流水线：
D. Full Data Flow Comparison Animation
The animation below shows how the same SRT file is processed through both frameworks. Left: lmms_eval manual pipeline. Right: VLMEvalKit pysubs2 pipeline:
00:00:01,000 --> 00:00:03,500
Hello everyone, welcome.
→ manual string split
+ int(s) + int(ms)/1000
→ {(1.0, 3.5): "Hello..."}
→ loop & window match
+ matched subtitle text
00:00:01,000 --> 00:00:03,500
<font>Hello everyone</font>
→ auto format detect
→ HTML tags stripped ✓
fps=30, frames=idx)
→ absolute ms timestamps
if s.start <= t < s.end]
+ clean subtitle text
E. 脏数据处理实战
真实世界的 SRT 文件经常包含 HTML 样式标签。以下是同一段脏字幕在两个框架中的不同命运：
E. Dirty Data Handling in Practice
Real-world SRT files frequently contain HTML styling tags. Here's how the same dirty subtitle is handled differently:
<font color="#ffffff"><i>Hello everyone, welcome to the show.</i></font>
❌ HTML 标签残留，污染 Prompt，可能误导模型❌ HTML tags leak into prompt, may confuse model
✅ 干净文本，pysubs2 自动剥离所有标签✅ Clean text, pysubs2 auto-strips all tags
• lmms_eval 的优势在于透明性——所有逻辑一目了然，适合快速定制和学术研究。但其手工 Regex 在面对脏数据、非 SRT 格式、BOM 编码时较脆弱。
• VLMEvalKit 的优势在于工程健壮性——pysubs2 天然支持多格式、多编码、HTML 标签清洗。
partial() 配置模式也更 Pythonic，适合大规模评测。
• 两者在功能上互补：lmms_eval 的 YAML 声明式配置更利于可复现研究，VLMEvalKit 的编程式配置更利于生产环境部署。
• lmms_eval excels at transparency—all logic is visible, ideal for quick customization and academic research. But its hand-crafted Regex breaks on dirty data, non-SRT formats, and BOM encodings.
• VLMEvalKit excels at engineering robustness—pysubs2 natively supports multi-format, multi-encoding, HTML tag stripping. The
partial() config pattern is more Pythonic for large-scale evaluation.
• The two are complementary: lmms_eval's declarative YAML is better for reproducible research; VLMEvalKit's programmatic config is better for production deployment.
8深度案例：Qwen3-VL 的极简哲学
当我们审视当今最强的开源视听模型 Qwen3-VL 时，会发现一个令人惊讶的现象：在模型网络结构层面，Qwen3-VL 并没有为“字幕”设计任何特殊的模块。
为什么不需要架构改动？因为 Qwen3-VL 拥有极其强悍的长上下文 Tokenizer，能够原生无缝处理交织的文本和图像 Token。因此，字幕的处理完全被抽象成了一个数据集构造（Dataset）的问题。
在评测运行中，Qwen3-VL 会将采样帧数推向极限：frame_num=768（而常规模型通常仅为 8 到 32 帧，VLMEvalKit 默认也是保守的 128 帧）。极高密度的视频帧与成百上千字的字幕文本被简单粗暴地拼接在一起，输入模型。这种“大力出奇迹”的极简哲学，恰恰证明了强大的基础模型不需要花哨的插件工程。
8Deep Case: The Minimalist Philosophy of Qwen3-VL
When examining today's strongest open-source audio-visual model, Qwen3-VL, a surprising phenomenon emerges: at the model network architecture level, Qwen3-VL incorporates absolutely zero special modules for "subtitles".
Why are no architectural changes needed? Because Qwen3-VL possesses an overwhelmingly powerful long-context Tokenizer capable of natively processing seamlessly interleaved text and image Tokens. Therefore, subtitle processing is entirely abstracted as a Dataset construction problem.
During evaluation, Qwen3-VL pushes the sampled frames to the extreme: frame_num=768 (whereas conventional models typically sample 8 to 32 frames, and VLMEvalKit defaults conservatively to 128). An incredibly high density of video frames combined with thousands of words of subtitle text are simply and brutally concatenated into the model. This minimalist "brute force" philosophy exactly proves that powerful base models do not require flashy engineering plugins.
9Allen AI 的 Molmo2：训练阶段融入字幕
与仅在评测阶段拼接字幕的方法不同，Allen AI 的 Molmo2 采用了一种全新的范式：将字幕处理视为训练层面的核心任务，而非仅仅是评测层面的工程手段。通过在训练阶段引入特定的字幕问答数据集，Molmo2 能够原生理解字幕与视频画面之间的关联，让模型在学习期间就“看过”并“读过”字幕。
9Allen AI's Molmo2: Integrating Subtitles at Training Time
Unlike approaches that merely concatenate subtitles during evaluation, Allen AI's Molmo2 adopts a completely new paradigm: treating subtitle processing as a core training concern, rather than just an engineering mechanism for evaluation. By introducing specific subtitle QA datasets during the training phase, Molmo2 natively understands the correlation between subtitles and video frames, enabling the model to have "seen" and "read" subtitles during its learning process.
与 lmms_eval 不同，Molmo2 使用自定义的 subtitles.json 格式（从 HuggingFace 仓库 allenai/Molmo2-VideoMME 下载），将字幕以 {(start, end): text} 字典形式附加到每个样本，并通过 style 字段区分有/无字幕模式。
Unlike lmms_eval, Molmo2 uses a custom subtitles.json format (downloaded from HuggingFace repo allenai/Molmo2-VideoMME), attaching subtitles as {(start, end): text} dictionaries to each example, and distinguishing subtitle/no-subtitle modes via the style field.
Molmo2 将评测任务明确分为“有字幕”和“无字幕”两大组，支持通过 --tasks=video_subtitle 或 --tasks=video_no_subtitle 命令行参数选择性运行，实现了字幕测试的清晰隔离。
Molmo2 explicitly divides evaluation tasks into "with subtitle" and "no subtitle" groups, supporting selective execution via --tasks=video_subtitle or --tasks=video_no_subtitle CLI arguments, achieving clean isolation of subtitle testing.
10总结：走向 AGI 的分岔路口
回顾从 VideoMME 到 VideoMME-v2，再到 MMOU 的演进，我们清晰地看到视频评测中针对声音与文本的探索，已经分化出三种主流范式：
- 显式文本流（Explicit Text）：如 VideoMME，将外部获取的 SRT/JSONL 作为坚实的提示词后盾，适合需要极高精度知识提取的场景。
- 纯视觉流（Pure Visual）：如 MLVU，刻意剥离声音和字幕，强迫模型提升对长视频纯视觉情节和微小动作的洞察力。
- 隐式全能流（Implicit Omni）：如 MMOU，代表着未来的终极形态。模型必须直面嘈杂的原生音频流，在视听交融的混沌中提取真理。
未来的研究留下了激动人心的开放问题：随着模型 OCR 能力的突破，未来的 MLLM 是否能直接“读懂”画面中烧录（Burned-in）的字幕，从而彻底终结外部字幕文件的历史？让我们拭目以待。
10Conclusion: Crossings on the Path to AGI
Looking back at the evolution from VideoMME to VideoMME-v2, and on to MMOU, we clearly see that the exploration of audio and text in video evaluation has branched into three mainstream paradigms:
- Explicit Text Stream: Like VideoMME, utilizing externally acquired SRT/JSONL as a solid prompt backbone. Ideal for scenarios requiring extreme precision in knowledge extraction.
- Pure Visual Stream: Like MLVU, deliberately stripping away audio and subtitles to force models to enhance their insight into pure visual plots and subtle actions in long videos.
- Implicit Omni Stream: Like MMOU, representing the ultimate future form. Models must confront noisy raw audio streams, extracting truth from the chaos of audio-visual fusion.
Future research leaves thrilling open questions: As model OCR capabilities break through, will future MLLMs directly "read" burned-in subtitles from the frames, thereby entirely ending the history of external subtitle files? We wait with anticipation.