Overview概览

LLaVA-OneVision-1.5-RL presents an RL post-training stage utilizing 67K curated examples with discrepancy-based selection to generate explicit chain-of-thought reasoning, achieving significant performance gains on STEM, coding, and reasoning benchmarks while maintaining visual understanding capabilities.

LLaVA-OneVision-1.5-RL 提出了一套强化学习后训练阶段，利用 6.7 万条经精筛的数据，并采用差异驱动的数据选择策略来激发显式链式思维推理，在保持视觉理解能力的同时，显著提升了模型在 STEM、代码生成与复杂推理基准上的表现。

Our contributions are threefold:

我们的贡献主要体现在三个方面：

RL Data Strategy强化学习数据策略

Discrepancy-Driven Selection差异驱动筛选

If a model can solve a task given enough attempts (high Pass@N) but rarely gets it right on the first try (low Pass@1), it already has the latent capability — it just needs to learn to use it reliably. We select tasks with this gap for RL training, filtering out tasks that are too easy (high Pass@1, nothing to learn) or too hard (low Pass@N, beyond current capability).

如果模型在给出足够多次尝试时能够解出某个任务（高 Pass@N），但首次作答却很难做对（低 Pass@1），说明它已经具备相应的潜在能力，只是尚未学会稳定地调用这种能力。我们将这类存在差距的样本用于 RL 训练，同时过滤掉过于简单（高 Pass@1，几乎没有学习空间）或过于困难（低 Pass@N，超出当前能力边界）的任务。

Reward-Based Sampling基于奖励的采样

Multiple candidate responses are filtered by average reward scores to exclude trivial and unsolvable cases, focusing on medium-difficulty instances that provide optimal learning signals.

我们通过候选回答的平均奖励分数进行过滤，剔除过于简单或几乎不可解的样本，将训练重点放在能提供最优学习信号的中等难度实例上。

Reward System Architecture奖励系统架构

Our RL setup employs a rule-based reward paradigm, where rewards are derived directly from task outcomes rather than learned preference models. Since different answer types require distinct verification strategies, we design answer-type-specific scoring rules via the reward/ module.

我们的 RL 方案采用基于规则的奖励范式：奖励直接由任务结果计算，而不是依赖学习式偏好模型。由于不同答案类型需要不同的验证策略，我们通过 reward/ 模块设计了针对答案类型的专用评分规则。

def math_reward_fn(completions, answer):
    model_answer = extract_answer(completion, format_strict=True)
    gold_parsed = parse(gt_answer, extraction_config=[
        StringExtractionConfig(), LatexExtractionConfig(), ExprExtractionConfig(),
    ])
    correct = verify(answer_parsed, gold_parsed)
    if not correct:
        correct = is_equal(gt_answer, model_answer)  # SymPy fallback
    return 1.0 if correct else 0.0

def bbox_reward_fn(completions, answer):
    xA = max(predicted[0], gt[0]); yA = max(predicted[1], gt[1])
    xB = min(predicted[2], gt[2]); yB = min(predicted[3], gt[3])
    inter = max(0, xB-xA) * max(0, yB-yA)
    iou = inter / float(areaA + areaB - inter)
    return iou  # ∈ [0, 1]

def multiplechoice_reward_fn(completions, answer):
    predicted = extract_boxed_content(completions)[-1]
    predicted = predicted.strip().strip('.()')
    predicted = predicted[0].upper() if predicted else ""
    return 1 if predicted == answer.upper() else 0

def number_reward_fn(completions, answer):
    answer_str = extract_boxed_content(completions)[-1]
    match = re.findall(r"([0-9\\.]+)", answer_str)
    count = match[-1] if match else ""
    return float(count.strip() == answer.strip())

def html_reward_fn(completions, answer):
    token_score = calculate_token_overlap(gen, ref)
    structure_score = calculate_tag_structure_similarity(gen, ref)
    reward = 0.6 * token_score + 0.4 * structure_score
    return max(0.0, min(1.0, reward))

def svg_reward_fn(completions, answer):
    token_score = calculate_token_overlap(gen, ref)
    structure_score = calculate_structure_similarity(gen, ref)
    image_score = calculate_image_similarity(gen_png, ref_png)  # SSIM
    reward = 0.5 * image_score + 0.25 * (token_score + structure_score)
    return reward  # ∈ [0, 1]

def ocr_reward_fn(completions, answer):
    dist = levenshtein_distance(gt, det)
    length = max(len(target), len(det))
    reward = 1 - min(values)
    return reward if reward >= 0.5 else 0  # threshold

Two-Stage Training Procedure两阶段训练流程

Interactive Training Pipeline: We utilize Group Relative Policy Optimization (GRPO) within the asynchronous AReaL framework. Expand the shared hyperparameters to view the global training configuration.

交互式训练流程：我们在异步 AReaL 框架中采用 Group Relative Policy Optimization（GRPO）。展开共享超参数区域即可查看全局训练配置。

Extended Capability Analysis扩展能力分析

Spatial & Grounding: RL “fast mode” significantly enhances fine-grained perception on SAT and Ref-L4 benchmarks.

空间与定位：RL 的“fast mode”在 SAT 与 Ref-L4 基准上显著提升了细粒度感知能力。

Coding: “Thinking” mode achieves highest scores on Design2Code and UniSVG, demonstrating chain-of-thought benefits for structural code generation.

代码：“Thinking” 模式在 Design2Code 与 UniSVG 上取得了最高分，说明链式思维对结构化代码生成具有明显优势。

Performance Results性能结果

GRPO Algorithm & AReaL Async FrameworkGRPO 算法与 AReaL 异步框架

GRPO

GRPO eliminates the critic by sampling G = 16 completions per prompt and using group-normalized rewards as the baseline.

GRPO 通过为每个提示采样 G = 16 个回答，并使用组内归一化奖励作为基线，从而移除了 critic。

# utils/functional.py — ppo_actor_loss_fn
ratio = torch.exp(logprobs - proximal_logprobs)
clipped_ratio = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip_higher)
pg_loss = torch.max(-advantages * ratio, -advantages * clipped_ratio)

# Behavior importance weight (off-policy correction)
behav_kl = proximal_logprobs - old_logprobs
behav_imp_weight = torch.clamp(behav_kl.exp(), max=behav_imp_weight_cap)
pg_loss = pg_loss * behav_imp_weight

AReaL Async TrainingAReaL 异步训练

AReaL decouples rollout from gradient computation — 2.77× throughput by eliminating GPU idle time.

AReaL 将 rollout 与梯度计算解耦，通过消除 GPU 空闲时间实现了 2.77× 吞吐提升。

# trains/grpo.py — main training loop
for global_step in range(start_step, max_steps):
    batch = rollout.prepare_batch(train_dataloader, workflow=workflow)
    batch["prox_logp"] = actor.compute_logp(batch)
    actor.compute_advantages(batch)
    actor.ppo_update(batch)

    rollout.pause()
    actor.update_weights(weight_update_meta)
    rollout.set_version(global_step + 1)
    rollout.resume()

Training Configuration训练配置

Acknowledgements致谢

We thank the following projects and frameworks:

我们感谢以下项目与框架：

• AReaL: Lightning-Fast RL for LLM Reasoning and Agents面向大模型推理与智能体的高吞吐强化学习框架
• sglang: Fast serving framework for LLMs and vision language models面向大语言模型与视觉语言模型的高性能服务框架
• lmms-eval: Standardized evaluation framework标准化评测框架
• LLaVA: Large Language-and-Vision Assistant大型语言-视觉助手
• LLaVA-NeXT: Next-generation multi-modal assistant新一代多模态助手

Citation引用

@inproceedings{LLaVA-OneVision-1.5,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
  booktitle={arXiv},
  year={2025}
}

@inproceedings{xie2025region,
  title={Region-based Cluster Discrimination for Visual Representation Learning},
  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
  booktitle={ICCV},
  year={2025}
}

@article{lillava,
  title={LLaVA-OneVision: Easy Visual Task Transfer},
  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
  journal={Transactions on Machine Learning Research},
  year={2024}
}