AReaL

Share link

input_ids ["123+456=?", "The", "answer", "is", "579", "."]
logprobs [-0.12, -0.41, -0.35, -0.09, -0.22]
loss_mask 0, 0, 0, 1, 1, 1, 1, 1
rewards 1.0
advantages [0.12, 0.34, 0.56, 0.78, 0.91]
versions ["v2", "v2", "v2", "v2", "v2"]

GPU 0
GPU 1
GPU 2
GPU 3
Util: 0%
GPU 0
GPU 1
GPU 2
GPU 3
Util: 0%

Architecture paths

⚡ Weight Update!
P1
P2
P3

⌊Nr−1B⌋≤i+η\left\lfloor \frac{N_r - 1}{B} \right\rfloor \leq i + \eta
42.0 -
1 41.8 42.1
42.2
35.7 41.0
∞ 34.0 36.9

JPPO(θ)=E[min⁡(ut(θ)⋅A^t,  clip(ut(θ), 1−ϵ, 1+ϵ)⋅A^t)]J_{\text{PPO}}(\theta) = \mathbb{E}\left[\min\left(u_t(\theta) \cdot \hat{A}_t,\; \text{clip}(u_t(\theta),\, 1{-}\epsilon,\, 1{+}\epsilon) \cdot \hat{A}_t\right)\right] where ut(θ)=πθ(at∣st)πold(at∣st)\text{where } u_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\text{old}}(a_t | s_t)}

J(θ)=E[∑tπproxπbehav⋅min⁡(utprox⋅A^t,  clip(utprox, 1−ϵ, 1+ϵ)⋅A^t)]J(\theta) = \mathbb{E}\left[\sum_t \frac{\textcolor{#27ae60}{\pi_{\text{prox}}}}{\textcolor{#c0392b}{\pi_{\text{behav}}}} \cdot \min\left(u_t^{\text{prox}} \cdot \hat{A}_t,\; \text{clip}(u_t^{\text{prox}},\, 1{-}\epsilon,\, 1{+}\epsilon) \cdot \hat{A}_t\right)\right] where utprox(θ)=πθ(at∣st)πprox(at∣st)\text{where } u_t^{\text{prox}}(\theta) = \frac{\pi_\theta(a_t | s_t)}{\textcolor{#27ae60}{\pi_{\text{prox}}}(a_t | s_t)}

JPPO(θ)=E[min⁡(ut(θ)⋅A^t,  clip(ut(θ), 1−ϵ, 1+ϵ)⋅A^t)]J_{\text{PPO}}(\theta) = \mathbb{E}\left[\min\left(u_t(\theta) \cdot \hat{A}_t,\; \text{clip}(u_t(\theta),\, 1{-}\epsilon,\, 1{+}\epsilon) \cdot \hat{A}_t\right)\right] where ut(θ)=πθ(at∣st)πold(at∣st)\text{where } u_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\text{old}}(a_t | s_t)}

J(θ)=E[∑tπproxπbehav⋅min⁡(utprox⋅A^t,  clip(utprox, 1−ϵ, 1+ϵ)⋅A^t)]J(\theta) = \mathbb{E}\left[\sum_t \frac{\textcolor{#27ae60}{\pi_{\text{prox}}}}{\textcolor{#c0392b}{\pi_{\text{behav}}}} \cdot \min\left(u_t^{\text{prox}} \cdot \hat{A}_t,\; \text{clip}(u_t^{\text{prox}},\, 1{-}\epsilon,\, 1{+}\epsilon) \cdot \hat{A}_t\right)\right] where utprox(θ)=πθ(at∣st)πprox(at∣st)\text{where } u_t^{\text{prox}}(\theta) = \frac{\pi_\theta(a_t | s_t)}{\textcolor{#27ae60}{\pi_{\text{prox}}}(a_t | s_t)}

JGRPO(θ)=E[1G∑i=1Gmin⁡(ρi(θ)A^i,  clip(ρi(θ), 1−ϵclip, 1+ϵclip)A^i)]J_{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\min\left(\rho_i(\theta)\hat{A}_i,\; \text{clip}(\rho_i(\theta),\, 1{-}\epsilon_{\text{clip}},\, 1{+}\epsilon_{\text{clip}})\hat{A}_i\right)\right] ρi(θ)=πθ(oi∣q)πold(oi∣q),A^i=r~i−μGσG,μG=1G∑j=1Gr~j,σG=1G∑j=1G(r~j−μG)2+δ\rho_i(\theta)=\frac{\pi_\theta(o_i|q)}{\pi_{\text{old}}(o_i|q)},\quad \hat{A}_i = \frac{\tilde r_i - \mu_G}{\sigma_G},\quad \mu_G = \frac{1}{G}\sum_{j=1}^{G}\tilde r_j,\quad \sigma_G = \sqrt{\frac{1}{G}\sum_{j=1}^{G}(\tilde r_j-\mu_G)^2 + \delta} r~i=reward_scaling⋅ri+reward_bias\tilde r_i = \text{reward\_scaling}\cdot r_i + \text{reward\_bias}

max⁡θ  L(θ)=Et ⁣[πθ(at∣st)πθold(at∣st)A^t]s.t.D‾KL(πθold∥πθ)≤δ\max_\theta \; L(\theta) = \mathbb{E}_t\!\left[\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)} \hat{A}_t\right] \quad \text{s.t.} \quad \overline{D}_{\text{KL}}(\pi_{\theta_{\text{old}}} \| \pi_\theta) \le \delta

θk+1=θk+2δg⊤F−1g  F−1g\theta_{k+1} = \theta_k + \sqrt{\frac{2\delta}{\mathbf{g}^\top \mathbf{F}^{-1} \mathbf{g}}} \; \mathbf{F}^{-1} \mathbf{g} where g=∇θL(θ)∣θk,F=∇θ2D‾KL(πθk∥πθ)∣θk\text{where } \mathbf{g} = \nabla_\theta L(\theta)\big|_{\theta_k},\quad \mathbf{F} = \nabla^2_\theta \overline{D}_{\text{KL}}(\pi_{\theta_k} \| \pi_\theta)\big|_{\theta_k}

ratio=πθ(at∣st)πold(at∣st)=exp⁡ ⁣(log⁡πθ(at∣st)−log⁡πold(at∣st))\text{ratio} = \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{old}}(a_t|s_t)} = \exp\!\bigl(\log\pi_\theta(a_t|s_t) - \log\pi_{\text{old}}(a_t|s_t)\bigr)

Ref Model Old Model Current Model
✅ ✅ ❌
❌ ❌ ✅
ref_logprobs old_logprobs current_logprobs
—

ratio

↑