Reinforcement Learning with Verifier Rewards

The fine-grained verifier score is a drop-in dense reward for both off-policy and on-policy RL, improving sample efficiency by ≈1.8× on LIBERO and ≈1.1× on MATH.

Off-policy RL: dense progress rewards for SAC

When fine-tuning a \(\pi_0\) policy on LIBERO with DSRL-SAC, each rollout is relabeled with the verifier’s per-step progress score \(\rho_t\) (see Progress Tracking) as a dense shaped reward; the transitions are stored in the replay buffer \(\mathcal{D}\) and the SAC critic is trained on returns sampled from \(\mathcal{D}\):

\[ r_t = r^{\text{env}}_t + \lambda\,\rho_t \qquad\quad \mathcal{D} \leftarrow \mathcal{D} \cup \{(s_t, a_t, r_t, s_{t+1})\} \]

Sample efficiency on LIBERO (\(\pi_0\) + DSRL-SAC): environment timesteps required to reach each target success rate (averaged over \(n{=}5\) seeds).

Target SR (%)	Sparse	LLM-as-a-Verifier	Steps saved	Speedup
20	132,800	74,300	58,500	1.79×
40	309,400	168,500	140,900	1.84×
60	993,800	600,000	393,800	1.66×

On-policy RL: dense reasoning rewards for GRPO

When fine-tuning Qwen3-8B on MATH with GRPO, sparse correctness rewards give no gradient when every sampled answer is wrong. Each completion’s reasoning trace is scored with the Probabilistic Pivot Tournament, and this reasoning-quality score is added to the correctness and format rewards to provide additional signal:

\[ r_i = r_{\mathrm{correct},i} + r_{\mathrm{format},i} + \beta\, r_{\mathrm{reasoning},i} \]

Sample efficiency on MATH (Qwen3-8B + GRPO): sampled completions required to reach each target success rate (averaged over \(n{=}3\) seeds; each step samples 1024).

Target SR (%)	Sparse	LLM-as-a-Verifier	Completions saved	Speedup
20	40,890	36,010	4,880	1.14×
40	47,990	43,450	4,540	1.10×
60	55,860	50,780	5,080	1.10×

Note

Both integrations use the verifier zero-shot — no reward-model fine-tuning is involved. The progress reward comes from llm_verifier.track / ProgressTracker, and the reasoning reward from llm_verifier.select-style tournaments over sampled completions.