Benchmark Results

Benchmark Results#

LLM-as-a-Verifier achieves state-of-the-art performance across coding, robotics, and medical domains: Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%).

State-of-the-art across domains

Test-time scaling#

Across challenging benchmarks such as Terminal-Bench V2, SWE-Bench Verified, and MedAgentBench, LLM-as-a-Verifier — using Gemini 2.5 Flash as the verifier model — outperforms frontier models including Claude Opus 4.8, GPT-5.5, and Gemini models. Results for Terminal-Bench and SWE-Bench are reported from the official leaderboards.

Benchmark

Baseline #1

Baseline #2

Baseline #3

Pass@1

Oracle

Ours

Terminal-Bench V2

GPT-5.5 (84.7%)

Opus 4.7 (80.2%)

Gemini 3.1 Pro (80.2%)

83.1%

92.1%

86.5%

SWE-Bench Verified

Opus 4.5 (76.8%)

Gemini 3 Flash (75.8%)

MiniMax M2.5 (75.8%)

76.1%

84.4%

78.2%

MedAgentBench

Opus 4.8 (70.2%)

Gemini 3.5 Flash (66.3%)

GPT-5.5 (65.1%)

70.2%

75.0%

73.3%

Preference accuracy on RoboRewardBench#

For RoboRewardBench, pairs of rollout videos are curated that follow the same natural-language instruction but make different amounts of progress; the reward model must output a preference indicating which rollout makes more progress. Applied zero-shot with a Qwen 3.6 35B VLM, LLM-as-a-Verifier outperforms reward models fine-tuned specifically on robotics data, and reduces MAE against human annotations from 1.11 to 0.72.

Method

Accuracy (%)

TOPReward

74.7

Robometer-4B

78.8

RoboReward-8B

81.4

LLM-as-a-Judge (Discrete)

70.8

LLM-as-a-Verifier (Ours)

87.4

Progress tracking#

Progress tracking is quantified with the Value-Order Correlation (VOC) — the Spearman rank correlation between a step’s chronological index and the verifier’s predicted value for the prefix ending at that step. A verifier that tracks progress assigns monotonically higher scores to later prefixes of a successful rollout (\(\mathrm{VOC}\to 1\)):

\[ \mathrm{VOC} = \mathrm{rank\text{-}correlation}\!\left( \mathrm{argsort}(s_{t_1}, s_{t_2}, \cdots, s_{t_K}),\ (t_1, t_2, \cdots, t_K) \right) \]

VOC on Terminal-Bench V2. Successful rollouts show near-monotonic progress, while failed rollouts correlate more weakly:

Trajectory outcome

Spearman VOC

Successful

0.848

Failed

0.769

Success − Failed (gap)

+0.079

VOC on RoboRewardBench. LLM-as-a-Verifier attains the highest correlation between step index and predicted progress, outperforming fine-tuned robotics reward models:

Method

Spearman VOC

LLM-as-a-Verifier (Qwen 3.6 35B)

0.966

RoboReward-8B

0.877

Robometer-4B

0.780

TOPReward (Qwen 3.6)

0.565

Reinforcement learning#

See Reinforcement Learning with Verifier Rewards for the full LIBERO (SAC, ≈1.8× sample efficiency) and MATH (GRPO, ≈1.1×) results.