Fine-Grained Reward Estimation

Fine-Grained Reward Estimation#

This page explains the core method behind LLM-as-a-Verifier. The implementation lives in llm_verifier/fine_grained_reward.py.

Motivation: generation is not the bottleneck — verification is#

Generation vs. verification

Most agents already “know” how to solve their tasks. On Terminal-Bench, repeatedly sampling 100 trajectories per task can outperform frontier models and nearly solve the entire benchmark — but the agent doesn’t know which trajectory is correct, particularly on long-horizon tasks. Standard LLM-as-a-Judge scoring fails to provide sufficiently fine-grained feedback: when comparing complex solutions, judges often assign the same discrete score, resulting in a tie. Coarse scoring leads to 27% ties on Terminal-Bench V2.

The scoring prompt#

Given a task prompt \(x\), a language model \(p_\theta\), a criterion \(c\), and two candidate trajectories \(\tau_i\) and \(\tau_j\), we construct scoring prompts and obtain the conditional distributions \(p_{\theta}(v \mid x,c,\tau_i)\) and \(p_{\theta}(v \mid x,c,\tau_j)\) by extracting the logprobs from the <score_A> and <score_B> tags:

You are an expert [domain] reviewer. You will see a task description and two trajectories.

Evaluation Criteria: [domain specific criteria]

Task: {task prompt}
Trajectory A: {A}   Trajectory B: {B}

Carefully analyze each trajectory, then provide your final scores:

<score_A> INTEGER_1_TO_20 </score_A>
<score_B> INTEGER_1_TO_20 </score_B>

Rating Rules: Rate correctness on a 1–20 scale based on evaluation criteria
(1 = incorrect, 10 = borderline, 20 = correct)

Note

A letter-based scale (A–T) is used internally instead of digits so that each score level is a single token, enabling logprob extraction at any granularity.

The reward#

Rather than reducing each distribution into a single discrete score (as in LLM-as-a-Judge), LLM-as-a-Verifier approximates the reward of a trajectory \(\tau\) on task \(x\) as:

\[ R(x, \tau) = \frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \sum_{g=1}^{G} p_{\theta}(v_g \mid x, c, \tau)\,\phi(v_g) \]

where:

  • \(C\) = number of evaluation criteria

  • \(K\) = number of repeated verifications

  • \(G\) = number of score tokens (granularity level; the default GRANULARITY is 20)

  • \(p_{\theta}(v_g \mid x, c, \tau)\) = probability assigned by model \(\theta\) to score token \(v_g\)

  • \(\phi(v_g)\) = maps each scoring token to a scalar value

  • \(V_{\text{score}} = \{v_1, \ldots, v_G\}\) = ordered set of discrete score tokens

This probabilistic formulation substantially reduces tie rates when comparing complex solutions: the continuous reward captures the verifier’s belief, including its uncertainty, instead of collapsing it to one integer.

From rewards to preferences#

For ranking, the continuous rewards convert to a pairwise preference:

\[ p(a \succ b) = \sigma(R_a - R_b) \]

These preferences are aggregated by the Probabilistic Pivot Tournament when selecting the best of N candidates.

Score caching#

Every scored (criterion, task, A, B, repeat) tuple is cacheable. Pass cache="path/to/cache.json" to select (the bundled benchmarks each define their own cache under cache/): re-running with the same cache re-scores only comparisons not seen before, so interrupted runs resume cheaply and repeated experiments are free. Failed calls scored as ties (on_error="tie") are never persisted to the cache.