Installation

A pure-Python package (Python 3.9+, dependencies: google-genai, openai, tqdm):

pip install llm-verifier

To also self-host the verifier model with vLLM, install the vllm extra (GPU required):

pip install "llm-verifier[vllm]"

Or from source — this also gives you the bundled benchmarks (run.py, data/, criteria/):

git clone https://github.com/llm-as-a-verifier/llm-as-a-verifier.git
cd llm-as-a-verifier
pip install -e .

Set up a verifier backend

LLM-as-a-Verifier extracts token-level logprobs from the verifier model. The client is picked automatically from the environment.

Option 1: Gemini via Vertex AI (default)

The default verifier model is gemini-2.5-flash. Put your Vertex AI API key in a .env file (or export it):

echo "VERTEX_API_KEY=your_key_here" > .env

Option 2: OpenAI-compatible server (vLLM / SGLang)

Any OpenAI-compatible server that returns logprobs works — e.g. a local open model served by vLLM:

vllm serve Qwen/Qwen3.5-9B --port 8000
export OPENAI_BASE_URL=http://localhost:8000/v1

OPENAI_BASE_URL takes precedence over VERTEX_API_KEY, and the served model is auto-detected — no model= argument needed (OPENAI_API_KEY is only needed for authenticated endpoints). On this backend each <score_A> / <score_B> value is read by prefilling the tag with the score position constrained to the 20 scale letters, so the distribution stays calibrated even for models that don’t reliably emit the tags — see Logit-Restricted Frontier Models, which also covers frontier models that withhold logprobs entirely.

You can also pass a pre-built openai or google-genai client via the client argument; then no environment variable is needed.