Home / Features / Evaluators

LLM Evaluators - Evals, LLM Judges and Deployment Gates

60+ LLM judges, code, and deterministic checks. Frozen datasets, gated deployments — online evals keep watch.

The evaluator library

Confidence, scored

Start from ready-made evaluators — or build your own in minutes — to score every response on the quality that matters to you, from helpfulness to safety.

  • Templates to start from
  • LLM judges, code, and rule-based checks
  • One clear pass/fail score
Failures & annotations

Learn from what breaks

When outputs miss the mark, Infere groups the failures and turns them into new evaluators — so the same mistake never ships twice.

  • Failures clustered automatically
  • One click to build a guard
  • Human review keeps judges honest
Runs & regression gates

Ship without fear

Run your prompts against frozen datasets, compare versions side by side, and automatically block any change that regresses. Quality becomes a gate, not a guess.

  • Compare versions before you merge
  • Auto-gate bad deploys
  • Watch quality on live traffic
agent-trace-quality
All SectionsEvaluator
1Basic Info
2Evaluator Type
3Tier
4LLM Config
5Criteria
6Dimensions
7Rubric
8Review
Evaluator Type
Step 2 of 8
Choose the type of evaluator that best fits your use case.
LLM BooleanLLM
True/false judgment
LLM RangeLLM
Score 0.0–1.0 on criteria
LLM CategoricalLLM
Pick from named categories
LLM ComparativeLLM
Rank/compare 2–5 responses
Python CodeCode
Execute Python in sandbox
Text SimilarityDeterministic
BLEU, ROUGE, cosine, etc.
HeuristicDeterministic
Latency, length, token efficiency
LastMile AIIntegration
LastMile AI Integration
qa-jul16-response-length
qa-jul16-response-length
Passes when the response is non-empty
CODE
Overview Config Stats Usage
Details
Created:about 1 month ago
Updated:about 1 month ago
Type:CODE
Tags:None
Quick Stats
1
Total Runs
1.00
Avg Score
Usage
Experiment
Last used about 1 month ago
Git Provenance Link to Git
New policy
New policy
Bind a prompt and dataset to a threshold policy. Regression runs against this binding gate deploys when the thresholds are not met.
BINDING
Repository
my-demo-repo
Prompt
New Template (new-template)
Token hash (optional)
Leave blank to apply to any token
Dataset
my-eval-dataset (v1)
Enabled
Disabled policies are saved but do not block deploys.
EVALUATOR VERSIONS 0 selected
No evaluator versions selected yet.
Evaluator
qa-jun30-gate-judge
Frozen version
v1
THRESHOLD POLICY
JSON object whose keys describe per-metric gates. Schema is intentionally open in v1 — must parse and be an object.
{ "minPassAtK": 0.85, "maxMeanRegression": 0.02 }
↗

Prompt Repositories

Branch and deploy prompts.

Explore →
≡

Datasets

Curate and freeze examples.

Explore →
◉

Observability

Trace every request.

Explore →

Create your first evaluator

Start from a template, freeze it, and gate a deploy.

Create evaluator → Read EML docs