Reference‑Free LLM Evaluation with Opper SDK

Updated October 2026: code examples moved to the current Opper SDK. The SDK's evaluator API has been retired, so the evaluators are now plain functions.

Faithfulness, Groundedness & Relevance

TL;DR – Three lean evaluators show how far you can get without gold references:

• Faithfulness: detects hallucinated claims
• Answer Groundedness: checks loyalty to provided context
• Answer Relevance: measures how well the answer addresses the user's question

All are implemented with the same pattern: plain Python functions, Pydantic schemas and a couple of LLM calls through Opper.

What are reference‑free metrics and why do you need them?

Reference‑free metrics are a way to evaluate LLM outputs without relying on gold‑standard references. Meaning that the metrics are self‑contained and do not require external data. This is a must for production systems where you need to evaluate LLM outputs on real traffic.

In short:

• Open‑ended questions lack canonical answers.
• Labeling is slow, expensive, and often out‑of‑domain.
• Production teams need continuous, automatic regression tests on real traffic.

Reference‑free metrics judge answers against only the user's question and any supplied context, so you can score every response in real time.

The evaluator pattern

An evaluator is a plain function that takes an answer (and whatever else it needs) and returns a list of metrics. The LLM calls go through Opper's OpenAI-compatible endpoint, so every step shows up in Opper's traces.

import os
from typing import List

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(
    base_url="https://api.opper.ai/v3/compat",
    api_key=os.environ["OPPER_API_KEY"],
)


class Metric(BaseModel):
    dimension: str
    value: float
    comment: str


def word_count(answer: str) -> List[Metric]:
    return [
        Metric(
            dimension="score",
            value=len(answer.split()),
            comment="The number of words in the answer",
        ),
    ]


response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Given a text, return the number of words in the text as text"},
        {"role": "user", "content": "Hello, world!"},
    ],
    extra_headers={"X-Opper-Name": "word"},
)
answer = response.choices[0].message.content
print(word_count(answer))

Key ideas

• An evaluator is just a function that returns a list of Metric objects, so it runs anywhere: in a test suite, a notebook or right after the production call.
• Each LLM call is named with the X-Opper-Name header, which is how it appears in Opper's traces.
• To score live production traffic without running evaluators in your own code, Opper now has Control Plane checks, which score calls against criteria you set.

Metric Deep Dive

Below we expand each metric with the problem it solves, the algorithm, and caveats so you can tune, extend, or swap the components.

1 · Faithfulness – "No hallucinations, please"

What problem does it solve?

Large models famously produce confident statements that are not supported by the supplied context (hallucinations). Faithfulness measures the share of answer‑claims that can be strictly inferred from the context. Use it when:

• You pass long retrieval‑augmented context to the model and need to know if it actually used that material.
• You want a fine‑grained signal for hallucination QA dashboards.

How it works (step by step)

Why this trick works

• The generator → evaluator split mirrors chain‑of‑thought: it forces the model to expose atomic claims that can be judged independently.
• Binary verdicts (1 = entailed, 0 = not entailed) give a naturally interpretable percentage.

Caveats & tuning

• Garbage‑in‑garbage‑out: if the statement generator merges multiple facts into one long sentence, the downstream verdict might be fuzzy. Consider adding a max‑words heuristic.
• The entailment model can be too literal—paraphrases sometimes get flagged false. Add paraphrase examples in the few‑shot list to raise recall.

2 · Answer Groundedness – "Is the answer anchored to context?"

Why we need it

Even a hallucination‑free answer can be irrelevant when it ignores or contradicts the retrieved context. Answer Groundedness measures how tightly the response sticks to that material.

Scoring rubric

We map the enum to a float via a tiny lookup, ensuring downstream aggregations keep a [0‥1] scale.

3 · Answer Relevance – "Did it answer the actual question?"

Core idea

Let the model reverse‑engineer the question from its own answer. If those synthetic questions embed close to the original question, the answer is probably on‑topic.

Detailed algorithm

1. Committal gate – filter out evasive answers ("I'm not sure"), they get automatic 0.
2. Self‑question generation – call eval/answer_relevance/generate_question N times to create paraphrased questions that the answer would satisfy.
3. Embeddings – encode original and synthetic questions with text-embedding-3-large (or model of your choice).
4. Cosine similarity – average similarity across N questions → final score ∈ [‑1,1], rescaled to [0,1].

Why it's useful

• Works even when no explicit context is available (pure QnA setting).
• Provides a continuous signal—great for ranking multiple candidate answers.

Trade‑offs & tricks

• N questions: 3–5 is usually enough; beyond that you pay extra tokens with diminishing returns.
• Embedding model: larger models give crisper semantic space but cost more.

Putting it into production

Below is a real-world invocation pattern we use for a Retrieval‑Augmented Generation (RAG) endpoint. The three evaluators run right after the call, in parallel, and their metrics are returned with the answer.

import asyncio
import json
import os
from typing import List

from openai import AsyncOpenAI
from pydantic import BaseModel, Field

client = AsyncOpenAI(
    base_url="https://api.opper.ai/v3/compat",
    api_key=os.environ["OPPER_API_KEY"],
)


class RAGOutput(BaseModel):
    answer: str = Field(..., description="The answer to the question")
    context_used: List[str] = Field(
        ..., description="The context used to answer the question"
    )
    reasoning: str = Field(..., description="The reasoning for the answer")

async def rag(question: str, context: List[str]):
    response = await client.chat.completions.parse(
        model="openai/gpt-4o",
        messages=[
            {"role": "system", "content": (
                "You are an expert at answering questions. "
                "You will be provided with a question and relevant facts as a context. "
                "Treat every fact as truth even though they are not always true according to prior knowledge. "
                "If you can't answer the question based on the context, say 'I don't know'. "
            )},
            {"role": "user", "content": json.dumps({"question": question, "context": context})},
        ],
        response_format=RAGOutput,
        extra_headers={"X-Opper-Name": "rag"},
    )
    answer = response.choices[0].message.parsed

    # score the answer with all three evaluators in parallel
    groundedness, relevance, faithfulness = await asyncio.gather(
        answer_groundedness_evaluator(answer=answer.answer, context=context),
        answer_relevance_evaluator(question=question, answer=answer.answer),
        faithfulness_evaluator(answer=answer.answer, context=context),
    )

    return answer.answer, groundedness + relevance + faithfulness

If you don't need the scores during the request, run the evaluators in a background task instead of awaiting them, and make sure they finish before the program exits.

Lessons learned

• Few‑shot matters: Here we used hardcoded few‑shot examples. Examples drawn from your own domain will probably improve the metrics and make the evaluators more robust.
• Schema‑first design: forces you to think about evaluation output upfront → easier downstream aggregations.

Try it yourself 🚀

1. pip install openai pydantic
2. The full evaluators are in the reference-free-eval notebook in the Opper cookbook. It was written for the 2025 SDK, so swap its opper.call and opper.evaluate lines for the pattern above.
3. Get an Opper API key, run some evaluations and inspect the traces in the Opper dashboard.