I built a faithfulness evaluator for a real RAG problem

About 8 min read · Applied AI engineering · Build 01 of paper-to-production

I’m Saran, co-founder of Tekvo. This is Build 01 of an open series: read a paper → implement the useful part → measure whether it holds up under real constraints.

At a glance

• Problem: A listing-style RAG system can answer fluently and still invent prices, amenities, or merge facts across properties.
• Question: Are the claims in the answer supported by the retrieved context? (Not “does it sound right,” and not world truth.)
• Build: A from-scratch faithfulness loop — decompose → verify → |V|/|S| — plus judge audit (κ / replicates) and domain failure fixtures.
• Measured: 102 labelled cases / 193 claim labels · heuristic κ 0.733 · LLM judge κ 1.000 · ~$0.00013/eval · p50 ~978 ms (gpt-4o-mini).

The actual problem

I did not start this series because I wanted to “implement RAGAS.”

I started because I have a grounded QA problem: a listing-style RAG system can answer fluently, but I still need to know whether the answer is actually supported by the retrieved context.

That is the applied-engineering version of faithfulness:

not “does the answer sound right?”

but “are the claims in the answer supported by what the system retrieved?”

For a property or voice assistant use case, wrong prices, invented amenities, and merged facts across listings are not minor errors. They are system failures.

So the job for Build 01 was:

1. build a faithfulness evaluator for a real domain
2. make the failure cases explicit
3. measure the evaluator itself before trusting the metric

Concept in one picture

The answer as a pipeline: split into single facts, block whatever the documents don’t back, then audit the checker itself.

What the papers gave me

RAGAS: the evaluation loop

RAGAS gave me the shape of the evaluator:

1. decompose the answer into claims
2. verify each claim against retrieved context
3. score faithfulness as supported / total

That is the useful contribution for an applied engineer: a workable evaluation architecture.

What RAGAS does not give you is permission to trust the number automatically.

Faithfulness is about grounding to retrieved context. It is not world truth. If retrieval is wrong, the answer can still look “faithful” to bad context.

FActScore: claim granularity matters

FActScore’s big lesson is that factuality should be judged at atomic fact level.

That matters immediately in RAG.

If the answer says:

“Unit 4B is 2BHK and has a gym.”

and only one half is supported, a coarse claim can inflate the score. An atomic split makes the evaluator tell the truth:

• Unit 4B is 2BHK
• Unit 4B has a gym

One fact can survive while the other fails.

That is not a cosmetic improvement. It changes whether the metric catches the bug.

Reliability without Validity: audit the judge

The third paper is the one most people skip, and it is the reason this build matters.

If your evaluator uses an LLM judge, exact agreement is not enough. A judge can look stable and still mislead you.

So the evaluator itself needs validation:

• compare against human labels
• report Cohen’s κ, not only exact agreement
• run replicates
• audit order bias

That is how the metric stops being a dashboard ornament and becomes something closer to a control.

What I built in the repo

In the repo, the three papers became one production-shaped evaluator:

• RAGAS became the decompose → verify → score loop
• FActScore shaped the decomposition discipline
• Reliability without Validity became an MVVP-lite audit path

Core modules (all on GitHub):

Module	Role
pipeline.py	End-to-end evaluate + costmeter
decompose.py	Atomic claims
verify.py	Support vs context
score.py	|V| / |S|
mvvp.py	Judge audit
cli.py	faithfulness CLI

The score (the whole formula)

def compute_faithfulness(verdicts: list[ClaimVerdict]) -> float | None:
    """Return supported/total, or None when there are no claims."""
    if not verdicts:
        return None
    supported = sum(1 for v in verdicts if v.supported)
    return supported / len(verdicts)

Run it locally

git clone https://github.com/saran-io/paper-to-production.git
cd paper-to-production
python3.12 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest -q
faithfulness fixtures --mode heuristic
faithfulness audit --mode heuristic --max-examples 8

LLM mode needs OPENAI_API_KEY and pip install -e ".[llm]".

The evaluator is domain-shaped around listing answers, not generic benchmarks. Failure buckets include: fully supported · partially supported · contradictory · correct but unsupported · poor retrieval · wrong price or feature · facts merged across two properties · empty retrieval.

How I used the papers in a real project

Paper	What I took	How I used it
RAGAS	evaluator loop	built claim-level faithfulness scoring
FActScore	atomic fact discipline	split listing answers into smaller claims
Reliability without Validity	judge validation protocol	added κ, replicates, and order audit

Applied AI engineering is not:

“I read three papers and built a clone.”

It is:

“I extracted the useful parts from three papers, adapted them to a real problem, and built a measurement loop I can actually use.”

Results from the full labelled run

The publish numbers are from the 102-case labelled set (193 claim-level human labels) across the listing failure buckets — not the 8-case CI seed. Run date: 2026-08-03. Details in the results README.

Heuristic vs LLM judge (claim-level vs human)

Mode	Cohen’s κ	Raw agreement	Cost / eval	p50	p95
heuristic	0.733	0.865	$0	~0.02 ms	~0.02 ms
llm_judge (fixed human claims + LLM verify)	1.000	1.000	~$0.000078	906 ms	1314 ms
full llm (decompose + verify)	1.000	1.000	~$0.000134	978 ms	1459 ms

Model: gpt-4o-mini.

The gap between heuristic κ (0.733) and LLM κ (1.000) is the point: lexical overlap is a CI harness, not the production judge.

MVVP-lite audit on the LLM judge (full set, 3 replicates)

Check	Value
Cohen’s κ	1.000
Replicate stability	1.000
Mean order disagreement	0.000
Paradox flag	false
Full audit cost (102 × 3 + order checks)	~$0.040

Regression behavior (full LLM pipeline)

• wrong-price / feature hallucinations: none scored fully faithful
• partially supported answers: score 0.5 (atomic split working)
• empty retrieval / contradictory: score 0.0

How to read κ = 1.0

On this listing-authored labelled set, the LLM verifier matches human support labels exactly. That is a strong domain result for clear price / amenity / support questions.

It is not the same as multi-annotator agreement on messy open-world answers. The third paper’s job here is completed for this corpus — κ, replicates, order audit, paradox check — and the honest production question becomes: keep the set hard enough that a perfect κ still means something.

What this build proves

1. the evaluator shape works on real listing-style failures
2. LLM mode catches unsupported prices and features; heuristic mode is weaker (κ 0.733)
3. the repo has regression fixtures, cost/latency instrumentation, and an MVVP audit command
4. the three papers are now runnable artifacts — not a reading list

I did not just read the papers. I turned them into a working evaluator for a real RAG problem, measured it on 102 labelled cases, and audited the judge.

When I would not use this

I would not treat this score as a hard production gate if:

• the labelled set was still only the CI seed (8 cases) — that bar is cleared here at 102
• retrieval quality was the main bottleneck and “faithful to bad context” would green-light wrong answers
• I could not afford ~$0.00013 / eval and ~1s p50 for the LLM path (use heuristic only for CI regressions)
• I confused κ = 1.0 on a clean listing corpus with multi-annotator validity on messy real traffic

And more broadly: I would not use paper implementations as a signaling trick. If the paper does not change a real engineering decision, it should stay a note, not become a build.

Decision record

See ADR-001 in the repo.

Next in the series: Build 02 — endpointing and turn-taking latency.