Architecting Trust: Building an AI Evaluation Pipeline

If you've worked on a Retrieval Augmented Generation (RAG) application, you already know the uncomfortable truth: the old way of testing software just doesn't cut it anymore. We learned this the hard way while building and testing RAG-based systems, and this post walks through what we did about it.

Why the Old Testing Playbook Falls Short

Traditional applications are deterministic. You give them an input, you know exactly what output to expect, and you write a test that checks for it. Simple, predictable, and honestly, pretty comfortable to work with.

RAG and LLM-based systems break that comfort. The same question asked twice can come back phrased differently both times. "Correct" isn't a single fixed string anymore, it's a moving target that depends on relevance, context, and whether the answer is actually grounded in real data instead of confidently made up.

Here's a simple way to see the difference. Imagine a project status application where someone asks "how many projects are complete?"

Traditional code flow: The system runs a SQL query like SELECT COUNT(*) FROM projects WHERE status='complete'. Testing this is straightforward, you assert that the output equals the expected number. Done.

AI-based RAG flow: Now imagine the same person asks, "Show me details of ongoing projects in Q4." The system has to recognize the intent, retrieve relevant documents from a vector database, and have an LLM generate a natural-language answer using that retrieved context. Testing this means checking relevance, faithfulness, hallucination, bias, security, and even cost, not just whether a number matches.

A Test Strategy Built for RAG

Once we accepted that traditional testing wasn't going to be enough, we built a strategy around one core goal: make sure the system gives accurate, relevant, faithful answers and catches hallucinations, bias, andsecurity gaps before they reach a real user.

This means testing the whole pipeline, not just the final answer:

• Data ingestion and indexing - how documents are chunked, embedded, and stored in the vector database
• Intent recognition and routing - how the system classifies what a user is actually asking
• Retrieval - whether the right context actually gets pulled
• Generation - how well the LLM turns that context into an answer
• Output quality - accuracy, faithfulness, coherence, and absence of bias
• Security and responsible AI - privacy, access control, and resistance to adversarial prompts
• Performance and cost - speed, stability, and resource usage at scale

Tools That Made This Practical

We didn't build this from scratch. A few tools did a lot of the heavy lifting:

• MLflow for tracking and comparing experiments centrally
• Ragas for automated RAG-specific metrics like faithfulness and relevance
• LangChain (or similar LLM frameworks) for building and running the pipelines
• Locust for load testing and performance checks
• Human-in-the-loop review for the things a metric alone can't judge well, like conciseness or subtle bias

We also built a "golden query dataset," a curated, version-controlled set of queries with known-good answers and contexts, including edge cases and adversarial prompts designed to try and break the system.

The Metrics That Actually Matter

Not every query needs the same scrutiny. We mapped metrics to query type:

• Simple queries (e.g., "list projects using Python"): answer relevance above 0.85, faithfulness above 0.90, hallucination rate under 0.05
• Complex, multi-condition queries: same thresholds, plus context relevance above 0.85 and a human check for completeness.
• Out-of-scope queries (e.g., "who is the PM of India?"): the system should recognize it's out of bounds, intent routing accuracy above 0.95, safety score above 0.99.
• Guardrail tests (e.g., prompts trying to get the system to manipulate or mislead users): safety score above 0.99, bias score under 0.10, data leakage under 0.01
• Conversational, multi-turn queries: context relevance and answer relevance both above 0.85, with coherence checked by a human reviewer
• Latency: end-to-end p90 latency under 5 seconds, throughput above 10 queries per second

These aren't arbitrary numbers, they should be calibrated to your domain. A healthcare or financial application probably needs tighter thresholds on faithfulness and data leakage than a general-purpose internal tool.

Two Stories From the Field

Story 1: When "Hi" Broke the System

We noticed something odd in our metrics: when users sent simple greetings like "Hi," the system was pulling in large chunks of irrelevant context and generating unfocused answers. Faithfulness scores for these queries were sitting below 0.50, and token usage was needlessly high.

The fix was straightforward once we saw it in the data: update intent recognition to flag generic greetings, and have the retriever skip fetching context altogether for these cases. After the change, faithfulness on these queries jumped above 0.90, token usage dropped, and responses got noticeably faster.

The bigger lesson: tracking faithfulness and token metrics side-by-side in MLflow made an inefficiency visible that we probably would have missed otherwise.

Story 2: Fixing Vague Answers with Better Prompts

In another case, a user asked about revenue growth for a project in a given quarter. The original prompt design produced something like: "Project Alpha had revenue growth in Q1 2026. The growth was good compared to previous quarters."

Technically not wrong, but useless. No numbers, no specifics - and answer relevance scores reflected that.

The retrieved context was actually fine, so the problem was the prompt itself. We rewrote it with clearer rules: use only the provided context, call out gaps explicitly, avoid vague language, and follow a structured process for identifying and synthesizing the relevant information.

The result: "Project Alpha achieved 23% revenue growth in Q1 2026, increasing from $4.2M (Q4 2025) to $5.2M (Q1 2026)." Specific, grounded, and verifiable - and answer relevance and faithfulness scores both moved into the "high" range.

Where This Is Heading

Evaluation for AI systems isn't standing still. As more teams move from simple Q&A chatbots toward multi-step, agentic AI systems; ones that plan, call tools, and chain several actions together, evaluation has to stretch further too. It's no longer enough to check whether a single answer was faithful; you increasingly need to evaluate whether an agent chose the right tool, took a sensible sequence of steps, and recovered gracefully when something didn't go as planned.

One trend worth watching is the rise of "LLM-as-a-judge" evaluation, where a separate model is used to score outputs against a rubric, scaling up what used to require constant human review. It's not a replacement for human-in-the-loop checks, especially for nuanced things like bias and tone, but it's a useful way to run evaluation continuously rather than periodically. Teams are also starting to build evaluation directly into their CI/CD pipelines, so a regression in faithfulness or a spike in hallucination rate gets caught before it ships, the same way a failing unit test would.

The core idea from this article doesn't change with any of that: trust in an AI system has to be earned through measurement, not assumed. As these systems get more capable and more autonomous, that discipline only becomes more important, not less.

Wrapping Up

Moving from traditional testing to AI-based evaluation isn't optional anymore, it's a response to genuinely new risks that come with non-deterministic, data-driven systems. By mapping the right metrics to the right query types, building a solid golden dataset, and treating evaluation as a continuous process rather than a one-time checklist, teams can build AI systems that are not just functional, but genuinely trustworthy.

This blog is part of ThoughtForce, an initiative by Xoriant to showcase insights from its House of XFactors, driving thought leadership through collective expertise.