How to properly test RAG apps for speed and accuracy in CI/CD pipelines

Performance testing for Retrieval-Augmented Generation (RAG) applications requires a fundamentally different approach than traditional API testing. Unlike standard endpoints that return complete responses instantly, RAG systems combine slow retrieval phases with streaming generation, creating two distinct failure modes: poor response times and factual inaccuracies. Testing must therefore address both speed and answer quality to ensure a reliable user experience.

Why traditional load testing fails for RAG systems

Standard performance testing tools measure total request duration, which obscures critical details about RAG application behavior. A RAG endpoint doesn’t simply return a single response—it first retrieves relevant context from a vector database or search index, then generates an answer token by token. This streaming process creates three distinct latency components:

• Time to First Token (TTFT): The initial wait before any content appears
• Inter-Token Latency (ITL): How consistently subsequent tokens arrive
• Total generation time: The cumulative duration of streaming

A system with fast generation but slow TTFT feels broken to end users staring at empty screens. Conversely, slow generation with acceptable TTFT works for brief answers but frustrates users requesting long-form summaries. Aggregating these metrics into a single response time measurement hides these critical user experience differences.

Implementing dual-gate testing for RAG applications

Effective RAG testing requires two parallel evaluation gates that run against the same endpoint:

• Performance gate: Measures speed and scalability under load
• Quality gate: Verifies answer accuracy and contextual grounding

Tools like k6 handle the performance gate by simulating user traffic and measuring metrics such as TTFT, ITL, tokens per second, and p95/p99 latency. For the quality gate, solutions like DeepEval evaluate answer faithfulness and relevancy using an LLM-as-judge approach. This dual-gate system ensures that neither speed nor accuracy is sacrificed at the expense of the other.

Essential metrics for RAG performance evaluation

Speed-related metrics

| Metric | What it reveals | Why it matters | |--------|-----------------|----------------| | TTFT (Time to First Token) | Initial response delay | Most visible to users; determines perceived responsiveness | | ITL (Inter-Token Latency) | Streaming consistency | Affects reading comfort for long answers | | Tokens/sec | Generation speed | Critical for document summarization tasks | | p95/p99 latency | Tail end performance | Identifies worst-case scenarios under load |

TTFT deserves special attention because most traditional load testing tools weren’t designed to isolate this metric. They typically measure atomic request/response cycles rather than streaming responses, making them unsuitable for RAG applications without modification.

Quality-related metrics

Four metrics provide the clearest picture of RAG answer quality:

• Faithfulness: Measures whether the answer stays true to retrieved context
• Answer relevancy: Determines if the response actually addresses the question
• Context precision: Evaluates whether retrieved chunks are appropriately ranked
• Context recall: Identifies if critical information was omitted from retrieval

These metrics split their diagnostic weight across two system components. Low faithfulness with high context recall suggests a prompting problem—the retriever did its job but the generation ignored it. Conversely, high faithfulness with low context precision indicates retrieval issues where irrelevant chunks contaminated the response.

Building quality gates with DeepEval in CI/CD pipelines

CI/CD integration requires quality metrics that can fail builds automatically when regressions occur. DeepEval fits this requirement by treating evaluations as pytest test cases with configurable pass/fail thresholds. Unlike some alternatives that lock users into specific LLM providers, DeepEval accepts any judge model, providing vendor flexibility.

Consider this example test case for a documentation assistant responding to "How do I run JMeter in non-GUI mode?":

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.models import GeminiModel
import os

# Configure judge model (Gemini 3.5 Flash in this example)
judge_model = GeminiModel(
    model="gemini-3.5-flash",
    api_key=os.getenv("GEMINI_API_KEY"),
)

# Set thresholds for acceptable performance
faithfulness_metric = FaithfulnessMetric(threshold=0.75, model=judge_model)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.8, model=judge_model)

def test_jmeter_non_gui_mode_answer():
    # Query the RAG application
    question = "How do I run JMeter in non-GUI mode?"
    result = query_rag_app(question)
    
    # Create test case with input, output, and retrieved context
    test_case = LLMTestCase(
        input=question,
        actual_output=result["answer"],
        retrieval_context=result["retrieved_chunks"],
    )
    
    # Evaluate against both metrics
    for metric in [faithfulness_metric, answer_relevancy_metric]:
        metric.measure(test_case)
        status = "PASS" if metric.success else "FAIL"
        print(f"[{status}] {metric.__class__.__name__}: {metric.score:.3f}")
    
    # Collect failed metrics for build failure
    failed = [m for m in [faithfulness_metric, answer_relevancy_metric] if not m.success]

When integrated into GitHub Actions, this test automatically blocks deployments when either performance regressions (measured by k6) or quality issues (flagged by DeepEval) are detected during pull requests. The combination prevents both slow responses and hallucinated answers from reaching production environments.

Looking ahead to more robust RAG evaluation

As RAG applications become more sophisticated, the testing landscape will likely evolve beyond these two fundamental gates. Future enhancements may include real-time user feedback integration, automated prompt optimization based on failure patterns, and more granular context quality scoring. The key takeaway remains unchanged: comprehensive RAG testing requires simultaneous evaluation of performance and quality to deliver reliable, useful AI-powered experiences to end users.