Enterprise RAG Benchmark: Retrieval Quality, Groundedness, Latency and Cost

Benchmark protocol, October 2026. No experiments have been run for this article. Acadify does not claim measured RAG performance, model rankings or cost savings here.

Executive summary

Direct answer: An enterprise RAG benchmark evaluates retrieval-augmented generation against a fixed, permissioned corpus and held-out questions. It separately measures retrieval relevance, factual grounding, citation support, abstention, end-to-end latency, cost and security. This article is a reproducible methodology, not a report of completed experiments.

Key takeaways

• Freeze corpus, question set and system configurations.
• Evaluate retrieval separately from generated factual claims.
• Check whether citations actually support answers.
• Include unanswerable queries and access-control tests.
• Measure latency, failure rates and real cost accounting.
• Publish comparative results only after verified experiments.

Dataset and reproducibility

Start with a documented benchmark objective: compare two retrievers, test a reranker, or approve a production release. Record the user population, supported languages, document types, response constraints, access policies and expected query mix. Avoid combining unrelated workflows into one leaderboard.

Freeze source document identifiers, content hashes, parser versions, chunking parameters, embedding model, index settings, reranker and generation model identifiers. Capture the ingestion date and document-level permissions. Candidates must be evaluated on comparable authorized evidence. Separate tuning and development questions from held-out test queries; identify duplicate and near-duplicate items.

Build a trustworthy evaluation set

Include simple fact lookup, multi-document synthesis, ambiguous queries, contradictory evidence, stale information, unanswerable questions and tenant-restricted requests. Label the supporting passage for each answerable question and record why unanswerable questions lack adequate evidence. Human reviewers should validate important labels and adjudicate disagreement. Synthetic questions can be useful, but label them as synthetic and check them against actual source passages.

Retrieval metrics

Retrieval should be evaluated before generation because a model cannot reliably cite evidence it never received. Define the evaluation unit: source document, passage or chunk. Record the retrieval cutoff k and the completeness of relevance judgments.

Recall at k

Recall@k equals relevant retrieved items in the top k divided by all known relevant items for that query. If the judgments are incomplete, call the metric judged recall rather than implying perfect ground truth. Questions with no relevant documents need separate reporting.

Precision, ranking and coverage

Precision@k equals relevant retrieved items divided by k. Mean reciprocal rank averages the reciprocal rank of the first relevant result, assigning zero when none is retrieved. nDCG@k accounts for graded relevance and position. Compare per-query outcomes and segment by question type; an average can hide poor performance on rare but important queries.

Interpretation

High Recall@5 does not prove that an answer is correct, cited, secure or fast. It only describes retrieval performance under the documented relevance labels and cutoff. Retain query-level examples for investigation.

Answer and citation evaluation

Assess generated answers separately from retrieval. Divide an answer into independently checkable factual claims, then mark each claim supported, contradicted or not established by the evidence actually supplied to the model. Report the fraction of supported claims among reviewed claims, with the denominator and reviewer method.

Groundedness versus correctness

An answer may happen to be correct based on model knowledge but not supported by retrieved documents. Conversely, a retrieved passage may support an outdated claim. Record answer correctness against the authoritative source and groundedness against supplied context as different fields.

Citation verification

For every factual citation, confirm that the referenced passage exists, is accessible to the user and supports the attached claim. Count unsupported citations and missing citations separately. Automated evaluators can triage examples, but high-impact cases require human calibration and review. Do not equate a model-generated grade with objective ground truth.

Abstention and security

A reliable enterprise assistant should say when authorized evidence is insufficient. Include questions with no answer in the corpus, ambiguous questions and requests whose relevant documents belong to another tenant. Measure correct abstention, false refusal and unsupported answers separately.

Permission-aware retrieval

Enforce identity, tenant and resource authorization before a passage is passed to the model. Test that the same question produces appropriately different evidence access for users with different permissions. Never treat semantic similarity as authorization.

Adversarial tests

Include malicious instructions embedded in retrieved pages, unsafe citation URLs, attempts to reveal hidden system context, index staleness, retrieval timeouts and model failures. Evaluate actual data-access or tool side effects, not only the model's text. Any confirmed cross-tenant disclosure is a release blocker regardless of average retrieval quality.

Latency and cost

Measure the full user-visible request, not only model generation. Record retrieval, reranking, prompt assembly, generation and post-processing spans using a consistent clock and tracing method. Report p50, p95 and p99 end-to-end latency alongside request counts, concurrency, geographic region and timeout policy.

Comparable operating conditions

Run baseline and candidate under comparable infrastructure, caching, workload and rate-limit conditions. Separate cold-index or cold-cache measurements from warmed runs. Do not silently exclude timed-out requests or retries: those are real operational outcomes.

Cost per completed request

Record input and output tokens, embedding and reranking charges, infrastructure allocation assumptions, retries and failed calls. State the date and currency of pricing inputs. When shared infrastructure allocation is uncertain, report an estimated range instead of a precise but unjustified number. Do not extrapolate monthly costs without a documented request distribution.

Experimental controls

Run baseline and candidate on the same held-out questions, source snapshot and permission policies. Pin model identifiers, prompts, decoding settings, retrieval depth, reranker, region and caching. Randomize run order if operating conditions may change, and repeat stochastic generations where variability matters.

Illustrative evaluation pseudocode

for question in held_out_questions:
    for candidate in [baseline, proposed]:
        result = evaluate(candidate, question, corpus_version)
        record_retrieval(result)
        record_claim_and_citation_reviews(result)
        record_abstention_latency_cost_and_errors(result)
compare_paired_results()
report_sample_sizes_and_uncertainty()

This is explanatory pseudocode, not executed benchmark code. Preserve failed and timed-out requests in denominators. Compare paired query outcomes and report uncertainty, exclusions and subgroup regressions. Do not claim statistical significance without an appropriate method and sample size.

Results: not measured

No RAG benchmark experiment has been executed for this publication. The following reporting fields are intentionally unfilled. They are not observations, estimates or rankings.

To publish actual results, include corpus and question-set manifests, configuration identifiers, sample sizes, raw or auditable query-level outcomes, evaluation rules, uncertainty and limitations. Any future performance claim must trace to verified measurements.

Proposed implementation

Week 1: Scope and baseline

Identify the business decision and risk level. Approve corpus usage, access controls, query segmentation and immutable dataset manifests. Define which production baseline will be evaluated.

Week 2: Evidence labels

Construct held-out questions with supporting passages, unanswerable cases and permission-restricted scenarios. Review labels, document disagreements and freeze the test set.

Week 3: Controlled execution

Run baseline and candidate under recorded configurations. Capture retrieval results, generated claims, citations, latency, usage, failures and security outcomes. Preserve incomplete and failed runs.

Week 4: Review and decision

Adjudicate factual claims, analyze paired results and failure segments, then document whether the candidate meets preapproved release gates. This is a proposed schedule, not completed experimental work.

Acceptance checklist

• Same authorized corpus and held-out questions
• Versioned configurations and reproducible runs
• Reviewed relevance, claim and citation labels
• Measured abstention and permission failures
• End-to-end latency and cost accounting
• Documented limitations, owners and rollback plan

Frequently asked questions

References

NIST TREC; Original RAG paper; Ragas documentation; OWASP GenAI Security.