RAG Architecture and Answer Quality

Consider a fictional HR assistant that answered employees’ policy questions from the company handbook and cited the passages it used in each answer. Three months after launch, four complaints reached the team. An employee was told she could carry over twenty days of leave, as the cited passage said; the policy had changed to five in January, but the old version was still in the index next to the new one. A contractor was told he was entitled to sixteen weeks of parental leave; the cited sentence was real, but it sat under a heading, Applies to employees only, that had not come with it. An intern saw a line from the executive compensation policy among the citations. And several people who asked about working from abroad were told the handbook did not cover it, although it did, in a section the search had ranked seventh while only the top three passages were sent to the model.

The team’s first move in this example was to switch to a larger generator, but the evidence and authorization paths remained unchanged. A larger generator cannot reliably reconstruct missing conditions or enforce access rules outside the model. It might change how the answer is phrased or whether the model abstains. Diagnose the evidence path and the answer together before choosing a remedy; this case does not establish what proportion of RAG errors originate at any particular stage.

Two pipelines

Retrieval-augmented generation (RAG) answers questions from a body of documents by finding relevant passages and giving them to a language model with the question. It is built as two pipelines that run at different times. The ingestion pipeline runs ahead of time: it collects documents, extracts their text, splits it into passages, computes embeddings or other search representations, and stores them in a search index. The query pipeline runs on each question: it searches the index, selects passages, assembles them with the question into a prompt, and generates an answer. Google’s RAG Engine documentation lists the same stages, ingestion, transformation, embedding, indexing, retrieval, and generation, and describes the last as the step where “the retrieved information becomes the context added to the original user query”.

The split matters for diagnosis. Several stages can contribute to one failure. Extraction can lose a condition; version selection can admit a superseded rule; authorization can fail when a query or cached answer is served. Even with incomplete evidence, generation still determines whether to qualify, abstain, or overstate an answer. The code for each stage, including splitting, keyword and vector search, and fusing their results, is worked through in Retrieval-Augmented Generation and Tool-Using Agents. This article concerns the design decisions and the ways they fail.

Documents become chunks, and chunks lose context

Before anything can be searched, text must be extracted correctly. Scanned pages need recognition, tables need their structure kept, and repeated decorative headers, footers, and navigation text should be distinguished from meaningful headings, effective dates, and footnotes before removal. Extraction errors can be silent: the index looks full, and the missing table simply never appears in any answer.

Extracted text is then split into passages by chunking. Size is a trade-off. Google’s RAG Engine guide describes smaller chunks as more focused and larger ones as potentially losing specific details in a broad representation. This is design guidance, not a guarantee that smaller chunks retrieve better on every task. The more serious risk is separation: a sentence cut away from the heading that limits it, as with the parental-leave rule, or a table row cut away from its column names. Structure-aware splitting and title or section prefixes preserve some relationships. They do not restore conditions found in an earlier paragraph, a footnote, or another section. Retain or retrieve those dependencies as well; the indexed chunk and the expanded evidence sent to the generator need not be the same unit. Each chunk should also carry metadata that later stages depend on: a stable identifier, source document and version, effective date, audience, and access rules.

Search finds candidates; ranking chooses evidence

Retrieval usually has two stages. A first-stage search finds candidates efficiently across the indexed corpus. Inverted indexes and approximate nearest-neighbor indexes reduce the records compared; searching a corpus does not require scanning every record. Keyword search is strong on exact terms, names, codes, and rare words; embedding search is strong on paraphrase, where the question and the passage use different words for the same idea. Combining the two, often called hybrid search, can improve coverage and is a useful baseline to evaluate for policy and technical documents, which are full of exact terms. The first stage returns a generous candidate list, perhaps fifty passages.

A second stage, reranking, reads the question and each candidate together and rescores them for “how well a document answers a given query”, in the words of Google’s ranking documentation, rather than for topical similarity. Only the best few go into the model’s context. This is where the working-abroad question failed: the right passage was among the candidates but ranked seventh, and only three were sent. A reranker may move it into the final selection, or a sufficiently larger selection within the context window may include it. Increasing the cutoff from three to five would still miss an unchanged seventh-place result. Inclusion enables an answer but does not guarantee the generator will interpret the evidence correctly. Rescoring alone cannot add a passage absent from its candidate pool. Reranking costs time: Google RAG Engine’s September 2026 comparison lists under 100 milliseconds for its dedicated ranking API and one to two seconds for an LLM reranker. These are product comparison figures, not a workload-specific p95 or latency guarantee, and that time comes out of the response’s latency budget.

Check the input limit at every stage. A chunk that fits the embedding model and generator can still be truncated by the reranker. Google’s Ranking API counts title and content together: models through version 003 accept 512 tokens per record, while version 004 and later accept 1,024. A 1,024-token chunk plus a heading can exceed that budget, and different tokenizers can count it differently. Preserve source spans while selecting or splitting ranker inputs, and distinguish the retrieved text, actual ranking input, and final prompt in the trace.

Freshness and conflicting sources

An index is a derived copy of its sources, and it is only as current as the last ingestion that touched each document. Data freshness for a RAG system means stating how quickly a source change must appear in answers and measuring whether it does. The carry-over failure was not a delay, though. The new policy had been ingested; the old one remained eligible for current-policy answers. The pipeline appended versions without a reliable applicability selection rule. Define whether the service answers only current-policy questions or also historical and future-effective questions. Keep version identity, effective intervals, audience, and region so a default current view can select the applicable version without deleting history needed for as-of questions. Prepare and validate the new version’s complete chunk set before activating it; avoid serving a mixture of old and new chunks. Publication time alone does not determine when a rule applies.

Some conflicts are real rather than accidental: a policy says one thing, an FAQ written later says another, and a regional supplement overrides both in one country. Instructing the model to prefer the most recent or most authoritative source is not enough, because it cannot see dates or authority it was not given. The conflict should be resolved by rules where rules exist, such as a stated order of authority by document type and a filter by the user’s country, and shown to the user where they do not, with both sources cited, rather than settled silently by whichever passage ranked higher.

Permissions belong in retrieval

The intern’s citation came from a document the intern could not open in the document system. The index held every document’s chunks, and nothing checked the user’s access before ranking. Telling the model not to reveal restricted material does not help: once a passage is in the prompt, it can appear in the answer or its citations, and a prompt instruction is not an access control. Permissions have to be enforced in retrieval, by filtering candidates to those the current user may read before they are ranked or shown. The same principle appears in row-level security for reports: a rule enforced in one central place, on the path every request takes, is far harder to bypass than one repeated in each consumer.

Authorization must combine a trusted user and tenant identity with the effective permissions on the requested document, including relevant inheritance and deny rules. Fresh group membership alone is insufficient if the document ACL is stale.

Define the permitted revocation delay as part of the security policy, then meet it through authoritative checks or versioned, expiring permission data and invalidation. A periodic refresh is not an adequate substitute when access must stop immediately. If the system cannot establish permission, withhold the material.

Apply these rules to expanded passages, cached answers, and citations as well as initial retrieval; protect diagnostic traces that contain restricted content.

Citations and groundedness

Asking the model to cite the passage behind each statement makes answers checkable, but only if someone checks. The application should confirm that each cited identifier refers to a passage that was actually supplied, and that the passage supports the statement. Groundedness is the measure of that support. Google’s grounding check sets a strict bar, requiring that “every claim in the answer candidate must be supported by one or more of the given facts”, with partly supported claims counted as unsupported. If support is missing, narrow the answer, retrieve further authorized evidence, ask for clarification, or state that the available evidence is insufficient. A failed check of selected passages does not prove that the whole handbook lacks an answer. Do not disclose the existence of inaccessible documents through the fallback response. An unsupported claim delivered fluently is the familiar hallucination.

Groundedness and correctness differ, but the exact claim matters. A faithful summary of an old policy can be supported by that passage while being unsuitable for a current-policy question. Saying that its rule applies today may add an unsupported temporal claim. Similarly, a statement about employees does not by itself establish a contractor’s entitlement. Copying its words is not enough: the subject, scope, time, and conditions of the answer must follow from the evidence. A checker can miss this distinction, especially when extraction has removed a qualifier. Its score is fallible evidence, not a proof of entailment or authorization.

Retrieval failure or generation failure

When an answer is wrong, these questions identify candidate failure points, not a unique cause. Inspect the full set of evidence needed for the answer, and compare controlled changes to test the diagnosis. Models may participate in extraction, query rewriting, reranking, and generation; changing one can affect several stages without fixing the underlying permissions or source coverage.

Question	If no, investigate	Typical fix
Is the required evidence set in the corpus, extracted correctly?	Coverage or extraction	Add the source; repair parsing of tables and scans
Is the required evidence in the first-stage candidates?	Search or chunking	Hybrid search; structure-aware chunks; query rewriting
Did it reach the model’s context?	Ranking or selection	Reranking; a larger final selection
Was it the right evidence: current, applicable, permitted?	Freshness, scope, or access	Select applicable versions; enforce date, audience, and permission rules
Does the full answer follow the supplied evidence and its conditions?	Generation	Clearer instructions; groundedness checks; a different model

Answering these questions for a sample of real failures requires logging, per request, which passages were retrieved, their scores and ranks, which were sent to the model, and the answer with its citations. Without that record, every wrong answer looks like a model problem.

Evaluate candidate retrieval, final context, and answers separately. For each answerable question, Recall@k is the fraction of labeled relevant items found in the top k, while Hit@k records whether at least one appears. They coincide when there is exactly one relevant item per question. Define the item as a document, chunk, or required fact, account for duplicate spans, and state how scores are averaged. A multi-part question can have Hit@k of one while missing essential evidence. Use explicit candidate and final cutoffs; reordering the same full pool does not change its membership. Measure correctness, completeness, claim support, and appropriate abstention separately. Label unanswerable cases against a stated corpus, date, and permission scope; report them separately from retrieval recall, whose relevant-item denominator is otherwise zero. Follow a fixed evaluation protocol, separating development and regression cases from independent confirmation data. Repeatedly choosing settings on the same cases makes their scores selection results, not independent evidence of final quality.

What a reviewer should ask

• When a source document changes, how does the index learn of it, and how are versions activated and selected for the question’s effective date?
• Does each chunk carry its heading, source, version, effective date, audience, and access rules?
• Are permissions enforced in retrieval, before ranking, and how current are they?
• How many candidates does search return, and how many passages reach the model?
• How are conflicting sources resolved, and when is the conflict shown to the user?
• Are citations checked against the supplied passages?
• Are retrieval and answer quality measured separately, including unanswerable questions?
• Is each request’s retrieval logged well enough to locate a failure?

Three things are offered as evidence of a good RAG system and are not. Answers with citations, which show that sources were attached, not that they support the claims or were the right sources. A high groundedness score, which does not certify current, applicable, or permitted evidence and may miss unsupported scope changes. And a demonstration on questions the builders chose, which rarely includes the question the documents cannot answer.

For the fictional HR team, the revised design selects complete document versions by effective date and scope, retains necessary historical evidence, and checks source-to-index consistency. Chunks retain headings and referenced conditions, with expansion where a rule depends on another passage. Retrieval checks both the user’s identity and the document’s effective permissions, including on cache hits. Hybrid retrieval and reranking become candidates to evaluate rather than assumed fixes. Suppose the team builds 250 development questions, 40 labeled unanswerable under the agreed corpus and access scope. Retrieval metrics use the 210 answerable cases; the other 40 test appropriate abstention. A hypothetical 36 correct abstentions out of 40 is 90% on that subset, not overall accuracy: refusing every question would score 40 out of 40 there while failing all answerable tasks. Track excessive refusal, final-answer quality, and paired uncertainty by relevant segment. Confirm the selected pipeline on representative data that did not guide these choices before reporting a quality improvement.

References:

• Google Cloud, RAG Engine overview (Creative Commons Attribution 4.0).
• Google Cloud, RAG Engine: Fine-tune RAG transformations (Creative Commons Attribution 4.0).
• Google Cloud, Improve search and RAG quality with ranking API (Creative Commons Attribution 4.0).
• Google Cloud, Reranking for RAG Engine (Creative Commons Attribution 4.0).
• Google Cloud, Check grounding with RAG (Creative Commons Attribution 4.0).
• OWASP, Authorization Cheat Sheet — request-level authorization and deny-by-default behavior.