+852 5807 1860 Request a proposal
Home / Blog / Engineering
Engineering

Context Engineering, Not Prompt Engineering: Why Your RAG Fails in Ways Prompting Can't Fix

Context engineering is the discipline of designing everything a model sees before it answers: what is retrieved, how it is chunked, how it is ordered, what is withheld, and what the model is permitted to conclude. Prompt engineering is one line in that discipline. Most RAG systems that "need better prompts" actually need a different pipeline.

Cyberwisdom Group2026-10-21 · 11 min read

On this page

  1. What is context engineering, and how is it different from prompt engineering?
  2. What are the three ways RAG fails?
  3. Which retrieval parameters should you tune?
  4. What does reranking actually buy you?
  5. When is GraphRAG worth the cost?
  6. When should you not use RAG at all?
  7. How do you handle permission inheritance in a retrieval system?
  8. What can RAG not do?

What is context engineering, and how is it different from prompt engineering?

Prompt engineering optimises the instructions you give a model. Context engineering optimises everything the model sees before it answers: which documents are retrieved, how they were chunked, how they are ordered and truncated, what is excluded, what the model may conclude. Prompt engineering is the smallest part of it.

Context engineering is defined as the systematic design of the information, tools, memory and instructions supplied to a model at inference time, together with the ordering, filtering and truncation applied to them, in order to maximise answer correctness under a cost and latency budget.

DimensionPrompt engineeringContext engineering
What is optimisedThe wording of the instructionInput assembly: documents, history, tools, permissions, format
When it appliesAt request construction, after context existsAcross indexing, retrieval, ranking, assembly, generation
Typical failureAmbiguous instruction, wrong format, missing stepRight document truncated, mis-ranked, out of scope, or unindexed
DiagnosisRead the instruction; run it by handTrace the request: retrieved, filtered, sent, cited
Who owns itUsually the application teamApplication, data and platform teams, with one named owner

The practical test is simple: if you improve the prompt and the answer does not change, the problem was never the prompt. In enterprise RAG across Singapore, Hong Kong SAR and Asia Pacific, most failures sit upstream — the document was never chunked retrievably, the right passage was out-ranked, or retrieval was never permission-filtered.

What are the three ways RAG fails?

Three failure modes account for most enterprise RAG complaints, each with a distinct signature, diagnostic and fix.

Failure modeWhat goes wrongHow to diagnose itWhat fixes it
1. Chunking destroys semanticsThe answering passage is split across chunks, or a table loses its headers, so the retrievable unit no longer stands aloneSearch the answer text directly; if the document is in the corpus but never returns, inspect its chunksStructure-aware chunking: keep sections and tables intact, carry headers forward, keep a minimum length for rows
2. Recall is right, ranking is wrongThe right chunk is retrieved but buried, and the model attends most to what is top of the promptCompare the first-pass list against the reranked list; if the chunk is present but not near the top, this is your failureReranking, then truncating to the context budget. Check whether assembly drops the best chunk
3. Generation ignores the contextThe model answers from prior knowledge, or blends retrieved text with its own assumptionsCompare answers with and without context; if nearly identical, the context is unusedConstrain the instruction to the supplied context, require citation, make absence explicit

Two compound failures are worth naming. A permission leak occurs when retrieval is not filtered by the user's entitlements, so a technically correct answer is an unauthorised one. A stale index occurs when the corpus changed after the last indexing run, so the system cites a superseded policy. Neither shows in a quality score; both surface in an incident review.

Prompt engineering asks for better phrasing. Context engineering asks for better decisions.

Both are real skills. Only one survives contact with an enterprise corpus, a changing policy set, and a regulator asking why the agent said that.

Prompt engineering
  • Optimises the phrasing of the instruction
  • Iterates on wording until the output looks right
  • Evaluated by reading the output
  • Improves with a skilled individual
  • Degrades silently when the context changes
  • Cannot explain why it worked Tuesday and not Wednesday
Context engineering
  • Optimises what the agent knows when it decides
  • Treats retrieval, ontology, granularity and freshness as design parameters
  • Evaluated against a test set with a stated threshold
  • Improves with a systematic method anyone can repeat
  • Detects and attributes degradation when the corpus changes
  • Can explain which stage failed and what would fix it

Which retrieval parameters should you tune?

Here is the checklist people ask for. Every parameter below is tunable in a well-built vector database enterprise deployment, and the last two columns describe what happens when you move the value each way — the useful way to read a setting.

ParameterWhat it controlsIf you increase itIf you decrease it
Chunk sizeText embedded as one retrievable unitMore context per hit, blurrier embeddings, more noiseSharper matches, but the answer may split
OverlapText repeated between adjacent chunksA fact near a boundary stays whole, at more index sizeLeaner index, but boundary-spanning answers are lost
Minimum retained table textFloor on how much of a table row or header is keptRows stay interpretable, at more tokensCompact tables, but rows lose meaning
Semantic versus keyword weightHow hybrid retrieval balances vector similarity against exact matchingBetter conceptual matches, worse on identifiers and quoted clausesPrecise on exact strings, weaker on paraphrase
Initial candidate countChunks pulled before any rankingHigher recall, the answer likelier in the pool, at more computeFaster and cheaper, but the right chunk may never enter the pool
Number sent to rerankingCandidates scored by the rerankerBetter ordering, more latency and costCheaper and faster, but a strong candidate may be dropped
Reranking score thresholdMinimum quality a candidate must reach to be keptFewer weak chunks, but higher risk of returning nothingMore material, more noise, longer prompt
Final recall countChunks placed in the final contextMore evidence, but diluted attention and costSharper focus, but the answer may need dropped evidence
Query rewritingWhether the question is normalised or expanded before searchHandles conversational phrasing, at the cost of driftFaithful to the question, but misses corpus vocabulary
Multi-route retrievalWhether several specialised retrievers run over one questionCovers structured and unstructured sources together, more merge logicSimpler and faster, but one index serves every question type

Work in that order and re-measure after every change. Measure with an evaluation set — fixed real questions with known sources — not by impression.

Every stage has a parameter, and every parameter has a defensible default

Six stages. The expertise is setting them per document class and knowing when to move one. The defaults are defensible; they are not always correct for your corpus.

01

Ingest

Sources, permissions and effective dates captured at ingestion, not inferred later.

1
02

Parse

Headings, clauses, tables and exceptions preserved as structure, not flattened to text.

2
03

Chunk

Granularity per document class. A policy and a contract want different settings, and one size for both is expensive.

3
04

Index

With version and lineage, so an answer can name the document version it relied on.

4
05

Retrieve

Permission and freshness filters applied before ranking, not after.

5
06

Assemble

Within an explicit context budget. What does not fit is a decision, not an accident.

6

What does reranking actually buy you?

Reranking in RAG is defined as a second scoring pass, using a model more expensive than the first-pass retriever, over a shortlist of candidates, to produce a final ordering that reflects relevance more accurately than embedding similarity alone. It is the highest-value change most enterprise RAG systems can make, and it costs latency.

Why it works: embedding similarity is blunt. Two chunks can sit close in vector space because they share a topic, not because one answers the question. A cross-encoder reads query and passage together and judges whether the passage responds.

The cost is real and should be budgeted. A reranker adds a model call per candidate, so latency scales with the number reranked, and the reranking model is typically larger and slower than the generator. Three patterns keep it affordable: rerank a modest candidate set; rerank only the top tier; rerank selectively, when the first-pass score is weak. Where a wrong answer is expensive, pay for it.

Measure the trade honestly: latency per query, and accuracy on a fixed evaluation set. "We added reranking" without those numbers learns nothing.

When is GraphRAG worth the cost?

GraphRAG uses a knowledge graph of entities and their relationships, rather than a flat collection of text chunks, as the retrieval substrate — instead of, or layered above, vector search.

GraphRAG earns its cost in three situations. Multi-hop questions ask you to traverse a chain — which supplier contracts with which entity, which of those is owned by which person. Entity relationship questions treat the link itself as the answer, which flat text retrieval handles poorly because the connection is implied by structure, not stated. Global aggregation questions require a view across the corpus — themes, trends, recurring clauses — which no chunk holds.

It does not earn its cost on single-document lookup. "What is the liability cap in contract 4471" is a retrieval-and-read task, and a well-tuned vector pipeline with reranking beats a graph approach on accuracy and latency. The reasons to stay with vector RAG are concrete: the graph index is expensive to build and keep current, it needs an entity resolution process that never quite finishes, and it adds latency to questions that never needed it. GraphRAG also fits poorly where the text is the point — narrative documents, commentary, anything whose meaning is not carried by relations.

A defensible middle path: route by question type, using graph retrieval for the subset that needs it and vector retrieval with reranking for the rest — the benefit without paying graph cost on every query.

When should you not use RAG at all?

Retrieval augmented generation is not the default answer to every knowledge question, and treating it as one is a costly habit.

When the knowledge is small. If the whole authoritative body of content fits comfortably in a prompt — a short policy set, a fixed FAQ, a data dictionary — passing it directly is more accurate, because you have removed the possibility of a bad retrieval. If the corpus fits, do not build a pipeline to fetch part of it.

When the content changes faster than you can index. If policy pages or price lists change daily, a vector index is a snapshot that is wrong by the time anyone asks. Either go to the live source at query time, or accept that the answer carries an "as at" timestamp.

When privacy constraints forbid searching a corpus. If a user may only see one authorised record, retrieval is the wrong mechanism, because it ranks candidates from a pool the user should not be exposed to. Fetch the authorised record and let the model reason over that. Deployments that search first and filter later are compliance hazards.

Enterprise programmes in Singapore, Hong Kong SAR and Asia Pacific hit a fourth case often: structured query instead of retrieval. If the answer lives in a database, query the database — retrieval on structured data produces confident prose that is quietly wrong.

How do you handle permission inheritance in a retrieval system?

This is the failure with the worst consequences and the one most often left to the end. The principle is simple: an agent must not see a document the requesting user could not see — whether it answers a person, calls a tool, or retrieves for another.

Enforcement must happen before the candidate set exists. If you retrieve the top fifty chunks and then filter by permission, the filter may leave nothing to answer from, and the ranking was already influenced by content the user was not entitled to see. The correct shape is a hard entitlement filter inside the retrieval query, carried into the vector store and source systems rather than trusted to the app.

Three details separate a working implementation from a plausible one. Permissions must be resolved at query time from the current source of truth, not copied into the index at ingestion, because entitlements change and stale copies leak. The filter must apply across every retrieval route — vector, keyword, graph, tool, cache — because one unfiltered route leaks. And the cache is part of the trust boundary: its keys must carry entitlement context, not the query.

For a Singapore, Hong Kong SAR or Asia Pacific organisation, the practical version is that PDPA and PDPO obligations, and your own data classification rules, are enforced structurally rather than by policy. Permission inherited by construction survives an audit; permission enforced by a reviewer trusting a filter generates incidents.

What can RAG not do?

  • It cannot make absent knowledge present. If the answer is not in the corpus, no retrieval tuning surfaces it. Assuming the model should know is where invention starts.
  • It cannot guarantee truth, only traceability. A grounded answer traces to a source. Whether that source is authoritative and current is a question the system does not answer.
  • It cannot rescue a corpus that was never curated. Chunking every file on a shared drive produces a system that confidently retrieves the wrong document. Corpus design — scope, exclusions, authoritative version — comes first.
  • It cannot eliminate hallucination. Hallucination reduction enterprise work comes from grounding, citation, refusal behaviour and evaluation, not retrieval alone.
  • It cannot carry live operational state. A retrieval index is a snapshot. Where the truth is a current balance, a live status or a pending approval, the answer must come from the system of record.

What RAG does well, with a curated corpus and tuned parameters, is narrow but valuable: it gives a model access to organisation-specific text it could not otherwise cite, in a way that is inspectable, measurable and tunable. It is not the same as knowing, and an organisation that confuses the two will govern a problem it does not have while leaving the one it does have alone.

For the retrieval and knowledge layer we deploy, see our knowledge and context capability page; for how evaluation closes the loop on answer quality, see our evaluation and quality page.

Three ways retrieval-augmented generation fails

We are not going to tell you these cannot happen. We will tell you which of the three you have, because the fix is different in each case and most vendors only address the third.

The answer was there

Retrieval misses

The right document was retrieved but not the right passage. Symptom: the agent is vague where it should be specific. Cause: granularity, not the model.

The answer was stale

The wrong version

A superseded policy outranks the current one because nothing encoded the effective date. Symptom: it contradicts your current policy while quoting your own document.

The answer was wrong

Retrieval works, reasoning fails

The right text was in front of it and it still concluded wrongly, often because the rule needs arithmetic or a multi-step dependency. Cause: task design.

FAQ

Frequently asked

What is the difference between context engineering and prompt engineering?

Prompt engineering optimises the instructions you give a model for one task. Context engineering optimises everything the model sees before it answers: which documents are retrieved, how they are chunked, how they are ordered, what is excluded, and what the model is permitted to do with them. Prompt engineering is one component of context engineering, and it is usually the smallest one.

What are the three ways RAG fails?

Three failure modes account for most RAG problems. Chunking destroys semantics so the right passage is never a retrievable unit. Retrieval is correct but ranking is wrong, so the best chunk is retrieved and then buried. Or generation ignores the retrieved context entirely, because the instruction did not constrain it. Each needs a different fix, and diagnosing which one you have is the first task.

Which RAG parameters should we tune first?

Start with chunk size and overlap, then the number of candidates that reach reranking, then the reranking score threshold, then the final recall count. Table handling and semantic-versus-keyword weighting matter for document sets dominated by tables or by exact identifiers. The order matters because changing the top of the pipeline invalidates every measurement you made below it.

When is GraphRAG worth the extra cost?

GraphRAG earns its cost on questions that need multiple hops, that traverse entity relationships, or that require a global aggregation across the whole corpus — "how many of our contracts contain an uncapped liability" rather than "what is the cap in contract 4471". For single-document lookup, a well-tuned vector pipeline with reranking is faster, cheaper and more accurate.

When should we not use RAG at all?

Three cases. When the knowledge is small enough to fit in the prompt reliably, retrieval adds failure modes for no benefit. When the content changes faster than you can re-index, stale retrieval is worse than no retrieval. And where strong privacy constraints apply, do not retrieve at all — fetch the specific authorised record instead of searching a corpus.

Does reranking slow the system down too much?

Yes, and that is the trade: reranking exchanges latency for accuracy by scoring candidates with a more expensive model than the generator uses. It is worth it when a wrong answer is expensive. A common pattern is to rerank only the top candidates and only on queries where the first-pass score is weak, which captures most of the benefit at a fraction of the cost.

Read next

Related

Engineering

ISO 42001 Implementation Guide

Building an AI management system that survives audit, including the tier model for agent autonomy.

Governance9 min read
Learning

LMS vs TMS vs HRIS

Why enterprises that buy only one of these three end up operating three broken systems.

Learning8 min read

Talk to a practitioner about this

WhatsApp +852 5807 1860

Talk to us about your workflow

The fastest route to a practitioner. If this article describes your situation, the useful conversation is about which workflow to start with.

Learn more →
info@cyberwisdom.net

Send a brief

For RFPs, tender documents and anything that needs a paper trail you can forward internally.

Learn more →
Singapore

Group headquarters

34 Floor, 77 Robinson Road, Singapore 068896 — enterprise AI consulting and group strategy.

Learn more →
Hong Kong SAR

Enterprise learning headquarters

LMS, TMS, custom courseware, academy and partner enablement.

Learn more →

Where to go next

Consulting

Agentic Business

The core proposition: what an organisation looks like when a meaningful share of decision and execution is delegated to agents.

Learn more →
Where to start

Agentic Operations

Which workflows to delegate first, in what order, and with what supervision.

Learn more →
Selection

20 Industries

Twenty sectors, each with a named first workflow, a named decision-maker, and where we are the wrong answer.

Learn more →
Platform

Evaluation & Quality

How evaluation becomes a production system rather than a checkpoint before go-live.

Learn more →
Topics we cover

What this page answers

Every topic below is covered in the sections above, and each links to the page that goes deeper on it.

context engineering vs prompt engineeringcontext engineering vs RAGwhat is context engineeringRAG tuning parameters guideAI agent platformenterprise AI agent platformAI agent platform evaluation criteriaAI agent runtimeagent orchestration platformmulti-agent orchestration patternsmulti-agent system architectureAI agent architecture comparisonReAct vs ReWoo vs plan-and-executewhat is ReAct agentplan-and-execute agent designdeep research agentAI agent development platformAI agent DevOpsagentic workflow design patternsAI workflow automation enterpriseAI agent tool callingMCP vs API integrationModel Context Protocol enterpriseMCP server enterprise deploymentA2A protocol enterprise AIagent-to-agent interoperabilityAI agent memory architecturecontext window management for agentsLLM as judge evaluationLLM as judge best practicesAI agent evaluation frameworkAI agent testing frameworkregression testing for AI agentsAI agent observability tracingLLM tracing and debuggingLLM token cost controlAI agent cost optimisationAI cost governance financehallucination reduction enterprisehallucination detection LLMprompt injection defence enterpriseAI firewall enterprisehuman in the loop AI designhuman in the loop vs fully autonomous AIAI agent sandboxingprivate deployment AI agentshybrid AI deployment enterprisemulti-model routing enterprisemodel gateway enterpriseAI agent observability best practicesbad case analysis LLMAI data flywheelAI agent performance monitoringAI agent SLA designenterprise agent identity and accessagent RBAC designwhat is enterprise ontologyenterprise ontology consultingenterprise ontology servicesenterprise ontology vs business glossarybusiness ontology examplesontology in AI agentsontology driven AIontology for generative AIknowledge graph vs ontologyenterprise knowledge graphenterprise knowledge graph servicesenterprise data engineering servicesenterprise data engineering consultingdata engineering for AIdata foundation for AI agentssemantic layer business intelligenceenterprise semantic layerdata mesh vs data fabricdata mesh vs central data platformmaster data management vs ontologysingle source of truth databusiness semantics governancemetadata management enterprisedata governance frameworkdata quality for AIdata lineage enterpriseunified data platform enterprisecontext engineering servicescontext engineering best practicesGraphRAG enterpriseGraphRAG vs vector RAGagentic RAGagentic RAG architectureadvanced RAG tuning parametersvector RAG tuning parametersreranking in RAGhybrid retrieval enterpriseRAG evaluation frameworkenterprise knowledge management AIknowledge management for AI agentsdocument intelligence enterpriseenterprise search AIwhat is AI agent platformhow to implement AI agent platformhow to choose a AI agent platform providerhow to evaluate AI agent platform vendorshow to build an AI agent platform roadmapAI agent platform best practicesAI agent platform implementation guideAI agent platform for mid-market companiesAI agent platform cost and pricingAI agent platform ROI and business caseAI agent platform governance and controlsAI agent platform implementation roadmapAI agent platform vs traditional approachesAI agent platform vs manual processbenefits of AI agent platform for bankingchallenges of AI agent platform for bankingAI agent platform readiness checklistAI agent platform maturity modelAI agent platform KPI frameworkAI agent platform team and skills neededAI agent platform integration with existing systemsAI agent platform pilot to production
Next step

Bring us one question your RAG cannot answer correctly.

We trace it end to end — chunk, retrieval, ranking, assembly, generation — and hand you a diagnosis of which of the three failure modes you have, plus the parameter changes that would address it. One question, one working session.

Book a session → All platforms

Talk to us about this topic

If this raises a specific question in your organisation — a workflow, a learning gap, or a governance obligation — message us on WhatsApp and a practitioner will pick it up. Usually the same working day.

WhatsApp +852 5807 1860 info@cyberwisdom.net Request a proposal