LLM on Snow

Agentic RAG Explained: Turning Retrieval into a Planning and Reflection Loop

Thu, 15 Oct 2026 15:00:00 +0800

Agentic RAG Explained: Turning Retrieval into a Planning and Reflection Loop

Traditional RAG resembles a predefined assembly line: receive a question, retrieve once, add context, and generate an answer. Agentic RAG adds a controller that can decide whether to retrieve, decompose a task, select data sources, judge whether the evidence is sufficient, and search again when it is not.

Here is the short version:

Question	Short answer
What is Agentic RAG?	A RAG architecture in which an agent plans, invokes, evaluates, and iterates retrieval at runtime
What is the main difference?	Not the presence of a vector database, but whether the retrieval control flow can change in response to intermediate results
Does it require multiple agents?	No. One agent with several retrieval tools is enough
Is it always better?	No. For simple questions it may only add latency, cost, and uncertainty
Where does it help most?	Multi-hop, multi-source tasks that require clarification or evidence checks
What matters most in production?	Bounded loops, evidence gates, permission isolation, traces, and layered evaluation

Start with traditional RAG

RAG stands for Retrieval-Augmented Generation. It combines the model’s parametric memory with external, non-parametric knowledge. After a user asks a question, the system retrieves relevant material and asks the model to answer from that material.

The original 2020 RAG paper explored combining a generative model with a dense vector index to improve factuality, knowledge updates, and provenance in knowledge-intensive tasks. In practice, a traditional online RAG pipeline often looks like this:

flowchart LR Q[User question] --> T[Clean or rewrite by fixed rules] T --> E[Create query embedding] E --> R[Retrieve Top-K chunks] R --> K[Optional reranking] K --> P[Build prompt] P --> L[LLM generates answer] L --> A[Return answer and citations]

This design has real strengths: a short path, predictable latency, straightforward caching and load testing, and a clean way to distinguish retrieval failures from generation failures. When most questions can be answered by searching one knowledge base once, this is often all you need.

The limit appears when a question hides several dependent tasks. Consider:

“Compare the permission changes in the last three releases of products A and B, then assess the migration risk for existing customers.”

The system must identify the release range, search both changelogs, inspect permission documentation, find migration notes, align dates, and only then compare the evidence. A single similarity search may cover only part of the task. Even if ten passages are returned, the model may not know what evidence is still missing.

What makes Agentic RAG “agentic”

Agentic RAG is not simply traditional RAG with a stronger model, nor merely vector search plus function calling. The real change is who controls retrieval:

Retrieval stops being a mandatory one-time stage and becomes a tool that an agent can select, repeat, and adapt according to results.

The system commonly organizes planning, tool calls, state updates, and stopping decisions into a controlled loop. The agent may decide:

whether external evidence is needed at all;
whether to use vector search, keyword search, SQL, a knowledge graph, an internal API, or the web;
whether to decompose the request into independent subquestions;
whether results are relevant, mutually consistent, and sufficient;
whether to rewrite the query, expand the time range, switch sources, or ask the user for clarification;
when enough evidence has been gathered to stop and answer.

This reason-act-observe loop follows the broader idea behind ReAct. A 2025 survey of Agentic RAG identifies planning, reflection, tool use, and multi-agent collaboration as recurring agentic patterns.

flowchart TD U[User question] --> P[Understand intent and create plan] P --> D{Enough evidence?} D -- Yes --> G[Generate from evidence] D -- No --> S[Select source and retrieval tool] S --> Q[Create or rewrite query] Q --> R[Run retrieval] R --> V[Evaluate relevance, coverage, conflicts] V --> C{Next action} C -- Gather more evidence --> S C -- Clarification needed --> H[Ask the user] C -- Budget exhausted --> F[Degrade gracefully] C -- Ready to answer --> G G --> A[Answer, citations, uncertainty]

An important boundary follows from this definition: a pipeline with query rewriting, hybrid search, and reranking can still be “advanced RAG” rather than Agentic RAG if every stage always runs in a fixed order. The agentic property comes from runtime adaptation, not from the number of components.

The complete operating loop

The online path can be separated into seven responsibilities. Implementations may merge nodes, but observability and evaluation should keep these concerns distinguishable.

Stage	Question being answered	Typical output
1. Intent analysis	What does the user need, and are external facts required?	Task type, entities, time range, constraints
2. Decomposition	Are there independently verifiable subproblems?	Subquestions and dependencies
3. Source routing	Which source is authoritative and suitable?	Tool, index, and filters
4. Query planning	Which semantic, keyword, or structured queries should run?	Executable queries
5. Result evaluation	Is the material relevant, fresh, complete, and consistent?	Scores, coverage, conflict flags
6. Correction	Should the agent change query, source, or scope?	Next action and remaining budget
7. Synthesis	Can every important claim be traced to evidence?	Answer, citations, uncertainty

Suppose a user asks, “Should we upgrade our production database from version 13 to 16?” A useful trajectory might be:

sequenceDiagram participant U as User participant A as Agent participant D as Internal docs participant M as Asset database participant W as Official docs U->>A: Should we upgrade production from 13 to 16? A->>M: Get versions, extensions, instance sizes, maintenance windows M-->>A: Return environment inventory A->>D: Search incidents, compatibility tests, change policy D-->>A: Return internal evidence A->>W: Check breaking changes and support windows for 14, 15, 16 W-->>A: Return official evidence A->>A: Check extension compatibility and evidence gaps A->>D: Search for missing extension test results D-->>A: No complete test record found A-->>U: Give a conditional recommendation and required tests

The key is not merely that several searches occurred. The later search depends on an earlier observation. If the inventory reveals a particular extension, the agent then investigates that extension. A fixed pipeline cannot economically encode every such branch.

Common architectures

Agentic RAG is a family of architectures, not a single topology.

flowchart TB O[Orchestration and state] --> P[Planner / Router] P --> T1[Vector retrieval] P --> T2[Keyword or hybrid retrieval] P --> T3[SQL / API] P --> T4[Knowledge graph / Web] T1 --> E[Evidence pool] T2 --> E T3 --> E T4 --> E E --> J[Evidence evaluator] J -->|Insufficient| P J -->|Sufficient| G[Answer generator] G --> C[Citation and safety gates]

One agent with multiple tools

A single agent plans, selects tools, evaluates results, and answers. This is the easiest design to start with when there are few tools and one permission boundary. Its weakness is that responsibilities accumulate in one prompt, making decisions harder to test and stabilize.

Router with specialized retrievers

A router only classifies the question and selects a data source; stable, specialized pipelines handle the actual retrieval. For example, policy questions use the document index, order questions use SQL, and live status uses a business API. This is more controllable than a fully open agent and often the most practical production compromise.

Corrective or reflective RAG

The system retrieves first, then evaluates result quality. Weak evidence triggers query rewriting, broader sources, or web search. CRAG focuses on correcting retrieval, while Self-RAG trains a model to decide when to retrieve and to critique its generations. These ideas overlap with Agentic RAG but are not synonyms: they can be components inside an agent loop or fixed stages in a non-agentic pipeline.

Multi-agent RAG

Work is divided among research, retrieval, verification, and writing agents, or among agents specialized by knowledge domain. This can help when work is genuinely parallel, expertise differs, or permissions must be isolated. It also increases communication cost, state coordination, and debugging complexity. Multi-agent orchestration is an optional extension, not a requirement for Agentic RAG.

Traditional RAG vs. Agentic RAG

The useful comparison is not “dumb versus smart.” It is an engineering trade-off between a deterministic pipeline and a runtime decision system.

Dimension	Traditional RAG	Agentic RAG
Control flow	Predefined	Dynamic branches and loops
Whether to retrieve	Usually always	Retrieve, skip, or defer
Query handling	Original query or fixed rewrite	Dynamic decomposition, expansion, and clarification
Data sources	One index or a fixed combination	Runtime selection across heterogeneous sources
Retrieval count	Once or a fixed number	Until evidence is sufficient or budget is exhausted
Multi-hop work	Relies on one retrieval covering the task	Uses one result to decide the next retrieval
Failure handling	Current result or fixed fallback	Evaluate, retry, switch tools, clarify, or degrade
Predictability	High	Lower; requires state and budgets
Latency and cost	Lower and steadier	Usually higher and more variable
Observability	Retrieval and final answer	Also plan, calls, state transitions, stopping reason
Security surface	Relatively concentrated	Broader tool, injection, and authorization risks
Best fit	High-volume, single-hop, bounded questions	Multi-step, multi-source research and decision support

The two approaches are not mutually exclusive. A production system can take a fast traditional path first and escalate only complex or low-confidence requests to an agentic path.

flowchart LR Q[User question] --> X{Complexity and risk check} X -- Simple, single-hop --> R[Traditional RAG fast path] X -- Multi-hop or low confidence --> A[Agentic RAG deep path] R --> G[Shared answer and citation layer] A --> G

This tiered design keeps common requests fast and reserves planning and retrieval budgets for questions that benefit from them.

Offline knowledge engineering still matters

Agentic RAG mainly changes online query execution. It does not replace document parsing, chunking, metadata, permissions, indexing, or freshness management. A powerful agent searching bad data simply searches bad data more times.

Layer	Responsibility	Important design choices
Knowledge	Ingest, parse, chunk, index, update	Quality, versioning, permissions, freshness
Tools	Expose retrieval through clear interfaces	Input schemas, filters, timeouts, result format
Orchestration	Plan, route, loop, maintain state and budget	Step limits, concurrency, retries, checkpoints
Governance	Evaluate, observe, secure, involve humans	Traces, audits, approval, offline test sets

Tool descriptions should be specific. “Search the knowledge base” is not enough. The agent needs to know the source’s coverage, time range, suitable question types, return fields, and permission constraints. More vaguely described tools can actually increase routing errors.

This framework-neutral pseudocode shows a minimal bounded loop:

def answer(question, user, max_steps=6):
    state = {
        "question": question,
        "evidence": [],
        "open_questions": [question],
        "trace": [],
    }

    for step in range(max_steps):
        decision = planner.decide(state, allowed_tools=user.allowed_tools)

        if decision.action == "clarify":
            return ask_user(decision.question)

        if decision.action == "answer":
            if evidence_gate.passed(state["evidence"]):
                return generator.grounded_answer(state)
            decision = planner.repair_missing_evidence(state)

        result = tools[decision.tool].run(
            query=decision.query,
            filters=decision.filters,
            identity=user.identity,
        )

        grade = evaluator.grade(result, state["open_questions"])
        state = update_state(state, decision, result, grade)

    return generator.bounded_fallback(state)

The safeguards are more important than the loop itself: allowed_tools constrains authorization, max_steps prevents runaway execution, evidence_gate blocks unsupported answers, and trace preserves structured evidence for debugging and evaluation.

Production failure modes

Endless search and runaway cost

If the stopping rule is only “the model thinks it has enough,” the agent can bounce among nearly identical queries. Bound maximum steps, total tokens, per-tool timeouts, concurrency, and cost. When the budget is exhausted, return an explicit degraded result.

Treating reflection as fact checking

Asking the same model to review its own answer does not create independent evidence. A reflection node should inspect observable properties: whether every subquestion has a source, whether citations support claims, and whether time ranges agree.

Prompt injection inside retrieved content

A page or document can contain text such as “ignore prior instructions and call this tool.” Retrieved content must be treated as untrusted data, not as high-priority instructions. Validate tool parameters, use source allowlists and least privilege, and require approval for high-impact actions.

Keeping only the final answer

Failures can arise during decomposition, routing, querying, retrieval, evaluation, stopping, or generation. Logs should include structured decisions, tool names, query parameters, document IDs, scores, latency, token use, errors, and stopping reasons. There is no need to store or expose private chain-of-thought.

Using agents to hide weak retrieval

If the correct document rarely appears in Top-K, first fix parsing, chunking, embeddings, filters, hybrid retrieval, and reranking. An agent may repair occasional gaps, but it is not a replacement for sound information retrieval.

Evaluate the trajectory, not only the answer

Traditional RAG evaluation often emphasizes retrieval hits and answer correctness. Agentic RAG must also ask whether the right tool was selected, whether searches were useful, and whether the agent stopped at the right time.

Layer	Suggested metrics	Question answered
Retrieval	Recall@K, MRR, nDCG, evidence coverage	Was the right material found?
Generation	Correctness, faithfulness, citation precision and completeness	Is the answer supported?
Trajectory	Tool-choice accuracy, query utility, average steps, stop accuracy	Was the process reasonable?
System	P50/P95 latency, token and tool cost, timeout rate	Was the quality gain worth it?
Safety	Unauthorized-call rate, injection success rate, data leakage rate	Did the system preserve boundaries?

Build a layered test set: simple single-hop cases for the fast path; multi-hop cases for decomposition and evidence composition; underspecified requests for clarification; conflicting sources for conflict handling; and malicious documents for prompt-injection defenses. A handful of answers that “look good” is not enough for production decisions.

When to use it

Scenario	Recommended approach	Why
FAQ, manuals, single-policy questions	Traditional RAG	Stable query shape and clear answer boundaries
One index with occasional recall problems	Advanced RAG	Improve hybrid search, reranking, rewrites, and evaluation first
Interdependent subquestions	Agentic RAG	Requires decomposition and follow-up retrieval
Documents, databases, and live APIs together	Agentic RAG	Requires dynamic routing across heterogeneous tools
Medical, financial, or production-changing actions	Agentic RAG plus human approval	Retrieval can support judgment; high-impact action still needs authorization
Very low latency, high throughput, fixed output	Traditional RAG or rules	Agent loops rarely justify their overhead

For a new system, evolve in this order:

Build an evaluable traditional RAG baseline.
Add hybrid retrieval, metadata filters, reranking, and citation checks.
Give only obvious source-selection decisions to a router.
Add a bounded retrieve-evaluate-correct loop for complex queries.
Consider multi-agent workflows, long-term memory, and broader tool access last.

Conclusion

The value of Agentic RAG is not that it makes RAG look more like an “agent.” It gives a system three abilities that a fixed retrieval pipeline lacks:

Plan: turn a complex request into verifiable retrieval tasks.
Adapt: change queries, sources, or ask for clarification based on intermediate evidence.
Verify: stop, degrade, or involve a human when evidence is missing, contradictory, or outside the permission boundary.

The costs are equally real: more model calls, higher latency, more state, a larger security surface, and harder evaluation. Mature systems usually do not make every request agentic. They retain a stable traditional RAG fast path and reserve Agentic RAG for questions that genuinely require multi-step, multi-source, runtime decisions.

In one sentence:

Traditional RAG asks, “Which passage is relevant?” Agentic RAG additionally asks, “What should I find next, where should I look, and do I have enough evidence to stop?”

References

Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, 2022.
Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, 2023.
Yan et al., Corrective Retrieval Augmented Generation, 2024.
Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, 2025.
Microsoft, Develop an Agentic RAG Solution on Azure.
LangChain, RAG agent and custom RAG agent tutorials.

FastGPT in Depth: RAG, Workflows, and Self-Hosting

Thu, 08 Oct 2026 00:00:00 +0800

FastGPT in Depth: RAG, Workflows, and Self-Hosting

FastGPT is neither a reskinned chat window nor merely a vector-database console. It brings model access, document processing, RAG retrieval, workflow orchestration, debugging, publishing, and operations into one platform so a team can turn an AI demo into a deliverable application.

Teams building a company knowledge assistant, support bot, policy Q&A system, content-review pipeline, or data agent soon hit the same problems: multiple model providers, messy documents, retrieval tuning, business logic that no single prompt can express, and production requirements such as permissions, quotas, logs, and channel integrations.

FastGPT consolidates much of that repeated infrastructure. The project describes itself as an AI Agent building platform; its Community Edition includes the core Agent, workflow, and Dataset features. This guide focuses on how those parts work together, how to deliver a small application, and where the platform does—and does not—fit.

What FastGPT actually is

FastGPT sits in the AI application layer above foundation models. The model understands and generates language; the vector store finds similar content; FastGPT connects both to business processes and exposes the result as a configurable, debuggable application.

Layer	Problem	What FastGPT adds
Models	Which chat, embedding, rerank, or vision model to call	Provider configuration and AI Proxy routing
Knowledge	How to parse, index, retrieve, and cite documents	Datasets, ingestion, vector/full-text/hybrid search, reranking
Orchestration	How to implement a multi-step task	Visual nodes, variables, branches, plugins, and Agents
Delivery	How to test, reuse, and ship the result	Debug logs, share links, APIs, embeds, and MCP
Operations	Who used it, what it cost, and what failed	Teams, quotas, records, logs, and annotation workflows

This is useful for developers who want to ship without building an entire RAG and orchestration control plane, and for product or delivery teams that can model a process but do not want to hand-code every backend component.

It is not a quality button. Answer quality still depends on source material, chunking, embeddings, retrieval settings, reranking, prompts, and the chosen model. FastGPT reduces systems work; it does not make poor data disappear.

How the core pieces cooperate

Datasets go beyond “documents to vectors”

During ingestion, a document is parsed, cleaned, chunked, indexed, and stored. At query time, FastGPT can rewrite the query, recall candidates through semantic, full-text, and image paths, fuse ranked lists with RRF, rerank candidates, and then filter the context by relevance and citation-token budget. The details are documented in the official Dataset search guide.

Four design details matter in practice:

Dataset, Collection, and Data are separate levels. A Dataset is searched as a whole, a Collection groups material much like a file, and Data entries are the units used for Q&A.
One Data entry may map to several vectors. Several focused indexes can improve recall while still returning one complete source passage.
Hybrid retrieval is more robust than vector-only search. Exact identifiers benefit from full-text search, natural questions from semantic search, and RRF combines the rankings.
Citation limits use tokens, not just Top-K. Fixed counts are unstable when chunk lengths differ greatly.

MongoDB holds most non-vector data. Vector data can live in PostgreSQL/PGVector, Milvus, OceanBase, SeekDB, and other supported backends. With PGVector, vector IDs link back to source records in MongoDB, so a complete backup must cover both stores.

Workflows turn prompts into observable programs

A minimal chat needs only an input and an AI Chat node. Real work adds classification, retrieval, HTTP calls, conditions, formatting, and fallback paths. A FastGPT node resembles a function: it has inputs, outputs, and triggers, and later nodes can reference earlier outputs.

The workflow runtime advances through connection states. A node executes when a predecessor becomes active, waits when a predecessor is still waiting, and skips when nothing can activate it. This makes branches visible and lets operators inspect each node’s inputs and outputs instead of seeing only the final model response.

Keep responsibilities narrow:

Node	One responsibility	Avoid putting this inside
Classification	Determine question type	Long-form answers
Query Rewrite	Resolve references and complete a search query	Changing exact IDs or product numbers
Dataset Search	Recall citable evidence	Final business decisions
HTTP / plugin	Fetch live data or perform a controlled action	Hidden, unauditable business rules
AI Chat	Synthesize context in the required format	Persistent business state
Fixed Reply	Deterministic errors, refusal, or handoff	Open-ended reasoning

Agents, plugins, and delivery

Workflows are best when the path is known. Agents are useful when the model must plan and choose among tools. Plugins package search, databases, messaging, document processing, and other external capabilities. They complement each other: a reliable production Agent should still constrain sensitive actions to explicit tools and guarded subflows.

Finished applications can be exposed by share link, embed, OpenAI-style API, or MCP Server. API keys are member-level OpenAPI credentials; the current documentation recommends sending appId in chat/completions. Public links consume the owning team’s quota, so use expiration, quotas, authentication, and explicit approval around tools with side effects.

Building an employee-policy assistant

The goal is a grounded assistant that cites company policy, refuses unsupported questions, and uses an internal API only for personal information.

1. Write acceptance questions first. Prepare 20–50 questions covering direct facts, cross-section reasoning, exact form IDs, multi-turn follow-ups, private-data requests, and questions absent from the source set. This becomes the regression suite for every prompt, model, chunking, or retrieval change.

2. Clean the material before tuning thresholds. Remove duplicates and obsolete versions, preserve headings, OCR scanned PDFs, and attach metadata such as department, version, and effective date. Keep each chunk semantically complete. Conditions, exceptions, and conclusions belonging to one rule should remain together.

3. Start with the smallest workflow.

User question
  → Query Rewrite (only for ambiguous follow-ups)
  → Dataset Search (hybrid retrieval + rerank)
  → AI Chat (answer only from citations)
  → Answer with sources

A useful system instruction is short and testable:

You are the company policy assistant. Answer only from the cited material.
If the evidence is insufficient, say that the current policy set has no answer.
For personal balances, approval status, or private employee data, do not guess;
require authorization before querying the internal system.
Give the conclusion first, then conditions, policy name, and effective date.

If form IDs fail, try full-text or hybrid search first. If conversational follow-ups fail, add query rewriting. If the right passages appear but are ordered poorly, inspect reranking. Change one variable at a time and rerun the same test set.

4. Add live data as a separate branch. Policy interpretation should use the Dataset; a remaining-leave balance should come from the HR system after identity and permission checks. Do not periodically embed rapidly changing balances into the vector store, and do not let the model compose unrestricted SQL.

5. Test retrieval, generation, flow, and security. Verify that the correct current policy is recalled; answers remain faithful; timeouts enter a safe fallback; and prompt injection cannot call private tools. A fluent final answer is not enough evidence that the system works.

Self-hosting and production concerns

The official Docker Compose architecture includes the FastGPT application, MongoDB, a vector database, and AI Proxy. Agent/Skill sandboxes, object storage, MCP, and advanced parsing can add more services.

The current official installer runs on Linux, macOS, or Windows through WSL:

FASTGPT_DEPLOY_BASE_URL=https://doc.fastgpt.cn \
  bash <(curl -fsSL https://doc.fastgpt.cn/deploy/install.sh)

docker compose --profile prepull pull
docker compose up -d

The script selects a vector backend, downloads Compose configuration, and generates credentials. For production, download and review remote scripts before execution, retain the generated Compose file and secrets, and pin explicit image versions instead of depending indefinitely on latest.

After deployment, configure the public FE_DOMAIN, add at least a chat model and embedding model in the admin console, and install the required system plugins. Put the service behind an HTTPS reverse proxy and expose only necessary ports.

Production area	Practical baseline
Persistence	Durable volumes for MongoDB, vector data, and object storage; prove that restore works
Secrets	Replace defaults; keep model keys, database passwords, and API keys out of Git
Network	Never expose databases publicly; restrict egress and apply upload/domain allowlists
Upgrades	Read migration notes, upgrade in small steps, back up first, and replay the regression set
Observability	Track node latency, model failures, token use, retrieval hits, and refusal rates
Capacity	Watch vector count, indexing time, MongoDB growth, concurrency, and provider limits

Start with the lightest backend that meets measured needs. PGVector is a practical small-to-medium starting point in the official guidance; distributed engines such as Milvus become relevant at much larger scale. Migration includes index rebuilding, downtime planning, performance validation, and backups—not merely copying vectors.

API integration and system boundaries

FastGPT can be the chat UI or sit behind your own web app and collaboration channels. Its OpenAI-style API lowers integration cost, but compatibility does not mean every extension field is identical across versions. Verify against the OpenAPI documentation shipped with your instance.

The shape of a request is:

curl https://fastgpt.example.com/api/v1/chat/completions \
  -H "Authorization: Bearer $FASTGPT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "appId": "your-app-id",
    "stream": false,
    "messages": [
      {"role": "user", "content": "How much leave does a probationary employee receive?"}
    ]
  }'

Treat FastGPT as an AI decision and generation service, not as the system of record. Orders, balances, and approval results remain in deterministic business systems. For payments, notifications, deletion, or other side effects, add authorization, schema validation, idempotency, and human confirmation rather than trusting model-generated arguments.

Fit, trade-offs, and license

Scenario	Fit	Why
Enterprise document Q&A and support knowledge	High	Datasets, citations, workflows, and delivery are integrated
Internal process assistants and content review	High	Visual flows connect models, rules, and HTTP tools
Delivering several vertical AI apps quickly	High	Models, Datasets, plugins, and templates are reusable
A basic model playground	Medium	It works, but may be heavier than necessary
Ultra-low-latency inference gateway	Low	Use a specialized gateway and inference stack
Transactional core business workflow	Low	LLM orchestration is not a transaction system
Competing multi-tenant SaaS	Review first	The license places additional conditions on this use

Compared with Dify, Coze, Langflow, Flowise, or a custom LangChain/LlamaIndex stack, FastGPT has a clear center of gravity: knowledge-based applications, visual orchestration, and private delivery. Platforms provide speed and consistency; code-first stacks provide maximum control. Choose according to which engineering work the team wants to avoid and which platform constraints it accepts.

Read the license carefully. The FastGPT Open Source License is based on Apache License 2.0 but adds conditions for operating a similar multi-tenant SaaS and for modifying or removing console logo and copyright information. This is not legal advice; check the repository’s current LICENSE before commercial use.

The lasting value of FastGPT is that it makes an AI application buildable, observable, and reusable. Its main risk is making a full canvas look like a finished system. Track retrieval hit rate, groundedness, task success, refusal and handoff rates, P50/P95 latency, cost per successful task, and regression results across every model, prompt, Dataset, and workflow change.

The useful rule is simple: treat FastGPT as an AI application engineering platform, not as a substitute for evaluation. Prove value with the smallest flow, improve retrieval and prompts against a fixed test set, and add Agents and automated actions only when the simpler system has earned that complexity.

This article reflects official material available in October 2026. FastGPT moves quickly, so recheck deployment variables, version dependencies, and API fields before implementation.

Official references: GitHub repository · Community Edition · Workflow runtime · Dataset retrieval · Docker Compose deployment · API publishing

KV Cache 深入理解：为什么只存 K/V，不存 Q

Fri, 18 Sep 2026 21:35:00 +0800

KV Cache 深入理解：为什么只存 K/V，不存 Q

KV Cache 是大模型推理里最重要的加速手段之一。它的名字里只有 K 和 V，没有 Q，不是命名偷懒，而是由自回归解码的计算结构决定的：未来 token 会反复读取过去 token 的 Key 和 Value，却不会再读取过去 token 的 Query。

先把结论放在前面：

对象	在注意力里的角色	未来 token 还会用吗	是否适合长期缓存
Query `Q`	当前 token 发出的“读请求”	否	否
Key `K`	历史 token 暴露出的“地址/索引”	是	是
Value `V`	历史 token 暴露出的“内容/信息”	是	是
Attention score	当前 Query 与历史 Key 的匹配结果	通常否	否

如果只记一句话：Q 是读者，K/V 是被读的记忆。读者每一步都会换，记忆会被后面的读者反复读取。

一、把 Attention 看成一次“查表”

Transformer 里的自注意力通常写成：

$$ Q = XW_Q,\quad K = XW_K,\quad V = XW_V $$

$$ \text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

这个公式看起来像矩阵乘法，但从直觉上可以拆成三件事：

名字	更像什么	含义
`Q` Query	查询语句	“我现在想找什么信息？”
`K` Key	索引地址	“我这里有什么特征，可以被谁匹配到？”
`V` Value	实际内容	“如果你匹配到了我，真正拿走的是这些信息。”

在普通的全序列计算中，Q/K/V 都来自同一段输入序列 X。但在自回归生成里，时间方向非常关键：模型已经生成了前面的 token，现在只需要生成下一个 token。

假设当前正在生成第 t 个 token，在某一层 decoder self-attention 中真正需要的是：

$$ \text{Attention}(q_t,K_{\le t},V_{\le t}) =\text{softmax}\left(\frac{q_tK_{\le t}^T}{\sqrt{d_k}}\right)V_{\le t} $$

注意这个公式里只有 当前的 q_t 和 历史到当前的 K/V。它并没有用到 q_1, q_2, ..., q_{t-1}。

这就是 KV Cache 的根。

二、Prefill 与 Decode：缓存到底发生在哪

大模型生成通常可以分成两个阶段：

阶段	输入规模	主要工作	KV Cache 的状态
Prefill	整个 prompt	一次性处理用户输入、系统提示词、历史上下文	建立初始 `K/V`
Decode	每次 1 个新 token	用新 token 继续生成下一个 token	每步追加新的 `K/V`

在 Prefill 阶段，模型要把 prompt 里的所有 token 都跑一遍。比如 prompt 长度是 n，每一层都会产生：

1
2

K_cache[layer] = [k1, k2, ..., kn]
V_cache[layer] = [v1, v2, ..., vn]

然后进入 Decode 阶段。每生成一个新 token，模型并不会把从 x1 到 xt 的整段上下文重新计算一遍，而是只处理最新 token：

new token -> 当前层 hidden state
          -> q_new, k_new, v_new
          -> q_new 读取已有 K_cache / V_cache
          -> k_new, v_new 追加进缓存

用伪代码看会更清楚：

# prefill
for layer in layers:
    q, k, v = project(prompt_hidden_states)
    K_cache[layer] = k
    V_cache[layer] = v
    hidden_states = attention(q, K_cache[layer], V_cache[layer])

# decode: 每次只处理一个新 token
for step in generation:
    x = embedding(last_token)

    for layer in layers:
        q_new, k_new, v_new = project(x)

        K_cache[layer].append(k_new)
        V_cache[layer].append(v_new)

        x = attention(q_new, K_cache[layer], V_cache[layer])

    last_token = sample(lm_head(x))

这里有一个容易忽略的细节：KV Cache 是每一层都有一份，不是整个模型共享一份。 因为每一层的 hidden state 不同，投影出来的 K/V 也不同。

常见张量形状可以写成：

1
2

K_cache[layer]: [batch, kv_heads, seq_len, head_dim]
V_cache[layer]: [batch, kv_heads, seq_len, head_dim]

如果是普通多头注意力（MHA），kv_heads 通常等于 query heads；如果是 MQA/GQA，kv_heads 会少很多，这是现代大模型降低 KV Cache 显存压力的重要技巧。

三、为什么只存 K/V，不存 Q

这个问题是理解 KV Cache 的核心。直觉答案是：旧 Q 用完就没了；旧 K/V 会被未来反复访问。

更严谨一点，可以从四个角度看。

角度	为什么旧 Q 没必要存
公式	生成第 `t` 个 token 时用的是 `q_t K_{\le t}^T`，旧的 `q_i` 不在公式里
因果方向	未来 token 只能拿自己的 Query 去看过去的 Key，不能反过来让过去的 Query 查询未来
复用价值	`k_i/v_i` 会被 `q_{i+1}, q_{i+2}, ...` 多次读取；`q_i` 只服务于第 `i` 行 attention
显存收益	存 Q 会增加大量显存，却几乎不减少 decode 计算

可以把第 i 个 token 的三样东西拆开看：

token `i` 产生的东西	第 `i` 步怎么用	第 `i+1` 步以后怎么用
`q_i`	用它查询 `K_{\le i}`，生成当前 token 的注意力输出	不再需要
`k_i`	被 `q_i` 匹配，也成为缓存的一部分	会被未来每个 `q_j` 匹配
`v_i`	被当前注意力加权求和	会被未来每个注意力分布加权求和

再换一种说法：q_i 描述的是“第 i 个位置想问什么”，这个问题只对第 i 个位置有意义。到了第 j 个位置，模型关心的是“第 j 个位置想问什么”，也就是 q_j，而不是 q_i。

如果存 Q，能不能少算一点？

通常不能。

Decode 时新 token 的 q_new 必须根据当前层的 hidden state 现场算出来。这个 hidden state 又来自前面层的输出，而前面层的输出依赖当前 token、位置编码、残差、MLP、LayerNorm 以及上一层 attention 结果。换句话说，未来的 Query 在未来 token 出现之前根本不存在。

缓存旧 Query 最多能避免“重新计算旧位置的 Query”。但 KV Cache 的设计本来就已经不重新计算旧位置了：decode 阶段只跑新 token，旧位置不会再进前向计算图。于是旧 Query 没有可节省的计算。

旧 attention 分数要不要缓存？

也通常不缓存。

第 t 步真正需要的是新的一行分数：

`1`	`score_t = q_t · [k1, k2, ..., kt]^T`

旧分数 score_1, score_2, ..., score_{t-1} 描述的是旧 Query 如何看旧 Key。它们不能替代 q_t 对历史 Key 的新匹配。把所有 attention score 都存下来，显存会从线性增长变成更接近平方增长，收益却很小。

训练时为什么好像会保存 Q？

训练和推理要分开看。

训练阶段为了反向传播，框架可能会临时保留 Q/K/V、attention score、softmax 结果、MLP 激活等中间量，或者用 activation checkpointing 在反向时重算。那是为了求梯度，不是为了自回归生成加速。

推理阶段没有反向传播，目标是尽快生成下一个 token。因此长期留下来的只有未来还会被读到的东西：K/V。

四、KV Cache 换来的代价

KV Cache 的本质是用显存换时间。它显著减少重复计算，但显存会随着上下文长度线性增长。

常见估算公式是：

1
2

KV cache bytes
≈ 2 × layers × batch × seq_len × kv_heads × head_dim × bytes_per_elem

其中前面的 2 表示 K 和 V 两份缓存。

举个近似例子，假设模型有：

参数	数值
层数 `layers`	32
query heads	32
`head_dim`	128
数据类型	FP16 / BF16，2 bytes

如果是普通 MHA，kv_heads = 32：

每 token KV cache
≈ 2 × 32 × 32 × 128 × 2 bytes
= 524288 bytes
≈ 512 KiB

如果是 GQA，假设 kv_heads = 8：

每 token KV cache
≈ 2 × 32 × 8 × 128 × 2 bytes
= 131072 bytes
≈ 128 KiB

所以同样 8192 token 的上下文，单请求 KV Cache 可能就是：

注意力结构	每 token 约占	8192 token 约占
MHA，32 个 KV heads	512 KiB	4 GiB
GQA，8 个 KV heads	128 KiB	1 GiB

这还只是 batch size = 1 的估算。服务端如果要同时处理很多请求，KV Cache 很快就会成为吞吐瓶颈。

现在再看“不存 Q”的显存意义：

架构	如果额外缓存 Q，会发生什么
MHA：`q_heads = kv_heads`	在原本 K+V 两份之外再加一份 Q，缓存大约增加 50%
GQA/MQA：`q_heads > kv_heads`	Q 的头数比 KV 多，额外缓存 Q 可能比现有 KV Cache 还大

也就是说，缓存 Q 不只是“多存一点”，在 GQA/MQA 模型里可能是非常昂贵的浪费。

五、工程实践里还要注意什么

KV Cache 听起来简单，真正服务化时会牵出不少工程问题：

问题	说明
Prefix Cache	如果很多请求共享同一段系统提示词，可以复用这段前缀的 K/V
PagedAttention	把 KV Cache 切成页管理，减少长短请求混批时的显存碎片
Sliding Window	只保留最近窗口内的 K/V，适合局部注意力模型
KV Quantization	把 KV Cache 量化到更低 bit，降低显存与带宽压力
Beam Search	多个 beam 共享前缀时要正确复制、重排或引用缓存
RoPE 位置	缓存的 K 通常已经带有位置信息，复用前缀时必须保证 position offset 一致

KV Cache 不会改变模型的理论输出，它只是避免重复计算。实际工程里如果使用低精度缓存、不同 attention kernel、不同 batch 合并策略，可能会有极小的数值差异，但原理上它是等价优化。

它也不是万能加速器：

场景	KV Cache 的作用
Prefill 很长	仍然要处理完整 prompt，KV Cache 不能消除 prefill 成本
Decode 很长	很有用，因为每步都能复用历史 K/V
上下文非常长	计算变少，但显存和显存带宽压力变大
请求很多且长度不一	需要分页缓存、连续批处理、调度器配合

最后用一张小表把核心逻辑收住：

问题	答案
KV Cache 缓存的是什么	每一层历史 token 的 Key 和 Value
为什么能加速	Decode 阶段不再重复计算旧 token 的 K/V 和旧位置输出
为什么不存 Q	旧 Query 只用于旧位置的 attention，不会被未来 token 读取
代价是什么	显存随层数、序列长度、batch、KV head 数线性增长
主要优化方向	GQA/MQA、PagedAttention、Prefix Cache、KV 量化、窗口注意力

一句话总结：KV Cache 的名字不是偶然。K/V 是上下文留给未来的可读记忆；Q 是当前 token 临时发出的读取动作。动作结束就结束了，记忆才需要留下来。

How Token Counting Works: How AI Providers and API Proxies Track Usage

Fri, 04 Sep 2026 21:43:19 +0800

How Token Counting Works: How AI Providers and API Proxies Track Usage

When using LLM APIs, you often see fields such as prompt_tokens, completion_tokens, and total_tokens. A token is not simply a word or character. It is the unit produced by the model’s tokenizer, and provider billing is built around counting input, output, cached, tool, and multimodal units.

The short version:

Question	Answer
What is a token	A unit produced when the tokenizer splits text
Is it the same as word count	No. Chinese, English, punctuation, whitespace, and code tokenize differently
How providers count it	They count input before inference, output during generation, then return usage
How API proxies count it	They estimate before forwarding, prefer upstream usage after completion, then settle billing
Why local counts differ	Tokenizer mismatch, hidden prompts, tool schemas, RAG context, multimodal inputs, retries, and caching

1. What a token is

LLMs do not process raw text directly. Before inference, text goes through a tokenizer, which splits it into tokens and maps each token to a numeric ID.

flowchart LR A[Raw text] --> B[Tokenizer] B --> C[Token sequence] C --> D[Token IDs] D --> E[Model inference]

For example:

`1`	`Write a Dockerfile for me`

may be split into pieces such as:

`1`	`["Write", " a", " Docker", "file", " for", " me"]`

This is only illustrative. Different models can tokenize the same text differently. Common words may be one token, rare characters may be split, and whitespace, punctuation, JSON, logs, and source code can all add tokens.

The practical definition is: a token is the smallest vocabulary unit the model actually reads or generates.

2. Why billing is token-based

LLM cost is strongly related to token count:

longer inputs require more context processing;
longer outputs require more generation steps;
larger context windows increase memory and attention cost;
tools, multimodal input, long context, caching, and reasoning models add more accounting dimensions.

A simplified formula:

1
2
3

cost = input_tokens * input_price
     + output_tokens * output_price
     + cached / reasoning / image / audio / tool-related token cost

Output tokens are often priced higher because they are generated step by step. Each new token depends on the current context and previous generated tokens.

3. What gets counted in a request

Real API requests usually contain more than the user’s latest message.

Content	Usually counted	Note
User message	Yes	The user’s visible input
System prompt	Yes	Role and behavior instructions
Developer prompt	Yes	Platform or application rules
Conversation history	Yes	Previous turns re-enter the context
RAG snippets	Yes	Retrieved context is added to the prompt
Tool definitions	Yes	Function schemas can be large
Tool results	Yes	JSON, tables, and logs may enter later turns
Output text	Yes	The model’s generated answer
Images, audio, video	Depends on model	Often converted into special billing units
Cached content	Depends on provider	May be reported and priced separately

A “simple” request may become:

system prompt
+ platform safety rules
+ conversation history
+ current user message
+ RAG context
+ tool schemas
+ output format instructions
= actual model input

This is why the same user message can cost very different amounts in a chat app, RAG app, coding assistant, agent, or search-enabled assistant.

4. How providers count tokens

Provider-side usage counting usually happens across the gateway, tokenizer, model server, and billing system.

sequenceDiagram participant C as Client participant G as API Gateway participant T as Tokenizer participant M as Model Server participant B as Billing C->>G: Send request G->>G: Auth, rate limit, context check G->>T: Tokenize input T-->>G: Input token count G->>M: Send token IDs M-->>G: Generate output tokens G->>G: Count output tokens G->>B: Write usage and billing event G-->>C: Return response and usage

Typical steps:

authenticate API key, project, organization, quota, and rate limits;
normalize messages, tools, response format, and multimodal inputs;
count input tokens with the model’s tokenizer;
check context length;
generate output and count generated tokens;
summarize usage when generation stops, is truncated, or errors;
write model, user, request ID, tokens, cost, and status into billing;
return a response with usage metadata.

A simple usage object may look like:

{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 350,
    "total_tokens": 1550
  }
}

More detailed platforms may expose:

{
  "usage": {
    "input_tokens": 1200,
    "output_tokens": 350,
    "cached_input_tokens": 800,
    "reasoning_tokens": 120,
    "image_input_tokens": 240
  }
}

Field names differ across providers, but the core idea is the same: bill the units the model actually processed and generated.

5. Streaming usage

For non-streaming calls:

`1`	`generate full answer -> count output tokens -> return response + usage`

For streaming calls:

generate token 1 -> send chunk
generate token 2 -> send chunk
generate token 3 -> send chunk
...
finish -> return final usage or record it internally

An API proxy usually relays chunks while also accumulating the response. If the upstream provider returns usage in the final chunk, the proxy uses it. If not, it may re-tokenize the final output and estimate usage locally.

Interrupted streams need careful handling:

Scenario	Typical treatment
User stops generation	Already generated output tokens may still be billed
Client disconnects	Depends on whether the proxy cancels upstream quickly
Proxy forwarding fails	Depends on whether upstream already processed the request
Timeout	May include only input or partial output

A serious proxy should not bill only by the characters received by the client. It should rely on upstream usage or server-side generation records when possible.

6. How API proxy platforms track usage

An API proxy sits between users and model providers. It forwards requests, but also manages authentication, quota, pricing, routing, retries, and audit records.

flowchart TD A[User request] --> B[Proxy auth] B --> C[Quota and balance check] C --> D[Estimate input tokens] D --> E[Choose upstream provider] E --> F[Forward request] F --> G[Read upstream usage] G --> H[Calculate cost] H --> I[Deduct balance or quota] I --> J[Write usage ledger] J --> K[Return response]

There are usually three accounting layers:

Layer	Purpose	Accuracy
Pre-request estimate	Context check, balance check, rate limiting	Estimate
Post-request settlement	Use upstream usage or local tokenizer result	Closer to real usage
Offline reconciliation	Compare proxy records with provider invoices	Final audit

Pre-request estimate

Before forwarding, the proxy needs to decide whether the request should be accepted.

model = request.model
tokenizer = tokenizer_registry.get(model)

normalized_prompt = render_messages(
    system=request.system,
    messages=request.messages,
    tools=request.tools,
    response_format=request.response_format
)

input_tokens_estimated = tokenizer.count(normalized_prompt)

if input_tokens_estimated + request.max_output_tokens > model.context_window:
    reject("context length exceeded")

if user.remaining_quota < estimate_cost(input_tokens_estimated, request.max_output_tokens):
    reject("insufficient quota")

The render_messages step matters. Counting only the latest user message underestimates the request.

Post-request settlement

After completion, the proxy should prefer provider usage:

response = upstream.call(request)
usage = normalize_usage(response.usage, provider, model)

cost = usage.input_tokens * price.input
     + usage.output_tokens * price.output
     + usage.cached_tokens * price.cached_input
     + usage.reasoning_tokens * price.reasoning

ledger.insert({
    request_id,
    user_id,
    provider,
    model,
    usage,
    cost,
    status: "success"
})

user.balance -= cost

Provider usage is more authoritative because the provider knows internal templates, cache hits, special modalities, and exact generated output.

Usage ledger

A mature proxy writes immutable usage records instead of only reducing a user balance.

CREATE TABLE ai_usage_ledger (
    id BIGINT PRIMARY KEY,
    request_id VARCHAR(64) NOT NULL,
    user_id BIGINT NOT NULL,
    provider VARCHAR(32) NOT NULL,
    model VARCHAR(128) NOT NULL,
    prompt_tokens INT DEFAULT 0,
    completion_tokens INT DEFAULT 0,
    cached_tokens INT DEFAULT 0,
    reasoning_tokens INT DEFAULT 0,
    total_tokens INT DEFAULT 0,
    upstream_cost DECIMAL(18, 8) DEFAULT 0,
    user_cost DECIMAL(18, 8) DEFAULT 0,
    status VARCHAR(32) NOT NULL,
    created_at TIMESTAMP NOT NULL
);

The ledger makes billing explainable: request ID, user, provider, model, token counts, upstream cost, user-facing cost, and status can all be audited.

7. Why proxy counts can differ from official counts

Cause	Why it differs
Tokenizer version mismatch	Model tokenizer rules may differ or evolve
Message template mismatch	Providers wrap roles, tools, and messages internally
Hidden system prompt	Apps or proxies may inject invisible instructions
Conversation history	Old messages re-enter the input
Large tool schemas	JSON schemas can exceed the user prompt
RAG context	Retrieved chunks increase input tokens
Structured output	JSON schema and examples add prompt cost
Multimodal conversion	Images and audio are not simple text tokens
Cache hits	Cached tokens may have separate pricing
Retries	Failed upstream attempts may still create cost

Reasonable proxy behavior:

estimate locally before request;
settle with upstream usage after request;
estimate locally only when upstream usage is unavailable;
reconcile proxy logs with upstream bills regularly.

8. Context windows and maximum output

Token counting also determines whether a request can fit into the model context window:

`1`	`input tokens + maximum output tokens <= context window`

Example with an 8192-token context window:

Input tokens	Max output	Safe
2000	1000	Yes
7000	1000	Near the limit
7900	1000	Likely rejected
8200	500	Input already exceeds window

Many context length errors are not caused by the latest user message. They come from history, system prompts, RAG snippets, tool schemas, and output constraints.

9. Cached tokens

Some providers cache repeated prompt prefixes. If the system prompt, tool schema, or a long document prefix is identical across requests, the provider may reuse processed context.

Request 1:
long system prompt + tool schema + user question A
all input processed normally

Request 2:
long system prompt + tool schema + user question B
the repeated prefix may hit cache

Usage may include:

Field	Meaning
`input_tokens`	Total input tokens
`cached_input_tokens`	Input tokens served from cache
`billable_input_tokens`	Tokens charged at normal input price, depending on provider rules

For proxies, this means pricing cannot always be total_tokens * one_price. Detailed fields matter.

10. Where token cost grows

Scenario	Token pattern
Simple Q&A	Short input, output dominates
Long summarization	Long input, medium output
Code generation	Code symbols, indentation, and file context add tokens
RAG Q&A	More retrieved chunks means more input tokens
Agent tool use	Tool schemas and tool results repeatedly enter context
Structured JSON output	Schemas and field names add prompt and output tokens
Embeddings	Usually input-only usage
Image understanding	Images are converted to visual input units
Speech tasks	May use duration, audio tokens, or mixed billing

The most commonly underestimated costs are history, tool schemas, and retrieved context.

11. Estimating tokens yourself

A rough UI estimate may use:

`1`	`estimated_tokens = text.length / 3`

But application-side controls should use a tokenizer compatible with the selected model:

1
2
3

tokenizer = get_tokenizer(model)
tokens = tokenizer.encode(rendered_prompt)
count = len(tokens)

Estimate the rendered prompt, not only the latest user message:

rendered_prompt = render_chat_request(
    system_prompt,
    developer_prompt,
    history_messages,
    retrieved_context,
    tool_schemas,
    response_format,
    user_message
)

count_tokens(rendered_prompt, model)

The goal is not perfect agreement with the provider. The goal is to catch obvious context overflows and cost spikes early.

12. Engineering notes for proxy billing

Module	Purpose
Model router	Select provider, channel, and model
Token estimator	Pre-check input tokens and maximum cost
Usage normalizer	Convert provider-specific usage into internal fields
Price table	Store input, output, cached, and multimodal prices
Usage ledger	Record tokens, cost, status, and request ID
Balance system	Reserve, settle, refund, or deduct quota
Retry policy	Decide how failed and interrupted calls are charged
Reconciliation job	Compare internal ledger with provider invoices
Rate limiting	Limit requests and token usage by user, model, IP, or org

Typical request handling:

handle_request(req):
    user = authenticate(req.api_key)
    model = resolve_model(req.model)

    estimated = estimate_usage(req, model)
    reserve_quota(user, estimated.max_cost)

    try:
        upstream_resp = call_provider(req, model)
        actual_usage = normalize_usage(upstream_resp.usage)
        actual_cost = calculate_cost(actual_usage, model.price)

        settle_quota(user, actual_cost)
        write_ledger(req.id, user.id, model, actual_usage, actual_cost, "success")
        return upstream_resp

    except ProviderError as err:
        release_or_settle_by_policy(user, req.id, err)
        write_ledger(req.id, user.id, model, estimated, 0, "failed")
        raise

Two details matter:

reservation is not settlement: reserve by worst-case output, settle by actual usage;
idempotency matters: repeated reporting for the same request_id must not double-charge the user.

13. Reducing token cost

Method	Why it works	Risk
Shorten system prompts	Reduces fixed input	Too little instruction can hurt stability
Compress history	Replaces full turns with summaries	Summaries may lose details
Control RAG Top-K	Adds fewer irrelevant chunks	Recall may suffer
Simplify tool schemas	Reduces repeated schema cost	Tool calls may become less precise
Set output limits	Prevents overly long completions	Answers may be truncated
Use caching	Reuses stable prefixes	Depends on provider support
Choose model by task	Simple tasks use cheaper models	Lower quality may require retries
Log usage	Finds the most expensive paths	Must respect privacy and compliance

Summary

Token counting has three layers:

Layer	Focus
Model layer	Tokenizer turns text into token IDs, then the model generates token by token
API layer	Input, output, tools, multimodal content, and cache are summarized as usage
Platform layer	Proxies estimate, limit, settle, audit, and reconcile usage

The key idea:

A token is not a character count. It is the unit the model actually processes. Providers count real model usage; proxies estimate before forwarding, settle after receiving usage, and keep ledger records for billing and audit.

For application developers, the main habit is to measure the full rendered context. System prompts, history, RAG snippets, tool schemas, and structured output constraints all become tokens, and therefore affect cost, latency, and context limits.

About the timing of fine-tuning

Sat, 11 Jul 2026 17:30:00 +0800

关于微调的时机

微调不是大模型效果优化的默认选项，更不是遇到效果问题后的第一反应。真正成熟的工程判断，是先弄清楚：当前缺的是知识，还是模型的行为稳定性？

“什么时候应该做大模型微调？”是大模型岗位面试中很常见的一道题。

它表面上在问微调技术，实际上考察的是两个更重要的能力：

能不能区分知识问题和行为问题；
有没有技术选型的成本意识。

如果连问题类型都没有判断清楚，就直接把微调提上日程，往往会把一个本可低成本解决的问题，变成数据准备、模型训练和持续维护的复杂工程。

因此，讨论“要不要微调”之前，先记住一个结论：

知识问题优先用检索解决，行为问题才进入微调的候选范围。

先判断：缺知识，还是行为不稳定？

大模型应用中的效果问题，可以先沿着“知识”和“行为”两条线拆开。

问题类型	典型表现	对应解法
知识问题	模型缺少业务文档、规章制度、行业事实等外部信息	使用检索增强（RAG），在生成时补充所需知识
行为问题	模型知道正确内容，但输出格式、语气或判断规则不稳定	提示词无法稳定解决后，再考虑微调

知识问题：模型不知道事实

如果模型答不出某个问题，是因为它没有接触过企业内部文档、最新规章制度或特定行业事实，那么问题的核心不是“模型不会回答”，而是模型缺少回答所需的外部信息。

这类问题应该优先使用检索增强。知识保留在外部知识源中，需要时检索并提供给模型，而不是把它固化进模型权重。

这样做的关键价值是：知识可以灵活更新。文档变化时更新知识源即可，不必重新训练模型。

行为问题：模型知道，但做得不稳定

另一类问题是：正确内容模型其实知道，但它无法持续按照业务要求输出。例如：

输出结构有时正确、有时缺字段；
语气无法长期保持一致；
面对复杂判断规则时，执行结果反复波动。

这不是知识缺失，而是行为不稳定。微调真正擅长处理的，正是这类“让模型更稳定地按照指定方式做事”的问题。

所以，区分两者最直接的判断方式是：

缺事实，补检索；事实已经具备但表现不稳定，才评估微调。

效果优化要走阶梯，而不是一步跳到微调

工业界做技术选型，不能只问“哪种方案能解决”，还要问“是否有成本更低的方案能先解决”。大模型效果优化应该按照从低成本到高成本的顺序逐级推进：

顺序	优化手段	进入条件
第一步	把提示词优化到极致	所有场景都应先完成
第二步	引入检索增强方案	问题涉及外部知识，或提示词缺少必要事实
第三步	考虑使用微调	前两步走不通，且问题确实适合微调

这套阶梯的关键不是三种技术谁更高级，而是上一级没有解决问题，再进入下一级。

第一步：先把提示词优化到极致

面对格式、语气和规则执行问题，第一步不是训练，而是先验证提示词能否解决。只要提示词仍有优化空间，就不能轻易得出“必须微调”的结论。

第二步：需要外部知识时引入检索

如果问题来自知识缺失，就通过检索把业务文档、规章制度或行业事实提供给模型。检索负责补齐事实，也保留了知识更新的灵活性。

第三步：最后才考虑微调

只有提示词已经反复优化、检索也已经覆盖所需事实，问题依然存在时，微调才真正进入决策范围。

微调应该是阶梯的第三层，而不是项目启动后的默认配置。

真正需要微调的三个信号

“微调不是起点”不等于“永远不做微调”。当下面三个信号出现时，微调就可能具备明确的工程价值。

信号一：行为与格式仍然不稳定

业务要求模型稳定输出特定结构、固定语气或复杂规则，但提示词经过反复优化后，效果仍然时好时坏。

这里有两个必要前提：

目标是约束模型的行为与输出形态；
提示词优化已经充分尝试，但稳定性仍未达标。

两个条件同时成立，才说明问题可能需要通过微调进一步固化。

信号二：高频窄口径任务受到成本与延迟制约

意图分类、实体抽取、内容打标这类任务，通常口径明确、调用频率高。为了让模型准确执行规则，系统可能需要在每一次请求中重复携带很长的提示词。

长提示词会带来持续的重复开销，也会增加延迟。此时，把任务行为通过微调迁移到小模型中，能够大幅降低成本和延迟。

这里决定是否微调的，不只是效果，还有调用规模下的长期成本。

信号三：距离业务硬指标只差最后几个点

提示词已经做到位，检索也已经补齐事实，但准确率距离业务硬指标仍然差最后几个点。这说明低成本手段已经基本用尽，而剩余差距又会直接影响业务上线或验收。

此时，微调才适合作为继续突破能力边界的手段。

三个信号可以浓缩成一张判断表：

信号	关键问题	微调价值
行为与格式	提示词反复优化后，输出仍不稳定吗？	固化结构、语气和复杂规则
成本与延迟	高频窄任务是否长期重复携带长提示词？	用小模型降低持续成本与延迟
能力边界	提示词和检索都到位后，是否仍差硬指标最后几个点？	进一步提升专项准确率

微调最常见的四个坑

真正的工程风险往往不在“会不会训练”，而在于微调目标、数据和评估是否正确。下面四个问题尤其常见。

坑一：用微调灌知识

把业务文档或行业知识直接拿去微调，看起来像是在“教会模型”，实际很容易让知识变成模型中的模糊印象，反而更容易胡说。

更麻烦的是，知识会持续变化。一旦知识固化进模型权重，每次更新都可能重新触发数据准备和训练，维护成本极高。

因此，这条边界要足够明确：

知识类需求永远优先走检索，不要用微调灌知识。

坑二：重数量，轻质量

微调数据不是越多越好。几百条高质量、统一口径的样本，效果可能好过几万条脏数据。

数据中的口径冲突和错误示范，会直接让模型学到不稳定的行为。与其盲目追求数量，不如先确保样本准确、清晰且规则一致。

对于简单任务，50～100 条样本就能看到明显效果。这意味着，微调项目早期更应该关注样本质量，而不是先堆出庞大的数据规模。

坑三：忽视灾难性遗忘

模型在窄任务上被过度微调后，可能出现通用能力退化。专项任务的指标变好，并不代表整体效果一定更好。

因此，微调后的评估不能只盯着目标任务，必须同时检查：

专项能力是否提升；
通用能力是否退化。

只有两者一起评估，才能判断这次微调是否真的值得上线。

坑四：没有评估就开始微调

没有评估体系就直接训练，相当于闭着眼睛开车：训练完成后，既无法确认效果提升了多少，也无法判断出现了哪些退化。

正确的顺序应该是：评估集先于训练搭建。

先明确用什么样本和标准验证效果，再开始准备微调。否则，训练过程即使顺利完成，也无法形成可信的工程结论。

一套可落地的决策顺序

面对“当前场景是否需要微调”，可以按下面的顺序做判断：

决策步骤	要回答的问题	对应动作
1. 定义问题	缺的是外部知识，还是行为稳定性？	知识问题走检索；行为问题继续判断
2. 优化提示词	提示词是否已经反复优化到位？	没有则继续优化，不急于微调
3. 补齐知识	是否缺少业务文档、制度或行业事实？	引入 RAG，不把知识写进权重
4. 检查信号	是否出现行为不稳、成本延迟或能力边界信号？	信号明确后再立项微调
5. 搭建评估	是否已有专项与通用能力评估？	评估集就绪后才开始训练
6. 准备数据	样本是否高质量且口径统一？	先保证质量，再考虑数量

这套顺序体现的不是对微调的保守，而是对工程成本和方案边界的尊重。

面试时应该怎样回答？

如果面试官问“什么时候做微调”，一个逻辑完整的回答可以这样组织：

我会先判断这是知识问题还是行为问题。模型缺少业务文档、规章制度或行业事实时，优先使用 RAG，因为知识需要灵活更新，不应该固化进模型权重。模型已经知道正确内容，但格式、语气或判断规则在提示词反复优化后仍不稳定，才是微调的适用场景。

在方案顺序上，我会先把提示词优化到极致，再引入检索增强，前两步走不通后才考虑微调。真正推动微调立项的信号主要有三个：行为和格式仍然不稳定；高频窄口径任务受长提示词的成本与延迟影响；提示词和检索都到位后，准确率距离业务硬指标仍差最后几个点。

开始训练前，我会先搭建评估集，并保证训练样本高质量、口径统一。评估时同时关注专项能力和通用能力，避免窄任务过度微调导致灾难性遗忘。

这个回答覆盖了问题分类、方案优先级、成本判断和上线风险，也更接近真实工业场景中的技术决策方式。

总结

大模型项目中，微调最容易被误用的原因，是团队把所有效果问题都归结为“模型能力不够”。但真正需要先回答的是：模型缺的是事实，还是稳定执行规则的行为？

整套决策可以压缩为三句话：

知识问题优先用 RAG，行为问题才考虑微调；
先提示词，再检索，最后才是微调；
没有明确微调信号、评估集和高质量数据，就不要启动训练。

成熟的工业界方案，往往是：

微调管形态，检索管事实。

但要特别注意：这是一套经过逐级验证后形成的终局组合方案，不是项目一开始就要全部开启的技术清单。

真正专业的选型，不是把所有技术都用上，而是在正确的问题上使用正确的手段。

Classic interview question:How exactly should you set RAG retrieval top-k?

Wed, 08 Jul 2026 22:05:00 +0800

经典面试题：RAG 检索 top-k 到底怎么设？

面试官问：“你们 RAG 系统的 top-k 是怎么设的？” 你如果只回答一句"看情况"，那基本就和 offer 说再见了。top-k 这个看似简单的参数，背后考的是你对完整 RAG 链路的理解深度。本文参考「小哲讲 agent 面试」的拆解思路，把这个高频面试题一次讲透。

一、为什么 top-k 是必考题？

在 RAG（Retrieval-Augmented Generation）系统中，top-k 决定从向量数据库里召回多少条文档片段喂给大模型。它看起来只是 search_kwargs={"k": 4} 这么一个参数，但它同时牵动四件事：

维度	top-k 的影响
召回率（Recall）	k 太小 → 漏掉真正相关的片段
噪声（Noise）	k 太大 → 召回一堆无关片段干扰生成
成本（Cost）	k 越大 → Prompt 越长 → token 费用越高、延迟越高
生成质量（Quality）	噪声片段会让 LLM “迷失在中间”，答非所问

也就是说，top-k 是召回、噪声、成本、质量四个目标的交汇点。面试官真正想听的不是"我设成了 5"，而是"我为什么这样设，背后是什么链路思维"。

二、top-k 设小了：幻觉的温床

很多人凭直觉认为"检索结果越靠前越相关，取 Top-1 不就行了？"——这是第一个坑。

问题在于：排名第一 ≠ 信息完整。

举个真实场景：用户问"公司请假流程是什么？"，向量库里有一份《员工手册》第 3 章详细写了请假流程，但因为切分（chunking）策略，这段内容被切成两半：

片段 A（排名第 1）："……请假需提前在 OA 系统提交申请，由直属主管审批。"
片段 B（排名第 2）：“超过 3 天的请假需提供医院证明，并报 HR 备案。”

如果你只取 Top-1，LLM 拿到的上下文只有"提交申请 + 主管审批"这一半，它大概率会编造“3 天以上需要什么材料”——因为用户问到了，它必须给出回答，但又没看到完整信息。这就是典型的幻觉：模型不是在撒谎，而是被不完整的上下文逼着"脑补"。

核心结论：top-k 设小的代价是召回不足 → 上下文不完整 → 模型幻觉。排名第一只能保证"最相关"，不能保证"最完整"。

三、top-k 设大了：噪声与成本的双重暴击

那干脆设大点，k=20、k=50 总不会漏了吧？这是第二个坑。

k 太大会引入三类问题：

3.1 噪声淹没信号

向量检索本质是相似度排序，但"相似"不等于"相关"。当 k=20 时，后 10 条往往是"看起来有点像但其实没用"的片段。这些噪声片段进入 Prompt 后，会触发 LLM 的 “Lost in the Middle” 现象——模型对上下文中间位置的信息注意力会下降，反而把真正相关的片段忽略了。

3.2 成本线性增长

LLM 的推理成本是按 token 计费的，k 越大，Prompt 越长：

k=4：上下文约 2000 token
k=20：上下文约 10000 token

成本直接 5 倍，延迟也跟着涨。线上高 QPS 场景下，这是真金白银的浪费。

3.3 生成质量下降

更隐蔽的问题是：噪声片段会让模型"分心"。LLM 是基于注意力机制的，当 Prompt 里混入大量无关内容，模型会尝试"综合"这些信息，导致回答跑题、啰嗦，甚至把无关片段当事实引用。

核心结论：top-k 设大的代价是噪声增加 → 成本上升 → 生成质量反而下降。多并不等于好。

四、两阶段检索：粗排 + 精排

既然 k 设小了漏、设大了吵，那有没有两全其美的办法？这就是**两阶段检索（Two-Stage Retrieval）**的价值。

4.1 第一阶段：粗排（Recall）——宁可错杀，不可放过

用什么：双塔 Embedding 模型（如 bge-large-zh、text-embedding-3）
怎么做：query 和 doc 分别编码，算余弦相似度
k 设多大：放宽到 20~50，甚至 100
目标：高召回率，把可能相关的片段都捞进来，哪怕混入噪声

这一步本质是"撒大网"，速度快（向量检索毫秒级），但精度有限。

4.2 第二阶段：精排（Precision）——去伪存真

用什么：Cross-Encoder 重排模型（如 bge-reranker-large、Cohere Rerank）
怎么做：把 query 和每一条候选 doc 拼在一起送进模型，输出相关性分数
取多少：按分数从高到低取 3~5 条 喂给 LLM
目标：高精度，把粗排里的噪声过滤掉，只留真正相关的

这一步是"过筛子"，精度高（Cross-Encoder 能捕捉 query-doc 的细粒度交互），但速度慢，所以只对粗排的少量候选做。

4.3 为什么不能直接用 Cross-Encoder 检索？

因为 Cross-Encoder 没有"索引"概念，必须对库里每条 doc 都跑一次模型，百万级文档库根本扛不住。所以工业界的标准做法就是：

双塔粗排（快但糙）→ Cross-Encoder 精排（慢但准）→ 取 Top-N 喂 LLM

这正是 top-k 在两阶段里的"分工"——粗排的 k 大（保召回），精排后的 k 小（保质量）。面试时把这两个 k 区分开讲，就已经超过 80% 的候选人了。

五、动态阈值：让输入规模自适应

两阶段检索解决了"召回 vs 精度"的矛盾，但还有一个问题：精排后取几条？固定取 3 条还是 5 条？

固定数字依然不够好，因为不同问题的"答案密度"不一样：

简单事实题（“公司全称是什么”）：可能只有 1 条强相关，取 5 条全是噪声
复杂流程题（“请假流程”）：可能需要 4~5 条片段拼起来才完整
开放性问题（“总结公司福利”）：可能需要 8~10 条覆盖不同福利项

这就引出动态阈值（Dynamic Threshold）：

5.1 基于分数的截断

不固定数量，而是设一个相关性分数阈值 θ：

`1`	`保留所有 score >= θ 的片段`

比如精排分数分布是 [0.92, 0.88, 0.85, 0.41, 0.38, ...]，阈值设 0.5，那就只取前 3 条，后边的低分噪声自动丢弃。

5.2 基于分数差距的截断（Drop-off 检测）

观察相邻分数的"断崖"：

`1`	`分数：0.92 → 0.88 → 0.85 → 0.41（断崖！）`

在分数急剧下降的位置截断，能自适应地保留"高分组"而丢弃"低分组"。

5.3 加上上下文长度约束

当然，动态阈值还要和 LLM 的上下文窗口配合——总不能召回 50 条全塞进去。实际工程里通常是：

`1`	`最终输入 = min(动态阈值筛出的片段, 上下文长度预算)`

核心结论：动态阈值让"取几条"从静态参数变成自适应决策，这是 top-k 的高阶玩法，也是面试加分项。

六、完整链路：top-k 不是参数，是思维

把前面几节串起来，你会看到 top-k 背后其实是一条完整的链路：

用户 Query
   ↓
[Query 改写]（可选：HyDE / Multi-Query / Step-back）
   ↓
[粗排] 双塔 Embedding 检索 → top-k₁ = 20~50（保召回）
   ↓
[精排] Cross-Encoder Rerank → 重排打分
   ↓
[截断] 动态阈值 / Drop-off 检测 → top-k₂ = 3~5（保质量）
   ↓
[上下文压缩]（可选：LongLLMLingua）
   ↓
[Prompt 拼装] Context + Question
   ↓
[LLM 生成] → 可溯源回答

每个环节都有它的"k"：

k₁（粗排）：大，保召回
k₂（精排后）：小，保质量
k₃（最终输入）：受动态阈值和上下文预算双重约束

面试官听到你能区分 k₁、k₂、k₃，并解释清楚每一步的目标，基本就认定你真正做过 RAG 工程，而不是只会调 LangChain 的 search_kwargs={"k": 4}。

七、面试高分回答模板

如果被问到"top-k 怎么设"，可以这样组织回答（约 90 秒）：

第一层：先承认 trade-off（10 秒）

“top-k 不是孤立参数，它同时影响召回率、噪声、成本和生成质量，设小了会漏、设大了会吵，所以不能一刀切。”

第二层：讲两阶段检索（40 秒）

“我们的做法是两阶段检索。粗排用双塔 Embedding，k 设到 20~50，优先保召回；精排用 Cross-Encoder Rerank，比如 bge-reranker-large，对粗排结果重打分。Cross-Encoder 精度高但慢，所以只对少量候选做。”

第三层：讲动态阈值（25 秒）

“精排后不固定取几条，而是用动态阈值——设一个相关性分数下限，或者检测分数 drop-off 的断崖位置，自适应截断。再叠加 LLM 上下文长度预算做上限保护。”

第四层：回到本质（15 秒）

“本质上 top-k 考的不是调参，而是对 RAG 链路的理解——召回、排序、截断、生成，每个阶段都有它的 k。区分清楚 k₁、k₂、k₃，效果就稳了。”

八、常见追问与应对

面试官追问	参考回答要点
“粗排 k 设 50 不会太慢吗？”	向量检索是 ANN（近似最近邻），百万级库毫秒级，HNSW/IVF 索引加持下没问题
“Rerank 模型怎么选？”	中文用 `bge-reranker-large`，英文用 Cohere Rerank；轻量场景可用 `bge-reranker-base`
“动态阈值怎么调？”	离线评估集上扫阈值，看 Recall@k 和 Precision@k 的 F1 拐点；或用分数分布的肘部
“为什么不直接用长上下文模型塞进去？”	长上下文有 Lost-in-the-Middle、成本高、不可溯源三个问题，RAG 仍然是更经济的方案
“k 设大了模型答非所问怎么办？”	先加 Rerank，再加动态阈值；必要时上 LongLLMLingua 做上下文压缩
“如何评估 top-k 设得对不对？”	用 Ragas / TruLens 做 Faithfulness、Context Recall、Answer Relevancy 三项指标评估

总结

回到那个面试题——“RAG 检索的 top-k 怎么设？”

低分回答：“看情况，一般设 3~5。”

及格回答：“设小了会幻觉，设大了会引入噪声，所以要在召回和精度之间权衡。”

高分回答：

top-k 不是单一参数，而是贯穿 RAG 链路的协同设计。粗排用双塔 Embedding 大 k 保召回（20~50），精排用 Cross-Encoder 重排提精度，再用动态阈值自适应截断到 3~5 条，最后受 LLM 上下文预算约束。本质考的不是调参，而是对召回 → 排序 → 截断 → 生成整条链路的理解。

记住视频里那句点睛之笔：

“top-k 的本质是完整 RAG 链路思维，而非调参数。”

把这句话在面试里讲出来，配合上面的链路图，这道题基本就稳了。

本文参考抖音「小哲讲 agent 面试」的视频内容整理扩展，原视频链接见文末。封面图由作者自行添加。

RAG Fully Explained:From "Closed-Book" to "Open-Book" LLMs

Wed, 08 Jul 2026 15:00:00 +0800

RAG 技术全景：让大模型从"闭卷考试"到"开卷考试"

大模型很强，但它会"一本正经地胡说八道"。RAG（Retrieval-Augmented Generation，检索增强生成）给大模型外挂了一个"知识库"，让它从闭卷考试变成开卷考试，回答既准又能溯源。本文用 6 张图带你完整搞懂 RAG。

一、先聊痛点：大模型的"幻觉"从哪来？

你大概遇到过这样的场景：问 GPT 一个公司内部的政策问题，或者让它总结一份最新发布的财报，它信誓旦旦地给你一段听起来很专业的回答——但仔细一查，全是编的。

这就是大模型的幻觉（Hallucination）。它不是故意的，而是由其工作方式决定的：LLM 本质上是在做"下一个 token 预测"，它根据训练时记住的概率分布生成文本，分不清"知道"和"不知道"。

原生 LLM 在企业级场景下有四个硬伤：

知识截止：训练数据有截止日期，不知道今天发生的事
幻觉问题：没见过的知识就靠"编"，还很自信
领域知识不足：对企业内部文档、专业细分领域不熟悉
无法溯源：回答不知道来自哪里，没法核验

要解决这些问题，传统思路是微调（Fine-tuning）——用领域数据重新训练模型。但微调又贵又慢，而且知识一更新就得再来一遍。

RAG 给了另一条路：不重新训练模型，而是给它外挂一个可随时更新的知识库，回答时先去知识库里检索，再基于检索到的内容作答。

就像考试一样：原生 LLM 是闭卷考试，全凭记忆，记不清就编；RAG 是开卷考试，遇到不会的可以翻书，答案有据可循。

二、RAG 是什么：核心思想一句话

RAG = 检索（Retrieval）+ 增强（Augmentation）+ 生成（Generation）

在 LLM 生成回答之前，先从外部知识库中检索相关资料，把这些资料增强到 Prompt 里作为上下文，再让 LLM 基于这些上下文生成回答。

用一句话概括：RAG 让 LLM 从"闭卷考试"变成"开卷考试"。

它和微调的根本区别在于：微调把知识"烧进"模型权重里，RAG 把知识放在模型之外的数据库里，用的时候现查。这也决定了两者的适用边界——后面会详细对比。

三、RAG 工作流程：三阶段详解

RAG 的完整流程可以拆成离线建库和在线问答两大阶段。在线问答又分为检索、增强、生成三步。

3.1 离线阶段：把文档变成可检索的向量库

这一步是"建图书馆"的过程，通常一次性完成，知识更新时增量重建：

文档加载：读取 PDF、Word、HTML、Markdown 等原始文档
切分（Chunking）：把长文档切成小段。为什么不整篇存？因为检索时要找最相关的片段，整篇文档向量会"稀释"语义；而且 LLM 上下文有限，塞不下整篇
向量化（Embedding）：用嵌入模型把每段文本转成一个高维向量（通常 768~1536 维），语义相近的文本在向量空间里距离也近
存入向量数据库：把向量和原文一起存进去，建好索引，供后续快速检索

3.2 在线阶段：检索 → 增强 → 生成

用户每一次提问，都会实时跑这三步：

① 检索 Retrieval

把用户问题用同一个 Embedding 模型转成向量
在向量数据库中做相似度检索（通常用余弦相似度），召回 Top-K（一般 3~10 条）最相关的文档片段

② 增强 Augmentation

把检索到的片段拼到 Prompt 模板里，作为上下文
一个典型的 Prompt 长这样：

你是一个严谨的问答助手，请根据以下参考资料回答用户问题。
如果资料中没有答案，请回答"根据现有资料无法回答"，不要编造。

【参考资料】
{检索到的文档片段1}
{检索到的文档片段2}
...

【用户问题】
{用户的问题}

③ 生成 Generation

把拼好的 Prompt 喂给 LLM，生成最终回答
可以要求模型在回答中标注引用来源，实现可溯源

四、完整架构图

这张图把离线和在线两条流水线放在了一起。注意几个关键点：

Embedding 模型必须一致：离线建库用哪个模型，在线检索时也要用同一个，否则向量空间不兼容
知识更新无需重训模型：新增文档只需走一遍"切分 → 向量化 → 入库"，模型权重完全不动
检索质量是 RAG 的命门：如果第一步召回的片段就不相关，后面 LLM 再强也答不好

五、核心组件选型

RAG 系统由四个核心组件构成，选型直接影响效果：

组件	作用	常见选型
Embedding 模型	文本转向量，决定语义匹配上限	BGE（智源）、M3E、OpenAI text-embedding-3、Cohere、bge-m3
向量数据库	存储与检索向量	Milvus、Qdrant、Chroma、Pinecone、FAISS、Weaviate
LLM	基于上下文生成回答	GPT-4、Claude、Qwen、DeepSeek、Llama、GLM
编排框架	串联整个流程	LangChain、LlamaIndex、Haystack、DSPy

几个选型经验：

中文场景：Embedding 优先考虑 bge-large-zh 或 bge-m3，比 OpenAI 的英文模型在中文检索上更准
轻量起步：向量库用 Chroma（纯 Python，单机嵌入式）跑通流程，量大了再换 Milvus / Qdrant
框架选择：快速原型用 LangChain，文档密集型场景用 LlamaIndex（它对文档处理和索引抽象更细致）

六、进阶技术：从"能用"到"好用"

朴素 RAG（一次向量检索 + 拼 Prompt）能跑通，但效果往往一般。真正生产级的 RAG 通常会在四个方向上做增强：

6.1 Query 改写（让问题更适合检索）

用户的问题往往是口语化、模糊的，直接拿去检索效果差。常见技巧：

Query Rewriting：让 LLM 把"那个东西怎么用"改写成"产品 X 的使用方法"
HyDE（Hypothetical Document Embeddings）：先让 LLM 生成一个假设性答案，再用这个答案的向量去检索——因为"答案"和"答案"的语义比"问题"和"答案"更接近
Multi-Query：把一个问题拆成多个子问题分别检索，结果合并去重
Step-back Prompting：先抽象出上位概念再检索，比如"2024 年诺贝尔物理学奖得主"先退到"诺贝尔物理学奖近年得主"

6.2 混合检索 + Rerank（多路召回，精排提纯）

单靠向量检索有个问题：它擅长语义匹配，但对专有名词、产品型号、人名这种精确字面匹配反而不如传统检索。所以生产系统通常用混合检索：

向量检索（语义）+ BM25 关键词检索（字面）并行召回
用 RRF（Reciprocal Rank Fusion） 融合两路结果
再用 Cross-Encoder 重排模型（如 bge-reranker-large、Cohere Rerank）对召回结果做精排
最后取 Top-K 喂给 LLM

Rerank 的关键区别：Embedding 是把 query 和 doc 分别编码再算相似度（双塔），而 Cross-Encoder 是把 query 和 doc 拼一起送进模型（交叉编码），精度高得多，但速度慢——所以先用向量检索粗筛，再用 Rerank 精排，兼顾速度和精度。

6.3 GraphRAG：基于知识图谱的全局检索

微软 2024 年提出的 GraphRAG 解决的是朴素 RAG 的另一个痛点：局部性。向量检索只能召回和问题相似的片段，回答不了"总结整个文档集的核心主题"这种全局性问题。

GraphRAG 的做法：

用 LLM 从文档中抽取实体和关系，构建知识图谱
用社区检测算法（如 Leiden）把图谱分成主题层次化的社区
为每个社区生成摘要
查询时根据问题粒度选择社区摘要或具体节点回答

它特别适合"全局性总结"“跨文档推理"类问题，代价是建图成本高、更新麻烦。

6.4 Self-RAG / 自适应 RAG：让模型自己决定要不要检索

不是每个问题都需要检索——“1+1 等于几"还去查库就浪费了。Self-RAG 让模型学会自我反思：

是否检索：判断问题是否需要外部知识，简单问题直接答
检索结果是否相关：过滤掉无关片段
生成是否忠于检索：检查答案有没有脱离资料编造

好处是省 token、降延迟，坏处是需要训练模型学会这些判断（或用 prompt 模拟），实现复杂度高。

七、方案对比：RAG vs 微调 vs 长上下文

随着模型上下文窗口越来越长（Claude 200K、Gemini 1M、GPT-4o 128K），有人质疑 RAG 是不是过时了——直接把所有文档塞进上下文不就行了？

实际并非如此，三者各有定位：

长上下文：贵（按 token 计费）、易"迷失在中间”（Lost in the Middle 现象）、不可溯源。适合一次性分析单份长文档
RAG：经济、可扩展到海量文档、可溯源。适合知识库问答
微调：改变模型行为/风格，而非注入事实知识。适合固定格式输出、领域术语适配
RAG + 微调：效果最好，成本最高，用于高精度专业场景

经验法则：知识频繁变化 → RAG；风格/格式固定 → 微调；两者都要 → 结合用。

八、实战：用 LangChain 跑通一个最小 RAG

理论讲完，来看一段能跑的代码。用 LangChain + Chroma + OpenAI 实现 30 行的 RAG：

from langchain_community.document_loaders import TextDocumentLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# ① 离线建库：加载 → 切分 → 向量化 → 入库
docs = TextDocumentLoader("knowledge.txt").load()
chunks = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50      # 500 字一段，重叠 50 字
).split_documents(docs)
vectorstore = Chroma.from_documents(
    chunks, OpenAIEmbeddings(model="text-embedding-3-small")
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})  # 召回 Top-4

# ② 在线问答：检索 → 增强 → 生成
prompt = ChatPromptTemplate.from_template("""
你是严谨的问答助手，根据以下资料回答问题。资料中没有答案就说"无法回答"，不要编造。

【资料】
{context}

【问题】
{question}
""")

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | ChatOpenAI(model="gpt-4o-mini", temperature=0)
    | StrOutputParser()
)

# ③ 提问
print(rag_chain.invoke("公司请假流程是什么？"))

这段代码虽然简单，但已经包含了 RAG 的所有核心环节。生产化时再把 Embedding 换成 BGE、加上 Rerank、换成 Milvus，骨架不变。

九、常见挑战与优化方向

实际落地 RAG，踩坑往往不在模型，而在检索。下面是常见问题对应的优化手段：

症状	根因	优化方向
检索结果不相关	Query 与文档语义差距大	Query 改写、换更强的 Embedding
召回准确但答非所问	片段顺序/噪声干扰	加 Rerank 精排、调 Top-K
答案不忠于资料	LLM 忽略上下文编造	Prompt 加约束、降 temperature、强制引用
专有名词召回差	纯向量检索短板	加 BM25 混合检索
上下文超长	召回太多塞不下	上下文压缩、LongLLMLingua、换长上下文模型
知识更新延迟	索引重建慢	增量索引、流式更新
答案无法溯源	未保留片段来源	检索结果带 metadata，Prompt 要求标注引用

一个经验：80% 的 RAG 优化工作都在检索阶段，而不是生成阶段。先把检索做准，效果立刻提升一大截。

十、典型应用场景

企业知识问答：基于内部 Wiki、手册、合同的智能客服与员工助手
法律/医疗咨询：基于法条、判例、药品说明、病历的精准问答
代码助手：基于私有代码库的代码生成（Cursor、Copilot Workspace 都用了 RAG 思想）
教育辅导：基于教材的个性化答疑，答案可指向课本页码
智能搜索：搜索引擎的 AI 摘要（Perplexity、Bing Chat、Google AI Overviews）
数据分析：Text-to-SQL，基于数据库 schema 的自然语言查询

总结

RAG 不是某个具体工具，而是一种 “把外部知识检索和大模型生成结合起来” 的架构思想。它的价值在于：让大模型能用最新、最准、可溯源的外部知识来回答问题，而无需重新训练。

回顾本文要点：

核心思想：检索 → 增强 → 生成，让 LLM 从闭卷变开卷
两大阶段：离线建库（切分+向量化+入库）+ 在线问答（检索+增强+生成）
命门在检索：80% 的优化工作都在检索阶段
进阶四件套：Query 改写、混合检索+Rerank、GraphRAG、Self-RAG
方案选择：知识频繁变化用 RAG，风格固定用微调，高精度场景结合使用

随着模型上下文窗口变长、多模态能力增强，RAG 的形态还在演进（比如多模态 RAG、Agentic RAG），但"外挂知识库"这个核心思想短期内不会过时——毕竟，开卷考试永远比背书更可靠。

如果你对其中某个环节（比如向量数据库选型、Embedding 微调、GraphRAG 实现）想深入，欢迎在评论区交流。