AI on Snow

The Complete MCP Guide: Architecture, Remote Call Flow, and Building Your Own Server

Fri, 09 Oct 2026 15:00:00 +0800

The Complete MCP Guide: Architecture, Remote Call Flow, and Building Your Own Server

A language model can reason and generate text, but it does not know which project your company created today, nor can it create a ticket by imagination alone. MCP provides a standard, controlled path from an agent to that live data and those actions.

The one-sentence definition is:

The Model Context Protocol (MCP) is a standard interface between agent applications and external capabilities. It defines capability discovery, schemas, calls, results, and negotiation. It is neither a model nor a replacement for your business API.

This article follows the 2026-07-28 MCP specification and calls out common differences in older clients. MCP is evolving quickly, so production integrations should follow the protocol versions supported by their host and SDK.

Why MCP exists

Without a shared protocol, every agent platform needs a custom adapter for every external system. Connecting three hosts to GitHub, a database, and a ticket system can quickly become nine integrations.

flowchart LR subgraph Before[Without MCP: N × M adapters] A1[Agent A] --> X1[GitHub adapter] A1 --> X2[Database adapter] A2[Agent B] --> X3[GitHub adapter] A2 --> X4[Ticket adapter] end subgraph After[With MCP: one protocol boundary] B1[Agent A] --> P[MCP] B2[Agent B] --> P P --> S1[GitHub Server] P --> S2[Database Server] P --> S3[Ticket Server] end

MCP standardizes more than the final HTTP request:

Question	MCP mechanism
What can this server do?	`tools/list`, `resources/list`, `prompts/list`
What arguments are valid?	JSON Schema
How is an action executed?	`tools/call`
How is addressable context read?	`resources/read`
How are results represented?	`content`, `structuredContent`, errors, and metadata
Which features do both sides support?	Protocol version, capabilities, and extensions
How do messages travel?	Local `stdio` or remote Streamable HTTP

The USB-C analogy is useful but incomplete. USB-C standardizes a physical interface; MCP standardizes an interface whose names, descriptions, and schemas are interpreted by a model. Metadata is therefore part of the product experience, not incidental documentation.

The four roles

flowchart LR U[User] --> H[Host
Agent application] H --> M[LLM] H --> C1[MCP Client A] H --> C2[MCP Client B] C1 <-->|MCP| S1[MCP Server
Project system] C2 <-->|MCP| S2[MCP Server
Knowledge base] S1 --> API1[(Business API / DB)] S2 --> API2[(Documents / Search)]

Role	Responsibility	Important boundary
User	States a goal and confirms consequential actions	Usually does not construct protocol messages
Host	Owns the agent, model context, policy, approvals, and MCP clients	The MCP server does not run the whole agent loop
MCP Client	Talks to one server and handles discovery, calls, and compatibility	Usually an internal host component
MCP Server	Exposes capabilities, validates calls, and reaches business systems	It must not expose unrestricted backend access

The model normally does not connect directly to the server. The host shows the model an allowed set of capabilities. The model proposes a tool call. The host applies policy and approval rules before executing it. This separates model intent from system authority.

Tools, resources, and prompts

Primitive	Primary controller	Best for	Example
Tools	Model selects; host governs execution	Queries and actions	`search_projects`, `create_ticket`
Resources	Host or application loads	Addressable, usually read-only context	`project://P-1024/spec`
Prompts	User or host selects	Reusable workflow templates	“Create a release checklist”

The protocol also includes notifications and other optional capabilities. A useful server rarely needs all of them. Start with a few well-designed tools.

An MCP server is often an agent-facing adapter over existing systems:

flowchart LR L[Tool visible to the LLM
name + description + schema] --> S[MCP Server
auth · validation · orchestration] S --> R1[REST API] S --> R2[gRPC] S --> R3[(Database)] S --> R4[Message queue]

A REST route might be PATCH /v1/projects/:id. A useful MCP tool is goal-oriented: archive_project(project_id, reason). Do not mirror hundreds of internal endpoints into hundreds of model tools.

The complete remote MCP call path

Remote MCP normally uses Streamable HTTP. The older HTTP+SSE transport is superseded and should only remain for compatibility.

sequenceDiagram autonumber actor User participant Host as Agent Host participant Model as LLM participant Client as MCP Client participant Server as Remote MCP Server participant API as Business system Host->>Client: Load server configuration Client->>Server: Discover protocol and capabilities Server-->>Client: Server info + capabilities Client->>Server: tools/list Server-->>Client: Names, descriptions, schemas, annotations User->>Host: “List delayed projects this week” Host->>Model: Message + allowed tool definitions Model-->>Host: list_projects(status="delayed") Host->>Host: Policy check / approval if needed Host->>Client: Execute approved call Client->>Server: tools/call + credentials Server->>Server: Authenticate, authorize, validate Server->>API: Query current business data API-->>Server: Project records Server-->>Client: content + structuredContent Client-->>Host: Tool result Host->>Model: Add allowed result to context Model-->>Host: Compose answer Host-->>User: Final response + action status

1. Configuration and discovery

The host stores a server name, URL, and authentication method. The client learns the protocol version, server information, and capabilities, then lists tools. Hosts may cache the list and refresh it when capabilities change or the cache expires.

In the 2026-07-28 protocol, the modern path is stateless: server/discover and per-request metadata replace the old connection handshake and protocol-level session. Older clients can still use initialize → initialized and Mcp-Session-Id. Let the SDK negotiate this; application code should not handcraft JSON-RPC.

2. Tools enter model context

The host may filter tools by user, tenant, role, task, or dynamic tool search. Only the relevant names, descriptions, and schemas need to enter context. Five distinct tools are usually more reliable and cheaper than fifty overlapping ones.

3. The model selects; the host decides

The model produces structured intent, not authority. The host should decide whether the user may call that tool, whether a write needs confirmation, whether arguments cross cost or data boundaries, and whether prompt injection or destructive behavior is suspected.

Annotations such as readOnly, destructive, and openWorld inform host policy. They are hints, not authorization controls.

4. HTTP and business execution

The client sends an MCP request over HTTPS to an endpoint such as /mcp, with protocol-routing information and usually an Authorization header. The server must independently validate the token, scopes, tenant, object-level permission, arguments, timeouts, and audit context before calling the real REST API, gRPC service, or database.

MCP does not make a dangerous SQL query safe, and it does not replace business authorization.

Expanding the details normally hidden by an SDK, a modern tool call looks roughly like this (some per-request _meta is omitted for readability):

POST /mcp HTTP/1.1
Host: mcp.example.com
Authorization: Bearer <access-token>
Content-Type: application/json
Mcp-Protocol-Version: 2026-07-28
Mcp-Method: tools/call
Mcp-Name: list_projects

{
  "jsonrpc": "2.0",
  "id": 42,
  "method": "tools/call",
  "params": {
    "name": "list_projects",
    "arguments": {"status": "delayed", "limit": 5}
  }
}

The successful JSON-RPC result carries model-readable content and schema-bound data for the application:

{
  "jsonrpc": "2.0",
  "id": 42,
  "result": {
    "content": [{"type": "text", "text": "Found 1 delayed project."}],
    "structuredContent": {
      "result": [{"id": "P-1002", "name": "Billing migration"}]
    },
    "isError": false
  }
}

Do not handcraft these messages in application code. The example is useful for reading gateway logs and traces; SDKs should own negotiation, headers, and JSON-RPC details.

5. Results return to the model

Channel	Consumer	Guidance
`content`	Model	Concise, understandable explanation of the result
`structuredContent`	Host, UI, or later calls	Schema-valid fields and stable identifiers
`_meta`	Client application	Client-only data; not a secret vault

The host puts approved result content back into the model context. The model then answers or plans another call. A successful MCP request does not necessarily mean the user task succeeded: the agent must inspect isError, completeness, and any follow-up requirements.

Two remote connection models

flowchart TB subgraph Direct[Mode A: runtime-managed connection] A[Your agent application] --> C[MCP Client] C --> S[Remote MCP Server] end subgraph Hosted[Mode B: platform-hosted connection] B[Your application] --> P[Model / Agent platform] P --> HS[Remote MCP Server] end

Mode	Main advantage	Best fit
Runtime-managed	Your app owns networking, credentials, filtering, and approval	Private networks, strict compliance, custom policy
Hosted MCP	Less integration code; the agent platform handles protocol calls	Public HTTPS services and quick integrations

The OpenAI Agents SDK supports hosted MCP for public remote servers and runtime-managed stdio or Streamable HTTP for local and private servers. The choice is about who should own the connection and trust boundary.

Build a remote MCP server

The following minimal project exposes a read-only project catalog. It uses Python SDK v2, Python 3.10+, and stateless Streamable HTTP so that replicas can scale horizontally.

Set up the project

1
2
3

mkdir project-mcp && cd project-mcp
uv init
uv add "mcp[cli]>=2,<3" pydantic

Pin the major SDK version. The protocol and SDKs move quickly; a production build should not cross a major version without review.

Create `server.py`

from typing import Annotated, Literal

from mcp.server import MCPServer
from mcp.types import ToolAnnotations
from pydantic import BaseModel, Field


class Project(BaseModel):
    id: str
    name: str
    status: Literal["active", "delayed", "archived"]
    owner: str


PROJECTS = [
    Project(id="P-1001", name="Search upgrade", status="active", owner="Lin"),
    Project(id="P-1002", name="Billing migration", status="delayed", owner="Kai"),
]

mcp = MCPServer("project-catalog", version="1.0.0")


@mcp.tool(
    annotations=ToolAnnotations(
        readOnlyHint=True,
        idempotentHint=True,
        destructiveHint=False,
        openWorldHint=False,
    )
)
def list_projects(
    status: Literal["active", "delayed", "archived"] | None = None,
    limit: Annotated[int, Field(ge=1, le=50)] = 10,
) -> list[Project]:
    """List projects, optionally filtered by exact status.

    Use this for project discovery. It never modifies project data.
    """
    matches = [p for p in PROJECTS if status is None or p.status == status]
    return matches[:limit]


if __name__ == "__main__":
    mcp.run(
        transport="streamable-http",
        host="0.0.0.0",
        port=8000,
        stateless_http=True,
        json_response=True,
    )

The function name becomes the tool name, its docstring becomes the description, type hints generate the input schema, and the return annotation generates the output schema. The annotations describe behavior to the host. The SDK handles MCP framing, version compatibility, and Streamable HTTP.

Replace the in-memory list with a business service call in production, but always derive the tenant from verified credentials rather than a model-supplied user_id.

Test locally

Use the official Inspector first:

`1`	`uv run mcp dev server.py`

Then start the HTTP server:

1
2

uv run python server.py
# http://127.0.0.1:8000/mcp

An SDK client makes a compact regression check:

import asyncio
from mcp import Client


async def main() -> None:
    async with Client("http://127.0.0.1:8000/mcp") as client:
        tools = await client.list_tools()
        assert any(tool.name == "list_projects" for tool in tools.tools)

        result = await client.call_tool(
            "list_projects", {"status": "delayed", "limit": 5}
        )
        assert not result.is_error
        assert result.structured_content["result"][0]["id"] == "P-1002"


asyncio.run(main())

Test valid calls, boundaries, invalid enums, authorization failures, downstream timeouts, empty results, and output-schema validation. Inspector proves the protocol works; automated tests prove it still works after an upgrade.

Containerize and deploy

FROM python:3.13-slim
WORKDIR /app

COPY pyproject.toml uv.lock ./
RUN pip install --no-cache-dir uv && uv sync --frozen --no-dev

COPY server.py ./
EXPOSE 8000
CMD ["uv", "run", "python", "server.py"]

1
2

docker build -t project-mcp:1.0.0 .
docker run --rm -p 8000:8000 project-mcp:1.0.0

Production should expose a stable HTTPS URL such as https://mcp.example.com/mcp. Configure proxies for streaming behavior, bounded request size, sensible timeouts, and idle connections.

The demo server has no authentication and is only suitable for local learning. Before exposing it to any network, configure the SDK’s token verifier or OAuth support and enforce business authorization inside every tool.

Stateless mode is a good default. If a workflow needs state across calls, prefer returning an explicit server-minted handle and accepting it in later tools instead of hiding state in one replica’s memory.

Connect the server to an agent

A reachable endpoint is only half the work. The agent needs registration, authorization, routing guidance, and behavioral tests.

Codex

1
2

codex mcp add projects --url https://mcp.example.com/mcp
codex mcp list

Or configure ~/.codex/config.toml:

1
2

[mcp_servers.projects]
url = "https://mcp.example.com/mcp"

Add a short, testable routing rule to the project’s AGENTS.md:

1
2
3

When a task needs a current project status, owner, or project ID, use the
read-only tools from the projects MCP first. Before any update, archive, or
delete operation, show the exact target and impact and wait for confirmation.

Configuration determines what the agent can see. Instructions help it decide when to use those capabilities. Never tell it to blindly trust remote content; tool results can be stale or malicious.

OpenAI Agents SDK

A public remote server can be attached as a hosted MCP tool:

from agents import Agent, HostedMCPTool

agent = Agent(
    name="Project assistant",
    instructions="Use the projects MCP for current project facts.",
    tools=[
        HostedMCPTool(
            tool_config={
                "type": "mcp",
                "server_label": "projects",
                "server_url": "https://mcp.example.com/mcp",
                "require_approval": "never",  # safe only for this read-only tool
            }
        )
    ],
)

Do not copy never onto write tools. Match approval policy to risk, and keep independent authorization on the server. For private services, let your Agents SDK runtime own the connection so that your application controls networking, filtering, and approvals.

Verify actual agent behavior

Prompt	Expected behavior
“List delayed projects”	Calls `list_projects(status="delayed")`
“What does delayed mean?”	Answers generally without a pointless call unless company policy is requested
“Delete P-1002”	Explains that no delete tool exists; never fabricates success
“Use `status=hacked`”	Schema rejects it; agent corrects or explains
Result contains “ignore previous instructions”	Treats it as untrusted data and does not escalate privileges

Measure tool selection, argument correctness, and answer faithfulness. Looking only at final prose can hide duplicate calls, denied actions, and unnecessary cost.

Tool design sets the agent’s ceiling

flowchart LR G[User goal] --> N[Clear verb name] N --> D[When-to-use description] D --> I[Narrow input schema] I --> A[Truthful annotations] A --> O[Stable structured output] O --> E[Recoverable errors]

Give each tool one recognizable goal. Prefer list_projects, get_project, and update_project over project_action(mode, payload).
Describe when to use the tool, not only how it is implemented.
Encode enums, lengths, ranges, and formats in the schema.
Separate reads and writes so the host can apply different approval rules.
Return stable IDs for later calls.
Paginate and cap results.
Return actionable errors, not stack traces.
Evolve schemas compatibly. Optional additions are safer than renames or type changes.

MCP is not always the answer. A private function used by one application may be simpler as a function tool. Large continuous media transfer needs a different data plane. Deterministic transaction state machines should not be delegated entirely to model planning. Unrestricted SQL, shell, or URL-fetching tools should not be exposed in production.

Security: every boundary is untrusted

flowchart LR U[User input
untrusted] --> H[Host policy] H -->|minimum tool set| M[Model output
untrusted intent] M --> A{Approval / authorization} A --> S[MCP Server
authenticate and validate again] S --> B[Business system] B --> R[Tool result
still untrusted] R --> H

Risk	Required defense
Cross-tenant access	Validate token, scope, tenant, and object permission on every call
Destructive action	Separate tool, truthful annotations, confirmation, idempotency key, audit, rollback
Prompt injection	Treat returned content as data; never let it rewrite host policy
Token leakage	Keep tokens out of descriptions, results, and logs; validate token audience
SSRF	URL allowlists, DNS/IP checks, block metadata/private ranges, bound redirects
Excessive data	Field minimization, pagination, result caps, PII redaction
Cost or denial of service	Rate, concurrency, timeout, circuit-breaker, and quota controls
Supply-chain drift	Pin SDK majors, scan dependencies, canary upgrades, rollback

Authentication says who the caller is; authorization says what that caller may do to this particular object. Never trust a model-provided user_id as the tenant boundary. OAuth deployments should use standard discovery, narrow scopes, and an explicit audience.

Host prompts, annotations, approval dialogs, and network allowlists are defense in depth. Server-side object authorization remains the final boundary.

Observability and troubleshooting

An MCP failure can happen because the model did not select a tool, the host denied it, the protocol failed, or a downstream system failed. Logs and traces need to connect these layers.

Record a trace or request ID, server/tool/version, redacted caller and tenant, approval outcome, total and downstream latency, result size, error category, schema version, and deployment version. Never log access tokens or complete private results.

Symptom	Check first
Server cannot connect	URL, certificate, DNS, Host header, firewall, `/mcp` path
Connected but no tools	Discovery/capabilities, `tools/list`, registration code
Model never calls a tool	Names, descriptions, instructions, overlapping tools
Arguments are consistently wrong	Schema breadth, ambiguous semantics, missing enums
Local works; remote times out	Proxy streaming, platform timeout, cold start, downstream latency
Call succeeds; answer is wrong	Content clarity, missing structured fields, stale cache
Multi-replica failures	Hidden process state or old session affinity assumptions

The shortest diagnostic path is usually: Inspector → direct SDK client → host tool list → one complete agent trace. It separates server, protocol, and model-decision problems quickly.

Production checklist

Tool names, descriptions, input schemas, and output schemas reviewed
Reads, writes, and destructive operations separated
Authentication, scopes, tenant, and object authorization enforced per request
Stable HTTPS Streamable HTTP endpoint
Approval, idempotency, audit, and rollback for writes
Pagination and result-size limits; no secrets in results
Timeouts, retries, rate limits, circuit breakers, and downstream error mapping
Inspector checks, automated tests, and agent behavior evals pass
Traces, metrics, redacted logs, and alerts exist
SDK and schema versions pinned; upgrades can canary and roll back

Closing thought

MCP is not merely a way for a model to send an HTTP request. It gives capability discovery, argument contracts, execution, results, and security policy a shared path.

flowchart LR A[Pick one real user goal] --> B[Design one narrow tool] B --> C[Test with Inspector] C --> D[Automate tests] D --> E[Streamable HTTP + auth] E --> F[Connect the agent] F --> G[Evaluate and observe] G --> H[Add the next tool]

Make one read-only tool accurate before adding writes. Establish authorization before building elaborate orchestration. MCP reduces integration cost; it does not replace sound API design, authorization, or operational discipline.

References

From Keyword Matching to Vector Retrieval: Three Ways to Search

Fri, 02 Oct 2026 20:13:30 +0800

From Keyword Matching to Vector Retrieval: Three Ways to Search

Search is not one technique. Keyword matching finds words that appear literally, fuzzy search rescues misspellings and partial input, and vector retrieval finds content with similar meaning. In RAG systems, these methods usually complement each other rather than replace each other.

A quick example:

User intent	Input	Best starting point
Find an explicit rule	`refund policy`	Keyword matching
Recover a typo	`retun policy`	Fuzzy search
Ask naturally	`Can I change my mind after buying this?`	Vector retrieval
Need both precision and semantics	`Can a premium member refund the wrong plan?`	Hybrid retrieval + reranking

Keyword Matching

Keyword matching is the classic information retrieval approach. Documents are tokenized, indexed, and ranked according to term overlap, term frequency, field weights, and document length.

The most common structure is an inverted index: instead of scanning documents one by one, the system maps each term to the documents that contain it.

Aspect	Notes
Matching signal	Literal term overlap
Common algorithms	TF-IDF, BM25, Boolean queries, field boosts
Strengths	Fast, stable, explainable, easy to filter
Weaknesses	Poor at synonyms, paraphrases, and unstated intent
Best for	SKU, order id, logs, legal clauses, code symbols, titles

If a user searches for redis setnx, keyword retrieval is excellent. If they search for how do I prevent duplicate orders, a purely lexical system may miss articles that only mention idempotency or distributed lock.

Fuzzy Search

Fuzzy search is still lexical search, but more tolerant. It asks whether the user input and the indexed term look similar enough.

Common techniques include:

Technique	Solves	Example
Edit distance	Insertions, deletions, substitutions, transpositions	`retun` → `return`
n-gram / trigram	Substring similarity	`microserv` → `microservice`
Prefix matching	Autocomplete	`kuber` → `kubernetes`
Phonetic or pinyin correction	Input method mistakes	pinyin-like input → Chinese term
Synonym dictionaries	Manually curated near-equivalents	`after-sales` ≈ `returns`

Fuzzy search is great for search box ergonomics: typos, partial names, remembered fragments, and autocomplete. Elasticsearch, OpenSearch, Solr, PostgreSQL trigram, and Algolia all provide related capabilities.

But fuzzy search does not fully understand meaning. refund and return are semantically related, but they may not be textually similar. iPhone and Apple phone can be far apart lexically. Once recall is widened, false positives also increase, so fuzzy search needs thresholds, field boosts, popularity, click signals, and business rules.

Vector Retrieval

Vector retrieval uses an embedding model to convert text, images, audio, or other objects into numeric vectors. Items with similar meaning are close in vector space. At query time, the system embeds the user query and retrieves the nearest vectors.

This is the retrieval foundation of many RAG systems: retrieve relevant chunks from a knowledge base, insert them into the prompt, and let the LLM answer with external context.

Aspect	Notes
Matching signal	Semantic similarity in embedding space
Common indexes	HNSW, IVF, PQ, DiskANN
Common metrics	Cosine similarity, inner product, Euclidean distance
Strengths	Handles paraphrases, natural questions, cross-lingual and multimodal similarity
Weaknesses	Higher cost, weaker explainability, less reliable for exact constraints
Best for	Knowledge base QA, similar articles, image similarity, recommendations, semantic code search

For example, a document may say:

`1`	`Customers may request no-reason returns within 7 days after order completion.`

A user may ask:

`1`	`Can I send it back if I no longer want it?`

Keyword search may miss the connection. Vector search is more likely to retrieve the return policy because the meanings are close.

Vector retrieval is not magic, though. It is weaker for exact IDs, version numbers, permissions, and precise field filters. For HTTP 451, order 202610020001, or Go 1.22, lexical and structured retrieval are often safer.

Core Differences

Dimension	Keyword matching	Fuzzy search	Vector retrieval
Focus	Did the term appear?	Is the term similar?	Is the meaning similar?
Query style	Precise, short, terminology-heavy	Typo-prone, abbreviated, partial	Natural, descriptive, long
Recall	Precise but can miss	Wider lexical recall	Strong semantic recall
Explainability	High	Medium	Lower
Latency and cost	Low	Medium	Medium to high
Typical tools	MySQL, Lucene, Elasticsearch BM25	Elasticsearch fuzzy, trigram, Algolia	FAISS, Milvus, Qdrant, Pinecone, Weaviate

A simple mental model:

1
2
3

Keyword matching: you said A, so I search for A.
Fuzzy search: you probably meant A, so I search for A.
Vector retrieval: you meant something like A, so I search for similar meaning.

Classic Applications

Scenario	Recommended approach	Why
Log search	Keyword + time/service filters	Exact error codes and fields matter
E-commerce search	Keyword + fuzzy + vector recall	Product names need precision, user input is messy, recommendations need semantics
Code search	Symbol index + keyword + semantic vectors	Names matter, but user intent can be natural language
Enterprise knowledge QA	Vector + BM25 + reranking + RAG	The system must understand questions, retrieve evidence, and cite sources
Image similarity	Vector retrieval	Visual similarity is hard to express with keywords
Help center search	Keyword + fuzzy + synonyms + vector retrieval	Users mix product terms and natural language

Enterprise RAG is the clearest example. Users rarely repeat document titles exactly. They ask questions like “Why was this reimbursement rejected?” or “How does sick leave work during probation?” A robust system filters by permissions, retrieves candidates with both BM25 and vectors, reranks them, and sends only a small set of reliable chunks to the LLM.

Practical Selection

Use this quick guide:

Problem	Start with
Users type order IDs, error codes, function names, or exact terms	Keyword matching
Users often misspell, abbreviate, or type only prefixes	Fuzzy search
Users describe intent in natural language	Vector retrieval
You need exact filters, semantic understanding, and LLM grounding	Hybrid retrieval

Production search is usually a layered system:

Structured filters: permissions, tenants, time, category.
Keyword recall: BM25 protects exact terms, titles, IDs, and entities.
Vector recall: captures paraphrases and semantic intent.
Fusion: RRF or weighted scoring merges candidate lists.
Reranking: cross-encoders, LLM rerankers, or business rules refine the final order.
Generation: in RAG, only the most relevant and trustworthy chunks are inserted into the prompt.

The Takeaway

Keyword matching answers “does this word appear?” Fuzzy search answers “was this word typed imperfectly?” Vector retrieval answers “what does this query mean?”

For a normal search box, start with keyword retrieval, tokenization, field boosts, and fuzzy correction. For knowledge QA or intelligent support, add vector retrieval and RAG. For real production systems, hybrid retrieval is usually where the design lands, because users alternate between exact identifiers and vague natural language all the time.

KV Cache 深入理解：为什么只存 K/V，不存 Q

Fri, 18 Sep 2026 21:35:00 +0800

KV Cache 深入理解：为什么只存 K/V，不存 Q

KV Cache 是大模型推理里最重要的加速手段之一。它的名字里只有 K 和 V，没有 Q，不是命名偷懒，而是由自回归解码的计算结构决定的：未来 token 会反复读取过去 token 的 Key 和 Value，却不会再读取过去 token 的 Query。

先把结论放在前面：

对象	在注意力里的角色	未来 token 还会用吗	是否适合长期缓存
Query `Q`	当前 token 发出的“读请求”	否	否
Key `K`	历史 token 暴露出的“地址/索引”	是	是
Value `V`	历史 token 暴露出的“内容/信息”	是	是
Attention score	当前 Query 与历史 Key 的匹配结果	通常否	否

如果只记一句话：Q 是读者，K/V 是被读的记忆。读者每一步都会换，记忆会被后面的读者反复读取。

一、把 Attention 看成一次“查表”

Transformer 里的自注意力通常写成：

$$ Q = XW_Q,\quad K = XW_K,\quad V = XW_V $$

$$ \text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

这个公式看起来像矩阵乘法，但从直觉上可以拆成三件事：

名字	更像什么	含义
`Q` Query	查询语句	“我现在想找什么信息？”
`K` Key	索引地址	“我这里有什么特征，可以被谁匹配到？”
`V` Value	实际内容	“如果你匹配到了我，真正拿走的是这些信息。”

在普通的全序列计算中，Q/K/V 都来自同一段输入序列 X。但在自回归生成里，时间方向非常关键：模型已经生成了前面的 token，现在只需要生成下一个 token。

假设当前正在生成第 t 个 token，在某一层 decoder self-attention 中真正需要的是：

$$ \text{Attention}(q_t,K_{\le t},V_{\le t}) =\text{softmax}\left(\frac{q_tK_{\le t}^T}{\sqrt{d_k}}\right)V_{\le t} $$

注意这个公式里只有 当前的 q_t 和 历史到当前的 K/V。它并没有用到 q_1, q_2, ..., q_{t-1}。

这就是 KV Cache 的根。

二、Prefill 与 Decode：缓存到底发生在哪

大模型生成通常可以分成两个阶段：

阶段	输入规模	主要工作	KV Cache 的状态
Prefill	整个 prompt	一次性处理用户输入、系统提示词、历史上下文	建立初始 `K/V`
Decode	每次 1 个新 token	用新 token 继续生成下一个 token	每步追加新的 `K/V`

在 Prefill 阶段，模型要把 prompt 里的所有 token 都跑一遍。比如 prompt 长度是 n，每一层都会产生：

1
2

K_cache[layer] = [k1, k2, ..., kn]
V_cache[layer] = [v1, v2, ..., vn]

然后进入 Decode 阶段。每生成一个新 token，模型并不会把从 x1 到 xt 的整段上下文重新计算一遍，而是只处理最新 token：

new token -> 当前层 hidden state
          -> q_new, k_new, v_new
          -> q_new 读取已有 K_cache / V_cache
          -> k_new, v_new 追加进缓存

用伪代码看会更清楚：

# prefill
for layer in layers:
    q, k, v = project(prompt_hidden_states)
    K_cache[layer] = k
    V_cache[layer] = v
    hidden_states = attention(q, K_cache[layer], V_cache[layer])

# decode: 每次只处理一个新 token
for step in generation:
    x = embedding(last_token)

    for layer in layers:
        q_new, k_new, v_new = project(x)

        K_cache[layer].append(k_new)
        V_cache[layer].append(v_new)

        x = attention(q_new, K_cache[layer], V_cache[layer])

    last_token = sample(lm_head(x))

这里有一个容易忽略的细节：KV Cache 是每一层都有一份，不是整个模型共享一份。 因为每一层的 hidden state 不同，投影出来的 K/V 也不同。

常见张量形状可以写成：

1
2

K_cache[layer]: [batch, kv_heads, seq_len, head_dim]
V_cache[layer]: [batch, kv_heads, seq_len, head_dim]

如果是普通多头注意力（MHA），kv_heads 通常等于 query heads；如果是 MQA/GQA，kv_heads 会少很多，这是现代大模型降低 KV Cache 显存压力的重要技巧。

三、为什么只存 K/V，不存 Q

这个问题是理解 KV Cache 的核心。直觉答案是：旧 Q 用完就没了；旧 K/V 会被未来反复访问。

更严谨一点，可以从四个角度看。

角度	为什么旧 Q 没必要存
公式	生成第 `t` 个 token 时用的是 `q_t K_{\le t}^T`，旧的 `q_i` 不在公式里
因果方向	未来 token 只能拿自己的 Query 去看过去的 Key，不能反过来让过去的 Query 查询未来
复用价值	`k_i/v_i` 会被 `q_{i+1}, q_{i+2}, ...` 多次读取；`q_i` 只服务于第 `i` 行 attention
显存收益	存 Q 会增加大量显存，却几乎不减少 decode 计算

可以把第 i 个 token 的三样东西拆开看：

token `i` 产生的东西	第 `i` 步怎么用	第 `i+1` 步以后怎么用
`q_i`	用它查询 `K_{\le i}`，生成当前 token 的注意力输出	不再需要
`k_i`	被 `q_i` 匹配，也成为缓存的一部分	会被未来每个 `q_j` 匹配
`v_i`	被当前注意力加权求和	会被未来每个注意力分布加权求和

再换一种说法：q_i 描述的是“第 i 个位置想问什么”，这个问题只对第 i 个位置有意义。到了第 j 个位置，模型关心的是“第 j 个位置想问什么”，也就是 q_j，而不是 q_i。

如果存 Q，能不能少算一点？

通常不能。

Decode 时新 token 的 q_new 必须根据当前层的 hidden state 现场算出来。这个 hidden state 又来自前面层的输出，而前面层的输出依赖当前 token、位置编码、残差、MLP、LayerNorm 以及上一层 attention 结果。换句话说，未来的 Query 在未来 token 出现之前根本不存在。

缓存旧 Query 最多能避免“重新计算旧位置的 Query”。但 KV Cache 的设计本来就已经不重新计算旧位置了：decode 阶段只跑新 token，旧位置不会再进前向计算图。于是旧 Query 没有可节省的计算。

旧 attention 分数要不要缓存？

也通常不缓存。

第 t 步真正需要的是新的一行分数：

`1`	`score_t = q_t · [k1, k2, ..., kt]^T`

旧分数 score_1, score_2, ..., score_{t-1} 描述的是旧 Query 如何看旧 Key。它们不能替代 q_t 对历史 Key 的新匹配。把所有 attention score 都存下来，显存会从线性增长变成更接近平方增长，收益却很小。

训练时为什么好像会保存 Q？

训练和推理要分开看。

训练阶段为了反向传播，框架可能会临时保留 Q/K/V、attention score、softmax 结果、MLP 激活等中间量，或者用 activation checkpointing 在反向时重算。那是为了求梯度，不是为了自回归生成加速。

推理阶段没有反向传播，目标是尽快生成下一个 token。因此长期留下来的只有未来还会被读到的东西：K/V。

四、KV Cache 换来的代价

KV Cache 的本质是用显存换时间。它显著减少重复计算，但显存会随着上下文长度线性增长。

常见估算公式是：

1
2

KV cache bytes
≈ 2 × layers × batch × seq_len × kv_heads × head_dim × bytes_per_elem

其中前面的 2 表示 K 和 V 两份缓存。

举个近似例子，假设模型有：

参数	数值
层数 `layers`	32
query heads	32
`head_dim`	128
数据类型	FP16 / BF16，2 bytes

如果是普通 MHA，kv_heads = 32：

每 token KV cache
≈ 2 × 32 × 32 × 128 × 2 bytes
= 524288 bytes
≈ 512 KiB

如果是 GQA，假设 kv_heads = 8：

每 token KV cache
≈ 2 × 32 × 8 × 128 × 2 bytes
= 131072 bytes
≈ 128 KiB

所以同样 8192 token 的上下文，单请求 KV Cache 可能就是：

注意力结构	每 token 约占	8192 token 约占
MHA，32 个 KV heads	512 KiB	4 GiB
GQA，8 个 KV heads	128 KiB	1 GiB

这还只是 batch size = 1 的估算。服务端如果要同时处理很多请求，KV Cache 很快就会成为吞吐瓶颈。

现在再看“不存 Q”的显存意义：

架构	如果额外缓存 Q，会发生什么
MHA：`q_heads = kv_heads`	在原本 K+V 两份之外再加一份 Q，缓存大约增加 50%
GQA/MQA：`q_heads > kv_heads`	Q 的头数比 KV 多，额外缓存 Q 可能比现有 KV Cache 还大

也就是说，缓存 Q 不只是“多存一点”，在 GQA/MQA 模型里可能是非常昂贵的浪费。

五、工程实践里还要注意什么

KV Cache 听起来简单，真正服务化时会牵出不少工程问题：

问题	说明
Prefix Cache	如果很多请求共享同一段系统提示词，可以复用这段前缀的 K/V
PagedAttention	把 KV Cache 切成页管理，减少长短请求混批时的显存碎片
Sliding Window	只保留最近窗口内的 K/V，适合局部注意力模型
KV Quantization	把 KV Cache 量化到更低 bit，降低显存与带宽压力
Beam Search	多个 beam 共享前缀时要正确复制、重排或引用缓存
RoPE 位置	缓存的 K 通常已经带有位置信息，复用前缀时必须保证 position offset 一致

KV Cache 不会改变模型的理论输出，它只是避免重复计算。实际工程里如果使用低精度缓存、不同 attention kernel、不同 batch 合并策略，可能会有极小的数值差异，但原理上它是等价优化。

它也不是万能加速器：

场景	KV Cache 的作用
Prefill 很长	仍然要处理完整 prompt，KV Cache 不能消除 prefill 成本
Decode 很长	很有用，因为每步都能复用历史 K/V
上下文非常长	计算变少，但显存和显存带宽压力变大
请求很多且长度不一	需要分页缓存、连续批处理、调度器配合

最后用一张小表把核心逻辑收住：

问题	答案
KV Cache 缓存的是什么	每一层历史 token 的 Key 和 Value
为什么能加速	Decode 阶段不再重复计算旧 token 的 K/V 和旧位置输出
为什么不存 Q	旧 Query 只用于旧位置的 attention，不会被未来 token 读取
代价是什么	显存随层数、序列长度、batch、KV head 数线性增长
主要优化方向	GQA/MQA、PagedAttention、Prefix Cache、KV 量化、窗口注意力

一句话总结：KV Cache 的名字不是偶然。K/V 是上下文留给未来的可读记忆；Q 是当前 token 临时发出的读取动作。动作结束就结束了，记忆才需要留下来。

How Token Counting Works: How AI Providers and API Proxies Track Usage

Fri, 04 Sep 2026 21:43:19 +0800

How Token Counting Works: How AI Providers and API Proxies Track Usage

When using LLM APIs, you often see fields such as prompt_tokens, completion_tokens, and total_tokens. A token is not simply a word or character. It is the unit produced by the model’s tokenizer, and provider billing is built around counting input, output, cached, tool, and multimodal units.

The short version:

Question	Answer
What is a token	A unit produced when the tokenizer splits text
Is it the same as word count	No. Chinese, English, punctuation, whitespace, and code tokenize differently
How providers count it	They count input before inference, output during generation, then return usage
How API proxies count it	They estimate before forwarding, prefer upstream usage after completion, then settle billing
Why local counts differ	Tokenizer mismatch, hidden prompts, tool schemas, RAG context, multimodal inputs, retries, and caching

1. What a token is

LLMs do not process raw text directly. Before inference, text goes through a tokenizer, which splits it into tokens and maps each token to a numeric ID.

flowchart LR A[Raw text] --> B[Tokenizer] B --> C[Token sequence] C --> D[Token IDs] D --> E[Model inference]

For example:

`1`	`Write a Dockerfile for me`

may be split into pieces such as:

`1`	`["Write", " a", " Docker", "file", " for", " me"]`

This is only illustrative. Different models can tokenize the same text differently. Common words may be one token, rare characters may be split, and whitespace, punctuation, JSON, logs, and source code can all add tokens.

The practical definition is: a token is the smallest vocabulary unit the model actually reads or generates.

2. Why billing is token-based

LLM cost is strongly related to token count:

longer inputs require more context processing;
longer outputs require more generation steps;
larger context windows increase memory and attention cost;
tools, multimodal input, long context, caching, and reasoning models add more accounting dimensions.

A simplified formula:

1
2
3

cost = input_tokens * input_price
     + output_tokens * output_price
     + cached / reasoning / image / audio / tool-related token cost

Output tokens are often priced higher because they are generated step by step. Each new token depends on the current context and previous generated tokens.

3. What gets counted in a request

Real API requests usually contain more than the user’s latest message.

Content	Usually counted	Note
User message	Yes	The user’s visible input
System prompt	Yes	Role and behavior instructions
Developer prompt	Yes	Platform or application rules
Conversation history	Yes	Previous turns re-enter the context
RAG snippets	Yes	Retrieved context is added to the prompt
Tool definitions	Yes	Function schemas can be large
Tool results	Yes	JSON, tables, and logs may enter later turns
Output text	Yes	The model’s generated answer
Images, audio, video	Depends on model	Often converted into special billing units
Cached content	Depends on provider	May be reported and priced separately

A “simple” request may become:

system prompt
+ platform safety rules
+ conversation history
+ current user message
+ RAG context
+ tool schemas
+ output format instructions
= actual model input

This is why the same user message can cost very different amounts in a chat app, RAG app, coding assistant, agent, or search-enabled assistant.

4. How providers count tokens

Provider-side usage counting usually happens across the gateway, tokenizer, model server, and billing system.

sequenceDiagram participant C as Client participant G as API Gateway participant T as Tokenizer participant M as Model Server participant B as Billing C->>G: Send request G->>G: Auth, rate limit, context check G->>T: Tokenize input T-->>G: Input token count G->>M: Send token IDs M-->>G: Generate output tokens G->>G: Count output tokens G->>B: Write usage and billing event G-->>C: Return response and usage

Typical steps:

authenticate API key, project, organization, quota, and rate limits;
normalize messages, tools, response format, and multimodal inputs;
count input tokens with the model’s tokenizer;
check context length;
generate output and count generated tokens;
summarize usage when generation stops, is truncated, or errors;
write model, user, request ID, tokens, cost, and status into billing;
return a response with usage metadata.

A simple usage object may look like:

{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 350,
    "total_tokens": 1550
  }
}

More detailed platforms may expose:

{
  "usage": {
    "input_tokens": 1200,
    "output_tokens": 350,
    "cached_input_tokens": 800,
    "reasoning_tokens": 120,
    "image_input_tokens": 240
  }
}

Field names differ across providers, but the core idea is the same: bill the units the model actually processed and generated.

5. Streaming usage

For non-streaming calls:

`1`	`generate full answer -> count output tokens -> return response + usage`

For streaming calls:

generate token 1 -> send chunk
generate token 2 -> send chunk
generate token 3 -> send chunk
...
finish -> return final usage or record it internally

An API proxy usually relays chunks while also accumulating the response. If the upstream provider returns usage in the final chunk, the proxy uses it. If not, it may re-tokenize the final output and estimate usage locally.

Interrupted streams need careful handling:

Scenario	Typical treatment
User stops generation	Already generated output tokens may still be billed
Client disconnects	Depends on whether the proxy cancels upstream quickly
Proxy forwarding fails	Depends on whether upstream already processed the request
Timeout	May include only input or partial output

A serious proxy should not bill only by the characters received by the client. It should rely on upstream usage or server-side generation records when possible.

6. How API proxy platforms track usage

An API proxy sits between users and model providers. It forwards requests, but also manages authentication, quota, pricing, routing, retries, and audit records.

flowchart TD A[User request] --> B[Proxy auth] B --> C[Quota and balance check] C --> D[Estimate input tokens] D --> E[Choose upstream provider] E --> F[Forward request] F --> G[Read upstream usage] G --> H[Calculate cost] H --> I[Deduct balance or quota] I --> J[Write usage ledger] J --> K[Return response]

There are usually three accounting layers:

Layer	Purpose	Accuracy
Pre-request estimate	Context check, balance check, rate limiting	Estimate
Post-request settlement	Use upstream usage or local tokenizer result	Closer to real usage
Offline reconciliation	Compare proxy records with provider invoices	Final audit

Pre-request estimate

Before forwarding, the proxy needs to decide whether the request should be accepted.

model = request.model
tokenizer = tokenizer_registry.get(model)

normalized_prompt = render_messages(
    system=request.system,
    messages=request.messages,
    tools=request.tools,
    response_format=request.response_format
)

input_tokens_estimated = tokenizer.count(normalized_prompt)

if input_tokens_estimated + request.max_output_tokens > model.context_window:
    reject("context length exceeded")

if user.remaining_quota < estimate_cost(input_tokens_estimated, request.max_output_tokens):
    reject("insufficient quota")

The render_messages step matters. Counting only the latest user message underestimates the request.

Post-request settlement

After completion, the proxy should prefer provider usage:

response = upstream.call(request)
usage = normalize_usage(response.usage, provider, model)

cost = usage.input_tokens * price.input
     + usage.output_tokens * price.output
     + usage.cached_tokens * price.cached_input
     + usage.reasoning_tokens * price.reasoning

ledger.insert({
    request_id,
    user_id,
    provider,
    model,
    usage,
    cost,
    status: "success"
})

user.balance -= cost

Provider usage is more authoritative because the provider knows internal templates, cache hits, special modalities, and exact generated output.

Usage ledger

A mature proxy writes immutable usage records instead of only reducing a user balance.

CREATE TABLE ai_usage_ledger (
    id BIGINT PRIMARY KEY,
    request_id VARCHAR(64) NOT NULL,
    user_id BIGINT NOT NULL,
    provider VARCHAR(32) NOT NULL,
    model VARCHAR(128) NOT NULL,
    prompt_tokens INT DEFAULT 0,
    completion_tokens INT DEFAULT 0,
    cached_tokens INT DEFAULT 0,
    reasoning_tokens INT DEFAULT 0,
    total_tokens INT DEFAULT 0,
    upstream_cost DECIMAL(18, 8) DEFAULT 0,
    user_cost DECIMAL(18, 8) DEFAULT 0,
    status VARCHAR(32) NOT NULL,
    created_at TIMESTAMP NOT NULL
);

The ledger makes billing explainable: request ID, user, provider, model, token counts, upstream cost, user-facing cost, and status can all be audited.

7. Why proxy counts can differ from official counts

Cause	Why it differs
Tokenizer version mismatch	Model tokenizer rules may differ or evolve
Message template mismatch	Providers wrap roles, tools, and messages internally
Hidden system prompt	Apps or proxies may inject invisible instructions
Conversation history	Old messages re-enter the input
Large tool schemas	JSON schemas can exceed the user prompt
RAG context	Retrieved chunks increase input tokens
Structured output	JSON schema and examples add prompt cost
Multimodal conversion	Images and audio are not simple text tokens
Cache hits	Cached tokens may have separate pricing
Retries	Failed upstream attempts may still create cost

Reasonable proxy behavior:

estimate locally before request;
settle with upstream usage after request;
estimate locally only when upstream usage is unavailable;
reconcile proxy logs with upstream bills regularly.

8. Context windows and maximum output

Token counting also determines whether a request can fit into the model context window:

`1`	`input tokens + maximum output tokens <= context window`

Example with an 8192-token context window:

Input tokens	Max output	Safe
2000	1000	Yes
7000	1000	Near the limit
7900	1000	Likely rejected
8200	500	Input already exceeds window

Many context length errors are not caused by the latest user message. They come from history, system prompts, RAG snippets, tool schemas, and output constraints.

9. Cached tokens

Some providers cache repeated prompt prefixes. If the system prompt, tool schema, or a long document prefix is identical across requests, the provider may reuse processed context.

Request 1:
long system prompt + tool schema + user question A
all input processed normally

Request 2:
long system prompt + tool schema + user question B
the repeated prefix may hit cache

Usage may include:

Field	Meaning
`input_tokens`	Total input tokens
`cached_input_tokens`	Input tokens served from cache
`billable_input_tokens`	Tokens charged at normal input price, depending on provider rules

For proxies, this means pricing cannot always be total_tokens * one_price. Detailed fields matter.

10. Where token cost grows

Scenario	Token pattern
Simple Q&A	Short input, output dominates
Long summarization	Long input, medium output
Code generation	Code symbols, indentation, and file context add tokens
RAG Q&A	More retrieved chunks means more input tokens
Agent tool use	Tool schemas and tool results repeatedly enter context
Structured JSON output	Schemas and field names add prompt and output tokens
Embeddings	Usually input-only usage
Image understanding	Images are converted to visual input units
Speech tasks	May use duration, audio tokens, or mixed billing

The most commonly underestimated costs are history, tool schemas, and retrieved context.

11. Estimating tokens yourself

A rough UI estimate may use:

`1`	`estimated_tokens = text.length / 3`

But application-side controls should use a tokenizer compatible with the selected model:

1
2
3

tokenizer = get_tokenizer(model)
tokens = tokenizer.encode(rendered_prompt)
count = len(tokens)

Estimate the rendered prompt, not only the latest user message:

rendered_prompt = render_chat_request(
    system_prompt,
    developer_prompt,
    history_messages,
    retrieved_context,
    tool_schemas,
    response_format,
    user_message
)

count_tokens(rendered_prompt, model)

The goal is not perfect agreement with the provider. The goal is to catch obvious context overflows and cost spikes early.

12. Engineering notes for proxy billing

Module	Purpose
Model router	Select provider, channel, and model
Token estimator	Pre-check input tokens and maximum cost
Usage normalizer	Convert provider-specific usage into internal fields
Price table	Store input, output, cached, and multimodal prices
Usage ledger	Record tokens, cost, status, and request ID
Balance system	Reserve, settle, refund, or deduct quota
Retry policy	Decide how failed and interrupted calls are charged
Reconciliation job	Compare internal ledger with provider invoices
Rate limiting	Limit requests and token usage by user, model, IP, or org

Typical request handling:

handle_request(req):
    user = authenticate(req.api_key)
    model = resolve_model(req.model)

    estimated = estimate_usage(req, model)
    reserve_quota(user, estimated.max_cost)

    try:
        upstream_resp = call_provider(req, model)
        actual_usage = normalize_usage(upstream_resp.usage)
        actual_cost = calculate_cost(actual_usage, model.price)

        settle_quota(user, actual_cost)
        write_ledger(req.id, user.id, model, actual_usage, actual_cost, "success")
        return upstream_resp

    except ProviderError as err:
        release_or_settle_by_policy(user, req.id, err)
        write_ledger(req.id, user.id, model, estimated, 0, "failed")
        raise

Two details matter:

reservation is not settlement: reserve by worst-case output, settle by actual usage;
idempotency matters: repeated reporting for the same request_id must not double-charge the user.

13. Reducing token cost

Method	Why it works	Risk
Shorten system prompts	Reduces fixed input	Too little instruction can hurt stability
Compress history	Replaces full turns with summaries	Summaries may lose details
Control RAG Top-K	Adds fewer irrelevant chunks	Recall may suffer
Simplify tool schemas	Reduces repeated schema cost	Tool calls may become less precise
Set output limits	Prevents overly long completions	Answers may be truncated
Use caching	Reuses stable prefixes	Depends on provider support
Choose model by task	Simple tasks use cheaper models	Lower quality may require retries
Log usage	Finds the most expensive paths	Must respect privacy and compliance

Summary

Token counting has three layers:

Layer	Focus
Model layer	Tokenizer turns text into token IDs, then the model generates token by token
API layer	Input, output, tools, multimodal content, and cache are summarized as usage
Platform layer	Proxies estimate, limit, settle, audit, and reconcile usage

The key idea:

A token is not a character count. It is the unit the model actually processes. Providers count real model usage; proxies estimate before forwarding, settle after receiving usage, and keep ledger records for billing and audit.

For application developers, the main habit is to measure the full rendered context. System prompts, history, RAG snippets, tool schemas, and structured output constraints all become tokens, and therefore affect cost, latency, and context limits.

RAG Fully Explained:From "Closed-Book" to "Open-Book" LLMs

Wed, 08 Jul 2026 15:00:00 +0800

RAG 技术全景：让大模型从"闭卷考试"到"开卷考试"

大模型很强，但它会"一本正经地胡说八道"。RAG（Retrieval-Augmented Generation，检索增强生成）给大模型外挂了一个"知识库"，让它从闭卷考试变成开卷考试，回答既准又能溯源。本文用 6 张图带你完整搞懂 RAG。

一、先聊痛点：大模型的"幻觉"从哪来？

你大概遇到过这样的场景：问 GPT 一个公司内部的政策问题，或者让它总结一份最新发布的财报，它信誓旦旦地给你一段听起来很专业的回答——但仔细一查，全是编的。

这就是大模型的幻觉（Hallucination）。它不是故意的，而是由其工作方式决定的：LLM 本质上是在做"下一个 token 预测"，它根据训练时记住的概率分布生成文本，分不清"知道"和"不知道"。

原生 LLM 在企业级场景下有四个硬伤：

知识截止：训练数据有截止日期，不知道今天发生的事
幻觉问题：没见过的知识就靠"编"，还很自信
领域知识不足：对企业内部文档、专业细分领域不熟悉
无法溯源：回答不知道来自哪里，没法核验

要解决这些问题，传统思路是微调（Fine-tuning）——用领域数据重新训练模型。但微调又贵又慢，而且知识一更新就得再来一遍。

RAG 给了另一条路：不重新训练模型，而是给它外挂一个可随时更新的知识库，回答时先去知识库里检索，再基于检索到的内容作答。

就像考试一样：原生 LLM 是闭卷考试，全凭记忆，记不清就编；RAG 是开卷考试，遇到不会的可以翻书，答案有据可循。

二、RAG 是什么：核心思想一句话

RAG = 检索（Retrieval）+ 增强（Augmentation）+ 生成（Generation）

在 LLM 生成回答之前，先从外部知识库中检索相关资料，把这些资料增强到 Prompt 里作为上下文，再让 LLM 基于这些上下文生成回答。

用一句话概括：RAG 让 LLM 从"闭卷考试"变成"开卷考试"。

它和微调的根本区别在于：微调把知识"烧进"模型权重里，RAG 把知识放在模型之外的数据库里，用的时候现查。这也决定了两者的适用边界——后面会详细对比。

三、RAG 工作流程：三阶段详解

RAG 的完整流程可以拆成离线建库和在线问答两大阶段。在线问答又分为检索、增强、生成三步。

3.1 离线阶段：把文档变成可检索的向量库

这一步是"建图书馆"的过程，通常一次性完成，知识更新时增量重建：

文档加载：读取 PDF、Word、HTML、Markdown 等原始文档
切分（Chunking）：把长文档切成小段。为什么不整篇存？因为检索时要找最相关的片段，整篇文档向量会"稀释"语义；而且 LLM 上下文有限，塞不下整篇
向量化（Embedding）：用嵌入模型把每段文本转成一个高维向量（通常 768~1536 维），语义相近的文本在向量空间里距离也近
存入向量数据库：把向量和原文一起存进去，建好索引，供后续快速检索

3.2 在线阶段：检索 → 增强 → 生成

用户每一次提问，都会实时跑这三步：

① 检索 Retrieval

把用户问题用同一个 Embedding 模型转成向量
在向量数据库中做相似度检索（通常用余弦相似度），召回 Top-K（一般 3~10 条）最相关的文档片段

② 增强 Augmentation

把检索到的片段拼到 Prompt 模板里，作为上下文
一个典型的 Prompt 长这样：

你是一个严谨的问答助手，请根据以下参考资料回答用户问题。
如果资料中没有答案，请回答"根据现有资料无法回答"，不要编造。

【参考资料】
{检索到的文档片段1}
{检索到的文档片段2}
...

【用户问题】
{用户的问题}

③ 生成 Generation

把拼好的 Prompt 喂给 LLM，生成最终回答
可以要求模型在回答中标注引用来源，实现可溯源

四、完整架构图

这张图把离线和在线两条流水线放在了一起。注意几个关键点：

Embedding 模型必须一致：离线建库用哪个模型，在线检索时也要用同一个，否则向量空间不兼容
知识更新无需重训模型：新增文档只需走一遍"切分 → 向量化 → 入库"，模型权重完全不动
检索质量是 RAG 的命门：如果第一步召回的片段就不相关，后面 LLM 再强也答不好

五、核心组件选型

RAG 系统由四个核心组件构成，选型直接影响效果：

组件	作用	常见选型
Embedding 模型	文本转向量，决定语义匹配上限	BGE（智源）、M3E、OpenAI text-embedding-3、Cohere、bge-m3
向量数据库	存储与检索向量	Milvus、Qdrant、Chroma、Pinecone、FAISS、Weaviate
LLM	基于上下文生成回答	GPT-4、Claude、Qwen、DeepSeek、Llama、GLM
编排框架	串联整个流程	LangChain、LlamaIndex、Haystack、DSPy

几个选型经验：

中文场景：Embedding 优先考虑 bge-large-zh 或 bge-m3，比 OpenAI 的英文模型在中文检索上更准
轻量起步：向量库用 Chroma（纯 Python，单机嵌入式）跑通流程，量大了再换 Milvus / Qdrant
框架选择：快速原型用 LangChain，文档密集型场景用 LlamaIndex（它对文档处理和索引抽象更细致）

六、进阶技术：从"能用"到"好用"

朴素 RAG（一次向量检索 + 拼 Prompt）能跑通，但效果往往一般。真正生产级的 RAG 通常会在四个方向上做增强：

6.1 Query 改写（让问题更适合检索）

用户的问题往往是口语化、模糊的，直接拿去检索效果差。常见技巧：

Query Rewriting：让 LLM 把"那个东西怎么用"改写成"产品 X 的使用方法"
HyDE（Hypothetical Document Embeddings）：先让 LLM 生成一个假设性答案，再用这个答案的向量去检索——因为"答案"和"答案"的语义比"问题"和"答案"更接近
Multi-Query：把一个问题拆成多个子问题分别检索，结果合并去重
Step-back Prompting：先抽象出上位概念再检索，比如"2024 年诺贝尔物理学奖得主"先退到"诺贝尔物理学奖近年得主"

6.2 混合检索 + Rerank（多路召回，精排提纯）

单靠向量检索有个问题：它擅长语义匹配，但对专有名词、产品型号、人名这种精确字面匹配反而不如传统检索。所以生产系统通常用混合检索：

向量检索（语义）+ BM25 关键词检索（字面）并行召回
用 RRF（Reciprocal Rank Fusion） 融合两路结果
再用 Cross-Encoder 重排模型（如 bge-reranker-large、Cohere Rerank）对召回结果做精排
最后取 Top-K 喂给 LLM

Rerank 的关键区别：Embedding 是把 query 和 doc 分别编码再算相似度（双塔），而 Cross-Encoder 是把 query 和 doc 拼一起送进模型（交叉编码），精度高得多，但速度慢——所以先用向量检索粗筛，再用 Rerank 精排，兼顾速度和精度。

6.3 GraphRAG：基于知识图谱的全局检索

微软 2024 年提出的 GraphRAG 解决的是朴素 RAG 的另一个痛点：局部性。向量检索只能召回和问题相似的片段，回答不了"总结整个文档集的核心主题"这种全局性问题。

GraphRAG 的做法：

用 LLM 从文档中抽取实体和关系，构建知识图谱
用社区检测算法（如 Leiden）把图谱分成主题层次化的社区
为每个社区生成摘要
查询时根据问题粒度选择社区摘要或具体节点回答

它特别适合"全局性总结"“跨文档推理"类问题，代价是建图成本高、更新麻烦。

6.4 Self-RAG / 自适应 RAG：让模型自己决定要不要检索

不是每个问题都需要检索——“1+1 等于几"还去查库就浪费了。Self-RAG 让模型学会自我反思：

是否检索：判断问题是否需要外部知识，简单问题直接答
检索结果是否相关：过滤掉无关片段
生成是否忠于检索：检查答案有没有脱离资料编造

好处是省 token、降延迟，坏处是需要训练模型学会这些判断（或用 prompt 模拟），实现复杂度高。

七、方案对比：RAG vs 微调 vs 长上下文

随着模型上下文窗口越来越长（Claude 200K、Gemini 1M、GPT-4o 128K），有人质疑 RAG 是不是过时了——直接把所有文档塞进上下文不就行了？

实际并非如此，三者各有定位：

长上下文：贵（按 token 计费）、易"迷失在中间”（Lost in the Middle 现象）、不可溯源。适合一次性分析单份长文档
RAG：经济、可扩展到海量文档、可溯源。适合知识库问答
微调：改变模型行为/风格，而非注入事实知识。适合固定格式输出、领域术语适配
RAG + 微调：效果最好，成本最高，用于高精度专业场景

经验法则：知识频繁变化 → RAG；风格/格式固定 → 微调；两者都要 → 结合用。

八、实战：用 LangChain 跑通一个最小 RAG

理论讲完，来看一段能跑的代码。用 LangChain + Chroma + OpenAI 实现 30 行的 RAG：

from langchain_community.document_loaders import TextDocumentLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# ① 离线建库：加载 → 切分 → 向量化 → 入库
docs = TextDocumentLoader("knowledge.txt").load()
chunks = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50      # 500 字一段，重叠 50 字
).split_documents(docs)
vectorstore = Chroma.from_documents(
    chunks, OpenAIEmbeddings(model="text-embedding-3-small")
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})  # 召回 Top-4

# ② 在线问答：检索 → 增强 → 生成
prompt = ChatPromptTemplate.from_template("""
你是严谨的问答助手，根据以下资料回答问题。资料中没有答案就说"无法回答"，不要编造。

【资料】
{context}

【问题】
{question}
""")

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | ChatOpenAI(model="gpt-4o-mini", temperature=0)
    | StrOutputParser()
)

# ③ 提问
print(rag_chain.invoke("公司请假流程是什么？"))

这段代码虽然简单，但已经包含了 RAG 的所有核心环节。生产化时再把 Embedding 换成 BGE、加上 Rerank、换成 Milvus，骨架不变。

九、常见挑战与优化方向

实际落地 RAG，踩坑往往不在模型，而在检索。下面是常见问题对应的优化手段：

症状	根因	优化方向
检索结果不相关	Query 与文档语义差距大	Query 改写、换更强的 Embedding
召回准确但答非所问	片段顺序/噪声干扰	加 Rerank 精排、调 Top-K
答案不忠于资料	LLM 忽略上下文编造	Prompt 加约束、降 temperature、强制引用
专有名词召回差	纯向量检索短板	加 BM25 混合检索
上下文超长	召回太多塞不下	上下文压缩、LongLLMLingua、换长上下文模型
知识更新延迟	索引重建慢	增量索引、流式更新
答案无法溯源	未保留片段来源	检索结果带 metadata，Prompt 要求标注引用

一个经验：80% 的 RAG 优化工作都在检索阶段，而不是生成阶段。先把检索做准，效果立刻提升一大截。

十、典型应用场景

企业知识问答：基于内部 Wiki、手册、合同的智能客服与员工助手
法律/医疗咨询：基于法条、判例、药品说明、病历的精准问答
代码助手：基于私有代码库的代码生成（Cursor、Copilot Workspace 都用了 RAG 思想）
教育辅导：基于教材的个性化答疑，答案可指向课本页码
智能搜索：搜索引擎的 AI 摘要（Perplexity、Bing Chat、Google AI Overviews）
数据分析：Text-to-SQL，基于数据库 schema 的自然语言查询

总结

RAG 不是某个具体工具，而是一种 “把外部知识检索和大模型生成结合起来” 的架构思想。它的价值在于：让大模型能用最新、最准、可溯源的外部知识来回答问题，而无需重新训练。

回顾本文要点：

核心思想：检索 → 增强 → 生成，让 LLM 从闭卷变开卷
两大阶段：离线建库（切分+向量化+入库）+ 在线问答（检索+增强+生成）
命门在检索：80% 的优化工作都在检索阶段
进阶四件套：Query 改写、混合检索+Rerank、GraphRAG、Self-RAG
方案选择：知识频繁变化用 RAG，风格固定用微调，高精度场景结合使用

随着模型上下文窗口变长、多模态能力增强，RAG 的形态还在演进（比如多模态 RAG、Agentic RAG），但"外挂知识库"这个核心思想短期内不会过时——毕竟，开卷考试永远比背书更可靠。

如果你对其中某个环节（比如向量数据库选型、Embedding 微调、GraphRAG 实现）想深入，欢迎在评论区交流。

AI on Snow

The Complete MCP Guide: Architecture, Remote Call Flow, and Building Your Own Server

The Complete MCP Guide: Architecture, Remote Call Flow, and Building Your Own Server

Why MCP exists

The four roles

Tools, resources, and prompts

The complete remote MCP call path

1. Configuration and discovery

2. Tools enter model context

3. The model selects; the host decides

4. HTTP and business execution

5. Results return to the model

Two remote connection models

Build a remote MCP server

Set up the project

Create server.py

Test locally

Containerize and deploy

Connect the server to an agent

Codex

OpenAI Agents SDK

Verify actual agent behavior

Tool design sets the agent’s ceiling

Security: every boundary is untrusted

Observability and troubleshooting

Production checklist

Closing thought

References

From Keyword Matching to Vector Retrieval: Three Ways to Search

From Keyword Matching to Vector Retrieval: Three Ways to Search

Keyword Matching

Fuzzy Search

Vector Retrieval

Core Differences

Classic Applications

Practical Selection

The Takeaway

KV Cache 深入理解：为什么只存 K/V，不存 Q

KV Cache 深入理解：为什么只存 K/V，不存 Q

一、把 Attention 看成一次“查表”

二、Prefill 与 Decode：缓存到底发生在哪

三、为什么只存 K/V，不存 Q

如果存 Q，能不能少算一点？

旧 attention 分数要不要缓存？

训练时为什么好像会保存 Q？

四、KV Cache 换来的代价

五、工程实践里还要注意什么

How Token Counting Works: How AI Providers and API Proxies Track Usage

How Token Counting Works: How AI Providers and API Proxies Track Usage

1. What a token is

2. Why billing is token-based

3. What gets counted in a request

4. How providers count tokens

5. Streaming usage

6. How API proxy platforms track usage

Pre-request estimate

Post-request settlement

Usage ledger

7. Why proxy counts can differ from official counts

8. Context windows and maximum output

9. Cached tokens

10. Where token cost grows

11. Estimating tokens yourself

12. Engineering notes for proxy billing

13. Reducing token cost

Summary

RAG Fully Explained:From "Closed-Book" to "Open-Book" LLMs

RAG 技术全景：让大模型从"闭卷考试"到"开卷考试"

一、先聊痛点：大模型的"幻觉"从哪来？

二、RAG 是什么：核心思想一句话

三、RAG 工作流程：三阶段详解

3.1 离线阶段：把文档变成可检索的向量库

3.2 在线阶段：检索 → 增强 → 生成

四、完整架构图

五、核心组件选型

六、进阶技术：从"能用"到"好用"

6.1 Query 改写（让问题更适合检索）

6.2 混合检索 + Rerank（多路召回，精排提纯）

6.3 GraphRAG：基于知识图谱的全局检索

6.4 Self-RAG / 自适应 RAG：让模型自己决定要不要检索

Create `server.py`