How Token Counting Works: How AI Providers and API Proxies Track Usage

When using LLM APIs, you often see fields such as prompt_tokens, completion_tokens, and total_tokens. A token is not simply a word or character. It is the unit produced by the model’s tokenizer, and provider billing is built around counting input, output, cached, tool, and multimodal units.

The short version:

1. What a token is

LLMs do not process raw text directly. Before inference, text goes through a tokenizer, which splits it into tokens and maps each token to a numeric ID.

For example:

1

Write a Dockerfile for me

may be split into pieces such as:

1

["Write", " a", " Docker", "file", " for", " me"]

This is only illustrative. Different models can tokenize the same text differently. Common words may be one token, rare characters may be split, and whitespace, punctuation, JSON, logs, and source code can all add tokens.

The practical definition is: a token is the smallest vocabulary unit the model actually reads or generates.

2. Why billing is token-based

LLM cost is strongly related to token count:

1. longer inputs require more context processing;
2. longer outputs require more generation steps;
3. larger context windows increase memory and attention cost;
4. tools, multimodal input, long context, caching, and reasoning models add more accounting dimensions.

A simplified formula:

1
2
3

cost = input_tokens * input_price
     + output_tokens * output_price
     + cached / reasoning / image / audio / tool-related token cost

Output tokens are often priced higher because they are generated step by step. Each new token depends on the current context and previous generated tokens.

3. What gets counted in a request

Real API requests usually contain more than the user’s latest message.

A “simple” request may become:

1
2
3
4
5
6
7
8

system prompt
+ platform safety rules
+ conversation history
+ current user message
+ RAG context
+ tool schemas
+ output format instructions
= actual model input

This is why the same user message can cost very different amounts in a chat app, RAG app, coding assistant, agent, or search-enabled assistant.

4. How providers count tokens

Provider-side usage counting usually happens across the gateway, tokenizer, model server, and billing system.

Typical steps:

1. authenticate API key, project, organization, quota, and rate limits;
2. normalize messages, tools, response format, and multimodal inputs;
3. count input tokens with the model’s tokenizer;
4. check context length;
5. generate output and count generated tokens;
6. summarize usage when generation stops, is truncated, or errors;
7. write model, user, request ID, tokens, cost, and status into billing;
8. return a response with usage metadata.

A simple usage object may look like:

1
2
3
4
5
6
7

{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 350,
    "total_tokens": 1550
  }
}

More detailed platforms may expose:

1
2
3
4
5
6
7
8
9

{
  "usage": {
    "input_tokens": 1200,
    "output_tokens": 350,
    "cached_input_tokens": 800,
    "reasoning_tokens": 120,
    "image_input_tokens": 240
  }
}

Field names differ across providers, but the core idea is the same: bill the units the model actually processed and generated.

5. Streaming usage

For non-streaming calls:

1

generate full answer -> count output tokens -> return response + usage

For streaming calls:

1
2
3
4
5

generate token 1 -> send chunk
generate token 2 -> send chunk
generate token 3 -> send chunk
...
finish -> return final usage or record it internally

An API proxy usually relays chunks while also accumulating the response. If the upstream provider returns usage in the final chunk, the proxy uses it. If not, it may re-tokenize the final output and estimate usage locally.

Interrupted streams need careful handling:

A serious proxy should not bill only by the characters received by the client. It should rely on upstream usage or server-side generation records when possible.

6. How API proxy platforms track usage

An API proxy sits between users and model providers. It forwards requests, but also manages authentication, quota, pricing, routing, retries, and audit records.

There are usually three accounting layers:

Pre-request estimate

Before forwarding, the proxy needs to decide whether the request should be accepted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

model = request.model
tokenizer = tokenizer_registry.get(model)

normalized_prompt = render_messages(
    system=request.system,
    messages=request.messages,
    tools=request.tools,
    response_format=request.response_format
)

input_tokens_estimated = tokenizer.count(normalized_prompt)

if input_tokens_estimated + request.max_output_tokens > model.context_window:
    reject("context length exceeded")

if user.remaining_quota < estimate_cost(input_tokens_estimated, request.max_output_tokens):
    reject("insufficient quota")

The render_messages step matters. Counting only the latest user message underestimates the request.

Post-request settlement

After completion, the proxy should prefer provider usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

response = upstream.call(request)
usage = normalize_usage(response.usage, provider, model)

cost = usage.input_tokens * price.input
     + usage.output_tokens * price.output
     + usage.cached_tokens * price.cached_input
     + usage.reasoning_tokens * price.reasoning

ledger.insert({
    request_id,
    user_id,
    provider,
    model,
    usage,
    cost,
    status: "success"
})

user.balance -= cost

Provider usage is more authoritative because the provider knows internal templates, cache hits, special modalities, and exact generated output.

Usage ledger

A mature proxy writes immutable usage records instead of only reducing a user balance.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

CREATE TABLE ai_usage_ledger (
    id BIGINT PRIMARY KEY,
    request_id VARCHAR(64) NOT NULL,
    user_id BIGINT NOT NULL,
    provider VARCHAR(32) NOT NULL,
    model VARCHAR(128) NOT NULL,
    prompt_tokens INT DEFAULT 0,
    completion_tokens INT DEFAULT 0,
    cached_tokens INT DEFAULT 0,
    reasoning_tokens INT DEFAULT 0,
    total_tokens INT DEFAULT 0,
    upstream_cost DECIMAL(18, 8) DEFAULT 0,
    user_cost DECIMAL(18, 8) DEFAULT 0,
    status VARCHAR(32) NOT NULL,
    created_at TIMESTAMP NOT NULL
);

The ledger makes billing explainable: request ID, user, provider, model, token counts, upstream cost, user-facing cost, and status can all be audited.

7. Why proxy counts can differ from official counts

Reasonable proxy behavior:

1. estimate locally before request;
2. settle with upstream usage after request;
3. estimate locally only when upstream usage is unavailable;
4. reconcile proxy logs with upstream bills regularly.

8. Context windows and maximum output

Token counting also determines whether a request can fit into the model context window:

1

input tokens + maximum output tokens <= context window

Example with an 8192-token context window:

Many context length errors are not caused by the latest user message. They come from history, system prompts, RAG snippets, tool schemas, and output constraints.

9. Cached tokens

Some providers cache repeated prompt prefixes. If the system prompt, tool schema, or a long document prefix is identical across requests, the provider may reuse processed context.

1
2
3
4
5
6
7

Request 1:
long system prompt + tool schema + user question A
all input processed normally

Request 2:
long system prompt + tool schema + user question B
the repeated prefix may hit cache

Usage may include:

For proxies, this means pricing cannot always be total_tokens * one_price. Detailed fields matter.

10. Where token cost grows

The most commonly underestimated costs are history, tool schemas, and retrieved context.

11. Estimating tokens yourself

A rough UI estimate may use:

1

estimated_tokens = text.length / 3

But application-side controls should use a tokenizer compatible with the selected model:

1
2
3

tokenizer = get_tokenizer(model)
tokens = tokenizer.encode(rendered_prompt)
count = len(tokens)

Estimate the rendered prompt, not only the latest user message:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

rendered_prompt = render_chat_request(
    system_prompt,
    developer_prompt,
    history_messages,
    retrieved_context,
    tool_schemas,
    response_format,
    user_message
)

count_tokens(rendered_prompt, model)

The goal is not perfect agreement with the provider. The goal is to catch obvious context overflows and cost spikes early.

12. Engineering notes for proxy billing

Typical request handling:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

handle_request(req):
    user = authenticate(req.api_key)
    model = resolve_model(req.model)

    estimated = estimate_usage(req, model)
    reserve_quota(user, estimated.max_cost)

    try:
        upstream_resp = call_provider(req, model)
        actual_usage = normalize_usage(upstream_resp.usage)
        actual_cost = calculate_cost(actual_usage, model.price)

        settle_quota(user, actual_cost)
        write_ledger(req.id, user.id, model, actual_usage, actual_cost, "success")
        return upstream_resp

    except ProviderError as err:
        release_or_settle_by_policy(user, req.id, err)
        write_ledger(req.id, user.id, model, estimated, 0, "failed")
        raise

Two details matter:

• reservation is not settlement: reserve by worst-case output, settle by actual usage;
• idempotency matters: repeated reporting for the same request_id must not double-charge the user.

13. Reducing token cost

Summary

Token counting has three layers:

The key idea:

A token is not a character count. It is the unit the model actually processes. Providers count real model usage; proxies estimate before forwarding, settle after receiving usage, and keep ledger records for billing and audit.

For application developers, the main habit is to measure the full rendered context. System prompts, history, RAG snippets, tool schemas, and structured output constraints all become tokens, and therefore affect cost, latency, and context limits.