Streaming

Set stream: true. The wire format is Server-Sent Events carrying chat.completion.chunk objects, terminated by data: [DONE] — the same shape every OpenAI SDK already parses.

stream = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[{"role": "user", "content": "Write a haiku about settlement."}],
    max_tokens=200,
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n\n{chunk.usage.total_tokens} tokens")

const stream = await client.chat.completions.create({
  model: 'gpt-5.6-terra',
  messages: [{ role: 'user', content: 'Write a haiku about settlement.' }],
  max_tokens: 200,
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
  if (chunk.usage) console.log(`\n\n${chunk.usage.total_tokens} tokens`);
}

Usage

With stream_options: {"include_usage": true}, every chunk carries "usage": null and one final chunk — after the finish_reason chunk, with an empty choices array — carries the real counts:

{
  "id": "chatcmpl-…",
  "object": "chat.completion.chunk",
  "created": 1757923200,
  "model": "gpt-5.6-terra",
  "choices": [],
  "usage": { "prompt_tokens": 412, "completion_tokens": 186, "total_tokens": 598 }
}

Without include_usage, no usage chunk is sent at all.

Reading the cost

A streamed response cannot carry x-nanorouter-charged-usdc. The figure is not known until the last token, long after the headers went out. What you get up front is x-nanorouter-request-id. Hold it, and read the settled figure back once the stream ends:

with client.chat.completions.with_raw_response.create(..., stream=True) as raw:
    request_id = raw.headers["x-nanorouter-request-id"]
    for chunk in raw.parse():
        ...

# after the stream closes
cost = requests.get(
    f"https://nanorouter-wcg9d.ondigitalocean.app/api/requests/{request_id}",
    headers={"Authorization": f"Bearer {session_token}"},
).json()

chargedUsd is null until the ledger closes the request, which happens within a moment of the stream ending. A null there means “not settled yet”, not “free”.

/api/requests/{id} takes a dashboard session token, not an nr_ API key. An API key on that endpoint is a 401.So a server-side integration cannot currently read its own settled costs back programmatically — the figure is visible on the dashboard, and in x-nanorouter-charged-usdc on non-streaming requests, but there is no key-authenticated way to fetch it for a stream.stream: false gets you the header on a fast request, but not reliably: the header is sent only when the response has not already begun. A buffered request that takes longer than 45 seconds gets the whitespace keepalive described below, which commits the headers before the ledger closes — and then the cost header is omitted exactly as it is on a stream.So there is currently no dependable key-authenticated way to read a settled cost. Long or reasoning-heavy requests are exactly the ones where it goes missing.

Reasoning

Reasoning models stream their summary as delta.reasoning_content, alongside the ordinary delta.content:

{ "choices": [{ "index": 0, "delta": { "reasoning_content": "The user is asking…" } }] }

This is why the summary exists at all: on a request that also searches, a model thinking in silence leaves the caller with nothing on the wire for a minute or more, with no way to tell a working request from a hung one. Reasoning is billed as output tokens. reasoning_effort defaults to low.

Tool activity

When the router runs web_search or web_fetch on the model’s behalf, those rounds happen between upstream calls and emit no message content. To keep that from looking like a stall, the router sends a chunk with an empty delta and a top-level nanorouter key:

{
  "choices": [{ "index": 0, "delta": {}, "finish_reason": null }],
  "nanorouter": {
    "type": "tool",
    "name": "web_search",
    "detail": "arc network usdc gas token",
    "arguments": "{\"query\":\"arc network usdc gas token\"}",
    "status": "started"
  }
}

A matching "status": "done" chunk follows, with ms for the duration and failed: true if the tool errored. It is a top-level key rather than something inside delta, because it is not part of the assistant’s message. Every OpenAI SDK ignores unknown top-level keys, so this is safe to ignore — but it is what lets you render “searching the web…” instead of a spinner.

Long, quiet requests

Two things the router does that you may notice: Streamed responses get SSE keepalives every 15 seconds, so an idle connection is not dropped by an intermediary. Buffered (stream: false) responses get whitespace keepalives. A non-streaming request sends nothing at all until the model finishes, and ingress proxies drop connections that have been silent too long. After 45 seconds the router commits the 200 and drips spaces until the real body is ready. JSON tolerates leading whitespace, so JSON.parse, httpx’s .json() and every OpenAI SDK skip them — but a hand-rolled parser that assumes the first byte is { will not.

Cancelling

Disconnect and the upstream call is aborted. You are still billed for the tokens generated before you disconnected — they were really generated — and the ledger closes with that partial usage. The partial answer is kept in your request history.

Get started

How it works

Guides

Usage

Reading the cost

Reasoning

Tool activity

Long, quiet requests

Cancelling

​Usage

​Reading the cost

​Reasoning

​Tool activity

​Long, quiet requests

​Cancelling

Usage

Reading the cost

Reasoning

Tool activity

Long, quiet requests

Cancelling