Streaming and the watchdog

The nastiest streaming failure is not an error. It is a provider that accepts the request, returns 200, sends a few bytes, and then nothing. No error, no close, no timeout. Every SDK’s default is to wait forever, and the symptom is the worst kind: none at all.

A request timeout does not catch it either. The request succeeded. What died is the gap between chunks.

Wider than this screen. Scroll the figure sideways.

Fig. Each chunk cancels the pending timer and arms a new one, so the ghosts are the time that was left. Only silence reaches the end of a window, and watch.classify() is what tells that abort apart from the caller pressing Stop.

Watching a stream

import { streamWatch, watchChunks } from "@providerkit/core";

const watch = streamWatch({ provider: "anthropic", signal: callerSignal });

try {
  const chunks = provider.createStream(messages, tools, {
    signal: watch.signal,
    onActivity: watch.sawActivity,
  });
  for await (const chunk of watchChunks(watch, chunks)) {
    handle(chunk);
  }
} finally {
  watch.dispose();
}

The watch runs two clocks, because “silent” means two different things:

• Idle, 60 seconds (idleMs, STREAM_IDLE_MS): no byte at all once the response has started. That is a dead socket. Every read resets it, keep-alives included, which is what onActivity is for. Leave it out and keep-alives don’t count: a provider that pings while it writes a long tool call gets killed at 60 seconds.
• Progress, 5 minutes (progressMs, STREAM_PROGRESS_MS): no chunk, counted from the moment the request is sent. That catches a stream that stays alive and goes nowhere, and it covers the wait for the response to start. Some backends think before they send a single byte: the ChatGPT backend can stay silent for minutes on a large prompt at high effort, and that turn is healthy. If you set a longer idleMs and leave progressMs out, progress waits as long as idle. When this clock runs out before the response started, the timeout carries shouldRetry: false, so withStreamRetry does not send the same prompt again: a retry would wait the same five minutes and pay for the thinking twice. It is still a timeout, so a fallback pool moves on to the next provider. The watch learns the response started from sawActivity (the headers) or the first chunk.

watchChunks calls sawByte() for you on each chunk, which resets both. When either runs out, watch.signal aborts, and watch.classify(err) turns the abort into a timeout. That is how you tell the provider went quiet from the caller pressed Stop, which both arrive as the same AbortError otherwise.

Three of the five codebases providerkit came from had each picked 60 seconds for the idle clock, on their own.

Always dispose()

The watchdog holds a timer. Without dispose() in a finally, a long-lived process leaks one per stream.

The composition, wrapped

Those lines (build a watch, hand the provider the WATCH’s signal and sawActivity, wrap the chunks) are easy to get subtly wrong, and the middle one fails silently in every test: pass the caller’s signal instead and the watchdog still arms, still fires, and cancels nothing, because the request it was meant to abort was never told about it. The composition ships as a wrapper so the next caller does not have to get it right by reading carefully:

import { withWatchdog } from "@providerkit/core";

const guarded = withWatchdog(provider, { idleMs: 60_000 });

for await (const chunk of guarded.createStream(messages, tools, { signal: callerSignal })) {
  handle(chunk);
}

guarded is an ordinary Provider, so retry and backup-model walkers wrap it unchanged. It passes onActivity for you. The wrapper catches the other way a stream fails silently too: a turn that ends without an answer throws instead of succeeding with an empty one. An answer is text or a tool call. Reasoning still streams as it arrives, but a turn that only thought has said nothing.

How it throws depends on why the turn was empty:

• It hit maxTokens (a length finish): invalid. On most providers thinking and answer share the cap, so a high-effort model can spend all of it thinking. Raise maxTokens or lower effort. It is never retried and never sent to a backup model, because the same cap would fail the same way.
• Anything else: overload, which is transient, so the retry and backup walkers try again.

One exception: an empty stop right after a tool result passes, because an agent whose answer is the tool call ends its turn exactly that way. requireContent: false turns the check off; onFirstChunk reports TTFT without the plumbing.

A stream that ends too early

A connection can also close cleanly in the middle of a turn: no finish reason, no closing event, no [DONE]. Every adapter hands over what arrived, then throws a network error, the same one a socket dying mid-read throws. So a half-written answer is never kept as the whole one, and the retry and backup walkers see a failure they know how to handle.

A connection that drops after the turn has finished is not a failure. Kimi, GLM and OpenCode Go close the socket right there, and the turn is kept. Only the usage sent after the finish may be lost.

A finish reason the adapter doesn’t know also fails, as a retryable overload. DeepSeek’s insufficient_system_resource and Gemini’s OTHER used to pass as a normal end. Other servers’ names for a normal end, like Together’s eos, still read as stop, and a full context window reads as length.

Bridging the caller’s signal

streamWatch composes your signal with its own using AbortSignal.any, which is deliberate: AbortSignal.any aborts synchronously when an input is already aborted. An event listener cannot catch that case: if the caller aborted before you subscribed, the event has already fired and your handler never runs.

Where the runtime supports it the timer is unref’d, so a pending watchdog never holds a Node process open on its own.

Time to first token

const ttft = watch.firstChunkMs(); // null until the first chunk arrives

TTFT is the number users actually feel. Total duration is dominated by answer length, but the wait before anything appears is the perceived latency. It is also the first thing to move when a provider degrades, usually well before the error rate does. Under withWatchdog, pass onFirstChunk and skip the plumbing.