Retries and fallback

Backoff

import { withRetry, backoffMs, sleep } from "@providerkit/core";

const result = await withRetry((attempt) => callProvider(attempt), {
  maxAttempts: 3,
  baseDelayMs: 1_000,
  maxDelayMs: 30_000,
  budgetMs: 60_000,
  signal,
  onRetry: ({ attempt, delayMs, error }) => log.warn({ attempt, delayMs, error }),
});

Defaults: 3 attempts, 1s base, 30s ceiling. backoffMs is full jitter: a random point in [0, min(max, base · 2^attempt)) rather than the exact doubling. Exact doubling synchronises every client that failed at the same moment into retrying at the same moment, which is how a provider blip turns into a self-inflicted thundering herd.

A server-requested retryAfterMs always wins over the computed delay.

budgetMs caps how long the retries go on: 60 seconds, counted from the first failure. A new attempt starts only if it would start inside that window, wait included. An attempt that is already running is never cut. Without it, three stalled streams at the watchdog’s five minutes each kept a caller waiting a quarter of an hour. Raise it for long jobs, or pass Infinity to turn it off. withStreamRetry takes the same option.

By default only transient kinds are retried, and sleep is abortable: a caller pressing Stop during a backoff wait returns immediately rather than after the delay.

Streams: retry only while nothing has been emitted

This is the rule that matters, and the one that is easy to get wrong.

Wider than this screen. Scroll the figure sideways.

Fig. The same failure, twice. withStreamRetry retries only to the left of the horizon; past it a retry would replay tokens the reader has already seen, so the error is handed to the caller instead.

import { withStreamRetry } from "@providerkit/core";

const stream = withStreamRetry(
  (signal, attempt) => provider.createStream(messages, tools, { signal }),
  { maxAttempts: 3 },
);

Once a single chunk has reached the caller, the turn is committed. Retrying after that replays tokens the user has already seen. The answer restarts mid-sentence, or worse, restarts in a different model’s voice. So withStreamRetry retries only while nothing has been yielded, and after the first chunk a failure propagates.

Each attempt gets its own AbortController, so abandoning an attempt actually cancels its in-flight request instead of leaving it running.

Backup models

import { streamWithBackupModels } from "@providerkit/core";

const stream = streamWithBackupModels(
  (model, signal) => provider.createStream(messages, tools, { model, signal }),
  {
    models: ["claude-sonnet-5", "claude-haiku-4-5", "gpt-5"],
    onFallback: ({ model, position, total }) =>
      log.info(`falling back to ${model} (${position}/${total})`),
  },
);

The list is [primary, ...backups]. Position 1 is the primary, so a fallback is anything past it. By default the walker advances only on isBackupEligible kinds (overload, rate), because those are the failures a different model actually fixes. An auth failure or a bad request will fail identically on every model in the list, and walking it just multiplies the same error.

The commitment rule applies to the walker too

streamWithBackupModels carries the same “nothing emitted yet” guard. A walker without it will switch models mid-answer and emit the second model’s response after the first model’s opening. The reader sees the answer restart in a different voice.

withBackupModels is the non-streaming equivalent.