Budgets and errors

Bounding what an optional feature can spend, and making sure nothing it does can take down the deterministic tool it lives inside.

The pricing rules themselves are cost and budgets; this page is about wiring them into a CLI that has to keep working.

Two failure classes

The distinction everything here rests on, and the one most often collapsed:

Class	How it arrives	Examples	Correct handling
Operational	thrown as InferenceError	missing API key, unknown provider name, unresolved selector	Catch at the boundary, translate, exit with your config-error code
Model	returned on run.error	validation failed after both attempts, provider API error, timeout	Never thrown. Record it, skip the subject, carry on

Collapse them and a fixable configuration mistake becomes an unhandled stack trace with the wrong exit code.

Do it

examples/budget-and-degrade.mjs

// Gating an optional feature on cost, and making every failure land softly.
// Runs with no API key: MockProvider stands in for a real provider.

import {
  InferenceError,
  MockProvider,
  completeValidatedJSON,
  costOfUsage,
  makeProvider,
  pricingFor,
} from "@hawkeyexl/inference";

// Your CLI's own error type. Your fail() handler maps this to an exit code —
// and only this, which is why a foreign error type must never escape.
class MyToolError extends Error {}

// Operational failures are thrown at construction. Translate them at the boundary.
function buildProvider(spec) {
  try {
    return makeProvider(spec);
  } catch (error) {
    if (error instanceof InferenceError) throw new MyToolError(`config: ${error.message}`);
    throw error;
  }
}

try {
  buildProvider({ provider: "anthropic", apiKeyEnv: "EXAMPLE_KEY_THAT_IS_NOT_SET" });
} catch (error) {
  console.log("translated:", error instanceof MyToolError);
  console.log("message:", error.message.slice(0, 46));
}

// A model failure is different: never thrown, always recorded on the run.
const schema = {
  type: "object",
  required: ["title"],
  properties: { title: { type: "string" } },
  additionalProperties: false,
};

const provider = new MockProvider(
  [
    { json: { title: "Authentication" } },
    { json: { title: "Rate limits" } },
    // Two consecutive rejects exhaust one document's attempts, since each call
    // retries once. One would simply be absorbed by the retry.
    { json: { nope: true } },
    { json: { nope: true } },
    { json: { title: "Webhooks" } },
  ],
  "claude-sonnet-4-5",
);

const pricing = pricingFor(provider.modelName());
const maxCostUsd = 0.009;
const documents = ["auth.md", "limits.md", "errors.md", "webhooks.md", "retries.md"];

let spent = 0;
let filled = 0;
let skipped = 0;

for (const doc of documents) {
  // Gate BEFORE the call. Checking afterwards means the overspend already happened.
  // The pricing guard matters: an unpriced model would make this check inert.
  if (pricing !== undefined && spent >= maxCostUsd) {
    console.log(`budget reached at $${spent.toFixed(6)} — skipping ${doc}`);
    skipped += 1;
    continue;
  }

  const run = await completeValidatedJSON({
    provider,
    system: "You propose a title for a documentation page.",
    user: `Path: ${doc}`,
    schema,
  });

  spent += costOfUsage(run.usage, pricing);

  if (run.error !== undefined) {
    // Warn and move on. The deterministic path is untouched.
    console.log(`${doc}: skipped — ${run.error.slice(0, 38)}`);
    skipped += 1;
    continue;
  }
  filled += 1;
}

console.log("filled:", filled, "skipped:", skipped);
console.log("spent usd:", spent.toFixed(6), "ceiling usd:", maxCostUsd.toFixed(6));

// The ceiling is soft in two ways, both worth designing around:
//
//  1. Gating before each call bounds the overshoot to one call. It does not
//     prevent one — the call that crosses the line has already been paid for.
//  2. A retried request reports only the SUCCESSFUL attempt's usage, so the two
//     rejected attempts on errors.md cost real input tokens that never reached
//     this total.
//
// Leave headroom rather than treating the number as exact.
console.log("calls actually made:", provider.requests.length);
console.log("calls charged to the total:", filled);

translated: true
message: config: Anthropic provider needs EXAMPLE_KEY_T
errors.md: skipped — Response failed schema validation:  mu
budget reached at $0.009000 — skipping retries.md
filled: 3 skipped: 2
spent usd: 0.009000 ceiling usd: 0.009000
calls actually made: 5
calls charged to the total: 3

Translate InferenceError at the boundary

function buildProvider(spec) {
  try {
    return makeProvider(spec);
  } catch (error) {
    if (error instanceof InferenceError) throw new MyToolError(`config: ${error.message}`);
    throw error;
  }
}

Your fail() handler maps your error type to an exit code. A foreign type escapes it and becomes a crash. Every existing consumer of this package wrote this same six-line boundary — do not rediscover it.

Keep the rethrow for anything that is not an InferenceError. Swallowing unknown errors is how a real bug becomes a silent no-op.

Gate before the call, not after

if (pricing !== undefined && spent >= config.maxCostUsd) break;

Checking afterwards means the overspend has already happened.

The pricing guard is not optional

Without pricing !== undefined, an unpriced model makes spent permanently 0 and the loop becomes unbounded. The gate is not satisfied — it is inert.

Your ceiling is soft, in three ways

The last two lines of the sample output are the point: five calls were made, three were charged.

1. Gating before each call bounds the overshoot to one call — while the loop is sequential. It does not prevent one: the call that crosses the line has already been paid for.
2. A retried request reports only the successful attempt’s usage. The two rejected attempts on errors.md cost real input tokens that never reached the total.
3. Concurrency widens the bound. The loop above is sequential, so exactly one call can be in flight when the gate opens. Run subjects concurrently — which we recommend for ensembles — and up to K calls can clear the gate before any of them adds to spent. The overshoot becomes K calls, not one.

Rule 1 and concurrency do not compose

The one-call bound is a property of sequential iteration, not of the gate. If you add a worker pool, recompute your headroom against the pool size. See running at scale.

Leave headroom. Treat the number as a bound with slack, not a measurement.

Degrade, do not fail

if (run.error !== undefined) {
  warn(`${doc.path}: skipped — ${run.error}`);
  continue;            // the deterministic path is untouched
}

Three situations must all end in a warning and a clean exit, never a stack trace:

Situation	What happens	Why it is safe
No API key	InferenceError at construction	Translated, reported as configuration
Read-only cache directory	Warns once, run continues	The cache is an optimization, never a dependency
Budget exhausted	Loop breaks, remaining subjects skipped	Reported in the summary, exit code unchanged
Model returns nothing usable	run.error set, subject skipped	Never coerced into your output

That last row is the invariant the whole layer is built on: an errored run is recorded, never dropped and never coerced.

Next

• Cost and budgets — pricing resolution and the price table
• Cost reference — signatures
• Errors and type index — everything InferenceError covers