#Running in Production

The next step after Quickstart is flipping one action from ghost to enforce for real traffic. This page is what to have on your monitor when you do it.

Two reading paths:

• Just finished Quickstart, ready to enforce one action? → §1 → §2 → §4 → §5 → §6. Read §6 before you flip, not after — §5's last check is that you already know it.
• Evaluating Anvil for your team? → §1 → §3 → §5 → §6.

#1. What "production" means with Anvil

Going to production with Anvil isn't a deploy. It's a toggle in /app — you flip one action from Ghost to Enforce, and that call is now protected in real traffic. Anvil Cloud has been running the control plane, dedup, receipts, and audit log the whole time since you ran anvil protect.

Small responsibility boundary. Nothing sneaks in.

The SDK is a wrapper, not a runtime. No long-lived processes, no sidecar. And an action Anvil has resolved to ghost still executes even if Anvil is unreachable — that invariant holds through outages. (See Fail-modes for the one case it cannot: a process that has never once reached Anvil, and so has no idea which mode you are in.)

#2. What happens when you flip to enforce

Path: /app → Actions → your action row → toggle Ghost → Enforce.

Three things change at runtime. Budget about eight seconds for the flip to reach every process — the plane holds its policy snapshot for up to 5 s and each SDK process its own control cache for up to 3 s. Nothing is lost in that window: calls keep running under the previous mode and every one is still recorded.

• Verdicts stop being hypothetical. would_block becomes blocked. would_dedup becomes duplicate. The call throws instead of running. The class thrown is AnvilBlockedError, which carries retryable, durableKey and blockReason; it extends AnvilPolicyBlockedError, so a catch written against either name keeps working. Both are exported from @useanvil/sdk.
• No new network calls. On the hosted transport every guarded call already makes two trips to api.anvilcontrol.ai — one before your call to claim the key and evaluate the policy, one after to record the outcome, each with a 5 s timeout. Flipping to enforce doesn't add a trip; it changes what the first one is permitted to answer.
• Fail-mode activates. Governed by ANVIL_FAIL_MODE (see below).

Nothing else about your app changes. No redeploy, no restart, no config file to edit.

#Fail-modes — what happens when Anvil is unreachable

The one env-var you should have an opinion on. Set on the client, before deploy — not in the dashboard. Fail mode defines what each value does; this section is about picking one and recognising it in your logs.

One case reaches you as an ordinary policy block rather than one of those errors. If the action carries a durable_key_dedup rule and Anvil already knows the shared store is down, closed refuses at evaluation time with dedup_store_unavailable — so you get an AnvilBlockedError (retryable: true) naming the fail mode, and the call appears in your decision log like any other block instead of only as an exception. The guarantee is the same either way: the call does not execute. Retry it with the same key once the store is reachable.

"Already knows" is deliberately narrow. A connection that is merely reconnecting is not an outage — a managed-Redis failover takes a few hundred milliseconds, and refusing every guarded payment during that window would be a far worse failure than the one this guards against. Anvil answers this question without a network round trip, from the connection state it already holds, and only a connection that has genuinely ended counts. A store that stays down is caught a moment later at the claim, by the fail mode above, where your call's amount is known and graceful can weigh it.

These refusals are never counted as money saved. Anvil could not evaluate the call; no rule of yours is why it did not run. They are excluded from every prevented figure on the Impact page, in anvil audit stats and in anvil cc, and reported on their own line instead — N held — Anvil could not reach its store. The same holds for the spend-cap store: an unreadable cap fails closed under ANVIL_FAIL_MODE=closed, and the refusal is recorded as spend_store_unavailable — an availability event, never a cap breach and never money a rule saved.

Ghost is exempt from all of this. During an outage, an action Anvil has resolved to ghost still executes your call and records the observation locally as nonDurable. The promise "ghost never blocks" holds even when Anvil is down — so a bad flip to enforce during an outage is a real risk to know about, and one dashboard click to undo.

One thing enforce does not change: an action with no rule behind it. Flip an action to enforce while your workspace has no activated policy and the call still runs — carried by the always-on floor and by nothing else, because there are no rules of yours to evaluate. The SDK says so once per action: "is set to ENFORCE but this workspace has no activated policy." Enforce is the switch that lets your rules block; it is not itself a rule. Activate a policy in Policy Studio to give it something to enforce.

#The mode that governs is the one you set here, not the one in your .env

anvil protect writes ANVIL_MODE=ghost into your .env on first run. Nothing ever rewrites it — not a dashboard flip, not a CLI command, not the API. It is a local default for a self-hosted runtime, and on the hosted transport it is not the authority: the mode that decides whether a call blocks is the one you set per action in the dashboard, resolved server-side.

That distinction only becomes visible during an outage, which is exactly when it matters:

• Anvil has told this process which mode the action is in (any successful guarded call does it, at no extra cost) → that mode governs. Resolved ghost executes; resolved enforce follows the table above. An answer older than 30 seconds is re-checked first with a lightweight mode read (up to 2 seconds, at most once every 5 seconds while it fails), so a flip you just made wins even when the guard path is failing or rate-limited. If Anvil cannot answer at all, the last answer stands: changing a mode is a write to Anvil, so it cannot happen while Anvil is unreachable.
• Anvil has never answered this process — a cold start during a full outage → Anvil does not know whether you are enforcing, and ANVIL_MODE=ghost is not evidence that you are not. ANVIL_FAIL_MODE decides, and it defaults to closed, so the call is held rather than run unprotected. Set ANVIL_FAIL_MODE=open if you would rather have availability in that window.

Every degradation is recorded either way. A held call and a call that ran unprotected both write an audit record naming the degraded dependency, before the side effect runs — so "how many calls executed money unguarded today?" is a question with an answer. A fail-open is a legitimate choice; a silent one is not. Both land in the ledger as unavailable, never as a block.

#One case where enforce does not block

Rare, visible, and reversible: Anvil can set a per-workspace enforcement ceiling. While one is in force, every action in your workspace resolves to ghost no matter what its own mode says — your calls run, nothing is blocked, and every decision is still recorded, with a would-be block landing in the log as a ghost would_block. Dedup and receipts are untouched. Your policy is unchanged and starts blocking again the moment enforcement is restored.

The ceiling only ever reduces enforcement. There is no path by which it turns an action you set to ghost into one that blocks.

You will see it on the dashboard. The runtime pill reads "Observing · paused by Anvil", and the detail underneath names us as the cause: "An Anvil operator has paused enforcement for this workspace, so your rules are being observed and recorded but nothing is being blocked. Your policy is unchanged and will start blocking again the moment enforcement is restored." Three things are worth knowing before you need them:

• Only Anvil can lift it. There is no CLI command and no API route for you to restore enforcement yourself — email henry@anvilcontrol.ai.
• No email or webhook — but it is machine-readable. The dashboard banner is the human signal. GET /v1/modes?action=<action> returns cappedByOperator: true while a ceiling is in force, and every decision made under one carries enforcementCeiling: "observe" in the ledger. Alert on either if an unenforced window matters to you.
• If you poll that route, handle 503. GET /v1/modes answers 503 {"code":"mode_unavailable"} when Anvil cannot read your control config — it will never guess a mode to fill the gap, because the SDK uses that answer to decide whether a degraded call may run unprotected. Treat a 503 as unknown, retry, and never read it as "not capped" or "in ghost". A 200 always carries a real answer, including configured: false for a workspace that has genuinely never flipped anything.
• Custom rules stop blocking too — they still run in your process, they just no longer hold a call. Two windows where one still can: for a few seconds after the pause, a process holding a freshly-cached enforce mode doesn't re-ask; and a process that can't reach Anvil at all falls back to ANVIL_MODE. Both close as soon as the SDK next hears from the plane.

We use this when our own enforcement is the thing doing damage — a rule misfiring across a whole workspace, blocking traffic that should be running. It is rare, it is visible, and it is reversible.

#3. Security & data

Everything a security reviewer wants, on one screen.

#What Anvil sees on the wire

Per-call, Anvil Cloud receives: action name, mode, dedup key, amount, currency, service, any workflow id, and any metadata you explicitly pass. Your function's arguments and its return value are never sent.

One exception, and you opt into it. The webhook_signature rule needs webhook: { rawBody, headers } — a signature cannot be verified without the bytes it signs. So for that rule, and only that rule, the raw body and headers do reach Anvil. Don't use it if the body is sensitive.

The CLI sends one more thing, and not at call time. anvil protect and anvil sync publish a code map: the repo-relative paths and line numbers where your protected calls live, so the dashboard can tell you where an action is and + Protect new can search your project at all. Paths and line numbers only — never the source, never the contents of those lines. Anvil says so the first time it publishes one rather than burying it here. Set ANVIL_NO_CODE_MAP=1 to never send it, and anvil sync --forget to delete what is already stored.

You can read the whole request body yourself: it's the /v1/guard fetch in src/api-client.ts.

#Auth

Keys are workspace-scoped, and a workspace can hold several. anvil protect writes the one it mints to .env as ANVIL_API_KEY; anvil keys list shows every live key and flags the one your shell is using. That is what makes anvil keys rotate safe on a running service — the replacement is minted before the old key is retired, so the workspace is never without one. The last live key cannot be revoked, for the same reason. The dashboard uses Google sign-in — no separate account to create.

Key rotation and revocation — self-serve, from your terminal.

anvil keys list             # what can reach my money path right now?
anvil keys rotate --yes     # replace this shell's key, atomically

rotate mints the replacement and retires the old key in one transaction, so the workspace is never without a live key. Your running service is — the old key is dead the moment the command returns, and every call still presenting it is refused until you put the new key in ANVIL_API_KEY and redeploy. Stage that deploy before you run it.

A revoked key fails closed under enforce, whatever ANVIL_FAIL_MODE says: the SDK raises AnvilCredentialRevokedError and the side effect does not run. Those fail-open knobs exist for outages, and a revoked credential is not one.

Every revoke and rotate lands in your tamper-evident ledger as anvil.key.revoked / anvil.key.rotated, on the same hash chain as your guard decisions. An attacker holding a leaked key can retire your others — but never invisibly. Command reference and the leaked-key incident path: anvil keys.

#Where your decisions land

Every decision — ghost and enforce, allow and block — is written to your workspace's ledger, and that is what the dashboard, anvil audit, and anvil verify read. On a local-development install the ledger is ./artifacts/anvil-audit.jsonl. On hosted, or with ANVIL_POSTGRES_URL set, the store is authoritative and the JSONL file is a best-effort local mirror — production is told to turn it off entirely with ANVIL_LOCAL_SINKS=disabled. Override the mirror's path with ANVIL_AUDIT_LOG=/some/other/path.jsonl.

#Set the URL, then run the migration

ANVIL_POSTGRES_URL on its own does not make the database your ledger — the tables have to exist:

anvil db migrate

Until you run it, Anvil keeps reading the local JSONL and says so rather than showing you an empty page: the Impact page and anvil audit stats both report "Your database has no Anvil tables yet" and name the store they read instead. Nothing is lost in the meantime — the file is still the record — and the moment the migration succeeds, the database becomes the ledger for every read.

The distinction matters because the two failures need different fixes and look identical from the outside: a database Anvil cannot reach is an outage (check the connection); a database Anvil reached that holds no Anvil tables is one command away from done. Anvil tells you which.

#Tamper-evident chain

On Anvil Cloud your ledger is hash-chained server-side — nothing to enable, and anvil verify checks it.

ANVIL_AUDIT_SIGNING_KEY is the local file's equivalent, for self-hosted installs and --local runs: set it to a 32+ character secret (openssl rand -hex 32) and every local record from that point forward is HMAC-SHA-256 chained to the previous one. ANVIL_AUDIT_TAMPER_EVIDENCE=1 makes the key mandatory at startup — the process refuses to boot without it.

#What Anvil Cloud runs on

Hosted infrastructure with managed Postgres + Redis. For teams that need data to stay in their own perimeter — regulated industries, air-gapped environments, signed SLAs — see Where Anvil Runs and §6 below.

#4. Configuration — the two knobs that matter

anvil protect writes everything the SDK needs to talk to Anvil Cloud — into .env.

The only knobs you should routinely tune:

#Self-hosting the dashboard

Running Anvil's own server? Two flags decide what it will do, and both are off by default — the API server is an API server until you say otherwise.

In short: ANVIL_SERVE_WEB serves the dashboard, ANVIL_CONTROL_TOKEN authorises writes, and ANVIL_DEV_NO_AUTH is the local-development shortcut that skips the token. There is a third way in that you may already have: a verified ANVIL_API_KEY authorises writes to its own workspace, and it is checked before any token — so a self-hosted plane whose callers already hold keys may not need a control token at all.

A first run that serves the UI and accepts writes locally:

ANVIL_SERVE_WEB=1 ANVIL_DEV_NO_AUTH=1 npx anvil serve

Open /app with the first flag missing and the page tells you which one is off. That is deliberate: a missing opt-in is configuration, not a crash, and it should never render as one.

The full env-var reference lives in the CLI Reference. Everything else Anvil writes to .env — ANVIL_API_KEY, ANVIL_API_BASE_URL — should stay as anvil protect set it.

Upgrading the SDK: npm update @useanvil/sdk. Anvil is pre-1.0, so read semver the pre-1.0 way: the minor is where breaking changes are allowed, not the major. A ^0.5.x range stops at that boundary, so npm update gives you patches inside your minor and those are safe. Crossing a minor (0.5.x → 0.6.0) is the one to take deliberately: pin ~0.5.x if you'd rather not cross it automatically, and read the release notes before you do.

Limits during beta. Anvil Cloud allows 600 guarded calls per minute, per workspace. Past that you get 429 with a Retry-After header — and a 429 is routed through ANVIL_FAIL_MODE like any other answer Anvil could not give. On the default closed the call is held, not executed, so retrying after the wait is safe and cannot double-charge; under open — or graceful below its threshold — it executes unprotected, and a blind retry there is a second execution. Separately, the hosted path is measured to stay inside its latency budget up to roughly 25 guarded calls in flight at the same instant; beyond that calls get slower, they do not fail. Neither limit binds ordinary agent traffic — 600/min is ten calls a second, and both exist so one runaway loop cannot slow every other tenant. The one workload that meets them is a batch job — bulk payouts, mass refunds, a retry storm — so space those out rather than firing the whole batch at once. If you need a higher ceiling, email henry@anvilcontrol.ai and we will raise it.

#5. Pre-flight checklist — you're ready to flip when…

Five checks. Every one is a yes, or you wait.

• anvil doctor's Traffic Readiness section reads "Ready to take risky traffic in ghost mode." — it prints above the Quick Start list, not at the very bottom
• Dashboard shows ≥ 1 would_block matching a rule you actually wrote (proof the observation path is real, not empty).
• You've decided which ANVIL_FAIL_MODE this action wants and set it in your app's environment.
• The team knows where the Ghost↔Enforce toggle lives (/app → Actions → your action row).
• You know the runbook in §6 and where to find it under pressure.

Rollback: flip Enforce → Ghost in the dashboard. The dashboard requires a 5+ character reason on any downgrade — that's the audit trail. No redeploy.

The full "read the log, rehearse the rollback, flip, watch 30 min" journey lives in the Ghost → Enforce guide. Read that once before your first flip.

#6. When it breaks — the runbook

#Stop enforcing, right now

You do not need the dashboard, a browser, or a second credential. With ANVIL_API_KEY set:

anvil control kill-switch on --reason "why, 5+ characters"

Every action in your workspace resolves to ghost — your calls run, nothing is blocked, and every decision is still recorded. Your policy, per-action modes, dedup and receipts are untouched, so lifting it restores your exact previous posture:

anvil control kill-switch off

To roll back one action instead of everything:

anvil control set-action-mode refund.issue ghost --reason "..."
anvil control show --action refund.issue      # verify against the plane, not the printout

The command prints the window it applied within — the plane's snapshot TTL, 5s by default, on every replica. On the hosted path add each SDK process's own ~3s control cache on top of that, so budget up to ~8 seconds before the switch is in force everywhere rather than the 5 the command reports. Engaging requires a reason and is written to the append-only control history; lifting does not, because making the safe direction harder is how a lever goes unused in an incident.

Read it back with anvil control show or anvil cc. Both distinguish a switch that is genuinely off from a control store they could not read, so neither will print a confident Off over an armed one.

If the output says capped by operator, an Anvil operator has capped your workspace to observe and that cap outranks your own setting — deliberately, so an incident response cannot be undone from outside. Your setting is stored and applies the moment the cap lifts. Email henry@anvilcontrol.ai.

#Then diagnose

Five ordered steps. Copy-paste-friendly under pressure.

1. Ask from the terminal first. anvil status prints what the status page shows, needs no key and no auth, and works from anywhere — so it answers is it us or them? without opening a tab. It exits 0 operational, 2 degraded, 3 major outage, 1 unreachable, which makes it scriptable into an alert.
2. Check the status page. status.anvilcontrol.ai shows Anvil Cloud health in real time. If it's red, the runbook below applies but the root cause isn't yours.
3. Run anvil doctor. Ends with "Ready to take risky traffic in [ghost|enforce] mode." or "Not ready…" plus which check failed.
4. Read the last decisions — both kinds. A rule of yours and an outage of ours file under different statuses, and in an incident the second is usually the one you want:

   anvil audit query --status blocked --since 1h       # your rules fired
   anvil audit query --status unavailable --since 1h   # Anvil could not evaluate
   

Or the Decision log in /app. Look at reason and blockedBy on the recent rows — the ground truth for what happened at the call site. (On a local-development install those same rows are in ./artifacts/anvil-audit.jsonl.)

1. Email henry@anvilcontrol.ai with the anvil doctor output attached. We respond same-day during beta.

#Symptom → cause → fix

#Need something Anvil Cloud can't do?

On-prem deploy. Signed SLA. Data has to stay inside your perimeter. Design-partner pricing on multi-tenant. Anything else the hosted default doesn't cover — email henry@anvilcontrol.ai. See Where Anvil Runs for the shape of the self-hosted conversation.