Skip to content
T threat.cafe

· Desmond Okafor-Hale (editorial persona) · Economics and measurement · on threat.cafe

Your AI keys live in more places than you think: what Fireworks, OpenCode, Railway and Zeabur taught us in October

In October 2026, OpenAI, Anthropic and Google were not breached, yet their customers' keys leaked anyway, because Fireworks AI, OpenCode Console, Zeabur and Railway each held copies on customers' behalf and each disclosed an incident within about two weeks. The risk sat with the custodian. Map every copy of every key, issue one key per service, cap spend at the provider, rotate at the issuer, and prefer workload identity.

The consensus, stated fairly

The mainstream view among security teams is that AI provider keys are dangerous mainly because of the providers themselves, so the sensible response is to scrutinise OpenAI, Anthropic and Google, read their trust pages, negotiate their contracts and watch their status feeds. The view has real merit, because providers concentrate enormous value, run the billing systems that turn a stolen key into money, and would expose every customer at once if breached. A careful risk register ranks that tail event first. Most procurement processes are built around this belief. Smart people hold it.

The incentive structure reinforces the consensus, since a provider assessment is one document with one counterparty while the services holding copies of a key are scattered across teams, credit cards and free tiers. The coordination cost of tracking them falls on nobody in particular. The principal-agent problem does the rest. Reviewing the provider feels like diligence. Reviewing the long tail feels like overhead.

The cracked assumption: custody, not the provider

The October incidents show that the consensus rests on an assumption that no longer holds, namely that a key lives mainly with its issuer and with you. A single production key now routinely sits in a developer's shell profile, a coding agent's config, a hosted console, an evaluation service, a deploy target's environment variables and a CI secret store. Each custodian sets its own encryption, staff access and disclosure speed. The issuer sees one key. The custodians number six.

The alternative mechanism fits in one sentence: attackers who want inference keys get better returns from intermediaries that hold thousands of customers' keys than from the providers that issue them. The customer carries the loss. Storing one more copy is free; losing it is not. That perverse incentive has a second-order effect, because shared keys make attribution impossible, so when five services hold the same key and one leaks, all five must be rotated and the billing log cannot say which one failed. The cost stays invisible until it lands.

The Cloud Security Alliance research note of 23 October 2026 reaches a similar conclusion. The note states plainly that it "was generated with AI assistance and has not undergone official CSA review and approval processes". Each claim below is therefore checked against the vendor's own words.

Three lines of evidence, strongest last

Fireworks AI, the first case, detected the activity on 6 October 2026 and published its bulletin on 9 October, updated on 10 October. Internal Fireworks credentials were misused to read environment variables used in evaluation jobs in "a limited number of customer accounts", and API keys and tokens stored as secrets were captured in some of them. Fireworks says it found no evidence of access to customer models, datasets or inference traffic. One sentence from that bulletin belongs on every wall: "Removing a stored secret alone does not invalidate the underlying credential."

OpenCode Console, the second case, showed the money, because OpenCode said attackers reached an older system and that saved provider keys belonging to about 1.4% of Console users (OpenCode's own figure, October 2026) were exposed, with earliest possible exposure at 00:30 UTC on 17 October, as reported by BleepingComputer and TechCrunch. Users reported unauthorised charges in the hundreds of dollars per account, with a few above $1,000, according to TechCrunch on 19 October 2026. Co-founder Dax Raad said the open source CLI was not affected because the keys were confined to Console, although OpenCode has not said how attackers got in or how keys were stored.

Railway, the third custodian, disclosed on 21 October that an unauthorised party may have accessed encrypted environment variables and, for some, "obtained the decrypted contents", according to Railway's disclosure. Railway believes the incident belongs to a series of attacks on hosting companies aimed at AI inference providers, notified upstream providers so they could revoke keys, and told customers to roll every environment variable. No customer count or root cause has been published.

Zeabur supplies the strongest evidence, because its numbers show what custody at scale looks like when it fails. Zeabur's preliminary report of 19 October 2026, written by founder Yuanlin Lin, confirms that 321,810 accounts and 3,218,965 service environment variable records were read from its database during activity on 14 and 15 October. The incident followed an August intrusion, and the report admits that the August rotation "did not cover the credentials used this time", naming its own root cause as long-lived admin credentials stored in plaintext and usable from the public internet. Lin told RuntimeWire on 22 October that "the keys involved were for OpenRouter, OpenAI, Anthropic, and Gemini", though RuntimeWire notes this does not establish which keys were exfiltrated or used. Zeabur will shut down on 31 December 2026, which it calls an operational decision separate from the incident.

One custodian. Four issuers. 321,810 accounts.

What to check today

Custody work on AI keys is mostly bookkeeping, and the bookkeeping is cheap compared with an overnight bill on a stolen key, so the steps below start with inventory and end with architecture.

  1. Build the threat.cafe Key Custody Map below for each production AI key, listing every service, file and secret store that holds a copy, and ask teams directly because dashboards are easy to forget.
  2. Rotate at the issuer, in the order the runbook below sets out, if any custodian on the map has disclosed; Fireworks, Zeabur and Railway all say that deleting a key from the intermediary does not revoke it.
  3. Issue one key per service and per app, named after its custodian (for example prod-railway-api), so a leak points at its source and revocation has a small blast radius.
  4. Set provider-side spend caps and alerts close to expected consumption, per key or per project where the provider supports it.
  5. Review provider billing and usage logs from mid-September, as the CSA note suggests, or from the earliest disclosed exposure date for each custodian you used.
  6. Prefer workload identity and short-lived tokens over long-lived static keys where provider and host support them, and inject secrets at runtime from your own secrets manager rather than pasting them into third-party dashboards.
  7. Add key custody to vendor reviews: encryption, who can decrypt, notification speed, and whether the vendor notifies the issuing provider.
THE THREAT.CAFE KEY CUSTODY MAP (one row per key; one custodian per key is the goal)

key name          | issuer     | every place a copy lives              | scope           | spend cap      | rotation owner | last rotated
------------------+------------+---------------------------------------+-----------------+----------------+----------------+-------------
prod-railway-api  | Anthropic  | Railway env vars; 1Password vault     | prod workspace  | $500/month     | infra team     | 2026-10-21
eval-fireworks    | OpenAI     | Fireworks secrets; CI secret store    | eval project    | $200/month     | ML team        | 2026-10-09
dev-console-alice | Google AI  | OpenCode Console; ~/.config (laptop)  | dev project     | $50/month      | Alice          | ????
shared-legacy     | OpenRouter | Zeabur env; .env in repo; Slack DM?   | org-wide (!)    | none (!)       | unknown (!)    | unknown

Red flags: more than one custodian per key, org-wide scope, no cap,
no named owner, "unknown" in any column.
ROTATION ORDER RUNBOOK (per exposed key)

1. Contain     Lower or set the spend cap at the issuer now.
2. Issue       Create a new key at the issuer, scoped to one custodian.
3. Replace     Update that custodian only. Redeploy so processes pick it up.
4. Revoke      Revoke the old key AT THE ISSUER. Deleting it from the
               intermediary's dashboard does not revoke it.
               (Zeabur advises revoking before redeploying new keys;
               choose the order that matches your downtime tolerance.)
5. Verify      Confirm the old key fails. Watch issuer usage for 72 hours.
6. Review      Check issuer billing back to the earliest possible exposure.
7. Record      Update the custody map: new key name, owner, date.
8. Repeat      For every other secret the custodian held: database
               passwords, registry tokens, cloud keys, signing keys.
THE THREAT.CAFE CUSTODY INVERSION TABLE

decision             | consensus answer                 | inverted answer (custody view)
---------------------+----------------------------------+------------------------------------------
where to assess      | the AI provider                  | every service holding a copy of the key
how many keys        | one per provider, shared         | one per custodian and per app
what limits damage   | provider security                | provider-side spend cap on each key
what "remove" means  | delete it from the dashboard     | revoke it at the issuer
what to buy next     | a better provider contract       | workload identity, short-lived tokens

Where this would be wrong

The custody argument would be wrong if a provider itself disclosed a breach of issued keys, because the concentration would then sit back with the issuer and the consensus would regain its ranking, and nothing in the October disclosures says that happened. The argument would also weaken if the four incidents turn out to share a single upstream cause that per-key hygiene could not have touched, such as a compromised tool used by all four custodians. Neither condition holds today. Both are worth watching.

The consensus still wins in one place, and the concession matters: spend caps, key scoping and revocation all live at the provider, so provider controls remain the lever that limits damage even when the leak happened somewhere else entirely.

What to tell the people who sign off

  • Finance: unauthorised charges from leaked OpenCode Console keys reached the hundreds of dollars per account, a few above $1,000 (TechCrunch, 19 October 2026), and a spend cap per key bounds that figure in advance.
  • Security lead: blast radius equals the number of custodians sharing a key, so one key per custodian turns an organisation-wide rotation into a single-service one.
  • Legal and compliance: add custody questions (encryption, decryption access, notification speed, notice to issuers) to third-party risk assessments, as the CSA note recommends.
  • Operations: Railway paused all deployments during its investigation, so a rotation plan that does not depend on a custodian's deploy pipeline is worth having before the next disclosure.

What we don't know yet

The open questions about October 2026 sit mainly with attribution and scope, and none of them changes the rotation advice above. No vendor has publicly linked these incidents to one actor; the "campaign" framing comes from Railway, and the CSA note says so. Entry routes into OpenCode Console and Railway remain undisclosed, as does Railway's count of affected customers. Exposure is not misuse, and Zeabur's founder naming four issuers does not confirm which keys were used. The "distillation" motive that OpenCode and Railway describe is unexplained, and TechCrunch reports that OpenCode did not explain the label.

The threat.cafe board records known-bad agent artifacts; key custody is the part only you can map.

Volatility: VOLATILE (event-triggered review). Review when Railway or OpenCode publishes a root cause, or when any vendor attributes these incidents to a single actor.

Sources

#api-keys #secrets-management #third-party-risk #key-rotation #ai-providers

All posts