Skip to content
T threat.cafe

· Tobias Lindqvist-Rao (editorial persona) · Incident response · on threat.cafe

LiteLLM 1.107.0 from an agent operator's seat: the agent config key nobody revoked

A malicious litellm==1.107.0 sat on PyPI from 11:39 to 13:23 UTC on 21 October 2026, about 100 minutes, and poisoned Docker Hub tags, latest included, lasted until 14:20. LiteLLM first blamed GitHub tokens leaking into gateway logs. Its corrected account blames an agent config file: a master key committed publicly in May and not revoked. If you ran 1.107.0, rebuild and rotate everything that host could reach.

The symptom: a release nobody qualified

LiteLLM's release pipeline published version 1.107.0 on 21 October 2026 from source that the project did not tag. The first postmortem, by Krrish Dholakia, Ishaan Jaffer, Oliver Jensen and Nate Armstrong, states that the official v1.107.0 source tag does not contain the malicious files. The artifacts came from an attacker-controlled repository.

The payload runs at Python start-up through cook.pth and sitecustomize.py, so importing LiteLLM is not required and any Python process in an affected environment triggers it, including pip.

Agent gateways make this worse, because a LiteLLM proxy typically holds every provider key in its database, plus master and virtual keys. One poisoned install on that host reaches all of them.

LidorMachluf caught the release within about 20 minutes, according to LiteLLM's acknowledgements, which also credit Socket Threat Research and Trail of Bits.

Telemetry: the 21 October release timeline

LiteLLM's first postmortem gives the release sequence in UTC, reconstructed from GitHub and registry records. The times matter for anyone matching their own CI logs against the window.

  • 10:08:29: GitHub branch protection rejected an unauthorised change to the private release workflow.
  • 10:08:34: a credential tied to a privileged developer account overrode the protection.
  • 10:46:39: the modified workflow ran a staging build from lookalike repository Berri-AI/litellm. The real organisation is BerriAI, without the hyphen.
  • 11:11:18: a second developer's credential satisfied the production approval gate.
  • 11:39:51 to 11:40:13: seven malicious wheels uploaded to PyPI.
  • 11:46:56: Docker Hub and GHCR tags v1.107.0, latest, main-latest and main-stable repointed to image index sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e.
  • 11:53 to 11:54: a third developer's credential deleted the production and staging workflow records.
  • By 13:23: PyPI removed 1.107.0.
  • 13:51: GHCR tags unavailable.
  • 14:20: Docker Hub tags removed or repointed to clean artifacts.

Three developer credentials played three roles: one overrode protection, one approved, one erased evidence. The deletion step is the misleading signal: it limited the logs LiteLLM had to work with on the day.

Failed hypothesis: tokens in gateway logs

LiteLLM's first theory, published on 21 October, pointed at its own gateway logs. The post said LiteLLM had "observed specific situations in which Claude Code will embed a user's GitHub CLI OAuth token into a response body". It added that "in situations where response bodies are logged, these tokens may appear in LiteLLM logs". It also reported an unexpected user account on an internal, internet-exposed LiteLLM deployment.

The hypothesis was plausible, since gateways that log full request and response bodies capture whatever a tool prints, and an attacker inside such a gateway would find tokens there.

The evidence did not hold. An update at the top of the same page, dated 23 October, says some assumptions "were later discovered to be inaccurate or incomplete" and points to a canonical record, which states: "The request logs show the first token came from a different service."

The threat.cafe review found no statement from Anthropic about GitHub CLI tokens in Claude Code response bodies. Beam's write-up by Yash Thakker reports the same gap. The Claude Code claim therefore rests only on LiteLLM's first post, which LiteLLM has since superseded.

Root cause: a master key in a public agent config file

LiteLLM's canonical record, by Oliver Jensen and Mateo Wang, dated 23 October, traces the chain back five months.

  • 27 May 2026: an opencode.json containing the master key for LiteLLM's internal Agent Platform was committed to a public repository. In the record's words, "key revocation did not occur for the master key."
  • 10 October: the attacker used that key against the Agent Platform API. A request listing managed agents returned the GitHub CLI OAuth token of one of the organisation's two owners.
  • 10 and 12 October: with the owner's token, the attacker pushed temporary branches whose workflows exfiltrated more secrets, including other maintainers' tokens. The runs were then deleted.
  • 21 October: the release above, with the owner's token overriding branch protection.

The record notes that GitHub CLI tokens are "post-2fa and not subject to GitHub's 'super-user' 2fa reauth flows". An owner's gh token could therefore relax protections without a fresh second factor. The record does not mention Claude Code or Anthropic at any point.

Agent operators should read the chain as their own failure mode: a coding agent's config file carried a master key, and an agent orchestration service stored an owner's GitHub token and returned it to any holder of that key.

Am I affected

Affected hosts are any that installed litellm==1.107.0 or pulled the compromised images during the window. LiteLLM warns against starting with pip install, because any Python start re-runs the malicious start-up files, so check from the filesystem first.

find / -path /proc -prune -o \( -name 'litellm-1.107.0.dist-info' -o -name 'cook.pth' -o -name 'cook.py' -o -name 'sitecustomize.py' \) -print 2>/dev/null
ls -la /tmp/remote_code_loader.lock 2>/dev/null
docker images --digests | grep litellm
  1. Inspect every hit next to a LiteLLM install. sitecustomize.py can exist legitimately elsewhere.
  2. Search lockfiles, CI caches and agent sandboxes for litellm==1.107.0.
  3. Compare image digests with sha256:1bbcb1443bc9... above. LiteLLM also lists digests for litellm-database and litellm-non_root on GHCR.
  4. Treat any pull of latest, main-latest or main-stable between 11:46 and 14:20 UTC on 21 October as suspect.

LiteLLM is explicit about the limits: finding the files confirms exposure, but not finding them does not prove a host is clean.

The fix

Remediation for an affected host follows LiteLLM's guidance: rebuild from scratch pinned to litellm==1.106.2, then rotate from a clean machine. Rotation covers cloud roles, every provider key in the proxy database, master and virtual keys, the database password and process environment secrets. Our post "Your agent installed a known-bad package: the first hour" has the sequence.

Prevention for agent operators comes down to six controls.

  1. Search repos for agent config files (opencode.json, .claude/, .mcp.json) containing keys, and revoke at the issuer anything ever committed.
  2. Keep agent admin APIs and gateway admin UIs off the public internet, and replace default master keys.
  3. Keep owner tokens out of agent tooling; use fine-grained, short-lived gh tokens for automation.
  4. Redact token patterns in gateway logs, logging metadata by default.
  5. Pin exact versions and image digests, and apply a release-age cooldown (see our post on release-age cooldowns).
  6. Allowlist egress from CI runners and gateway hosts (see our post on egress allowlisting).
THE THREAT.CAFE LITELLM 1.107.0 CONTROL TIMELINE

when (UTC)          event                                  control that would have fired
------------------  -------------------------------------  ----------------------------------------------
27 May              master key committed in opencode.json  secret scanning on push; revoke-on-detect
10 Oct              Agent Platform API called remotely     admin API not reachable from the internet
10 Oct              owner gh token returned by API         no owner tokens stored in agent tooling
10-12 Oct           temp branches exfiltrate secrets       CI egress allowlist; alert on run deletion
21 Oct 10:08        branch protection overridden           protection one owner token cannot bypass;
                                                           alert on org audit-log overrides
21 Oct 10:46        build from Berri-AI/litellm            release workflow bound to the real source org
21 Oct 11:39-13:23  1.107.0 on PyPI                        exact version pin; release-age cooldown
21 Oct 11:46-14:20  Docker tags repointed                  pin by digest, not latest/main-stable
after install       start-up files run on the host         egress allowlist on gateway hosts
any time            tokens printed into gateway logs       body redaction (hygiene, not this root cause)

Incurred trade-offs

Each control above costs something, and the costs land on different teams. A release-age cooldown delays legitimate security fixes as well as poisoned releases, so it needs a documented override that someone owns. Egress allowlists break builds when a dependency moves hosts, and the list needs a maintainer. Log redaction removes detail that on-call engineers use for debugging token and routing errors. Short-lived gh tokens add re-authentication friction for maintainers, and branch protection that no single owner can bypass slows emergency releases.

None of these costs is new, and all of them are cheaper than rotating every key a gateway holds.

The strongest counterargument

The best case against this framing says the incident was a LiteLLM operational failure, not an agent problem. On that reading, a leaked master key and a release pipeline with expanded permissions would have been exploitable whatever tools the team used.

The evidence still points at agent tooling. The key sat in a coding agent's config file, and the token came out of an agent orchestration service; both are artefacts that agent operators now create daily.

Where this would be wrong

The agent-operator framing would be wrong if LiteLLM's next update shows the Agent Platform played no real role, or that the owner token was obtained another way. The framing would also weaken if Anthropic or LiteLLM confirms the Claude Code response-body behaviour as a separate live risk; that would add a second lesson rather than replace this one.

What to tell the people who sign off

  • Finance: the cost is rotation effort and rebuild time on every host that ran 1.107.0; LiteLLM has published no exposure counts to estimate from.
  • Security lead: blast radius is every secret reachable from an affected host, including all provider keys in the proxy database.
  • Legal and compliance: LiteLLM advises assuming exfiltration from any affected host with outbound internet, which may trigger your own notification duties.
  • Operations: rebuild images and virtual environments rather than uninstalling; rollback target is litellm==1.106.2.

What we don't know yet

Several open questions remain about the LiteLLM 1.107.0 incident as of 28 October 2026. LiteLLM has not published install or pull counts for the window. The canonical record shows remote API calls but does not state whether the Agent Platform was directly internet-facing. The Cloud Security Alliance's AI-assisted research note, not officially reviewed, calls the incident related to the Fireworks, OpenCode and Railway intrusions but not confirmed as linked. LiteLLM said it would share process changes the following week.

The threat.cafe board is where hooks and gateways look up known-bad packages like this one; check it for the current verdict on any LiteLLM version you run.

Volatility: VOLATILE (event-triggered review). Review when LiteLLM publishes its process changes or detection guidance, or when Anthropic comments on GitHub tokens in Claude Code output.

Sources

#litellm #pypi #github-tokens #agent-gateways #release-cooldowns

All posts