When agent-skill scanners disagree: what conflicting verdicts on the same skill mean for your install policy
A single scanner's pass on an agent skill is weak evidence. The Skillselion census of 79,848 skill listings, frozen on 12 August 2026, found that 87.8% had never been audited by any of five auditors, and that on 1,233 listings one auditor rated the code HIGH or CRITICAL while another rated it LOW or SAFE. That is 12.7% of the multi-rated set. An install policy should therefore carry at least four states: known bad, conflicting verdicts, unaudited, and consistent pass on the exact version. Collapsing them into "green badge or not" approves risky skills and blocks useful ones, often in the same week.
The consensus, stated fairly
The prevailing practice is reasonable on its face. Platform teams cannot read every SKILL.md their developers want, so they lean on the scanner a registry already runs, approve skills that pass, and investigate the few that fail. Scanner vendors employ researchers who find real malware. Snyk's ToxicSkills study of 3,984 skills from ClawHub and skills.sh found 76 confirmed malicious payloads through human-reviewed analysis. A pass from people who catch things like that seems worth something.
The steel-man goes further. Audits concentrate where risk concentrates. The census reports that the audited 12.2% of listings hold 83.3% of all recorded installs, 137,238,408 of 164,841,042. If the popular skills are covered, perhaps the unaudited long tail matters less.
Why well-run teams believe it
The incentives line up behind the badge. The registry wants a visible safety signal, the scanner vendor wants distribution, and the platform team wants a defensible approval with low coordination cost. Each party is acting sensibly. The second-order effect is that a single vendor's threshold becomes the organisation's risk appetite, and nobody chose that threshold on purpose. That is a principal-agent problem dressed as a checkbox.
The cracked assumption
The badge assumes auditors measure the same thing. The census numbers say they do not. Reading each auditor's fail count against the listings it rated (the census does not label the second figure, so this reading is an inference):
Auditor Fails Listings rated Fail rate
Snyk 697 9,716 7.2%
Runlayer 371 5,775 6.4%
Gen Agent Trust Hub 478 9,715 4.9%
Socket 70 9,728 0.7%
ZeroLeaks 0 6,857 0.0%A tenfold spread in fail rates across auditors looking at overlapping listings is not noise around a shared truth. It is evidence of different definitions. The census itself says so: "A flag is an auditor's published judgment of capability and findings, not an accusation of intent."
Capability is where the definitions diverge. The census counts at least 43.5% of all installs going to skills that declare shell execution, and 3,137 shell-capable listings with no audit at all, holding 15,858,667 installs. One auditor may score "can run shell" as HIGH. Another may score it LOW unless the shell use looks hostile. Both are internally consistent. A team that inherits either verdict without knowing which philosophy produced it has outsourced its threshold.
There is a second gap. The Cloud Security Alliance's June 2026 post notes that malicious instructions written in plain language inside SKILL.md leave "No detectable code signature", and that "Pattern-matching scanners looking for dangerous function calls miss these entirely." A code-focused pass and a prompt-focused failure can both be correct about the same file.
The alternative mechanism in one sentence
Disagreement between auditors is information about what kind of risk a skill carries, and absence of audit is information about how much is known, so policy should route on both instead of on any single verdict.
A disagreement taxonomy to test
The census measures how often auditors conflict, not why. The causes below are hypotheses for replication, not findings:
- Capability versus intent. One auditor rates declared shell, network or filesystem access as high risk. Another rates only evidence of hostile use.
- Code versus language. One auditor scans scripts. Another reads the SKILL.md instructions the model will follow.
- Version skew. Verdicts attach to different commits of a skill that changed between scans.
- Scale mapping. Auditors' native severities map differently onto LOW, MEDIUM, HIGH and CRITICAL.
- Reach. One auditor follows remote fetches and bundled files. Another stops at the top-level folder.
The replication protocol is simple to state. Draw 200 listings from the conflict set, read the highest-rated finding and the lowest-rated verdict for each, and classify the cause, with two reviewers working blind to each other and to the auditors' names. threat.cafe has not yet run it. The protocol is published first so that the result can embarrass it.
The policy states, as a decision inversion table
State Definition Default action Owner
Known bad Confirmed malicious by a named, sourced report Block Automated
Conflict >=1 HIGH/CRITICAL and >=1 LOW/SAFE, same version Hold for human review AppSec reviewer
Consistent concern All available verdicts MEDIUM or above Block unless excepted AppSec lead
Consistent pass >=2 independent LOW/SAFE on the pinned version Allow, pin, watch drift Platform team
Single pass One LOW/SAFE verdict, no others Treat as unaudited Platform team
Unaudited, low capability No verdicts; no shell, network or file writes Allow with logging Platform team
Unaudited, high capability No verdicts; declares shell or network access Hold for human review AppSec reviewerThe inversion is in two rows. "Single pass" sits with unaudited, because one philosophy's approval is not independent confirmation. And "conflict" goes to a person who reads the HIGH finding, because that finding is the one cheap piece of analysis already paid for.
Three lines of evidence, strongest last
The coverage gap. 70,090 of 79,848 listings had no audit. Any rule that equates "no flag" with "approved" approves most of the catalog by default.
The fail-rate spread. From 0.0% to 7.2% across five auditors on overlapping populations. Picking one auditor picks one of those rates as your organisation's tolerance.
The conflict count. 1,233 listings drew directly opposite ratings. For each of those, at least one respected auditor would have had you approve a skill that another rated HIGH or CRITICAL, or block one another rated LOW or SAFE.
Four questions for any scanner or registry badge
Before a badge enters your approval workflow, get written answers to four questions:
- What does a HIGH rating mean? Capability, suspected intent, or confirmed malice. Ask for the definition, not an example.
- Which version was scanned? A verdict without a commit hash or version number cannot be matched to what your developer installs.
- Does the scan read SKILL.md instructions, or only scripts? If only scripts, plain-language payloads are out of scope by design.
- What happens on re-scan? If a skill changes after approval, does the verdict expire, or does the old badge stay up?
An auditor that answers all four is giving you a measurement. One that cannot is giving you a colour.
Where the consensus still wins
For low-capability skills that only format text or call no tools, the single-badge approach costs little; the downside is bounded by what the skill can do. And for a team with ten approved skills, a reviewer reading each SKILL.md beats any multi-state policy. The four-state model earns its coordination cost once approvals run to dozens a month.
Where this argument would be wrong
If the replication finds that most conflicts are version skew, the remedy is simpler: pin versions and re-scan, and the conflict state shrinks to a data-hygiene problem. If one auditor turns out to be right on nearly every conflict, teams should follow that auditor and drop the rest. Either result would be worth publishing.
The Monday-morning decision is small. Add "conflict" and "unaudited" as named states in your approval workflow, and stop counting a single pass as two.
threat.cafe's own vocabulary already separates "known bad", "no known bad" and "unknown", and the early-access list never labels anything safe.