14 Oct 2026 · Margot Reyes (editorial persona) · Governance and leadership · on threat.cafe

Six agent-security indicators a CISO can report every quarter without vanity metrics

Six indicators tell a board whether AI-agent risk is under control: inventory coverage, checked-install share, unknown-artifact rate, time to answer "are we exposed?", canary pass rate, and approval-tier mix. Each one is a ratio with a stated denominator, and each one changes a decision when it moves. Raw counts do not: CVEs disclosed, alerts fired, skills scanned. Those grow with activity, not with risk. The six below can be computed from hook logs, endpoint inventory and incident tickets, without buying a new product.

The reporting problem in numbers

The burden is measurable. Panaseer's 2026 Security Leaders Peer Report says teams spend 34% of their week collecting, analysing and presenting data, that only 38% of CISOs are "truly confident" their reports are clear and comprehensive, and that 50% find proving control effectiveness a major challenge. The page does not state a sample size, so treat those as directional.

Board time is short. An IANS report of 3 March 2026 says CISOs get "roughly 30 minutes" with boards, and 47% of directors surveyed said reporting on AI-driven risk needs improvement. The director sample was 17 people. That is a small base for a percentage, and the finding should be read as a signal, not a rate.

Trend Micro's July 2026 analysis gives the plainest advice: "Stop reporting raw CVE counts to the board." It projects about 66,000 CVEs in 2026 and puts the actively exploited share "well under 1%". A count that rises 30% while exploited exposure falls tells a board the opposite of the truth.

Why counts mislead: the denominator problem

Counts have no denominator. "We blocked 412 risky installs" could mean the control is working, or that developers attempted 40,000 installs and 412 happened to match. "Zero incidents" could mean safety, or no detection. Every indicator below therefore pairs a numerator with the population it is drawn from.

The indicator definitions sheet

#  Indicator                    Formula                                              Data source                      Decision it drives
1  Inventory coverage           endpoints with agent inventory < 7 days old          MDM device list + config scan    Where controls cannot reach
                                / endpoints known to run agents
2  Checked-install share        agent-initiated installs that passed a policy check  hook logs vs package-manager     Whether enforcement is real
                                / all agent-initiated installs observed              and agent transcripts
3  Unknown-artifact rate        checked installs with verdict "unknown"              hook logs                        Review capacity; list coverage
                                / all checked installs
4  Time to exposure answer      median hours from public report of a malicious      incident tickets                 Inventory and search speed
                                artifact to a fleet-wide yes/no
5  Canary pass rate             canary tests blocked as expected                     canary evidence log              Whether controls fail open
                                / canary tests run
6  Approval-tier mix            share of agent actions auto-allowed, human-          hook and permission logs         Whether human approval is a
                                approved, denied                                                                      control or a formality

1. Inventory coverage

The denominator is the hard part. "Endpoints known to run agents" comes from software inventory, plus any machine whose user holds an agent seat. Report the gap as a count of machines, not just a percentage. Ninety per cent coverage of 2,000 endpoints is still 200 machines nobody can search.

2. Checked-install share

This measures enforcement, not intent. The numerator comes from your policy hook's log. The denominator comes from a source the hook does not control: package-manager logs, proxy logs or agent transcripts. If both numbers come from the hook, the indicator only proves the hook saw what it saw.

3. Unknown-artifact rate

Use three verdict states: known bad, no known bad, unknown. "Unknown" means nobody has reviewed that exact version. A rising unknown rate is not a failure; it is a capacity signal. It tells the board how much of what agents install sits outside any reviewed list. Never relabel "no known bad" as "safe" in a board pack.

4. Time to exposure answer

When a malicious package or skill is published, how long until you can say whether any machine installed it? Measure in hours. Public timestamps show why. In the 2026 TanStack npm compromise, the maintainers' postmortem records 84 malicious versions published from 19:20 UTC and the last registry removal at 23:55 UTC. A team that answers in days is answering after the window has closed. Report the median and the worst case for the quarter.

5. Canary pass rate

A canary is a harmless test entry that a working policy must block. Report passes over runs, per agent client version. Anything below 100% is a finding, because a failed canary means a control fails open without telling anyone.

6. Approval-tier mix

Report the split of agent actions into auto-allowed, human-approved and denied. Then report the human approval rate on its own. Anthropic's engineering post on auto mode says "Claude Code users approve 93% of permission prompts"; the post gives no sample size. If your own approval rate is in that range, human approval is acting as a formality, and the board should know which actions rely on it.

What the board page looks like

Illustrative example: the numbers below are hypothetical and round, to show the layout, not a benchmark.

Indicator                    Q2           Q3           Denominator (Q3)
Inventory coverage           70%          90%          1,000 agent endpoints
Checked-install share        50%          80%          20,000 agent installs
Unknown-artifact rate        30%          25%          16,000 checked installs
Time to exposure answer      48 h         6 h          2 public incidents
Canary pass rate             8 of 10      10 of 10     10 canaries, 1 client version
Human approval rate          95%          95%          5,000 prompts

One line of commentary per row is enough. In this example the story is that coverage and enforcement improved, exposure answers got faster, and human approval still clears nearly everything, so the next quarter's ask is to move more actions behind policy instead of behind a prompt.

Competing explanations to rule out before you report a change

Mix shift. Checked-install share can rise because developers installed less, not because enforcement improved. Report the denominator next to the ratio.
Definition drift. A new agent client changes what counts as an install. Footnote any change in definitions, and restate the prior quarter on the new basis.
Survivorship. Time-to-answer only covers incidents you noticed. Pair it with inventory coverage, which bounds what you could have noticed.

What not to report

Raw CVE counts or alert counts.
"Skills scanned" or "packages analysed" without a denominator.
Percent "safe". No list can support that word.
Vendor-wide breach statistics presented as your organisation's risk.

Limits in plain sight

These six indicators are definitions, not benchmarks. No published dataset gives peer values for them yet, so there are no industry targets to quote. Set internal targets after two quarters of your own data. Indicators 2 and 6 need hook logging, which some agent clients only support through managed settings. Indicator 4 needs at least one real or exercised incident per quarter to produce a number; run a tabletop with a public incident if none occurs.

Decision consequences, and what would falsify the set

Adopt two this quarter: inventory coverage and canary pass rate. Both are cheap and both expose gaps that the other four depend on. Add the rest as logging matures.

The set is wrong if, after two quarters, none of the six has changed a funding or policy decision. In that case they are reporting labour, the very cost Panaseer measured, and should be cut.

The threat.cafe early-access page describes the planned team tier, which logs every agent check in the three-state vocabulary these indicators use.

Sources

#ciso #security-metrics #board-reporting #ai-agents #governance

All posts