Skip to content
T threat.cafe

· Nadia Ferreira-Strand (editorial persona) · Enforcement architecture · on threat.cafe

Prove your agent policy fires: a harmless canary suite for Claude Code hooks and MCP allowlists

A deployed agent policy proves nothing until something it should block has been blocked. Add harmless canary entries to your hook policy and MCP allowlist: a marker command, a package name in a scope you own, a server you never approved. Then test that each one is stopped. Run the offline tests in CI on every policy change and the live tests after every agent client upgrade. Several documented settings fail open without an error, so a policy that worked last month can quietly stop working. The canary result is your audit evidence.

Outcome, time and access

Outcome. A ten-case canary suite, an offline test script, and a dated evidence log.

Time. About a day to build. Ten minutes per release to run.

Access you need.

  • Write access to the repository that holds your hook scripts and policy files.
  • A staging machine that receives the same managed settings as developer laptops.
  • Admin rights to your GitHub Copilot enterprise managed settings, if you use Copilot.

Why controls fail without telling you

The fail-open paths are in the vendors' own documentation. Anthropic's Claude Code hooks reference says exit code 2 blocks a PreToolUse call, but "Claude Code treats exit code 1 as a non-blocking error, even though 1 is the conventional Unix failure code." It also says "A timed-out command, http, or mcp_tool hook doesn't block the tool call." Blocking on failure needs "onFailure": "block", available from Claude Code v2.1.295. A matcher written as mcp__memory "is compared as an exact string and matches no tool"; you need mcp__memory__.*.

GitHub's MCP allowlist changelog of 6 August 2026 is better on one point: "Policies fail closed, meaning a malformed or unverifiable configuration is blocked rather than allowed." It is clear on another: matching on serverName is "only supplied as a convenience, not a security control, since users can rename servers."

None of these failures throws an alert. A canary is the only way to see them.

Use this runbook if / do not use it if

Use it if you enforce agent policy with Claude Code hooks, Claude Code managed MCP settings, or Copilot enterprise MCP allowlists.

Do not use it as a substitute for reviewing the policy itself. A canary proves the block path works. It does not prove the list behind it is right.

Rules for a safe canary

  1. Harmless if it runs. If the control fails, the canary executes. Pick actions whose worst case is a temp file.
  2. Unsquattable. Never use an unclaimed public package name. Someone can register it after you publish your test. Use a scope your organisation owns on npm, such as @yourorg/tc-canary, and run it with --dry-run --ignore-scripts.
  3. Unique marker. Use a string that appears nowhere else, such as tc-canary-, so log searches return only test events.
  4. No real indicators. Never use a real malicious package name, hash or domain as a canary.
  5. Observable result. Each canary must leave or not leave a checkable trace.

The canary suite

ID   Control under test               Canary action                                       Expected      Evidence
C0   Hook does not over-block         Bash: ls                                            allowed       exit 0
C1   Hook blocks a Bash marker        Bash: touch /tmp/tc-canary-c1                       blocked       file absent
C2   Exit-code path                   Hook variant that exits 1 on match                  NOT blocked   proves test sees fail-open
C3   Timeout fails closed             Hook sleeps past a 2 s timeout, onFailure block     blocked       file absent
C4   Missing hook fails closed        Hook command path renamed, onFailure block          blocked       file absent
C5   Package policy                   npm install --dry-run --ignore-scripts @yourorg/tc-canary   blocked   hook log entry
C6   MCP matcher covers plugin tools  tool_name mcp__plugin_acme_canary__echo (offline)   blocked       exit 2
C7   Managed hooks survive users      user settings: disableAllHooks true, then C1        blocked       file absent
C8   MCP allowlist denies unknown     Copilot: add stdio server, command /usr/bin/true    refused       client policy message
C9   Allowlist ignores renamed names  Same server renamed to an approved serverName      refused       client policy message

C2 is a test of your test. It must show the canary running. If it is blocked, your test script is not measuring what you think.

Step-by-step

  1. Add the canary entries to policy. Put tc-canary- and @yourorg/tc-canary in the hook's block list. Verify: grep tc-canary on the deployed policy file returns both lines.
  2. Run the offline tests. The hooks reference notes you can run a hook by hand by piping a sample payload to it. This script does that for C0, C1, C5 and C6:
#!/usr/bin/env bash
# canary-hooks.sh: offline canary tests for a PreToolUse policy hook
set -u
HOOK="${1:?usage: canary-hooks.sh /path/to/hook}"
[ -x "$HOOK" ] || { echo "FAIL: hook not executable: $HOOK"; exit 1; }
command -v timeout >/dev/null || { echo "FAIL: timeout(1) not found"; exit 1; }
pass=0; fail=0
run_case() {
  local id="$1" want="$2" payload="$3" got
  printf '%s' "$payload" | timeout 10 "$HOOK" >/dev/null 2>&1
  got=$?
  if [ "$got" -eq "$want" ]; then
    echo "PASS $id (exit $got)"; pass=$((pass+1))
  else
    echo "FAIL $id (want $want, got $got)"; fail=$((fail+1))
  fi
}
run_case C0 0 '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"ls"}}'
run_case C1 2 '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"touch /tmp/tc-canary-c1"}}'
run_case C5 2 '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"npm install --dry-run --ignore-scripts @yourorg/tc-canary"}}'
run_case C6 2 '{"hook_event_name":"PreToolUse","tool_name":"mcp__plugin_acme_canary__echo","tool_input":{}}'
echo "passed=$pass failed=$fail"
[ "$fail" -eq 0 ]

Verify: the script exits 0. A timeout exit code of 124 means the hook hung, which is a failure. If your hook denies with JSON and exit 0 instead of exit 2, change the check to read permissionDecision from stdout.

  1. Wire it into CI. Run the script on every change to the hook or its policy file. Do not merge on failure.
  2. Run live canaries on staging. On a machine with production managed settings, ask the agent to run C1, C3, C4 and C7 by name. Then check for /tmp/tc-canary-* files. Verify: no files exist. Delete any you find and treat them as a failed control.
  3. Run the allowlist canaries. In Copilot, add the C8 server, then C9. Verify: both are refused. If C9 loads, your allowlist matches on serverName. Change it to serverCommand for local servers or serverUrl for remote ones.
  4. Test a broken policy in staging. Deploy a deliberately malformed policy. GitHub documents fail-closed behaviour. Anthropic's managed settings page says an invalid allowedMcpServers value is "Enforced as an empty allowlist until the value is fixed". Verify: user-added servers are refused. Then restore the valid file.
  5. Log the evidence. Record date, client version, canary ID, expected result, observed result and the tester's name.

Three common failure modes

C1 passes offline but fails live. The hook is not deployed where you think, or a user-level copy overrides it. Fix: deploy through managed settings and set allowManagedHooksOnly. Re-run C7.

C3 or C4 lets the canary run. onFailure is missing or the client is older than v2.1.295. Fix: add "onFailure": "block", set a short explicit timeout, and pin the minimum client version.

C6 fails. The matcher names the server without __.*, or ignores plugin-bundled servers, which use mcp__plugin_<plugin-name>_<server-name>__<tool>. Fix: match on mcp__.* and decide inside the script.

Rollback and health check

Canary entries can block a real workflow if a marker collides with a real name. Keep the marker unique and the canary scope private. If a canary blocks legitimate work, remove that entry, not the hook.

Health check each quarter: all ten canaries pass on the current client version, the evidence log has an entry for every client upgrade, and C2 still shows the fail-open path the suite exists to catch.

The threat.cafe early-access list ships with a local hook designed to exit 2 on a match, and canaries like these are how you should check it.

Sources

#claude-code #hooks #mcp-allowlist #control-validation #canary

All posts