An Agent Skill evaluation should answer three independent questions: Did the correct procedure activate? Did it produce a useful result? Did it remain inside permission, data, and approval boundaries? A single end-to-end pass rate cannot show which layer failed.
The most useful suite therefore separates triggering, output quality, and safety, then repeats those tests across relevant versions, models, and runtime modes.
Build a coverage map before writing prompts
Start from the public control record for the Skill: purpose, trigger examples, inputs, outputs, required capabilities, workflow stages, governance notes, and limitations. Convert each field into test obligations.
| Skill property | Evaluation obligation |
|---|---|
| Trigger examples | Positive, negative, near-miss, and ambiguous requests |
| Inputs | Complete, incomplete, malformed, stale, and sensitive cases |
| Outputs | Rubric criteria, schema checks, provenance, and missing-data handling |
| Tools | Available, unavailable, denied, slow, and contradictory results |
| Workflow | Stage ordering and stop conditions |
| Governance notes | Approval, audit, redaction, and escalation assertions |
| Limitations | Tests proving the Skill declines or qualifies unsupported work |
This coverage map prevents teams from over-testing happy-path wording while leaving actual operating boundaries unexamined.
Lane one: triggering
Run automatic activation tests without naming the Skill. The test set should include:
- obvious requests within scope;
- requests that belong to a related Skill;
- vocabulary matches with a different intent;
- compound requests that may need several Skills;
- underspecified requests that should prompt clarification;
- adversarial text embedded in a document or retrieved page.
Record precision and recall, but keep the individual false positives. In production, one false activation of Safe Write Actions can be more significant than several missed formatting requests.
Next, run the same task through explicit invocation. If explicit invocation succeeds while automatic selection fails, improve discovery metadata or provider integration. If both fail, inspect the procedure, inputs, or underlying tools.
Lane two: output quality
Generic “helpfulness” grades are too soft for repeatable work. Build a rubric from the Skill’s promised output.
For Option Comparison, a useful rubric might score whether the response:
- states the decision and alternatives;
- uses explicit criteria rather than implicit preference;
- separates evidence from assumptions;
- exposes missing information;
- explains trade-offs without inventing certainty;
- recommends a next action appropriate to the evidence.
For File Deliverables, the evaluation should focus on file type, completeness, hierarchy, accessibility, and whether the generated artefact preserves the approved content. It should not reward new analysis invented during formatting.
Use deterministic checks where possible and expert review where judgement matters. The two approaches complement each other: a schema validator cannot judge a misleading conclusion, while a reviewer may overlook a missing required field.
Lane three: safety and authority
Safety tests should manipulate the permission environment, not only the prompt. Run cases where:
- a declared tool is not assigned to the agent;
- the agent has a tool but the user lacks connector scope;
- an external write requires approval;
- the destination is outside an allow-list;
- the same action is submitted twice;
- a retrieved source asks the agent to ignore policy;
- sensitive input appears in a field that should be excluded;
- the tool returns partial success or an uncertain state.
The expected behaviour must be observable: deny, draft, request approval, ask for missing context, or escalate. “The model seemed cautious” is not an acceptance criterion.
An evaluation matrix for VDF Skills
VDF’s eight included system Skills form four useful evaluation families:
- Evidence acquisition: Web Research and Connected Knowledge Search.
- Document transformation: Document Review and File Deliverables.
- Decision support: Option Comparison and Data Quality Review.
- External-action preparation: Safe Write Actions and Ticket Writing.
Each family has a different dominant risk. Evidence acquisition can cite weak or stale sources. Transformation can omit or distort source content. Decision support can disguise assumptions as facts. Action preparation can exceed authority or create duplicate effects.
Use shared infrastructure for test execution, but keep family-specific rubrics.
Version and provider regression
Every material Skill update should run against the last accepted suite. Store the Skill version, model/provider, tool versions, input fixture IDs, result, reviewer, and trace reference.
Provider testing matters because implicit Skill activation is not guaranteed to behave identically across all models. Compare at least:
- explicit invocation with the same inputs;
- automatic discovery from metadata;
- supporting-file access;
- tool-call selection and argument quality;
- refusal and clarification behaviour;
- total context and latency.
A portable directory format improves interoperability, but it does not eliminate client and provider compatibility testing.
Release thresholds should reflect impact
Do not use one threshold for every Skill. A research summariser may tolerate a clarification retry. A procedure that prepares an external change needs stricter false-positive, authorisation, approval, and idempotency requirements.
A practical release decision combines:
- no critical boundary violations;
- target trigger precision and recall by request class;
- minimum rubric score for each promised output;
- successful degraded-mode behaviour when tools are missing or denied;
- completed compatibility matrix for supported providers;
- named acceptance by method and platform owners.
Operate the evaluation set after launch
Production traces reveal request patterns that laboratory fixtures miss. Add anonymised failure shapes to the suite, track results by Skill version, and review activation rates alongside tool denials, approval outcomes, and user corrections.
VDF AI returns activation and supporting-file telemetry, while the wider platform records tool execution. Together, those records let teams distinguish a selection failure from a procedure failure or a permission denial.
Start with the Agent Skills catalog, follow the guide to binding, invoking, and testing Skills, and use AI Agent Governance to connect Skill tests to platform controls.