---
title: 'Demystifying Agent Skills: Why They Work — Until They Don''t'
description: >-
  Skills help agents by stabilizing action, not by injecting facts. But the same
  abstraction that makes them useful also creates a new failure surface. A deep
  dive into the Princeton study that breaks down 8,135 trials into 12 failure
  modes.
date: 2026-08-18T00:00:00.000Z
heroImage:
  src: /_astro/hero.NfgRxfa4.webp
  width: 1360
  height: 768
  format: webp
tags:
  - agent-skills
  - llm-agents
  - procedural-memory
  - retrieval
  - taxonomy
categories:
  - The Explainer
published: true
featured: true
draft: false
author: Jad
href: /posts/demystifying-agent-skills/
slug: demystifying-agent-skills
---
Do you also feel lost when someone says "agent skills"?
You are not alone. The term sounds precise. It is not.

A skill is a markdown file. It sits in a directory. It tells an
agent what to do, what to check, and what to avoid. That is it.
The pitch is simple — compress messy execution experience into a
reusable artifact. The catch is that nobody had rigorously
explained *when* the artifact helps, *why* it works, or *where*
it breaks.

A team from Princeton, Stanford, USC, and Johns Hopkins just
filled that gap. They ran 8,135 controlled trials across two
agent frameworks, three benchmarks, and four research questions.
They distilled the results into a 12-mode taxonomy of skill
success and failure. The paper is called "Demystifying Agent
Skills: Why They Work — Until They Don't"[^paper], and it
changes how we should think about skill design.

Let's walk through what they found. Not the summary. The
mechanism.

**What's in this post:**

- The one rule that explains skill utility
- The experiment: same experience, three representations
- The trajectory mixture grid: what happens at the extremes
- The 12-mode taxonomy: what skills fix, what they don't
- The group-level shift: SC1, SC2, SC3 and what they mean
- How the taxonomy was built (and why you should trust it)
- The retrieval collapse: precision falls from 29.6% to 3.3%
- The confusability stress test: why similar distractors break
  retrieval
- The recall paradox: agents find the right skill but won't
  commit to it
- The outcome annotation gap: 74.62% vs 40.00%
- Cross-framework transfer: why skills travel better than
  traces
- The token-cost tradeoff: effectiveness vs efficiency
- The lightweight baselines: how much of skill value is just
  structure
- What a good skill actually looks like (the validated
  template)
- The concrete example: a paired trajectory in full
- Implications for skill library design
- The larger lesson

## The One Rule

Here is the rule and I want you to hold it for the whole post:

**Skills stabilize action. They do not inject missing facts.**

The study found that `procedural_anchor` — the mechanism where a
skill gives a usable procedure, ordering, checklist, or tool
sequence — accounts for **65.7%** of skill utility. Explicit
`knowledge_injection` — the skill supplies domain knowledge the
agent otherwise lacked — accounts for **4.5%**.

That ratio is not a rounding error. It is the thesis.

Asking a skill to teach an agent new facts is asking a compass
to draw the map. The compass does not draw maps. It points.
Skills point. They say: run these steps, in this order, check
these things, avoid these pitfalls. The agent already has the
reasoning capacity. The skill gives it a stable path through the
execution.

Notice what this means for skill design. A skill that reads
like a textbook entry — definitions, context, background — is
working against the grain. A skill that reads like a pre-flight
checklist is working with it.

Hold on to this rule. Every subsequent section tests it.

## The Experiment: Same Experience, Three Representations

The study design is clean. The team collected raw agent
trajectories on Terminal-Bench 2.0, Terminal-Bench Pro, and
SkillsBench — three benchmarks that require multi-step terminal
execution, debugging, and verification. Then they held the
source experience fixed and varied only how it was represented:

1. **Raw** — no prior experience. The agent starts from
   scratch.
2. **Workflow Memory** — cleaned procedural traces from prior
   runs, appended to the task instruction.
3. **Skill** — the same traces distilled into a standardized
   `SKILL.md` file placed in the agent's execution environment.

The key control: Workflow Memory and Skill are built from the
*same* source trajectories. The only variable is representation.
If Skill outperforms Workflow Memory, the gain comes from
distillation, not from more experience.

The result: Skill reaches **61.9%** success versus **55.9%**
for Workflow Memory — a **+6.06 point** difference with a 95%
bootstrap confidence interval of [+0.76, +11.36][^website].

And notice: Workflow Memory actually *underperforms* Raw
execution (55.9% vs 59.1%). Giving the agent more procedural
residue — long explorations, failed attempts, low-level
debugging paths — can burden it. The agent drowns in process
noise. Skills strip that noise.

That is the representation argument. It is not "more experience
helps." It is "the same experience, packaged better, helps."

## The Trajectory Mixture Grid

The team did not just compare representations. They varied the
*quality* of the source experience. For each task, they
collected both successful and failed raw trajectories, then
constructed a composition grid from 5-success-0-failure (5s0f)
through 0-success-5-failure (0s5f).

This is a six-point axis. At one end, the skill is built from
five successful runs. At the other, it is built from five
failures. The middle points mix them in decreasing success
ratios.

The data for Codex + GPT-5.3-Codex on Terminal-Bench-2 tells
the story:

| Mix | Workflow | Skill | Raw baseline |
|-----|----------|-------|--------------|
| 5s0f| 44.5%    | 75.5% | 59.4%        |
| 3s2f| 41.9%    | 78.1% | 59.4%        |
| 1s4f| 27.1%    | 71.0% | 59.4%        |
| 0s5f| 28.4%    | 51.6% | 59.4%        |

Read the skill column top to bottom. At 5s0f, skill reaches
75.5%. At 3s2f, it goes *up* to 78.1%. At 1s4f, it drops to
71.0% but still beats raw. At 0s5f — built entirely from
failures — it collapses to 51.6%, below the raw baseline.

Notice what happened at 3s2f. Adding two failed trajectories to
a pool of three successes *improved* skill performance. The
failures were not noise. They were signal. The skill creator saw
what went wrong and encoded the avoidance into the skill.

But at 0s5f, the signal inverts. The skill is built from nothing
but failures. The creator has no positive procedure to distill.
The skill becomes a catalog of what not to do, with no path
toward what to do. That is not a skill. That is a warning
label.

The same pattern holds for Gemini CLI on Terminal-Bench-2. At
5s0f, skill reaches 79.2%. At 0s5f, it drops to 47.7%, below
the raw baseline of 50.0%. The shape is consistent across
agent-model pairings and benchmarks.

The practical lesson: a skill built from pure failures is worse
than no skill at all. A skill built from a mix of successes and
failures can be better than a skill built from successes alone —
*if* the creator knows which traces failed.

Which brings us to the next finding.

## The 12-Mode Taxonomy: What Skills Fix, What They Don't

The aggregate numbers tell us skills help. They don't tell us
*what changes* in the agent's behavior. To answer that, the team
built a contrastive taxonomy. They normalized 8,135 trial
records, open-coded 240 sampled trajectories, retained 238 valid
labels, and merged them into 12 canonical modes across three
categories[^repo]:

**SC1 — Successful procedural anchoring (the wins):**

- `skill_guided_success` — 61.6% of skill-arm cases
- `workflow_guided_success` — 54.5% of workflow-arm cases
- `autonomous_clean_success` — 48.7% of raw-arm cases

**SC2 — Execution-layer and verification failures (what skills
fix):**

- `environment_infrastructure_failure` — drops from 5.3% (raw)
  to 0.2% (skill)
- `output_format_schema_mismatch` — drops from 7.4% to 3.2%
- `background_service_lifecycle_failure` — drops from 2.7% to
  0.8%
- `shell_code_corruption` — drops from 1.1% to 0.2%
- `algorithmic_logic_error` — stays at 7.4% (skills don't fix
  this)
- `static_verification_without_runtime` — stays at 11.7%
  (skills don't fix this either)

**SC3 — Invocation and boundary failures (what skills break):**

- `timeout_budget_exhaustion` — 4.4% for skills, 10.6% for
  workflow memory
- `skill_guidance_misapplied_or_ignored` — 10.0% of skill-arm
  cases, vs 0.8% for raw
- `capability_or_safety_limit` — 0.4% for skills

Let's read this carefully. Skills are very good at fixing
environment setup, output formatting, and service lifecycle.
These are operational fragilities — the kind of failure where
the agent knows *what* to do but keeps stumbling on *how* to set
it up. Once a reliable setup sequence has been discovered, it
compresses cleanly into a skill. The environment failure rate
drops from 5.3% to 0.2%. That is a 96% reduction. Almost
eradicated.

But skills do not fix algorithmic errors. They do not fix
static verification without runtime checks. These failures
require deeper problem reformulation or stronger validation
logic — not procedural anchoring. The skill cannot think for
you. It can only keep you on the path. If the path itself is
wrong, the skill will help you follow it efficiently into the
ditch.

And then there is the new failure surface.
`skill_guidance_misapplied_or_ignored` appears in 10% of
skill-arm cases. The agent has the skill. It reads the skill. It
applies the skill mechanically, misses a condition, or carries
over assumptions that no longer hold. The skill contains
plausible guidance, but the agent fails to decide *when* and
*how* it should govern the current execution.

The dominant narrative is that skills are safe — they are just
markdown files, how much damage can they do? But actually, the
same abstraction that stabilizes action also creates a new
boundary. A skill is not self-executing. The agent must judge
applicability, adapt to context, and know when to abandon the
script.

## The Group-Level Shift: SC1, SC2, SC3

The 12 modes are useful, but the group-level shift tells the
cleaner story. The team tracked how trajectories move between
the three categories across arms:

- **SC1 (success):** 294/528 for Workflow Memory, 326/528 for
  Skill
- **SC2 (execution failure):** 197/528 for Raw, 176/528 for
  Workflow Memory, 124/528 for Skill
- **SC3 (invocation failure):** 19/528 for Raw, 78/528 for
  Skill

Skills don't just fix failures. They move trajectories from
"execution struggle" into "procedural success." The SC2
reduction is 37.3% → 23.5%. The SC1 increase is 55.7% → 61.7%.
But SC3 — the category that barely exists in raw execution —
jumps from 3.6% to 14.8% for skills.

This is the tradeoff in one sentence: **skills eliminate
operational failures by introducing invocation failures.**

Whether that tradeoff is worth it depends on which failure is
cheaper to debug. An environment setup failure is
time-consuming but straightforward — the agent retries, finds
the right package, moves on. A skill misapplication failure is
insidious — the agent appears to follow instructions, produces
plausible output, and fails in a way that is harder to trace
because the skill *looked* like it was helping.

## How the Taxonomy Was Built

A taxonomy is only as credible as its construction. The team did
not write the 12 modes by intuition. They built a pipeline.

First, they normalized 8,135 trial records from the controlled
experiments into a shared manifest — task identity, execution
arm, verifier outcome, injected artifact, and trajectory
transcript.

Then they sampled 240 trajectories using stratified cells
defined by benchmark, setting, arm, and outcome. The
stratification prevents the label pool from being dominated by
the largest benchmark or by one execution arm. Each trajectory
was passed to Claude Sonnet 4.6 through a headless CLI with
tool use and session persistence disabled — forcing the model
to judge only the evidence in the prompt, not external context.

The context budget was fixed: task instruction truncated to
3,000 characters, skill artifact to 3,000, trajectory
transcript to a 6,000-character head and a 12,000-character
tail. The larger tail budget reflects an empirical observation:
final errors, verifier-facing decisions, and timeout behavior
are usually concentrated near the end of the trajectory.

The open-coding stage produced 240 raw labels. Two were
discarded as invalid. The remaining 238 were fed into a
two-round batched induction: first, batches of ~60 records each
proposed 8-14 local modes; second, a merge prompt consolidated
batch-level modes into 9-14 global modes. The final merge
produced 12.

Then came the human validation. For each of the 238 raw labels,
a human annotator inspected three supporting trajectories — 714
trajectory-label checks total. All labels were confirmed as
grounded in the recorded behavior. The annotator then
independently mapped all 238 labels to the 12 canonical modes
using only the taxonomy definitions. The result: **95.8% exact
agreement** with the LLM aggregation, and **Cohen's κ = 0.952**.

That κ score is not a courtesy check. It is near-perfect
inter-rater reliability. The taxonomy is not one model's
opinion. It is a stable mapping that a human reproduces
independently.

The paired comparison stage then constructed 528 triples — one
raw, one workflow-memory, one skill trajectory for the same
task — and asked an LLM judge to assign a mode to each arm,
record pairwise changes, and label the mechanism. This is where
the "skill fixed environment setup that raw encountered" or
"skill introduced a misapplication failure" attributions come
from. The judge was constrained by strict JSON schemas and
required evidence quotes — each mode assignment traceable to a
verbatim snippet from the trajectory, instruction, or skill
artifact.

## The Retrieval Collapse

The retrieval study is the most unsettling part of the paper.

The team tested what happens when skills are selected from a
library rather than handed directly to the agent. They used
SkillsBench, which provides native task-to-skill annotations,
and constructed candidate pools of increasing size and
confusability. Three independent arms:

- **Arm 1** — embedding-based ranking (Qwen3-Embedding-0.6B)
- **Arm 2** — explicit agent selection without execution
- **Arm 3** — full-pool real execution with skill-use parsing

The results:

| Pool size | Arm 1 P | Arm 2 P | Arm 3 P | Arm 3 success |
|-----------|---------|---------|---------|---------------|
| 5         | 88.3%   | 70.0%   | 29.6%   | 36.4%         |
| 10        | 85.5%   | 68.3%   | 15.2%   | 37.3%         |
| 20        | 84.1%   | 67.8%   | 9.4%    | 37.6%         |
| 50        | 81.0%   | 65.5%   | 4.8%    | 38.0%         |
| 100       | 76.9%   | 63.7%   | 3.3%    | 39.3%         |

Read that table twice. Embedding precision drops gently — from
88.3% to 76.9%. That is manageable. Agent selection drops from
70.0% to 63.7%. Also manageable. But Arm 3 — actual-use
precision, the fraction of skills the agent *actually invokes*
during execution that are ground-truth — collapses from 29.6%
to 3.3%.

And downstream success? It stays flat. Around 36-39% across all
pool sizes. Slightly *increases*, if anything.

The missing middle here is obvious once you see it. The agent
inspects multiple candidates. It invokes several. It does not
reliably restrict use to the ground-truth skill. But related
non-ground-truth skills still provide partial procedural
support — a setup step here, a verification check there. The
task succeeds without the "correct" skill because procedural
anchoring is not about finding the perfect artifact. It is about
stabilizing enough of the execution to get through.

This is both reassuring and alarming. Reassuring because the
system is graceful — imperfect retrieval does not collapse
performance. Alarming because it means your skill library's
retrieval quality is almost invisible in your success metrics.
You can have a broken retriever and not know it.

## The Confusability Stress Test

Pool size is not the dominant stressor. Semantic confusability
is.

The team constructed three distractor regimes:

- **Random** — distractors sampled from unrelated skills
- **Dissimilar** — distractors selected from embedding-space
  far-away skills
- **Similar** — distractors selected as embedding-space
  near-neighbors

Arm 1 (embedding) precision at pool size 100:

- Random pools: 84.1%
- Dissimilar pools: 93.2%
- Similar pools: 53.4%

The gap between dissimilar (93.2%) and similar (53.4%) is 40
points. That is not pool size. That is confusability. When
distractors look like the ground-truth skill, the embedding
retriever gets confused. When they look different, it does not.

The same asymmetry appears in agent selection. At pool size 5,
precision on similar pools is 54.3% for Gemini and 51.9% for
Codex. On random pools, it is 74.4% and 81.8%. The agent is
better at rejecting obviously irrelevant skills than at choosing
between semantically adjacent ones.

The practical lesson: if your skill library has many skills that
describe similar procedures with different vocabulary, your
retrieval system will struggle. The fix is not a bigger
embedding model. The fix is either fewer, more distinct skills,
or skill descriptions that emphasize what makes each one
different from its neighbors.

## The Recall Paradox

Here is the strangest finding in the retrieval study. Arm 3
recall — the fraction of ground-truth skills the agent
inspected or invoked at any point — stays at 54.3-73.6% even at
pool size 100, despite precision being only 0.7-8.1%.

That means the agent *did* look at the right skill. It just also
looked at 12-15 wrong ones. And it did not commit.

This combination — high recall, low precision, stable success —
reveals something about how agents use skills. They do not
select-then-execute. They browse. They pull multiple skills into
context. They cherry-pick steps from whichever ones seem
relevant. The ground-truth skill contributes, but so do its
neighbors. The task succeeds not because the agent found the
one right answer, but because enough procedural fragments from
enough skills collectively stabilized the execution.

This is the "partial procedural support" mechanism. It is why
retrieval precision can collapse without success collapsing. And
it has an uncomfortable implication for skill library design:
maybe retrieval quality matters less than people think, and
skill *content* matters more. If the wrong skills still help
because they contain reusable fragments, then the marginal value
of perfect retrieval is lower than the marginal value of better
skill content.

## The Outcome Annotation Gap

The next finding is about how skills are *constructed*, not how
they are used.

The team created two variants of every skill. In the **normal**
setting, the skill creator sees which source trajectories
succeeded and which failed. In the **no-hint** setting, those
labels are removed — the creator sees the same traces but does
not know which ones worked.

When the source pool contains only successful trajectories
(5s0f), withholding labels barely matters. The creator has
nothing to confuse. Every trace is a positive example. The
distillation is clean either way.

But once failed trajectories enter the pool, the gap opens fast.
For Gemini on Terminal-Bench-2 at the 3-success-2-failure
mixture, normal skills reach **74.62%** versus **40.00%**
without outcome hints[^website]. A 34-point gap. The same
pattern holds across all completed Gemini Terminal-Bench-2 and
SkillsBench ratios.

The lesson is conditional, and the condition matters. Outcome
labels are not always important. They are important *when failed
traces are present*. Without the label, the creator cannot
distinguish a procedure that worked from one that failed. It may
distill a failed approach into a "reliable procedure" and encode
the bug as a feature.

The outcome label is not metadata. It is a quality signal that
shapes what gets distilled. If you are building skills from
mixed trajectory pools — and the mixture grid shows that mixed
pools can outperform success-only pools — you must label your
traces. The 34-point gap is the price of not doing so.

## Cross-Framework Transfer

The team built skills and workflow memories from Codex
trajectories, then evaluated them in Gemini CLI. The source
experience was held fixed. The target framework changed —
different prompting style, different tool interface, different
execution loop.

Skills transferred more cleanly than Workflow Memory. The
reason is structural. Workflow Memory preserves trace-level
details — specific commands, framework-specific tool calls,
prompting conventions. When you move to a different framework,
those details become noise. Skills, by contrast, compress the
procedure into framework-agnostic steps. "Check the output
schema before returning" survives a framework switch. "Run
`codex exec --json`" does not.

This is the portability argument for distillation. The more
framework-coupled your procedural memory, the less it travels.
Skills pay an upfront distillation cost to buy portability.

And the implication is practical. If you are building a skill
library that will be used across multiple agent frameworks —
Codex, Gemini CLI, Claude Code, custom scaffolds — distillation
is not optional. Workflow Memory tied to one framework's tool
interface will degrade when the framework changes. Skills that
describe procedures in framework-agnostic language will not.

## The Token-Cost Tradeoff

The team measured token usage on a matched 83-task intersection
where Raw, Workflow Memory, and Skill runs all had complete
metadata. To avoid over-weighting tasks with more completed
trials, they averaged within each task first, then across tasks.

The numbers:

| Representation | Success | Input   | Output | Total   |
|----------------|----------|---------|--------|---------|
| Raw            | 64.1%    | 541.5K  | 14.2K  | 555.7K  |
| Workflow Memory| 64.8%    | 417.9K  | 8.3K   | 426.2K  |
| Skill          | 69.6%    | 511.7K  | 9.8K   | 521.5K  |

Workflow Memory is the most token-efficient representation. It
substantially reduces both input and output tokens relative to
Raw, with nearly unchanged success. Skill improves success by
5.5 points over Raw while still reducing token use by 34.2K. But
Skill costs 95.3K more tokens than Workflow Memory.

The tradeoff: Skill is better but costlier. Workflow Memory is
cheaper but weaker. Raw is both expensive and mediocre.

If your constraint is token budget, Workflow Memory gives you
the best bang per token. If your constraint is success rate,
Skill wins. The 95.3K token premium buys 4.8 points of success.
Whether that is worth it depends on your cost-of-failure. In a
CI pipeline where a failed run means a human gets paged, 4.8
points is cheap. In a high-volume batch job where tokens are the
dominant cost, Workflow Memory may be the better choice.

Notice that Skill's output tokens (9.8K) are close to Workflow
Memory's (8.3K). The premium is almost entirely in input tokens.
Skills are loaded into context; they do not cause the agent to
generate more. The cost is the skill file itself, not the
agent's response to it.

## The Lightweight Baselines

The team tested two compact procedural baselines on 26
Terminal-Bench-2 tasks, 5 trials each, 130 trials per
condition:

- **Short plan** — a concise instruction-derived plan with 3-5
  high-level steps
- **Test-first template** — a workflow-derived validation
  template emphasizing success conditions, intermediate checks,
  and final verification

The results:

| Condition         | Source     | Success rate |
|-------------------|------------|--------------|
| Raw               | None       | 50.0%        |
| Short plan        | Task instr | 47.7%        |
| Test-first        | Workflow   | 59.2%        |
| Workflow Memory   | Workflow   | 62.3%        |
| Skill             | Workflow   | 79.2%        |

The short plan *underperforms* Raw. A vague 3-5 step outline is
worse than no guidance at all. It gives the agent a false sense
of structure without the operational specifics to execute.

The test-first template reaches 59.2% — close to Workflow
Memory's 62.3%. A validation template that emphasizes
intermediate checks and final verification captures most of
Workflow Memory's value. The template is compact, reusable, and
does not require full trajectory preservation.

But the jump from test-first (59.2%) to Skill (79.2%) is 20
points. That gap is the value of structured failure modes and
recovery procedures. The test-first template tells you what to
check. The skill tells you what to check, what goes wrong, how
to detect it, and how to fix it. That last 25% of skill value —
the part that takes you from 59% to 79% — is in the failure
modes section.

If you are building skills and skipping the failure modes
section because it feels like extra work, you are leaving 20
points on the table.

## What a Good Skill Looks Like

The paper's skill-creator prompt — included in Appendix B —
reveals the validated template. A skill is not a free-form
document. It has a structure:

```markdown
---
name: {{skill-name}}
description: {{one-line description}}
---
# {{Skill Name}}
## Use This Skill When
- {{condition 1}}
- {{condition 2}}
## Preconditions
- {{what must be true before starting}}
## Steps
1. {{step 1}}
2. {{step 2}}
## Common Failure Modes To Avoid
- {{failure mode 1: signal and mitigation}}
- {{failure mode 2: signal and mitigation}}
## If A Failure Happens
1. Stop and inspect the latest output.
2. Map the error to the failure modes above
   and apply the fix.
3. Re-run verification before finishing.
## Verify
- {{how to confirm the skill completed successfully}}
```

Notice what is *not* there. There is no "Background" section.
No "Introduction." No "Why This Matters." The skill starts with
applicability conditions, lists preconditions, gives steps,
names failure modes with signals and mitigations, provides a
recovery procedure, and ends with verification.

This is a pre-flight checklist, not a textbook chapter. Every
section does procedural work. If a section does not stabilize
action, it is not in the skill.

The "Use This Skill When" section is the applicability boundary.
It is the defense against the 10% misapplication failure mode.
If the skill says "use when X, do not use when Y," the agent has
a fighting chance of deciding whether to invoke it. If the skill
has no boundary, the agent will apply it whenever it seems
vaguely relevant — and 10% of the time, that will be wrong.

The "Common Failure Modes To Avoid" section is the defense
against the operational failures that skills are so good at
fixing. Each failure mode has a signal (how to detect it) and a
mitigation (what to do about it). This is not "avoid X." It is
"if you see Y, do Z." That is a procedure, not a warning.

The "If A Failure Happens" section is the recovery procedure.
The paper's template has a fixed three-step recovery: stop,
map, re-verify. This is the section that turns a skill from a
script into a resilient procedure. Scripts break on
unanticipated conditions. Resilient procedures have a plan for
when they break.

The "Verify" section is the exit criterion. Without it, the
agent does not know when the skill is done. It may stop early,
or it may keep going past completion. Verification is the
difference between "I ran the steps" and "I completed the task."

## The Concrete Example: A Paired Trajectory

The paper includes a paired trajectory example that makes the
procedural anchoring mechanism visible. The task is
`react-performance-debugging` on SkillsBench. The agent must
fix a Next.js API route that is too slow. The setting is 1s4f —
one successful trace, four failed traces.

**Raw execution (reward 0):**

The agent finds that `/api/products` is returning 500 in
production. It checks server logs, rebuilds, and restarts the
production server. It hits `EADDRINUSE: address already in use
:::3000`. The verifier reports:

- `test_checkout_fast` PASSED
- `test_external_api_actually_called` FAILED
- AssertionError: Checkout API too fast (7 ms) — external API
  may be bypassed

The agent bypassed the external service check to make the route
fast. It optimized the wrong thing.

**Workflow Memory (reward 0):**

The agent identifies the waterfalls — `fetchUserFromService`,
then `fetchConfigFromService`, then `fetchProfileFromService`.
It implements targeted fixes. The checkout patch serializes all
three calls. The verifier reports:

- `test_external_api_actually_called` PASSED
- `test_checkout_fast` FAILED
- AssertionError: Checkout took 915 ms (should be < 800 ms)

The agent preserved correctness but left the dependent profile
request serialized. Checkout takes 915ms. The latency test
fails.

**Skill (reward 1):**

The injected skill says:

> Eliminate server/API waterfalls:
> - Convert independent awaits to `Promise.all`.
> - Start promises early, await late.
> - For partially dependent flows, fetch independent data in
>   parallel, then trigger the dependent fetch as soon as its
>   prerequisite resolves.

The agent starts the profile fetch the moment the user fetch
resolves, parallelizes it with the config fetch, and passes all
11 tests. Warm average: 711ms.

Notice what the skill did. It did not teach the agent about
`Promise.all`. The agent already knew that. It did not explain
what a waterfall is. The agent already identified it. The skill
provided a *procedural anchor* — a specific pattern to apply
("start promises early, await late") that stabilized the
execution and prevented the agent from leaving the dependent
call serialized.

That is the 65.7% mechanism. Not knowledge. Procedure.

And notice what the skill did *not* do. It did not solve the
algorithmic problem. The agent had to figure out which calls
were independent and which were dependent. The skill gave the
pattern. The agent applied the pattern to the specific code.
That division of labor — skill provides pattern, agent provides
adaptation — is the whole point.

## Implications for Skill Library Design

The retrieval study has a counterintuitive implication. If
downstream success stays flat while retrieval precision
collapses, then retrieval quality is not the bottleneck. Skill
content is.

This does not mean retrieval does not matter. It means the
marginal value of improving retrieval from 76.9% to 88.3% is
lower than the marginal value of improving skill content from
generic to structured. If your skills are well-written with
failure modes and recovery procedures, even the wrong skill
provides partial procedural support. If your skills are vague
textbook entries, even the right skill will not help much.

The investment priority is:

1. **Skill content first.** Structured skills with failure
   modes, recovery procedures, and verification criteria.
2. **Applicability boundaries second.** "Use when X, do not use
   when Y" to reduce the 10% misapplication rate.
3. **Retrieval third.** Only invest in retrieval infrastructure
   when your skill pool is large enough that confusability
   becomes the dominant stressor — and even then, focus on
   skill *distinctness* (making descriptions differentiate) not
   on bigger embedding models.
4. **Outcome labels always.** If you are building skills from
   mixed trajectory pools, label your traces. The 34-point gap
   is not a rounding error.

## The Larger Lesson

The larger lesson is that skill use is a lifecycle problem, not
a memory-injection problem. A skill must be represented
correctly, constructed with outcome signals, retrieved from a
confusable pool, invoked at the right time, and adapted during
execution. Each stage has its own failure mode. Aggregate
success rates hide all of them.

The paper's contribution is not "skills work." We knew that.
The contribution is decomposing *how* they work into observable
mechanisms, and *where* they fail into a taxonomy that a human
annotator can reproduce with 95.8% agreement. That decomposition
is what makes skill design a principled practice rather than
heuristic iteration.

From now on, when you write a skill, ask yourself three
questions:

1. **Does it stabilize action?** If it reads like a textbook
   entry, rewrite it as a checklist. The 65.7% vs 4.5% ratio is
   your guide.
2. **Does it name failure modes with signals?** If it says
   "avoid X" without saying how to detect X, add the signal. The
   20-point gap between test-first templates and full skills is
   in the failure modes section.
3. **Does it state when it does not apply?** If it has no
   applicability boundary, the agent will misuse it 10% of the
   time. That is the difference between SC1 and SC3.

The era of writing skills by intuition is closing. The era of
writing them by taxonomy has begun.

---

[^paper]: Jiang et al., "Demystifying Agent Skills: Why They
    Work — Until They Don't," arXiv:2608.14036, August 2026.
    [PDF][arxiv-pdf], [HuggingFace][hf-papers].

[^website]: Project website with interactive results:
    [demystify-agent-skills.github.io][project-site].

[^repo]: Code, artifacts, and reproduction pipeline:
    [github.com/zhiyuanjiang04/demystify-agent-skills][repo].

[arxiv-pdf]: https://arxiv.org/pdf/2608.14036
[hf-papers]: https://huggingface.co/papers/2608.14036
[project-site]: https://zhiyuanjiang04.github.io/demystify-agent-skills/
[repo]: https://github.com/zhiyuanjiang04/demystify-agent-skills