Demystifying Agent Skills: Why They Work — Until They Don't

Do you also feel lost when someone says “agent skills”? You are not alone. The term sounds precise. It is not.

A skill is a markdown file. It sits in a directory. It tells an agent what to do, what to check, and what to avoid. That is it. The pitch is simple — compress messy execution experience into a reusable artifact. The catch is that nobody had rigorously explained when the artifact helps, why it works, or where it breaks.

A team from Princeton, Stanford, USC, and Johns Hopkins just filled that gap. They ran 8,135 controlled trials across two agent frameworks, three benchmarks, and four research questions. They distilled the results into a 12-mode taxonomy of skill success and failure. The paper is called “Demystifying Agent Skills: Why They Work — Until They Don’t”¹, and it changes how we should think about skill design.

Let’s walk through what they found. Not the summary. The mechanism.

What’s in this post:

• The one rule that explains skill utility
• The experiment: same experience, three representations
• The trajectory mixture grid: what happens at the extremes
• The 12-mode taxonomy: what skills fix, what they don’t
• The group-level shift: SC1, SC2, SC3 and what they mean
• How the taxonomy was built (and why you should trust it)
• The retrieval collapse: precision falls from 29.6% to 3.3%
• The confusability stress test: why similar distractors break retrieval
• The recall paradox: agents find the right skill but won’t commit to it
• The outcome annotation gap: 74.62% vs 40.00%
• Cross-framework transfer: why skills travel better than traces
• The token-cost tradeoff: effectiveness vs efficiency
• The lightweight baselines: how much of skill value is just structure
• What a good skill actually looks like (the validated template)
• The concrete example: a paired trajectory in full
• Implications for skill library design
• The larger lesson

The One Rule

Here is the rule and I want you to hold it for the whole post:

Skills stabilize action. They do not inject missing facts.

The study found that procedural_anchor — the mechanism where a skill gives a usable procedure, ordering, checklist, or tool sequence — accounts for 65.7% of skill utility. Explicit knowledge_injection — the skill supplies domain knowledge the agent otherwise lacked — accounts for 4.5%.

That ratio is not a rounding error. It is the thesis.

Asking a skill to teach an agent new facts is asking a compass to draw the map. The compass does not draw maps. It points. Skills point. They say: run these steps, in this order, check these things, avoid these pitfalls. The agent already has the reasoning capacity. The skill gives it a stable path through the execution.

Notice what this means for skill design. A skill that reads like a textbook entry — definitions, context, background — is working against the grain. A skill that reads like a pre-flight checklist is working with it.

Hold on to this rule. Every subsequent section tests it.

The Experiment: Same Experience, Three Representations

The study design is clean. The team collected raw agent trajectories on Terminal-Bench 2.0, Terminal-Bench Pro, and SkillsBench — three benchmarks that require multi-step terminal execution, debugging, and verification. Then they held the source experience fixed and varied only how it was represented:

1. Raw — no prior experience. The agent starts from scratch.
2. Workflow Memory — cleaned procedural traces from prior runs, appended to the task instruction.
3. Skill — the same traces distilled into a standardized SKILL.md file placed in the agent’s execution environment.

The key control: Workflow Memory and Skill are built from the same source trajectories. The only variable is representation. If Skill outperforms Workflow Memory, the gain comes from distillation, not from more experience.

The result: Skill reaches 61.9% success versus 55.9% for Workflow Memory — a +6.06 point difference with a 95% bootstrap confidence interval of [+0.76, +11.36]².

And notice: Workflow Memory actually underperforms Raw execution (55.9% vs 59.1%). Giving the agent more procedural residue — long explorations, failed attempts, low-level debugging paths — can burden it. The agent drowns in process noise. Skills strip that noise.

That is the representation argument. It is not “more experience helps.” It is “the same experience, packaged better, helps.”

The Trajectory Mixture Grid

The team did not just compare representations. They varied the quality of the source experience. For each task, they collected both successful and failed raw trajectories, then constructed a composition grid from 5-success-0-failure (5s0f) through 0-success-5-failure (0s5f).

This is a six-point axis. At one end, the skill is built from five successful runs. At the other, it is built from five failures. The middle points mix them in decreasing success ratios.

The data for Codex + GPT-5.3-Codex on Terminal-Bench-2 tells the story:

Mix	Workflow	Skill	Raw baseline
5s0f	44.5%	75.5%	59.4%
3s2f	41.9%	78.1%	59.4%
1s4f	27.1%	71.0%	59.4%
0s5f	28.4%	51.6%	59.4%

Read the skill column top to bottom. At 5s0f, skill reaches 75.5%. At 3s2f, it goes up to 78.1%. At 1s4f, it drops to 71.0% but still beats raw. At 0s5f — built entirely from failures — it collapses to 51.6%, below the raw baseline.

Notice what happened at 3s2f. Adding two failed trajectories to a pool of three successes improved skill performance. The failures were not noise. They were signal. The skill creator saw what went wrong and encoded the avoidance into the skill.

But at 0s5f, the signal inverts. The skill is built from nothing but failures. The creator has no positive procedure to distill. The skill becomes a catalog of what not to do, with no path toward what to do. That is not a skill. That is a warning label.

The same pattern holds for Gemini CLI on Terminal-Bench-2. At 5s0f, skill reaches 79.2%. At 0s5f, it drops to 47.7%, below the raw baseline of 50.0%. The shape is consistent across agent-model pairings and benchmarks.

The practical lesson: a skill built from pure failures is worse than no skill at all. A skill built from a mix of successes and failures can be better than a skill built from successes alone — if the creator knows which traces failed.

Which brings us to the next finding.

The 12-Mode Taxonomy: What Skills Fix, What They Don’t

The aggregate numbers tell us skills help. They don’t tell us what changes in the agent’s behavior. To answer that, the team built a contrastive taxonomy. They normalized 8,135 trial records, open-coded 240 sampled trajectories, retained 238 valid labels, and merged them into 12 canonical modes across three categories³:

SC1 — Successful procedural anchoring (the wins):

• skill_guided_success — 61.6% of skill-arm cases
• workflow_guided_success — 54.5% of workflow-arm cases
• autonomous_clean_success — 48.7% of raw-arm cases

SC2 — Execution-layer and verification failures (what skills fix):

• environment_infrastructure_failure — drops from 5.3% (raw) to 0.2% (skill)
• output_format_schema_mismatch — drops from 7.4% to 3.2%
• background_service_lifecycle_failure — drops from 2.7% to 0.8%
• shell_code_corruption — drops from 1.1% to 0.2%
• algorithmic_logic_error — stays at 7.4% (skills don’t fix this)
• static_verification_without_runtime — stays at 11.7% (skills don’t fix this either)

SC3 — Invocation and boundary failures (what skills break):

• timeout_budget_exhaustion — 4.4% for skills, 10.6% for workflow memory
• skill_guidance_misapplied_or_ignored — 10.0% of skill-arm cases, vs 0.8% for raw
• capability_or_safety_limit — 0.4% for skills

Let’s read this carefully. Skills are very good at fixing environment setup, output formatting, and service lifecycle. These are operational fragilities — the kind of failure where the agent knows what to do but keeps stumbling on how to set it up. Once a reliable setup sequence has been discovered, it compresses cleanly into a skill. The environment failure rate drops from 5.3% to 0.2%. That is a 96% reduction. Almost eradicated.

But skills do not fix algorithmic errors. They do not fix static verification without runtime checks. These failures require deeper problem reformulation or stronger validation logic — not procedural anchoring. The skill cannot think for you. It can only keep you on the path. If the path itself is wrong, the skill will help you follow it efficiently into the ditch.

And then there is the new failure surface. skill_guidance_misapplied_or_ignored appears in 10% of skill-arm cases. The agent has the skill. It reads the skill. It applies the skill mechanically, misses a condition, or carries over assumptions that no longer hold. The skill contains plausible guidance, but the agent fails to decide when and how it should govern the current execution.

The dominant narrative is that skills are safe — they are just markdown files, how much damage can they do? But actually, the same abstraction that stabilizes action also creates a new boundary. A skill is not self-executing. The agent must judge applicability, adapt to context, and know when to abandon the script.

The Group-Level Shift: SC1, SC2, SC3

The 12 modes are useful, but the group-level shift tells the cleaner story. The team tracked how trajectories move between the three categories across arms:

• SC1 (success): 294/528 for Workflow Memory, 326/528 for Skill
• SC2 (execution failure): 197/528 for Raw, 176/528 for Workflow Memory, 124/528 for Skill
• SC3 (invocation failure): 19/528 for Raw, 78/528 for Skill

Skills don’t just fix failures. They move trajectories from “execution struggle” into “procedural success.” The SC2 reduction is 37.3% → 23.5%. The SC1 increase is 55.7% → 61.7%. But SC3 — the category that barely exists in raw execution — jumps from 3.6% to 14.8% for skills.

This is the tradeoff in one sentence: skills eliminate operational failures by introducing invocation failures.

Whether that tradeoff is worth it depends on which failure is cheaper to debug. An environment setup failure is time-consuming but straightforward — the agent retries, finds the right package, moves on. A skill misapplication failure is insidious — the agent appears to follow instructions, produces plausible output, and fails in a way that is harder to trace because the skill looked like it was helping.

How the Taxonomy Was Built

A taxonomy is only as credible as its construction. The team did not write the 12 modes by intuition. They built a pipeline.

First, they normalized 8,135 trial records from the controlled experiments into a shared manifest — task identity, execution arm, verifier outcome, injected artifact, and trajectory transcript.

Then they sampled 240 trajectories using stratified cells defined by benchmark, setting, arm, and outcome. The stratification prevents the label pool from being dominated by the largest benchmark or by one execution arm. Each trajectory was passed to Claude Sonnet 4.6 through a headless CLI with tool use and session persistence disabled — forcing the model to judge only the evidence in the prompt, not external context.

The context budget was fixed: task instruction truncated to 3,000 characters, skill artifact to 3,000, trajectory transcript to a 6,000-character head and a 12,000-character tail. The larger tail budget reflects an empirical observation: final errors, verifier-facing decisions, and timeout behavior are usually concentrated near the end of the trajectory.

The open-coding stage produced 240 raw labels. Two were discarded as invalid. The remaining 238 were fed into a two-round batched induction: first, batches of ~60 records each proposed 8-14 local modes; second, a merge prompt consolidated batch-level modes into 9-14 global modes. The final merge produced 12.

Then came the human validation. For each of the 238 raw labels, a human annotator inspected three supporting trajectories — 714 trajectory-label checks total. All labels were confirmed as grounded in the recorded behavior. The annotator then independently mapped all 238 labels to the 12 canonical modes using only the taxonomy definitions. The result: 95.8% exact agreement with the LLM aggregation, and Cohen’s κ = 0.952.

That κ score is not a courtesy check. It is near-perfect inter-rater reliability. The taxonomy is not one model’s opinion. It is a stable mapping that a human reproduces independently.

The paired comparison stage then constructed 528 triples — one raw, one workflow-memory, one skill trajectory for the same task — and asked an LLM judge to assign a mode to each arm, record pairwise changes, and label the mechanism. This is where the “skill fixed environment setup that raw encountered” or “skill introduced a misapplication failure” attributions come from. The judge was constrained by strict JSON schemas and required evidence quotes — each mode assignment traceable to a verbatim snippet from the trajectory, instruction, or skill artifact.

The Retrieval Collapse

The retrieval study is the most unsettling part of the paper.

The team tested what happens when skills are selected from a library rather than handed directly to the agent. They used SkillsBench, which provides native task-to-skill annotations, and constructed candidate pools of increasing size and confusability. Three independent arms:

• Arm 1 — embedding-based ranking (Qwen3-Embedding-0.6B)
• Arm 2 — explicit agent selection without execution
• Arm 3 — full-pool real execution with skill-use parsing

The results:

Pool size	Arm 1 P	Arm 2 P	Arm 3 P	Arm 3 success
5	88.3%	70.0%	29.6%	36.4%
10	85.5%	68.3%	15.2%	37.3%
20	84.1%	67.8%	9.4%	37.6%
50	81.0%	65.5%	4.8%	38.0%
100	76.9%	63.7%	3.3%	39.3%

Read that table twice. Embedding precision drops gently — from 88.3% to 76.9%. That is manageable. Agent selection drops from 70.0% to 63.7%. Also manageable. But Arm 3 — actual-use precision, the fraction of skills the agent actually invokes during execution that are ground-truth — collapses from 29.6% to 3.3%.

And downstream success? It stays flat. Around 36-39% across all pool sizes. Slightly increases, if anything.

The missing middle here is obvious once you see it. The agent inspects multiple candidates. It invokes several. It does not reliably restrict use to the ground-truth skill. But related non-ground-truth skills still provide partial procedural support — a setup step here, a verification check there. The task succeeds without the “correct” skill because procedural anchoring is not about finding the perfect artifact. It is about stabilizing enough of the execution to get through.

This is both reassuring and alarming. Reassuring because the system is graceful — imperfect retrieval does not collapse performance. Alarming because it means your skill library’s retrieval quality is almost invisible in your success metrics. You can have a broken retriever and not know it.

The Confusability Stress Test

Pool size is not the dominant stressor. Semantic confusability is.

The team constructed three distractor regimes:

• Random — distractors sampled from unrelated skills
• Dissimilar — distractors selected from embedding-space far-away skills
• Similar — distractors selected as embedding-space near-neighbors

Arm 1 (embedding) precision at pool size 100:

• Random pools: 84.1%
• Dissimilar pools: 93.2%
• Similar pools: 53.4%

The gap between dissimilar (93.2%) and similar (53.4%) is 40 points. That is not pool size. That is confusability. When distractors look like the ground-truth skill, the embedding retriever gets confused. When they look different, it does not.

The same asymmetry appears in agent selection. At pool size 5, precision on similar pools is 54.3% for Gemini and 51.9% for Codex. On random pools, it is 74.4% and 81.8%. The agent is better at rejecting obviously irrelevant skills than at choosing between semantically adjacent ones.

The practical lesson: if your skill library has many skills that describe similar procedures with different vocabulary, your retrieval system will struggle. The fix is not a bigger embedding model. The fix is either fewer, more distinct skills, or skill descriptions that emphasize what makes each one different from its neighbors.

The Recall Paradox

Here is the strangest finding in the retrieval study. Arm 3 recall — the fraction of ground-truth skills the agent inspected or invoked at any point — stays at 54.3-73.6% even at pool size 100, despite precision being only 0.7-8.1%.

That means the agent did look at the right skill. It just also looked at 12-15 wrong ones. And it did not commit.

This combination — high recall, low precision, stable success — reveals something about how agents use skills. They do not select-then-execute. They browse. They pull multiple skills into context. They cherry-pick steps from whichever ones seem relevant. The ground-truth skill contributes, but so do its neighbors. The task succeeds not because the agent found the one right answer, but because enough procedural fragments from enough skills collectively stabilized the execution.

This is the “partial procedural support” mechanism. It is why retrieval precision can collapse without success collapsing. And it has an uncomfortable implication for skill library design: maybe retrieval quality matters less than people think, and skill content matters more. If the wrong skills still help because they contain reusable fragments, then the marginal value of perfect retrieval is lower than the marginal value of better skill content.

The Outcome Annotation Gap

The next finding is about how skills are constructed, not how they are used.

The team created two variants of every skill. In the normal setting, the skill creator sees which source trajectories succeeded and which failed. In the no-hint setting, those labels are removed — the creator sees the same traces but does not know which ones worked.

When the source pool contains only successful trajectories (5s0f), withholding labels barely matters. The creator has nothing to confuse. Every trace is a positive example. The distillation is clean either way.

But once failed trajectories enter the pool, the gap opens fast. For Gemini on Terminal-Bench-2 at the 3-success-2-failure mixture, normal skills reach 74.62% versus 40.00% without outcome hints². A 34-point gap. The same pattern holds across all completed Gemini Terminal-Bench-2 and SkillsBench ratios.

The lesson is conditional, and the condition matters. Outcome labels are not always important. They are important when failed traces are present. Without the label, the creator cannot distinguish a procedure that worked from one that failed. It may distill a failed approach into a “reliable procedure” and encode the bug as a feature.

The outcome label is not metadata. It is a quality signal that shapes what gets distilled. If you are building skills from mixed trajectory pools — and the mixture grid shows that mixed pools can outperform success-only pools — you must label your traces. The 34-point gap is the price of not doing so.

Cross-Framework Transfer

The team built skills and workflow memories from Codex trajectories, then evaluated them in Gemini CLI. The source experience was held fixed. The target framework changed — different prompting style, different tool interface, different execution loop.

Skills transferred more cleanly than Workflow Memory. The reason is structural. Workflow Memory preserves trace-level details — specific commands, framework-specific tool calls, prompting conventions. When you move to a different framework, those details become noise. Skills, by contrast, compress the procedure into framework-agnostic steps. “Check the output schema before returning” survives a framework switch. “Run codex exec --json” does not.

This is the portability argument for distillation. The more framework-coupled your procedural memory, the less it travels. Skills pay an upfront distillation cost to buy portability.

And the implication is practical. If you are building a skill library that will be used across multiple agent frameworks — Codex, Gemini CLI, Claude Code, custom scaffolds — distillation is not optional. Workflow Memory tied to one framework’s tool interface will degrade when the framework changes. Skills that describe procedures in framework-agnostic language will not.

The Token-Cost Tradeoff

The team measured token usage on a matched 83-task intersection where Raw, Workflow Memory, and Skill runs all had complete metadata. To avoid over-weighting tasks with more completed trials, they averaged within each task first, then across tasks.

The numbers:

Representation	Success	Input	Output	Total
Raw	64.1%	541.5K	14.2K	555.7K
Workflow Memory	64.8%	417.9K	8.3K	426.2K
Skill	69.6%	511.7K	9.8K	521.5K

Workflow Memory is the most token-efficient representation. It substantially reduces both input and output tokens relative to Raw, with nearly unchanged success. Skill improves success by 5.5 points over Raw while still reducing token use by 34.2K. But Skill costs 95.3K more tokens than Workflow Memory.

The tradeoff: Skill is better but costlier. Workflow Memory is cheaper but weaker. Raw is both expensive and mediocre.

If your constraint is token budget, Workflow Memory gives you the best bang per token. If your constraint is success rate, Skill wins. The 95.3K token premium buys 4.8 points of success. Whether that is worth it depends on your cost-of-failure. In a CI pipeline where a failed run means a human gets paged, 4.8 points is cheap. In a high-volume batch job where tokens are the dominant cost, Workflow Memory may be the better choice.

Notice that Skill’s output tokens (9.8K) are close to Workflow Memory’s (8.3K). The premium is almost entirely in input tokens. Skills are loaded into context; they do not cause the agent to generate more. The cost is the skill file itself, not the agent’s response to it.

The Lightweight Baselines

The team tested two compact procedural baselines on 26 Terminal-Bench-2 tasks, 5 trials each, 130 trials per condition:

• Short plan — a concise instruction-derived plan with 3-5 high-level steps
• Test-first template — a workflow-derived validation template emphasizing success conditions, intermediate checks, and final verification

The results:

Condition	Source	Success rate
Raw	None	50.0%
Short plan	Task instr	47.7%
Test-first	Workflow	59.2%
Workflow Memory	Workflow	62.3%
Skill	Workflow	79.2%

The short plan underperforms Raw. A vague 3-5 step outline is worse than no guidance at all. It gives the agent a false sense of structure without the operational specifics to execute.

The test-first template reaches 59.2% — close to Workflow Memory’s 62.3%. A validation template that emphasizes intermediate checks and final verification captures most of Workflow Memory’s value. The template is compact, reusable, and does not require full trajectory preservation.

But the jump from test-first (59.2%) to Skill (79.2%) is 20 points. That gap is the value of structured failure modes and recovery procedures. The test-first template tells you what to check. The skill tells you what to check, what goes wrong, how to detect it, and how to fix it. That last 25% of skill value — the part that takes you from 59% to 79% — is in the failure modes section.

If you are building skills and skipping the failure modes section because it feels like extra work, you are leaving 20 points on the table.

What a Good Skill Looks Like

The paper’s skill-creator prompt — included in Appendix B — reveals the validated template. A skill is not a free-form document. It has a structure:

---
name: {{skill-name}}
description: {{one-line description}}
---
# {{Skill Name}}
## Use This Skill When
- {{condition 1}}
- {{condition 2}}
## Preconditions
- {{what must be true before starting}}
## Steps
1. {{step 1}}
2. {{step 2}}
## Common Failure Modes To Avoid
- {{failure mode 1: signal and mitigation}}
- {{failure mode 2: signal and mitigation}}
## If A Failure Happens
1. Stop and inspect the latest output.
2. Map the error to the failure modes above
   and apply the fix.
3. Re-run verification before finishing.
## Verify
- {{how to confirm the skill completed successfully}}

Notice what is not there. There is no “Background” section. No “Introduction.” No “Why This Matters.” The skill starts with applicability conditions, lists preconditions, gives steps, names failure modes with signals and mitigations, provides a recovery procedure, and ends with verification.

This is a pre-flight checklist, not a textbook chapter. Every section does procedural work. If a section does not stabilize action, it is not in the skill.

The “Use This Skill When” section is the applicability boundary. It is the defense against the 10% misapplication failure mode. If the skill says “use when X, do not use when Y,” the agent has a fighting chance of deciding whether to invoke it. If the skill has no boundary, the agent will apply it whenever it seems vaguely relevant — and 10% of the time, that will be wrong.

The “Common Failure Modes To Avoid” section is the defense against the operational failures that skills are so good at fixing. Each failure mode has a signal (how to detect it) and a mitigation (what to do about it). This is not “avoid X.” It is “if you see Y, do Z.” That is a procedure, not a warning.

The “If A Failure Happens” section is the recovery procedure. The paper’s template has a fixed three-step recovery: stop, map, re-verify. This is the section that turns a skill from a script into a resilient procedure. Scripts break on unanticipated conditions. Resilient procedures have a plan for when they break.

The “Verify” section is the exit criterion. Without it, the agent does not know when the skill is done. It may stop early, or it may keep going past completion. Verification is the difference between “I ran the steps” and “I completed the task.”

The Concrete Example: A Paired Trajectory

The paper includes a paired trajectory example that makes the procedural anchoring mechanism visible. The task is react-performance-debugging on SkillsBench. The agent must fix a Next.js API route that is too slow. The setting is 1s4f — one successful trace, four failed traces.

Raw execution (reward 0):

The agent finds that /api/products is returning 500 in production. It checks server logs, rebuilds, and restarts the production server. It hits EADDRINUSE: address already in use :::3000. The verifier reports:

• test_checkout_fast PASSED
• test_external_api_actually_called FAILED
• AssertionError: Checkout API too fast (7 ms) — external API may be bypassed

The agent bypassed the external service check to make the route fast. It optimized the wrong thing.

Workflow Memory (reward 0):

The agent identifies the waterfalls — fetchUserFromService, then fetchConfigFromService, then fetchProfileFromService. It implements targeted fixes. The checkout patch serializes all three calls. The verifier reports:

• test_external_api_actually_called PASSED
• test_checkout_fast FAILED
• AssertionError: Checkout took 915 ms (should be < 800 ms)

The agent preserved correctness but left the dependent profile request serialized. Checkout takes 915ms. The latency test fails.

Skill (reward 1):

The injected skill says:

Eliminate server/API waterfalls:

• Convert independent awaits to Promise.all.
• Start promises early, await late.
• For partially dependent flows, fetch independent data in parallel, then trigger the dependent fetch as soon as its prerequisite resolves.

The agent starts the profile fetch the moment the user fetch resolves, parallelizes it with the config fetch, and passes all 11 tests. Warm average: 711ms.

Notice what the skill did. It did not teach the agent about Promise.all. The agent already knew that. It did not explain what a waterfall is. The agent already identified it. The skill provided a procedural anchor — a specific pattern to apply (“start promises early, await late”) that stabilized the execution and prevented the agent from leaving the dependent call serialized.

That is the 65.7% mechanism. Not knowledge. Procedure.

And notice what the skill did not do. It did not solve the algorithmic problem. The agent had to figure out which calls were independent and which were dependent. The skill gave the pattern. The agent applied the pattern to the specific code. That division of labor — skill provides pattern, agent provides adaptation — is the whole point.

Implications for Skill Library Design

The retrieval study has a counterintuitive implication. If downstream success stays flat while retrieval precision collapses, then retrieval quality is not the bottleneck. Skill content is.

This does not mean retrieval does not matter. It means the marginal value of improving retrieval from 76.9% to 88.3% is lower than the marginal value of improving skill content from generic to structured. If your skills are well-written with failure modes and recovery procedures, even the wrong skill provides partial procedural support. If your skills are vague textbook entries, even the right skill will not help much.

The investment priority is:

1. Skill content first. Structured skills with failure modes, recovery procedures, and verification criteria.
2. Applicability boundaries second. “Use when X, do not use when Y” to reduce the 10% misapplication rate.
3. Retrieval third. Only invest in retrieval infrastructure when your skill pool is large enough that confusability becomes the dominant stressor — and even then, focus on skill distinctness (making descriptions differentiate) not on bigger embedding models.
4. Outcome labels always. If you are building skills from mixed trajectory pools, label your traces. The 34-point gap is not a rounding error.

The Larger Lesson

The larger lesson is that skill use is a lifecycle problem, not a memory-injection problem. A skill must be represented correctly, constructed with outcome signals, retrieved from a confusable pool, invoked at the right time, and adapted during execution. Each stage has its own failure mode. Aggregate success rates hide all of them.

The paper’s contribution is not “skills work.” We knew that. The contribution is decomposing how they work into observable mechanisms, and where they fail into a taxonomy that a human annotator can reproduce with 95.8% agreement. That decomposition is what makes skill design a principled practice rather than heuristic iteration.

From now on, when you write a skill, ask yourself three questions:

1. Does it stabilize action? If it reads like a textbook entry, rewrite it as a checklist. The 65.7% vs 4.5% ratio is your guide.
2. Does it name failure modes with signals? If it says “avoid X” without saying how to detect X, add the signal. The 20-point gap between test-first templates and full skills is in the failure modes section.
3. Does it state when it does not apply? If it has no applicability boundary, the agent will misuse it 10% of the time. That is the difference between SC1 and SC3.

The era of writing skills by intuition is closing. The era of writing them by taxonomy has begun.

---

Footnotes

1. Jiang et al., “Demystifying Agent Skills: Why They Work — Until They Don’t,” arXiv:2608.14036, August 2026. PDF, HuggingFace. ↩

2. Project website with interactive results: demystify-agent-skills.github.io. ↩ ↩²

3. Code, artifacts, and reproduction pipeline: github.com/zhiyuanjiang04/demystify-agent-skills. ↩