Fixing flawed agent skills helps, but how much depends on the model reading them

A new preprint finds that counting the inconsistencies in an AI agent's "skill" files is a weak guide to how much they will hurt, because the damage depends on how the model reacts once it reads the flaw.

Agent skills are reusable packages, typically a documentation file plus scripts and configuration, that tell an AI agent how to do a job. When those files disagree, for example documentation promising a feature the code never provides, the authors call it a contract inconsistency. Researchers at Huazhong University of Science and Technology audited 575 skills from GitHub and a benchmark called SkillBench and report 1,182 manually confirmed inconsistencies. About two thirds of skills had at least one: 67.4% of GitHub skills and 64.4% of SkillBench skills.

What they tested. The team built 49 runnable tasks, each designed around one specific inconsistency. They ran three models (Claude Haiku 4.5, Qwen3.6-35B and DeepSeek-V4-Flash) three times each under three conditions: no skill, the original flawed skill, and a minimally repaired skill. Scoring used programmatic checks, not an AI judge.

What they found. Repair helped every model, but unevenly. Average pass rates rose by 0.136 for Haiku, 0.085 for Qwen and 0.023 for DeepSeek. Agents read the skill in 97.1% of runs, so the gap was not about whether the flaw was seen. It was about what happened next. Haiku followed the flaw into a failed result in 30% of runs, Qwen in 24% and DeepSeek in 12%, and DeepSeek recovered in 56% of the runs where it met the flaw. Most harm came from "over-promised" flaws, where documentation describes a capability the code lacks: 41 of the 97 failures that followed a flaw.

The paper does not say flawed skills are harmless, only that the raw flaw count is a poor proxy for risk.

What remains uncertain. This is one preprint, submitted to arXiv on 13 October 2026, and nothing we found indicates it has been peer reviewed. The authors caution that the tasks were built to trigger each flaw, so the high exposure rate does not reflect everyday use, that repairs might have side effects, and that results should not be generalised to every inconsistency type. Only three models were tested, one in 4-bit quantised form. Guth News found no independent write-up, reply or replication of this paper, so everything here rests on the authors' own account.

Why it matters. Teams that audit skill libraries with static scanners may be tempted to rank them by defect count. This study suggests checking which defects agents actually act on, starting with promises the code does not keep, and pairing static checks with runtime tests.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. Correction: the earlier version said static skill inconsistencies rarely cause performance loss. The paper reports that repairing them raised pass rates for all three models it tested; its point is that the number of inconsistencies is a poor guide to how much harm they do. The earlier version remains available as revision 1.