Bad agent skills did not trip up every model equally, preprint finds

Repairing contradictions in AI agents’ reusable “skill” packages improved results for all three models in a study posted to arXiv on October 13, 2026—but the gains varied considerably. The researchers argue that a tally of defects misses something important: how the agent responds when it encounters them.

Skills bundle instructions with resources such as scripts and configuration files. Those pieces can disagree. Documentation might advertise a feature the code does not provide, for instance, or give conflicting dependency requirements. In one example described in the paper, a visualization skill names D3.js v6 in one place and d3@7.9.0 in another.

The team, from Huazhong University of Science and Technology, reports finding 1,182 manually validated inconsistencies across 575 skills collected from GitHub and SkillBench. To examine their effects, the researchers built 49 executable tasks around selected inconsistencies, comparing agents given no skill, the original skill or a minimally repaired version.

Claude Haiku 4.5 showed the largest reported improvement after repairs, followed by Qwen3.6-35B; DeepSeek-V4-Flash gained much less. The execution records offered a more revealing contrast: the authors classified Haiku as following an inconsistency into failure in 30% of runs, compared with 24% for Qwen and 12% for DeepSeek. DeepSeek recovered in 56% of runs where it was exposed to an inconsistency, according to the paper.

Simply overlooking the skill was not the main explanation, the authors say. Agents read the skill in 97.1% of runs. The important divergence came afterward—whether they carried the faulty instructions through to the result or managed to work around them.

That does not make flawed skills harmless: repairs helped every model tested. Nor is this preprint a general reliability ranking. Its runtime tests were constructed to make selected inconsistencies reachable and checkable, rather than to measure their frequency in everyday use. The practical distinction is between finding a contradiction in a package and showing what a particular agent does with it.