synodic-ai AI SECURITY
AI Security Desk · Adversarial Evaluation

A Check That Passes Once: Exploit-Guided Patching, Composed Skill Attacks, and Ten-Query Jailbreaks

2026-10-15

Three preprints from 2026-10-14 describe the same weakness from three directions: a security check that is run once, against fixed evidence, says little about an adversary who gets to try again. A patch validated against the exploit that motivated it, a skill audited one package at a time, and a model's refusals measured against a fixed prompt set can each pass the check they were given and still fail against an attacker who adapts.

The shared failure: validating against the evidence you started from

The P2Patch authors state the problem directly for automated program repair. Language-model agents that patch vulnerabilities through multi-step reasoning and tool use typically validate a patch against the same evidence used to construct it: a static alert, a sanitizer trace, a published exploit, or a test suite. A patch that blocks that one piece of evidence can look successful while the program remains vulnerable to the same flaw reached a slightly different way.

The same structure appears in the other two papers. CoRedSkill shows that agent skills audited one package at a time can each look benign while their composition carries out an unauthorized operation. JSB shows that an attacker who receives graded, multi-dimensional feedback from a judge reaches an average attack success rate of 89.8% within a budget of ten queries to the target. In each case the defender's check is fixed and the adversary's next attempt is not.

P2Patch: treat every patch as a new attack target

P2Patch is a generative adversarial repair system. Starting from a normalized vulnerability finding, an exploitation agent constructs a reproducing proof of vulnerability (PoV), a patching agent repairs the program, and the exploitation agent then attacks the patched program again, looking for a distinct exploit of the same vulnerability. Every successful exploit becomes a permanent repair requirement, so the final patch must block all PoVs accumulated during the process. The authors keep construction-time PoVs strictly separate from the held-out oracles used for evaluation, which is what makes the comparison meaningful: the system is not graded on the evidence it trained its patch against.

Across 101 C/C++ and Java CVE subjects, the adversarial loop surfaces a distinct, confirmed exploit in 41% of runs after the patch has already blocked its original construction-time PoV. That figure is the paper's central measurement of the problem: in roughly two runs out of five, a patch that would have passed single-evidence validation was still exploitable. Against single-pass repair from the same finding by an off-the-shelf coding agent, P2Patch raises held-out fix-closed coverage from 78.1% to 84.7% and beyond-upstream coverage from 42.1% to 62.5%. Under the same held-out oracles, the authors report that it also certifies more attack paths as blocked than recent state-of-the-art agentic repair systems on overlapping subjects, and they conclude that adversarial testing and independent security oracles expose repair gaps that one-shot validation misses.

CoRedSkill: the intent lives in the composition

Agents extend their capabilities through reusable skills that package instructions, executable code, and resources. Composing skills lets an agent run complex workflows, but, as the CoRedSkill authors argue, dependencies between skills can also connect malicious operations across packages, so that the combined intent is hard to identify by auditing each package on its own.

CoRedSkill is a compositional red-teaming method built to test that claim. It decomposes an existing workflow into interdependent skills and embeds attack operations inside task functions by rewriting their abstract syntax trees. Skill instructions guide the agent to select and invoke the relevant skills by itself, and the workflow's dependencies and intermediate artifacts connect the injected operations while the task runs. Each skill keeps its original utility.

The authors built a benchmark of 150 cases covering 50 tasks and three attack objectives, scored on attack success, task completion, and audit detection. In the main evaluation CoRedSkill reaches 98.0% attack success and 95.3% task success with a 3.3% detection rate, and a joint success rate of 90.7%, meaning attack success, task completion, and audit evasion in the same case. The four baselines reach between 6.7% and 42.7% on the same joint measure. The conclusion the authors draw is that skill compositions can complete legitimate tasks while carrying out unauthorized operations that evade auditing.

JSB: a richer score makes the attacker cheaper

Automated jailbreak attacks optimize adversarial prompts against evaluation feedback. The JSB authors observe that existing methods often lack fine-grained diagnostic information about why an attempt failed. Their Jev-Guided Strategy Bandit uses multi-dimensional probabilistic feedback to guide prompt optimization, combining continuous jailbreak scoring with targeted refinement, strategy switching, and reseeding from the best state found so far, so that the search balances exploring new strategies against refining a promising one.

The evaluation covers two harmful-request datasets across five open-source and three closed-source language models. With a budget of only ten queries to the target model, JSB reaches an average HarmBench attack success rate of 89.8% across ten open-source settings and consistently outperforms existing automated jailbreak attacks; further analyses report that it stays effective under different query budgets and attacker models. The authors' defense experiments also reveal limitations in existing filtering mechanisms and point to output filtering based on the same multi-dimensional scoring as a possible counter.

What the three results share, and where each stops

All three are preprints, and every number above is the authors' own, measured on their chosen subjects, benchmarks, and baselines. P2Patch reports results on 101 C/C++ and Java CVE subjects; CoRedSkill on a benchmark its authors constructed; JSB on HarmBench under a fixed query budget. None of them claims a general guarantee.

Read together, they make one structural point. A defense evaluated against a fixed artifact, whether the original PoV, a single skill package, or a fixed prompt set, measures resistance to that artifact. P2Patch turns the adversary into part of the repair loop and still finds new exploits in 41% of runs. CoRedSkill shows the per-package audit missing what only exists across packages. JSB suggests that the quality of the feedback signal, not only the number of attempts, governs how quickly an attacker converges. The defensive lesson in each paper is the same: the check has to adapt as the attacker does, or it has to look at the composed system rather than the part.

What security teams can take from this

None of these papers is a product, but each suggests a concrete change to how existing controls are validated.

Sources