With a standardized “Resilience Gap” metric, MLCommons introduces a common framework to stress-test AI models under adversarial pressure, and it invites the community to close the gap.
Safety Is Not Static; Robustness Must Be Measured
Over recent years, AI developers have layered safety systems, filters, and alignment mechanisms around large models. But those defenses often assume well-behaved inputs. The real test happens when an attacker exploits weaknesses.
MLCommons just released the Jailbreak Benchmark v0.5, which establishes a quantitative “Resilience Gap” metric. The resilience gap is defined as the drop in safety performance when a model is under attack vs. its baseline. This is not merely a benchmark: it reframes AI safety as a continuous arms race.
Why We Needed This Benchmark
Deploying AI in high-risk settings (healthcare, finance, critical infrastructure) demands that models remain safe even when attackers probe for weaknesses. A model that performs well under benign conditions but generates unsafe outputs under subtle attacks is a major liability.
Until now, many defenses and red-teaming efforts were private, bespoke, and differed across models or vendors. There was no shared metric for how much safety degrades under adversarial conditions. The Jailbreak Benchmark v0.5 is designed to address that blind spot.
What Jailbreak Benchmark v0.5 Does
The benchmark follows a three-phase process:
- Baseline Safety Evaluation – Models are first tested on benign input sets to obtain a baseline safety score.
- Adversarial / Jailbreak Testing – The same models face curated jailbreak prompts that use techniques (role-play, obfuscation, prompt chaining, etc.) designed to force unsafe behavior.
- Compute Resilience Gap – The Resilience Gap is the difference between baseline and post-attack performance, evaluated with the same evaluator, so comparisons remain consistent.
In this initial release, v0.5 covers 39 text-to-text models and 5 text+image → text systems. MLCommons also publishes a taxonomy of 13 hazard categories, with 7 covered tests in this version. (Totaling ~43,090 prompt items) (arXiv)
They include a grading and aggregation scheme, and they release ModelBench — a platform and toolset to run the tests and report on performance. (MLCommons)
Key Takeaways from the Results
- Every model degrades under attack. Most of the tested systems performed worse in the adversarial phase than under their baseline.
- Magnitude of breakdown matters. Many text models fell by ~20 percentage points; multimodal models dropped even more in some cases.
- Vulnerability is heterogeneous. Different models fail under different attack types and hazard categories.
- Baseline safety is not a guarantee. A high score behind closed doors doesn’t assure resilience in real use.
These findings clearly illustrate the urgent need to measure resilience, not just perimeter safety.
Why This Matters Now
Just like with all new products and ideas, we have to ask “why?” and “why now?”
- Operational Risk Becomes Quantifiable – Deployers of AI now have a defensible metric, the Resilience Gap, to integrate in risk models, audit processes, and governance reporting.
- A Common Ground for Accountability – Vendors and researchers can no longer claim safety without showing resilience data. The benchmark aligns the language of security across the ecosystem.
- Defense Innovation Incentive – Because the benchmark is open, anyone can propose new attacks or mitigations and test them against the same standard. That accelerates innovation.
- Governance Alignment – The benchmark helps compliance teams and standard bodies (e.g. ISO/IEC 42001, AI risk frameworks) get objective data instead of opaque claims.
- Community Power – Because contributions are open, individuals or smaller organizations can impact its trajectory (submit new attack types, extend to new modalities, or refine evaluation).
Standards-driven innovation has been the foundation of some of the most impactful technologies in modern times. This is where we “slow down to speed up” which is particularly important with security and compliance.
How You Can Contribute Today
There are many ways that you can contribute to this initiative. Whether you are an individual contributor as a consultant, or a large-scale vendor building out AI-infused solutions.
| You Are… | What You Can Do | Why It Matters |
| Attack Researchers | Submit new jailbreak strategies (e.g. cross-modal, obfuscated prompts) | Broaden the benchmark’s coverage and expose hidden weaknesses |
| Model Builders / Vendors | Run your models through v0.5, analyze Resilience Gaps, publish defenses | Improve your product trustworthiness and set higher standards |
| Infrastructure / DevOps Teams | Embed the benchmark in CI / regression testing | Detect regressions early and guard against safety drift |
| Academic & Lab Researchers | Join Working Groups, publish studies using the benchmark | Influence next versions and build your research reputation |
| Application Developers / End-users | Ask vendors for Resilience Gap data, demand transparency | Shift market pressure toward robust, not just “safe,” models |
MLCommons specifically solicits help in these areas: security workstreams (new attack vectors), analytics & scaling, multimodal expansion, and feedback toward v1.0.
Risks, Challenges & Next Steps
How can we assure success with this initiative? AI deployments are not slowing down. The AI technology community has an opportunity to build in more AI safety and security, provided we follow a pragmatic approach.
- Evaluator quality is critical – If the core evaluator is inconsistent or biased, the results lose credibility.
- Coverage remains narrow – v0.5 only tests a small slice of possible attacks and modalities; many real-world cases may escape its net.
- Benchmark overfitting risk – Vendors might optimize specifically for the benchmark rather than for general robustness.
- Multilingual and cultural gaps – The current version is limited (English, general chat); its findings may not generalize globally.
- Model evolution is continuous – Attacks and defenses will coevolve; the benchmark must evolve, version, and backtest.
MLCommons is already planning a v1.0 release that increases rigor, broadens modalities, supports more personas and use cases, and strengthens the evaluator. (MLCommons)
From “Safe Enough” to “Robust by Design”
Jailbreak Benchmark v0.5 catalyzes a shift: we must stop treating safety as a checkbox and start treating resilience as a measurable, evolving objective. For AI to truly scale into higher-risk systems, resilience must be built in, tested continuously, and made transparent.
This is a rare moment: the community, vendors, and researchers can shape the benchmark’s trajectory. Together, we can:
- Attack, defend, publish.
- Embed resilience testing.
- Demand accountability.
The next decade of AI depends on models that do more than just behave well. AI safety depends on models that are built to resist the pressure of continuous, active threats.
