Open-weight AI models match closed models in cyberattack tests
New benchmark results show open AI models like DeepSeek achieve comparable or better performance to closed models like GPT-6-Sol Daybreak in simulated network attacks, at a fraction of the cost, challenging cybersecurity assumptions.
Open-weight AI models are now almost as capable of hacking into computer networks as the best closed models, according to new research. Vals, a company that measures AI capabilities, tested AI models on mock corporate networks and found the best frontier models, locked up inside labs and available only through trusted access programs, are now matched by open models that anybody can download. This rapid improvement challenges the security advantage of closed systems and forces defenders to adapt their strategies.
In Vals' CyberOffense benchmark, models were run as autonomous agents in two simulated cyber ranges: Nova, with 21 objectives, and Lighthouse, with 31 objectives. Each model ran in the open-source Pi agent on an isolated Kali Linux machine, with a 48-hour limit and a 500-million-token budget. Vals credited an objective only when hidden markers appeared in the command output, never on the model's own claim.
The results demonstrate that closed models no longer have a definitive edge in cyber skills. OpenAI’s GPT-6-Sol Daybreak model completed about 81 percent of objectives across the two tests. DeepSeek, a Chinese open-weight model, completed about 73 percent. On the Nova range, DeepSeek averaged 16.3 objectives to Daybreak's 15.7, effectively matching it. DeepSeek completed nothing in its first 25 million tokens and made most of its progress after 100 million, leading Vals to conclude that short, compute-capped tests understate cheap open models.
| Model | Avg. Objectives Completed (Nova & Lighthouse) | Cost per Run (Nova) |
|---|---|---|
| GPT-6-Sol Daybreak | 81% | ~$22 |
| DeepSeek V4.1 Flash | 73% | under $3 |
Cost efficiency of open models challenges closed system assumptions
The dramatic price difference between open and closed models suggests open models offer significantly better value for security testing. Vals found that using GPT-6-Sol cost between $17 and $46 in tokens per attack, while DeepSeek cost just $2 to $3. Rayan Krishnan, CEO of Vals, said cheaper open models provide a strong incentive for defenders to adopt them. Other models tested, including Kimi K3.1 Pro, Meta's Wonder, and Grok 4.7, also finished behind DeepSeek in the benchmark.
Industry leaders warn of impending capabilities surge
Anthropic and other companies are preparing for the widespread availability of advanced AI attack capabilities within 6 to 24 months. Logan Graham, frontier red team lead at Anthropic, said they need to prepare for broad availability of these capabilities in that timeframe. This aligns with Anthropic's own strategy, which began a limited release of new models starting with Claude Mythos Preview in April, based on the idea that gated access would buy time for defenders. Project Glasswing at Anthropic provides trusted access to new models for defenders to gain a head start.
Open models identified as key security risk
Experts note that cheaper open models enable more widespread attack capabilities while also providing tools for defenders. Alex Zenla, cofounder and CTO of Edera, said open-weight models can produce advanced attacks, especially with ablated models. He also noted that closed labs may have unreleased models exceeding current open-weight capabilities. AI has automated core hacking tasks, increasing the speed and scale of vulnerability discovery, though experts debate whether it has fundamentally altered cybersecurity defense or just amplified existing challenges.
Performance among other open models varied. Z.ai's GLM 5.3 Flash came last of seven models in Vals' test, completing just five objectives on each range, which Vals called the clearest dead end. However, other evaluations presented a different picture. The firm Irregular found GLM 5.3 completed 3 of 13 multi-stage attack scenarios, the best result it has recorded from an openly available model. NIST called GLM 5.3 the most capable open-weight cyber model released to date and estimated it was about four months behind leading U.S. systems. Anthropic's own assessment said it built working exploits in 12% of attempts, against 14% for its own Claude Mythos model. OpenAI did not respond to a request for comment on the findings. The industry now faces the need to prepare for broad availability of these AI attack capabilities within 6, 12, or 24 months.
