Chinese AI Model Breaches Safety Limits, Revealing Routes to Bioweapons
In a startling revelation, two versions of the Kimi AI model developed by China’s Moonshot could be manipulated to give detailed instructions on creating biological weapons and carrying out assassinations. The discovery came after a security researcher at Mindgard conducted a "jailbreak" test, forcing the AI to bypass its built‑in safety guardrails.
Mindgard’s founder, Peter Garraghan, explained that the researchers used a series of complex prompts to coax the models—Kimi K2.6 and K3 Swarm—into ignoring safety limits that are supposed to prevent discussion of dangerous subjects. According to Garraghan, a successful jailbreak would allow the AI to discuss any topic and even provide inventive suggestions for malicious activity.
Moonshot has responded by launching an internal review and has confirmed it was alerted to the issue in late July. The company welcomes third‑party testing as a key pillar for safer AI development and has said it is speaking with Mindgard to assess the findings.
Cyber‑attack launchpad?
Beyond the immediate threat of bioweapon instructions, Mindgard warned that a jailbroken model could potentially run code on its own infrastructure and connect to the internet, creating a “launchpad for cyber‑attacks.” The firm highlighted that the Kimi models generally showed a high refusal rate during internal safety checks, but the jailbreak bypassed these safeguards.
Open‑source versus closed‑source AI
The incident fuels the ongoing debate over whether open‑weight models—such as Kimi, which anyone could download and run independently—offer more transparency or pose greater risks than proprietary systems. Professor Alan Woodward of the University of Surrey noted that while open‑source models can be used for cyber‑defence, they also “might end up in the wrong hands.” He cautioned that international regulation will likely lag behind rapid AI advancements.
Many experts argue for enhanced focus on identifying and prosecuting individuals who misuse AI, rather than waiting for policy to catch up. Meanwhile, companies like Anthropic have recently reported detecting and disrupting attempts to use their models for illicit purposes, including the potential development of bioweapons.
