Explaining LLM Insecurity: Why We Can Jailbreak Every Major Model

The issue of LLM security isn’t just about being jailbroken. The reality was that they were never properly built.

That’s why CyberArk Labs researchers developing Fuzzy AI, a framework that claims to jailbreak virtually every large language model on the market, is a major wake-up call. From ChatGPT to Claude, from Gemini to the latest programming assistants, none have proven immune to exploitation.

Why LLM insecurity matters

Jailbreaking LLMs is not the same as doing it to a phone or an application.

Eran Shimony, principal security researcher at CyberArk, who has spent the last two years diving deep into AI security after six years of traditional vulnerability research, tells the difference: “For instance, you’re using Copilot in your organization, and you use it to automate some tasks about budget management. If I’m able to trick Copilot by jailbreaking it and causing it to follow different instructions than the instructions it’s supposed to follow, and you trust the data without a second authentication or review, the results can be quite catastrophic.”

Yet the implications are immediate and tangible, especially as enterprises rush to integrate AI agents into critical business processes without understanding the security implications. Shai Dvash, CyberArk’s senior software engineer for AI engineering working on the Fuzzy AI project, puts it clearly: “LLM integrations should only be used for non-critical functions in organizations, only because it’s in the baby phase of the technology. People don’t quite understand it and don’t understand the risks behind it.”

The academic-reality gap

Then there is the disconnect between academic AI security research and real-world vulnerabilities that is creating a dangerous blind spot. When CyberArk researchers tested academic jailbreak techniques against current models, they found most weren’t reproducible. The problem isn’t with academic rigor but the breakneck pace of AI development.

“The pipeline of creating research, reviewing it, doing peer review, and then publishing it is amazing,” Shimony acknowledges. “But this world is changing so fast. You’re researching old models that are irrelevant and guardrails that have been fixed or changed. It’s very difficult to compare this world and the world we’re living in at the moment.”

This temporal mismatch has serious consequences for LLM resiliency. While researchers publish papers on months-old vulnerabilities, new models with fresh attack surfaces deploy weekly. The result? A security landscape where yesterday's fixes become tomorrow’s false confidence.

Operation Grandma and the art of creative destruction

Among Fuzzy AI’s most successful techniques is something CyberArk calls “Operation Grandma” (named after an Israeli movie). The attack demonstrates how historical framing can bypass virtually any content filter by presenting malicious requests within seemingly academic contexts.

“You design the prompt in a historical context, and you make it passive — like writing a paper — but essentially you can almost ask anything, and the LLM will happily supply answers,” Shimony explains. This technique has proven “incredibly successful against almost every LLM” because it exploits the fundamental tension between helpful AI and safe AI.

The sophistication of these attacks raises uncomfortable questions about the current state of AI security. If a simple historical reframing can bypass multi-billion-dollar safety systems, what does that say about our readiness to deploy these models in critical infrastructure?

The Agentic AI amplification problem

The stakes escalate dramatically as the industry pivots toward agentic AI — systems where large language models don’t just generate text but actively execute tasks and make decisions. These AI agents can trigger system calls, modify files, and interact with external services, transforming a simple jailbreak from an embarrassing chatbot conversation into a potential system compromise.

“We see it as a multiplier of danger,” Shimony warns. “There must be security products to protect the agentic AI framework. Currently, we have not seen ones that are mature enough.”

The challenge goes beyond traditional access controls. In agentic AI systems, the line between legitimate automation and malicious activity becomes blurred by design. An AI agent authorized to manage your organization’s budget could, if compromised, execute financial transactions that appear entirely legitimate within its granted permissions.

Dvash emphasizes the core problem: “Elements by themselves do not embed security mechanisms of some sort. It’s always another layer you need to use when you integrate an LLM to your application or service. The first line of defense is the software architect who designs systems that use large language models.”

The observability black hole

Perhaps most concerning is the observability problem. As AI agents become more sophisticated, understanding their decision-making processes becomes exponentially harder. Users see successful task completion but remain blind to the data collection, inference chains, and potential security compromises happening beneath the surface.

“Even now, if you just look at the pipeline of an agentic AI framework and run it, it's difficult to understand what all the moving parts are accessing,” Shimony notes. “Observability is a key issue, but observability is not enough. You also need to interpret the data that you see to understand what the correct system behavior is.”

This opacity creates perfect conditions for sophisticated attacks. A compromised agent could exfiltrate sensitive data while appearing to perform its designated tasks normally, with organizations none the wiser until the breach is discovered through other means.

The security-by-design deficit

The fundamental issue is philosophical. Current LLM development prioritizes functionality and user satisfaction over security considerations. Models are trained to be helpful, harmless, and honest, but these goals often conflict in practice.

“We cannot support a claim that LLMs are designed with security in mind because clearly it’s not really the case,” Shimony states.

The industry’s approach mirrors historical patterns in software development: build first, secure later. But with AI systems, this approach carries amplified risks because the attack surface isn’t just code but language itself, with all its ambiguity and contextual complexity.

Transparency is the first step

CyberArk’s decision to open-source Fuzzy AI reflects a belief that transparency, not obscurity, will drive better security practices. “Our mindset is to create better security from an offensive mindset,” Shimony explains. “You must know your opponent to create a more secure product.”

The framework represents just “one leg of the chair of AI security,” as the researchers put it. Comprehensive AI security requires addressing jailbreaking, privilege management, observability, and data leakage — none of which current systems handle adequately.

As enterprises continue their AI adoption sprint, the security gap widens. The question isn’t whether AI systems will be compromised but whether organizations will recognize the compromise when it happens. Until security becomes a first-class concern in AI development rather than an afterthought, we’re building our digital future on foundations of algorithmic sand.

The jailbreak is just the door. What matters is what walks through it.

Image credit: iStockphoto/Mihaela Rosu