What is Offensive AI?

Offensive AI is the malicious use of artificial intelligence to power, automate, and scale cyberattacks, as well as attacks that target AI systems themselves. It takes two forms:

  • Attacks using AI: AI makes the attacker faster, cheaper, and more convincing, from automated reconnaissance, intrusion, and vulnerability discovery to adaptive malware, deepfakes, and social engineering at scale.
  • Attacks against AI: the adversary exploits the AI that people and organizations rely on, through prompt injection, agent hijacking, jailbreaks, data poisoning, model theft, and adversarial examples that evade AI-based defenses.

From Prediction to Reality

When we founded the lab in 2020, offensive AI was largely a prediction, with a handful of early incidents such as the poisoning of Microsoft’s Tay chatbot1 and adversarial evasion of an AI-based antivirus.2 Since then, large language models (LLMs) and AI agents have made it part of the everyday threat landscape:

  • Deepfakes and impersonation. In 2024, a finance worker at the engineering firm Arup transferred about $25 million after a video call in which every other participant was a deepfake of the company’s executives.3 In 2025, the FBI warned of an ongoing campaign using AI-generated voice messages to impersonate senior US officials.4
  • AI in the attacker’s toolkit. Microsoft and OpenAI reported state-affiliated groups from Russia, North Korea, Iran, and China using LLMs to support their operations,5 and Google has since found malware that queries an LLM while it runs to rewrite itself and evade detection.6
  • AI agents carrying out attacks. In 2025, Anthropic reported a criminal using its Claude Code agent to automate a data-theft and extortion campaign against at least 17 organizations.7 Months later, it disrupted what it called the first reported AI-orchestrated cyber-espionage campaign, attributed to a state-sponsored group, in which AI performed 80–90% of the work.8
  • Fraud at scale. Our own interviews with 145 scam-compound insiders found that LLMs are already used in romance-baiting scams, and in a blinded study an LLM agent out-performed human scammers at building trust.9
  • Attacks on AI systems. As LLMs become agents with access to email, files, and tools, prompt injection is ranked the top risk for LLM applications.10 In 2025, researchers disclosed EchoLeak, a since-patched zero-click prompt injection that could make Microsoft 365 Copilot leak internal data through a single email.11

Are We Prepared?

Security agencies expect the threat to keep growing. In its 2024 assessment, the UK’s National Cyber Security Centre concluded:

“Artificial intelligence (AI) will almost certainly increase the volume and heighten the impact of cyber attacks over the next two years.”12

Defenses are struggling to keep up. Our recent work shows that popular safety filters miss AI-driven romance scams,9 that AI agents can be hijacked through the content they read,13 and that LLMs reviewing code can be fooled into misreading what it does.14 To counter offensive AI, we need to anticipate the adversary’s next move and build countermeasures before new attacks become mainstream, rather than racing to patch them afterward.

Our Lab

image

The Offensive AI Research Lab was founded in 2020 at Ben-Gurion University of the Negev with the goal of producing research that helps the world prepare for offensive AI. Instead of just “patching” the problem, we work on ways to give defenders the upper hand by putting the adversary at a disadvantage.

Today our research spans AI agent security, LLM safety and monitoring, AI-driven fraud and deepfakes, and adversarial machine learning. Our work appears at top venues such as USENIX Security, NDSS, ICLR, EMNLP, and ACSAC, has been presented at Black Hat and DEF CON, and has been covered by outlets including WIRED and The Washington Post. In 2025, the lab was awarded an ERC Starting Grant (AGI-Safety). Explore our featured research and full list of publications.

We are a friendly group of researchers and hackers with a passion for AI and cyber security. We strongly believe that AI should never be used to cause physical, financial, or psychological harm. However, cyber criminals do not hold by the same code of ethics. It is therefore our mission to identify, counter, and mitigate the threat of offensive AI to help protect society.

Join Us!

We are currently seeking excellent students and post-docs who are eager to research offensive AI. Whether you are drawn to AI agent security, LLM safety, deepfakes, or breaking machine learning models, we would love to work with you! You can make a difference in the world, and we’d be honored to make that difference with you. Generous stipends are available for suitable candidates.

If you are interested in joining or have any questions, please reach out to Dr. Mirsky by email: yisroel [at] bgu [dot] ac [dot] il

References

  1. Microsoft, “Learning from Tay’s introduction”, March 2016. ↩

  2. Skylight Cyber, “Cylance, I Kill You!”, July 2019. ↩

  3. CNN, “British engineering giant Arup revealed as $25 million deepfake scam victim”, May 2024. ↩

  4. FBI IC3, “Senior US Officials Impersonated in Malicious Messaging Campaign”, Public Service Announcement I-051525-PSA, May 2025. ↩

  5. Microsoft Threat Intelligence, “Staying ahead of threat actors in the age of AI”, February 2024. ↩

  6. Google Threat Intelligence Group, “GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools”, November 2025. ↩

  7. Anthropic, “Detecting and countering misuse of AI: August 2025”, August 2025. ↩

  8. Anthropic, “Disrupting the first reported AI-orchestrated cyber espionage campaign”, November 2025. ↩

  9. Gressel et al., “Love, Lies, and Language Models: Investigating AI’s Role in Romance-Baiting Scams”, USENIX Security 2026. ↩ ↩2

  10. OWASP Gen AI Security Project, “LLM01:2025 Prompt Injection”, OWASP Top 10 for LLM Applications 2025. ↩

  11. Reddy and Gujral, “EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System”, 2025 (CVE-2025-32711). ↩

  12. UK National Cyber Security Centre, “The near-term impact of AI on the cyber threat”, January 2024. ↩

  13. Eliav et al., “Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents”, ACSAC 2026. ↩

  14. Bernstein et al., “Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias”, NDSS 2026. ↩