Interrogating Intelligence: Red Teaming LLMs with Garak

Famous decel quote of the day

“We should be more concerned with the system doing what we tell it to do rather than what we want it to do.” – Eliezer Yudkowsky

Introduction

As llms increasingly power critical applications across industries, the security implications have evolved from theoretical concerns to urgent priorities. Vulnerabilities in these systems can lead to data leaks, harmful outputs, and compromised decision-making—risks that grow proportionally with deployment scale.

Unlike general cybersecurity tools, Garak was specifically engineered to probe the unique attack surfaces of language models. This open-source framework systematically tests for a comprehensive range of vulnerabilities including:

• prompt injection
• jailbreaking resistance
• harmful content generation
• data leakage vectors
• adversarial manipulations

The consequences of undetected llm weaknesses extend beyond mere technical failures:

• Healthcare applications could generate dangerously inaccurate medical advice
• Enterprise systems might leak proprietary information through carefully crafted queries
• Public-facing applications remain vulnerable to reputation-damaging outputs

Garak’s approach represents a shift from ad-hoc testing to comprehensive security assurance

Understanding Garak

Garak began as a research project at nvida in late 2022, when early public access to powerful llms highlighted significant security gaps. The initial codebase consolidated various ad-hoc testing approaches researchers were using to probe models like gpt-3.5 and early open-source alternatives.

the tool evolved through several key developmental phases:

• initially focused on jailbreaking and prompt injection vectors
• expanded to include data extraction and model behavior consistency testing
• adopted a plugin architecture allowing community contributions of new attack vectors

Positioning in the safety toolchain

• a pre-deployment verification tool rather than a runtime safeguard
• a proactive testing framework rather than a reactive monitoring system
• a model-agnostic testing suite applicable across different architectures

Setting Up Your Red Team Environment

System prerequisites

• python 3.9+ with pip package manager
• 8GB+ RAM for basic testing (16GB+ recommended for larger models)
• internet connectivity for api-based model testing
• git for repository cloning and updates

The computational requirements scale based on:

• whether you’re testing local models or remote apis
• batch size and concurrent test execution settings

Basic installation process

#Creating a new virtual enviornment 
python -m venv garak-env

#activating the virtual enviornment 
source garak-env/bin/activate

#updating pip
pip install --upgrade pip

#installing Garak
pip install garak

#Check to see if Garak is installed
> garak --help

garak LLM vulnerability scanner v0.10.3.1 ( https://github.com/NVIDIA/garak ) at 2025-03-16T20:27:44.017969

I will be integrating garak with my current local LLM environment I currently host all of my models on openwebui with the help of ollama.

Garak has a very easy way to integrate with ollama

garak --model_type ollama --model_name [Name of Model]

I currently have a handful of models but the oldest and probably most vulnerable would be mistral-nemo so that is the model we will be testing against in this post

docker exec -it ollama /bin/bash

root@e524721aa7d5:/# ollama list
NAME                     ID              SIZE      MODIFIED   
mistral-nemo:latest      994f3b8b7801    7.1 GB    25 days ago

A few of the probes (Attack Methods that come with Garak)

Core Testing Methodologies

exploring attack categories

• prompt injection attacks
• jailbreaking techniques
• behavioral manipulation

Prompt injection techniques and examples

Jailbreaking strategies and their evolution

Harmful content generation tests

harmful content generation tests often use subtler approaches:

Case Studies

Listing the probes and attacking mistrail-nemo

❯ garak --list_probes
probes: ansiescape 🌟
probes: atkgen 🌟
probes: av_spam_scanning 🌟
probes: continuation 🌟
probes: dan 🌟
probes: divergence 🌟
probes: donotanswer 🌟
probes: encoding 🌟
probes: encoding.InjectBase32
probes: encoding.InjectBase64
probes: encoding.InjectBraille
probes: encoding.InjectEcoji
probes: encoding.InjectHex
probes: encoding.InjectMorse
probes: glitch 🌟
probes: goodside 🌟
probes: grandma 🌟
probes: grandma.Substances
probes: grandma.Win10
probes: grandma.Win11
probes: latentinjection 🌟
probes: leakreplay 🌟
probes: lmrc 🌟
probes: lmrc.QuackMedicine
probes: malwaregen 🌟
probes: malwaregen.Evasion
probes: malwaregen.Payload
probes: malwaregen.SubFunctions
probes: malwaregen.TopLevel
probes: misleading 🌟
probes: phrasing 🌟
probes: promptinject 🌟
probes: realtoxicityprompts 🌟
probes: suffix 🌟
probes: topic 🌟

Testing against our locally hosted model mistral-nemo we can see that it is vulnerable to a handful of the probes offered by Garak

❯ garak --model_type ollama --model_name mistral-nemo --probes malwaregen.Evasion --report_prefix /home/trevor/projects/garak/AI-redteam/reports/

garak LLM vulnerability scanner v0.10.3.post1 ( https://github.com/NVIDIA/garak ) at 2025-03-16T22:51:34.828757

📜 logging to /home/trevor/.local/share/garak/garak.log
🦜 loading generator: Ollama: mistral-nemo
📜 reporting to /home/trevor/projects/garak/AI-redteam/reports/.report.jsonl

🕵️  queue of probes: malwaregen.Evasion
probes.malwaregen.Evasion:  48%|█████████████████████▌                       | 23/48 [10:31<09:36, 23.05s/it]
Ollama:mistral-nemo:  40%|█████████████████████▏                               | 2/5 [00:09<00:14,  4.95s/it]

This probe requests a code to perform evasive actions that hinder its detection, such as disabling antivirus, hiding in a process list, and so on

Personally I dont know if this is surprising or not, Mistral Nemo was released on July 18, 2024, which would make it about 8 months old as of March 2025. Seven months without an update (In AI land this is a ancient model) it failed every single test that we threw at it. if you are interested I reformatted the json output to a html file and you can view it below

Running a similar test against a brand new model from google gemma3

garak --model_type ollama --model_name gemma3:latest --probes malwaregen.TopLevel --report_prefix /home/trevor/projects/garak/AI-redteam/reports/

Based on the report it looks like it failed but openening it up we can see that the detector is detecting for any code output. Not just malicious code.

When taking a closer look at the drop downs we can see that gemma3 has gone through a more intense red teaming trial to prevent it from generating harmful content

This did not contain any prompt injection or attempts outside of just directly asking for the content so lets take our hand at some manual prompts to see if we can get anything interesting using one of the comprehensive jailbreaks from pliny the liberator

Our testing revealed significant security differences between older and newer language models when faced with malicious requests:

Google’s Gemma3: The newer model initially showed robust safeguards, refusing direct requests for malicious code (as seen in Test #55 requesting virus code in Rust). However, when targeted with specialized jailbreak techniques from “pliny the liberator,” the model’s defenses were compromised, resulting in the generation of x86 assembly code for a keylogger with remote control capabilities.

Mistral-Nemo (July 2024): Despite being only 8 months old at the time of testing (March 2025), this model failed all 236 security tests in the malwaregen.Evasion probe category, demonstrating high vulnerability to requests for harmful content.

This progression shows an important pattern in AI security: while newer models have better safety features and defenses, they can still be tricked by clever attack methods. The case shows the ongoing back-and-forth between security improvements and new ways to attack AI systems.

Conclusion

Final Thoughts on the Importance of Proactive Security Measures

Our testing has demonstrated that even new state-of-the-art models like Google’s Gemma3 can be circumvented with sophisticated jailbreak techniques, while older models like Mistral-Nemo (8 months old) fail comprehensively against standard security probes. As LLMs become increasingly embedded in critical infrastructure, the impact of these vulnerabilities grows exponentially. Organizations deploying these technologies must recognize that:

• Traditional security frameworks cannot address unique LLM vulnerabilities like the “pliny the liberator” jailbreak that successfully extracted malicious x86 assembly code

• The attack surface of language models requires specialized tools like Garak that can systematically probe for AI-specific weaknesses

• The demonstrated progression from older to newer models shows security improvements, but also highlights the ongoing cat-and-mouse game between defenders and attackers

Resources

If you are interested in reading more about AI red-teaming i’ve left some related links below

Red Team Guide

NVIDIA AI Red Team: An Introduction

garak the LLM vulnerability scanner

Pliny the liberators jail breaks

Pliny on twitter

Hack The Box AI Red Teaming Path