# AI Red Teaming AI red teaming is the practice of adversarially testing [[Large Language Models (LLMs)|LLM]]-based systems to surface failures *before* real users or attackers do. It is the agent-era equivalent of penetration testing, with a broader scope: in addition to security boundaries, red teaming probes safety, alignment, factuality, jailbreak resistance, and capability misuse. The discipline is essential because LLM systems fail in ways traditional QA cannot anticipate. The same input can succeed one day and fail the next; an attacker can wrap a malicious instruction inside ordinary-looking text; a tool-using agent can be coaxed into actions outside its sanctioned scope. ## Common Attack Surfaces - **Prompt injection**: hostile input (in user messages, retrieved documents, web pages, emails, file contents) overrides the system prompt and redirects the model. - **Indirect prompt injection**: payload arrives via a tool call result rather than from the user directly; the model dutifully executes attacker instructions hidden in the data it just fetched. - **Jailbreaks**: prompts engineered to bypass safety training and extract disallowed content. - **Tool misuse**: the model is tricked or persuaded into calling powerful tools with attacker-chosen arguments (sending email, executing code, transferring funds). - **Data exfiltration**: the model is induced to leak system prompts, secrets, training data, or contents of other users' sessions. - **Resource abuse**: prompts engineered to maximize token spend, infinite tool loops, or compute exhaustion. - **Bias and harm probes**: targeted inputs to surface unsafe outputs across protected categories. - **Hallucination probes**: questions designed to make the model confidently invent facts. ## Methodology - **Threat model first**: who are the attackers, what are their goals, what surface do they touch? - **Baseline before changes**: capture the current pass/fail rate on a curated red-team set. - **Mix manual and automated**: humans find creative attacks; automated fuzzers cover breadth. - **Real attack data**: convert real abuse incidents into permanent test cases. - **Adversarial replay**: replay full traces (system + user + tool history) so attacks composed across steps are caught. - **Measure capability + safety together**: a fix that destroys utility is not a fix. - **Red-team newer models too**: capability improvements often unlock new attack patterns the old model could not execute. ## Where Red Teaming Lives Red teaming is the *Evaluate* and *Operate* stages of the [[Agent Development Lifecycle (ADLC)]]. Production systems run red-team sets as gates on every model, prompt, or tool change. ## Related Disciplines - Application security and penetration testing. - ML evaluation and benchmark design. - AI alignment and safety research. - Incident response for AI products. ## Related - [[Agent Development Lifecycle (ADLC)]] - [[AI Agents]] - [[AI Agent Permissions]] - [[AI Agent Harness]] - [[Large Language Models (LLMs)]] - [[LLM Tool Calling]] - [[AI Verifiability]] - [[AI and Trust]] - [[Agentic Engineering]] - [[Model Context Protocol (MCP)]]