# Promptfoo Promptfoo is an open-source CLI and library for testing LLM applications the way we test regular code. You describe prompts, test cases, and assertions in a [[Yet Another Markup Language (YAML)|YAML]] file (TypeScript works too), run `promptfoo eval`, and get a results matrix comparing outputs across prompts, models, and providers. A local web viewer shows everything side by side. The same engine also does automated red teaming: it probes your app for prompt injections, jailbreaks, data leaks, and dozens of other vulnerability classes. That security angle is what led OpenAI to acquire the company in March 2026. The project stays open source (MIT), and teams at OpenAI, Anthropic, and a large chunk of the Fortune 500 were already using it before the acquisition. ## What it gives you - **Declarative evals** — prompts, test cases, and expected outputs live in versioned config files next to your code. No more eyeballing outputs in a playground. - **Two assertion styles** — deterministic checks (valid JSON, regex, word counts, cost and latency thresholds) and model-graded ones (LLM-as-judge for tone, factuality, relevance). Prefer deterministic checks when you can; judges add cost and latency. - **Model comparison** — run the same test suite against OpenAI, Anthropic, Google, [[Ollama]] models, and more. Answers "can I switch to the cheaper model without losing quality?" with data instead of gut feeling. - **Red teaming** — automated adversarial probes covering 50+ vulnerability types, from prompt injection to business rule violations. - **CI/CD integration** — evals as a quality gate in GitHub Actions, GitLab, Jenkins, etc. Prompt regressions get caught before they ship. - **Local-first** — runs on your machine. Prompts and outputs stay on your infra unless you opt into sharing. ## When I'd reach for it - Prompt tweaks keep breaking things elsewhere, and manually spot-checking outputs doesn't scale anymore. - I want to compare models on MY workload before committing (quality, cost, latency). - An LLM feature faces real users and needs a security check before launch. ## Trade-offs - **Node.js based** and wants a recent version; fine for JS/TS shops, one more runtime for everyone else. - **Model-graded assertions multiply API calls.** Keep them for the checks that genuinely need judgment. - **Now owned by OpenAI.** Still open source and multi-provider today, but worth watching how neutral a testing tool remains when one of the providers owns it. ## Alternatives - [[LangSmith]] — hosted platform bundling evals with tracing and prompt management. - [[Langfuse]] — open source, self-hostable observability with eval features. ## References - Website: https://www.promptfoo.dev/ - GitHub: https://github.com/promptfoo/promptfoo - Docs: https://www.promptfoo.dev/docs/intro/ - Release notes: https://www.promptfoo.dev/docs/releases/ - Discord: https://discord.gg/promptfoo - Testing AI prompts and comparing models with Promptfoo (Tim Deschryver): https://timdeschryver.dev/blog/testing-ai-prompts-and-comparing-models-with-promptfoo - Acquisition announcement: https://www.promptfoo.dev/blog/promptfoo-joining-openai/ ## Related - [[LangSmith]] - [[Langfuse]] - [[Helicone]] - [[Prompt Engineering]] - [[Prompt Engineering Best Practices]] - [[LLM Monitoring]] - [[AI Observability]] - [[OpenAI]] - [[Anthropic]]