Imagine you've just hired someone to run your front desk. Before they start taking calls, you'd want to know: what happens if a rude customer tries to manipulate them? What if someone tries to get information they shouldn't have?
Prompfoo does that, but for AI. It automatically fires thousands of tricky scenarios at your AI agent — trying to trick it, confuse it, or get it to leak things it shouldn't — and then shows you what broke.
On March 9th this year, OpenAI paid roughly $86 million for this tool. Then they did something unusual: they kept it completely free and open for anyone to use. That's not a PR move. That's a signal that testing AI for security flaws is no longer optional — it's becoming standard practice, like having a smoke alarm before you open a restaurant.
The teams building AI at both OpenAI and Anthropic use this tool on their own work. So do over 150 Fortune 500 companies. And yet a small agency or independent founder can run the exact same tests, for free, on their own machine.
If you're using AI to answer customer questions, process documents, or make decisions — this is the kind of tool your developer should know exists. Before your AI goes live, someone should be asking: what happens when someone tries to break it?
AI agent — an AI that doesn't just answer questions, but takes actions: booking things, sending emails, looking up data on your behalf.
Prompt injection — a trick where someone sneaks hidden instructions into text your AI reads, trying to make it behave in ways you didn't intend.
Open source — software whose inner workings are public and free to use. Anyone can inspect it, improve it, or build on top of it.
Red teaming — intentionally trying to break something before a real problem does. Originally a military concept, now common in cybersecurity.