Five open-source prompt-injection testing tools compared: promptfoo, garak, Giskard, PyRIT, sentinel-scan-cli

A hands-on comparison published on dev.to takes on the question every team shipping an LLM application eventually faces: whether anyone genuinely tested the system for prompt injection, or just reviewed the prompts by eye. The author worked through the five open-source tools that come up most often — promptfoo, garak, Giskard, PyRIT and sentinel-scan-cli — and the result reads less like a ranking and more like a decision framework.

Start with the layer you are testing

The post's central argument is that most confusion comes from conflating two distinct activities. App-layer testing checks whether your prompts, guardrails, tool-calling logic and retrieval pipeline resist attacks when wired together end to end. Model-layer testing checks whether the underlying model itself behaves exploitably, regardless of what is wrapped around it.

According to the author, promptfoo, Giskard and sentinel-scan-cli are primarily app-layer tools, NVIDIA's garak is model-layer, and Microsoft's PyRIT could be configured for either. Settling which layer you actually need, the post says, decides most of the choice before you ever compare feature lists.

How each tool stacks up

promptfoo comes across as the most broadly adopted option. It can be launched straight through npx with no install step, ships more than 50 built-in red-team plugins, and includes report presets aligned to the OWASP LLM Top 10, NIST and MITRE ATLAS. The author also cites the project's repository claim that it is used internally at OpenAI and Anthropic, and passes along an update from March 2026: promptfoo is now part of OpenAI, though it remains open source under the MIT licence. The tradeoff is setup cost — the plugin surface is large enough that a useful first run takes real wiring time.

garak, maintained by NVIDIA with roughly 8,100 GitHub stars, targets the model itself: jailbreak susceptibility, encoding tricks and data leakage across its 50-plus probes. It suits teams evaluating which base model to build on, but the post warns it is not a substitute for testing your own application, and it is not natively organised around the OWASP LLM Top 10, so any compliance mapping falls on you.

Giskard offers app-layer scanning with OWASP-mapped detectors in a similar spirit to promptfoo. The catch, per the post, is that the continuous-scan Hub — the component that would re-run scans against a live app over time — is a paid product. The free scanner is real and usable, but expect a one-off snapshot rather than ongoing monitoring.

PyRIT, Microsoft's framework for multi-turn attack orchestration, was archived on GitHub on 27 March 2026, according to the post, leaving the repository read-only with no further commits, releases or issue triage. The author characterises it as having always aimed at security researchers scripting structured engagements rather than developers running pre-ship checks, and says nothing directly replaces its multi-turn orchestration today; promptfoo's red-team plugins or a small custom harness come closest.

sentinel-scan-cli gets a section with an explicit disclosure: the author works on it. It is a dependency-free CLI available in both Python and Node with identical output across the two, covering 15 attack patterns, each tagged with its OWASP LLM Top 10 category. A demo mode runs with zero configuration and no API keys, delivering pass/fail results tagged by OWASP category in under a minute. It is pitched as a fast first pass or CI gate rather than broad coverage — the author concedes that anyone past the quick-check stage should reach for promptfoo or Giskard instead.

The short version

The post's recommendations: promptfoo for the most thorough free app-layer coverage, garak for raw model evaluation, Giskard if you want OWASP mapping and might later pay for continuous monitoring, and sentinel-scan-cli for a quick zero-install smoke test. Multi-turn red-teaming is left with a gap now that PyRIT is archived. The author's suggested stack layers several tools rather than choosing one: a lightweight CI gate via sentinel-scan-cli or a promptfoo quick-start, a full promptfoo pass before launch, and garak when evaluating model choice.

Why it matters

Prompt injection is the security question every RAG pipeline, tool-calling agent and customer-facing chatbot inherits the moment untrusted text reaches a model, and this comparison turns a crowded tool landscape into practical guidance keyed to the layer you are testing and the setup budget you have. It is also a snapshot of how fast the ecosystem is moving: within the same year, one major tool was archived and another was absorbed by OpenAI, so status checks belong in any tool-selection process. One caveat worth keeping in mind: this is a single author's assessment, and that author works on one of the five tools under review, so the recommendations deserve the same scrutiny as the products themselves.