Version 1
100/100 · 46 tokensSummarize the incident report below for the on-call engineering team in 5 bullet points. Include the root cause, the customer impact, and the fix. Return a markdown list.
<report>
{{report}}
</report>Prompt testing
Prompt testing
Treat prompts as source files. One command checks every prompt in a repository against a quality bar, a token budget, and a severity threshold, and fails the build when one of them is broken.
- CLI
- Offline
- Build from source
What it does
A prompt that worked last month can be broken by a well-meant edit: a duplicated paragraph, a removed output format, a pasted key. Prompt testing catches that at review time instead of in production.
The promptflow check command reads the prompt files you point it at and fails when a file scores below your minimum, uses more tokens than your maximum, or has a finding at or above a chosen severity. The default threshold is critical, so a leaked credential always fails. Settings live in a .promptflowrc.json file at the repository root.
These are static tests. They examine the prompt text and never call a model, which is why they are fast, free, and identical on every run. They do not replace output testing against a model; they sit in front of it and remove the failures that do not need a model to find. Model-based testing is on the roadmap and is not available today.
Practical example
A regression caught by compare
Two versions of the same prompt, compared by the engine when this page was built. The edit looked harmless and removed the output format.
Version 2
84/100 · 36 tokensSummarize the incident report below for the on-call engineering team. Include the root cause, the customer impact, the fix, and some other details if relevant.
{{report}}- Quality
- -16
- Tokens
- -10
- New issues
- 3
- warningTemplate variable not delimited: {{report}}.
- infoNo output format is specified.
- infoNo length limit for generated content.
Computed by the engine when this page was built. With the configuration below, version 2 passes the gate at a minimum score of 70.
.promptflowrc.json
{
"include": ["prompts/**/*.md"],
"model": "gpt-4.1",
"minScore": 70,
"maxTokens": 2000,
"failOn": "critical"
}shell
promptflow check
promptflow compare prompts/summary.v1.md prompts/summary.v2.mdKey benefits
No flaky tests
The same file always produces the same result, so a red build means the prompt changed.
No API keys in CI
Nothing is sent to a provider, so there are no secrets to manage and nothing to pay per run.
Fast enough for every commit
A repository of prompt files is checked in seconds.
Readable failures
Each failure names the file, the rule, and the fix, and links to the rule reference.
Features
check
The gate: minimum score, maximum tokens, and fail-on severity across a set of globs.
validate
Exit code 1 on any critical finding, for a simple pass or fail.
compare
Token, quality, and findings difference between two versions of a prompt.
scan
Security-only run that fails on a critical security finding.
JSON output
Every command accepts --json for use in scripts and dashboards.
Per-repository config
Include globs, model, thresholds, and ignored rules in .promptflowrc.json.
Who it is for
- Teams that keep prompts in gitGive prompt changes the same automatic review that code changes already get.
- Platform and DevEx engineersEnforce one standard for prompt files across many repositories with a small config file.
- Engineers on call for an LLM featureStop the class of incident where a prompt edit quietly doubles token cost or removes a constraint.
Use cases
- 01
Pull request check
Run check in a GitHub Actions job so a prompt change cannot merge below the bar.
- 02
Token budget enforcement
Set maxTokens to keep system prompts within the allowance you planned for.
- 03
Secret prevention
Leave fail-on at critical so a committed key or password fails immediately.
- 04
Pre-commit hook
Run validate on staged prompt files for feedback before a push.
Frequently asked questions
Does prompt testing run my prompts against a model?
No. The checks are static: they analyze the prompt text with deterministic rules. Running prompts against a model and scoring the output is on the roadmap and is not available today.
Is the CLI on npm?
Not yet. Until the package is published, build the CLI from a checkout of the repository. The CLI documentation has the commands.
What file types can it check?
Any text file. Point the include globs at the files that hold your prompts, such as Markdown or .prompt.txt files.
Can I ignore a rule?
Yes. List rule codes under ignore in .promptflowrc.json to skip them for the whole repository.
Related PromptFlowEngine capabilities
Every product runs the same engine, so they combine without surprises.
- ToolPrompt EvaluationA published scoring formula across eight dimensions, with version comparison.
- ToolPrompt AnalyzerFind ambiguity, missing context, conflicts, and wasted tokens in any prompt.
- ToolPrompt SecurityDetect injection phrases, leaked secrets, and personal data, then redact them.
- DocumentationCLI and CI gateReference and setup details.
- DocumentationScoring modelReference and setup details.
Learn the technique behind it
Guides from the prompt engineering knowledge center that explain the ideas this product applies.
- GuidePrompt testing and evaluation: how to measure a promptTest sets, metrics, static checks, model graders, and running prompt tests in CI.
- GuidePrompt management and versioningStoring, versioning, reviewing, testing, and rolling out prompts as a team.
- GuidePrompt optimization: an iterative methodA repeatable loop for improving a prompt by measurement instead of by feel.
Try it on your own prompt.
Free, deterministic, and private. No sign-up.