Literal assertions
Required tools, forbidden tools, sequence, arguments, and action counts are checked directly.
Repeatability before release
Deterministic AI agent testing for failures that only appear sometimes.
The same release can ask for approval on one run and skip it on the next, preserve the right target once and change it later, or repeat an external action intermittently. Konsista makes that hidden drift visible with repeated mocked runs and literal checks.
One passing run can hide the release-breaking run.
Deterministic AI agent testing does not require identical wording. It requires the declared action path to remain observable and the verdict to come from tool calls, order, values, and limits rather than another model's interpretation.
Repeated scenarios
The same case runs multiple times so intermittent behavior cannot hide behind one successful execution.
Baseline comparison
Compare a known configuration with the candidate prompt, model, tool schema, connector, or policy.
Value drift
Track whether critical recipients, record IDs, amounts, and destinations change across runs.
Multiplicity drift
Separate a changed value from an action that starts executing zero, two, or several times.
Explicit coverage
The report states what was observed and what the test cannot claim about the wider system.
One pass cannot measure an intermittent gate.
A useful regression suite repeats the same high-risk procedure and keeps correctness separate from consistency.
- Define the procedure and the exact observable controls.
- Run a baseline configuration against mocked tools.
- Run the candidate configuration with the same scenario and mock responses.
- Compare per-run findings, aggregate stability, and critical values.
- Block on a failed declared control even when the unsafe behavior is perfectly consistent.
Baseline vs candidate
required sequence6/6 → 2/6
critical targetstable
behavior consistency1.00 → 0.67
candidate verdictred finding
Consistency is not correctness.
An agent that reliably skips approval is consistent and unsafe. Konsista reports that as stable unsafe behavior, not as a successful test. A passed scenario means only that the declared controls held for the tested configuration, data, mocks, and repeats.
Related AI agent testing guides
Repeat the procedure your release depends on.
Send the risky action, required sequence, critical values, and model, prompt, tool, or policy change. The first run uses synthetic records and mocked tools.
Test one intermittent risk