Problem
Promptfoo's normalized trajectory steps already preserve statusCode and statusMessage, but deterministic trajectory assertions can currently check tool names, arguments, order, and counts—not whether a matched tool/command/span succeeded or failed.
For agent reliability evaluation this makes an important case hard to express: a run may call the correct tool with the correct arguments, yet the tool may fail; conversely, a policy may require that a dangerous tool never returns an error or that a failed call is followed by a successful retry.
Proposal
Add a deterministic trajectory:step-status assertion that matches a trajectory step by name/pattern/type and status (success, error, or an exact numeric status code), with an optional status-message pattern. It should reuse the existing TrajectoryStep.statusCode and statusMessage fields and support the existing not- assertion convention.
Example:
- type: trajectory:step-status
value:
name: delete_file
status: errorScope of the first implementation
- Add pure status-matching semantics and focused unit tests.
- Add/register the assertion, schema, documentation, and failure reasons.
- Add a local CLI fixture/E2E test for successful and failed tool spans.
- Treat status code
0 or a missing status as unset; do not match it as success unless the design is revised with maintainer feedback.
- Keep existing trajectory assertions backward compatible.
I plan to submit this as the first step in a small, test-first series that makes agent trajectory evaluation cover invocation, arguments, ordering, and execution outcome. The implementation will reference this issue and include unit tests before the CLI E2E validation.
Problem
Promptfoo's normalized trajectory steps already preserve
statusCodeandstatusMessage, but deterministic trajectory assertions can currently check tool names, arguments, order, and counts—not whether a matched tool/command/span succeeded or failed.For agent reliability evaluation this makes an important case hard to express: a run may call the correct tool with the correct arguments, yet the tool may fail; conversely, a policy may require that a dangerous tool never returns an error or that a failed call is followed by a successful retry.
Proposal
Add a deterministic
trajectory:step-statusassertion that matches a trajectory step by name/pattern/type and status (success,error, or an exact numeric status code), with an optional status-message pattern. It should reuse the existingTrajectoryStep.statusCodeandstatusMessagefields and support the existingnot-assertion convention.Example:
Scope of the first implementation
0or a missing status as unset; do not match it as success unless the design is revised with maintainer feedback.I plan to submit this as the first step in a small, test-first series that makes agent trajectory evaluation cover invocation, arguments, ordering, and execution outcome. The implementation will reference this issue and include unit tests before the CLI E2E validation.