Skip to content

Add consensusPolicy: fail-on-disagreement option for assertions#50

Open
Ipseeta wants to merge 1 commit into
mainfrom
ipseeta/consensus-policy
Open

Add consensusPolicy: fail-on-disagreement option for assertions#50
Ipseeta wants to merge 1 commit into
mainfrom
ipseeta/consensus-policy

Conversation

@Ipseeta
Copy link
Copy Markdown
Contributor

@Ipseeta Ipseeta commented May 12, 2026

When the primary (Claude) and secondary (Gemini) assertion models reach opposite verdicts, Passmark calls a third arbiter model to break the tie. This commit makes that behavior configurable via
configure({ assertions: { consensusPolicy } }).

  • "consult-arbiter-on-disagreement" (default): unchanged; the arbiter decides. Existing users see no behavior change.
  • "fail-on-disagreement": treats any disagreement as a failure immediately, with no arbiter call. The synthesized reasoning includes both models' verdicts and confidence scores so the failure is actionable. Useful when you'd rather surface ambiguity in the UI under test than risk a single model swinging the result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant