Evidence-bound release decisions for AI systems.
EvalSeal is a deterministic release gate. It takes an evidence pack containing
the risks, controls, evaluation cases, findings, and approval decision for an AI
release candidate, combines it with organization-owned governance, tailoring,
and change-trigger policies, and returns APPROVE, CONDITIONAL, or REJECT
with a machine-readable reason for every blocking issue. It validates 13 schemas
and enforces exact artifact hashes, evidence freshness, approval authority, and
decision chronology.
Trust boundary: The gate validates supplied records, file integrity, policy rules, and recorded authority. It cannot determine whether a report is truthful or whether a test actually reached the intended system. Its value is procedural evidence binding and fail-closed release control, not independent AI assurance.
git clone https://github.com/jonathanjasare/evalseal
cd evalseal
python -m pip install dist/evalseal-0.1.0-py3-none-any.whl
evalseal gate \
--pack examples/fictional-support-assistant/evidence-pack \
--governance examples/fictional-support-assistant/governance.json \
--tailoring config/tailoring.reference.json \
--change-policy config/change-triggers.reference.json \
--as-of 2026-07-31T12:30:00Z \
--allow-fictionalExpected: Decision: SIMULATED_APPROVE, Authorizes deployment: false, exit
3. Without --allow-fictional, the example returns REJECT; it cannot
authorize deployment.
The wheel is committed so protected CI can verify and install the exact release artifact without rebuilding source. Attach that same wheel to the corresponding GitHub Release.
EvalSeal helps engineering, product, security, privacy, legal, and risk teams structure the evidence and accountable decision for an LLM release candidate. It covers six connected layers:
- Threat modelling and risk assessment
- Quality and task evaluation
- Bias, fairness, calibration, and abstention
- Security and adversarial testing
- Governance and accountability
- Release gates, monitoring, and incident response
The method must be tailored to the system's intended purpose, affected people, autonomy, data, jurisdictions, and potential harms. It is not a certification scheme, a substitute for legal advice, or evidence of conformity with any standard or law.
python -m pip install dist/evalseal-0.1.0-py3-none-any.whl
evalseal init my-assessment --candidate-id CAND-MY-SYSTEM-001 --name "My system"A plain install resolves the pinned jsonschema[format] dependency from PyPI. Protected pipelines should instead use the hash-locked install path in the operating guide.
Start with the ten-minute starter, then use the operating guide. Adopt organization-owned governance, tailoring, and change policies before assessing a real candidate. The supplied example and reference policies cannot authorize a production deployment.
The model proposes. Deterministic controls and accountable people decide what the system may accept or do.
- Treat model input and output as untrusted at every security boundary.
- Enforce authorization, data access, transaction limits, and tool permissions outside the model.
- Derive evaluation coverage and release thresholds from the system's risks and requirements. Do not copy universal scores or sample sizes.
- Prefer deterministic tests and qualified human review where they can directly establish the requirement.
- Treat model-graded evaluation as a fallible measurement instrument that must be validated for the task.
- A control is not implemented until evidence shows it exists. It is not verified until a current test shows it works.
Every control, test, and governance record must use one of these statuses:
| Status | Meaning | Minimum evidence |
|---|---|---|
| Proposed | An intention or recommendation only | Owner and target date |
| Implemented | The control exists in the assessed version | Code, configuration, procedure, or deployed resource |
| Verified | Current, scoped evidence shows the control met its acceptance criterion | Dated test or review result linked to the assessed version |
| Ineffective | Evidence shows the control failed or is insufficient | Finding and remediation owner |
| Not applicable | A documented scope decision excludes the control | Rationale and approver |
| Retired | The control is no longer used | Replacement or risk disposition |
Terms such as active, complete, compliant, and production-ready must not be used without a linked evidence record and an identified approver.
| Document | Purpose |
|---|---|
| 01-framework.md | Core components, measurement rules, lifecycle, and release gate |
| 02-threat-modelling-for-llm-apps.md | System boundaries, threat analysis, OWASP and ATLAS use, and risk treatment |
| 03-golden-set-design.md | Evaluation-set design, RAG evaluation, model-graded evaluation, and statistical rules |
| 04-governance-mapping.md | Accountability, evidence status, records, release decisions, monitoring, and incidents |
| 05-security-testing.md | Authorized adversarial testing and deterministic security controls |
| 06-bias-and-fairness.md | Harm analysis, disaggregated evaluation, fairness metrics, and legal cautions |
| SOURCE_REGISTER.md | Authoritative sources and the claims they support |
| CLAIMS_LEDGER.md | Historical fact-check and claim corrections |
| VERIFICATION_NOTES.md | Executable release checks, trust dependencies, and limitations |
| docs/OPERATING_GUIDE.md | Installation, evidence workflow, preflight, final gate, and protected CI requirements |
| docs/ARCHITECTURE_AND_TRUST_BOUNDARY.md | Why the gate exists, what it can prove, and what remains outside its trust boundary |
| docs/QUICKSTART.md | Minimal Tier 1 scaffold, automated hashing, and the shortest safe onboarding path |
| docs/TAILORING_GUIDE.md | Risk-tier scaling and conditional evidence requirements |
| docs/CHANGE_TRIGGERS.md | Partial, full, incident, and documentation reassessment rules |
| docs/GOVERNANCE_SETUP.md | Real roles, authority limits, identity boundary, and separation of duties |
| evidence-pack-template/ | Copyable production evidence-pack structure |
| machine-readable schemas | JSON Schema Draft 2020-12 contracts for policies and evidence |
| fictional worked example | Completed Tier 3 example that demonstrates the whole decision flow |
| docs/REFERENCE_EXAMPLE.md | Walkthrough and expected output for the fictional example |
| CONTRIBUTING.md | Branch naming, pull-request content, and review expectations |
| SECURITY.md | Supported versions and private vulnerability reporting |
| CHANGELOG.md | Release history and notable changes |
- Classify the system. Document intended and prohibited uses, users and affected people, data, jurisdictions, autonomy, and worst credible harms.
- Model the system and threats. Include all external models, retrieval sources, tools, identities, human decisions, and downstream renderers.
- Create the evidence pack. Build requirement-linked tests, risk treatments, data records, human-review protocols, and an incident plan.
- Set release criteria before testing. Define blocking scenarios, statistical decision rules, acceptable residual risk, and named approval authority.
- Evaluate the exact release candidate. Record model and prompt versions, code revision, retrieval index, tool configuration, environment, and repeated-run policy.
- Run preflight. Use the executable gate to identify missing, stale, inconsistent, or unauthorized evidence before asking people to approve.
- Make an evidence-based decision. Approve, approve with conditions, or reject. Bind the decision to the exact evidence and candidate. A failed blocking requirement cannot be averaged away by a high overall score.
- Run the final gate and deploy the exact candidate. Preserve the machine-readable gate report with the deployment record.
- Monitor and reassess. Re-evaluate after material changes, incidents, drift, provider updates, evidence expiry, and changes in law or intended use.
- NIST AI RMF 1.0 is voluntary, use-case agnostic guidance organized around GOVERN, MAP, MEASURE, and MANAGE. NIST states that AI RMF 1.0 is being revised, so mappings must be reviewed when the revision is published.
- OWASP Top 10 for LLM Applications 2025 supplies a risk taxonomy. OWASP does not assign the custom Critical, High, or Medium priorities that appeared in the earlier draft.
- MITRE ATLAS is a living knowledge base of adversary tactics and techniques. Use stable tactic and technique identifiers in records; do not rely on a hard-coded item count.
- ISO/IEC 42001:2023 specifies requirements for an artificial intelligence management system. This methodology can support an AIMS but does not establish conformity or certification. Exact clause or Annex A mappings require access to the licensed standard and review by a qualified assessor.
- The EU AI Act, Regulation (EU) 2024/1689, as amended by Regulation (EU) 2026/1744, applies different duties according to role and risk classification. The 2026 amendment moved the application of Chapter III Sections 1 to 3 to 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Article 6(1)/Annex I systems. Obtain a current applicability assessment rather than treating this repository as a compliance checklist.
| Term | Definition |
|---|---|
| Evaluation set | Versioned cases used to measure a system against specified requirements and risks. "Golden set" is used only as a familiar label, not as a claim that labels are infallible. |
| Model-graded evaluation | Use of a model to classify or score outputs against a rubric. It requires task-specific validation and human escalation rules. |
| Faithfulness | The degree to which claims in a response are supported by the supplied evidence. It is distinct from whether the evidence itself is true or complete. |
| Retrieval quality | The ability of a retriever to return relevant, authorized evidence with useful ranking and coverage. |
| Non-inferiority gate | A pre-specified statistical decision that a candidate is not worse than a baseline by more than an accepted margin. A raw score delta alone is not such a test. |
| Abstention | A system response that declines, defers, requests clarification, or routes to a human when the required evidence or authority is absent. |
| Calibration | Agreement between a declared probability and the observed frequency of the event that probability represents. |
| Release gate | An automated or human-controlled decision point that evaluates required evidence and blocks a release when a blocking criterion fails. |
- Review external references at least every six months and after a material legal or framework update.
- Record source access dates and methodology changes in SOURCE_REGISTER.md, CLAIMS_LEDGER.md, and VERIFICATION_NOTES.md.
- Recheck the claims ledger before a major version is issued.
Copyright 2026 Jonathan Asare. Licensed under the Apache License 2.0, including its explicit patent grant. The license does not certify, warrant, or approve any system assessed with this software.