Reported
Information supplied by a source but not yet independently verified.
Shared standardVersioned evidence
Evidence travels with the work.
The Labs record connects the request, execution conditions, observed behavior, technical analysis, review decision, correction history, and release state. A score without that chain is an observation, not an acceptance record. This page defines the standard; it does not claim that every record already exists.
Analytical language.
These terms separate source claims, direct evidence, interpretation, review decisions, and approved release.
Observed
Behavior or state directly present in preserved artifacts or execution records.
Reproduced
The same material behavior observed again under documented conditions.
Inferred
An interpretation supported by evidence but not directly observed as a fact.
Decided
An accountable disposition recorded after review of evidence and limitations.
Released
A version approved for its stated public or commercial use.
Evidence model.
Five layers preserve enough context for another qualified reviewer to reconstruct the decision.
Identity
Stable program, task, environment, evaluator, attempt, and release identifiers connect every artifact to one version.
Lineage
Source, rights posture, author, reviewer, timestamps, revisions, and supersession preserve who changed what and why.
Conditions
Model, harness, tools, seed, network policy, resources, and environment state make attempts comparable.
Behavior
Commands, outputs, artifacts, evaluator results, requirement outcomes, and intervention points preserve what occurred.
Decision
Findings, corrections, limitations, disagreements, approvals, and disposition explain why a state advanced or stopped.
Acceptance package.
An accepted unit must carry the artifacts necessary to inspect scope, validity, behavior, and accountability.
Specification
Capability question, task contract, starting state, constraints, acceptance criteria, and approval boundaries.
Rights record
Authorship, source history, permitted evaluation, training, demonstration, and redistribution uses, holdout status, confidentiality review, and unresolved restrictions.
Environment manifest
Dependencies, images, services, reset procedure, health checks, tools, network, and resource profile.
Reference record
Verified expected behavior, legitimate alternatives, expected artifacts, and known non-goals.
Evaluator record
Requirement-to-test map, reward semantics, invalid-state handling, control coverage, and shortcut or exploit checks.
Attempt ledger
Every valid, invalid, retried, or intervened run with versioned conditions, reward components, and artifact links.
Analysis dossier
Observed outcome, competing explanations, evidence, interventions, classification, and confidence.
Review record
Independent reproduction, findings, adjudication, corrections, limitations, and accountable sign-off.
Release history
Version, date, disposition, superseded record, regression impact, and rerun or regrade decision.
Attempt ledger.
The ledger is the join between the released package and the analysis. It prevents selective reporting and incomparable aggregation.
- Task, environment, evaluator, harness, and release version
- Model or system configuration, tools, seed, limits, and trial purpose
- Start state, reset result, health status, timestamps, and infrastructure incidents
- Raw outcome, component outcomes, artifacts, score, and normalized disposition
- Invalid-run reason, retry relationship, human intervention, and comparable-attempt group
- Failure domain, supporting evidence, reviewer decision, and follow-up action
Interpretation controls.
These rules keep attractive summaries from outrunning the underlying record.
Component is not whole
A passing subscore cannot be presented as complete-task success.
Version changes the result
Attempts from materially different task or evaluator versions are not silently aggregated.
Invalid is not failure
Broken setup, service incidents, corrupted state, and unauthorized interventions are classified separately.
Retry stays visible
The successful rerun does not erase the earlier attempt or the reason it was repeated.
Absence is explicit
Missing evidence remains unknown; it is never converted into a passing state.
Release disposition.
Review ends in an explicit state with a named owner and next action.
Accept
Evidence is internally consistent, independently reproducible, and sufficient for the stated claim.
Revise
The intended work remains valid, but a correctable defect requires another validation cycle.
Quarantine
A material dependency, rights, infrastructure, or evidence issue prevents interpretation or sharing.
Reject
The package cannot validly measure the intended capability or meet the release standard.
Minimum publication test.
A public claim must identify the artifact, version, date, reviewer, evidence, observed result, and limitation it relies on.
- Required
- Reproducible package, visible controls, independent decision, and known limitations
- Insufficient
- Inventory totals, screenshots, file-presence checks, isolated scores, or unsupported failure labels