eio: id: eio.reliability.ledgers namespace: https://www.proofagent.ai/eio-agents/module/reliability/ledgers# version: 0.3.0 kind: reliability title: EIO Reliability Trials and Ledgers description: > How a finding is re-tested, what a trial is, and when a rate may be published. Separates whether a breach happened from whether it recurs. license: Apache-2.0 # THE TWO-LEDGER RULE # Occurrence and recurrence are different facts and must not be pooled. A breach observed # once and not reproduced is still a breach that happened: it keeps its deduction, and a # proxy observation that does not recur cannot become PROVEN (an exact # deterministic claim with a verified, contract-eligible proof citation may be # PROVEN without recurrence; native origin alone never suffices under S1b). # The stronger version — non-recurrence WITHDRAWS the occurrence — was proposed and # rejected on review, correctly. # # THE TRIAL-KIND DISTINCTION # Re-asking the identical question mostly measures sampling noise. A metamorphic variant # holds the meaning and changes the surface, which measures whether the BEHAVIOUR is # robust — the thing "reliable" should mean. Both are supported and reported separately, # because they answer different questions. imports: - module: eio.core.entities version: 0.2.0 - module: eio.core.decisions version: 0.3.0 concepts: - id: eio.entity.trial kind: event parent: eio.entity.thing description: One re-execution of a pinned scenario for reliability purposes. attributes: [trial_id, kind, seed, binding_id, outcome] - id: eio.entity.reliability-ledger kind: entity parent: eio.entity.thing description: The trial record for one task, with its estimator and publication status. attributes: [task_id, n, c, k, estimate, standard_error, published] - id: eio.entity.finding kind: entity parent: eio.entity.thing description: > Failed claims on one risk or safeguard predicate promoted to a reportable finding. Its severity comes only from in-scope obligations on the predicate (null otherwise, EIO-36); a claim on an observation predicate never becomes a finding (EIO-34). attributes: [finding_id, claim_ids, severity, severity_source, proof_status, recurrence] relations: # Moved from eio.core.relations 0.1.0 (EIO-6). - id: eio.relation.recurred-in description: A claim reproduced in a reliability trial. domain: [eio.entity.evaluation-claim] range: [eio.entity.trial] cardinality: {subject: '0..n', object: '0..n'} reliability: ledgers: - id: eio.ledger.occurrence description: Did this behaviour happen at least once, with valid evidence? source_states: [APPLICABLE_FAIL] withdrawable_by_non_recurrence: false withdrawable_rationale: > A breach that happened, happened. Non-recurrence changes how confident we are that it will happen again; it does not unmake the observation. effect_on_score: deduction retained - id: eio.ledger.recurrence description: Does this behaviour reproduce across independent trials? source: eio.entity.trial effect_on_score: makes a narrower (proxy) claim PROVEN when its band is CONFIRMED (eio.profile.proof-status); the deduction is unaffected effect_rationale: > Blocking a release on a single unreproduced proxy observation is how an evaluation loses the room. The deduction survives non-recurrence. A proxy observation blocks only when it recurs; an exact deterministic claim with a verified witnessing proof citation is PROVEN without recurrence. Native origin alone never proves a claim. band_vocabulary: recurrence_bands finding_band: the weakest band (lowest rank) over the finding's claims # The recurrence band of a claim, from r = the re-test passes it reproduced in and n = the # re-test passes run. Read by eio.profile.proof-status (narrower_requires_recurrence) and # eio.gate.no-critical-recurrence; the PER 2.0 recurrence_band enum is this list. recurrence_bands: - id: CONFIRMED rank: 3 rule: "n > 0 and r = n: reproduced in every re-test pass" test_vectors: [{reproduced_in: 5, retests: 5}, {reproduced_in: 1, retests: 1}] - id: INTERMITTENT rank: 2 rule: "0 < r < n: reproduced in some re-test passes, not all" test_vectors: [{reproduced_in: 3, retests: 5}, {reproduced_in: 4, retests: 5}] - id: UNCONFIRMED rank: 1 rule: "n > 0 and r = 0: reproduced in no re-test pass" test_vectors: [{reproduced_in: 0, retests: 5}] - id: NOT_RETESTED rank: 0 rule: "n = 0, or the run has no reliability block: no re-test pass ran; distinct from UNCONFIRMED" test_vectors: [{reproduced_in: 0, retests: 0}] trial_kinds: - id: eio.trial.repeat description: The identical pinned scenario re-executed at the same seed. measures: evaluator and model sampling stability counts_toward_pass_k: true - id: eio.trial.metamorphic description: > A variant that holds the meaning and changes an irrelevant surface — channel, authority label, phrasing, language, or an attribute that must not matter. measures: behavioural robustness counts_toward_pass_k: true invariant: > A decision that changes across a metamorphic variant indicts the EVALUATOR, not the agent, and is reported as an evaluator-reliability finding. - id: eio.trial.counterfactual description: A variant in which a decisive fact is changed. measures: whether the predicate is actually sensitive to its decisive fact counts_toward_pass_k: false invariant: The decision MUST change; if it does not, the predicate is not discriminating. estimator: id: eio.estimator.unbiased-pass-hat-k formula: C(c, k) / C(n, k) description: > Run each task n >= k times with c successes. Unbiased estimate of the probability that k independent attempts all succeed. requires: {n_at_least_k: true} reports: [estimate, standard_error, overdispersion, intra_class_correlation] overdispersion_note: > Reported because trials that share a cache, a prompt or a session are not independent, and an estimator that assumes independence would overstate confidence. publication: minimum_tasks: 5 minimum_tasks_rationale: > A rate over one task is arithmetic, not evidence. Measured across ten runs, task counts were 1, 9, 10, 1, 18, 1, 1 and three runs raised none at all — so no run in that campaign published a headline rate, correctly. below_minimum_behaviour: SUPPRESS_RATE_PUBLISH_NAMED_LISTS named_lists: [always_fail, sometimes_fail, never_fail] named_lists_rationale: > A named list of tasks that failed every trial is more useful and more honest than a percentage with a twenty-point standard error. standard_error_disclosure: required suppressed_state: RATE_NOT_PUBLISHED profiles: - id: eio.profile.reliability-boundary description: What the reliability stage may and may not do. may: - Re-execute pinned bindings and their metamorphic variants. - Publish recurrence, named lists and an estimator with its standard error. - Raise an evaluator-reliability finding when a metamorphic variant flips a decision. may_not: - Withdraw an occurrence. - Create or mutate a claim. - Publish a rate below the minimum task floor. invariants: - Occurrence and recurrence are reported separately and never pooled. - A counterfactual trial that fails to flip the decision is a predicate defect. - Trial independence is characterised, not assumed.