InvarLock 0.13.0: one signed evaluation transaction

Release: InvarLock 0.13.0 — one closed request, one canonical evidence bundle, one independently replayable decision

Highlights

• A closed request.yaml authenticates the baseline and subject artifacts, dataset, paired schedule, providers, runtimes, metric or scorer, policy, execution mode, and evidence destination.
• invarlock evaluate request.yaml executes or imports both sides over one deterministic schedule and atomically publishes an Ed25519-signed invarlock/evidence-pack-v1 bundle.
• invarlock verify evidence/ treats that bundle as untrusted, replays the decision against verifier-owned anchors, and writes a separately signed verification receipt outside the bundle.
• invarlock report evidence/ authenticates and renders the same canonical comparison as console text or optional standalone HTML.
• Hugging Face text execution remains the built-in provider. GGUF/llama.cpp, Hugging Face vision-text, TensorRT-LLM, and observation-only diagnostics ship as coordinated optional distributions.

The product model is now one closed transaction

The main interface is no longer a loose set of baseline and subject flags followed by producer-owned report formats. It is one versioned request and one evidence destination:

invarlock evaluate request.yaml
invarlock verify evidence/
invarlock report evidence/

In run mode, the host authenticates local model artifacts and JSONL bytes, prepares the ordered schedule, launches constrained per-side OCI workers, validates both provider outputs, derives paired scores and uncertainty, publishes the closed bundle, and signs it without exposing the evidence-signing key to either worker. Import mode accepts complete authenticated provider sidecars and record-level material from another controlled execution; aggregate scores alone are not enough.

The Evaluation Request guide covers both modes. Schedule and Policy defines the source-byte identity, field mapping, ordered record identities, metric selection, threshold, and optional record-count and interval-width qualification.

The verifier owns the decision

The producer can publish a canonical comparison, but it cannot choose the verifier's expected values from inside the submitted bundle. Verification requires independent policy bytes plus expected baseline and subject artifact identities, canonical schedule digest, runtime digests, evidence-signer fingerprint, verifier identity, and verifier key. A trust profile can carry that closed input set.

Strict replay checks the fixed inventory, canonical JSON, paths and symlinks, checksums, evidence signature, versioned schemas, cross-bindings, record order, scores, paired statistics, interval endpoints, policy arithmetic, and final verdict. The result is an external signed receipt that binds the exact evidence and anchors used. Runtime-provider receipts remain canonical files inside the signed bundle; they are not separate runtime attestations.

The distinction matters. A valid evidence signature identifies which key signed which bundle bytes. A runtime digest identifies declared image bytes. Neither proves that an honest runtime executed or that the selected schedule is representative. The Trust Model, Threat Model, and Assurance Case keep those residual assumptions explicit.

Paired statistics are part of verifier replay

The request selects exactly one acceptance path:

• exact_match records baseline-only regressions, subject-only improvements, the exact two-sided McNemar probability, and a paired Newcombe 95% effect-size interval. Policy reads the interval's conservative lower bound.
• normalized_nll_per_utf8_byte compares arithmetic means of teacher-forced expected-continuation NLL per UTF-8 byte and uses 2,048 deterministic paired schedule-resampling replicates. Policy reads the interval's upper bound.
• An explicitly authorized deterministic text scorer can emit per-record values in [0,1]. Core verification still owns aggregation, paired resampling, policy interpretation, and verdict calculation.

When tokenizer contracts and paired token counts are comparable, normalized-NLL evidence can include a verifier-derived token-weighted perplexity ratio as interpretation. It has no separate policy or verdict authority. The exact arithmetic and legacy v1 replay rule are documented in Decision Semantics.

Runtime providers are explicit package boundaries

Install the core for verification and reporting, or add the built-in Hugging Face text execution dependencies:

python -m pip install invarlock
python -m pip install "invarlock[hf]"

Other first-party providers are separate coordinated packages:

The release also maintains complete example journeys for LM Evaluation Harness import, PEFT LoRA merge, TorchAO INT8 materialization, GGUF conversion and execution, vision-text, TensorRT-LLM, and offline evidence handoff. Those integrations preserve their own transformation or runtime boundary while ending in the same request, evidence, verification, and report transaction. See Runtime Providers and Change Scenarios.

Diagnostics moved out of acceptance

Spectral, random-matrix, and variance calculations now live in invarlock-diagnostics. Their canonical JSON can be attached as an authenticated observation and rendered in a separate report section, but it cannot determine whether the paired metric passes policy.

This is a deliberate boundary, not a rename of the old guard pipeline. The package produces descriptive observations; it does not ship calibrated safety, anomaly, or quality verdicts. If diagnostic work motivates a future acceptance rule, that rule must be defined, calibrated, reviewed, and versioned outside the current bundle. See Optional Diagnostics.

Public evidence is qualification-specific

The v0.13.0 public index admits only invarlock/evidence-pack-v1 comparisons tied to a checked-in qualification-suite manifest, balanced pinned 400-record MMLU-Pro or MMMU-Pro schedules, exact model and runtime identities, policy, evidence signer, verifier, and independent signed receipt. Where the defined large-sample evidence is absent, the status is Evidence not yet created rather than failure.

The current entries cover qualified Hugging Face text, Hugging Face vision-text, and TensorRT-LLM comparisons under their recorded suite conditions. They do not establish general quality or safety beyond those conditions. See Publish Public Evidence.

Migrating from 0.12.1

The former catalog and calibration command families, built-in model editing/training/quantization operations, masked-LM and sequence-to-sequence execution, custom observability stack, and duplicate report formats are removed. Existing signed legacy comparison-report v1 evidence remains replayable with its original interval method; new evaluations emit comparison-report v2.

For the immutable source record, read the v0.13.0 GitHub Release and tagged changelog.