eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add filesystem checkpoint compatibility tests

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-18 23:27:36 +00:00
parent e70450c401
commit 2cba6d4512
4 changed files with 1397 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

481
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,481 @@
# Test Suite Policy Review Implementation
## Purpose
This document is the decision-complete execution plan for implementing the
target revisions in [Test Suite Policy Review](tests.md). Follow the stages in
order. The feature roadmap owns the review evidence, desired test ownership,
required dispositions, and target state; this document owns sequencing,
file-level work, validation, and stop conditions.
This is test-suite harmonization, not a production feature change. Do not
modify production behavior. If a new durable test exposes incorrect production
behavior, preserve the failing evidence, stop the affected stage, and report
the defect separately for explicit scoping.
## Governing Policies
Before implementation, read and follow:
- [Testing Policy](../policy/testing.md), especially behavioral ownership,
semantic duplication, test doubles, exact diagnostics, and deletion criteria;
- [Architecture](../policy/architecture.md), especially dependency direction,
typed artifact boundaries, validation ownership, checkpoint safety, and
source/reference separation; and
- [Documentation Policy](../policy/documentation.md), especially canonical
ownership, current-versus-future behavior, and maintenance of test-routing
links.
The following constraints apply to every stage:
- preserve unrelated user changes in a dirty worktree;
- make no production-code changes unless separately authorized after reporting
a confirmed production defect;
- add no coverage-percentage target and do not use line coverage as a success
metric;
- do not mechanically convert tests to tables, consolidate leaf registration
tests, or introduce new test frameworks;
- keep the default suite deterministic, offline, credential-free, and safe for
repeated and parallel execution;
- use real filesystem collaborators with `t.TempDir()` for checkpoint behavior;
- assert typed/structured outcomes or stable category fragments rather than
complete incidental error wording; and
- do not update fixtures automatically or add a golden-update path.
## Stage 0 - Establish the implementation baseline
1. Read [Test Suite Policy Review](tests.md) completely, including the retained
ownership map and every required finding.
2. Inspect the worktree and preserve unrelated changes. Limit planned edits to
tests and the internal documentation routing identified below.
3. Run:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/framework/pipeline ./internal/framework/llm ./internal/cli ./internal/modules/integration
```
4. If the baseline fails for a reason unrelated to the planned work, stop and
report it. Do not rewrite expectations merely to obtain a clean baseline.
Stage 0 is complete when the starting state and any pre-existing worktree
changes are recorded and all baseline commands pass.
## Stage 1 - Protect filesystem checkpoint compatibility and recovery
Complete this stage before deleting or consolidating existing tests so the
highest-risk protection is strengthened first.
### 1.1 Checkpoint identity
Create `internal/framework/checkpoint/identity_test.go`. Test through
`NewIdentity` and `Identity.RelativePath`, not private normalization helpers.
Build one representative `pipeline.ResolvedPipeline` with two artifact lanes,
an input module, stable pipeline ID/digest, selected lanes, runtime
fingerprints, reference provenance, and provenance fingerprints. Add tests that
prove:
- reordering selected lanes, runtime fingerprints, references, provenance
fingerprints, or resolved lanes does not change the resulting identity;
- duplicate or blank selected-lane entries do not change the normalized lane
set, and blank fingerprint entries do not change the identity;
- changing each meaningful identity input independently changes the digest:
pipeline ID or digest, input key, raw/source digest, selected lane set,
runtime override value, reference digest or target identity, and provenance
fingerprint;
- an omitted explicit input key falls back to the resolved input module;
- missing pipeline ID, pipeline digest, input key, and both source/input digests
are rejected with stable category context; and
- `RelativePath` is deterministic, slash-separated, relative, confined, and
includes the normalized pipeline/input and digest-derived hierarchy without
asserting incidental private prefix lengths beyond the documented layout.
Use independent inputs for each mutation so one case cannot pass because a
different field also changed.
### 1.2 All-stage recorder/loader round trips
Create `internal/framework/checkpoint/filesystem_test.go`; leave the existing
schema-identifier test in `recorder_test.go`.
Using one real root and one identity, exercise the exported recorder and loader
for:
- source success with a valid document, self-references, metadata, and digest;
- extract success with at least one serialized artifact, schema identity,
content metadata, chunk provenance, warnings, dependency fingerprints, and
one rejected output so `StatusSucceededWithRejections` is round-tripped;
- merge success with one serialized artifact, warnings, and dependencies; and
- normalize success with one serialized artifact, warnings, and dependencies.
For each stage, assert the loader returns `Reused`, restores the meaningful
values, and preserves serialized bytes and codec identity. Mutate the original
inputs after recording and mutate one loaded result before reloading; neither
mutation may alter persisted or subsequently loaded state.
Inspect representative created directories and files to retain the `0700`/
`0600` permission contract on platforms where Unix permission bits are
meaningful. Do not snapshot the full directory tree or complete JSON documents.
### 1.3 Invalid and incompatible checkpoint state
Seed valid state through the recorder, then copy or edit one artifact per
subtest. Drive every case through the exported loader method for that stage.
Require a non-reused decision and a short category fragment for:
- missing artifact and malformed JSON;
- `WorkspaceSchemaVersionV1` and an unknown workspace schema version;
- mismatched checkpoint identity digest;
- wrong stage, lane, module, terminal status, or dependency fingerprint;
- incomplete serialized artifact kind/schema/schema digest;
- malformed base64 and content-digest mismatch;
- source document validation or source/output digest mismatch; and
- extract, merge, and normalize output-digest mismatch.
Include one successful `StatusSucceededWithRejections` extract case and prove
that non-reusable running, failed, pending, or invalidated statuses remain
non-reused. Assert categories, not complete sentences. Missing state should be
a normal non-reuse decision; corrupt or incompatible state must never panic or
silently reuse.
### 1.4 Stage validation
Run:
```sh
go test ./internal/framework/checkpoint ./internal/framework/pipeline ./internal/cli
go test -race ./internal/framework/checkpoint ./internal/framework/pipeline ./internal/cli
```
Stage 1 is complete when identity selection, every persisted stage, mutation
ownership, compatibility rejection, and corrupt-state recovery are protected at
the real filesystem boundary without production changes.
## Stage 2 - Remove fake-only, obsolete, and misleading tests
### 2.1 Framework contracts and helpers
In `internal/framework/contracts/contracts_test.go`, delete exactly:
- `TestFakeExtractorReturnsTypedOutput`;
- `TestFakeChunkerReturnsSourcePlan`;
- `TestFakeChunkerReceivesPerRunContext`;
- `TestFakeExtractorReceivesChunkAndAmbientContext`;
- `TestFakeMergeNormalizeAndOutputContracts`; and
- `TestReferenceSetDataTypes`.
After removing those tests, delete fake methods/types or imports only when they
have no remaining test use. Retain the compile-time interface assertions and
the reference/material/artifact clone and JSON non-leakage tests.
Delete `TestHelpersReturnValidationResults` from
`internal/framework/validate/validate_test.go`. Retain `TestApproved` and
`TestRejectedTrimsReasonAndMessage`.
Delete `internal/modules/generic/normalize/noop/typed_test.go`. Do not remove the
no-op normalizer's resolver, registration, runner, production-composition, or
maintained-example coverage. Retain the direct append-order merger ordering
test.
### 2.2 Obsolete Scriptorium grounding
Delete `internal/framework/llm/scriptorium_api_test.go` in full. Do not move its
unused API inventory elsewhere. Retain and run `scriptorium_client_test.go`,
`asset_registry_test.go`, module-local prompt preparation, cancellation,
validation, profile, and credential-redaction tests.
### 2.3 Misleading D&D integration tests
In `internal/modules/integration/dnd_spells_runner_test.go`, delete exactly:
- `TestRunnerDoesNotExtractSpellMentionedOnlyInPartyReference`;
- `TestRunnerCarriesDNDSpellCastWithInvalidSourceRefToSerializedOutput`; and
- `TestRunnerRejectsMalformedDNDSpellsArtifactAtSerializationBoundary`.
Remove helpers/imports only if unused afterward. Retain
`TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor` and
`TestRunnerPassesPartyAndGlossaryReferencesToDNDSpellsPrompt` as the two
representative cross-family workflows.
### 2.4 Stage validation
Run:
```sh
go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/validate
go test ./internal/modules/generic/normalize/noop ./internal/modules/integration ./internal/modules/dnd/...
```
Stage 2 is complete when all named low-value tests are gone, retained owners
still pass, and no production file changed.
## Stage 3 - Consolidate configuration, CLI state, and resolution ownership
### 3.1 Version 3 configuration tests
Before deleting `internal/core/config/v3_test.go`, preserve its only distinct
protections:
- add a version-2 input to the strict file-decoding cases in
`file_config_contract_test.go`; require rejection and the migration category,
not the complete diagnostic; and
- ensure `env_contract_test.go` positively proves that empty configured cache
roots resolve to distinct `notarius/chunk-plans` and
`notarius/checkpoints` descendants of the supplied per-user cache root.
Then delete `v3_test.go`. Do not duplicate its defaults, precedence,
redaction, invalid-source, or removed-field cases elsewhere.
### 3.2 CLI state and maintained examples
Move `emptyLookup` from `internal/cli/state_surfaces_test.go` to
`internal/cli/contract_test_helpers_test.go`, then delete
`state_surfaces_test.go` in full. Confirm its remaining behavior is still owned
by command/run contracts and `state_hardening_test.go`; do not transplant its
tests.
In `internal/cli/production_contract_test.go`:
- keep `TestDefaultCLICompositionResolvesMaintainedConfigurations`, but rename
it to `TestDefaultCLICompositionValidatesRepresentativeConfiguration` and
reduce it to one representative `config validate` command using empty/default
`Options`; this test owns fallback production composition, not
maintained-example enumeration;
- remove maintained-example resolution from
`TestProductionCatalogCoversMaintainedConfigurations` while retaining
required production registry members, typed codec/variant wiring, catalog
conversion, and the exact documented spell validator chain; and
- remove the maintained-example loop from
`TestProductionConfigValidationCoversModuleAndVariantFailures`, retaining one
valid baseline and each distinct failure mutation.
Do not weaken `example_contract_test.go`; it remains the sole owner for loading,
resolving, listing, and executing the maintained examples.
### 3.3 Default and Seriatim resolver duplication
Delete `internal/framework/pipeline/default_modules_test.go` in full.
Delete `internal/modules/seriatim/input/transcript/config_test.go` in full.
Delete `internal/modules/seriatim/input/transcript/testdata/pipeline.yml` with
it; the fixture is owned only by that deleted test file. Retain adapter parsing,
leaf registration, the Seriatim runner integration, generic resolver tests,
and production examples.
### 3.4 D&D capability integration
Rewrite `internal/modules/integration/dnd_spells_config_test.go` to own only two
cross-family capability failures:
1. removing `source.transcript` from the Seriatim input spec must make the D&D
spell extractor incompatible; and
2. removing `dnd.spell_casts` from the spell extractor spec must make the
append-order merger incompatible.
Use one compact table over a programmatically constructed profile and the
smallest catalog capable of resolution. Delete the successful-resolution,
stable-digest, and unknown-lane cases. Simplify or remove fixture/catalog
helpers that become unnecessary, but preserve helpers used by the retained
runner integration in sibling test files.
### 3.5 Stage validation
Run:
```sh
go test ./internal/core/config ./internal/cli ./internal/framework/pipeline
go test ./internal/modules/seriatim/... ./internal/modules/integration
```
Stage 3 is complete when each behavior has the owner specified above, the
maintained examples are not redundantly enumerated, and all distinct migration,
default-composition, and capability protections remain.
## Stage 4 - Rewrite brittle assertions at durable behavioral boundaries
### 4.1 Source validation ownership and diagnostics
In `internal/core/source/source_test.go`:
- consolidate malformed `SourceRef` categories under the `ValidateRef` tests;
- reduce `TestValidateDocumentUnitReferences` to one case proving nested
reference failures receive unit/document context and one case proving the
document-only self-reference invariant;
- retain valid documents, required document/unit fields, duplicate IDs,
non-empty units, valid/reversed/missing references, unit lookup, and digest
sensitivity/determinism; and
- replace exact complete error equality with the minimum stable field/category
fragments needed to distinguish each failure.
Do not add typed production errors during this pass. If stable fragments cannot
distinguish meaningful categories without a production change, retain the
narrowest current assertion and report that limitation rather than changing
production code.
### 4.2 Private LLM response schemas
In `internal/modules/dnd/chunk/scenes/schema_test.go`, replace the nested
`map[string]any` schema-structure walk with actual JSON Schema validation.
Add a small test helper that parses the instance and schema with
`jsonschema.UnmarshalJSON`, registers the schema with
`jsonschema.NewCompiler().AddResource`, compiles it, and calls
`schema.Validate`, matching the existing production validator boundary. Use a
representative valid scene response and mutations that reject:
- obsolete segment-based boundary fields;
- non-positive start/end unit IDs;
- invalid `primary_mode` and `boundary_confidence` values;
- empty boundary caveats; and
- unknown properties.
Retain identity/hash validity, DTO integer decoding, and mutation safety.
In `internal/modules/dnd/extract/spells/schema_test.go`, validate one legal
private LLM response whose references omit `source_id`, and prove that adding a
canonical `source_id` is rejected. Retain response identity/hash, mutation
safety, and diagnostics non-leakage. Do not conflate this private transport
schema with the durable codec schema or alter the maintained durable fixture.
### 4.3 Framework schema enumeration
In `internal/framework/llm/schema_registry_test.go`, remove only the assertion
that `RegisteredResponseSchemas` has exactly two entries. Continue to assert:
- returned keys are sorted;
- both required framework test schemas are present and valid;
- returned bytes are mutation-safe;
- diagnostics omit raw schema content; and
- D&D schemas are not registered in the domain-neutral framework registry.
### 4.4 Stage validation
Run:
```sh
go test ./internal/core/source ./internal/framework/llm
go test ./internal/modules/dnd/chunk/scenes ./internal/modules/dnd/extract/spells ./internal/modules/dnd/codec/spells
```
Stage 4 is complete when failures express behavioral categories, private
schemas are validated by accepted/rejected instances, and durable schema
compatibility coverage remains unchanged.
## Stage 5 - Make composition and architecture checks extension-friendly
### 5.1 Family registrars
Rewrite the success assertions in:
- `internal/modules/generic/register/register_test.go`;
- `internal/modules/seriatim/register/register_test.go`; and
- `internal/modules/dnd/register/register_test.go`.
Use required-membership helpers rather than exact equality for registered keys
and asset names. Prove representative family-owned entries are retrievable or
buildable through their registry boundary. Preserve:
- validation of all required registry/asset dependencies before any mutation;
- contextual failure on duplicate family registration;
- absence of cross-family composition where that is an explicit architectural
ownership rule; and
- exact D&D spell default-validator order, because it is documented production
policy.
Do not delete leaf spec/constructor/registration tests and do not move the CLI
production catalog into family tests.
### 5.2 Central import-boundary enforcement
Move the two rules from
`internal/framework/chunkplan/import_boundaries_test.go` into
`internal/modules/import_boundaries_test.go`:
- production files under `internal/core/source` may import neither
`internal/framework` nor `internal/modules`; and
- production files under `internal/framework/chunkplan` may not import
`internal/modules`.
Extend the centralized checker so its repository walk enforces those rules.
Add rule-level synthetic cases for both allowed and forbidden imports, using
the existing table/checker style. Retain the existing rejected fixture that
proves generic-to-concrete enforcement. Once both unique rules and checker
activation are protected centrally, delete
`internal/framework/chunkplan/import_boundaries_test.go`.
### 5.3 Internal documentation routing
Update only the `Tests To Inspect` routing needed to match the final suite in:
- `docs/internal/state.md`;
- `docs/internal/pipeline.md`;
- `docs/internal/modules.md`; and
- `docs/internal/llm.md`.
Replace deleted or nonexistent names such as
`internal/cli/state_surfaces_test.go` and `internal/cli/run_test.go` with concise
links or paths to the retained command/run contracts, state-hardening,
production-composition, maintained-example, checkpoint filesystem, and
cross-family integration owners. Do not create an exhaustive test inventory or
repeat subsystem contracts owned elsewhere.
### 5.4 Stage validation
Run:
```sh
go test ./internal/modules/... ./internal/framework/chunkplan
go test ./internal/cli ./internal/framework/pipeline ./internal/framework/llm
```
Stage 5 is complete when legitimate family additions no longer require closed
inventory edits, all architectural rules remain executable in one checker, and
internal documentation names only existing retained owners.
## Stage 6 - Repository acceptance and handoff
1. Review the final diff against every required finding and retained-owner
statement in [Test Suite Policy Review](tests.md). Confirm that no production
`.go` file changed.
2. Run formatting on changed Go test files, then run:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
go test -race ./internal/framework/pipeline ./internal/framework/llm ./internal/framework/checkpoint ./internal/cli ./internal/modules/integration
go test ./... -shuffle=on -count=5
```
3. Generate one coverage report for investigation. Confirm the new checkpoint
tests execute source, extract, merge, normalize, manifest-validation, and
corruption paths. Do not compare the percentage to the review baseline and
do not add tests merely to increase it.
4. Check documentation links and `git diff --check`. Confirm no fixtures were
updated automatically and no secrets or external-service requirements were
introduced.
Implementation is complete when:
- every checkpoint stage has a real filesystem round trip and corrupted or
incompatible state is never silently reused;
- every deletion or rewrite in the feature roadmap is complete while its named
retained owner still passes;
- maintained examples have one example-contract owner plus one narrow default
production-composition smoke path;
- registrar and architecture checks permit legitimate extension without
weakening documented ownership or validator order;
- the full suite is deterministic, offline, credential-free, race-clean, and
diagnostically useful;
- internal documentation points to existing test owners; and
- all validation commands pass with no production behavior change.
## Open Questions
None. The feature roadmap contains enough evidence and policy decisions to
implement every stage without additional testing-policy choices. If a new test
reveals a production defect, that is a scope boundary rather than an open
planning question: stop the affected stage and request explicit authorization
before changing production behavior.

324
docs/roadmap/tests.md
View File

@@ -23,8 +23,9 @@ important behavior.
## Objective
Review the complete Notarius test suite against the Testing Policy, identify
both low-value coverage and meaningful protection gaps, and then develop a
decision-complete staged implementation plan for the justified changes.
both low-value coverage and meaningful protection gaps, and define the
justified target revisions. The ordered execution plan is maintained separately
in [Implementation](implementation.md).
The review must answer:
@@ -151,27 +152,6 @@ group shares one clear disposition.
Distinguish required changes from optional cleanup. Absence of a finding is not
evidence that a package needs more tests.
## Implementation-Plan Deliverable
After completing the review, write a decision-complete staged implementation
plan based on the supported findings. The plan must:
- order work in small, independently verifiable package or behavior groups;
- state exactly which protections are retained when tests are consolidated,
rewritten, or deleted;
- add tests only for identified meaningful risks;
- avoid production behavior changes unless a separately identified production
defect is expressly brought into scope;
- identify focused and repository-wide validation for each stage; and
- include acceptance criteria demonstrating that the resulting suite remains
deterministic, offline, diagnostically useful, and sufficient under the
Testing Policy.
The plan must not establish a coverage-percentage target, require mechanical
conversion to table-driven tests, restore old tests wholesale, or equate fewer
tests with success. It should prefer correct, idiomatic, and maintainable tests
even when achieving the durable boundary requires more immediate work.
## Completion Criteria
The review is complete when:
@@ -183,5 +163,299 @@ The review is complete when:
- redundant or brittle protection is distinguished from valuable intentional
overlap;
- production defects, if any, are reported separately; and
- the resulting implementation plan can be executed without requiring the
implementing agent to make additional testing-policy decisions.
- every target revision is specific enough to support a decision-complete
implementation roadmap.
## Revisions Needed
### Review baseline
The review was completed on 2026-07-18 against 90 committed Go test files,
536 named tests, approximately 21,900 lines of test code, and the committed
fixtures under `testdata/` and `examples/`.
The baseline is clean:
- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass;
- `go test -race` passes for `internal/framework/pipeline`,
`internal/framework/llm`, `internal/cli`, and
`internal/modules/integration`;
- five shuffled repetitions of the full suite pass; and
- the default suite is offline, credential-free, and fast. The only HTTP
behavior uses local test servers or deliberately unreachable loopback
endpoints without making provider calls.
Coverage was used only to investigate risk. It confirms broad behavioral
coverage in CLI, configuration, source, pipeline, and production module
packages. It also exposes the material checkpoint gap described below: most
filesystem loader paths and all source, merge, and normalize checkpoint
round trips are unexercised. No production defect was identified during this
review.
### Test ownership that should be retained
The following overlap is intentional and should remain:
- CLI command, run, cache, reference, state-hardening, production-composition,
and maintained-example tests own exit classification, option mapping,
physical state placement, default production wiring, and representative
assembled workflows. Package tests continue to own the underlying parsing,
resolution, and persistence rules.
- Configuration contract tests own file decoding, precedence, validation,
redaction, and effective resolution. Pipeline resolver tests own module,
capability, typed-variant, validator-chain, and reference resolution after
configuration has produced a profile.
- Pipeline runner tests own bounded concurrency, deterministic ordering,
cancellation, retries, rejection propagation, checkpoint decisions, debug
recording, candidate/final encoding, and terminal failure behavior. The one
cross-family concurrency test remains valuable because it proves that the
framework worker bound and shared provider scheduler remain independent in
an assembled production-style pipeline.
- Source, chunk-plan store, file I/O, debug-bundle, output encoder, artifact
codec, and manifest tests own data integrity, confinement, serialization,
compatibility, atomic publication, redaction, and round trips. The exact
checkpoint schema identifier and exact artifact-schema digest tests are
compatibility tests, not incidental constant assertions, and must remain.
- Seriatim adapter, D&D scene chunker and spell extractor, D&D validators,
generic chunker and validators, prompt preparation, and durable spell codec
tests own their domain rules and external or durable schemas. Leaf
constructor/spec/registration tests remain justified because the extension
contract explicitly requires each leaf to be independently buildable.
- The centralized production import-boundary test remains a valuable
executable architecture rule. Its synthetic rejected fixture is necessary
to prove that the checker itself is active.
Absence from the revisions below means the current behavioral group has an
appropriate owner and no justified change was found.
### Required findings
#### P0 - add filesystem checkpoint compatibility and recovery coverage
**Disposition: add.** `internal/framework/checkpoint/recorder_test.go` currently
proves only that an empty successful extract checkpoint can be written and
loaded. Coverage confirms that `FilesystemLoader.Source`, `Merge`, `Normalize`,
manifest validation, payload digest validation, and most recorder status paths
are otherwise silent. CLI resume coverage proves root selection and one happy
reuse, while pipeline checkpoint tests use controlled collaborators; neither
owns the filesystem format.
Add package-level tests at the real recorder/loader boundary that:
- prove `NewIdentity` is deterministic under reordered lanes, references, and
fingerprints, changes when pipeline, source/input, selected lanes, runtime
overrides, references, or provenance change, and produces a confined stable
relative path;
- round-trip representative source, extract, merge, and normalize payloads,
including serialized artifact identity, warnings, extract rejections,
metadata, and dependency fingerprints;
- prove caller mutation cannot alter recorded or loaded values and retain the
existing restrictive-permission expectation through real files; and
- mutate one persisted artifact at a time to prove that missing or malformed
JSON, old or unknown workspace schema versions, wrong identity/stage/lane/
module/status/dependencies, incomplete codec identity, invalid base64, and
content/output-digest mismatches yield a non-reused decision with useful
category context rather than a panic or silent reuse.
Use relationships and category fragments rather than snapshotting complete
manifests or error sentences. This addition protects cache correctness,
compatibility, recovery, and sensitive durable state; it is not intended to
raise a coverage percentage.
#### P1 - remove tests that exercise only their own fakes or obsolete APIs
**Disposition: delete.** Remove the fake-behavior tests in
`internal/framework/contracts/contracts_test.go`:
`TestFakeExtractorReturnsTypedOutput`, `TestFakeChunkerReturnsSourcePlan`,
`TestFakeChunkerReceivesPerRunContext`,
`TestFakeExtractorReceivesChunkAndAmbientContext`,
`TestFakeMergeNormalizeAndOutputContracts`, and `TestReferenceSetDataTypes`.
They detect changes in test helpers or Go struct assignment, not defects in
production contracts. Retain the compile-time interface assertions and the
clone, JSON omission, and content-ownership tests, which protect type
compatibility and non-leakage invariants.
**Disposition: delete.** Remove
`internal/framework/llm/scriptorium_api_test.go`. It was introduced as
pre-adapter API grounding and still mirrors unused Scriptorium request/result
fields, option constructors, and sentinel errors. Production compilation now
grounds the API actually used, while `scriptorium_client_test.go` and module
prompt-preparation tests protect adapter mapping, local integration,
cancellation, validation, profile selection, and credential redaction.
**Disposition: delete.** Remove
`TestHelpersReturnValidationResults` from
`internal/framework/validate/validate_test.go`; it is a compile-only assignment
that adds no protection beyond the function signatures and the two behavioral
helper tests.
**Disposition: delete.** Remove the direct
`TestTypedNormalizerPreservesValue` test for the generic no-op normalizer. Its
single assignment is already exercised through resolver, runner, production
composition, and maintained-example output tests. Retain the direct append-order
merger test because input ordering is a meaningful transformation invariant.
#### P1 - remove misleading D&D integration claims
**Disposition: delete.** In
`internal/modules/integration/dnd_spells_runner_test.go`, delete:
- `TestRunnerDoesNotExtractSpellMentionedOnlyInPartyReference`, whose fake LLM
is programmed to return no spells and therefore cannot prove the claimed
extraction policy;
- `TestRunnerCarriesDNDSpellCastWithInvalidSourceRefToSerializedOutput`, which
omits the production validator chain and duplicates the extractor's explicit
invalid-evidence handoff test; and
- `TestRunnerRejectsMalformedDNDSpellsArtifactAtSerializationBoundary`, whose
final-codec invariant is already owned by the codec and runner candidate/
final-encoding tests.
Retain the representative Seriatim-to-spell assembled workflow and the test
that proves party/glossary references and provenance cross the production
module boundary. The policy that reference material is not source evidence is
durably protected by extractor mapping, prompt-input assertions, source-ref
validators, and the production validator-chain composition test; it cannot be
proved by controlling an LLM stub's semantic answer.
#### P1 - consolidate superseded configuration and CLI state tests
**Disposition: consolidate.** Delete `internal/core/config/v3_test.go` after
moving its two distinct protections into the current contract owners:
- add the version-2 migration rejection case to the strict file-decoding cases
in `file_config_contract_test.go`; and
- ensure the positive per-user chunk-plan/checkpoint root separation remains
in `env_contract_test.go`.
The remaining default, precedence, invalid-source, redaction, and removed-field
assertions are already more completely owned by `file_config_contract_test.go`,
`env_contract_test.go`, `validation_contract_test.go`, and
`redaction_test.go`.
**Disposition: consolidate.** Delete `internal/cli/state_surfaces_test.go`.
Move the shared `emptyLookup` helper to
`contract_test_helpers_test.go`. Its debug flag syntax, allocation timing,
no-debug absence, version-3 validation, and removed-field cases are already
owned by `run_contract_test.go`, `state_hardening_test.go`,
`command_contract_test.go`, and the configuration contract tests. Preserve the
state matrix, failure retention, no-debug terminal-writer, and pre-resolution
debug allocation protections in those current owners.
**Disposition: rewrite.** Keep
`TestDefaultCLICompositionResolvesMaintainedConfigurations` as a narrow default
composition smoke test, but run one representative config-validation command
through empty `Options` rather than revalidating every maintained example.
`example_contract_test.go` owns both maintained examples. Remove the repeated
maintained-example loops from `TestProductionCatalogCoversMaintainedConfigurations`
and `TestProductionConfigValidationCoversModuleAndVariantFailures`; retain the
production registry/codec/default-chain checks and the distinct production
failure cases.
#### P1 - consolidate resolver and module-composition duplication
**Disposition: delete.** Remove
`internal/framework/pipeline/default_modules_test.go`. Its large local catalog
only re-proves default binding selection, already owned by
`profile_test.go` and `effective_config_contract_test.go`; production default
keys and wiring are separately exercised by leaf registration, the production
catalog, default CLI composition, and the maintained end-to-end example.
**Disposition: delete.** Remove
`internal/modules/seriatim/input/transcript/config_test.go`. It builds an
entire fake catalog to repeat generic resolver success, deterministic digest,
missing-capability, and unknown-lane behavior. Retain Seriatim parsing and leaf
registration tests, the Seriatim runner integration, D&D cross-family
capability tests, resolver contract tests, and production CLI examples.
**Disposition: consolidate.** In
`internal/modules/integration/dnd_spells_config_test.go`, retain only the two
cross-family capability contracts: Seriatim must provide `source.transcript`
to the spell extractor, and the spell extractor must provide
`dnd.spell_casts` to append-order. Express them as one compact table over a
programmatic profile/catalog. Remove the generic successful-resolution,
stable-digest, and unknown-lane cases, which are already exercised by the
retained runner integration and resolver/configuration owners.
#### P1 - replace brittle assertions with behavioral ones
**Disposition: rewrite.** In `internal/core/source/source_test.go`, make
`ValidateRef` the owner of malformed reference categories. `ValidateDocument`
should retain one propagation/context case plus the document-only self-reference
invariant instead of repeating missing, foreign, and reversed-reference cases.
Replace complete internal error-sentence equality with the smallest stable
category and field fragments. Retain exact acceptance/rejection boundaries,
duplicate detection, deterministic digests, and reference ordering.
**Disposition: rewrite.** Replace the decoded-schema-structure walk in
`internal/modules/dnd/chunk/scenes/schema_test.go` with representative JSON
Schema validation: accept a valid source-unit-boundary response and reject old
segment fields, non-positive bounds, invalid enums, empty caveats, and unknown
properties. Likewise, change the private spell response-schema assertion in
`internal/modules/dnd/extract/spells/schema_test.go` to accept the LLM transport
shape without `source_id` and reject a response that supplies canonical
`source_id`. Retain schema identity/hash, mutation safety, diagnostics
non-leakage, DTO decoding, and the codec's durable fixture tests. These rewrites
preserve schema regression protection without coupling tests to nested map
layout or unsafe type assertions.
**Disposition: rewrite.** In `internal/framework/llm/schema_registry_test.go`,
assert that returned keys are sorted and required framework test schemas are
present without asserting that the registry contains exactly two entries.
Retain the explicit negative D&D lookup because it protects the framework/
domain ownership boundary.
#### P2 - make composition and architecture tests open to legitimate extension
**Disposition: rewrite.** The family registrar tests in
`internal/modules/generic/register`, `internal/modules/seriatim/register`, and
`internal/modules/dnd/register` currently assert closed-world key and asset
inventories that duplicate the CLI production catalog. Change them to require
the family-owned registrations they need, permit unrelated future additions,
and prove representative entries can be built or prepared. Keep exact order
for the D&D default spell validator chain because that order is documented
production policy. Keep missing-dependency-before-mutation and contextual
duplicate-registration cases.
**Disposition: consolidate.** Move the source/framework independence rules
from `internal/framework/chunkplan/import_boundaries_test.go` into the
centralized `internal/modules/import_boundaries_test.go` checker and delete the
second repository walker. Preserve both unique rules: `internal/core/source`
may import neither framework nor modules, and `internal/framework/chunkplan`
may not import modules. Add rule-level cases so a broken checker still fails.
#### Documentation alignment
**Disposition: rewrite.** When the affected tests move or are deleted, update
the `Tests To Inspect` sections in `docs/internal/state.md`,
`docs/internal/pipeline.md`, `docs/internal/modules.md`, and
`docs/internal/llm.md`. They currently name files such as
`internal/cli/state_surfaces_test.go` and `internal/cli/run_test.go` that will
be deleted or do not exist. Point each document at the retained contract,
state-hardening, production-composition, example, checkpoint, and integration
owners without recreating an exhaustive test inventory.
### Optional cleanup
No additional cleanup is recommended now. In particular, do not mechanically
convert the large resolver and runner suites to table-driven form, merge all
leaf registration tests into family registrars, add tests for trivial accessor
coverage, or introduce a golden-update framework. Those changes do not provide
enough additional confidence to justify their immediate cost.
### Target state
The revision is complete when:
- every checkpoint stage has a real filesystem round trip and incompatible or
corrupted state is demonstrably recomputed rather than silently reused;
- all protection named as retained above remains present at its stated owner;
- the fake-only, obsolete, misleading, duplicate, and closed-world assertions
named in the required findings are removed or rewritten exactly as specified;
- maintained examples are each owned by one example contract plus one narrow
default-composition smoke path, rather than repeated across production tests;
- default tests remain deterministic, offline, credential-free, order
independent, and race-clean;
- failures identify the violated behavioral category without snapshotting
complete incidental diagnostics;
- internal documentation points to existing retained test owners; and
- the suite reaches this state without changing production behavior.