notarius/docs/roadmap/tests.md

Test Suite Policy Review

Context

Notarius now has a canonical Testing Policy. The policy defines a risk-based approach that favors durable behavioral coverage while removing redundant, brittle, misleading, or obsolete tests whose lifetime cost exceeds their protective value.

Much of the existing test suite predates that policy. Recent CLI and configuration work has already applied several of its principles, but the suite has not been reviewed consistently as a whole. This roadmap calls for that review before additional domain capabilities materially expand the number of tests and fixtures.

The review is not based on a presumption that old tests are bad, that direct tests of package-private behavior must be removed, or that the suite should be made smaller at any cost. Existing tests should be retained when they provide durable and nonredundant protection for a meaningful risk. The goal is a leaner and clearer allocation of test ownership together with credible protection for important behavior.

Objective

Review the complete Notarius test suite against the Testing Policy, identify both low-value coverage and meaningful protection gaps, and then develop a decision-complete staged implementation plan for the justified changes.

The review must answer:

• Which important contracts, invariants, failure modes, and integration boundaries does the current suite protect?
• Where do multiple tests protect the same behavior without providing distinct failure isolation or integration confidence?
• Which tests are coupled to incidental implementation details, duplicated policy literals, closed-world inventories, unstable formatting, or mock choreography?
• Which important risks can still fail silently despite the existing suite?
• Which tests should be retained, consolidated, rewritten, or deleted, and which new tests are warranted?

Review Scope

Review all committed Go tests and their supporting fixtures. Organize the work by behavioral layer rather than treating test count or package coverage as the unit of quality.

CLI and configuration

Review command parsing, configuration decoding and precedence, validation, effective configuration, run controls, reference selection, output/cache/debug state, production composition, and maintained examples.

Pay particular attention to:

• duplicated assertions across parser, resolver, CLI, and assembled-run tests;
• tests that restate complete defaults or registry contents rather than protecting operator-visible behavior;
• exact error or output assertions broader than the documented CLI contract;
• fixture mutation that can silently stop establishing a test precondition; and
• whether representative CLI workflows provide sufficient assembled coverage.

Framework and durable state

Review pipeline preparation and execution, artifact contracts, validation and retry behavior, LLM scheduling and transport seams, chunk-plan storage, checkpoint compatibility, debug bundles, and file persistence.

Presume durable protection is important for data integrity, serialization, compatibility, cancellation, concurrency, cache correctness, resume behavior, atomic or failure-safe persistence, and recovery. Look for opportunities to replace many narrow structural tests with a smaller invariant, round-trip, or behavior-level test only when protection is not weakened.

Modules and domain behavior

Review generic, seriatim, and D&D module tests, including codecs, chunking, extraction, merging, normalization, validators, registration, prompt assets, and module integration.

Confirm that domain rules and artifact schemas have clear test ownership. Identify tests that merely reproduce schemas, prompt asset inventories, or implementation structure, while preserving tests that protect compatibility, source provenance, normalization, validation, or other consequential domain invariants.

Cross-cutting suite quality

Across all packages, evaluate:

• deterministic, offline, credential-free execution;
• isolation from mutable machine and process-global state;
• appropriate use of real collaborators, fakes, stubs, and mocks;
• stable behavioral assertions and useful failure diagnostics;
• golden files and large snapshots;
• helper and fixture complexity;
• test runtime, race safety, repetition stability, and parallel-execution assumptions; and
• semantic ownership and duplication across package, integration, and end-to-end layers.

Use coverage only as a diagnostic to locate unexpectedly untested critical branches. Do not recommend tests solely to increase a percentage or make coverage uniform across packages.

Review Method

1. Read the Testing Policy and the canonical documentation for each subsystem before judging its tests.
2. Establish a clean baseline with the repository validation commands, focused race tests where concurrency or shared state is relevant, and a coverage report used only for investigation.
3. Inventory tests and map each meaningful test or closely related group to the contract, invariant, integration boundary, or regression it protects.
4. Inspect production code only as needed to understand the protected behavior, identify the stable boundary, and detect untested risk. Do not infer a contract merely from current implementation detail.
5. Evaluate marginal value across layers. Similar assertions are not redundant when one owns a package contract and another distinctly proves production wiring or end-to-end integration.
6. Record evidence for every proposed change. Name the affected test or fixture, the realistic defect it currently catches or fails to catch, and why the recommendation improves confidence or reduces unnecessary friction.
7. Check historical context when a test appears unusually specific. Preserve a regression test when the underlying defect remains plausible and consequential, even if its purpose is not obvious from the current code.

Do not modify production code or tests during the review. If the review reveals incorrect production behavior, report it separately from test-suite harmonization rather than treating a changed test expectation as the fix.

Review Deliverable

Produce an evidence-backed report organized by priority and subsystem. Each finding must classify the proposed disposition as one of:

• retain: valuable protection at an appropriate boundary;
• consolidate: overlapping protection that can be represented more simply;
• rewrite: meaningful protection expressed through a brittle or misleading boundary;
• delete: no sufficient plausible defect or distinct protection justifies the lifetime cost; or
• add: a consequential risk lacks credible protection.

For consolidate, rewrite, delete, and add findings, describe the protected risk, current evidence, recommended boundary, and expected effect on confidence and maintenance. Do not produce a raw list of every test when a package or related group shares one clear disposition.

Distinguish required changes from optional cleanup. Absence of a finding is not evidence that a package needs more tests.

Implementation-Plan Deliverable

After completing the review, write a decision-complete staged implementation plan based on the supported findings. The plan must:

• order work in small, independently verifiable package or behavior groups;
• state exactly which protections are retained when tests are consolidated, rewritten, or deleted;
• add tests only for identified meaningful risks;
• avoid production behavior changes unless a separately identified production defect is expressly brought into scope;
• identify focused and repository-wide validation for each stage; and
• include acceptance criteria demonstrating that the resulting suite remains deterministic, offline, diagnostically useful, and sufficient under the Testing Policy.

The plan must not establish a coverage-percentage target, require mechanical conversion to table-driven tests, restore old tests wholesale, or equate fewer tests with success. It should prefer correct, idiomatic, and maintainable tests even when achieving the durable boundary requires more immediate work.

Completion Criteria

The review is complete when:

• the full suite has been considered at an appropriate behavioral grouping;
• important test ownership and integration boundaries are mapped;
• every recommended change is supported by a concrete risk and evidence;
• high-risk behavior without credible protection is identified;
• redundant or brittle protection is distinguished from valuable intentional overlap;
• production defects, if any, are reported separately; and
• the resulting implementation plan can be executed without requiring the implementing agent to make additional testing-policy decisions.