Add testing policy documentation

docs/roadmap/future.md

@@ -41,12 +41,6 @@ future work only.
 - Optional generated example output fixtures with a regeneration procedure.
 - Additional diagnostics or reporting views if operator workflows need them.
 
-## Candidate Developer Work
-
-- Reassess broad CLI and configuration regression coverage, including whether
-  to retain a legacy-checkpoint compatibility fixture and adopt a coverage
-  policy.
-
 ## Candidate Workspace Work
 
 - Default-idempotent run behavior with an explicit force override.

docs/roadmap/tests.md

@@ -0,0 +1,187 @@
+# Test Suite Policy Review
+
+## Context
+
+Notarius now has a canonical [Testing Policy](../policy/testing.md). The policy
+defines a risk-based approach that favors durable behavioral coverage while
+removing redundant, brittle, misleading, or obsolete tests whose lifetime cost
+exceeds their protective value.
+
+Much of the existing test suite predates that policy. Recent CLI and
+configuration work has already applied several of its principles, but the suite
+has not been reviewed consistently as a whole. This roadmap calls for that
+review before additional domain capabilities materially expand the number of
+tests and fixtures.
+
+The review is not based on a presumption that old tests are bad, that direct
+tests of package-private behavior must be removed, or that the suite should be
+made smaller at any cost. Existing tests should be retained when they provide
+durable and nonredundant protection for a meaningful risk. The goal is a leaner
+and clearer allocation of test ownership together with credible protection for
+important behavior.
+
+## Objective
+
+Review the complete Notarius test suite against the Testing Policy, identify
+both low-value coverage and meaningful protection gaps, and then develop a
+decision-complete staged implementation plan for the justified changes.
+
+The review must answer:
+
+- Which important contracts, invariants, failure modes, and integration
+  boundaries does the current suite protect?
+- Where do multiple tests protect the same behavior without providing distinct
+  failure isolation or integration confidence?
+- Which tests are coupled to incidental implementation details, duplicated
+  policy literals, closed-world inventories, unstable formatting, or mock
+  choreography?
+- Which important risks can still fail silently despite the existing suite?
+- Which tests should be retained, consolidated, rewritten, or deleted, and
+  which new tests are warranted?
+
+## Review Scope
+
+Review all committed Go tests and their supporting fixtures. Organize the work
+by behavioral layer rather than treating test count or package coverage as the
+unit of quality.
+
+### CLI and configuration
+
+Review command parsing, configuration decoding and precedence, validation,
+effective configuration, run controls, reference selection, output/cache/debug
+state, production composition, and maintained examples.
+
+Pay particular attention to:
+
+- duplicated assertions across parser, resolver, CLI, and assembled-run tests;
+- tests that restate complete defaults or registry contents rather than
+  protecting operator-visible behavior;
+- exact error or output assertions broader than the documented CLI contract;
+- fixture mutation that can silently stop establishing a test precondition;
+  and
+- whether representative CLI workflows provide sufficient assembled coverage.
+
+### Framework and durable state
+
+Review pipeline preparation and execution, artifact contracts, validation and
+retry behavior, LLM scheduling and transport seams, chunk-plan storage,
+checkpoint compatibility, debug bundles, and file persistence.
+
+Presume durable protection is important for data integrity, serialization,
+compatibility, cancellation, concurrency, cache correctness, resume behavior,
+atomic or failure-safe persistence, and recovery. Look for opportunities to
+replace many narrow structural tests with a smaller invariant, round-trip, or
+behavior-level test only when protection is not weakened.
+
+### Modules and domain behavior
+
+Review generic, seriatim, and D&D module tests, including codecs, chunking,
+extraction, merging, normalization, validators, registration, prompt assets,
+and module integration.
+
+Confirm that domain rules and artifact schemas have clear test ownership.
+Identify tests that merely reproduce schemas, prompt asset inventories, or
+implementation structure, while preserving tests that protect compatibility,
+source provenance, normalization, validation, or other consequential domain
+invariants.
+
+### Cross-cutting suite quality
+
+Across all packages, evaluate:
+
+- deterministic, offline, credential-free execution;
+- isolation from mutable machine and process-global state;
+- appropriate use of real collaborators, fakes, stubs, and mocks;
+- stable behavioral assertions and useful failure diagnostics;
+- golden files and large snapshots;
+- helper and fixture complexity;
+- test runtime, race safety, repetition stability, and parallel-execution
+  assumptions; and
+- semantic ownership and duplication across package, integration, and
+  end-to-end layers.
+
+Use coverage only as a diagnostic to locate unexpectedly untested critical
+branches. Do not recommend tests solely to increase a percentage or make
+coverage uniform across packages.
+
+## Review Method
+
+1. Read the Testing Policy and the canonical documentation for each subsystem
+   before judging its tests.
+2. Establish a clean baseline with the repository validation commands, focused
+   race tests where concurrency or shared state is relevant, and a coverage
+   report used only for investigation.
+3. Inventory tests and map each meaningful test or closely related group to the
+   contract, invariant, integration boundary, or regression it protects.
+4. Inspect production code only as needed to understand the protected behavior,
+   identify the stable boundary, and detect untested risk. Do not infer a
+   contract merely from current implementation detail.
+5. Evaluate marginal value across layers. Similar assertions are not redundant
+   when one owns a package contract and another distinctly proves production
+   wiring or end-to-end integration.
+6. Record evidence for every proposed change. Name the affected test or fixture,
+   the realistic defect it currently catches or fails to catch, and why the
+   recommendation improves confidence or reduces unnecessary friction.
+7. Check historical context when a test appears unusually specific. Preserve a
+   regression test when the underlying defect remains plausible and
+   consequential, even if its purpose is not obvious from the current code.
+
+Do not modify production code or tests during the review. If the review reveals
+incorrect production behavior, report it separately from test-suite
+harmonization rather than treating a changed test expectation as the fix.
+
+## Review Deliverable
+
+Produce an evidence-backed report organized by priority and subsystem. Each
+finding must classify the proposed disposition as one of:
+
+- **retain:** valuable protection at an appropriate boundary;
+- **consolidate:** overlapping protection that can be represented more simply;
+- **rewrite:** meaningful protection expressed through a brittle or misleading
+  boundary;
+- **delete:** no sufficient plausible defect or distinct protection justifies
+  the lifetime cost; or
+- **add:** a consequential risk lacks credible protection.
+
+For consolidate, rewrite, delete, and add findings, describe the protected risk,
+current evidence, recommended boundary, and expected effect on confidence and
+maintenance. Do not produce a raw list of every test when a package or related
+group shares one clear disposition.
+
+Distinguish required changes from optional cleanup. Absence of a finding is not
+evidence that a package needs more tests.
+
+## Implementation-Plan Deliverable
+
+After completing the review, write a decision-complete staged implementation
+plan based on the supported findings. The plan must:
+
+- order work in small, independently verifiable package or behavior groups;
+- state exactly which protections are retained when tests are consolidated,
+  rewritten, or deleted;
+- add tests only for identified meaningful risks;
+- avoid production behavior changes unless a separately identified production
+  defect is expressly brought into scope;
+- identify focused and repository-wide validation for each stage; and
+- include acceptance criteria demonstrating that the resulting suite remains
+  deterministic, offline, diagnostically useful, and sufficient under the
+  Testing Policy.
+
+The plan must not establish a coverage-percentage target, require mechanical
+conversion to table-driven tests, restore old tests wholesale, or equate fewer
+tests with success. It should prefer correct, idiomatic, and maintainable tests
+even when achieving the durable boundary requires more immediate work.
+
+## Completion Criteria
+
+The review is complete when:
+
+- the full suite has been considered at an appropriate behavioral grouping;
+- important test ownership and integration boundaries are mapped;
+- every recommended change is supported by a concrete risk and evidence;
+- high-risk behavior without credible protection is identified;
+- redundant or brittle protection is distinguished from valuable intentional
+  overlap;
+- production defects, if any, are reported separately; and
+- the resulting implementation plan can be executed without requiring the
+  implementing agent to make additional testing-policy decisions.