Add testing policy documentation

docs/development.md

@@ -16,6 +16,7 @@ implemented component map.
 | Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. |
 | Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
 | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
+| Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
 | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
 | Production modules or validators | [Module Internals](internal/modules.md) | It documents implemented module contracts, capabilities, assets, and registration. |
 | LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |

docs/policy/documentation.md

@@ -61,6 +61,7 @@ secret values.
 | Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
 | Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
 | Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
+| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
 | CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
 | Configuration contract | `docs/config.md` | Discovery and precedence, file schema, fields, defaults, environment overrides, validation rules, and user-selectable module or validator keys. | Complete example files, CLI syntax, runtime state lifecycle, module implementation details. |
 | Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. |

docs/policy/testing.md

@@ -0,0 +1,296 @@
+# Testing Policy
+
+## Purpose
+
+Our tests exist to make **incorrect changes expensive and correct changes cheap**.
+
+We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
+
+## Every test has a cost
+
+Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
+
+A test must be:
+
+- written and reviewed;
+- understood by future maintainers and coding agents;
+- executed in local and CI workflows;
+- diagnosed when it fails;
+- updated when legitimate behavior changes;
+- maintained as fixtures, APIs, and dependencies evolve; and
+- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
+
+Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
+
+A test is warranted only when the confidence it provides justifies these costs.
+
+Apply this cost-benefit analysis at two levels:
+
+1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
+2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
+
+The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
+
+Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
+
+The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
+
+## Default testing style
+
+Use a **classical/Detroit-style** approach:
+
+- Test observable behavior, resulting state, contracts, and invariants.
+- Use real internal collaborators when they are fast and deterministic.
+- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
+- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
+- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
+
+Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
+
+## Test execution requirements
+
+Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
+
+Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
+
+## What deserves tests
+
+Prioritize tests for:
+
+1. Public and package-level contracts.
+2. Domain rules and important invariants.
+3. Boundary conditions and malformed input.
+4. Failure handling, cancellation, retries, recovery, and partial success.
+5. Serialization, schemas, compatibility, and round trips.
+6. Previously observed or plausible regressions.
+7. Representative integration and end-to-end workflows.
+
+A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
+
+For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
+
+Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
+
+## Choose the right test boundary
+
+Test through the narrowest stable boundary that expresses the behavior clearly.
+
+This is often the package API, but it may instead be:
+
+- a smaller pure function when dense domain logic is most clearly isolated there;
+- a package-level operation when several internal collaborators jointly produce the behavior; or
+- a larger integration boundary when correctness emerges from interaction with a real dependency.
+
+Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
+
+## Test behavior, not implementation
+
+A test should protect a decision, contract, or invariant—not memorialize the current implementation.
+
+Before adding or retaining a test, ask:
+
+> What realistic defect would this test catch?
+
+A test is suspect when its main purpose is to detect that someone:
+
+- changed an internal constant;
+- renamed or split a private helper;
+- reordered equivalent internal operations;
+- changed incidental formatting;
+- replaced one correct algorithm with another; or
+- refactored internal object structure without changing behavior.
+
+Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
+
+A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
+
+## Expected effects of different changes
+
+Use the following expectations when evaluating test failures and test maintenance:
+
+| Change | Expected effect on tests |
+|---|---|
+| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
+| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
+| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
+| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
+
+A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
+
+## Separate mechanism from policy
+
+Configurable thresholds and defaults must not be duplicated throughout the test suite.
+
+For example, do not encode an internal concurrency limit indirectly:
+
+```go
+// Production policy:
+const maxConcurrency = 4
+
+// Brittle test:
+err := startProcesses(5)
+require.Error(t, err)
+```
+
+Instead, test the mechanism relationally:
+
+```go
+const limit = 2
+runner := NewRunner(limit)
+
+require.NoError(t, runner.Start(limit))
+require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
+```
+
+The test should prove:
+
+- the configured limit is accepted; and
+- one beyond the configured limit is rejected.
+
+The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
+
+Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
+
+For concurrency limits, test both kinds of behavior when relevant:
+
+1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
+2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
+
+Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
+
+## Avoid semantic duplication across layers
+
+Each behavior should have a clear test owner.
+
+- Parser tests own parsing cases.
+- Validator tests own validation rules.
+- Domain tests own transformations and invariants.
+- Adapter tests own external integration behavior.
+- Orchestrator tests own coordination and failure propagation.
+- CLI tests own argument and configuration mapping.
+- End-to-end tests prove that representative assembled workflows work.
+
+Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
+
+Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
+
+## Use test doubles deliberately
+
+Choose the least elaborate test double that provides the required control or observation.
+
+As a default:
+
+1. Prefer real collaborators when they are fast and deterministic.
+2. Use small in-memory fakes when realistic stateful behavior is helpful.
+3. Use stubs when a dependency only needs to provide controlled responses.
+4. Use mocks when the interaction itself is contractual.
+
+Mocks are appropriate when the contract includes facts such as:
+
+- a notification is sent exactly once;
+- a transaction is committed only after successful writes;
+- cancellation reaches a subprocess;
+- an expensive API is called no more than once; or
+- a security audit event is emitted.
+
+Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
+
+## Go-specific guidance
+
+Use:
+
+- table-driven tests for meaningful behavioral categories and boundaries;
+- `t.TempDir()` for real filesystem behavior;
+- `httptest.Server` for realistic HTTP interactions;
+- fuzz tests for parsers, normalization, path handling, and broad input spaces;
+- golden files only when the complete output is intentionally stable;
+- integration tests where correctness depends on component interaction; and
+- a small number of representative end-to-end tests.
+
+Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
+
+At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
+
+Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
+
+Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
+
+## Coverage
+
+Coverage is a diagnostic, not a target.
+
+Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
+
+Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
+
+Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
+
+## Regression tests
+
+A bug fix should normally include a regression test that fails before the fix and passes afterward.
+
+Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
+
+Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
+
+## Deleting or rewriting tests
+
+Tests are maintained code, not permanent historical artifacts.
+
+Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
+
+Strong candidates include tests that:
+
+- require updates after harmless internal changes;
+- directly assert private constants without protecting a real contract;
+- duplicate the same policy across several layers;
+- verify mock choreography rather than outcomes;
+- snapshot large amounts of incidental output;
+- test trivial private helpers already exercised through stable package behavior;
+- protect risks already covered more effectively elsewhere;
+- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
+- no longer correspond to a plausible failure mode.
+
+Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
+
+Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
+
+## Reviewing a proposed test
+
+Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
+
+1. What realistic defect would it catch?
+2. How likely is that defect?
+3. How consequential would it be?
+4. Is the behavior already protected elsewhere?
+5. At which layer should this behavior be owned?
+6. Does the test assert a durable contract or an incidental implementation detail?
+7. Could the implementation be refactored without changing the behavior and without editing this test?
+8. What should cause this test to fail?
+9. What legitimate changes should not cause this test to fail?
+10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
+11. Is there a smaller or more direct test that protects the same risk?
+
+Do not add the test when its expected lifetime cost exceeds its expected protective value.
+
+When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
+
+## Definition of sufficient
+
+A test suite is sufficient when:
+
+- important contracts and invariants are protected;
+- meaningful boundaries and failure modes are exercised;
+- realistic and consequential regressions are credibly protected against silent recurrence;
+- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
+- important external boundaries have realistic integration coverage;
+- representative complete workflows are tested;
+- failures provide useful signal rather than redundant noise;
+- legitimate internal changes usually do not require test edits; and
+- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
+
+Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
+
+The governing rule is:
+
+> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.

docs/roadmap/future.md

@@ -41,12 +41,6 @@ future work only.
 - Optional generated example output fixtures with a regeneration procedure.
 - Additional diagnostics or reporting views if operator workflows need them.
 
-## Candidate Developer Work
-
-- Reassess broad CLI and configuration regression coverage, including whether
-  to retain a legacy-checkpoint compatibility fixture and adopt a coverage
-  policy.
-
 ## Candidate Workspace Work
 
 - Default-idempotent run behavior with an explicit force override.

docs/roadmap/tests.md

@@ -0,0 +1,187 @@
+# Test Suite Policy Review
+
+## Context
+
+Notarius now has a canonical [Testing Policy](../policy/testing.md). The policy
+defines a risk-based approach that favors durable behavioral coverage while
+removing redundant, brittle, misleading, or obsolete tests whose lifetime cost
+exceeds their protective value.
+
+Much of the existing test suite predates that policy. Recent CLI and
+configuration work has already applied several of its principles, but the suite
+has not been reviewed consistently as a whole. This roadmap calls for that
+review before additional domain capabilities materially expand the number of
+tests and fixtures.
+
+The review is not based on a presumption that old tests are bad, that direct
+tests of package-private behavior must be removed, or that the suite should be
+made smaller at any cost. Existing tests should be retained when they provide
+durable and nonredundant protection for a meaningful risk. The goal is a leaner
+and clearer allocation of test ownership together with credible protection for
+important behavior.
+
+## Objective
+
+Review the complete Notarius test suite against the Testing Policy, identify
+both low-value coverage and meaningful protection gaps, and then develop a
+decision-complete staged implementation plan for the justified changes.
+
+The review must answer:
+
+- Which important contracts, invariants, failure modes, and integration
+  boundaries does the current suite protect?
+- Where do multiple tests protect the same behavior without providing distinct
+  failure isolation or integration confidence?
+- Which tests are coupled to incidental implementation details, duplicated
+  policy literals, closed-world inventories, unstable formatting, or mock
+  choreography?
+- Which important risks can still fail silently despite the existing suite?
+- Which tests should be retained, consolidated, rewritten, or deleted, and
+  which new tests are warranted?
+
+## Review Scope
+
+Review all committed Go tests and their supporting fixtures. Organize the work
+by behavioral layer rather than treating test count or package coverage as the
+unit of quality.
+
+### CLI and configuration
+
+Review command parsing, configuration decoding and precedence, validation,
+effective configuration, run controls, reference selection, output/cache/debug
+state, production composition, and maintained examples.
+
+Pay particular attention to:
+
+- duplicated assertions across parser, resolver, CLI, and assembled-run tests;
+- tests that restate complete defaults or registry contents rather than
+  protecting operator-visible behavior;
+- exact error or output assertions broader than the documented CLI contract;
+- fixture mutation that can silently stop establishing a test precondition;
+  and
+- whether representative CLI workflows provide sufficient assembled coverage.
+
+### Framework and durable state
+
+Review pipeline preparation and execution, artifact contracts, validation and
+retry behavior, LLM scheduling and transport seams, chunk-plan storage,
+checkpoint compatibility, debug bundles, and file persistence.
+
+Presume durable protection is important for data integrity, serialization,
+compatibility, cancellation, concurrency, cache correctness, resume behavior,
+atomic or failure-safe persistence, and recovery. Look for opportunities to
+replace many narrow structural tests with a smaller invariant, round-trip, or
+behavior-level test only when protection is not weakened.
+
+### Modules and domain behavior
+
+Review generic, seriatim, and D&D module tests, including codecs, chunking,
+extraction, merging, normalization, validators, registration, prompt assets,
+and module integration.
+
+Confirm that domain rules and artifact schemas have clear test ownership.
+Identify tests that merely reproduce schemas, prompt asset inventories, or
+implementation structure, while preserving tests that protect compatibility,
+source provenance, normalization, validation, or other consequential domain
+invariants.
+
+### Cross-cutting suite quality
+
+Across all packages, evaluate:
+
+- deterministic, offline, credential-free execution;
+- isolation from mutable machine and process-global state;
+- appropriate use of real collaborators, fakes, stubs, and mocks;
+- stable behavioral assertions and useful failure diagnostics;
+- golden files and large snapshots;
+- helper and fixture complexity;
+- test runtime, race safety, repetition stability, and parallel-execution
+  assumptions; and
+- semantic ownership and duplication across package, integration, and
+  end-to-end layers.
+
+Use coverage only as a diagnostic to locate unexpectedly untested critical
+branches. Do not recommend tests solely to increase a percentage or make
+coverage uniform across packages.
+
+## Review Method
+
+1. Read the Testing Policy and the canonical documentation for each subsystem
+   before judging its tests.
+2. Establish a clean baseline with the repository validation commands, focused
+   race tests where concurrency or shared state is relevant, and a coverage
+   report used only for investigation.
+3. Inventory tests and map each meaningful test or closely related group to the
+   contract, invariant, integration boundary, or regression it protects.
+4. Inspect production code only as needed to understand the protected behavior,
+   identify the stable boundary, and detect untested risk. Do not infer a
+   contract merely from current implementation detail.
+5. Evaluate marginal value across layers. Similar assertions are not redundant
+   when one owns a package contract and another distinctly proves production
+   wiring or end-to-end integration.
+6. Record evidence for every proposed change. Name the affected test or fixture,
+   the realistic defect it currently catches or fails to catch, and why the
+   recommendation improves confidence or reduces unnecessary friction.
+7. Check historical context when a test appears unusually specific. Preserve a
+   regression test when the underlying defect remains plausible and
+   consequential, even if its purpose is not obvious from the current code.
+
+Do not modify production code or tests during the review. If the review reveals
+incorrect production behavior, report it separately from test-suite
+harmonization rather than treating a changed test expectation as the fix.
+
+## Review Deliverable
+
+Produce an evidence-backed report organized by priority and subsystem. Each
+finding must classify the proposed disposition as one of:
+
+- **retain:** valuable protection at an appropriate boundary;
+- **consolidate:** overlapping protection that can be represented more simply;
+- **rewrite:** meaningful protection expressed through a brittle or misleading
+  boundary;
+- **delete:** no sufficient plausible defect or distinct protection justifies
+  the lifetime cost; or
+- **add:** a consequential risk lacks credible protection.
+
+For consolidate, rewrite, delete, and add findings, describe the protected risk,
+current evidence, recommended boundary, and expected effect on confidence and
+maintenance. Do not produce a raw list of every test when a package or related
+group shares one clear disposition.
+
+Distinguish required changes from optional cleanup. Absence of a finding is not
+evidence that a package needs more tests.
+
+## Implementation-Plan Deliverable
+
+After completing the review, write a decision-complete staged implementation
+plan based on the supported findings. The plan must:
+
+- order work in small, independently verifiable package or behavior groups;
+- state exactly which protections are retained when tests are consolidated,
+  rewritten, or deleted;
+- add tests only for identified meaningful risks;
+- avoid production behavior changes unless a separately identified production
+  defect is expressly brought into scope;
+- identify focused and repository-wide validation for each stage; and
+- include acceptance criteria demonstrating that the resulting suite remains
+  deterministic, offline, diagnostically useful, and sufficient under the
+  Testing Policy.
+
+The plan must not establish a coverage-percentage target, require mechanical
+conversion to table-driven tests, restore old tests wholesale, or equate fewer
+tests with success. It should prefer correct, idiomatic, and maintainable tests
+even when achieving the durable boundary requires more immediate work.
+
+## Completion Criteria
+
+The review is complete when:
+
+- the full suite has been considered at an appropriate behavioral grouping;
+- important test ownership and integration boundaries are mapped;
+- every recommended change is supported by a concrete risk and evidence;
+- high-risk behavior without credible protection is identified;
+- redundant or brittle protection is distinguished from valuable intentional
+  overlap;
+- production defects, if any, are reported separately; and
+- the resulting implementation plan can be executed without requiring the
+  implementing agent to make additional testing-policy decisions.