188 lines
8.6 KiB
Markdown
188 lines
8.6 KiB
Markdown
# Test Suite Policy Review
|
|
|
|
## Context
|
|
|
|
Notarius now has a canonical [Testing Policy](../policy/testing.md). The policy
|
|
defines a risk-based approach that favors durable behavioral coverage while
|
|
removing redundant, brittle, misleading, or obsolete tests whose lifetime cost
|
|
exceeds their protective value.
|
|
|
|
Much of the existing test suite predates that policy. Recent CLI and
|
|
configuration work has already applied several of its principles, but the suite
|
|
has not been reviewed consistently as a whole. This roadmap calls for that
|
|
review before additional domain capabilities materially expand the number of
|
|
tests and fixtures.
|
|
|
|
The review is not based on a presumption that old tests are bad, that direct
|
|
tests of package-private behavior must be removed, or that the suite should be
|
|
made smaller at any cost. Existing tests should be retained when they provide
|
|
durable and nonredundant protection for a meaningful risk. The goal is a leaner
|
|
and clearer allocation of test ownership together with credible protection for
|
|
important behavior.
|
|
|
|
## Objective
|
|
|
|
Review the complete Notarius test suite against the Testing Policy, identify
|
|
both low-value coverage and meaningful protection gaps, and then develop a
|
|
decision-complete staged implementation plan for the justified changes.
|
|
|
|
The review must answer:
|
|
|
|
- Which important contracts, invariants, failure modes, and integration
|
|
boundaries does the current suite protect?
|
|
- Where do multiple tests protect the same behavior without providing distinct
|
|
failure isolation or integration confidence?
|
|
- Which tests are coupled to incidental implementation details, duplicated
|
|
policy literals, closed-world inventories, unstable formatting, or mock
|
|
choreography?
|
|
- Which important risks can still fail silently despite the existing suite?
|
|
- Which tests should be retained, consolidated, rewritten, or deleted, and
|
|
which new tests are warranted?
|
|
|
|
## Review Scope
|
|
|
|
Review all committed Go tests and their supporting fixtures. Organize the work
|
|
by behavioral layer rather than treating test count or package coverage as the
|
|
unit of quality.
|
|
|
|
### CLI and configuration
|
|
|
|
Review command parsing, configuration decoding and precedence, validation,
|
|
effective configuration, run controls, reference selection, output/cache/debug
|
|
state, production composition, and maintained examples.
|
|
|
|
Pay particular attention to:
|
|
|
|
- duplicated assertions across parser, resolver, CLI, and assembled-run tests;
|
|
- tests that restate complete defaults or registry contents rather than
|
|
protecting operator-visible behavior;
|
|
- exact error or output assertions broader than the documented CLI contract;
|
|
- fixture mutation that can silently stop establishing a test precondition;
|
|
and
|
|
- whether representative CLI workflows provide sufficient assembled coverage.
|
|
|
|
### Framework and durable state
|
|
|
|
Review pipeline preparation and execution, artifact contracts, validation and
|
|
retry behavior, LLM scheduling and transport seams, chunk-plan storage,
|
|
checkpoint compatibility, debug bundles, and file persistence.
|
|
|
|
Presume durable protection is important for data integrity, serialization,
|
|
compatibility, cancellation, concurrency, cache correctness, resume behavior,
|
|
atomic or failure-safe persistence, and recovery. Look for opportunities to
|
|
replace many narrow structural tests with a smaller invariant, round-trip, or
|
|
behavior-level test only when protection is not weakened.
|
|
|
|
### Modules and domain behavior
|
|
|
|
Review generic, seriatim, and D&D module tests, including codecs, chunking,
|
|
extraction, merging, normalization, validators, registration, prompt assets,
|
|
and module integration.
|
|
|
|
Confirm that domain rules and artifact schemas have clear test ownership.
|
|
Identify tests that merely reproduce schemas, prompt asset inventories, or
|
|
implementation structure, while preserving tests that protect compatibility,
|
|
source provenance, normalization, validation, or other consequential domain
|
|
invariants.
|
|
|
|
### Cross-cutting suite quality
|
|
|
|
Across all packages, evaluate:
|
|
|
|
- deterministic, offline, credential-free execution;
|
|
- isolation from mutable machine and process-global state;
|
|
- appropriate use of real collaborators, fakes, stubs, and mocks;
|
|
- stable behavioral assertions and useful failure diagnostics;
|
|
- golden files and large snapshots;
|
|
- helper and fixture complexity;
|
|
- test runtime, race safety, repetition stability, and parallel-execution
|
|
assumptions; and
|
|
- semantic ownership and duplication across package, integration, and
|
|
end-to-end layers.
|
|
|
|
Use coverage only as a diagnostic to locate unexpectedly untested critical
|
|
branches. Do not recommend tests solely to increase a percentage or make
|
|
coverage uniform across packages.
|
|
|
|
## Review Method
|
|
|
|
1. Read the Testing Policy and the canonical documentation for each subsystem
|
|
before judging its tests.
|
|
2. Establish a clean baseline with the repository validation commands, focused
|
|
race tests where concurrency or shared state is relevant, and a coverage
|
|
report used only for investigation.
|
|
3. Inventory tests and map each meaningful test or closely related group to the
|
|
contract, invariant, integration boundary, or regression it protects.
|
|
4. Inspect production code only as needed to understand the protected behavior,
|
|
identify the stable boundary, and detect untested risk. Do not infer a
|
|
contract merely from current implementation detail.
|
|
5. Evaluate marginal value across layers. Similar assertions are not redundant
|
|
when one owns a package contract and another distinctly proves production
|
|
wiring or end-to-end integration.
|
|
6. Record evidence for every proposed change. Name the affected test or fixture,
|
|
the realistic defect it currently catches or fails to catch, and why the
|
|
recommendation improves confidence or reduces unnecessary friction.
|
|
7. Check historical context when a test appears unusually specific. Preserve a
|
|
regression test when the underlying defect remains plausible and
|
|
consequential, even if its purpose is not obvious from the current code.
|
|
|
|
Do not modify production code or tests during the review. If the review reveals
|
|
incorrect production behavior, report it separately from test-suite
|
|
harmonization rather than treating a changed test expectation as the fix.
|
|
|
|
## Review Deliverable
|
|
|
|
Produce an evidence-backed report organized by priority and subsystem. Each
|
|
finding must classify the proposed disposition as one of:
|
|
|
|
- **retain:** valuable protection at an appropriate boundary;
|
|
- **consolidate:** overlapping protection that can be represented more simply;
|
|
- **rewrite:** meaningful protection expressed through a brittle or misleading
|
|
boundary;
|
|
- **delete:** no sufficient plausible defect or distinct protection justifies
|
|
the lifetime cost; or
|
|
- **add:** a consequential risk lacks credible protection.
|
|
|
|
For consolidate, rewrite, delete, and add findings, describe the protected risk,
|
|
current evidence, recommended boundary, and expected effect on confidence and
|
|
maintenance. Do not produce a raw list of every test when a package or related
|
|
group shares one clear disposition.
|
|
|
|
Distinguish required changes from optional cleanup. Absence of a finding is not
|
|
evidence that a package needs more tests.
|
|
|
|
## Implementation-Plan Deliverable
|
|
|
|
After completing the review, write a decision-complete staged implementation
|
|
plan based on the supported findings. The plan must:
|
|
|
|
- order work in small, independently verifiable package or behavior groups;
|
|
- state exactly which protections are retained when tests are consolidated,
|
|
rewritten, or deleted;
|
|
- add tests only for identified meaningful risks;
|
|
- avoid production behavior changes unless a separately identified production
|
|
defect is expressly brought into scope;
|
|
- identify focused and repository-wide validation for each stage; and
|
|
- include acceptance criteria demonstrating that the resulting suite remains
|
|
deterministic, offline, diagnostically useful, and sufficient under the
|
|
Testing Policy.
|
|
|
|
The plan must not establish a coverage-percentage target, require mechanical
|
|
conversion to table-driven tests, restore old tests wholesale, or equate fewer
|
|
tests with success. It should prefer correct, idiomatic, and maintainable tests
|
|
even when achieving the durable boundary requires more immediate work.
|
|
|
|
## Completion Criteria
|
|
|
|
The review is complete when:
|
|
|
|
- the full suite has been considered at an appropriate behavioral grouping;
|
|
- important test ownership and integration boundaries are mapped;
|
|
- every recommended change is supported by a concrete risk and evidence;
|
|
- high-risk behavior without credible protection is identified;
|
|
- redundant or brittle protection is distinguished from valuable intentional
|
|
overlap;
|
|
- production defects, if any, are reported separately; and
|
|
- the resulting implementation plan can be executed without requiring the
|
|
implementing agent to make additional testing-policy decisions.
|