notarius/docs/roadmap/implementation.md

CLI and Configuration Contract Test Plan

This plan adds deliberate contract and risk-based test coverage for internal/cli and internal/core/config. It does not pursue a statement coverage target. Coverage reports are diagnostic input: they may reveal an important untested branch, but a percentage alone does not justify a test.

The audience is a relatively small LLM coding agent. Implement one stage per prompt, in order. Each stage must leave the repository compiling and passing all tests.

Scope and fixed decisions

Tests added by this plan protect:

• external contracts documented in docs/cli.md, docs/config.md, docs/operations.md, and maintained examples;
• internal contracts around configuration loading, precedence, validation, resolution, cloning, and redaction;
• high-risk CLI composition paths involving run selection, references, LLM profiles, output, cache, resume, and debug state; and
• production composition and the shortest maintained end-to-end workflows.

The following decisions are fixed:

• Do not set or enforce a repository-wide coverage percentage.
• Do not restore deleted test files wholesale. The former suite predates ADR-0006 and includes obsolete workspace and diagnostics behavior. Historical tests immediately before commit 26142f0 may be consulted for fixture and harness ideas only; every restored assertion must be checked against current code and canonical documentation.
• Prefer behavior tests at the narrowest useful boundary. Pure configuration rules belong in internal/core/config; CLI tests should prove parsing, composition, process-facing behavior, and cross-component wiring rather than repeat every configuration permutation.
• Test stable observable facts: exit code, output stream, selected collaborator, logical artifact, path, or resolved field. Avoid exact whole-error strings, entire JSON snapshots, timestamps, random identifiers, and internal call sequences unless those are the contract under test.
• Tests must be deterministic, offline, and secret-free. Inject catalogs, registries, clocks, run-ID generators, cache collaborators, and fake LLM clients. Use t.TempDir, t.Setenv, and t.Chdir where appropriate. Never use a real provider, credential, user cache directory, or shared output root.
• Do not weaken a test to match an apparent defect. If current behavior conflicts with a canonical contract, stop the stage and report the conflict. Production fixes are outside this test-coverage plan unless separately authorized.
• Preserve the focused ADR-0006 state tests already present. Consolidate helpers only when it materially reduces duplication and does not obscure the contract being tested.
• Checkpoint wire compatibility is already owned by internal/framework/checkpoint tests. Do not add a second legacy checkpoint fixture in CLI tests unless a distinct CLI compatibility boundary is found.

Execution rules for every stage

Before editing:

1. Read docs/development.md and follow its task-specific reading guide.
2. Read both files under docs/policy/.
3. Read this plan and all current source, tests, and canonical documents named by the stage.
4. Run git status --short; preserve all existing changes.
5. Run the stage's package tests once to establish a baseline.

While editing:

• Add tests only for the cases listed in the stage. Do not expand into adjacent subsystem redesign.
• Use table-driven tests for validation and syntax matrices when the setup and assertion are genuinely shared.
• Give failures enough context to identify the contract case.
• Assert both the positive outcome and important negative side effects, such as a root not being resolved or a file not being created.
• Do not run environment- or working-directory-mutating tests in parallel.

At the end of each stage:

1. Run the focused tests named by that stage.
2. Run go test ./....
3. Run go vet ./....
4. Run go build ./cmd/notarius.
5. Run git diff --check.
6. Review the diff for assertions tied to obsolete implementation details.

Do not mark a stage complete until all checks pass.

Stage 1: File schema, defaults, and decoding contracts

Status: Not started

Objective

Protect the version 3 file format and the translation from YAML into the runtime configuration model.

Read first

• docs/config.md, through Module Bindings
• internal/core/config/config.go
• internal/core/config/file_config.go
• internal/core/config/v3_test.go

Implement

Add focused tests in internal/core/config/file_config_contract_test.go. Reuse small helper constructors within that file; do not reproduce complete example files for unit cases.

Cover these contracts:

1. Default returns the documented concurrency, output, cache-family, debug, and empty-pipeline defaults. Mutating maps in one returned configuration must not affect a later Default result.
2. A minimal version: 3 file parses and applies over defaults.
3. Missing, version 2, and unsupported versions fail with actionable version context before ordinary field decoding.
4. Unknown top-level, pipeline, lane, and module-binding fields are rejected. Include removed workspace, diagnostics, and llm_profiles fields as representative compatibility failures, without testing every old field.
5. Shorthand and object module bindings preserve module, profile, retries, options, references, and the distinction between omitted validators and an explicit empty validator chain.
6. Pipeline, target-local, and lane compatibility reference maps apply with the documented precedence, including extract.references overriding the lane-level alias.
7. Chunk, extract, merge, and normalize stage-local validator bindings retain configured order and fields.
8. Scriptorium source, concurrency, output, chunk-plan cache, checkpoint cache, and debug sections apply without coupling their roots.
9. Keys that collide after trimming—pipeline IDs, lane IDs, and representative reference slots—are rejected. Do not enumerate the same normalization rule at every possible reference location.
10. LoadFileConfig reports both missing-file and malformed-YAML context.

Where v3_test.go already proves a case completely, either leave it there or move it without duplicating it.

Validation

go test ./internal/core/config -run 'Test(FileConfig|Default|Version3)'

Exit criteria

The maintained version 3 schema, defaults, binding forms, state sections, and normalization rules can be refactored without silently changing their contract.

Stage 2: Precedence and validation matrices

Status: Not started

Objective

Protect configuration precedence and high-risk invalid combinations after all sources have been applied.

Read first

• docs/config.md, especially Discovery, Environment Overrides, Concurrency, Module Bindings, and State Surfaces
• internal/core/config/env.go
• internal/core/config/validation.go
• current tests from Stage 1

Implement

Add internal/core/config/env_contract_test.go and internal/core/config/validation_contract_test.go.

Precedence tests must prove:

1. File values override built-in defaults.
2. Every current NOTARIUS_* operational variable overrides the corresponding file value: total LLM concurrency, extract workers, output directory, chunk-plan mode and directory, checkpoint directory, and debug directory.
3. Extract workers default to the final effective total when not explicitly configured, but an explicit file or environment worker value is retained.
4. Empty cache-directory fields in a file remain valid and select deferred per-user defaults; empty directory environment overrides are errors.
5. Invalid integers and chunk-cache modes report the responsible environment variable and do not panic.
6. Removed provider environment variables are ignored. Do not place credential values in assertions or failure output.

Validation tables must cover one representative case for each rule family:

• non-positive total concurrency; extract worker below one, above total, and at both valid boundaries; unknown and blank worker keys;
• mutually exclusive Scriptorium profile sources;
• blank required output/debug roots, NUL in every physical root family, and an invalid chunk-plan mode;
• empty and trim-duplicated pipeline/lane/reference identifiers;
• negative retries and whitespace-only explicit LLM profiles;
• references on unsupported input/output bindings;
• invalid validator bindings: empty module, retries, references, nested validators, and validator chains on unsupported stages; and
• deprecated non-empty lane-level validators rejected while stage-local omitted, empty, and non-empty overrides remain valid structurally.

Assert stable contextual fragments rather than complete error text.

Validation

go test ./internal/core/config -run 'Test(Env|Precedence|Validate)'

Exit criteria

Every documented precedence edge and every materially different validation rule family has a direct, readable regression test.

Stage 3: Effective configuration, cloning, and redaction

Status: Not started

Objective

Protect the internal contract that validated configuration resolves into an isolated, auditable pipeline without leaking sensitive options.

Read first

• docs/policy/architecture.md, especially Pipeline Composition And Ownership, Configuration And Provenance, and State, Output, And Safety
• internal/core/config/effective_config.go
• internal/core/config/redaction.go
• internal/core/config/redaction_test.go
• internal/framework/pipeline/profile.go

Implement

Add internal/core/config/effective_config_contract_test.go and extend redaction_test.go only for missing cases.

Use a minimal fake pipeline.ModuleCatalog with enough variants and capabilities to make each resolution outcome explicit. Cover:

1. Empty and unknown pipeline IDs fail; a map key with harmless surrounding whitespace resolves to its normalized ID.
2. Only selects exactly the requested lanes, rejects unknown lanes, and does not mutate the source configuration.
3. Default chunk, merge, normalize, and output modules are materialized through the catalog.
4. Unknown modules, missing capabilities, missing artifact variants, invalid module options, and invalid validator options retain pipeline/lane/stage context.
5. An LLM-profile override applies to LLM-capable pipeline module bindings before digest calculation, does not override validator-specific profiles, and changes the digest when effective behavior changes.
6. Omitted, explicitly empty, and configured validator chains resolve distinctly and preserve configured order.
7. Returned EffectiveConfig, resolved bindings, nested options, reference slices/content, and validator chains do not alias the input configuration or resolution inputs.
8. Both redacted summary payloads cover every binding and recursively redact sensitive keys in untyped and typed/aliased option containers while preserving safe neighbors and excluding materialized reference content.

Do not duplicate detailed reference-selector behavior owned by Stage 6.

Validation

go test ./internal/core/config -run 'Test(Resolve|Effective|Redacted)'

Exit criteria

Resolution defaults, selection, compatibility failures, profile overrides, digests, deep-copy isolation, and redaction are directly protected.

Stage 4: CLI command and configuration-loading contracts

Status: Not started

Objective

Protect the public command surface, exit-code classification, configuration discovery, validation command, and pipeline listing.

Read first

• docs/cli.md
• docs/config.md#discovery
• command dispatch, runConfigValidate, runPipelinesList, and configuration loading in internal/cli/run.go
• internal/cli/state_surfaces_test.go

Implement

Add internal/cli/command_contract_test.go. Use injected options and temporary files; no case may require production credentials or provider access.

Cover:

1. No arguments and each documented help spelling write usage to stdout and return 0.
2. Unknown commands/subcommands and malformed command syntax write to stderr and return 2.
3. Configuration discovery precedence is explicit --config, then non-empty NOTARIUS_CONFIG, then the compiled default path. Test the first two with temporary files. For the compiled path, inspect its host state first: assert selection when it is a regular file, or the documented not-found error when it is absent. Never create, replace, or remove the system path.
4. Missing and malformed configuration files return 1 with actionable path or parse context.
5. config validate succeeds for a valid file, resolves a requested pipeline against an injected catalog, rejects an unknown pipeline/lane, requires --pipeline with --only, and returns 2 for malformed --only syntax.
6. pipelines list sorts normalized IDs in text and JSON forms; JSON is decoded and compared structurally rather than as raw formatting.
7. Removed structural/state flags such as --diagnostics-dir are rejected with exit 2 instead of being silently ignored.
8. Representative malformed run flags and valid-runtime failures establish the documented 2 versus 1 boundary.

Validation

go test ./internal/cli -run 'Test(Command|Help|ConfigDiscovery|ConfigValidate|PipelinesList|ExitCode)'

Exit criteria

The documented command grammar, streams, discovery order, listing behavior, and exit-code classes are protected without invoking a real pipeline.

Stage 5: Run controls, profile selection, and process-facing results

Status: Not started

Objective

Protect the public run controls and the CLI-to-pipeline composition boundary.

Read first

• docs/cli.md#run
• internal/cli/run.go
• internal/cli/scriptorium_profiles.go
• internal/cli/state_hardening_test.go and its fake harness

Implement

Add internal/cli/run_contract_test.go. Reuse or minimally generalize the existing deterministic state-test harness. Keep helper implementations at the bottom of a test file or in one clearly named test_helpers_test.go file.

Cover:

1. Missing pipeline ID, missing --input, unknown flags, blank values, and multiple positional IDs return 2 without allocating output or debug state.
2. Unknown pipeline/lane and unreadable input are valid invocations that return 1; when debug is requested after allocation, its path is reported.
3. A successful run writes the logical output files, reports normalized and rejected counts, and passes the same deterministic run ID and start time to the manifest and debug bundle.
4. --only executes and reports only selected lanes.
5. --output-dir and --debug-dir override environment, file, and default roots; --debug-dir without --debug returns 2 and creates nothing.
6. --llm-profile reaches every effective LLM-capable pipeline module binding, leaves validator-specific profiles unchanged, selects the expected factory profile when exactly one is effective, and validates explicit Scriptorium profile IDs without making provider calls.
7. effectiveLLMProfileIDs ignores deterministic stages and returns stable, deduplicated ordering across chunk, lane stages, and LLM-backed validators.
8. --session-id rejects missing/blank values and passes a trimmed explicit value through prompt-facing requests. With no override, the parsed source document ID is used as the session identifier.
9. LLM factory, preparation, pipeline execution, output persistence, and debug persistence failures return 1, preserve the primary error, and do not print a success message.
10. Successful warnings remain exit 0, are counted on stderr, and appear in durable output and requested debug summary.

Do not repeat the output-collision and terminalization matrices already covered by run_id_test.go and state_hardening_test.go.

Validation

go test ./internal/cli -run 'TestRun'

Exit criteria

Every public non-reference run flag and every major CLI-to-runner handoff has a contract test, including representative failures.

Stage 6: CLI reference selector and override contracts

Status: Not started

Objective

Protect the high-risk selector grammar and target-resolution behavior without duplicating reference materialization internals.

Read first

• reference sections of docs/cli.md and docs/config.md
• reference parsing and resolution helpers in internal/cli/run.go
• reference resolution/materialization tests under internal/framework/pipeline

Implement

Add internal/cli/reference_contract_test.go. Build a compact catalog with chunk, extractor, merger, and normalizer specs declaring distinct, shared, optional, and required slots, including at least two artifact kinds where variant lookup matters.

Cover:

1. Parse and apply every documented selector form: flat, chunk.slot, merge.slot, lane.slot, and explicit lane extract/merge/normalize.
2. Flat and lane selectors succeed only with one eligible selected target. Ambiguity errors identify viable explicit selectors.
3. Selected lanes constrain reference discovery; selectors for unselected or unknown lanes fail before reference file materialization.
4. Malformed bindings—missing selector, path, separator, or excess selector segments—and malformed unbind selectors return 2.
5. Repeated overrides use the final explicit binding for that exact target; target-specific overrides do not alter same-named slots on other targets.
6. --without-reference removes optional configured bindings and fails when a required chunk, extract, merge, or normalize slot would remain unbound.
7. CLI paths resolve from the working directory while config paths resolve from the config file directory. Assert recorded provenance, not private helper call order.
8. Target lookup uses the selected artifact-kind variant and returns useful context when the requested merger/normalizer/extractor variant is absent.

Use small UTF-8 temporary reference files. Media-type and malformed-content details already proven by pipeline materialization tests need only one CLI smoke case.

Validation

go test ./internal/cli -run 'Test.*Reference'

Exit criteria

All public selector forms, ambiguity boundaries, lane selection, binding precedence, unbinding, and path-origin rules are protected.

Stage 7: Cache, resume, output, and debug integration risks

Status: Not started

Objective

Fill state-integration gaps without duplicating the focused ADR-0005 and ADR-0006 tests already present.

Read first

• docs/operations.md
• docs/internal/state.md
• ADR-0005 and ADR-0006
• internal/cli/run_id_test.go
• internal/cli/state_surfaces_test.go
• internal/cli/state_hardening_test.go
• tests under internal/framework/chunkplan and internal/framework/checkpoint

Implement

Add internal/cli/cache_contract_test.go only for uncovered CLI composition contracts. Extend an existing state test when it already owns the behavior.

Cover or confirm existing coverage for:

1. Chunk-plan mode precedence: CLI flag, environment, file, default. Invalid CLI syntax returns 2; invalid environment/file modes return 1.
2. An explicit chunk-plan root is passed through exactly. An empty configured root selects <UserCacheDir>/notarius/chunk-plans. bypass never calls UserCacheDir, constructs a store, or creates a root.
3. Root-resolution and store-construction failures preserve context and create no durable output.
4. auto reuses a plan across independent invocations with the same source digest even when pipeline, chunk module settings, references, or selected lanes differ. refresh replaces only after valid chunking; bypass does no cache I/O. Reuse details below the CLI remain owned by chunk-plan package tests.
5. --resume alone constructs checkpoint loader and recorder under the exact configured or per-user checkpoint root. Without it, neither root resolution nor checkpoint I/O occurs.
6. Chunk-plan reuse and checkpoint resume remain independent across the relevant mode matrix. Reuse a compact table rather than duplicating every existing state-surface assertion.
7. Existing tests continue to prove exclusive output allocation, atomic files, retained new partial output, shared run identity, debug opt-in, restrictive debug permissions, redacted summaries, and exactly-once terminal reports. Add a case only if one of these facts is not directly asserted.
8. Config-oriented commands do not resolve or create output, cache, or debug state.

Do not introduce automatic cleanup tests: Notarius intentionally retains output and requested debug bundles.

Validation

go test ./internal/cli -run 'Test.*(Cache|ChunkPlan|Checkpoint|Resume|State|Output|Debug|RunID)'
go test ./internal/framework/chunkplan ./internal/framework/checkpoint

Exit criteria

CLI state-root selection, opt-in I/O, cache mode precedence, cross-run reuse, and cache-family independence are covered at their composition boundary.

Stage 8: Production composition and maintained examples

Status: Not started

Objective

Protect the small number of production and end-to-end contracts most likely to break while unit tests continue to pass.

Read first

• README.md
• maintained files under examples/
• docs/config.md#implemented-production-modules
• internal/cli/catalog.go
• internal/modules/integration tests and helpers

Implement

Add internal/cli/production_contract_test.go and internal/cli/example_contract_test.go. Reuse production registries and prompt assets, but always inject a deterministic structured fake LLM client.

Production composition tests must prove:

1. The production catalog exposes every module, artifact codec, validator, default validator chain, and prompt asset named by current configuration and maintained examples.
2. Catalog/registry conversion preserves typed artifact codec variants and validator-chain registrations.
3. Production prompt assets allow the configured D&D scene chunker and spell extractor to prepare without reading provider credentials.
4. config validate --pipeline accepts the maintained production module graph and rejects representative unknown modules, unknown validators, invalid artifact variants, and deterministic validators with LLM profiles.
5. A small test configuration selecting dnd/scenes records that chunker and its expected warning/provenance fields during a fake-LLM run.

Example tests must prove:

1. Every maintained example configuration parses, validates, resolves its documented pipeline, and appears correctly in pipelines list.
2. The README/minimal example invocation runs with a fake LLM and produces the documented logical JSON bundle. Decode manifest, index, warning, rejection, and artifact files and assert contract fields; do not snapshot whitespace or incidental timestamps.
3. The documented --only spells invocation selects exactly the spells lane.
4. One malformed input fixture returns 1, produces no successful output bundle, and records a failure report only when debug is explicitly enabled.

If an example requires a provider profile, supply a test-only fake through cli.Options; never modify the example to embed credentials.

Validation

go test ./internal/cli -run 'Test(Production|Example|Maintained)'
go test ./internal/modules/integration

Exit criteria

Production registration, prompts, maintained configuration, and the shortest end-to-end user workflow are protected by offline tests.

Stage 9: Coverage audit and plan closure

Status: Not started

Objective

Confirm that contract and high-risk coverage is complete without converting coverage percentage into a goal.

Read first

• all tests added by Stages 1–8
• current canonical CLI, configuration, operations, integration, and internal state documents
• docs/policy/documentation.md

Implement

1. Build a temporary checklist mapping every current CLI command/flag and every current configuration section/source to at least one owning test. Do not commit the checklist if the test names themselves make ownership clear.
2. Run statement coverage for internal/cli and internal/core/config. Inspect uncovered functions and branches. Add a test only when the uncovered code is a documented contract, safety boundary, destructive/stateful path, complex selector/precedence branch, or failure path likely to regress.
3. Remove redundant cases that prove no additional contract or risk. Keep a small end-to-end layer and more numerous narrow unit tests.
4. Confirm tests do not access the network, depend on host credentials, use the real user cache directory, write outside temporary roots, or depend on test ordering.
5. Run race-enabled tests for the two changed packages. Fix test races; report production races separately unless authorized to change production code.
6. Update docs/development.md only if a stable testing procedure beyond its existing validation commands is now necessary. Do not document a coverage threshold.
7. Once all checks pass, delete this completed implementation plan. The tests and canonical current-behavior documents remain authoritative.

Validation

go test -coverprofile=/tmp/notarius-cli.cover ./internal/cli
go test -coverprofile=/tmp/notarius-config.cover ./internal/core/config
go tool cover -func=/tmp/notarius-cli.cover
go tool cover -func=/tmp/notarius-config.cover
go test -race ./internal/cli ./internal/core/config
go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check

The /tmp coverage profiles are inspection artifacts and must not be committed.

Exit criteria

Every current CLI and configuration contract has an identifiable owning test; high-risk state, safety, precedence, selector, and failure paths are covered; the suite remains deterministic and maintainable; and no numeric coverage target has been introduced.