Document configuration and CLI internals

docs/development.md

@@ -17,8 +17,8 @@ implemented component map.
 | Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
 | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
 | Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
-| CLI composition or command behavior | [CLI Reference](cli.md) | It owns public command syntax; focused CLI internals are documented separately as they are introduced. |
+| CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. |
-| Configuration loading, resolution, or user-visible configuration behavior | [Configuration](config.md) | It owns the configuration contract; focused configuration internals are documented separately as they are introduced. |
+| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
 | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
 | Production modules or validators | [Module Internals](internal/modules.md) and [D&D integration contracts](integrations/) | The generic module guide owns extension mechanics; D&D artifact contracts own durable output shapes. |
 | LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |

docs/internal/cli.md

@@ -0,0 +1,133 @@
+# CLI Internals
+
+This document describes **internal/cli**, Notarius's production composition
+root. The [CLI reference](../cli.md) owns command syntax and exit statuses;
+[Configuration](../config.md) owns configuration values; and
+[Operations](../operations.md) owns filesystem layout, recovery, and operator
+procedures.
+
+## Inputs, Outputs, And Boundaries
+
+The CLI accepts process arguments, standard streams, and injectable options
+used by tests and embedding code. It writes command results to the supplied
+streams and returns a process exit status. For a run, it also creates the
+production catalog and runtime collaborators, hands a prepared pipeline and
+source bytes to the framework, and places the logical files returned by the
+runner.
+
+It is the only boundary allowed to compose concrete registries, LLM clients,
+cache/checkpoint collaborators, debug recorders, and physical output paths.
+Pipeline modules receive interfaces and request data rather than CLI streams or
+filesystem roots. The [Architecture](../policy/architecture.md) defines this
+composition-root boundary; [Pipeline Internals](pipeline.md) owns resolution,
+preparation, and runner mechanics after their inputs are supplied.
+
+## Dispatch And Configuration Handoff
+
+The root dispatcher handles help, configuration validation, pipeline listing,
+and a pipeline run. It normalizes injectable options before dispatch so that a
+missing production dependency fails as a command error rather than reaching
+execution.
+
+Commands that need configuration use one shared loader. The CLI discovers the
+file, parses it through **internal/core/config**, starts from defaults, applies
+the file and supported environment overrides, and then validates it for the
+command. The configured discovery and precedence contract is in
+[Configuration](../config.md), while the loading and resolution mechanics are
+in [Configuration Internals](configuration.md).
+
+Configuration validation without a selected pipeline checks structural
+configuration only. Validation with a selected pipeline also builds the
+effective catalog, resolves the pipeline, and verifies explicitly selected
+Scriptorium profiles. Pipeline listing validates configuration before returning
+normalized, sorted identifiers.
+
+## Production Composition
+
+The production composition helper allocates every framework registry and the
+prompt-asset registry, then registers the generic, Seriatim, and D&D module
+families in that order. The resulting registries provide both the module
+catalog used for resolution and the concrete constructors used for preparation.
+Tests may provide a catalog or registries instead; production code must not
+silently merge an injected partial catalog with production registrations.
+
+The production LLM factory builds the Scriptorium-backed client from resolved
+configuration, creates one scheduler from the effective global LLM limit, and
+wraps the client before it reaches modules. Registration and LLM construction
+errors are returned before a pipeline is prepared. Concrete module keys and
+validator chains are public configuration choices and remain documented in
+[Configuration](../config.md).
+
+## Run Orchestration
+
+After parsing and validating a run invocation, the CLI performs this ordered
+handoff:
+
+1. load and validate configuration, then apply command-level operational
+   overrides;
+2. create and validate a safe run identity, then allocate a debug bundle only
+   when requested;
+3. build the effective catalog, resolve requested reference changes, resolve
+   the effective pipeline, and verify explicit Scriptorium profiles;
+4. materialize external or generated references and record redacted invocation
+   and resolution provenance when debug capture is enabled;
+5. construct registries, the scheduled LLM client, prepared modules, and the
+   requested cache/checkpoint collaborators;
+6. read the source input and invoke the framework runner; and
+7. write the runner's logical output files only after a successful run, then
+   complete the command report and user-facing result.
+
+Preparation happens before source parsing, so module construction and
+dependency failures cannot begin stage execution. The CLI also preserves the
+framework's result and warning information when it writes summaries and the
+final command result. Detailed state lifecycle, resume handling, and physical
+path confinement are maintained in [Run State Internals](state.md) and
+[Operations](../operations.md).
+
+## Failure Mapping And Terminal Reporting
+
+Argument, flag, and invocation-combination failures are reported to standard
+error before runtime composition and use the syntax error class. Once an
+invocation is syntactically valid, configuration loading and validation,
+resolution, registration, profile checks, reference materialization, module
+construction, input reads, runner failures, output publication, and requested
+debug handling use the runtime failure class. The public status numbers and
+stream contract are defined in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
+
+When debug capture has been allocated, one command-state value records the
+known run result. Guarded terminalization writes a success report once, or
+attempts a failure report and error record once. A persistence failure is
+reported in addition to the original failure and never replaces it. If a debug
+path exists, failure output includes that path so the retained diagnostic data
+is discoverable.
+
+## Invariants To Preserve
+
+- Only the CLI composes production implementations and physical runtime roots.
+- Configuration and resolved composition failures occur before module
+  preparation or source parsing.
+- A runner's logical files are published only after a successful run.
+- Production registries and a caller-supplied catalog or registries are
+  alternative composition sources, not an implicit mixture.
+- A requested debug bundle has one terminal report attempt; its persistence
+  errors supplement rather than obscure the primary command error.
+- User-facing flags, paths, exit codes, and configuration fields are defined
+  by their public documentation, not duplicated here.
+
+## Focused Tests
+
+- **internal/cli/command_contract_test.go** covers dispatch, help, syntax and
+  runtime error classes, discovery, validation, and listing.
+- **internal/cli/run_contract_test.go** covers the run handoff, publication,
+  debug reporting, and command-owned state collaborators.
+- **internal/cli/production_contract_test.go** covers registrar composition,
+  production catalog contents, assets, and representative configuration
+  validation.
+- **internal/cli/reference_contract_test.go** covers CLI reference overrides,
+  origin separation, and materialization boundaries.
+- **internal/cli/state_hardening_test.go** covers safe run identity, state
+  roots, and failure ordering.
+
+Run **go test ./internal/cli** after changing command composition or command
+behavior. Pair it with **go test ./internal/core/config** when the configuration
+handoff changes.

docs/internal/configuration.md

@@ -0,0 +1,129 @@
+# Configuration Internals
+
+This document describes the maintainer-facing configuration boundary in
+**internal/core/config**. The [Configuration](../config.md) reference owns the
+file format, fields, defaults, precedence contract, and selectable keys. The
+[CLI reference](../cli.md) owns command syntax; this document does not redefine
+either interface.
+
+## Boundary
+
+The configuration package turns a selected YAML file and supported environment
+values into a validated, independently owned configuration. It then resolves a
+requested pipeline against a module catalog before the framework prepares or
+runs anything.
+
+| Boundary | Inputs | Outputs | Does not own |
+| --- | --- | --- | --- |
+| Loading | Selected file path and environment lookup | Parsed file model and a populated **Config** | Choosing the file path or reporting a command result. |
+| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
+| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
+| Summary | **Config** or **EffectiveConfig** | Detached redacted payload suitable for debug summaries | Redacting arbitrary process state or provider traffic. |
+
+The CLI discovers a configuration file, invokes this package, and supplies the
+result to the framework. Configuration never reads an input file, constructs a
+module, or creates output, cache, or debug paths. Those responsibilities remain
+at their respective [CLI](cli.md), [pipeline](pipeline.md), and
+[run-state](state.md) boundaries.
+
+## Loading And Validation
+
+The CLI loads configuration in this order:
+
+1. parse the selected YAML file strictly into the file model;
+2. start from **Default**;
+3. apply the file model; and
+4. apply the supported environment overrides.
+
+This establishes the public precedence order without giving environment input a
+second file schema. Loading and application reject malformed YAML, unsupported
+file versions, unknown fields, invalid values, and identifiers that are empty
+or collide after whitespace normalization. The file application also makes the
+effective extraction-worker default follow the effective LLM limit.
+
+**Config.Validate** checks configuration-only invariants before resolution. It
+rejects incompatible profile sources, invalid state-surface values, unsupported
+concurrency settings, malformed bindings and references, invalid retries, and
+invalid pipeline, step, or lane structure. Its errors retain the closest known
+pipeline, lane, and binding context. It deliberately does not require modules
+to be registered: that requires a catalog and belongs to resolution.
+
+The exact user-selectable values and validation rules are defined in
+[Configuration](../config.md). Keep additions to the file model, an
+environment override, its validation, and that reference in the same change.
+
+## Effective Resolution
+
+**Config.Resolve** first recomputes derived concurrency defaults and validates
+the configuration. It normalizes the requested pipeline ID, copies the selected
+profile, applies a non-empty command-level LLM profile override to the
+LLM-capable stage bindings, and calls the framework resolver with the requested
+lane selection and reference changes.
+
+The command-level override does not replace an explicitly selected validator
+profile. Validator bindings remain part of the resolved validator chain and
+are resolved under their own declared configuration.
+
+The framework resolver supplies defaults, selects lanes, resolves validator
+chains, checks registered module and artifact compatibility, validates module
+options, and returns the fixed ordered pipeline shape. The resulting
+**EffectiveConfig** retains the selected ID, requested selection and reference
+changes, a clone of the input configuration, and the resolved pipeline.
+Callers may therefore retain or modify their input slices and maps without
+changing the resolved result, and later consumers cannot mutate the original
+configuration through the effective value.
+
+Resolution failures stop before module construction and source parsing. They
+include an error path for an unconfigured pipeline, missing module, missing
+capability, incompatible artifact variant, invalid option, invalid reference,
+or invalid lane selection. CLI code maps these valid-invocation failures to the
+runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
+
+## Resolved Identity And Redaction
+
+The framework assigns the resolved pipeline a deterministic SHA-256 digest
+after defaults, lane selection, module bindings, reference bindings, validator
+chains, and artifact schema identity have been resolved. The digest excludes
+its own stored value. It identifies resolved composition rather than raw YAML
+bytes, a debug payload, or all runtime state. The CLI records it as invocation
+provenance before execution; cache and checkpoint identity have additional
+owners in [Run State Internals](state.md).
+
+Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
+**RedactedResolvedPipelinePayload**, never a direct configuration marshal.
+Those methods copy every binding and nested option container, replace values
+whose key is credential-shaped with **[REDACTED]**, and omit materialized
+reference content while retaining safe binding and reference provenance. The
+payload must not alias the source configuration or resolved pipeline. This
+redaction is deliberately narrow: it protects configuration summaries and does
+not authorize recording arbitrary environment values or provider requests.
+
+## Invariants To Preserve
+
+- Defaults, YAML values, and environment values are applied in one direction;
+  later sources may override only their supported operational settings.
+- A configuration is structurally valid before it is resolved, and a resolved
+  pipeline is compatible with the supplied catalog before preparation begins.
+- Whitespace-normalized identifiers are unique wherever they identify a
+  pipeline, step, lane, worker, or reference slot.
+- Resolution and summary generation return detached data. Redaction must cover
+  every configured and resolved binding, including nested validator bindings.
+- The resolved digest changes when resolved composition changes and never
+  includes itself.
+
+## Focused Tests
+
+- **internal/core/config/file_config_contract_test.go** covers strict file
+  parsing, normalization, file application, and structural rejection.
+- **internal/core/config/env_contract_test.go** covers supported operational
+  overrides and their precedence.
+- **internal/core/config/validation_contract_test.go** covers configuration
+  invariants and contextual failures.
+- **internal/core/config/effective_config_contract_test.go** covers defaults,
+  selections, overrides, resolution context, digest changes, and ownership.
+- **internal/core/config/redaction_test.go** covers recursive credential
+  redaction, reference-content exclusion, and non-aliasing payloads.
+
+Run **go test ./internal/core/config** after changing this boundary. Changes to
+the handoff or resolved-composition semantics also need the focused framework
+pipeline tests.

docs/internal/overview.md

@@ -181,6 +181,10 @@ interfaces and request data, not physical state roots.
 
 ## Focused Documentation
 
+- [Configuration Internals](configuration.md): loading, validation, effective
+  resolution, redaction, and resolved-composition identity.
+- [CLI Internals](cli.md): command dispatch, production composition, run
+  orchestration, terminal reporting, and focused CLI tests.
 - [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
   checkpoint/debug hooks, and result assembly.
 - [Module Internals](modules.md): production modules, validators, assets,