diff --git a/docs/adr/0018-adopt-configkit.md b/docs/adr/0018-adopt-configkit.md index 011cf197..6538fde5 100644 --- a/docs/adr/0018-adopt-configkit.md +++ b/docs/adr/0018-adopt-configkit.md @@ -5,9 +5,9 @@ ## Context -Notarius's configuration boundary currently owns strict YAML parsing, input +Notarius's former configuration boundary owned strict YAML parsing, input presence bookkeeping, defaults, normalization, environment overlays, and -configuration inspection. Those general-purpose responsibilities have grown +configuration inspection. Those general-purpose responsibilities had grown alongside application-specific pipeline and module semantics. ConfigKit v0.3.0 provides strict typed YAML loading, imports, configuration @@ -47,8 +47,7 @@ boundary. ## Consequences -The implementation migrates configuration behavior in bounded increments, but -the released application will have one ConfigKit-backed loader and one version -5 contract. Operator documentation and maintained examples must move with the -cutover. ConfigKit metadata is retained only at the configuration publication -boundary; its types do not become module or runtime pipeline interfaces. +The released application has one ConfigKit-backed loader and one version 5 +contract. Operator documentation and maintained examples move with the cutover. +ConfigKit metadata is retained only at the configuration publication boundary; +its types do not become module or runtime pipeline interfaces. diff --git a/docs/integrations/pkg-configkit.md b/docs/integrations/pkg-configkit.md new file mode 100644 index 00000000..f0994e93 --- /dev/null +++ b/docs/integrations/pkg-configkit.md @@ -0,0 +1,13 @@ +# ConfigKit Integration + +Notarius uses the public root package +[`gitea.maximumdirect.net/eric/ConfigKit`](https://gitea.maximumdirect.net/eric/ConfigKit) +at v0.3.0 for version 5 configuration acquisition, imports, profiles, explicit +overrides, diagnostics, and safe inspection rendering. + +Notarius owns the typed document, domain defaults and validation, runtime +conversion, catalog-dependent pipeline resolution, source-aware configured +paths, and command discovery. It uses only ConfigKit's public path, result, +diagnostic, and resolution-source APIs. Upstream documentation owns general +ConfigKit semantics; [Configuration](../config.md) owns Notarius's file and +operator contract. diff --git a/docs/internal/cli.md b/docs/internal/cli.md index 64196240..2d3d6fa9 100644 --- a/docs/internal/cli.md +++ b/docs/internal/cli.md @@ -25,7 +25,8 @@ preparation, and runner mechanics after their inputs are supplied. ## Dispatch And Configuration Handoff The root dispatcher handles help, version reporting, configuration validation, -pipeline listing, and a pipeline run. Version reporting resolves build +effective-configuration inspection, explanation inspection, pipeline listing, +and a pipeline run. Version reporting resolves build information through `internal/buildinfo` before production composition, so the diagnostic remains available without configuration or runtime collaborators. The public syntax, streams, exit classes, and version semantics are defined by @@ -33,14 +34,15 @@ the [CLI reference](../cli.md). Other root commands normalize injectable options before dispatch so that a missing production dependency fails as a command error rather than reaching execution. -Commands that need configuration use one shared loader. The CLI discovers the -file, parses it through **internal/core/config**, starts from defaults, applies -the file and supported environment overrides, and then validates it for the -command. The configured discovery and precedence contract is in +Commands that need configuration use one shared ConfigKit-backed loader. The +CLI discovers the file, passes the requested profile and only explicit +configuration-valued run overrides, and receives a **LoadedConfig**. The +configured discovery and precedence contract is in [Configuration](../config.md), while the loading and resolution mechanics are in [Configuration Internals](configuration.md). -Configuration validation without a selected pipeline checks structural +`config effective` and `config explain` only render the loaded ConfigKit view; +they do not resolve pipelines or construct providers. Configuration validation without a selected pipeline checks structural configuration only. Validation with a selected pipeline also builds the effective catalog, resolves the pipeline, and verifies every explicit effective PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator @@ -84,8 +86,7 @@ field. After parsing and validating a run invocation, the CLI performs this ordered handoff: -1. load and validate configuration, then apply command-level operational - overrides; +1. load and validate configuration with ConfigKit command overrides; 2. create and validate a safe run identity, then allocate a debug bundle only when requested; 3. build the effective catalog, resolve requested reference changes, resolve diff --git a/docs/internal/configuration.md b/docs/internal/configuration.md index dde1adaa..fdde4087 100644 --- a/docs/internal/configuration.md +++ b/docs/internal/configuration.md @@ -8,17 +8,17 @@ either interface. ## Boundary -The configuration package turns a selected YAML file and supported environment -values into a validated, independently owned configuration. It then resolves a +The configuration package turns a selected ConfigKit-composed YAML document +into a validated, independently owned configuration. It then resolves a requested pipeline against a module catalog before the framework prepares or runs anything. | Boundary | Inputs | Outputs | Does not own | | --- | --- | --- | --- | -| Loading | Selected file path and environment lookup | Parsed file model and a populated **Config** | Choosing the file path or reporting a command result. | +| Loading | Selected file path, profile, overrides, and invocation directory | **LoadedConfig** with ConfigKit inspection metadata and a populated **Config** | Choosing the file path or reporting a command result. | | Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. | | Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. | -| Summary | **Config** or **EffectiveConfig** | Detached redacted payload suitable for debug summaries | Redacting arbitrary process state or provider traffic. | +| Inspection | **LoadedConfig** and an optional JSON Pointer | Redacted effective YAML or explanation text | Pipeline selection, module preparation, or provider work. | The CLI discovers a configuration file, invokes this package, and supplies the result to the framework. Configuration never reads an input file, constructs a @@ -28,23 +28,15 @@ at their respective [CLI](cli.md), [pipeline](pipeline.md), and ## Loading And Validation -The CLI loads configuration in this order: - -1. parse the selected YAML file strictly into the file model; -2. start from **Default**; -3. apply the file model; and -4. apply the supported environment overrides. - -This establishes the public precedence order without giving environment input a -second file schema. Loading and application reject malformed YAML, unsupported -file versions, unknown fields, invalid values, and identifiers that are empty -or collide after whitespace normalization. The file application also makes the -effective extraction-worker default follow the effective LLM limit. A present -PromptKit local-backend object requires and trims its endpoint, defaults its -omitted concurrency limit to zero, and is copied so the parsed file model -cannot alias the populated **Config**. A pipeline `llm_profile` is -presence-aware: omission remains empty, while a present blank value is -rejected and a non-empty file value is trimmed before it reaches **Config**. +ConfigKit acquires the selected root, imports, optional selected profile, and +explicit CLI overrides in precedence order. Its document definition supplies +typed defaults, source-aware path normalization, and structural validation +before conversion to the independently owned runtime **Config**. Only +`NOTARIUS_CONFIG` participates in discovery; retired field-level environment +variables do not participate in loading. Version 5 decoding rejects scalar +module bindings and scalar external references. ConfigKit retains redacted, +historical inspection metadata while **LoadedConfig** keeps that implementation +detail private behind effective and explanation writers. **Config.Validate** checks configuration-only invariants before resolution. It rejects incompatible profile sources, invalid state-surface values, unsupported @@ -57,13 +49,14 @@ It deliberately does not require modules to be registered: that requires a catalog and belongs to resolution. The exact user-selectable values and validation rules are defined in -[Configuration](../config.md). Keep additions to the file model, an -environment override, its validation, and that reference in the same change. +[Configuration](../config.md). Keep document schema, source-aware normalization, +runtime conversion, and that reference aligned in the same change. ## Effective Resolution -**Config.Resolve** first recomputes derived concurrency defaults and validates -the configuration. It normalizes the requested pipeline ID, copies the selected +**Config.Resolve** derives an absent extraction-worker value in an owned copy +and validates the configuration. A programmatically supplied worker value is +always explicit. It normalizes the requested pipeline ID, copies the selected profile, and passes the non-empty command-level LLM profile override, requested lane selection, and reference changes to the framework resolver. @@ -110,43 +103,37 @@ bytes, a debug payload, or all runtime state. The CLI records it as invocation provenance before execution; cache and checkpoint identity have additional owners in [Run State Internals](state.md). -Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or -**RedactedResolvedPipelinePayload**, never a direct configuration marshal. -Those methods copy every binding and nested option container, replace values -whose key is credential-shaped with **[REDACTED]**, and omit materialized -reference content while retaining safe binding and reference provenance. The -payload must not alias the source configuration or resolved pipeline. -PromptKit's local endpoint and concurrency limit are preserved as non-secret -configuration metadata in the independently owned summary; the object contains -no credential value. This redaction is deliberately narrow: it protects -configuration summaries and does not authorize recording arbitrary environment -values or provider requests. +Configuration publication uses ConfigKit's redacted effective YAML. Resolved +pipeline publication uses **RedactedResolvedPipelinePayload**, which copies +bindings and nested option containers, replaces credential-shaped values with +**[REDACTED]**, and omits materialized reference content while retaining safe +provenance. This redaction is deliberately narrow and does not authorize +recording arbitrary environment values or provider requests. ## Invariants To Preserve -- Defaults, YAML values, and environment values are applied in one direction; - later sources may override only their supported operational settings. +- ConfigKit composition applies defaults, imports, an optional profile, and + explicit overrides in one direction with retained origin metadata. - A configuration is structurally valid before it is resolved, and a resolved pipeline is compatible with the supplied catalog before preparation begins. - Whitespace-normalized identifiers are unique wherever they identify a pipeline, step, lane, worker, or reference slot. -- Resolution and summary generation return detached data. Redaction must cover - every configured and resolved binding, including nested validator bindings. +- Runtime conversion and resolved-pipeline publication return detached data. - The resolved digest changes when resolved composition changes and never includes itself. ## Focused Tests -- **internal/core/config/file_config_contract_test.go** covers strict file - parsing, normalization, file application, and structural rejection. -- **internal/core/config/env_contract_test.go** covers supported operational - overrides and their precedence. +- **internal/core/config/document_test.go** and + **operational_document_test.go** cover ConfigKit document loading, + normalization, imports, profiles, and overrides. - **internal/core/config/validation_contract_test.go** covers configuration invariants and contextual failures. - **internal/core/config/effective_config_contract_test.go** covers defaults, selections, overrides, resolution context, digest changes, and ownership. -- **internal/core/config/redaction_test.go** covers recursive credential - redaction, reference-content exclusion, and non-aliasing payloads. +- **internal/core/config/loaded_config_test.go** covers safe effective and + explanation rendering; **redaction_test.go** covers resolved-pipeline + redaction and reference-content exclusion. Run **go test ./internal/core/config** after changing this boundary. Changes to the handoff or resolved-composition semantics also need the focused framework diff --git a/docs/internal/overview.md b/docs/internal/overview.md index c2ef559d..e3a624ac 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -26,7 +26,7 @@ physical state roots. | --- | --- | --- | | Executable and command boundary | **cmd/notarius**, **internal/cli** | Process entry, command dispatch, configuration discovery, production composition, runtime collaborator setup, durable file placement, and user-facing reporting. | | Build information | **internal/buildinfo** | Resolves a stable linked release tag or build metadata for the diagnostic root version command. | -| Configuration | **internal/core/config** | Defaults, strict YAML parsing, environment overrides, structural validation, effective resolution, redaction, and resolved-composition summaries. | +| Configuration | **internal/core/config** | ConfigKit document definition, source-aware normalization, runtime conversion, structural validation, safe inspection, effective resolution, and resolved-composition redaction. | | Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. | | Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. | | LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. | diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 9ab8c7ee..832025d0 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -291,9 +291,11 @@ beneath this mandatory application-wide scheduler. ## Configuration And Provenance -Configuration loading, precedence, defaults, environment overrides, redaction, -and validation are centralized. Named pipeline definitions make structural -composition explicit and discoverable. Operational overrides are permitted +Configuration loading, precedence, defaults, redaction, and validation are +centralized through the ConfigKit-backed boundary. `NOTARIUS_CONFIG` is limited +to discovery; field-level environment overlays are not configuration input. +Named pipeline definitions make structural composition explicit and discoverable. +Explicit command overrides are permitted when they do not obscure the configured pipeline structure. Run preparation fails before stage execution when statically discoverable diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 53cb0df5..ceb2f814 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -44,7 +44,7 @@ Relevant current owners are: | Catalog checks and configured/CLI reference behavior | `internal/framework/pipeline/profile.go`, `references.go`, and CLI reference contract tests | | Dynamic integer consumers | `internal/modules/generic/chunk/units/chunker.go` and `internal/modules/generic/output/json/encoder.go` currently require native `int` | | Debug configuration and resolved-pipeline output | `internal/core/config/redaction.go`, `internal/core/debugbundle/summary.go`, and the CLI debug handoff | -| Maintained examples | `examples/dnd-minimal.config.yml`, `examples/dnd-complete.config.yml`, `examples/profiles/dnd-extraction.yml`, and CLI production contract tests | +| Maintained examples | `examples/dnd-minimal.config.yml`, `examples/dnd-complete.config.yml`, `examples/pipelines/dnd-session.yml`, `examples/llm-profiles/dnd-extraction.yml`, and CLI production contract tests | | Direct YAML test dependency | `internal/cli/production_contract_test.go` also parses embedded prompt definitions using `yaml.v3` | | Upstream contract | ConfigKit `docs/config.md`, `docs/consumers/README.md`, public `load.go`, `lifecycle.go`, `metadata.go`, `records.go`, `path.go`, and consumer examples at v0.3.0 | @@ -652,6 +652,8 @@ one. Verify all moved example links. ## Stage 14 — Final Architecture And Contract Verification +**Status:** Completed 2026-09-13. + **Goal:** Verify the completed architecture and reconcile canonical docs with the implemented boundaries.