8.2 KiB
Configuration Internals
This document describes the maintainer-facing configuration boundary in internal/core/config. The Configuration reference owns the file format, fields, defaults, precedence contract, and selectable keys. The CLI reference owns command syntax; this document does not redefine either interface.
Boundary
The configuration package turns a selected ConfigKit-composed YAML document into a validated, independently owned configuration. It then resolves a requested pipeline against a module catalog before the framework prepares or runs anything.
| Boundary | Inputs | Outputs | Does not own |
|---|---|---|---|
| Loading | Selected file path, profile, overrides, and invocation directory | LoadedConfig with ConfigKit inspection metadata and a populated Config | Choosing the file path or reporting a command result. |
| Validation | Config | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
| Resolution | Valid Config, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | EffectiveConfig with a ResolvedPipeline | Materializing reference bytes, preparing modules, execution, or filesystem state. |
| Inspection | LoadedConfig and an optional JSON Pointer | Redacted effective YAML or explanation text | Pipeline selection, module preparation, or provider work. |
The CLI discovers a configuration file, invokes this package, and supplies the result to the framework. Configuration never reads an input file, constructs a module, or creates output, cache, or debug paths. Those responsibilities remain at their respective CLI, pipeline, and run-state boundaries.
Loading And Validation
ConfigKit acquires the selected root, imports, optional selected overlay, and
explicit CLI overrides in precedence order. Its document definition supplies
typed defaults, source-aware path normalization, and structural validation
before conversion to the independently owned runtime Config. Only
NOTARIUS_CONFIG participates in discovery; retired field-level environment
variables do not participate in loading. Version 5 decoding rejects scalar
module bindings and scalar external references. ConfigKit retains redacted,
historical inspection metadata while LoadedConfig keeps that implementation
detail private behind effective and explanation writers.
Config.Validate checks configuration-only invariants before resolution. It rejects incompatible profile sources, invalid state-surface values, unsupported concurrency settings, malformed bindings and references, invalid retries, and invalid pipeline, step, or lane structure. PromptKit local-backend validation accepts only an absolute HTTP or HTTPS endpoint with a host and no user information, query, or fragment, and rejects a negative local concurrency limit. Its errors retain the closest known pipeline, lane, and binding context. It deliberately does not require modules to be registered: that requires a catalog and belongs to resolution.
The exact user-selectable values and validation rules are defined in Configuration. Keep document schema, source-aware normalization, runtime conversion, and that reference aligned in the same change.
Effective Resolution
Config.Resolve derives an absent extraction-worker value in an owned copy and validates the configuration. A programmatically supplied worker value is always explicit. It normalizes the requested pipeline ID, copies the selected profile, and passes the non-empty command-level LLM profile override, requested lane selection, and reference changes to the framework resolver.
After module and validator selection, the resolver applies the effective profile policy to LLM-backed bindings only: command override, binding profile, pipeline profile, then the prompt default. Deterministic bindings remain profile-free, and no second inheritance decision occurs during execution. The public field definitions and precedence are owned by Configuration.
The resolver retains configured validation_policy overrides and derives one
detached concrete terminal policy for the chunk producer and every lane's
extract, merge, and normalize producers. That field-by-field inheritance is
complete before preparation, and the effective values contribute to pipeline
and checkpoint identity; execution does not interpret configuration defaults.
The framework resolver supplies defaults, selects lanes, resolves validator chains, checks registered module and artifact compatibility, validates module options, and returns the fixed ordered pipeline shape. Positive validator retry budgets require an LLM-backed selected validator; deterministic validators are rejected during resolution. Eligible LLM-backed producer specifications also contribute their declared correction protocol to the resolved metadata. The resulting EffectiveConfig retains the selected ID, requested selection and reference changes, a clone of the input configuration, and the resolved pipeline. Callers may therefore retain or modify their input slices and maps without changing the resolved result, and later consumers cannot mutate the original configuration through the effective value. This ownership includes the nested PromptKit local-backend value.
Resolution failures stop before module construction and source parsing. They include an error path for an unconfigured pipeline, missing module, missing capability, incompatible artifact variant, invalid option, invalid reference, or invalid lane selection. CLI code maps these valid-invocation failures to the runtime error class described in the CLI reference.
Resolved Identity And Redaction
The framework assigns the resolved pipeline a deterministic SHA-256 digest after defaults, lane selection, module bindings, reference bindings, validator chains, selected correction protocols, effective LLM profiles, and artifact schema identity have been resolved. The digest excludes its own stored value. It identifies resolved composition rather than raw YAML bytes, a debug payload, or all runtime state. The CLI records it as invocation provenance before execution; cache and checkpoint identity have additional owners in Run State Internals.
Configuration publication uses ConfigKit's redacted effective YAML. Resolved pipeline publication uses RedactedResolvedPipelinePayload, which copies bindings and nested option containers, replaces credential-shaped values with [REDACTED], and omits materialized reference content while retaining safe provenance. This redaction is deliberately narrow and does not authorize recording arbitrary environment values or provider requests.
Invariants To Preserve
- ConfigKit composition applies defaults, imports, an optional overlay, and explicit overrides in one direction with retained origin metadata.
- A configuration is structurally valid before it is resolved, and a resolved pipeline is compatible with the supplied catalog before preparation begins.
- Whitespace-normalized identifiers are unique wherever they identify a pipeline, step, lane, worker, or reference slot.
- Runtime conversion and resolved-pipeline publication return detached data.
- The resolved digest changes when resolved composition changes and never includes itself.
Focused Tests
- internal/core/config/document_test.go and operational_document_test.go cover ConfigKit document loading, normalization, imports, overlays, and overrides.
- internal/core/config/validation_contract_test.go covers configuration invariants and contextual failures.
- internal/core/config/effective_config_contract_test.go covers defaults, selections, overrides, resolution context, digest changes, and ownership.
- internal/core/config/loaded_config_test.go covers safe effective and explanation rendering; redaction_test.go covers resolved-pipeline redaction and reference-content exclusion.
Run go test ./internal/core/config after changing this boundary. Changes to the handoff or resolved-composition semantics also need the focused framework pipeline tests.