Add explicit pipeline configuration imports

This commit is contained in:
2026-08-30 12:33:43 +00:00
parent 49ea747b17
commit b97b12da7f
11 changed files with 793 additions and 38 deletions

View File

@@ -0,0 +1,48 @@
# Configuration Internals
User-visible fields, defaults, and selection behavior belong in the
[Configuration Reference](../config.md). This document describes the internal
pipeline-loading boundary implemented by `internal/config`.
## Pipeline Loading
`LoadPipeline` assembles and validates a pipeline in this order:
1. Parse the root YAML into a presence-aware composition tree. The tree retains
source names, full field paths, node kinds, declaration order, and explicit
zero, false, empty-map, and empty-list values.
2. Remove the root-only `composition` envelope and validate its explicit
`imports` list.
3. Open each import relative to the root pipeline directory through the
confined regular-file boundary. Imports must use a `.yml` or `.yaml`
extension and cannot traverse, use symlinks, repeat a file, import the root,
or contain another composition envelope.
4. Additively merge the root body and imports. Distinct map leaves compose;
repeated scalar or list paths and node-kind disagreements are conflicts.
5. Emit deterministic canonical YAML and strictly decode it into
`PipelineConfig`.
6. Apply pipeline defaults once, then resolve ordinary relative pipeline paths
from the root pipeline file.
This ordering preserves monolithic configuration behavior. Moving a field to
an imported fragment changes its source ownership, not its path base, default,
or schema semantics.
## Diagnostics And Runtime Metadata
Syntax, duplicate-key, composition, conflict, and schema failures include the
relevant source name and full field path. Additive conflicts report every
claiming source so operators can repair the split without repeatedly
rediscovering additional conflicts.
The loaded pipeline retains private runtime metadata for the absolute root
path, ordered imports, contributing sources, and field ownership. This metadata
does not participate in YAML decoding or alter the public configuration model.
## Test Surfaces
`composition_test.go` protects the presence and merge algebra independently of
the public schema. `pipeline_composition_test.go` exercises explicit imports,
confinement, conflicts, strict decoding, metadata, and root-relative path
behavior through `LoadPipeline`. Other configuration tests continue to protect
defaults and validation after assembly.

View File

@@ -28,7 +28,7 @@ progress and artifact services resolve durable inputs and outputs.
| --- | --- | --- |
| Executable | `cmd/narratio` | Process entry, standard stream wiring, argument handoff, and exit status. |
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, cleanup gates, and user-facing reporting. |
| Configuration | `internal/config` | Strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
| Configuration | [`internal/config`](configuration.md) | Presence-aware pipeline composition, strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
| Pipeline stages | `internal/stage` | Canonical stage registry, shared stage contract, execution dependencies, and implemented stage behavior. |
| External boundaries | `internal/adapters`, `internal/audio` | WhisperX HTTP, downstream subprocesses, notification, object storage, and S3 audio materialization behind Narratio contracts. |
| Manifests | `internal/manifest` | Durable session progress, invocation audit state, stage transitions, validation, and atomic persistence. |
@@ -82,6 +82,8 @@ trimmed transcript state: neither invalidates the other, while either can stale
## Focused Documentation
- [Configuration Internals](configuration.md): pipeline composition, import
confinement, field ownership, decoding, and root-relative path semantics.
- [Adapter Internals](adapters.md): external adapter boundaries, composition,
failure behavior, and test surfaces.
- [Artifact Internals](artifacts.md): source identities, runtime catalog,