Add explicit pipeline configuration imports
This commit is contained in:
@@ -55,6 +55,10 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
|
||||
- YAML decode is strict (`KnownFields(true)`) and accepts exactly one document:
|
||||
unknown fields or trailing documents fail load.
|
||||
- A pipeline file may explicitly import additive YAML fragments through the
|
||||
root-only `composition.imports` list. Imported files contribute fields to one
|
||||
logical pipeline document; they do not override fields supplied by the root
|
||||
or another import.
|
||||
- Configured timeout and retry-delay durations must be positive. An omitted
|
||||
artifact timeout continues to inherit its configured Scriptorium timeout.
|
||||
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||
@@ -68,6 +72,38 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
- local (`audio_dir` or `audio_files`), or
|
||||
- S3 (`audio_s3.prefix`).
|
||||
|
||||
### Pipeline imports
|
||||
|
||||
Large pipeline configurations may be split into explicitly named fragments:
|
||||
|
||||
```yaml
|
||||
composition:
|
||||
imports:
|
||||
- config/storage.yml
|
||||
- config/integrations.yaml
|
||||
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
```
|
||||
|
||||
Imports are resolved relative to the directory containing the root pipeline
|
||||
file and are loaded in declaration order. Narratio does not scan directories or
|
||||
infer fragments. Each import must be a confined regular `.yml` or `.yaml` file:
|
||||
absolute paths, traversal, symlinks, directories, duplicate files, and an
|
||||
import of the root pipeline itself are rejected. Only the root pipeline may
|
||||
contain `composition`; nested composition is rejected.
|
||||
|
||||
Composition is additive. A map may be extended by multiple files when every
|
||||
leaf is distinct, but a scalar, list, or map/list/scalar kind cannot be claimed
|
||||
more than once, even when the repeated values are identical. Conflict errors
|
||||
name the full field path and every source that claimed it. The assembled YAML is
|
||||
then decoded against the normal strict pipeline schema and defaults are applied
|
||||
once.
|
||||
|
||||
An imported field has the same meaning it would have in a monolithic root
|
||||
pipeline. In particular, ordinary relative pipeline paths continue to resolve
|
||||
from the root pipeline directory, not from the importing fragment's directory.
|
||||
|
||||
## Minimal Working Configuration
|
||||
|
||||
`pipeline.yml`
|
||||
@@ -155,6 +191,7 @@ Rules:
|
||||
|
||||
| Field | Type | Required | Default / Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `composition.imports[]` | list of strings | No | explicit additive pipeline fragments relative to the root pipeline directory; `.yml` or `.yaml` regular files only |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
|
||||
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
||||
|
||||
48
docs/internal/configuration.md
Normal file
48
docs/internal/configuration.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Configuration Internals
|
||||
|
||||
User-visible fields, defaults, and selection behavior belong in the
|
||||
[Configuration Reference](../config.md). This document describes the internal
|
||||
pipeline-loading boundary implemented by `internal/config`.
|
||||
|
||||
## Pipeline Loading
|
||||
|
||||
`LoadPipeline` assembles and validates a pipeline in this order:
|
||||
|
||||
1. Parse the root YAML into a presence-aware composition tree. The tree retains
|
||||
source names, full field paths, node kinds, declaration order, and explicit
|
||||
zero, false, empty-map, and empty-list values.
|
||||
2. Remove the root-only `composition` envelope and validate its explicit
|
||||
`imports` list.
|
||||
3. Open each import relative to the root pipeline directory through the
|
||||
confined regular-file boundary. Imports must use a `.yml` or `.yaml`
|
||||
extension and cannot traverse, use symlinks, repeat a file, import the root,
|
||||
or contain another composition envelope.
|
||||
4. Additively merge the root body and imports. Distinct map leaves compose;
|
||||
repeated scalar or list paths and node-kind disagreements are conflicts.
|
||||
5. Emit deterministic canonical YAML and strictly decode it into
|
||||
`PipelineConfig`.
|
||||
6. Apply pipeline defaults once, then resolve ordinary relative pipeline paths
|
||||
from the root pipeline file.
|
||||
|
||||
This ordering preserves monolithic configuration behavior. Moving a field to
|
||||
an imported fragment changes its source ownership, not its path base, default,
|
||||
or schema semantics.
|
||||
|
||||
## Diagnostics And Runtime Metadata
|
||||
|
||||
Syntax, duplicate-key, composition, conflict, and schema failures include the
|
||||
relevant source name and full field path. Additive conflicts report every
|
||||
claiming source so operators can repair the split without repeatedly
|
||||
rediscovering additional conflicts.
|
||||
|
||||
The loaded pipeline retains private runtime metadata for the absolute root
|
||||
path, ordered imports, contributing sources, and field ownership. This metadata
|
||||
does not participate in YAML decoding or alter the public configuration model.
|
||||
|
||||
## Test Surfaces
|
||||
|
||||
`composition_test.go` protects the presence and merge algebra independently of
|
||||
the public schema. `pipeline_composition_test.go` exercises explicit imports,
|
||||
confinement, conflicts, strict decoding, metadata, and root-relative path
|
||||
behavior through `LoadPipeline`. Other configuration tests continue to protect
|
||||
defaults and validation after assembly.
|
||||
@@ -28,7 +28,7 @@ progress and artifact services resolve durable inputs and outputs.
|
||||
| --- | --- | --- |
|
||||
| Executable | `cmd/narratio` | Process entry, standard stream wiring, argument handoff, and exit status. |
|
||||
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, cleanup gates, and user-facing reporting. |
|
||||
| Configuration | `internal/config` | Strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
|
||||
| Configuration | [`internal/config`](configuration.md) | Presence-aware pipeline composition, strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
|
||||
| Pipeline stages | `internal/stage` | Canonical stage registry, shared stage contract, execution dependencies, and implemented stage behavior. |
|
||||
| External boundaries | `internal/adapters`, `internal/audio` | WhisperX HTTP, downstream subprocesses, notification, object storage, and S3 audio materialization behind Narratio contracts. |
|
||||
| Manifests | `internal/manifest` | Durable session progress, invocation audit state, stage transitions, validation, and atomic persistence. |
|
||||
@@ -82,6 +82,8 @@ trimmed transcript state: neither invalidates the other, while either can stale
|
||||
|
||||
## Focused Documentation
|
||||
|
||||
- [Configuration Internals](configuration.md): pipeline composition, import
|
||||
confinement, field ownership, decoding, and root-relative path semantics.
|
||||
- [Adapter Internals](adapters.md): external adapter boundaries, composition,
|
||||
failure behavior, and test surfaces.
|
||||
- [Artifact Internals](artifacts.md): source identities, runtime catalog,
|
||||
|
||||
@@ -206,7 +206,7 @@ needed by imports and profiles without changing the public pipeline schema.
|
||||
|
||||
## Stage 2 — Explicit Additive Imports
|
||||
|
||||
**Status: Pending**
|
||||
**Status: Completed**
|
||||
|
||||
### Goal
|
||||
|
||||
|
||||
Reference in New Issue
Block a user