Add named pipeline profile composition

This commit is contained in:
2026-08-30 13:25:02 +00:00
parent 8c1171478d
commit f302488075
8 changed files with 802 additions and 24 deletions

View File

@@ -59,6 +59,10 @@ remote state with an unsafe legacy identity must be migrated before use.
root-only `composition.imports` list. Imported files contribute fields to one
logical pipeline document; they do not override fields supplied by the root
or another import.
- A root pipeline may declare named profiles. Exactly one profile is selected
by an option-aware caller or by `composition.default_profile`; a caller's
explicit selection takes precedence. Declaring profiles without either form
of selection is an error.
- Configured timeout and retry-delay durations must be positive. An omitted
artifact timeout continues to inherit its configured Scriptorium timeout.
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
@@ -72,15 +76,22 @@ remote state with an unsafe legacy identity must be migrated before use.
- local (`audio_dir` or `audio_files`), or
- S3 (`audio_s3.prefix`).
### Pipeline imports
### Pipeline composition
Large pipeline configurations may be split into explicitly named fragments:
Large pipeline configurations may be split into explicitly named fragments and
may declare one overlay per selectable profile:
```yaml
composition:
imports:
- config/storage.yml
- config/integrations.yaml
default_profile: production
profiles:
production:
overlay: profiles/production.yml
testing:
overlay: profiles/testing.yml
campaigns:
root: /usr/local/share/narratio/campaigns
@@ -100,6 +111,30 @@ name the full field path and every source that claimed it. The assembled YAML is
then decoded against the normal strict pipeline schema and defaults are applied
once.
Profile names are case-sensitive, non-empty, trimmed, and cannot contain
control characters. If `profiles` is present, it must contain at least one
entry and every entry must contain only an `overlay` path. An explicit profile
selection overrides `default_profile`; unknown and explicitly empty selections
fail. Narratio never selects the first profile implicitly and does not read a
profile selection from the environment.
Every declared overlay is resolved relative to the root pipeline directory and
must satisfy the same confined regular-YAML-file rules as an import. Narratio
parses every declared overlay even when it is not selected, then applies only
the selected one. Maps merge recursively, overlay scalars replace base scalars,
and overlay lists replace base lists completely. Explicit `false`, zero, empty
lists, and empty maps remain meaningful. YAML null cannot delete a value, and
kind changes are rejected. Profiles cannot inherit from or stack with other
profiles, and overlays cannot import files or declare profiles.
After composition, Narratio strictly decodes the result, applies centralized
defaults once, resolves ordinary paths, and computes a deterministic effective
configuration digest. The digest represents the normalized, secret-free
runtime pipeline mapping; it excludes composition declarations, source
provenance, profile identity, and raw environment secret values. Equivalent
effective mappings therefore have the same digest regardless of how fields are
split among the root and imports.
An imported field has the same meaning it would have in a monolithic root
pipeline. In particular, ordinary relative pipeline paths continue to resolve
from the root pipeline directory, not from the importing fragment's directory.
@@ -192,6 +227,8 @@ Rules:
| Field | Type | Required | Default / Rule |
| --- | --- | --- | --- |
| `composition.imports[]` | list of strings | No | explicit additive pipeline fragments relative to the root pipeline directory; `.yml` or `.yaml` regular files only |
| `composition.default_profile` | string | Conditional | selected when profiles exist and no caller explicitly selects one; must name a declared profile |
| `composition.profiles.<name>.overlay` | string | Conditional | required for every declared profile; one confined `.yml` or `.yaml` overlay relative to the root pipeline directory |
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |

View File

@@ -12,17 +12,25 @@ pipeline-loading boundary implemented by `internal/config`.
source names, full field paths, node kinds, declaration order, and explicit
zero, false, empty-map, and empty-list values.
2. Remove the root-only `composition` envelope and validate its explicit
`imports` list.
`imports`, `default_profile`, and named `profiles` declarations. A load
option retains the difference between omitted and explicitly empty profile
selection.
3. Open each import relative to the root pipeline directory through the
confined regular-file boundary. Imports must use a `.yml` or `.yaml`
extension and cannot traverse, use symlinks, repeat a file, import the root,
or contain another composition envelope.
4. Additively merge the root body and imports. Distinct map leaves compose;
4. Resolve and structurally parse every declared profile overlay through the
same confined regular-file boundary. Missing or malformed unselected
overlays fail the load. Overlays cannot contain a composition envelope.
5. Additively merge the root body and imports. Distinct map leaves compose;
repeated scalar or list paths and node-kind disagreements are conflicts.
5. Emit deterministic canonical YAML and strictly decode it into
6. Select exactly one declared profile from an explicit option or the default,
then recursively merge its overlay. Overlay leaves replace base leaves,
lists are atomic replacements, and null or kind changes fail.
7. Emit deterministic canonical YAML and strictly decode it into
`PipelineConfig`.
6. Apply pipeline defaults once, then resolve ordinary relative pipeline paths
from the root pipeline file.
8. Apply pipeline defaults once, resolve ordinary relative pipeline paths from
the root pipeline file, and digest the normalized effective mapping.
This ordering preserves monolithic configuration behavior. Moving a field to
an imported fragment changes its source ownership, not its path base, default,
@@ -36,13 +44,27 @@ claiming source so operators can repair the split without repeatedly
rediscovering additional conflicts.
The loaded pipeline retains private runtime metadata for the absolute root
path, ordered imports, contributing sources, and field ownership. This metadata
does not participate in YAML decoding or alter the public configuration model.
path, ordered imports, selected profile name and selection source, selected
overlay, contributing sources, effective digest, and leaf ownership. Base
leaves retain their root/import owners, replaced leaves belong to the selected
overlay, and centrally supplied values use the synthetic `default` owner. This
metadata does not participate in YAML decoding or alter the public
configuration model.
The effective digest is SHA-256 over deterministic canonical YAML produced from
the defaulted and path-resolved `PipelineConfig`. Because composition and
resolution metadata are private, the digest excludes source layout, profile
name, and ownership. Configuration stores environment variable names rather
than resolving raw credentials, so raw secret values are neither loaded nor
hashed. `recomputePipelineEffectiveDigest` is the single package-owned refresh
point for later runtime expansion.
## Test Surfaces
`composition_test.go` protects the presence and merge algebra independently of
the public schema. `pipeline_composition_test.go` exercises explicit imports,
confinement, conflicts, strict decoding, metadata, and root-relative path
behavior through `LoadPipeline`. Other configuration tests continue to protect
behavior through `LoadPipeline`. `pipeline_profiles_test.go` covers selection,
all-overlay validation, overlay behavior, provenance, option propagation, and
effective-digest stability. Other configuration tests continue to protect
defaults and validation after assembly.

View File

@@ -445,7 +445,7 @@ weakening extract validation or analyze's per-artifact granularity.
## Stage 7 — Named Profile Composition And Effective Digest
**Status: Pending**
**Status: Completed**
### Goal