Add named pipeline profile composition
This commit is contained in:
@@ -59,6 +59,10 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
root-only `composition.imports` list. Imported files contribute fields to one
|
||||
logical pipeline document; they do not override fields supplied by the root
|
||||
or another import.
|
||||
- A root pipeline may declare named profiles. Exactly one profile is selected
|
||||
by an option-aware caller or by `composition.default_profile`; a caller's
|
||||
explicit selection takes precedence. Declaring profiles without either form
|
||||
of selection is an error.
|
||||
- Configured timeout and retry-delay durations must be positive. An omitted
|
||||
artifact timeout continues to inherit its configured Scriptorium timeout.
|
||||
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||
@@ -72,15 +76,22 @@ remote state with an unsafe legacy identity must be migrated before use.
|
||||
- local (`audio_dir` or `audio_files`), or
|
||||
- S3 (`audio_s3.prefix`).
|
||||
|
||||
### Pipeline imports
|
||||
### Pipeline composition
|
||||
|
||||
Large pipeline configurations may be split into explicitly named fragments:
|
||||
Large pipeline configurations may be split into explicitly named fragments and
|
||||
may declare one overlay per selectable profile:
|
||||
|
||||
```yaml
|
||||
composition:
|
||||
imports:
|
||||
- config/storage.yml
|
||||
- config/integrations.yaml
|
||||
default_profile: production
|
||||
profiles:
|
||||
production:
|
||||
overlay: profiles/production.yml
|
||||
testing:
|
||||
overlay: profiles/testing.yml
|
||||
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
@@ -100,6 +111,30 @@ name the full field path and every source that claimed it. The assembled YAML is
|
||||
then decoded against the normal strict pipeline schema and defaults are applied
|
||||
once.
|
||||
|
||||
Profile names are case-sensitive, non-empty, trimmed, and cannot contain
|
||||
control characters. If `profiles` is present, it must contain at least one
|
||||
entry and every entry must contain only an `overlay` path. An explicit profile
|
||||
selection overrides `default_profile`; unknown and explicitly empty selections
|
||||
fail. Narratio never selects the first profile implicitly and does not read a
|
||||
profile selection from the environment.
|
||||
|
||||
Every declared overlay is resolved relative to the root pipeline directory and
|
||||
must satisfy the same confined regular-YAML-file rules as an import. Narratio
|
||||
parses every declared overlay even when it is not selected, then applies only
|
||||
the selected one. Maps merge recursively, overlay scalars replace base scalars,
|
||||
and overlay lists replace base lists completely. Explicit `false`, zero, empty
|
||||
lists, and empty maps remain meaningful. YAML null cannot delete a value, and
|
||||
kind changes are rejected. Profiles cannot inherit from or stack with other
|
||||
profiles, and overlays cannot import files or declare profiles.
|
||||
|
||||
After composition, Narratio strictly decodes the result, applies centralized
|
||||
defaults once, resolves ordinary paths, and computes a deterministic effective
|
||||
configuration digest. The digest represents the normalized, secret-free
|
||||
runtime pipeline mapping; it excludes composition declarations, source
|
||||
provenance, profile identity, and raw environment secret values. Equivalent
|
||||
effective mappings therefore have the same digest regardless of how fields are
|
||||
split among the root and imports.
|
||||
|
||||
An imported field has the same meaning it would have in a monolithic root
|
||||
pipeline. In particular, ordinary relative pipeline paths continue to resolve
|
||||
from the root pipeline directory, not from the importing fragment's directory.
|
||||
@@ -192,6 +227,8 @@ Rules:
|
||||
| Field | Type | Required | Default / Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `composition.imports[]` | list of strings | No | explicit additive pipeline fragments relative to the root pipeline directory; `.yml` or `.yaml` regular files only |
|
||||
| `composition.default_profile` | string | Conditional | selected when profiles exist and no caller explicitly selects one; must name a declared profile |
|
||||
| `composition.profiles.<name>.overlay` | string | Conditional | required for every declared profile; one confined `.yml` or `.yaml` overlay relative to the root pipeline directory |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
|
||||
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
||||
|
||||
@@ -12,17 +12,25 @@ pipeline-loading boundary implemented by `internal/config`.
|
||||
source names, full field paths, node kinds, declaration order, and explicit
|
||||
zero, false, empty-map, and empty-list values.
|
||||
2. Remove the root-only `composition` envelope and validate its explicit
|
||||
`imports` list.
|
||||
`imports`, `default_profile`, and named `profiles` declarations. A load
|
||||
option retains the difference between omitted and explicitly empty profile
|
||||
selection.
|
||||
3. Open each import relative to the root pipeline directory through the
|
||||
confined regular-file boundary. Imports must use a `.yml` or `.yaml`
|
||||
extension and cannot traverse, use symlinks, repeat a file, import the root,
|
||||
or contain another composition envelope.
|
||||
4. Additively merge the root body and imports. Distinct map leaves compose;
|
||||
4. Resolve and structurally parse every declared profile overlay through the
|
||||
same confined regular-file boundary. Missing or malformed unselected
|
||||
overlays fail the load. Overlays cannot contain a composition envelope.
|
||||
5. Additively merge the root body and imports. Distinct map leaves compose;
|
||||
repeated scalar or list paths and node-kind disagreements are conflicts.
|
||||
5. Emit deterministic canonical YAML and strictly decode it into
|
||||
6. Select exactly one declared profile from an explicit option or the default,
|
||||
then recursively merge its overlay. Overlay leaves replace base leaves,
|
||||
lists are atomic replacements, and null or kind changes fail.
|
||||
7. Emit deterministic canonical YAML and strictly decode it into
|
||||
`PipelineConfig`.
|
||||
6. Apply pipeline defaults once, then resolve ordinary relative pipeline paths
|
||||
from the root pipeline file.
|
||||
8. Apply pipeline defaults once, resolve ordinary relative pipeline paths from
|
||||
the root pipeline file, and digest the normalized effective mapping.
|
||||
|
||||
This ordering preserves monolithic configuration behavior. Moving a field to
|
||||
an imported fragment changes its source ownership, not its path base, default,
|
||||
@@ -36,13 +44,27 @@ claiming source so operators can repair the split without repeatedly
|
||||
rediscovering additional conflicts.
|
||||
|
||||
The loaded pipeline retains private runtime metadata for the absolute root
|
||||
path, ordered imports, contributing sources, and field ownership. This metadata
|
||||
does not participate in YAML decoding or alter the public configuration model.
|
||||
path, ordered imports, selected profile name and selection source, selected
|
||||
overlay, contributing sources, effective digest, and leaf ownership. Base
|
||||
leaves retain their root/import owners, replaced leaves belong to the selected
|
||||
overlay, and centrally supplied values use the synthetic `default` owner. This
|
||||
metadata does not participate in YAML decoding or alter the public
|
||||
configuration model.
|
||||
|
||||
The effective digest is SHA-256 over deterministic canonical YAML produced from
|
||||
the defaulted and path-resolved `PipelineConfig`. Because composition and
|
||||
resolution metadata are private, the digest excludes source layout, profile
|
||||
name, and ownership. Configuration stores environment variable names rather
|
||||
than resolving raw credentials, so raw secret values are neither loaded nor
|
||||
hashed. `recomputePipelineEffectiveDigest` is the single package-owned refresh
|
||||
point for later runtime expansion.
|
||||
|
||||
## Test Surfaces
|
||||
|
||||
`composition_test.go` protects the presence and merge algebra independently of
|
||||
the public schema. `pipeline_composition_test.go` exercises explicit imports,
|
||||
confinement, conflicts, strict decoding, metadata, and root-relative path
|
||||
behavior through `LoadPipeline`. Other configuration tests continue to protect
|
||||
behavior through `LoadPipeline`. `pipeline_profiles_test.go` covers selection,
|
||||
all-overlay validation, overlay behavior, provenance, option propagation, and
|
||||
effective-digest stability. Other configuration tests continue to protect
|
||||
defaults and validation after assembly.
|
||||
|
||||
@@ -445,7 +445,7 @@ weakening extract validation or analyze's per-artifact granularity.
|
||||
|
||||
## Stage 7 — Named Profile Composition And Effective Digest
|
||||
|
||||
**Status: Pending**
|
||||
**Status: Completed**
|
||||
|
||||
### Goal
|
||||
|
||||
|
||||
Reference in New Issue
Block a user