Files
narratio/docs/roadmap/pipeline-configuration-ergonomics.md

740 lines
32 KiB
Markdown

# Pipeline Configuration Ergonomics
## Status
Accepted target state. This document owns the intended feature scope, user
intent, policy choices, and target end state until the behavior is implemented.
## Goal
Make Narratio's pipeline configuration easier to organize, vary, inspect, and
reuse without weakening its strict configuration contract or turning it into a
general-purpose configuration or workflow language.
The feature should let operators:
- split a large pipeline configuration into explicitly imported partial files;
- maintain one production-default pipeline plus a selectable testing profile;
- keep stable settings separate from frequently edited models and artifacts;
- define a character-oriented artifact workflow once and expand it for every
character in the selected campaign;
- reuse one campaign-owned party roster across multiple artifact families and
the Notarius integration; and
- inspect and compare the fully resolved configuration before running a
session.
All composition and expansion must produce the same concrete, validated
`PipelineConfig` and concrete configured-artifact map consumed by the existing
stage, planning, manifest, artifact, and adapter boundaries.
## User Intent
Pipeline configuration currently combines long-lived platform and transcript
settings with model choices, prompt-backed artifact definitions, and publish
rules that change frequently during development. Keeping all of those concerns
in one file makes edits noisy and makes production/testing variation difficult
to review.
Character-specific artifacts introduce a second kind of repetition. Several
artifact families, including character meta-analysis and character-specific
item tracking, should execute once per campaign character. Their prompt,
inputs, dependencies, and output shape are shared, while player, character,
class, alias, and output identity vary by party member. Those campaign facts do
not belong in a global pipeline definition.
The desired model is therefore:
```text
pipeline.yml + explicit imports + one selected profile
campaign.yml + party.yml ─┤
resolved effective configuration
expand party-driven families
defaults and strict validation
existing concrete pipeline and artifact model
```
## Architectural Constraints
The feature must preserve the configuration and orchestration policies in
[`docs/policy/architecture.md`](../policy/architecture.md):
- YAML remains strict and rejects unknown fields.
- Defaults remain centralized and are applied exactly once.
- Composition and expansion are deterministic and testable.
- Session templating remains narrow; this feature does not introduce arbitrary
expressions, scripting, or a general template language.
- The canonical stage sequence and fixed invalidation relation remain
application-owned and non-configurable.
- Stages receive resolved Narratio-owned configuration and do not implement
import, profile, or family semantics independently.
- External tools continue to own their private runtime configuration and
defaults.
- Raw secrets remain indirect and are never introduced into composed YAML,
effective-configuration output, manifests, or diagnostics.
Configuration composition belongs to `internal/config`. CLI selection and
production loading belong to `internal/app`. Party parsing must be a
Narratio-owned campaign-input contract. Artifact-family expansion must finish
before existing configured-artifact validation, catalog construction,
planning, reconciliation, or execution begins.
## Configuration Imports
### Root document
Keep `pipeline.yml` as the selected pipeline entry point. Add an optional
top-level composition envelope with explicitly ordered imports:
```yaml
composition:
imports:
- conf.d/platform.yml
- conf.d/transcripts.yml
- conf.d/extraction.yml
- conf.d/artifacts.yml
- conf.d/publish.yml
default_profile: production
profiles:
production:
overlay: profiles/production.yml
testing:
overlay: profiles/testing.yml
```
An existing monolithic `pipeline.yml` without `composition` remains valid and
retains its current behavior. The root document may contain ordinary pipeline
fields alongside `composition`, allowing incremental migration.
`imports` is the public term. Imported files are partial pipeline
configuration documents rather than independently runnable pipelines.
### Import boundaries
- Only the root pipeline document may declare `composition`, imports, or
profiles. Imported and overlay documents cannot import other documents.
- Every import is explicit. Narratio does not automatically scan `conf.d` or
interpret unlisted files.
- Import paths are relative to the root pipeline file's directory, must remain
confined beneath that directory, and must resolve to supported regular YAML
files. Absolute paths, traversal, non-regular files, and cycles are rejected.
- All ordinary relative pipeline paths retain one stable base: the root
pipeline file's directory. Moving a field between imported files must not
silently change the meaning of its relative path.
- The declared import order is retained for diagnostics and provenance, but it
is not an implicit precedence mechanism between base imports.
- Composition metadata is removed before the effective mapping is decoded as
strict `PipelineConfig` data.
### Additive merge contract
Base configuration from the root document and its imports is additive:
- mappings merge recursively when their child fields are disjoint;
- two files may therefore contribute different entries beneath a shared map
such as `scriptorium.artifacts`;
- defining the same final field or keyed entry in more than one base source is
an error, even when the values are identical;
- lists are atomic values and cannot be contributed more than once at the same
base configuration path;
- mapping/scalar, mapping/list, and other incompatible type collisions are
errors; and
- duplicate YAML keys within one source remain errors.
Errors must identify the complete configuration path and every source file
that claims it. This contract prevents an import reorder or newly added file
from silently changing production behavior.
Composition must operate on a presence-aware YAML representation before Go
struct decoding. Merging already decoded structs is not sufficient because it
cannot reliably distinguish omission from explicitly configured `false`, zero,
an empty collection, or another meaningful zero value.
## Named Profiles
### Selection and precedence
Exactly one named profile may be active for an invocation. A composition root
may define `default_profile`; the maintained production-shaped configuration
uses `production`. Add `--profile <name>` to the common configuration flags for
every command that loads pipeline configuration:
```bash
narratio run SESSION
narratio run SESSION --profile testing
narratio session plan SESSION --profile testing
narratio regenerate-artifacts SESSION --profile testing
```
The explicit CLI value overrides `default_profile`. An unknown profile, a
profile selection against a configuration that defines no such profile, an
empty profile name, or a default naming an undefined profile is an error before
stage composition. Do not add an environment-variable selector in this
feature; profile selection should remain visible in the command or root
configuration.
Only one profile can be selected. Profiles cannot extend other profiles, and
callers cannot stack several profiles. A single overlay provides the required
production/testing variation without creating user-programmable precedence.
### Overlay semantics
The selected profile is the sole intentional override layer:
- mappings recursively merge with the assembled base mapping;
- scalar and boolean leaves replace base leaves;
- lists replace the complete base list rather than concatenate implicitly;
- keyed maps such as configured artifacts merge by key, allowing a testing
profile to add experimental artifacts;
- an existing artifact can be disabled explicitly with `enabled: false`;
- an overlay cannot change a value's YAML kind incompatibly; and
- the fully overlaid result undergoes the same strict decode, defaults, path
resolution, expansion, and validation as a monolithic configuration.
Generic deletion syntax is outside this feature. Shared definitions should
live in base imports, while production-only and testing-only definitions live
in their corresponding overlays. Existing enable/disable fields and complete
list replacement cover the required cases without a YAML patch language.
### Effective identity and provenance
Every loaded configuration must retain bounded, non-secret provenance:
- the root pipeline path;
- the selected profile name, including whether it came from the default or
CLI;
- the ordered imported and overlay source paths;
- a deterministic digest of the normalized effective pipeline configuration;
and
- enough source ownership information to explain composition errors and
effective values.
Run reporting and invocation state should identify the selected profile and
effective configuration digest. The profile name is provenance, not a blanket
cache key: changing profiles must invalidate only work whose result-affecting
effective configuration changed.
## Canonical Campaign Party Contract
### Ownership and schema
Make `party.yml` a strict, versioned Narratio campaign-input contract. A
canonical file has this shape:
```yaml
schema_version: narratio.party.v1
characters:
arannis:
player:
name: Eric
character:
name: Arannis
alias:
- Ari
- The Grey Owl
classes:
- name: wizard
level: 8
brenna:
player:
name: Jane
character:
name: Brenna
classes:
- name: paladin
level: 6
- name: warlock
level: 2
```
The contract has these semantics:
- `schema_version` is required and must equal `narratio.party.v1`.
- `characters` is a non-empty mapping.
- Each mapping key is the stable `character_id` and must satisfy the existing
configured-artifact key grammar.
- `player.name` and `character.name` are required, non-empty display strings.
- `character.alias` is optional and accepts a list of zero or more non-empty
alias strings. The singular field spelling `alias` is intentional.
- Leading or trailing whitespace and control characters in names, aliases, and
class names are rejected rather than silently normalized.
- A character alias cannot equal that character's primary name, another alias,
or another character's primary name or alias under case-insensitive
comparison. This keeps party-member grounding unambiguous. Player names may
repeat because one player may control more than one character.
- `character.classes` is required and non-empty. Every entry has a required
free-form `name` and an optional positive integer `level`.
- Class names are not restricted to a Narratio-owned D&D enumeration.
- Duplicate class names for one character are rejected case-insensitively.
- Class order and alias order are preserved.
- Narratio derives a deterministic class summary such as
`paladin 6 / warlock 2` and an alias summary that joins declared aliases with
`, ` in their declared order.
- Unknown fields and trailing YAML documents are rejected.
The stable `character_id`, rather than a display name, owns generated artifact
identity. Changing a display name, alias, class, or level retains that identity
and changes the relevant semantic inputs. Changing the mapping key is an
intentional remove-and-add operation.
### Load and prepare behavior
The selected campaign's effective `party_file` resolves relative to
`campaign.yml`. Narratio must load and validate canonical party data during
combined pipeline/campaign resolution so `session plan`, configuration
inspection, and artifact-family expansion see the same roster before stage
execution.
`prepare` continues to materialize the canonical document as
`inputs/party.yml`, record its source and content identity, and expose it as
`narratio.input.party`. The same canonical file is passed unchanged to
Notarius's `party` reference slot, which currently accepts YAML reference
material without imposing a competing roster schema.
### Players projection and compatibility
The canonical party document becomes the eventual single source of truth for
player-to-character relationships. Narratio derives a deterministic,
documented players-only YAML projection during prepare, materializes it at
`inputs/players.yml`, records its content identity, and exposes it through the
existing `narratio.input.players` source. This preserves the separate Notarius
`players` reference and existing Scriptorium source without requiring every
campaign to maintain duplicate facts.
The projection uses this versioned shape, with entries sorted by stable
character ID:
```yaml
schema_version: narratio.players.v1
players:
- name: Eric
character:
id: arannis
name: Arannis
alias:
- Ari
- The Grey Owl
```
There is one projection entry per character rather than one grouped entry per
display player name. This permits one player to control several characters and
does not conflate distinct players who happen to share a display name.
Backward compatibility must be narrow and removable:
- an unversioned legacy party file continues to be treated as opaque reference
material when the existing `players_file` is also configured;
- legacy mode preserves the current prepared `party` and `players` behavior
but cannot drive artifact-family expansion;
- configuring a party-driven artifact family with a legacy roster produces a
clear migration error;
- canonical `narratio.party.v1` mode derives players and rejects a separate
`players_file`, preventing contradictory authorities;
- parsing, validation, documentation, and tests for legacy mode live behind a
clearly identified compatibility boundary with comments stating that it is
intended for removal after migration; and
- the maintained examples migrate to canonical mode and demonstrate no
separate players file.
The derived players projection is a durable integration contract and must be
documented under `docs/integrations/` rather than left as incidental generated
YAML.
## Party-Driven Artifact Families
### Configuration model
Add `scriptorium.artifact_families`, separate from the existing concrete
`scriptorium.artifacts` map:
```yaml
scriptorium:
artifact_families:
character_meta:
enabled: true
for_each: party.characters
prompt_id: dnd.character_meta
profile_id: production
output_path_pattern: artifacts/characters/{character_id}/meta.md
inputs:
transcript:
source: narratio.transcript.final_trimmed
required: true
member_vars:
player_name: player.name
character_name: character.name
character_class: character.class_summary
character_aliases: character.alias_summary
character_items:
enabled: true
for_each: party.characters
prompt_id: dnd.character_items
profile_id: production
output_path_pattern: artifacts/characters/{character_id}/items.md
inputs:
transcript:
source: narratio.transcript.final_trimmed
required: true
item_occurrences:
source: narratio.extraction.item_occurrences
required: true
member_vars:
character_name: character.name
```
Families support the ordinary result-affecting fields shared by concrete
Scriptorium artifacts: enablement, prompt and profile IDs, timeout,
render-debug policy, ordinary dependencies, inputs, and static variables.
`output_path_pattern` replaces concrete `output_path` at the family level.
`for_each` is not an expression language. This feature accepts only the exact
source `party.characters`. `member_vars` maps a Scriptorium variable name to
one of a closed set of canonical values:
- `character_id`;
- `player.name`;
- `character.name`;
- `character.class_summary`; and
- `character.alias_summary`.
Static `vars` and resolved member variables merge deterministically. Duplicate
variable names across the two maps are rejected rather than assigned implicit
precedence. Narratio's existing reserved sticky-session variable remains
application-owned and is applied after expansion under its current rules.
### Narrow substitution
`{character_id}` is the only family substitution supported in this feature.
It is required in `output_path_pattern` and may appear exactly where documented
for family-owned publish destinations. Unknown, repeated in an invalid
position, unresolved, or malformed substitutions are errors. Names, aliases,
classes, environment values, and arbitrary YAML paths cannot be interpolated
into configuration strings.
### Concrete expansion
For each family and party character, resolution creates one ordinary concrete
artifact key by joining the normalized family key, an underscore, and the
stable character ID:
```text
character_meta_arannis
character_items_arannis
```
Expansion must be sorted, deterministic, and completed before existing
configured-artifact defaults and validation. Every generated artifact then
uses the existing catalog, dependency planner, fingerprinting, manifest,
reconciliation, execution, materialization, and publish boundaries.
Validation rejects:
- invalid family keys or character IDs;
- generated keys that fail the existing configured-artifact grammar;
- collisions between generated artifacts, explicit concrete artifacts, or
family names;
- duplicate or escaping output paths;
- a family with no canonical party source;
- invalid member-variable selectors;
- a selected profile that leaves an executable family incomplete; and
- any expanded artifact that fails ordinary concrete-artifact validation.
An added party character creates missing concrete artifacts. Removing a
character removes those artifacts from effective configuration so existing
analyze reconciliation classifies their old records as removed. A change to a
member field changes only fingerprints that actually consume its resolved
value or the party document as an input.
### Dependencies between families
Ordinary `depends_on` continues to name shared concrete artifacts such as
`session_recap`. Add a typed `member_dependencies` list for corresponding
members of another family:
```yaml
character_items:
member_dependencies:
- character_meta
```
For `arannis`, this expands to a dependency on
`character_meta_arannis`. The referenced family must use the same party source
and generate the same character ID. Cycles and missing members are rejected by
the existing concrete dependency validation after expansion.
When a family input consumes the corresponding member output, provide a narrow
pre-expansion member-artifact source form owned by configuration resolution.
The syntax is:
```yaml
inputs:
prior_meta:
source: narratio.member_artifact.character_meta
required: true
```
For the `arannis` member, this resolves to
`narratio.artifact.character_meta_arannis` before ordinary runtime validation.
The referenced family must also appear in `member_dependencies`. The
`narratio.member_artifact.*` form must not survive into stage or adapter
configuration as a new runtime artifact kind.
### Selection and reporting
Extend artifact selection so an exact family key selects all of that family's
concrete members. An exact generated key selects only that member. Explicitly
selecting a disabled family follows the existing rule for explicitly selected
disabled concrete artifacts: all generated targets must still have valid
executable fields. Mixed family and concrete selections are normalized and
deduplicated after expansion.
Planning, run summaries, artifact listing, and effective-configuration output
must show both family origin and concrete artifact identity. Execution remains
entirely concrete; a family is not a dynamic pipeline stage or runtime loop.
### Family publish rules
Allow a family to declare one typed publish policy so operators need not repeat
one publish output per character:
```yaml
publish:
enabled: true
required: false
dest_pattern: artifacts/characters/{character_id}/meta.md
```
Resolution expands this into ordinary concrete publish rules. When
`dest_pattern` is omitted, the generated artifact's output path is used where
the existing publish contract permits derivation. Conflicts with explicit
publish rules, duplicate destinations, unsafe paths, and generated sources
that are unavailable are rejected by the existing publish policy after
expansion. Family publish behavior must not introduce wildcard source matching
at runtime.
## Profile-Safe Resume And Invalidation
Named profiles are safe only if result-affecting model and configuration
changes cannot silently reuse incompatible successful stage results. Artifact
analysis already has a versioned semantic fingerprint that includes effective
prompt/profile identifiers, variables, dependencies, inputs, and output
identity. The broader pipeline requires the same principle at every applicable
stage boundary.
Audit each stage's resume contract and add a versioned semantic configuration
fingerprint wherever current evidence does not already cover all
Narratio-observable result-affecting settings. In particular, switching an
Audita model through a profile must not reuse polish output produced by the
testing model. Conversely, changing only an operational timeout, binary path,
workspace path, diagnostic location, or profile name must not invalidate
byte-equivalent semantic work unless that value genuinely affects the stage's
canonical result.
The selected profile name and complete effective-config digest are retained as
provenance. Resume decisions use stage-specific effective semantics rather
than the profile name as a blanket invalidator. This permits production and
testing profiles to share unaffected transcript work while correctly staling
changed stages and their fixed transitive dependents.
Private configuration, prompt, model, executable, and module contents loaded
inside an external tool remain outside Narratio's observable fingerprint
boundary unless an existing integration contract exposes their identity.
Changing such private inputs continues to require explicit force, and the
documentation must not claim otherwise.
## Configuration Inspection
Add a `config` command family that performs composition without running a
session:
```bash
narratio config validate [--config <pipeline.yml>] [--profile <name>]
narratio config show [--config <pipeline.yml>] [--profile <name>]
narratio config sources [--config <pipeline.yml>] [--profile <name>]
narratio config diff <left-profile> <right-profile> [--config <pipeline.yml>]
```
Required behavior:
- `validate` composes imports and the selected profile, resolves campaign data
when required for party-driven families, expands artifacts and publish rules,
applies defaults, and runs strict validation without creating session or run
state.
- `show` emits deterministic normalized effective YAML, including generated
concrete artifacts, without raw credentials or environment values.
- `sources` reports root, import, profile, campaign, and party provenance and
identifies which source owns effective configuration paths.
- `diff` compares normalized effective configurations rather than raw files
and clearly identifies additions, removals, and value changes.
- output order is deterministic and suitable for review, but output formatting
is not a substitute for the effective configuration digest.
If family expansion requires a campaign, these commands accept the existing
campaign selection mechanisms or report clearly that pipeline-only validation
cannot complete the party-dependent portion. They must not silently choose an
unintended campaign.
## Required Application Changes
The implemented feature will require coordinated changes across existing
owners rather than a parallel configuration subsystem:
- `internal/config`: presence-aware YAML composition, import confinement,
profile overlays, provenance, effective digesting, party parsing and
validation, players projection, family expansion, defaults, and final strict
validation;
- `internal/app`: `--profile` plumbing through every configuration-consuming
command, `config` inspection commands, campaign-aware inspection, profile
reporting, and one shared loading path;
- `internal/artifactpolicy` and `internal/artifacts`: generated concrete source
registration, family-aware selection metadata, derived-player identity, and
reuse of the existing configured-artifact grammar and path policy;
- `internal/stage`: consumption only of expanded concrete artifacts, party and
derived-player preparation, and stage-specific semantic resume evidence;
- `internal/manifest`: bounded profile/effective-config provenance and any
versioned stage semantic fingerprints required for safe reuse;
- publish planning: expansion of family publish declarations into existing
concrete rules before ordinary validation and execution;
- Notarius composition: continue passing prepared `party` and `players`
sources, with canonical mode sourcing both from the party-owned contract;
and
- Scriptorium composition: receive only resolved concrete artifacts and
string/bool variables under the existing adapter contract.
There must be one production configuration-loading path. Remote session
loading, restore, status, planning, helper commands, and execution must not
independently reimplement import, profile, campaign, or family selection.
## Documentation And Examples
When behavior is implemented, update the canonical owners in the same change:
- [`docs/config.md`](../config.md): import, merge, profile, party selection,
family fields, defaults, and validation contracts;
- [`docs/cli.md`](../cli.md): `--profile` and `config` command syntax and output
behavior;
- [`docs/operations.md`](../operations.md): recommended production/testing
bundle layout, migration, inspection, and profile-switching workflow;
- [`docs/policy/architecture.md`](../policy/architecture.md): only the durable
composition, campaign-data ownership, and semantic-resume invariants;
- `docs/integrations/`: the canonical party and derived players YAML contracts;
- [`docs/internal/overview.md`](../internal/overview.md) and focused internal
configuration, manifest, analyze, prepare, and publish documents: implemented
ownership and mechanics; and
- `examples/`: a validated split configuration bundle, production/testing
overlays, canonical party roster with aliases and multiclass data, at least
two party-driven artifact families, and representative family publishing.
Do not copy complete example bundles into reference prose. Maintained examples
must be secret-free and loaded by the repository's example validation test.
## Testing Expectations
Testing should protect the public contracts and high-risk composition and
state-reuse behavior without coupling to private merge helpers. Required
confidence includes:
- monolithic configuration compatibility;
- recursive mapping composition and all duplicate/type/list conflict classes;
- import confinement, missing files, non-regular files, unsupported nesting,
source-aware diagnostics, deterministic ordering, and duplicate YAML keys;
- explicit zero-value profile overrides, list replacement, keyed-map additions,
unknown profiles, invalid defaults, and absence of profile stacking;
- canonical party schema validation, aliases, alias/name ambiguity, multiclass
summaries, stable IDs, malformed data, and legacy/canonical mode separation;
- deterministic players projection and prepared-input provenance;
- deterministic multi-family expansion, generated identities and paths,
variable binding, same-member dependencies, collisions, additions,
removals, and family selection;
- expansion into ordinary analyze planning, fingerprints, manifests, partial
selection, failure handling, and publish rules;
- model/profile changes staling only stages whose semantic configuration
changed while unaffected transcript work remains reusable;
- representative CLI validation, show, sources, diff, plan, and run behavior;
and
- maintained monolithic, imported, production, and testing examples loading
and validating offline.
Use package-level behavioral tests through the narrowest stable owner. A small
number of assembled workflow tests should prove that composed profiles and
party-driven families reach the existing pipeline correctly; higher-level
tests should not repeat every parser and validation case.
## Compatibility And Migration
- Existing monolithic pipeline files remain supported.
- Omitting `--profile` retains existing behavior for configurations without
profiles and selects the declared production default for composed bundles.
- Existing explicit concrete Scriptorium artifacts and publish rules remain
valid and may coexist with non-colliding families.
- Legacy opaque party plus players files remain temporarily usable only through
the isolated compatibility mode and cannot power party-driven families.
- Migration consists of converting the campaign party document to
`narratio.party.v1`, removing the separate players file, and then enabling
party-driven families.
- Configuration inspection must make the migration result reviewable before a
session run.
- Compatibility code and documentation must clearly identify the eventual
removal boundary; new features must target canonical party mode rather than
extend legacy semantics.
## Non-Goals And Deferred Work
This feature does not introduce:
- automatic loading of every file in `conf.d`;
- recursive imports, imports in profile overlays, or profile inheritance;
- multiple simultaneously active profiles;
- environment-selected profiles;
- generic YAML patch/delete operations;
- arbitrary string interpolation, expressions, scripting, or loops;
- artifact expansion from arbitrary files, Notarius outputs, or runtime data;
- configurable pipeline stages or a workflow DAG;
- runtime wildcard artifact or publish sources;
- automatic hashing of private external-tool configuration, prompts, modules,
models, or executables; or
- immediate removal of the isolated legacy party/player compatibility path.
Named reusable parameter sets beyond the canonical party roster may be
considered later if a second non-party use case demonstrates a need. Additional
party-member types, campaign-domain fields, or family iteration sources should
be added only through a versioned schema decision rather than an untyped
extension map.
## Target End State
The feature is complete when:
- operators can organize a pipeline as a root file plus explicit additive
imports with deterministic, source-aware conflict errors;
- one production-default or explicitly selected testing profile applies a
deliberate override without duplicating stable configuration;
- every command uses the same resolved profile and reports its identity;
- a campaign owns one strict, versioned party roster containing stable
character IDs, players, character names, optional alias lists, and one or
more classes;
- Narratio derives the existing players input from that roster and supplies the
canonical campaign context to Notarius;
- multiple Scriptorium artifact families expand once per campaign character
into ordinary concrete artifacts, dependencies, variables, output paths, and
optional publish rules;
- adding, changing, or removing a character produces deterministic and
manifest-correct artifact reconciliation;
- profile model changes cannot silently reuse semantically incompatible stage
outputs and do not unnecessarily invalidate unaffected transcript work;
- operators can validate, display, trace, and compare effective configuration
without executing a session;
- existing monolithic and concrete-artifact configurations remain compatible;
and
- current user, operator, integration, architecture, internal, and example
documentation accurately owns the implemented contracts.