# Pipeline Configuration Ergonomics ## Status Accepted target state. This document owns the intended feature scope, user intent, policy choices, and target end state until the behavior is implemented. ## Goal Make Narratio's pipeline configuration easier to organize, vary, inspect, and reuse without weakening its strict configuration contract or turning it into a general-purpose configuration or workflow language. The feature should let operators: - split a large pipeline configuration into explicitly imported partial files; - maintain one production-default pipeline plus a selectable testing profile; - keep stable settings separate from frequently edited models and artifacts; - define a character-oriented artifact workflow once and expand it for every character in the selected campaign; - reuse one campaign-owned party roster across multiple artifact families and the Notarius integration; and - inspect and compare the fully resolved configuration before running a session. All composition and expansion must produce the same concrete, validated `PipelineConfig` and concrete configured-artifact map consumed by the existing stage, planning, manifest, artifact, and adapter boundaries. ## User Intent Pipeline configuration currently combines long-lived platform and transcript settings with model choices, prompt-backed artifact definitions, and publish rules that change frequently during development. Keeping all of those concerns in one file makes edits noisy and makes production/testing variation difficult to review. Character-specific artifacts introduce a second kind of repetition. Several artifact families, including character meta-analysis and character-specific item tracking, should execute once per campaign character. Their prompt, inputs, dependencies, and output shape are shared, while player, character, class, alias, and output identity vary by party member. Those campaign facts do not belong in a global pipeline definition. The desired model is therefore: ```text pipeline.yml + explicit imports + one selected profile │ campaign.yml + party.yml ─┤ ▼ resolved effective configuration │ expand party-driven families │ defaults and strict validation ▼ existing concrete pipeline and artifact model ``` ## Architectural Constraints The feature must preserve the configuration and orchestration policies in [`docs/policy/architecture.md`](../policy/architecture.md): - YAML remains strict and rejects unknown fields. - Defaults remain centralized and are applied exactly once. - Composition and expansion are deterministic and testable. - Session templating remains narrow; this feature does not introduce arbitrary expressions, scripting, or a general template language. - The canonical stage sequence and fixed invalidation relation remain application-owned and non-configurable. - Stages receive resolved Narratio-owned configuration and do not implement import, profile, or family semantics independently. - External tools continue to own their private runtime configuration and defaults. - Raw secrets remain indirect and are never introduced into composed YAML, effective-configuration output, manifests, or diagnostics. Configuration composition belongs to `internal/config`. CLI selection and production loading belong to `internal/app`. Party parsing must be a Narratio-owned campaign-input contract. Artifact-family expansion must finish before existing configured-artifact validation, catalog construction, planning, reconciliation, or execution begins. ## Configuration Imports ### Root document Keep `pipeline.yml` as the selected pipeline entry point. Add an optional top-level composition envelope with explicitly ordered imports: ```yaml composition: imports: - conf.d/platform.yml - conf.d/transcripts.yml - conf.d/extraction.yml - conf.d/artifacts.yml - conf.d/publish.yml default_profile: production profiles: production: overlay: profiles/production.yml testing: overlay: profiles/testing.yml ``` An existing monolithic `pipeline.yml` without `composition` remains valid and retains its current behavior. The root document may contain ordinary pipeline fields alongside `composition`, allowing incremental migration. `imports` is the public term. Imported files are partial pipeline configuration documents rather than independently runnable pipelines. ### Import boundaries - Only the root pipeline document may declare `composition`, imports, or profiles. Imported and overlay documents cannot import other documents. - Every import is explicit. Narratio does not automatically scan `conf.d` or interpret unlisted files. - Import paths are relative to the root pipeline file's directory, must remain confined beneath that directory, and must resolve to supported regular YAML files. Absolute paths, traversal, non-regular files, and cycles are rejected. - All ordinary relative pipeline paths retain one stable base: the root pipeline file's directory. Moving a field between imported files must not silently change the meaning of its relative path. - The declared import order is retained for diagnostics and provenance, but it is not an implicit precedence mechanism between base imports. - Composition metadata is removed before the effective mapping is decoded as strict `PipelineConfig` data. ### Additive merge contract Base configuration from the root document and its imports is additive: - mappings merge recursively when their child fields are disjoint; - two files may therefore contribute different entries beneath a shared map such as `scriptorium.artifacts`; - defining the same final field or keyed entry in more than one base source is an error, even when the values are identical; - lists are atomic values and cannot be contributed more than once at the same base configuration path; - mapping/scalar, mapping/list, and other incompatible type collisions are errors; and - duplicate YAML keys within one source remain errors. Errors must identify the complete configuration path and every source file that claims it. This contract prevents an import reorder or newly added file from silently changing production behavior. Composition must operate on a presence-aware YAML representation before Go struct decoding. Merging already decoded structs is not sufficient because it cannot reliably distinguish omission from explicitly configured `false`, zero, an empty collection, or another meaningful zero value. ## Named Profiles ### Selection and precedence Exactly one named profile may be active for an invocation. A composition root may define `default_profile`; the maintained production-shaped configuration uses `production`. Add `--profile ` to the common configuration flags for every command that loads pipeline configuration: ```bash narratio run SESSION narratio run SESSION --profile testing narratio session plan SESSION --profile testing narratio regenerate-artifacts SESSION --profile testing ``` The explicit CLI value overrides `default_profile`. An unknown profile, a profile selection against a configuration that defines no such profile, an empty profile name, or a default naming an undefined profile is an error before stage composition. Do not add an environment-variable selector in this feature; profile selection should remain visible in the command or root configuration. Only one profile can be selected. Profiles cannot extend other profiles, and callers cannot stack several profiles. A single overlay provides the required production/testing variation without creating user-programmable precedence. ### Overlay semantics The selected profile is the sole intentional override layer: - mappings recursively merge with the assembled base mapping; - scalar and boolean leaves replace base leaves; - lists replace the complete base list rather than concatenate implicitly; - keyed maps such as configured artifacts merge by key, allowing a testing profile to add experimental artifacts; - an existing artifact can be disabled explicitly with `enabled: false`; - an overlay cannot change a value's YAML kind incompatibly; and - the fully overlaid result undergoes the same strict decode, defaults, path resolution, expansion, and validation as a monolithic configuration. Generic deletion syntax is outside this feature. Shared definitions should live in base imports, while production-only and testing-only definitions live in their corresponding overlays. Existing enable/disable fields and complete list replacement cover the required cases without a YAML patch language. ### Effective identity and provenance Every loaded configuration must retain bounded, non-secret provenance: - the root pipeline path; - the selected profile name, including whether it came from the default or CLI; - the ordered imported and overlay source paths; - a deterministic digest of the normalized effective pipeline configuration; and - enough source ownership information to explain composition errors and effective values. Run reporting and invocation state should identify the selected profile and effective configuration digest. The profile name is provenance, not a blanket cache key: changing profiles must invalidate only work whose result-affecting effective configuration changed. ## Canonical Campaign Party Contract ### Ownership and schema Make `party.yml` a strict, versioned Narratio campaign-input contract. A canonical file has this shape: ```yaml schema_version: narratio.party.v1 characters: arannis: player: name: Eric character: name: Arannis alias: - Ari - The Grey Owl classes: - name: wizard level: 8 brenna: player: name: Jane character: name: Brenna classes: - name: paladin level: 6 - name: warlock level: 2 ``` The contract has these semantics: - `schema_version` is required and must equal `narratio.party.v1`. - `characters` is a non-empty mapping. - Each mapping key is the stable `character_id` and must satisfy the existing configured-artifact key grammar. - `player.name` and `character.name` are required, non-empty display strings. - `character.alias` is optional and accepts a list of zero or more non-empty alias strings. The singular field spelling `alias` is intentional. - Leading or trailing whitespace and control characters in names, aliases, and class names are rejected rather than silently normalized. - A character alias cannot equal that character's primary name, another alias, or another character's primary name or alias under case-insensitive comparison. This keeps party-member grounding unambiguous. Player names may repeat because one player may control more than one character. - `character.classes` is required and non-empty. Every entry has a required free-form `name` and an optional positive integer `level`. - Class names are not restricted to a Narratio-owned D&D enumeration. - Duplicate class names for one character are rejected case-insensitively. - Class order and alias order are preserved. - Narratio derives a deterministic class summary such as `paladin 6 / warlock 2` and an alias summary that joins declared aliases with `, ` in their declared order. - Unknown fields and trailing YAML documents are rejected. The stable `character_id`, rather than a display name, owns generated artifact identity. Changing a display name, alias, class, or level retains that identity and changes the relevant semantic inputs. Changing the mapping key is an intentional remove-and-add operation. ### Load and prepare behavior The selected campaign's effective `party_file` resolves relative to `campaign.yml`. Narratio must load and validate canonical party data during combined pipeline/campaign resolution so `session plan`, configuration inspection, and artifact-family expansion see the same roster before stage execution. `prepare` continues to materialize the canonical document as `inputs/party.yml`, record its source and content identity, and expose it as `narratio.input.party`. The same canonical file is passed unchanged to Notarius's `party` reference slot, which currently accepts YAML reference material without imposing a competing roster schema. ### Players projection and compatibility The canonical party document becomes the eventual single source of truth for player-to-character relationships. Narratio derives a deterministic, documented players-only YAML projection during prepare, materializes it at `inputs/players.yml`, records its content identity, and exposes it through the existing `narratio.input.players` source. This preserves the separate Notarius `players` reference and existing Scriptorium source without requiring every campaign to maintain duplicate facts. The projection uses this versioned shape, with entries sorted by stable character ID: ```yaml schema_version: narratio.players.v1 players: - name: Eric character: id: arannis name: Arannis alias: - Ari - The Grey Owl ``` There is one projection entry per character rather than one grouped entry per display player name. This permits one player to control several characters and does not conflate distinct players who happen to share a display name. Backward compatibility must be narrow and removable: - an unversioned legacy party file continues to be treated as opaque reference material when the existing `players_file` is also configured; - legacy mode preserves the current prepared `party` and `players` behavior but cannot drive artifact-family expansion; - configuring a party-driven artifact family with a legacy roster produces a clear migration error; - canonical `narratio.party.v1` mode derives players and rejects a separate `players_file`, preventing contradictory authorities; - parsing, validation, documentation, and tests for legacy mode live behind a clearly identified compatibility boundary with comments stating that it is intended for removal after migration; and - the maintained examples migrate to canonical mode and demonstrate no separate players file. The derived players projection is a durable integration contract and must be documented under `docs/integrations/` rather than left as incidental generated YAML. ## Party-Driven Artifact Families ### Configuration model Add `scriptorium.artifact_families`, separate from the existing concrete `scriptorium.artifacts` map: ```yaml scriptorium: artifact_families: character_meta: enabled: true for_each: party.characters prompt_id: dnd.character_meta profile_id: production output_path_pattern: artifacts/characters/{character_id}/meta.md inputs: transcript: source: narratio.transcript.final_trimmed required: true member_vars: player_name: player.name character_name: character.name character_class: character.class_summary character_aliases: character.alias_summary character_items: enabled: true for_each: party.characters prompt_id: dnd.character_items profile_id: production output_path_pattern: artifacts/characters/{character_id}/items.md inputs: transcript: source: narratio.transcript.final_trimmed required: true item_occurrences: source: narratio.extraction.item_occurrences required: true member_vars: character_name: character.name ``` Families support the ordinary result-affecting fields shared by concrete Scriptorium artifacts: enablement, prompt and profile IDs, timeout, render-debug policy, ordinary dependencies, inputs, and static variables. `output_path_pattern` replaces concrete `output_path` at the family level. `for_each` is not an expression language. This feature accepts only the exact source `party.characters`. `member_vars` maps a Scriptorium variable name to one of a closed set of canonical values: - `character_id`; - `player.name`; - `character.name`; - `character.class_summary`; and - `character.alias_summary`. Static `vars` and resolved member variables merge deterministically. Duplicate variable names across the two maps are rejected rather than assigned implicit precedence. Narratio's existing reserved sticky-session variable remains application-owned and is applied after expansion under its current rules. ### Narrow substitution `{character_id}` is the only family substitution supported in this feature. It is required in `output_path_pattern` and may appear exactly where documented for family-owned publish destinations. Unknown, repeated in an invalid position, unresolved, or malformed substitutions are errors. Names, aliases, classes, environment values, and arbitrary YAML paths cannot be interpolated into configuration strings. ### Concrete expansion For each family and party character, resolution creates one ordinary concrete artifact key by joining the normalized family key, an underscore, and the stable character ID: ```text character_meta_arannis character_items_arannis ``` Expansion must be sorted, deterministic, and completed before existing configured-artifact defaults and validation. Every generated artifact then uses the existing catalog, dependency planner, fingerprinting, manifest, reconciliation, execution, materialization, and publish boundaries. Validation rejects: - invalid family keys or character IDs; - generated keys that fail the existing configured-artifact grammar; - collisions between generated artifacts, explicit concrete artifacts, or family names; - duplicate or escaping output paths; - a family with no canonical party source; - invalid member-variable selectors; - a selected profile that leaves an executable family incomplete; and - any expanded artifact that fails ordinary concrete-artifact validation. An added party character creates missing concrete artifacts. Removing a character removes those artifacts from effective configuration so existing analyze reconciliation classifies their old records as removed. A change to a member field changes only fingerprints that actually consume its resolved value or the party document as an input. ### Dependencies between families Ordinary `depends_on` continues to name shared concrete artifacts such as `session_recap`. Add a typed `member_dependencies` list for corresponding members of another family: ```yaml character_items: member_dependencies: - character_meta ``` For `arannis`, this expands to a dependency on `character_meta_arannis`. The referenced family must use the same party source and generate the same character ID. Cycles and missing members are rejected by the existing concrete dependency validation after expansion. When a family input consumes the corresponding member output, provide a narrow pre-expansion member-artifact source form owned by configuration resolution. The syntax is: ```yaml inputs: prior_meta: source: narratio.member_artifact.character_meta required: true ``` For the `arannis` member, this resolves to `narratio.artifact.character_meta_arannis` before ordinary runtime validation. The referenced family must also appear in `member_dependencies`. The `narratio.member_artifact.*` form must not survive into stage or adapter configuration as a new runtime artifact kind. ### Selection and reporting Extend artifact selection so an exact family key selects all of that family's concrete members. An exact generated key selects only that member. Explicitly selecting a disabled family follows the existing rule for explicitly selected disabled concrete artifacts: all generated targets must still have valid executable fields. Mixed family and concrete selections are normalized and deduplicated after expansion. Planning, run summaries, artifact listing, and effective-configuration output must show both family origin and concrete artifact identity. Execution remains entirely concrete; a family is not a dynamic pipeline stage or runtime loop. ### Family publish rules Allow a family to declare one typed publish policy so operators need not repeat one publish output per character: ```yaml publish: enabled: true required: false dest_pattern: artifacts/characters/{character_id}/meta.md ``` Resolution expands this into ordinary concrete publish rules. When `dest_pattern` is omitted, the generated artifact's output path is used where the existing publish contract permits derivation. Conflicts with explicit publish rules, duplicate destinations, unsafe paths, and generated sources that are unavailable are rejected by the existing publish policy after expansion. Family publish behavior must not introduce wildcard source matching at runtime. ## Profile-Safe Resume And Invalidation Named profiles are safe only if result-affecting model and configuration changes cannot silently reuse incompatible successful stage results. Artifact analysis already has a versioned semantic fingerprint that includes effective prompt/profile identifiers, variables, dependencies, inputs, and output identity. The broader pipeline requires the same principle at every applicable stage boundary. Audit each stage's resume contract and add a versioned semantic configuration fingerprint wherever current evidence does not already cover all Narratio-observable result-affecting settings. In particular, switching an Audita model through a profile must not reuse polish output produced by the testing model. Conversely, changing only an operational timeout, binary path, workspace path, diagnostic location, or profile name must not invalidate byte-equivalent semantic work unless that value genuinely affects the stage's canonical result. The selected profile name and complete effective-config digest are retained as provenance. Resume decisions use stage-specific effective semantics rather than the profile name as a blanket invalidator. This permits production and testing profiles to share unaffected transcript work while correctly staling changed stages and their fixed transitive dependents. Private configuration, prompt, model, executable, and module contents loaded inside an external tool remain outside Narratio's observable fingerprint boundary unless an existing integration contract exposes their identity. Changing such private inputs continues to require explicit force, and the documentation must not claim otherwise. ## Configuration Inspection Add a `config` command family that performs composition without running a session: ```bash narratio config validate [--config ] [--profile ] narratio config show [--config ] [--profile ] narratio config sources [--config ] [--profile ] narratio config diff [--config ] ``` Required behavior: - `validate` composes imports and the selected profile, resolves campaign data when required for party-driven families, expands artifacts and publish rules, applies defaults, and runs strict validation without creating session or run state. - `show` emits deterministic normalized effective YAML, including generated concrete artifacts, without raw credentials or environment values. - `sources` reports root, import, profile, campaign, and party provenance and identifies which source owns effective configuration paths. - `diff` compares normalized effective configurations rather than raw files and clearly identifies additions, removals, and value changes. - output order is deterministic and suitable for review, but output formatting is not a substitute for the effective configuration digest. If family expansion requires a campaign, these commands accept the existing campaign selection mechanisms or report clearly that pipeline-only validation cannot complete the party-dependent portion. They must not silently choose an unintended campaign. ## Required Application Changes The implemented feature will require coordinated changes across existing owners rather than a parallel configuration subsystem: - `internal/config`: presence-aware YAML composition, import confinement, profile overlays, provenance, effective digesting, party parsing and validation, players projection, family expansion, defaults, and final strict validation; - `internal/app`: `--profile` plumbing through every configuration-consuming command, `config` inspection commands, campaign-aware inspection, profile reporting, and one shared loading path; - `internal/artifactpolicy` and `internal/artifacts`: generated concrete source registration, family-aware selection metadata, derived-player identity, and reuse of the existing configured-artifact grammar and path policy; - `internal/stage`: consumption only of expanded concrete artifacts, party and derived-player preparation, and stage-specific semantic resume evidence; - `internal/manifest`: bounded profile/effective-config provenance and any versioned stage semantic fingerprints required for safe reuse; - publish planning: expansion of family publish declarations into existing concrete rules before ordinary validation and execution; - Notarius composition: continue passing prepared `party` and `players` sources, with canonical mode sourcing both from the party-owned contract; and - Scriptorium composition: receive only resolved concrete artifacts and string/bool variables under the existing adapter contract. There must be one production configuration-loading path. Remote session loading, restore, status, planning, helper commands, and execution must not independently reimplement import, profile, campaign, or family selection. ## Documentation And Examples When behavior is implemented, update the canonical owners in the same change: - [`docs/config.md`](../config.md): import, merge, profile, party selection, family fields, defaults, and validation contracts; - [`docs/cli.md`](../cli.md): `--profile` and `config` command syntax and output behavior; - [`docs/operations.md`](../operations.md): recommended production/testing bundle layout, migration, inspection, and profile-switching workflow; - [`docs/policy/architecture.md`](../policy/architecture.md): only the durable composition, campaign-data ownership, and semantic-resume invariants; - `docs/integrations/`: the canonical party and derived players YAML contracts; - [`docs/internal/overview.md`](../internal/overview.md) and focused internal configuration, manifest, analyze, prepare, and publish documents: implemented ownership and mechanics; and - `examples/`: a validated split configuration bundle, production/testing overlays, canonical party roster with aliases and multiclass data, at least two party-driven artifact families, and representative family publishing. Do not copy complete example bundles into reference prose. Maintained examples must be secret-free and loaded by the repository's example validation test. ## Testing Expectations Testing should protect the public contracts and high-risk composition and state-reuse behavior without coupling to private merge helpers. Required confidence includes: - monolithic configuration compatibility; - recursive mapping composition and all duplicate/type/list conflict classes; - import confinement, missing files, non-regular files, unsupported nesting, source-aware diagnostics, deterministic ordering, and duplicate YAML keys; - explicit zero-value profile overrides, list replacement, keyed-map additions, unknown profiles, invalid defaults, and absence of profile stacking; - canonical party schema validation, aliases, alias/name ambiguity, multiclass summaries, stable IDs, malformed data, and legacy/canonical mode separation; - deterministic players projection and prepared-input provenance; - deterministic multi-family expansion, generated identities and paths, variable binding, same-member dependencies, collisions, additions, removals, and family selection; - expansion into ordinary analyze planning, fingerprints, manifests, partial selection, failure handling, and publish rules; - model/profile changes staling only stages whose semantic configuration changed while unaffected transcript work remains reusable; - representative CLI validation, show, sources, diff, plan, and run behavior; and - maintained monolithic, imported, production, and testing examples loading and validating offline. Use package-level behavioral tests through the narrowest stable owner. A small number of assembled workflow tests should prove that composed profiles and party-driven families reach the existing pipeline correctly; higher-level tests should not repeat every parser and validation case. ## Compatibility And Migration - Existing monolithic pipeline files remain supported. - Omitting `--profile` retains existing behavior for configurations without profiles and selects the declared production default for composed bundles. - Existing explicit concrete Scriptorium artifacts and publish rules remain valid and may coexist with non-colliding families. - Legacy opaque party plus players files remain temporarily usable only through the isolated compatibility mode and cannot power party-driven families. - Migration consists of converting the campaign party document to `narratio.party.v1`, removing the separate players file, and then enabling party-driven families. - Configuration inspection must make the migration result reviewable before a session run. - Compatibility code and documentation must clearly identify the eventual removal boundary; new features must target canonical party mode rather than extend legacy semantics. ## Non-Goals And Deferred Work This feature does not introduce: - automatic loading of every file in `conf.d`; - recursive imports, imports in profile overlays, or profile inheritance; - multiple simultaneously active profiles; - environment-selected profiles; - generic YAML patch/delete operations; - arbitrary string interpolation, expressions, scripting, or loops; - artifact expansion from arbitrary files, Notarius outputs, or runtime data; - configurable pipeline stages or a workflow DAG; - runtime wildcard artifact or publish sources; - automatic hashing of private external-tool configuration, prompts, modules, models, or executables; or - immediate removal of the isolated legacy party/player compatibility path. Named reusable parameter sets beyond the canonical party roster may be considered later if a second non-party use case demonstrates a need. Additional party-member types, campaign-domain fields, or family iteration sources should be added only through a versioned schema decision rather than an untyped extension map. ## Target End State The feature is complete when: - operators can organize a pipeline as a root file plus explicit additive imports with deterministic, source-aware conflict errors; - one production-default or explicitly selected testing profile applies a deliberate override without duplicating stable configuration; - every command uses the same resolved profile and reports its identity; - a campaign owns one strict, versioned party roster containing stable character IDs, players, character names, optional alias lists, and one or more classes; - Narratio derives the existing players input from that roster and supplies the canonical campaign context to Notarius; - multiple Scriptorium artifact families expand once per campaign character into ordinary concrete artifacts, dependencies, variables, output paths, and optional publish rules; - adding, changing, or removing a character produces deterministic and manifest-correct artifact reconciliation; - profile model changes cannot silently reuse semantically incompatible stage outputs and do not unnecessarily invalidate unaffected transcript work; - operators can validate, display, trace, and compare effective configuration without executing a session; - existing monolithic and concrete-artifact configurations remain compatible; and - current user, operator, integration, architecture, internal, and example documentation accurately owns the implemented contracts.