narratio/docs/roadmap/pipeline-configuration-ergonomics.md

Pipeline Configuration Ergonomics

Status

Accepted target state. This document owns the intended feature scope, user intent, policy choices, and target end state until the behavior is implemented.

Goal

Make Narratio's pipeline configuration easier to organize, vary, inspect, and reuse without weakening its strict configuration contract or turning it into a general-purpose configuration or workflow language.

The feature should let operators:

• split a large pipeline configuration into explicitly imported partial files;
• maintain one production-default pipeline plus a selectable testing profile;
• keep stable settings separate from frequently edited models and artifacts;
• define a character-oriented artifact workflow once and expand it for every character in the selected campaign;
• reuse one campaign-owned party roster across multiple artifact families and the Notarius integration; and
• inspect and compare the fully resolved configuration before running a session.

All composition and expansion must produce the same concrete, validated PipelineConfig and concrete configured-artifact map consumed by the existing stage, planning, manifest, artifact, and adapter boundaries.

User Intent

Pipeline configuration currently combines long-lived platform and transcript settings with model choices, prompt-backed artifact definitions, and publish rules that change frequently during development. Keeping all of those concerns in one file makes edits noisy and makes production/testing variation difficult to review.

Character-specific artifacts introduce a second kind of repetition. Several artifact families, including character meta-analysis and character-specific item tracking, should execute once per campaign character. Their prompt, inputs, dependencies, and output shape are shared, while player, character, class, alias, and output identity vary by party member. Those campaign facts do not belong in a global pipeline definition.

The desired model is therefore:

pipeline.yml + explicit imports + one selected profile
                          │
campaign.yml + party.yml ─┤
                          ▼
             resolved effective configuration
                          │
             expand party-driven families
                          │
              defaults and strict validation
                          ▼
        existing concrete pipeline and artifact model

Architectural Constraints

The feature must preserve the configuration and orchestration policies in docs/policy/architecture.md:

• YAML remains strict and rejects unknown fields.
• Defaults remain centralized and are applied exactly once.
• Composition and expansion are deterministic and testable.
• Session templating remains narrow; this feature does not introduce arbitrary expressions, scripting, or a general template language.
• The canonical stage sequence and fixed invalidation relation remain application-owned and non-configurable.
• Stages receive resolved Narratio-owned configuration and do not implement import, profile, or family semantics independently.
• External tools continue to own their private runtime configuration and defaults.
• Raw secrets remain indirect and are never introduced into composed YAML, effective-configuration output, manifests, or diagnostics.

Configuration composition belongs to internal/config. CLI selection and production loading belong to internal/app. Party parsing must be a Narratio-owned campaign-input contract. Artifact-family expansion must finish before existing configured-artifact validation, catalog construction, planning, reconciliation, or execution begins.

Configuration Imports

Root document

Keep pipeline.yml as the selected pipeline entry point. Add an optional top-level composition envelope with explicitly ordered imports:

composition:
  imports:
    - conf.d/platform.yml
    - conf.d/transcripts.yml
    - conf.d/extraction.yml
    - conf.d/artifacts.yml
    - conf.d/publish.yml

  default_profile: production

  profiles:
    production:
      overlay: profiles/production.yml
    testing:
      overlay: profiles/testing.yml

An existing monolithic pipeline.yml without composition remains valid and retains its current behavior. The root document may contain ordinary pipeline fields alongside composition, allowing incremental migration.

imports is the public term. Imported files are partial pipeline configuration documents rather than independently runnable pipelines.

Import boundaries

• Only the root pipeline document may declare composition, imports, or profiles. Imported and overlay documents cannot import other documents.
• Every import is explicit. Narratio does not automatically scan conf.d or interpret unlisted files.
• Import paths are relative to the root pipeline file's directory, must remain confined beneath that directory, and must resolve to supported regular YAML files. Absolute paths, traversal, non-regular files, and cycles are rejected.
• All ordinary relative pipeline paths retain one stable base: the root pipeline file's directory. Moving a field between imported files must not silently change the meaning of its relative path.
• The declared import order is retained for diagnostics and provenance, but it is not an implicit precedence mechanism between base imports.
• Composition metadata is removed before the effective mapping is decoded as strict PipelineConfig data.

Additive merge contract

Base configuration from the root document and its imports is additive:

• mappings merge recursively when their child fields are disjoint;
• two files may therefore contribute different entries beneath a shared map such as scriptorium.artifacts;
• defining the same final field or keyed entry in more than one base source is an error, even when the values are identical;
• lists are atomic values and cannot be contributed more than once at the same base configuration path;
• mapping/scalar, mapping/list, and other incompatible type collisions are errors; and
• duplicate YAML keys within one source remain errors.

Errors must identify the complete configuration path and every source file that claims it. This contract prevents an import reorder or newly added file from silently changing production behavior.

Composition must operate on a presence-aware YAML representation before Go struct decoding. Merging already decoded structs is not sufficient because it cannot reliably distinguish omission from explicitly configured false, zero, an empty collection, or another meaningful zero value.

Named Profiles

Selection and precedence

Exactly one named profile may be active for an invocation. A composition root may define default_profile; the maintained production-shaped configuration uses production. Add --profile <name> to the common configuration flags for every command that loads pipeline configuration:

narratio run SESSION
narratio run SESSION --profile testing
narratio session plan SESSION --profile testing
narratio regenerate-artifacts SESSION --profile testing

The explicit CLI value overrides default_profile. An unknown profile, a profile selection against a configuration that defines no such profile, an empty profile name, or a default naming an undefined profile is an error before stage composition. Do not add an environment-variable selector in this feature; profile selection should remain visible in the command or root configuration.

Only one profile can be selected. Profiles cannot extend other profiles, and callers cannot stack several profiles. A single overlay provides the required production/testing variation without creating user-programmable precedence.

Overlay semantics

The selected profile is the sole intentional override layer:

• mappings recursively merge with the assembled base mapping;
• scalar and boolean leaves replace base leaves;
• lists replace the complete base list rather than concatenate implicitly;
• keyed maps such as configured artifacts merge by key, allowing a testing profile to add experimental artifacts;
• an existing artifact can be disabled explicitly with enabled: false;
• an overlay cannot change a value's YAML kind incompatibly; and
• the fully overlaid result undergoes the same strict decode, defaults, path resolution, expansion, and validation as a monolithic configuration.

Generic deletion syntax is outside this feature. Shared definitions should live in base imports, while production-only and testing-only definitions live in their corresponding overlays. Existing enable/disable fields and complete list replacement cover the required cases without a YAML patch language.

Effective identity and provenance

Every loaded configuration must retain bounded, non-secret provenance:

• the root pipeline path;
• the selected profile name, including whether it came from the default or CLI;
• the ordered imported and overlay source paths;
• a deterministic digest of the normalized effective pipeline configuration; and
• enough source ownership information to explain composition errors and effective values.

Run reporting and invocation state should identify the selected profile and effective configuration digest. The profile name is provenance, not a blanket cache key: changing profiles must invalidate only work whose result-affecting effective configuration changed.

Canonical Campaign Party Contract

Ownership and schema

Make party.yml a strict, versioned Narratio campaign-input contract. A canonical file has this shape:

schema_version: narratio.party.v1

characters:
  arannis:
    player:
      name: Eric
    character:
      name: Arannis
      alias:
        - Ari
        - The Grey Owl
      classes:
        - name: wizard
          level: 8

  brenna:
    player:
      name: Jane
    character:
      name: Brenna
      classes:
        - name: paladin
          level: 6
        - name: warlock
          level: 2

The contract has these semantics:

• schema_version is required and must equal narratio.party.v1.
• characters is a non-empty mapping.
• Each mapping key is the stable character_id and must satisfy the existing configured-artifact key grammar.
• player.name and character.name are required, non-empty display strings.
• character.alias is optional and accepts a list of zero or more non-empty alias strings. The singular field spelling alias is intentional.
• Leading or trailing whitespace and control characters in names, aliases, and class names are rejected rather than silently normalized.
• A character alias cannot equal that character's primary name, another alias, or another character's primary name or alias under case-insensitive comparison. This keeps party-member grounding unambiguous. Player names may repeat because one player may control more than one character.
• character.classes is required and non-empty. Every entry has a required free-form name and an optional positive integer level.
• Class names are not restricted to a Narratio-owned D&D enumeration.
• Duplicate class names for one character are rejected case-insensitively.
• Class order and alias order are preserved.
• Narratio derives a deterministic class summary such as paladin 6 / warlock 2 and an alias summary that joins declared aliases with , in their declared order.
• Unknown fields and trailing YAML documents are rejected.

The stable character_id, rather than a display name, owns generated artifact identity. Changing a display name, alias, class, or level retains that identity and changes the relevant semantic inputs. Changing the mapping key is an intentional remove-and-add operation.

Load and prepare behavior

The selected campaign's effective party_file resolves relative to campaign.yml. Narratio must load and validate canonical party data during combined pipeline/campaign resolution so session plan, configuration inspection, and artifact-family expansion see the same roster before stage execution.

prepare continues to materialize the canonical document as inputs/party.yml, record its source and content identity, and expose it as narratio.input.party. The same canonical file is passed unchanged to Notarius's party reference slot, which currently accepts YAML reference material without imposing a competing roster schema.

Players projection and compatibility

The canonical party document becomes the eventual single source of truth for player-to-character relationships. Narratio derives a deterministic, documented players-only YAML projection during prepare, materializes it at inputs/players.yml, records its content identity, and exposes it through the existing narratio.input.players source. This preserves the separate Notarius players reference and existing Scriptorium source without requiring every campaign to maintain duplicate facts.

The projection uses this versioned shape, with entries sorted by stable character ID:

schema_version: narratio.players.v1
players:
  - name: Eric
    character:
      id: arannis
      name: Arannis
      alias:
        - Ari
        - The Grey Owl

There is one projection entry per character rather than one grouped entry per display player name. This permits one player to control several characters and does not conflate distinct players who happen to share a display name.

Backward compatibility must be narrow and removable:

• an unversioned legacy party file continues to be treated as opaque reference material when the existing players_file is also configured;
• legacy mode preserves the current prepared party and players behavior but cannot drive artifact-family expansion;
• configuring a party-driven artifact family with a legacy roster produces a clear migration error;
• canonical narratio.party.v1 mode derives players and rejects a separate players_file, preventing contradictory authorities;
• parsing, validation, documentation, and tests for legacy mode live behind a clearly identified compatibility boundary with comments stating that it is intended for removal after migration; and
• the maintained examples migrate to canonical mode and demonstrate no separate players file.

The derived players projection is a durable integration contract and must be documented under docs/integrations/ rather than left as incidental generated YAML.

Party-Driven Artifact Families

Configuration model

Add scriptorium.artifact_families, separate from the existing concrete scriptorium.artifacts map:

scriptorium:
  artifact_families:
    character_meta:
      enabled: true
      for_each: party.characters
      prompt_id: dnd.character_meta
      profile_id: production
      output_path_pattern: artifacts/characters/{character_id}/meta.md
      inputs:
        transcript:
          source: narratio.transcript.final_trimmed
          required: true
      member_vars:
        player_name: player.name
        character_name: character.name
        character_class: character.class_summary
        character_aliases: character.alias_summary

    character_items:
      enabled: true
      for_each: party.characters
      prompt_id: dnd.character_items
      profile_id: production
      output_path_pattern: artifacts/characters/{character_id}/items.md
      inputs:
        transcript:
          source: narratio.transcript.final_trimmed
          required: true
        item_occurrences:
          source: narratio.extraction.item_occurrences
          required: true
      member_vars:
        character_name: character.name

Families support the ordinary result-affecting fields shared by concrete Scriptorium artifacts: enablement, prompt and profile IDs, timeout, render-debug policy, ordinary dependencies, inputs, and static variables. output_path_pattern replaces concrete output_path at the family level.

for_each is not an expression language. This feature accepts only the exact source party.characters. member_vars maps a Scriptorium variable name to one of a closed set of canonical values:

• character_id;
• player.name;
• character.name;
• character.class_summary; and
• character.alias_summary.

Static vars and resolved member variables merge deterministically. Duplicate variable names across the two maps are rejected rather than assigned implicit precedence. Narratio's existing reserved sticky-session variable remains application-owned and is applied after expansion under its current rules.

Narrow substitution

{character_id} is the only family substitution supported in this feature. It is required in output_path_pattern and may appear exactly where documented for family-owned publish destinations. Unknown, repeated in an invalid position, unresolved, or malformed substitutions are errors. Names, aliases, classes, environment values, and arbitrary YAML paths cannot be interpolated into configuration strings.

Concrete expansion

For each family and party character, resolution creates one ordinary concrete artifact key by joining the normalized family key, an underscore, and the stable character ID:

character_meta_arannis
character_items_arannis

Expansion must be sorted, deterministic, and completed before existing configured-artifact defaults and validation. Every generated artifact then uses the existing catalog, dependency planner, fingerprinting, manifest, reconciliation, execution, materialization, and publish boundaries.

Validation rejects:

• invalid family keys or character IDs;
• generated keys that fail the existing configured-artifact grammar;
• collisions between generated artifacts, explicit concrete artifacts, or family names;
• duplicate or escaping output paths;
• a family with no canonical party source;
• invalid member-variable selectors;
• a selected profile that leaves an executable family incomplete; and
• any expanded artifact that fails ordinary concrete-artifact validation.

An added party character creates missing concrete artifacts. Removing a character removes those artifacts from effective configuration so existing analyze reconciliation classifies their old records as removed. A change to a member field changes only fingerprints that actually consume its resolved value or the party document as an input.

Dependencies between families

Ordinary depends_on continues to name shared concrete artifacts such as session_recap. Add a typed member_dependencies list for corresponding members of another family:

character_items:
  member_dependencies:
    - character_meta

For arannis, this expands to a dependency on character_meta_arannis. The referenced family must use the same party source and generate the same character ID. Cycles and missing members are rejected by the existing concrete dependency validation after expansion.

When a family input consumes the corresponding member output, provide a narrow pre-expansion member-artifact source form owned by configuration resolution. The syntax is:

inputs:
  prior_meta:
    source: narratio.member_artifact.character_meta
    required: true

For the arannis member, this resolves to narratio.artifact.character_meta_arannis before ordinary runtime validation. The referenced family must also appear in member_dependencies. The narratio.member_artifact.* form must not survive into stage or adapter configuration as a new runtime artifact kind.

Selection and reporting

Extend artifact selection so an exact family key selects all of that family's concrete members. An exact generated key selects only that member. Explicitly selecting a disabled family follows the existing rule for explicitly selected disabled concrete artifacts: all generated targets must still have valid executable fields. Mixed family and concrete selections are normalized and deduplicated after expansion.

Planning, run summaries, artifact listing, and effective-configuration output must show both family origin and concrete artifact identity. Execution remains entirely concrete; a family is not a dynamic pipeline stage or runtime loop.

Family publish rules

Allow a family to declare one typed publish policy so operators need not repeat one publish output per character:

publish:
  enabled: true
  required: false
  dest_pattern: artifacts/characters/{character_id}/meta.md

Resolution expands this into ordinary concrete publish rules. When dest_pattern is omitted, the generated artifact's output path is used where the existing publish contract permits derivation. Conflicts with explicit publish rules, duplicate destinations, unsafe paths, and generated sources that are unavailable are rejected by the existing publish policy after expansion. Family publish behavior must not introduce wildcard source matching at runtime.

Profile-Safe Resume And Invalidation

Named profiles are safe only if result-affecting model and configuration changes cannot silently reuse incompatible successful stage results. Artifact analysis already has a versioned semantic fingerprint that includes effective prompt/profile identifiers, variables, dependencies, inputs, and output identity. The broader pipeline requires the same principle at every applicable stage boundary.

Audit each stage's resume contract and add a versioned semantic configuration fingerprint wherever current evidence does not already cover all Narratio-observable result-affecting settings. In particular, switching an Audita model through a profile must not reuse polish output produced by the testing model. Conversely, changing only an operational timeout, binary path, workspace path, diagnostic location, or profile name must not invalidate byte-equivalent semantic work unless that value genuinely affects the stage's canonical result.

The selected profile name and complete effective-config digest are retained as provenance. Resume decisions use stage-specific effective semantics rather than the profile name as a blanket invalidator. This permits production and testing profiles to share unaffected transcript work while correctly staling changed stages and their fixed transitive dependents.

Private configuration, prompt, model, executable, and module contents loaded inside an external tool remain outside Narratio's observable fingerprint boundary unless an existing integration contract exposes their identity. Changing such private inputs continues to require explicit force, and the documentation must not claim otherwise.

Configuration Inspection

Add a config command family that performs composition without running a session:

narratio config validate [--config <pipeline.yml>] [--profile <name>]
narratio config show [--config <pipeline.yml>] [--profile <name>]
narratio config sources [--config <pipeline.yml>] [--profile <name>]
narratio config diff <left-profile> <right-profile> [--config <pipeline.yml>]

Required behavior:

• validate composes imports and the selected profile, resolves campaign data when required for party-driven families, expands artifacts and publish rules, applies defaults, and runs strict validation without creating session or run state.
• show emits deterministic normalized effective YAML, including generated concrete artifacts, without raw credentials or environment values.
• sources reports root, import, profile, campaign, and party provenance and identifies which source owns effective configuration paths.
• diff compares normalized effective configurations rather than raw files and clearly identifies additions, removals, and value changes.
• output order is deterministic and suitable for review, but output formatting is not a substitute for the effective configuration digest.

If family expansion requires a campaign, these commands accept the existing campaign selection mechanisms or report clearly that pipeline-only validation cannot complete the party-dependent portion. They must not silently choose an unintended campaign.

Required Application Changes

The implemented feature will require coordinated changes across existing owners rather than a parallel configuration subsystem:

• internal/config: presence-aware YAML composition, import confinement, profile overlays, provenance, effective digesting, party parsing and validation, players projection, family expansion, defaults, and final strict validation;
• internal/app: --profile plumbing through every configuration-consuming command, config inspection commands, campaign-aware inspection, profile reporting, and one shared loading path;
• internal/artifactpolicy and internal/artifacts: generated concrete source registration, family-aware selection metadata, derived-player identity, and reuse of the existing configured-artifact grammar and path policy;
• internal/stage: consumption only of expanded concrete artifacts, party and derived-player preparation, and stage-specific semantic resume evidence;
• internal/manifest: bounded profile/effective-config provenance and any versioned stage semantic fingerprints required for safe reuse;
• publish planning: expansion of family publish declarations into existing concrete rules before ordinary validation and execution;
• Notarius composition: continue passing prepared party and players sources, with canonical mode sourcing both from the party-owned contract; and
• Scriptorium composition: receive only resolved concrete artifacts and string/bool variables under the existing adapter contract.

There must be one production configuration-loading path. Remote session loading, restore, status, planning, helper commands, and execution must not independently reimplement import, profile, campaign, or family selection.

Documentation And Examples

When behavior is implemented, update the canonical owners in the same change:

• docs/config.md: import, merge, profile, party selection, family fields, defaults, and validation contracts;
• docs/cli.md: --profile and config command syntax and output behavior;
• docs/operations.md: recommended production/testing bundle layout, migration, inspection, and profile-switching workflow;
• docs/policy/architecture.md: only the durable composition, campaign-data ownership, and semantic-resume invariants;
• docs/integrations/: the canonical party and derived players YAML contracts;
• docs/internal/overview.md and focused internal configuration, manifest, analyze, prepare, and publish documents: implemented ownership and mechanics; and
• examples/: a validated split configuration bundle, production/testing overlays, canonical party roster with aliases and multiclass data, at least two party-driven artifact families, and representative family publishing.

Do not copy complete example bundles into reference prose. Maintained examples must be secret-free and loaded by the repository's example validation test.

Testing Expectations

Testing should protect the public contracts and high-risk composition and state-reuse behavior without coupling to private merge helpers. Required confidence includes:

• monolithic configuration compatibility;
• recursive mapping composition and all duplicate/type/list conflict classes;
• import confinement, missing files, non-regular files, unsupported nesting, source-aware diagnostics, deterministic ordering, and duplicate YAML keys;
• explicit zero-value profile overrides, list replacement, keyed-map additions, unknown profiles, invalid defaults, and absence of profile stacking;
• canonical party schema validation, aliases, alias/name ambiguity, multiclass summaries, stable IDs, malformed data, and legacy/canonical mode separation;
• deterministic players projection and prepared-input provenance;
• deterministic multi-family expansion, generated identities and paths, variable binding, same-member dependencies, collisions, additions, removals, and family selection;
• expansion into ordinary analyze planning, fingerprints, manifests, partial selection, failure handling, and publish rules;
• model/profile changes staling only stages whose semantic configuration changed while unaffected transcript work remains reusable;
• representative CLI validation, show, sources, diff, plan, and run behavior; and
• maintained monolithic, imported, production, and testing examples loading and validating offline.

Use package-level behavioral tests through the narrowest stable owner. A small number of assembled workflow tests should prove that composed profiles and party-driven families reach the existing pipeline correctly; higher-level tests should not repeat every parser and validation case.

Compatibility And Migration

• Existing monolithic pipeline files remain supported.
• Omitting --profile retains existing behavior for configurations without profiles and selects the declared production default for composed bundles.
• Existing explicit concrete Scriptorium artifacts and publish rules remain valid and may coexist with non-colliding families.
• Legacy opaque party plus players files remain temporarily usable only through the isolated compatibility mode and cannot power party-driven families.
• Migration consists of converting the campaign party document to narratio.party.v1, removing the separate players file, and then enabling party-driven families.
• Configuration inspection must make the migration result reviewable before a session run.
• Compatibility code and documentation must clearly identify the eventual removal boundary; new features must target canonical party mode rather than extend legacy semantics.

Non-Goals And Deferred Work

This feature does not introduce:

• automatic loading of every file in conf.d;
• recursive imports, imports in profile overlays, or profile inheritance;
• multiple simultaneously active profiles;
• environment-selected profiles;
• generic YAML patch/delete operations;
• arbitrary string interpolation, expressions, scripting, or loops;
• artifact expansion from arbitrary files, Notarius outputs, or runtime data;
• configurable pipeline stages or a workflow DAG;
• runtime wildcard artifact or publish sources;
• automatic hashing of private external-tool configuration, prompts, modules, models, or executables; or
• immediate removal of the isolated legacy party/player compatibility path.

Named reusable parameter sets beyond the canonical party roster may be considered later if a second non-party use case demonstrates a need. Additional party-member types, campaign-domain fields, or family iteration sources should be added only through a versioned schema decision rather than an untyped extension map.

Target End State

The feature is complete when:

• operators can organize a pipeline as a root file plus explicit additive imports with deterministic, source-aware conflict errors;
• one production-default or explicitly selected testing profile applies a deliberate override without duplicating stable configuration;
• every command uses the same resolved profile and reports its identity;
• a campaign owns one strict, versioned party roster containing stable character IDs, players, character names, optional alias lists, and one or more classes;
• Narratio derives the existing players input from that roster and supplies the canonical campaign context to Notarius;
• multiple Scriptorium artifact families expand once per campaign character into ordinary concrete artifacts, dependencies, variables, output paths, and optional publish rules;
• adding, changing, or removing a character produces deterministic and manifest-correct artifact reconciliation;
• profile model changes cannot silently reuse semantically incompatible stage outputs and do not unnecessarily invalidate unaffected transcript work;
• operators can validate, display, trace, and compare effective configuration without executing a session;
• existing monolithic and concrete-artifact configurations remain compatible; and
• current user, operator, integration, architecture, internal, and example documentation accurately owns the implemented contracts.