From 3b5e9db41f732a4675c9edaa291dec1dc5cec6de Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 30 Aug 2026 03:41:56 +0000 Subject: [PATCH] Plan pipeline configuration improvements --- docs/roadmap/implementation.md | 1250 +++++++++++++++++ .../pipeline-configuration-ergonomics.md | 739 ++++++++++ 2 files changed, 1989 insertions(+) create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/pipeline-configuration-ergonomics.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..d407838 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,1250 @@ +# Pipeline Configuration Ergonomics Implementation Plan + +## Purpose And Status + +This document is the executable implementation plan for +[`pipeline-configuration-ergonomics.md`](pipeline-configuration-ergonomics.md). +That feature roadmap owns the accepted user intent, policy choices, scope, and +target end state. This plan translates the roadmap into bounded stages suitable +for one `gpt-5.6-terra` implementation prompt apiece. + +All stages are pending and must be implemented in numeric order. Each stage is +intended to leave the repository correct, documented for the behavior that is +then usable, and independently reviewable. Later stages may rely on the tested +contracts established by earlier stages, but must not silently redesign them. + +## Settled Implementation Decisions + +The following choices make this plan decision-complete: + +- Effective pipeline resolution has one order: parse the root and declared + sources with presence information; additively merge root fields and imports; + apply exactly one selected profile overlay when profiles exist; strict-decode + the assembled mapping; apply centralized defaults; resolve ordinary paths + relative to the root pipeline; resolve the selected campaign and its party; + expand party-driven artifacts, dependencies, member sources, and publish + rules; run ordinary concrete validation; and render/digest the normalized + effective pipeline. Defaults must not be applied to individual fragments. +- `composition.imports` is additive and has no winner. A duplicate final path + in two base sources is an error even if its values are equal. The selected + profile overlay is the only overwrite layer. Lists are atomic in the base and + replace completely in an overlay; maps merge recursively; incompatible YAML + kinds and YAML null-as-deletion are rejected. +- Profile names are case-sensitive, must be non-empty, trimmed, and free of + control characters. If profiles are declared, exactly one must be selected by + an explicitly supplied CLI value or `default_profile`; there is no implicit + first profile. A configuration with imports but no profiles remains valid. +- Pipeline load options retain whether `--profile` was explicitly present so + an explicitly empty value is distinguishable from omission and rejected. + `LoadPipeline` remains a compatibility entry point for callers that do not + make an explicit selection; the option-aware loader is authoritative. +- The normalized effective digest covers the secret-free, defaulted, expanded + runtime pipeline mapping after composition. It excludes composition + declarations and runtime-only provenance. It uses logical configuration + values rather than machine-specific serialization accidents. Party-derived + concrete values and publish rules therefore affect the digest. The digest is + provenance only and is never a blanket resume key. +- Source ownership is tracked by complete YAML-style field path. Base leaves + retain their declaring root/import source, overlaid leaves belong to the + selected overlay, centralized defaults are marked as defaults, and generated + fields identify both the family declaration and canonical party as sources. + Source paths may appear in diagnostics and inspection output; raw secret + values may not. +- Semantic resume evidence uses a versioned, typed stage contract. The runner + compares a stage's current semantic fingerprint before reusing terminal + state, then invokes any existing stage-specific resume validator. Missing or + mismatched evidence makes the stage non-resumable and uses the existing fixed + transitive invalidation relation. Each stage hashes only Narratio-observable, + result-affecting values. Timeouts, concurrency, workspace and diagnostic + paths, executable paths, profile names, and raw credentials are excluded. +- Existing analyze artifact fingerprints remain the granular authority for the + `analyze` stage. Do not add an aggregate analyze fingerprint that would stale + every configured artifact after one member or model changes. Instead, audit + and extend the existing per-artifact fingerprint inputs where necessary. +- The canonical party source is campaign-owned. In canonical + `narratio.party.v1` mode, a separate campaign or session `players_file` and a + session-level `party_file` override are rejected. The isolated legacy mode + preserves the current effective party/players override behavior, treats the + unversioned party payload as opaque reference material, and cannot drive + families. Compatibility code must be grouped and commented for later + removal. +- A party document containing `schema_version` is an attempted canonical + document: an unsupported or malformed version is an error, not a legacy + fallback. Canonical documents are strict single-document YAML. Unversioned + legacy documents are not decoded into the canonical schema. +- The canonical players projection is emitted with + `schema_version: narratio.players.v1`, one entry per character sorted by + stable character ID, and aliases in declaration order. An absent or empty + alias list is omitted from a projection entry. `prepare` copies the canonical + party bytes unchanged and generates players bytes deterministically. +- `scriptorium.artifact_families` is a resolution-only declaration. Every + family expands for every canonical party character, including a disabled + family so explicit selection can use existing disabled-artifact semantics. + Runtime stages and adapters receive only the resulting ordinary concrete + artifacts plus bounded origin metadata. +- `{character_id}` must occur exactly once in each configured family output + path pattern and each non-empty family publish destination pattern. No other + brace token is accepted. Substitution is followed by the existing relative, + confined output/destination validation; arbitrary interpolation is not + introduced. +- Ordinary family `depends_on` entries name shared concrete artifacts. + Corresponding family members use `member_dependencies`. A + `narratio.member_artifact.` input is legal only in a family, must name + one of its `member_dependencies`, and is rewritten to the concrete + `narratio.artifact._` source before ordinary validation. +- Family selection is normalized to concrete keys before the existing + effective-artifact resolver. Selecting a family selects all members; + selecting a generated key selects only that member; mixed selections are + sorted and deduplicated. Typed optional family/character origin fields are + retained in planning, listing, run/analyze state, and reconciliation without + creating dynamic pipeline stages. +- An enabled family publish declaration requires an existing top-level publish + configuration and expands to ordinary concrete output rules. Omitting its + destination pattern derives each destination from the generated artifact's + output path. Top-level publish enablement retains its existing meaning. +- `config validate`, `show`, and `sources` accept the existing `--config`, + `--campaign`, and `--campaign-file` selection mechanisms plus `--profile`. + They do not discover a session or create runtime state. Campaign selection is + optional only when full resolution does not need party data. `config diff` + takes exactly two positional profile names and the same pipeline/campaign + flags, but no separate `--profile` flag. +- `config show` emits deterministic secret-free effective YAML. + `config sources` emits deterministic path/role/source records. + `config diff` emits sorted semantic records of the form added, removed, or + changed at a normalized field path, with values represented in deterministic + compact form; it does not diff raw source text. + +## Instructions For Every Stage + +Before changing code in each stage: + +1. Read `docs/development.md`, all documents in `docs/policy/`, this plan, and + the portions of the feature roadmap relevant to that stage. +2. Follow `docs/development.md` to the current canonical user, integration, and + internal documents for every subsystem the stage changes. +3. Inspect the current implementation and its focused tests before editing. + Prefer the codebase knowledge graph for discovery, then read the exact owning + files and symbols. +4. Confirm the worktree state and preserve unrelated user changes. + +During each stage: + +- Keep changes within that stage's scope and the accepted roadmap. Do not add a + general configuration language, workflow DAG, arbitrary templating, nested + imports, profile inheritance, or runtime family loop. +- Preserve strict YAML, centralized defaults, fixed application-owned stage + sequencing/invalidation, manifest authority, root-confined filesystem work, + indirect secret handling, and adapter ownership of private configuration. +- Use the one production loading/resolution path created by this plan. Tests + may call narrower package owners, but commands may not reimplement merging, + profile selection, party loading, or family expansion. +- Follow the testing policy: protect behavior at its narrowest stable owner, + use real filesystem behavior in `t.TempDir()`, fake only external or + nondeterministic boundaries, keep default tests offline, and avoid repeating + every parser case at CLI or assembled-workflow layers. +- Update each canonical current-behavior document in the same stage that makes + the behavior usable. Until then, leave unimplemented behavior only in the + roadmap and this plan. +- Preserve backward-compatible reads of existing manifests and monolithic + configurations. Emit only the new canonical representation after a new + writer is introduced. +- Run focused tests while iterating, format changed Go files, and finish with + `go test ./...`. Also run the applicable static, race, build, documentation, + and maintained-example checks listed in `docs/development.md`; the final stage + must run the complete repository validation set. + +At the end of each stage, leave a cohesive change that can be reviewed and +committed independently. Do not begin a later stage while an earlier stage has +failing tests, stale documentation, or unmet exit criteria. + +## Stage 1 — Presence-Aware Composition Engine + +**Status: Pending** + +### Goal + +Introduce the internal YAML representation and deterministic merge primitives +needed by imports and profiles without changing the public pipeline schema. + +### Required Work + +1. In `internal/config`, add a composition document model built from + `yaml.Node` or an equivalently presence-aware representation. Retain mapping + versus sequence versus scalar kind, explicit zero values, complete logical + field paths, source file identity, and declaration order for diagnostics. +2. Parse exactly one top-level mapping per source. Reject duplicate YAML keys, + aliases that produce ambiguous ownership, trailing documents, and malformed + YAML with source-qualified errors. Keep the existing strict decoder as the + final schema authority; this layer owns document mechanics and merge + presence, not a parallel field schema. +3. Implement separate operations for: + - additive base merge, which recursively joins disjoint maps and rejects any + duplicate final value/list/keyed entry; and + - overlay merge, which recursively merges maps, replaces same-kind scalar or + list values, and rejects incompatible kinds or deletion/null semantics. +4. Make conflicts report the complete configuration path and all claiming + sources. Preserve declared source order only for diagnostics, never as base + precedence. +5. Add deterministic traversal/rendering suitable for normalized YAML, + canonical digest input, semantic field comparison, and source reports. Map + keys must be sorted for effective output even when source declaration order + is retained separately. +6. Keep these APIs internal to the configuration owner. Do not accept a + `composition` field in `PipelineConfig` yet. + +### Tests And Exit Criteria + +- Package-level table tests cover explicit `false`, zero, empty maps/lists, + disjoint recursive maps, every duplicate class, list atomicity/replacement, + kind conflicts, null deletion attempts, duplicate YAML keys, non-mapping + documents, and trailing documents. +- Diagnostics identify the full field path and both or all relevant sources. +- Deterministic rendering and digest input are stable across map insertion and + source traversal order where semantics are equal. +- Existing `LoadPipeline` behavior and all monolithic examples remain + unchanged. + +## Stage 2 — Explicit Additive Imports + +**Status: Pending** + +### Goal + +Make root-owned `composition.imports` usable while preserving monolithic +configuration behavior and one root-relative path base. + +### Required Work + +1. Add the root-only composition envelope and ordered `imports` field. Strip + the envelope before strict `PipelineConfig` decoding. Root ordinary fields + participate as one additive base source alongside imported partial files. +2. Resolve import paths relative to the root pipeline directory. Accept only + non-empty relative `.yml` or `.yaml` paths that remain beneath that directory + and resolve through `Lstat` to regular files. Reject absolute paths, + traversal, symlinks/non-regular files, missing files, duplicate imports, and + an import of the root itself. +3. Parse every imported document through Stage 1. Reject `composition` in any + imported document, thereby preventing recursion, nested imports, profiles, + and cycles. +4. Additively merge root ordinary fields and imports, strict-decode the result, + apply existing defaults once, and resolve every ordinary relative pipeline + path against the root pipeline directory regardless of the declaring file. +5. Retain root/import paths and base field ownership on runtime-only resolution + metadata without exposing them as YAML fields or secrets. +6. Route `LoadPipeline` through the composed loader so monolithic and imported + roots cannot drift. Do not add an automatic `conf.d` scan. +7. Update `docs/config.md` and focused internal configuration documentation for + the implemented import schema, additive conflict contract, confinement, and + stable root-relative path semantics. + +### Tests And Exit Criteria + +- Tests cover monolithic compatibility; root-plus-import fields; disjoint + contributions beneath `scriptorium.artifacts`; duplicate root/import and + import/import leaves; identical duplicate values; lists; type conflicts; + missing/absolute/traversing/symlink/non-regular/unsupported-extension paths; + repeated imports; imported composition; root self-import; source-aware + errors; and deterministic source ordering. +- Moving the same relative path field between root and import resolves to the + same absolute runtime value. +- `TestExamplesLoadAndValidate` and existing strict loader tests pass without + fixture-wide changes. + +## Stage 3 — Versioned Stage Semantic Resume Framework + +**Status: Pending** + +### Goal + +Create one typed runner/manifest mechanism for result-affecting stage +configuration before selectable profiles can change those values. + +### Required Work + +1. Add an optional typed semantic-configuration fingerprint record to session + `StageRecord` and invocation `RunStageRecord`, containing a positive schema + version and lowercase SHA-256 digest. Use `omitempty` and preserve reads of + existing manifests without the field. +2. Add a narrow optional stage interface that returns the current versioned + semantic fingerprint from resolved `stage.Env` configuration. Provide one + helper that hashes deterministic JSON from stage-owned typed structs; reject + invalid versions and never hash raw secret values, arbitrary `map[string]any` + iteration, or the complete effective configuration digest. +3. In both execution and read-only planning, compare current evidence whenever + a terminal stage record would otherwise be reused. Missing legacy evidence + or a version/digest mismatch makes the stage non-resumable with a bounded, + actionable reason and follows the existing stale/dependent invalidation + path. +4. If the semantic comparison succeeds, continue to call the existing + stage-specific `ResumeValidator`; both checks are required. Forced execution + retains its current behavior and does not need a comparison to decide to + run. +5. Compute the fingerprint immediately before the stage decision, persist the + same value only with a successful or intentional skipped result, and copy it + to the run-stage record. Never promote fingerprint evidence from a failed or + interrupted execution. +6. Ensure a run-manifest skip records the reused semantic fingerprint for + invocation provenance without rewriting the authoritative session stage + result. +7. Update focused manifest, runner, plan, and resume internals. Do not claim + stage coverage until Stages 4–6 add it. + +### Tests And Exit Criteria + +- Manifest round-trip tests prove old records remain readable and new records + preserve version/digest in session and run state. +- Runner and plan tests cover matching, missing, mismatched, malformed, forced, + skipped, failed, and existing-resume-validator combinations. +- A mismatch marks the stage and only its fixed transitive dependents stale; + read-only plan predicts the same decision without mutation. +- Fingerprint tests use typed values and prove deterministic hashes without + asserting implementation-private serialization beyond the versioned + contract. + +## Stage 4 — Prepare, Transcribe, And Merge Semantic Contracts + +**Status: Pending** + +### Goal + +Protect reuse of the pipeline's input preparation and initial transcript work +with stage-specific semantic configuration evidence. + +### Required Work + +1. Implement Stage 3 fingerprint providers for `prepare`, `transcribe`, and + `merge`, each with an independently versioned private payload type. +2. The `prepare` payload must include resolved campaign/session input selection + semantics, local versus S3 audio selection, previous-session identity and + selected previous-artifact requirements, and any option that changes + prepared canonical bytes or names. Exclude absolute source/workspace paths + when their logical values and bytes are equivalent, cache/spool placement, + transfer tuning, and the complete profile/digest. Continue to rely on + prepared-input checksums and current source validation for content identity; + do not hash large audio a second time solely for configuration evidence. +3. The `transcribe` payload must include Narratio-visible recognition language + and service/model identity exposed by its configured adapter contract. It + must exclude retry, concurrency, timeout, credential, and diagnostic values. +4. The `merge` payload must include Seriatim operation/output schema and every + configured merge transformation that can change canonical transcript bytes. + Exclude executable, timeout, report, and retention settings. +5. Keep payload construction in the owning stage or a narrowly shared stage + helper; do not add a reflection-based whole-config hasher. +6. Update the focused prepare, transcribe, merge, operations, and resume + documentation for the evidence that is now observable and the external + private inputs that still require force. + +### Tests And Exit Criteria + +- Each stage has table-driven tests proving a representative semantic change + changes its fingerprint and representative operational/path-only changes do + not. +- Runner tests prove a changed prepare value stales its fixed descendants, a + changed transcribe value reuses prepare but stales transcribe descendants, + and a changed merge value reuses prepare/transcribe. +- Existing successful records without semantic evidence receive a safe one-time + rerun when the corresponding stage is selected. +- No test invokes live storage or transcription services. + +## Stage 5 — Polish, Normalize, Trim, And Render Semantic Contracts + +**Status: Pending** + +### Goal + +Make transcript refinement and rendering safe across profile-selected model and +prompt changes without invalidating unaffected upstream transcript work. + +### Required Work + +1. Implement independently versioned semantic payloads for `polish`, + `normalize`, `trim`, and `render`. +2. The `polish` payload must include every Narratio-visible Audita value that + can change canonical output, including model, validation model, module set, + transcript description, output schema, and explicitly selected external + configuration identity. Include a configured service endpoint when it can + select different semantics. Exclude executable path, timeout, concurrency, + report/debug paths, work retention, and credential environment names. +3. The `normalize` payload must include the Seriatim normalization operation, + schema, and transformation settings, while excluding operational runner + settings. +4. The `trim` payload must include its enablement, prompt/profile identifiers, + canonical input/output identity, Scriptorium variables and render policy + that affect canonical output, and relevant Seriatim transformation settings. + Exclude diagnostic render output, timeout, and executable path. +5. The `render` payload must include final format, title, timestamp/segment ID/ + metadata inclusion, output identity, and any other canonical render choice. +6. Document that Narratio cannot observe private prompt/module/model/config file + contents behind a stable identifier; changes to those contents still require + force. + +### Tests And Exit Criteria + +- A production/testing Audita model change makes `polish` non-resumable while + leaving `prepare`, `transcribe`, and `merge` reusable. +- Each stage's semantic and operational exclusions are covered at its package + boundary, with one runner-level invalidation test per distinct dependency + branch rather than duplicated exhaustive cases. +- Render-only changes do not stale `extract`; trim changes stale both render and + extract through the existing fixed invalidation relation. +- Current stage, operations, and manifest documentation accurately describes + the implemented behavior. + +## Stage 6 — Extract, Analyze, Publish, And Notify Resume Audit + +**Status: Pending** + +### Goal + +Complete semantic resume coverage for downstream artifact work without +weakening extract validation or analyze's per-artifact granularity. + +### Required Work + +1. Give `extract` a versioned semantic payload containing enablement, Notarius + pipeline identity, declared output contracts, reference-slot/source mapping, + and canonical output identities. Combine it with the existing reference and + output resume validator; do not duplicate checksum logic in the payload. + Exclude binary, timeout, working directory, and private Notarius config file + contents. +2. Audit the existing analyze artifact fingerprint schema. Ensure it includes + all Narratio-visible effective prompt/profile identifiers, static and + generated variables, dependencies, input source identities/content + evidence, render-debug policy that affects canonical output, and output + identity. Bump its schema version only if its payload changes. Keep + reconciliation and partial selection artifact-granular. +3. Do not implement the Stage 3 aggregate fingerprint interface for `analyze`. + Adapt plan/runner integration only as necessary so its existing + `ResumeValidator` remains the authority and later family members participate + as ordinary artifacts. +4. Give `publish` semantic evidence for enabled behavior, normalized concrete + source/destination/required rules, upload-run policy, static lock policy, + and remote destination identity (backend, bucket, region/endpoint identity, + and root prefix). Exclude credentials, timeout/retry tuning, local workspace, + and run IDs. Retain immediate lock revalidation and commit safety. +5. Give `notify` semantic evidence for its configured delivery mode and any + Narratio-visible message-shaping option. Never hash credentials or remote + response data. +6. Publish a concise implemented coverage table in the focused resume/manifest + internals, linking to stage and integration owners rather than duplicating + their schemas. + +### Tests And Exit Criteria + +- Extract tests prove semantic changes and reference/output corruption are + independently non-resumable and operational changes are reusable. +- Analyze tests prove a model, variable, dependency, or input change affects + only the relevant artifact and its artifact dependencies, not every analysis + record or upstream transcript stage. +- Publish tests prove a destination/rule/target change reruns publication while + credential names and operational tuning do not; existing lock and commit + tests remain authoritative for destructive behavior. +- All canonical stages now have explicit semantic reuse coverage or the + documented analyze artifact-level equivalent before named profiles become + selectable. + +## Stage 7 — Named Profile Composition And Effective Digest + +**Status: Pending** + +### Goal + +Add strict single-profile overlay resolution and complete effective +configuration provenance within `internal/config`. + +### Required Work + +1. Extend the root-only composition envelope with `default_profile` and a + profile map whose only field is `overlay`. Reject unknown fields, empty + profile names, invalid defaults, nested composition, imports in overlays, + profile inheritance, and any attempt to stack profiles. +2. Add option-aware pipeline loading that distinguishes an explicitly supplied + profile from omission. Apply the selection rules in Settled Decisions and + return errors before stage or adapter composition. +3. Resolve every declared overlay path under the same confinement and regular + YAML-file rules as imports. Structurally parse every declared overlay to + catch missing files, malformed YAML, duplicate keys, trailing documents, or + forbidden composition; apply only the selected overlay to the base. +4. Use Stage 1 overlay semantics. Preserve explicit `false`, zero, empty-list, + and keyed-map additions; replace lists completely; reject kind changes and + null/deletion syntax. +5. Strict-decode and default once after overlay. Retain selected profile name + and source (`default` or `cli`), ordered root/import/overlay sources, and + leaf ownership as runtime-only provenance. +6. Generate a deterministic effective digest from the normalized secret-free + runtime pipeline mapping. Provide one recomputation hook for later + party/family expansion rather than inventing a second digest. +7. Keep `LoadPipeline` as the omission wrapper and update `LoadWithSession*` + option types so profile presence can flow through without API duplication. +8. Update `docs/config.md` and internal configuration documentation for the + implemented schema, exact selection/overlay rules, digest meaning, and + absence of inheritance, stacking, deletion, or environment selection. Do + not document `--profile` until Stage 9 exposes it. + +### Tests And Exit Criteria + +- Tests cover default and explicit selection; explicit-over-default; omitted + selection with profiles; profile-free imports; explicit profile against no + profiles; unknown/empty profiles; invalid defaults; malformed/unselected + overlay sources; map recursion; false/zero overrides; list replacement; + keyed artifact addition/disablement; kind/null conflicts; and nested + composition. +- Equal normalized results produce equal digests regardless of source split; + a semantic value change changes the digest; no raw secret value is loaded or + represented. +- Monolithic and import-only configurations keep their current behavior and + source-relative paths. + +## Stage 8 — One Production Configuration Loading Path + +**Status: Pending** + +### Goal + +Refactor application configuration loading so a composed pipeline/profile is +loaded once and carried unchanged through campaign, local/remote session, and +command-specific resolution before CLI profile selection is exposed. + +### Required Work + +1. Refactor `internal/app/config_loader.go` around one loaded pipeline/campaign + context that retains the option-aware pipeline result and provenance. The + subsequent session resolver must consume that loaded value rather than call + `LoadPipeline` or reread the root path. +2. Provide one configuration-package resolution entry point that can combine + an already loaded pipeline with campaign and optional session data. Keep + compatibility wrappers thin and route them through that owner. +3. Move pipeline and campaign discovery, explicit `--campaign-file` handling, + registry selection, and mutual-exclusion rules into shared application + helpers used by runtime and later inspection commands. Do not silently pick + a campaign when no existing command rule authorizes it. +4. Route local sessions, remote session loading, restore, plan, run, status, + helper commands, single-stage commands, session init/validate, locks, + artifact listing, session cleanup, and `clean --all` through the shared + pipeline load where applicable. Preserve their existing behavior and + mutation boundaries. +5. Ensure remote-session download changes only the session source and cannot + discard the already selected pipeline/profile. Closing temporary session + resources must not invalidate retained configuration/provenance. +6. Remove or make private any alternate production loader that could bypass + import/profile selection. Keep narrow pure test helpers only when they call + the same configuration package APIs. +7. Update focused application configuration-loader internals. This is a + behavior-preserving refactor; do not add public flags or user documentation. + +### Tests And Exit Criteria + +- Existing command, discovery, campaign registry, remote session, restore, + clean, and session-init tests pass unchanged except where fixtures must call + the new shared API. +- Add a counting/in-memory loader seam or equivalent behavioral evidence proving + one command invocation does not reread/reselect the root pipeline while + resolving its session. +- A test mutation of the pipeline file between base and session resolution + cannot create a mixed invocation; the initially loaded value is retained. +- No user-visible command behavior or runtime state format changes in this + stage. + +## Stage 9 — Profile CLI Plumbing, Reporting, And Manifest Provenance + +**Status: Pending** + +### Goal + +Expose profile selection consistently through every configuration-consuming +command and retain bounded invocation provenance without using profile identity +as a cache key. + +### Required Work + +1. Add an explicit-presence `--profile ` flag to the shared configuration + flags. Reject duplicates and explicitly empty values under the same + structural parsing standard as other singleton options. +2. Thread the profile selection through the Stage 8 loader for `run`, `session + plan`, `regenerate-artifacts`, `run-stage`, `analyze`, `publish`, status, + restore, session init/validate, artifacts, locks, cleanup, and every other + command that loads a pipeline. Convenience commands and aliases must not own + an independent profile rule. +3. Add typed optional selected-profile name/source and effective-config digest + fields to session and run manifests. Preserve old manifest reads. The run + manifest records the exact invocation; the session manifest records the most + recently resolved invocation provenance without changing stage reuse + decisions. +4. Apply current provenance before the first persistent run mutation and copy + it into terminal run state. A failed run still retains which effective + configuration was attempted. Read-only plan must report it without writing + any manifest. +5. Include concise profile (or `none`) and digest reporting in run/plan/status + output. When status also shows persisted provenance, label current resolved + versus last persisted values so a profile switch is not ambiguous. +6. Verify secret loading remains after configuration resolution and that raw + environment/file secret values never enter the digest, manifests, logs, or + output. +7. Update `docs/cli.md`, `docs/operations.md`, troubleshooting where useful, + and focused manifest/command internals for selection, reporting, resume + implications, and the force requirement for private external-tool changes. + +### Tests And Exit Criteria + +- Representative shared parser tests and command tests cover omitted/default, + explicit, unknown, duplicate, and explicitly empty profile values, including + the `regenerate-artifacts` alias path. +- Local, remote-session, restore, plan, and one helper command prove they retain + the same selected profile through final resolution; do not repeat the same + assertion for every wrapper. +- Manifest round trips preserve new provenance and old fixtures remain valid. +- An assembled runner test switches only the Audita model: prepare through + merge remain reusable, polish and its fixed dependents become stale, and the + profile name itself causes no unrelated invalidation. + +## Stage 10 — Canonical Party Domain And Players Projection + +**Status: Pending** + +### Goal + +Implement the strict versioned party contract and deterministic derived players +document as a pure configuration/domain boundary before wiring it into campaign +resolution. + +### Required Work + +1. Add typed canonical party structures for `narratio.party.v1`, keyed + characters, nested player/character values, optional singular `alias` list, + and required class entries with optional pointer levels. Retain stable + character IDs and declaration order where the public contract requires it. +2. Parse canonical mode as strict single-document YAML with known fields only. + A top-level `schema_version` selects canonical parsing; a wrong value, + malformed value, or otherwise malformed canonical document is an error. + Classify a document with no version as legacy without decoding it into the + canonical structures. +3. Validate non-empty characters; the existing configured-artifact key grammar + for IDs; exact trimmed, non-control display strings; required player and + character names; optional non-empty aliases; global case-insensitive + ambiguity across all character primary names and aliases; required non-empty + classes; case-insensitive duplicate classes per character; and positive + levels when present. Player display names may repeat. +4. Derive class summaries by preserving declared class order and joining + entries as `` or ` ` with ` / `. Derive alias summaries by + joining declared aliases with `, `. Use Unicode-aware case-insensitive + comparison for ambiguity while retaining original spelling in outputs. +5. Produce the exact `narratio.players.v1` projection from canonical data, one + entry per character sorted by stable ID, with player name, character ID, + character name, and an optional alias list in declared order. Serialize it + deterministically with one trailing newline. +6. Return raw canonical bytes separately from normalized domain values so + prepare can later copy `party.yml` unchanged. Do not expose a general + campaign metadata extension map. +7. Keep legacy classification in a small, clearly named compatibility file or + boundary with a removal comment. It must not grow canonical transformation + behavior. + +### Tests And Exit Criteria + +- Table tests cover every schema rule, alias/primary collisions across and + within characters, Unicode case folding, repeated player names, invalid IDs, + whitespace/control characters, duplicate classes, absent/zero/negative + levels, unknown fields, trailing documents, unsupported versions, and + unversioned legacy classification. +- Multiclass and alias summaries preserve declaration order. +- Projection tests prove stable-ID sorting, repeated-player behavior, optional + alias omission, deterministic bytes, and no class leakage into the + players-only contract. +- Tests target exported/package domain behavior rather than each private YAML + walk helper. + +## Stage 11 — Campaign Party Resolution And Isolated Legacy Mode + +**Status: Pending** + +### Goal + +Make combined pipeline/campaign resolution own canonical party loading and +enforce one unambiguous canonical versus legacy input mode. + +### Required Work + +1. Resolve the campaign-owned `inputs.party_file` relative to the selected + `campaign.yml`, require a regular readable file, load/classify it through + Stage 10, and retain mode, canonical domain data/raw bytes, source path, and + source identity on runtime-only resolved configuration. +2. Split campaign validation into syntax/path-independent validation and final + combined input-mode validation. Do not require `players_file` before party + mode is known. +3. In canonical mode, reject a campaign or session `players_file` and reject a + session `party_file` override. Create a virtual resolved players input whose + source is the canonical party projection; it has no external source path. +4. In legacy mode, require the existing effective `players_file`, preserve + current campaign/session stable-input override semantics for both opaque + party and players files, and retain the exact prepared-input behavior. Add + explicit comments and names marking this compatibility surface for removal + after migration. +5. Make `Resolve`, `LoadWithSessionOptions`, plan/run loading, and any + pipeline-plus-campaign resolution entry point share this logic. A caller + cannot obtain a fully resolved canonical configuration while bypassing party + validation. +6. Add typed party/campaign provenance to the resolution metadata for later + family and `config sources` use. Do not place raw party contents in + manifests or logs. +7. Update `docs/config.md` and add/extend the canonical party integration + document under `docs/integrations/`. Describe legacy mode only as a bounded + migration path and link rather than duplicate the full schema elsewhere. + +### Tests And Exit Criteria + +- Combined-resolution tests cover canonical success, canonical separate + players rejection from campaign and session, canonical session party + override rejection, legacy success, missing legacy players, campaign/session + legacy overrides, missing/non-regular party files, and source-relative paths. +- The same canonical and legacy decision is observed through direct config + resolution, local command loading, and remote session loading with only one + representative application-level test. +- Existing legacy fixtures continue to load; new canonical fixtures do not + carry a separate players file. +- No party-driven family behavior is added yet. + +## Stage 12 — Canonical Party And Derived Players Preparation + +**Status: Pending** + +### Goal + +Materialize one canonical campaign roster and its deterministic players +projection through the existing prepared-input contracts consumed by Notarius +and Scriptorium. + +### Required Work + +1. Refactor `prepare` input materialization by party mode. In canonical mode, + copy the validated campaign party source bytes unchanged to + `inputs/party.yml` and atomically write Stage 10's projection to + `inputs/players.yml`. In legacy mode, keep the current two-file copy path in + the isolated compatibility owner. +2. Record separate `party` and `players` input records with their own content + checksums. Mark the players record with a stable source identifier such as + `derived_from_party`; do not pretend it came from a user `players_file`. + Retain party source/config provenance without recording its full content. +3. Use existing confined atomic/copy-if-changed filesystem helpers and current + group-readable permission policy. Remove an obsolete prior players output + safely when switching modes only through the normal prepared overwrite path. +4. Preserve `narratio.input.party` and `narratio.input.players` as the two + runtime source IDs. Ensure extract reference composition passes the prepared + canonical party unchanged to Notarius's `party` slot and the derived + projection to its `players` slot; do not add adapter-specific roster logic. +5. Ensure Scriptorium configured inputs resolve the derived players record by + the existing manifest-authoritative prepared-input lookup. Missing or + checksum-invalid generated bytes must fail at that owner. +6. Include party mode and projection schema version in prepare semantic + evidence so a legacy-to-canonical migration cannot reuse old prepared + inputs. +7. Complete the durable players integration document and update prepare, + artifact, Notarius, manifest, and operations documentation without copying + full maintained examples into prose. + +### Tests And Exit Criteria + +- Prepare tests compare canonical party bytes exactly, compare deterministic + projection bytes, and verify separate manifest checksums/source identity. +- Legacy prepare tests prove current opaque party and explicit players bytes + remain unchanged through the compatibility path. +- Extract/reference and analyze-input tests prove both source IDs resolve from + prepared manifest authority; missing/corrupt projection evidence fails + clearly without invoking live tools. +- Re-running prepare with identical inputs is byte-stable and does not perform + an unsafe partial write. + +## Stage 13 — Basic Party-Driven Artifact Family Expansion + +**Status: Pending** + +### Goal + +Expand a shared character family declaration into deterministic ordinary +Scriptorium artifacts with member-specific identity, paths, and variables. + +### Required Work + +1. Add strict `scriptorium.artifact_families` structures supporting the shared + concrete-artifact fields named by the feature roadmap plus + `for_each`, `output_path_pattern`, `member_vars`, + `member_dependencies`, and the later typed publish block. Reject unknown + fields through the existing strict pipeline decode. +2. Require the exact `for_each: party.characters`, a canonical party, a valid + family key, and exactly one literal `{character_id}` token in + `output_path_pattern`. Reject all other brace syntax before substitution. +3. For every family and canonical character sorted by family key then stable + character ID, create concrete key `_` and copy shared + enablement, prompt/profile, timeout, render-debug, ordinary dependencies, + inputs, and static variables into an ordinary + `ScriptoriumArtifactConfig`. +4. Resolve member variables only from the closed selectors in the roadmap and + merge their string values into concrete `vars`. Reject invalid selectors, + invalid destination variable names under the existing Scriptorium variable + rules, and collisions with static variables. Preserve the existing reserved + sticky-session variable ownership after expansion. +5. Substitute the stable character ID into output paths and then run the + existing configured-key, safe relative path, duplicate output, selected + executable-field, and Scriptorium variable validation. Do not create a new + runtime artifact type or loop. +6. Reject collisions among family keys, explicit concrete keys, generated + keys, and generated output paths. Expand disabled families too; ordinary + default effective selection will still omit their disabled members. +7. Store a runtime-only family catalog mapping family to sorted members and + each generated member to family/character origins and source ownership. + Remove resolution-only family declarations from the concrete runtime + Scriptorium configuration before adapters receive it. +8. Recompute the normalized effective digest after expansion so party member + values and generated concrete configuration are represented. Update + configuration and Scriptorium internal documentation for the implemented + fields and exact substitution/member-variable limits. + +### Tests And Exit Criteria + +- Tests cover two families over multiple characters, stable ordering, all + member selectors, multiclass/alias summaries, static vars, disabled families, + invalid iteration, legacy/no party, token errors, variable conflicts, + key/output collisions, unsafe paths, and ordinary post-expansion validation. +- Reordering source maps without changing party order semantics produces the + same concrete map, catalog, digest, and normalized output. +- Adapters and analyze planning receive only ordinary concrete artifacts and + current `map[string]any` string/bool variables. +- Adding or removing a character changes the generated member set + deterministically; reconciliation behavior is deferred to Stage 15. + +## Stage 14 — Same-Member Dependencies And Member Artifact Sources + +**Status: Pending** + +### Goal + +Support several coordinated character artifact families while resolving all +family-specific dependency syntax before runtime validation. + +### Required Work + +1. Validate `member_dependencies` as unique family keys. Each referenced family + must exist, use the exact same canonical party iteration source, and produce + the same character IDs. Reject self-dependency early with family/member + context. +2. For each generated member, append dependencies on the corresponding + `_` artifacts. Preserve separately declared + ordinary dependencies on shared concrete artifacts, normalize duplicates, + and let the existing concrete dependency planner detect transitive cycles. +3. Recognize `narratio.member_artifact.` only while expanding family + inputs. Require the referenced family to appear in that declaration's + `member_dependencies`, then rewrite it to + `narratio.artifact._` before artifact-policy and + analyze validation. +4. Reject member-artifact syntax in explicit concrete artifacts, references to + missing/non-member families, malformed suffixes, and any unresolved member + source reaching the concrete pipeline. +5. Attribute generated dependency/input paths to both the declaring family and + relevant party member in provenance. Do not register + `narratio.member_artifact.*` as a runtime artifact-policy source. +6. Update configuration, analyze, and artifact internals for the implemented + resolution boundary and examples no larger than needed to show the syntax. + +### Tests And Exit Criteria + +- Tests cover valid meta-to-items corresponding dependencies for every member, + shared concrete dependencies, missing families/members, self and transitive + cycles, duplicate dependencies, missing `member_dependencies` declarations, + malformed/member syntax in concrete artifacts, and deterministic rewritten + sources. +- Existing concrete dependency ordering, failure propagation, fingerprints, + and source validation work unchanged after expansion. +- A repository search/test assertion confirms no unresolved + `narratio.member_artifact.` source can reach stage or adapter configuration. + +## Stage 15 — Family Selection, Origin Reporting, And Reconciliation + +**Status: Pending** + +### Goal + +Make family declarations ergonomic at command boundaries while preserving +concrete execution, artifact-granular fingerprints, and manifest authority. + +### Required Work + +1. Extend application artifact selection normalization with the Stage 13 family + catalog. Expand an exact family key to all sorted member keys; retain an + exact generated/concrete key as one target; reject unknown values; and sort/ + deduplicate mixed family/member/concrete selections before calling the + existing effective-artifact resolver. +2. Explicit family selection must pass every generated member through the + existing explicitly-selected-disabled validation. Default selection still + includes only enabled concrete artifacts. An empty canonical party is + already invalid and therefore never turns a family selection into a silent + no-op. +3. Extend effective-artifact metadata with optional typed `family` and + `character_id` origins while keeping `Keys()` and concrete lookup behavior + stable for existing callers. +4. Add backward-compatible optional family/character fields to the authoritative + per-artifact analyze session and run records. Populate them from resolved + configuration, not by splitting concrete names. Explicit non-family + artifacts leave them absent. +5. Include origin plus concrete identity in `session plan`, run/analyze + summaries, and `artifacts list` output. Family headings may summarize, but + every executable/reused/failed artifact must remain identifiable by concrete + key. +6. Ensure the existing analyze reconciliation treats a newly added party member + as one or more new configured artifacts and a removed member as removed + records. Preserve unrelated current members and archive/remove outputs only + through existing safe reconciliation policy. +7. Ensure member-specific resolved variables and dependency sources enter the + existing artifact fingerprint so changing a class or alias stales only + consumers of that value and their artifact dependents. +8. Update CLI, analyze, artifact, manifest, and operations documentation for + family versus member selection and concrete runtime identity. + +### Tests And Exit Criteria + +- Selection tests cover family, member, explicit concrete, mixed/duplicate, + unknown, enabled/disabled, and invalid executable member cases. +- Plan and runner tests prove family selection and its normalized concrete + selection are behaviorally equivalent; no adapter receives a family key. +- Reconciliation tests cover adding, changing, and removing a character while + preserving unrelated artifact records and respecting member dependencies. +- Manifest compatibility tests read old analyze records and round-trip new + optional origin fields without deriving origin from names. + +## Stage 16 — Family Publish Rule Expansion + +**Status: Pending** + +### Goal + +Turn one optional family publish declaration into existing concrete publish +rules before ordinary validation and publication. + +### Required Work + +1. Implement the strict family `publish` block with `enabled`, `required`, and + optional `dest_pattern`. An absent or disabled block emits no rules. An + enabled block requires a top-level publish configuration but does not change + top-level publish enablement or storage policy. +2. For each generated family member, create one ordinary publish rule whose + source is `narratio.artifact.`, whose required bit is copied, + and whose destination is either the exactly-once substituted destination + pattern or the generated artifact output path. +3. Reject unknown/missing/repeated brace tokens, unsafe/escaping destinations, + conflicts with explicit rules for the same source, duplicate normalized + destinations, and sources unavailable under existing artifact policy. +4. Merge generated and explicit rules in deterministic destination/source order + before existing publish and static-lock validation. Runtime publish code + continues to receive a concrete list and performs no wildcard/family match. +5. Include generated rules in normalized effective configuration, effective + digest, source provenance, publish semantic fingerprint, plan output, and + static/remote lock applicability. +6. Update configuration, publish, artifact-policy, operations, and manifest + internals for the implemented expansion boundary. + +### Tests And Exit Criteria + +- Tests cover explicit destination patterns, omitted destination derivation, + disabled policy, missing top-level publish config, explicit/generated source + conflicts, duplicate destinations, token/path failures, required propagation, + static locks, and deterministic order. +- Publish stage tests use existing fake object storage to prove generated rules + enter the same staging/commit/lock path as explicit rules. +- No runtime source matcher or adapter accepts family wildcards. + +## Stage 17 — Read-Only `config validate` And `config show` + +**Status: Pending** + +### Goal + +Let operators validate and inspect the complete selected effective pipeline +without creating a session, workspace, run, or external adapter. + +### Required Work + +1. Add the top-level `config` command dispatcher and `validate`/`show` + subcommands. Reuse the shared explicit-presence profile flag and existing + pipeline/campaign selection parsers; reject session-only, stage-range, + force, and artifact-execution flags. +2. Add a shared read-only inspection resolver that uses the Stage 8 production + pipeline/campaign path but does not discover/load a session. If the selected + pipeline contains any party-driven family, require an explicit/unambiguous + campaign through the existing mechanisms and perform canonical party and + all family/publish expansion. Otherwise permit pipeline-only resolution. +3. `config validate` must run complete strict composition, selection, defaults, + path resolution, campaign/party loading when needed, expansion, concrete + artifact/publish validation, and effective digesting. On success, print a + concise root/profile/digest summary; on failure, retain source/field context. +4. `config show` must perform the same validation and emit one deterministic + normalized effective YAML document. Remove composition metadata, + resolution-only families, runtime provenance, and raw secret values; include + defaulted fields, expanded concrete artifacts, and generated publish rules. +5. Define a stable serializer over the logical effective representation rather + than marshaling runtime-only fields or relying on nondeterministic map + traversal. Use documented YAML scalar types and one trailing newline. +6. Both commands must be side-effect free: no workspace layout, manifest, + session lock, secret read, subprocess, network, object storage, or cleanup. +7. Register help/usage and update `docs/cli.md`, `docs/config.md`, and focused + command/config internals. `show` owns effective output, not source tracing or + profile comparison yet. + +### Tests And Exit Criteria + +- CLI tests cover help, unknown subcommands, default/explicit profiles, + pipeline-only success, campaign registry/file selection, family-without- + campaign errors, canonical party errors, and imported source diagnostics. +- `show` golden/semantic tests cover deterministic ordering and expanded + concrete values without overcoupling to incidental whitespace. Golden updates + require the repository's explicit-review convention. +- Side-effect tests prove no workspace/manifest is created and external fake + adapters/object stores are not invoked. +- `validate` and `show` resolve the same digest as plan/run for an equivalent + fully resolved pipeline and campaign. + +## Stage 18 — Read-Only `config sources` + +**Status: Pending** + +### Goal + +Expose enough deterministic source ownership to explain effective values and +generated configuration without leaking secrets or internal YAML machinery. + +### Required Work + +1. Add `config sources` on the Stage 17 resolver and flags. It must validate the + same complete effective configuration before reporting ownership. +2. Complete ownership propagation through defaults, overlay replacements, + canonical campaign/party resolution, derived players, concrete family + expansion, member dependencies/sources, and generated publish rules. +3. Emit sorted records with at least effective field path, source role, and + source path/identifier. Use repeat records when a generated value has both a + family declaration and party source. Mark centralized defaults explicitly + rather than assigning them to whichever source happened to be traversed. +4. Report the root, ordered imports, selected profile/selection source/overlay, + selected campaign, canonical or legacy party mode, party source, and + effective digest in a concise header. Legacy players provenance remains + visible only as a legacy source; canonical derived players point to party. +5. Normalize paths consistently, but never print raw file contents, + environment-resolved credential values, secrets directory contents, or + private adapter configuration contents. +6. Keep source reporting as a projection over configuration provenance. Do not + reparse files in the CLI or add annotations to runtime YAML structs solely + for formatting. +7. Update the CLI/config internal documentation and troubleshooting guidance + for diagnosing duplicate ownership and unexpected profile values. + +### Tests And Exit Criteria + +- Tests cover root/import ownership, profile replacement versus inherited + leaves, list ownership, defaults, campaign/party, derived players, family + variables, member sources/dependencies, generated publish rules, and stable + sorting. +- A source-aware conflict diagnostic and successful `config sources` output use + the same logical path convention. +- Secret sentinel values placed in environment/files never appear in output; + only configured indirect identifiers may appear where part of effective + configuration. +- The command remains side-effect free under the Stage 17 test harness. + +## Stage 19 — Semantic `config diff` + +**Status: Pending** + +### Goal + +Compare two fully resolved profiles by effective meaning rather than raw file +layout or formatting. + +### Required Work + +1. Add `config diff ` with exactly two non-empty + positional profile names plus `--config`, `--campaign`, and + `--campaign-file`. Reject a separate `--profile`, duplicate singleton flags, + extra/missing profiles, and identical unknown selections through shared + profile rules. +2. Resolve the left and right independently from the same already parsed root + source set and the same explicitly selected campaign/party. Do not let one + profile's mutable structs or expansion metadata contaminate the other. +3. Flatten the two normalized secret-free effective mappings to logical field + paths. Emit sorted `added`, `removed`, and `changed` records with deterministic + compact representations of the relevant value(s). Treat an atomic list + replacement as one changed path unless semantic child paths are independently + addressable by the normalized model. +4. Compare expanded concrete artifacts and publish rules, not family source + text. Profile-only source movement with equal effective values produces no + semantic difference; differing complete digests with no emitted semantic + difference is an internal error. +5. Return success with an explicit `no differences` result when equal. Use + normal command errors for invalid resolution; do not use a non-zero exit + merely because differences exist unless Narratio already has a documented + CLI convention for that behavior. +6. Reuse Stage 17's side-effect-free resolver and Stage 1's canonical traversal. + Do not invoke an external `diff` process or add a raw-text diff dependency. +7. Update `docs/cli.md`, `docs/operations.md`, and focused config command + internals with the semantic output and migration/review use case. + +### Tests And Exit Criteria + +- Tests cover scalar changes, additions/removals, explicit false, list + replacement, keyed artifact additions, disabled artifacts, party-expanded + model/variable changes, generated publish differences, equal effective + profiles, invalid profile/campaign inputs, and deterministic order. +- Reformatting or moving an equal value between eligible base imports produces + no diff after successful composition. +- Profile resolution does not mutate cached/shared base nodes and remains + deterministic when left/right argument order is reversed. +- The command performs no runtime state or external side effects. + +## Stage 20 — Maintained Split Configuration Bundle + +**Status: Pending** + +### Goal + +Provide a copyable, validated production/testing bundle and move maintained +example data to the new party source of truth. + +### Required Work + +1. Add or migrate one maintained example bundle with root `pipeline.yml`, + explicit `conf.d` imports for stable concerns, production/testing overlays, + `production` as the default, and no auto-loaded fragment assumptions. +2. Use testing/production model selections that are obvious placeholders and + secret-free. Demonstrate at least one testing-only or testing-disabled + artifact without relying on deletion syntax or profile inheritance. +3. Migrate the maintained campaign to `narratio.party.v1` with stable IDs, + aliases, repeated-safe player semantics, and multiclass data. Remove its + separate `players_file` and provide at least two character families (meta + analysis and item tracking), a same-member dependency/source, and one family + publish policy. +4. Extend `TestExamplesLoadAndValidate` or its focused helpers to discover each + maintained root, validate every declared profile intentionally, select the + required campaign for families, and assert that examples remain offline and + secret-free. Retain at least one monolithic/explicit-concrete compatibility + fixture if the maintained set no longer naturally provides it. +5. Update `examples/README.md` and only the minimum canonical configuration or + integration links needed to make the new bundle discoverable. Do not defer + field/command behavior documentation from the stages that implemented it. + +### Tests And Exit Criteria + +- The maintained example test loads monolithic compatibility and every profile + of the split canonical bundle with its campaign, expands expected concrete + members/rules, and rejects no unknown fields. +- Documentation checks pass; complete examples exist only under `examples/`; + and no secret/private infrastructure values are added. +- A new operator can validate, show, source-trace, and diff the example profiles + using the documented commands without creating runtime state. + +## Stage 21 — Canonical Documentation And Migration Convergence + +**Status: Pending** + +### Goal + +Make current user, operator, integration, policy, and internal documentation +describe the completed feature once, at the correct canonical owners. + +### Required Work + +1. Review every document changed in earlier stages against the implementation + and maintained Stage 20 bundle. Correct commands, flags, fields, defaults, + schemas, paths, output conventions, compatibility limits, and resume claims; + do not repeat complete examples outside `examples/`. +2. Complete `docs/config.md`, `docs/cli.md`, `docs/operations.md`, and integration + links with a concise migration path: split additively, define/select + profiles, convert party, remove players, add families, inspect sources/show, + compare profiles, then run. Link to the maintained bundle rather than copy + it. +3. Revise `docs/policy/architecture.md` only for the durable implemented + invariants: root-owned deterministic composition, campaign-owned canonical + party, pre-runtime family expansion, one loading path, and semantic + stage/artifact resume evidence. Do not move field syntax or implementation + inventory into policy. +4. Update `docs/internal/overview.md` and focused configuration, manifest, + prepare, analyze, artifact, adapter, and publish documents so ownership and + cross-links describe the final implementation without parallel contracts. +5. Review troubleshooting and README orientation for only stable summaries and + links. Remove stale players-file guidance from canonical workflows while + retaining clearly labeled legacy migration documentation at its canonical + owner. +6. Check that documentation distinguishes Narratio-observable semantic + fingerprints from private external-tool content changes that still require + force, and that it never presents the profile name/effective digest as a + blanket resume key. + +### Tests And Exit Criteria + +- Documentation checks and maintained-example validation pass. +- Each contract has one canonical owner under the documentation policy; other + documents contain only a short stable summary and link. +- Current-behavior documents contain no pending implementation language, and + roadmap documents do not masquerade as current reference material. +- No commands, flags, schema fragments, secret values, private infrastructure, + or complete example bundles are duplicated into the wrong owner. + +## Stage 22 — Assembled Workflow Regression And Final Validation + +**Status: Pending** + +### Goal + +Prove the completed feature reaches existing pipeline boundaries correctly and +remove redundant implementation/test scaffolding before release review. + +### Required Work + +1. Add a small number of assembled offline workflow tests using real config, + filesystem, manifest, planner, and reconciliation owners plus fakes only for + external adapters/storage. Do not reproduce every lower-level parser case. +2. Prove one representative split bundle and canonical campaign can: + - select the production default and explicit testing profile; + - prepare unchanged party plus derived players; + - pass both prepared references to Notarius; + - expand two families with member variables, dependencies, and member input; + - select a full family or one concrete member; + - reconcile an added, changed, and removed party character; and + - expand and execute family publication through concrete rules. +3. Prove profile switching reuses semantically unchanged transcript stages, + stales an Audita polish result when its model changes, and limits analyze + regeneration to artifacts whose effective fingerprint changed. Verify + session/run provenance reports the selected profile/digest throughout. +4. Exercise `config validate/show/sources/diff`, `session plan`, and a bounded + run against the same fixtures and assert consistent effective digest, + concrete artifact identity, and family origin. Inspection remains + side-effect free. +5. Audit the final implementation for alternate production config loaders, + unresolved family/member syntax, reflection/whole-config cache keys, + compatibility code outside its marked boundary, duplicated merge or + projection helpers, stale documentation, and redundant tests. Consolidate + only where behavior remains unchanged. +6. Re-run formatting and the complete validation set from + `docs/development.md`: + - `go test ./...` + - `go test -race ./...` + - `go vet ./...` + - `go build ./...` + - `go test ./internal/doccheck` + - `go test ./internal/config -run '^TestExamplesLoadAndValidate$'` +7. Record any platform limitation exactly as current development policy does; + do not claim native macOS/Windows runtime evidence from cross-compilation. + +### Tests And Exit Criteria + +- The assembled tests protect cross-package wiring and meaningful state + transitions while lower-level tests retain ownership of detailed parse and + validation matrices. +- All repository validation commands pass offline and without real + credentials, paid APIs, mutable services, or live Notarius/Scriptorium calls. +- The implemented code, examples, manifests, CLI output, current documentation, + feature roadmap target state, and this plan agree on the final behavior. +- The worktree contains no generated test artifacts or obsolete compatibility + fixtures outside the intentionally retained legacy boundary. + +## Open Questions + +None. The accepted feature roadmap and the settled decisions above are +sufficient to implement every stage without another product or architecture +choice. If implementation reveals a genuinely new decision that would change +the accepted target state, stop that stage and revise the roadmap/plan with the +user rather than choosing a broader behavior implicitly. diff --git a/docs/roadmap/pipeline-configuration-ergonomics.md b/docs/roadmap/pipeline-configuration-ergonomics.md new file mode 100644 index 0000000..fecdae0 --- /dev/null +++ b/docs/roadmap/pipeline-configuration-ergonomics.md @@ -0,0 +1,739 @@ +# Pipeline Configuration Ergonomics + +## Status + +Accepted target state. This document owns the intended feature scope, user +intent, policy choices, and target end state until the behavior is implemented. + +## Goal + +Make Narratio's pipeline configuration easier to organize, vary, inspect, and +reuse without weakening its strict configuration contract or turning it into a +general-purpose configuration or workflow language. + +The feature should let operators: + +- split a large pipeline configuration into explicitly imported partial files; +- maintain one production-default pipeline plus a selectable testing profile; +- keep stable settings separate from frequently edited models and artifacts; +- define a character-oriented artifact workflow once and expand it for every + character in the selected campaign; +- reuse one campaign-owned party roster across multiple artifact families and + the Notarius integration; and +- inspect and compare the fully resolved configuration before running a + session. + +All composition and expansion must produce the same concrete, validated +`PipelineConfig` and concrete configured-artifact map consumed by the existing +stage, planning, manifest, artifact, and adapter boundaries. + +## User Intent + +Pipeline configuration currently combines long-lived platform and transcript +settings with model choices, prompt-backed artifact definitions, and publish +rules that change frequently during development. Keeping all of those concerns +in one file makes edits noisy and makes production/testing variation difficult +to review. + +Character-specific artifacts introduce a second kind of repetition. Several +artifact families, including character meta-analysis and character-specific +item tracking, should execute once per campaign character. Their prompt, +inputs, dependencies, and output shape are shared, while player, character, +class, alias, and output identity vary by party member. Those campaign facts do +not belong in a global pipeline definition. + +The desired model is therefore: + +```text +pipeline.yml + explicit imports + one selected profile + │ +campaign.yml + party.yml ─┤ + ▼ + resolved effective configuration + │ + expand party-driven families + │ + defaults and strict validation + ▼ + existing concrete pipeline and artifact model +``` + +## Architectural Constraints + +The feature must preserve the configuration and orchestration policies in +[`docs/policy/architecture.md`](../policy/architecture.md): + +- YAML remains strict and rejects unknown fields. +- Defaults remain centralized and are applied exactly once. +- Composition and expansion are deterministic and testable. +- Session templating remains narrow; this feature does not introduce arbitrary + expressions, scripting, or a general template language. +- The canonical stage sequence and fixed invalidation relation remain + application-owned and non-configurable. +- Stages receive resolved Narratio-owned configuration and do not implement + import, profile, or family semantics independently. +- External tools continue to own their private runtime configuration and + defaults. +- Raw secrets remain indirect and are never introduced into composed YAML, + effective-configuration output, manifests, or diagnostics. + +Configuration composition belongs to `internal/config`. CLI selection and +production loading belong to `internal/app`. Party parsing must be a +Narratio-owned campaign-input contract. Artifact-family expansion must finish +before existing configured-artifact validation, catalog construction, +planning, reconciliation, or execution begins. + +## Configuration Imports + +### Root document + +Keep `pipeline.yml` as the selected pipeline entry point. Add an optional +top-level composition envelope with explicitly ordered imports: + +```yaml +composition: + imports: + - conf.d/platform.yml + - conf.d/transcripts.yml + - conf.d/extraction.yml + - conf.d/artifacts.yml + - conf.d/publish.yml + + default_profile: production + + profiles: + production: + overlay: profiles/production.yml + testing: + overlay: profiles/testing.yml +``` + +An existing monolithic `pipeline.yml` without `composition` remains valid and +retains its current behavior. The root document may contain ordinary pipeline +fields alongside `composition`, allowing incremental migration. + +`imports` is the public term. Imported files are partial pipeline +configuration documents rather than independently runnable pipelines. + +### Import boundaries + +- Only the root pipeline document may declare `composition`, imports, or + profiles. Imported and overlay documents cannot import other documents. +- Every import is explicit. Narratio does not automatically scan `conf.d` or + interpret unlisted files. +- Import paths are relative to the root pipeline file's directory, must remain + confined beneath that directory, and must resolve to supported regular YAML + files. Absolute paths, traversal, non-regular files, and cycles are rejected. +- All ordinary relative pipeline paths retain one stable base: the root + pipeline file's directory. Moving a field between imported files must not + silently change the meaning of its relative path. +- The declared import order is retained for diagnostics and provenance, but it + is not an implicit precedence mechanism between base imports. +- Composition metadata is removed before the effective mapping is decoded as + strict `PipelineConfig` data. + +### Additive merge contract + +Base configuration from the root document and its imports is additive: + +- mappings merge recursively when their child fields are disjoint; +- two files may therefore contribute different entries beneath a shared map + such as `scriptorium.artifacts`; +- defining the same final field or keyed entry in more than one base source is + an error, even when the values are identical; +- lists are atomic values and cannot be contributed more than once at the same + base configuration path; +- mapping/scalar, mapping/list, and other incompatible type collisions are + errors; and +- duplicate YAML keys within one source remain errors. + +Errors must identify the complete configuration path and every source file +that claims it. This contract prevents an import reorder or newly added file +from silently changing production behavior. + +Composition must operate on a presence-aware YAML representation before Go +struct decoding. Merging already decoded structs is not sufficient because it +cannot reliably distinguish omission from explicitly configured `false`, zero, +an empty collection, or another meaningful zero value. + +## Named Profiles + +### Selection and precedence + +Exactly one named profile may be active for an invocation. A composition root +may define `default_profile`; the maintained production-shaped configuration +uses `production`. Add `--profile ` to the common configuration flags for +every command that loads pipeline configuration: + +```bash +narratio run SESSION +narratio run SESSION --profile testing +narratio session plan SESSION --profile testing +narratio regenerate-artifacts SESSION --profile testing +``` + +The explicit CLI value overrides `default_profile`. An unknown profile, a +profile selection against a configuration that defines no such profile, an +empty profile name, or a default naming an undefined profile is an error before +stage composition. Do not add an environment-variable selector in this +feature; profile selection should remain visible in the command or root +configuration. + +Only one profile can be selected. Profiles cannot extend other profiles, and +callers cannot stack several profiles. A single overlay provides the required +production/testing variation without creating user-programmable precedence. + +### Overlay semantics + +The selected profile is the sole intentional override layer: + +- mappings recursively merge with the assembled base mapping; +- scalar and boolean leaves replace base leaves; +- lists replace the complete base list rather than concatenate implicitly; +- keyed maps such as configured artifacts merge by key, allowing a testing + profile to add experimental artifacts; +- an existing artifact can be disabled explicitly with `enabled: false`; +- an overlay cannot change a value's YAML kind incompatibly; and +- the fully overlaid result undergoes the same strict decode, defaults, path + resolution, expansion, and validation as a monolithic configuration. + +Generic deletion syntax is outside this feature. Shared definitions should +live in base imports, while production-only and testing-only definitions live +in their corresponding overlays. Existing enable/disable fields and complete +list replacement cover the required cases without a YAML patch language. + +### Effective identity and provenance + +Every loaded configuration must retain bounded, non-secret provenance: + +- the root pipeline path; +- the selected profile name, including whether it came from the default or + CLI; +- the ordered imported and overlay source paths; +- a deterministic digest of the normalized effective pipeline configuration; + and +- enough source ownership information to explain composition errors and + effective values. + +Run reporting and invocation state should identify the selected profile and +effective configuration digest. The profile name is provenance, not a blanket +cache key: changing profiles must invalidate only work whose result-affecting +effective configuration changed. + +## Canonical Campaign Party Contract + +### Ownership and schema + +Make `party.yml` a strict, versioned Narratio campaign-input contract. A +canonical file has this shape: + +```yaml +schema_version: narratio.party.v1 + +characters: + arannis: + player: + name: Eric + character: + name: Arannis + alias: + - Ari + - The Grey Owl + classes: + - name: wizard + level: 8 + + brenna: + player: + name: Jane + character: + name: Brenna + classes: + - name: paladin + level: 6 + - name: warlock + level: 2 +``` + +The contract has these semantics: + +- `schema_version` is required and must equal `narratio.party.v1`. +- `characters` is a non-empty mapping. +- Each mapping key is the stable `character_id` and must satisfy the existing + configured-artifact key grammar. +- `player.name` and `character.name` are required, non-empty display strings. +- `character.alias` is optional and accepts a list of zero or more non-empty + alias strings. The singular field spelling `alias` is intentional. +- Leading or trailing whitespace and control characters in names, aliases, and + class names are rejected rather than silently normalized. +- A character alias cannot equal that character's primary name, another alias, + or another character's primary name or alias under case-insensitive + comparison. This keeps party-member grounding unambiguous. Player names may + repeat because one player may control more than one character. +- `character.classes` is required and non-empty. Every entry has a required + free-form `name` and an optional positive integer `level`. +- Class names are not restricted to a Narratio-owned D&D enumeration. +- Duplicate class names for one character are rejected case-insensitively. +- Class order and alias order are preserved. +- Narratio derives a deterministic class summary such as + `paladin 6 / warlock 2` and an alias summary that joins declared aliases with + `, ` in their declared order. +- Unknown fields and trailing YAML documents are rejected. + +The stable `character_id`, rather than a display name, owns generated artifact +identity. Changing a display name, alias, class, or level retains that identity +and changes the relevant semantic inputs. Changing the mapping key is an +intentional remove-and-add operation. + +### Load and prepare behavior + +The selected campaign's effective `party_file` resolves relative to +`campaign.yml`. Narratio must load and validate canonical party data during +combined pipeline/campaign resolution so `session plan`, configuration +inspection, and artifact-family expansion see the same roster before stage +execution. + +`prepare` continues to materialize the canonical document as +`inputs/party.yml`, record its source and content identity, and expose it as +`narratio.input.party`. The same canonical file is passed unchanged to +Notarius's `party` reference slot, which currently accepts YAML reference +material without imposing a competing roster schema. + +### Players projection and compatibility + +The canonical party document becomes the eventual single source of truth for +player-to-character relationships. Narratio derives a deterministic, +documented players-only YAML projection during prepare, materializes it at +`inputs/players.yml`, records its content identity, and exposes it through the +existing `narratio.input.players` source. This preserves the separate Notarius +`players` reference and existing Scriptorium source without requiring every +campaign to maintain duplicate facts. + +The projection uses this versioned shape, with entries sorted by stable +character ID: + +```yaml +schema_version: narratio.players.v1 +players: + - name: Eric + character: + id: arannis + name: Arannis + alias: + - Ari + - The Grey Owl +``` + +There is one projection entry per character rather than one grouped entry per +display player name. This permits one player to control several characters and +does not conflate distinct players who happen to share a display name. + +Backward compatibility must be narrow and removable: + +- an unversioned legacy party file continues to be treated as opaque reference + material when the existing `players_file` is also configured; +- legacy mode preserves the current prepared `party` and `players` behavior + but cannot drive artifact-family expansion; +- configuring a party-driven artifact family with a legacy roster produces a + clear migration error; +- canonical `narratio.party.v1` mode derives players and rejects a separate + `players_file`, preventing contradictory authorities; +- parsing, validation, documentation, and tests for legacy mode live behind a + clearly identified compatibility boundary with comments stating that it is + intended for removal after migration; and +- the maintained examples migrate to canonical mode and demonstrate no + separate players file. + +The derived players projection is a durable integration contract and must be +documented under `docs/integrations/` rather than left as incidental generated +YAML. + +## Party-Driven Artifact Families + +### Configuration model + +Add `scriptorium.artifact_families`, separate from the existing concrete +`scriptorium.artifacts` map: + +```yaml +scriptorium: + artifact_families: + character_meta: + enabled: true + for_each: party.characters + prompt_id: dnd.character_meta + profile_id: production + output_path_pattern: artifacts/characters/{character_id}/meta.md + inputs: + transcript: + source: narratio.transcript.final_trimmed + required: true + member_vars: + player_name: player.name + character_name: character.name + character_class: character.class_summary + character_aliases: character.alias_summary + + character_items: + enabled: true + for_each: party.characters + prompt_id: dnd.character_items + profile_id: production + output_path_pattern: artifacts/characters/{character_id}/items.md + inputs: + transcript: + source: narratio.transcript.final_trimmed + required: true + item_occurrences: + source: narratio.extraction.item_occurrences + required: true + member_vars: + character_name: character.name +``` + +Families support the ordinary result-affecting fields shared by concrete +Scriptorium artifacts: enablement, prompt and profile IDs, timeout, +render-debug policy, ordinary dependencies, inputs, and static variables. +`output_path_pattern` replaces concrete `output_path` at the family level. + +`for_each` is not an expression language. This feature accepts only the exact +source `party.characters`. `member_vars` maps a Scriptorium variable name to +one of a closed set of canonical values: + +- `character_id`; +- `player.name`; +- `character.name`; +- `character.class_summary`; and +- `character.alias_summary`. + +Static `vars` and resolved member variables merge deterministically. Duplicate +variable names across the two maps are rejected rather than assigned implicit +precedence. Narratio's existing reserved sticky-session variable remains +application-owned and is applied after expansion under its current rules. + +### Narrow substitution + +`{character_id}` is the only family substitution supported in this feature. +It is required in `output_path_pattern` and may appear exactly where documented +for family-owned publish destinations. Unknown, repeated in an invalid +position, unresolved, or malformed substitutions are errors. Names, aliases, +classes, environment values, and arbitrary YAML paths cannot be interpolated +into configuration strings. + +### Concrete expansion + +For each family and party character, resolution creates one ordinary concrete +artifact key by joining the normalized family key, an underscore, and the +stable character ID: + +```text +character_meta_arannis +character_items_arannis +``` + +Expansion must be sorted, deterministic, and completed before existing +configured-artifact defaults and validation. Every generated artifact then +uses the existing catalog, dependency planner, fingerprinting, manifest, +reconciliation, execution, materialization, and publish boundaries. + +Validation rejects: + +- invalid family keys or character IDs; +- generated keys that fail the existing configured-artifact grammar; +- collisions between generated artifacts, explicit concrete artifacts, or + family names; +- duplicate or escaping output paths; +- a family with no canonical party source; +- invalid member-variable selectors; +- a selected profile that leaves an executable family incomplete; and +- any expanded artifact that fails ordinary concrete-artifact validation. + +An added party character creates missing concrete artifacts. Removing a +character removes those artifacts from effective configuration so existing +analyze reconciliation classifies their old records as removed. A change to a +member field changes only fingerprints that actually consume its resolved +value or the party document as an input. + +### Dependencies between families + +Ordinary `depends_on` continues to name shared concrete artifacts such as +`session_recap`. Add a typed `member_dependencies` list for corresponding +members of another family: + +```yaml +character_items: + member_dependencies: + - character_meta +``` + +For `arannis`, this expands to a dependency on +`character_meta_arannis`. The referenced family must use the same party source +and generate the same character ID. Cycles and missing members are rejected by +the existing concrete dependency validation after expansion. + +When a family input consumes the corresponding member output, provide a narrow +pre-expansion member-artifact source form owned by configuration resolution. +The syntax is: + +```yaml +inputs: + prior_meta: + source: narratio.member_artifact.character_meta + required: true +``` + +For the `arannis` member, this resolves to +`narratio.artifact.character_meta_arannis` before ordinary runtime validation. +The referenced family must also appear in `member_dependencies`. The +`narratio.member_artifact.*` form must not survive into stage or adapter +configuration as a new runtime artifact kind. + +### Selection and reporting + +Extend artifact selection so an exact family key selects all of that family's +concrete members. An exact generated key selects only that member. Explicitly +selecting a disabled family follows the existing rule for explicitly selected +disabled concrete artifacts: all generated targets must still have valid +executable fields. Mixed family and concrete selections are normalized and +deduplicated after expansion. + +Planning, run summaries, artifact listing, and effective-configuration output +must show both family origin and concrete artifact identity. Execution remains +entirely concrete; a family is not a dynamic pipeline stage or runtime loop. + +### Family publish rules + +Allow a family to declare one typed publish policy so operators need not repeat +one publish output per character: + +```yaml +publish: + enabled: true + required: false + dest_pattern: artifacts/characters/{character_id}/meta.md +``` + +Resolution expands this into ordinary concrete publish rules. When +`dest_pattern` is omitted, the generated artifact's output path is used where +the existing publish contract permits derivation. Conflicts with explicit +publish rules, duplicate destinations, unsafe paths, and generated sources +that are unavailable are rejected by the existing publish policy after +expansion. Family publish behavior must not introduce wildcard source matching +at runtime. + +## Profile-Safe Resume And Invalidation + +Named profiles are safe only if result-affecting model and configuration +changes cannot silently reuse incompatible successful stage results. Artifact +analysis already has a versioned semantic fingerprint that includes effective +prompt/profile identifiers, variables, dependencies, inputs, and output +identity. The broader pipeline requires the same principle at every applicable +stage boundary. + +Audit each stage's resume contract and add a versioned semantic configuration +fingerprint wherever current evidence does not already cover all +Narratio-observable result-affecting settings. In particular, switching an +Audita model through a profile must not reuse polish output produced by the +testing model. Conversely, changing only an operational timeout, binary path, +workspace path, diagnostic location, or profile name must not invalidate +byte-equivalent semantic work unless that value genuinely affects the stage's +canonical result. + +The selected profile name and complete effective-config digest are retained as +provenance. Resume decisions use stage-specific effective semantics rather +than the profile name as a blanket invalidator. This permits production and +testing profiles to share unaffected transcript work while correctly staling +changed stages and their fixed transitive dependents. + +Private configuration, prompt, model, executable, and module contents loaded +inside an external tool remain outside Narratio's observable fingerprint +boundary unless an existing integration contract exposes their identity. +Changing such private inputs continues to require explicit force, and the +documentation must not claim otherwise. + +## Configuration Inspection + +Add a `config` command family that performs composition without running a +session: + +```bash +narratio config validate [--config ] [--profile ] +narratio config show [--config ] [--profile ] +narratio config sources [--config ] [--profile ] +narratio config diff [--config ] +``` + +Required behavior: + +- `validate` composes imports and the selected profile, resolves campaign data + when required for party-driven families, expands artifacts and publish rules, + applies defaults, and runs strict validation without creating session or run + state. +- `show` emits deterministic normalized effective YAML, including generated + concrete artifacts, without raw credentials or environment values. +- `sources` reports root, import, profile, campaign, and party provenance and + identifies which source owns effective configuration paths. +- `diff` compares normalized effective configurations rather than raw files + and clearly identifies additions, removals, and value changes. +- output order is deterministic and suitable for review, but output formatting + is not a substitute for the effective configuration digest. + +If family expansion requires a campaign, these commands accept the existing +campaign selection mechanisms or report clearly that pipeline-only validation +cannot complete the party-dependent portion. They must not silently choose an +unintended campaign. + +## Required Application Changes + +The implemented feature will require coordinated changes across existing +owners rather than a parallel configuration subsystem: + +- `internal/config`: presence-aware YAML composition, import confinement, + profile overlays, provenance, effective digesting, party parsing and + validation, players projection, family expansion, defaults, and final strict + validation; +- `internal/app`: `--profile` plumbing through every configuration-consuming + command, `config` inspection commands, campaign-aware inspection, profile + reporting, and one shared loading path; +- `internal/artifactpolicy` and `internal/artifacts`: generated concrete source + registration, family-aware selection metadata, derived-player identity, and + reuse of the existing configured-artifact grammar and path policy; +- `internal/stage`: consumption only of expanded concrete artifacts, party and + derived-player preparation, and stage-specific semantic resume evidence; +- `internal/manifest`: bounded profile/effective-config provenance and any + versioned stage semantic fingerprints required for safe reuse; +- publish planning: expansion of family publish declarations into existing + concrete rules before ordinary validation and execution; +- Notarius composition: continue passing prepared `party` and `players` + sources, with canonical mode sourcing both from the party-owned contract; + and +- Scriptorium composition: receive only resolved concrete artifacts and + string/bool variables under the existing adapter contract. + +There must be one production configuration-loading path. Remote session +loading, restore, status, planning, helper commands, and execution must not +independently reimplement import, profile, campaign, or family selection. + +## Documentation And Examples + +When behavior is implemented, update the canonical owners in the same change: + +- [`docs/config.md`](../config.md): import, merge, profile, party selection, + family fields, defaults, and validation contracts; +- [`docs/cli.md`](../cli.md): `--profile` and `config` command syntax and output + behavior; +- [`docs/operations.md`](../operations.md): recommended production/testing + bundle layout, migration, inspection, and profile-switching workflow; +- [`docs/policy/architecture.md`](../policy/architecture.md): only the durable + composition, campaign-data ownership, and semantic-resume invariants; +- `docs/integrations/`: the canonical party and derived players YAML contracts; +- [`docs/internal/overview.md`](../internal/overview.md) and focused internal + configuration, manifest, analyze, prepare, and publish documents: implemented + ownership and mechanics; and +- `examples/`: a validated split configuration bundle, production/testing + overlays, canonical party roster with aliases and multiclass data, at least + two party-driven artifact families, and representative family publishing. + +Do not copy complete example bundles into reference prose. Maintained examples +must be secret-free and loaded by the repository's example validation test. + +## Testing Expectations + +Testing should protect the public contracts and high-risk composition and +state-reuse behavior without coupling to private merge helpers. Required +confidence includes: + +- monolithic configuration compatibility; +- recursive mapping composition and all duplicate/type/list conflict classes; +- import confinement, missing files, non-regular files, unsupported nesting, + source-aware diagnostics, deterministic ordering, and duplicate YAML keys; +- explicit zero-value profile overrides, list replacement, keyed-map additions, + unknown profiles, invalid defaults, and absence of profile stacking; +- canonical party schema validation, aliases, alias/name ambiguity, multiclass + summaries, stable IDs, malformed data, and legacy/canonical mode separation; +- deterministic players projection and prepared-input provenance; +- deterministic multi-family expansion, generated identities and paths, + variable binding, same-member dependencies, collisions, additions, + removals, and family selection; +- expansion into ordinary analyze planning, fingerprints, manifests, partial + selection, failure handling, and publish rules; +- model/profile changes staling only stages whose semantic configuration + changed while unaffected transcript work remains reusable; +- representative CLI validation, show, sources, diff, plan, and run behavior; + and +- maintained monolithic, imported, production, and testing examples loading + and validating offline. + +Use package-level behavioral tests through the narrowest stable owner. A small +number of assembled workflow tests should prove that composed profiles and +party-driven families reach the existing pipeline correctly; higher-level +tests should not repeat every parser and validation case. + +## Compatibility And Migration + +- Existing monolithic pipeline files remain supported. +- Omitting `--profile` retains existing behavior for configurations without + profiles and selects the declared production default for composed bundles. +- Existing explicit concrete Scriptorium artifacts and publish rules remain + valid and may coexist with non-colliding families. +- Legacy opaque party plus players files remain temporarily usable only through + the isolated compatibility mode and cannot power party-driven families. +- Migration consists of converting the campaign party document to + `narratio.party.v1`, removing the separate players file, and then enabling + party-driven families. +- Configuration inspection must make the migration result reviewable before a + session run. +- Compatibility code and documentation must clearly identify the eventual + removal boundary; new features must target canonical party mode rather than + extend legacy semantics. + +## Non-Goals And Deferred Work + +This feature does not introduce: + +- automatic loading of every file in `conf.d`; +- recursive imports, imports in profile overlays, or profile inheritance; +- multiple simultaneously active profiles; +- environment-selected profiles; +- generic YAML patch/delete operations; +- arbitrary string interpolation, expressions, scripting, or loops; +- artifact expansion from arbitrary files, Notarius outputs, or runtime data; +- configurable pipeline stages or a workflow DAG; +- runtime wildcard artifact or publish sources; +- automatic hashing of private external-tool configuration, prompts, modules, + models, or executables; or +- immediate removal of the isolated legacy party/player compatibility path. + +Named reusable parameter sets beyond the canonical party roster may be +considered later if a second non-party use case demonstrates a need. Additional +party-member types, campaign-domain fields, or family iteration sources should +be added only through a versioned schema decision rather than an untyped +extension map. + +## Target End State + +The feature is complete when: + +- operators can organize a pipeline as a root file plus explicit additive + imports with deterministic, source-aware conflict errors; +- one production-default or explicitly selected testing profile applies a + deliberate override without duplicating stable configuration; +- every command uses the same resolved profile and reports its identity; +- a campaign owns one strict, versioned party roster containing stable + character IDs, players, character names, optional alias lists, and one or + more classes; +- Narratio derives the existing players input from that roster and supplies the + canonical campaign context to Notarius; +- multiple Scriptorium artifact families expand once per campaign character + into ordinary concrete artifacts, dependencies, variables, output paths, and + optional publish rules; +- adding, changing, or removing a character produces deterministic and + manifest-correct artifact reconciliation; +- profile model changes cannot silently reuse semantically incompatible stage + outputs and do not unnecessarily invalidate unaffected transcript work; +- operators can validate, display, trace, and compare effective configuration + without executing a session; +- existing monolithic and concrete-artifact configurations remain compatible; + and +- current user, operator, integration, architecture, internal, and example + documentation accurately owns the implemented contracts.