1251 lines
66 KiB
Markdown
1251 lines
66 KiB
Markdown
# Pipeline Configuration Ergonomics Implementation Plan
|
||
|
||
## Purpose And Status
|
||
|
||
This document is the executable implementation plan for
|
||
[`pipeline-configuration-ergonomics.md`](pipeline-configuration-ergonomics.md).
|
||
That feature roadmap owns the accepted user intent, policy choices, scope, and
|
||
target end state. This plan translates the roadmap into bounded stages suitable
|
||
for one `gpt-5.6-terra` implementation prompt apiece.
|
||
|
||
All stages are pending and must be implemented in numeric order. Each stage is
|
||
intended to leave the repository correct, documented for the behavior that is
|
||
then usable, and independently reviewable. Later stages may rely on the tested
|
||
contracts established by earlier stages, but must not silently redesign them.
|
||
|
||
## Settled Implementation Decisions
|
||
|
||
The following choices make this plan decision-complete:
|
||
|
||
- Effective pipeline resolution has one order: parse the root and declared
|
||
sources with presence information; additively merge root fields and imports;
|
||
apply exactly one selected profile overlay when profiles exist; strict-decode
|
||
the assembled mapping; apply centralized defaults; resolve ordinary paths
|
||
relative to the root pipeline; resolve the selected campaign and its party;
|
||
expand party-driven artifacts, dependencies, member sources, and publish
|
||
rules; run ordinary concrete validation; and render/digest the normalized
|
||
effective pipeline. Defaults must not be applied to individual fragments.
|
||
- `composition.imports` is additive and has no winner. A duplicate final path
|
||
in two base sources is an error even if its values are equal. The selected
|
||
profile overlay is the only overwrite layer. Lists are atomic in the base and
|
||
replace completely in an overlay; maps merge recursively; incompatible YAML
|
||
kinds and YAML null-as-deletion are rejected.
|
||
- Profile names are case-sensitive, must be non-empty, trimmed, and free of
|
||
control characters. If profiles are declared, exactly one must be selected by
|
||
an explicitly supplied CLI value or `default_profile`; there is no implicit
|
||
first profile. A configuration with imports but no profiles remains valid.
|
||
- Pipeline load options retain whether `--profile` was explicitly present so
|
||
an explicitly empty value is distinguishable from omission and rejected.
|
||
`LoadPipeline` remains a compatibility entry point for callers that do not
|
||
make an explicit selection; the option-aware loader is authoritative.
|
||
- The normalized effective digest covers the secret-free, defaulted, expanded
|
||
runtime pipeline mapping after composition. It excludes composition
|
||
declarations and runtime-only provenance. It uses logical configuration
|
||
values rather than machine-specific serialization accidents. Party-derived
|
||
concrete values and publish rules therefore affect the digest. The digest is
|
||
provenance only and is never a blanket resume key.
|
||
- Source ownership is tracked by complete YAML-style field path. Base leaves
|
||
retain their declaring root/import source, overlaid leaves belong to the
|
||
selected overlay, centralized defaults are marked as defaults, and generated
|
||
fields identify both the family declaration and canonical party as sources.
|
||
Source paths may appear in diagnostics and inspection output; raw secret
|
||
values may not.
|
||
- Semantic resume evidence uses a versioned, typed stage contract. The runner
|
||
compares a stage's current semantic fingerprint before reusing terminal
|
||
state, then invokes any existing stage-specific resume validator. Missing or
|
||
mismatched evidence makes the stage non-resumable and uses the existing fixed
|
||
transitive invalidation relation. Each stage hashes only Narratio-observable,
|
||
result-affecting values. Timeouts, concurrency, workspace and diagnostic
|
||
paths, executable paths, profile names, and raw credentials are excluded.
|
||
- Existing analyze artifact fingerprints remain the granular authority for the
|
||
`analyze` stage. Do not add an aggregate analyze fingerprint that would stale
|
||
every configured artifact after one member or model changes. Instead, audit
|
||
and extend the existing per-artifact fingerprint inputs where necessary.
|
||
- The canonical party source is campaign-owned. In canonical
|
||
`narratio.party.v1` mode, a separate campaign or session `players_file` and a
|
||
session-level `party_file` override are rejected. The isolated legacy mode
|
||
preserves the current effective party/players override behavior, treats the
|
||
unversioned party payload as opaque reference material, and cannot drive
|
||
families. Compatibility code must be grouped and commented for later
|
||
removal.
|
||
- A party document containing `schema_version` is an attempted canonical
|
||
document: an unsupported or malformed version is an error, not a legacy
|
||
fallback. Canonical documents are strict single-document YAML. Unversioned
|
||
legacy documents are not decoded into the canonical schema.
|
||
- The canonical players projection is emitted with
|
||
`schema_version: narratio.players.v1`, one entry per character sorted by
|
||
stable character ID, and aliases in declaration order. An absent or empty
|
||
alias list is omitted from a projection entry. `prepare` copies the canonical
|
||
party bytes unchanged and generates players bytes deterministically.
|
||
- `scriptorium.artifact_families` is a resolution-only declaration. Every
|
||
family expands for every canonical party character, including a disabled
|
||
family so explicit selection can use existing disabled-artifact semantics.
|
||
Runtime stages and adapters receive only the resulting ordinary concrete
|
||
artifacts plus bounded origin metadata.
|
||
- `{character_id}` must occur exactly once in each configured family output
|
||
path pattern and each non-empty family publish destination pattern. No other
|
||
brace token is accepted. Substitution is followed by the existing relative,
|
||
confined output/destination validation; arbitrary interpolation is not
|
||
introduced.
|
||
- Ordinary family `depends_on` entries name shared concrete artifacts.
|
||
Corresponding family members use `member_dependencies`. A
|
||
`narratio.member_artifact.<family>` input is legal only in a family, must name
|
||
one of its `member_dependencies`, and is rewritten to the concrete
|
||
`narratio.artifact.<family>_<character_id>` source before ordinary validation.
|
||
- Family selection is normalized to concrete keys before the existing
|
||
effective-artifact resolver. Selecting a family selects all members;
|
||
selecting a generated key selects only that member; mixed selections are
|
||
sorted and deduplicated. Typed optional family/character origin fields are
|
||
retained in planning, listing, run/analyze state, and reconciliation without
|
||
creating dynamic pipeline stages.
|
||
- An enabled family publish declaration requires an existing top-level publish
|
||
configuration and expands to ordinary concrete output rules. Omitting its
|
||
destination pattern derives each destination from the generated artifact's
|
||
output path. Top-level publish enablement retains its existing meaning.
|
||
- `config validate`, `show`, and `sources` accept the existing `--config`,
|
||
`--campaign`, and `--campaign-file` selection mechanisms plus `--profile`.
|
||
They do not discover a session or create runtime state. Campaign selection is
|
||
optional only when full resolution does not need party data. `config diff`
|
||
takes exactly two positional profile names and the same pipeline/campaign
|
||
flags, but no separate `--profile` flag.
|
||
- `config show` emits deterministic secret-free effective YAML.
|
||
`config sources` emits deterministic path/role/source records.
|
||
`config diff` emits sorted semantic records of the form added, removed, or
|
||
changed at a normalized field path, with values represented in deterministic
|
||
compact form; it does not diff raw source text.
|
||
|
||
## Instructions For Every Stage
|
||
|
||
Before changing code in each stage:
|
||
|
||
1. Read `docs/development.md`, all documents in `docs/policy/`, this plan, and
|
||
the portions of the feature roadmap relevant to that stage.
|
||
2. Follow `docs/development.md` to the current canonical user, integration, and
|
||
internal documents for every subsystem the stage changes.
|
||
3. Inspect the current implementation and its focused tests before editing.
|
||
Prefer the codebase knowledge graph for discovery, then read the exact owning
|
||
files and symbols.
|
||
4. Confirm the worktree state and preserve unrelated user changes.
|
||
|
||
During each stage:
|
||
|
||
- Keep changes within that stage's scope and the accepted roadmap. Do not add a
|
||
general configuration language, workflow DAG, arbitrary templating, nested
|
||
imports, profile inheritance, or runtime family loop.
|
||
- Preserve strict YAML, centralized defaults, fixed application-owned stage
|
||
sequencing/invalidation, manifest authority, root-confined filesystem work,
|
||
indirect secret handling, and adapter ownership of private configuration.
|
||
- Use the one production loading/resolution path created by this plan. Tests
|
||
may call narrower package owners, but commands may not reimplement merging,
|
||
profile selection, party loading, or family expansion.
|
||
- Follow the testing policy: protect behavior at its narrowest stable owner,
|
||
use real filesystem behavior in `t.TempDir()`, fake only external or
|
||
nondeterministic boundaries, keep default tests offline, and avoid repeating
|
||
every parser case at CLI or assembled-workflow layers.
|
||
- Update each canonical current-behavior document in the same stage that makes
|
||
the behavior usable. Until then, leave unimplemented behavior only in the
|
||
roadmap and this plan.
|
||
- Preserve backward-compatible reads of existing manifests and monolithic
|
||
configurations. Emit only the new canonical representation after a new
|
||
writer is introduced.
|
||
- Run focused tests while iterating, format changed Go files, and finish with
|
||
`go test ./...`. Also run the applicable static, race, build, documentation,
|
||
and maintained-example checks listed in `docs/development.md`; the final stage
|
||
must run the complete repository validation set.
|
||
|
||
At the end of each stage, leave a cohesive change that can be reviewed and
|
||
committed independently. Do not begin a later stage while an earlier stage has
|
||
failing tests, stale documentation, or unmet exit criteria.
|
||
|
||
## Stage 1 — Presence-Aware Composition Engine
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Introduce the internal YAML representation and deterministic merge primitives
|
||
needed by imports and profiles without changing the public pipeline schema.
|
||
|
||
### Required Work
|
||
|
||
1. In `internal/config`, add a composition document model built from
|
||
`yaml.Node` or an equivalently presence-aware representation. Retain mapping
|
||
versus sequence versus scalar kind, explicit zero values, complete logical
|
||
field paths, source file identity, and declaration order for diagnostics.
|
||
2. Parse exactly one top-level mapping per source. Reject duplicate YAML keys,
|
||
aliases that produce ambiguous ownership, trailing documents, and malformed
|
||
YAML with source-qualified errors. Keep the existing strict decoder as the
|
||
final schema authority; this layer owns document mechanics and merge
|
||
presence, not a parallel field schema.
|
||
3. Implement separate operations for:
|
||
- additive base merge, which recursively joins disjoint maps and rejects any
|
||
duplicate final value/list/keyed entry; and
|
||
- overlay merge, which recursively merges maps, replaces same-kind scalar or
|
||
list values, and rejects incompatible kinds or deletion/null semantics.
|
||
4. Make conflicts report the complete configuration path and all claiming
|
||
sources. Preserve declared source order only for diagnostics, never as base
|
||
precedence.
|
||
5. Add deterministic traversal/rendering suitable for normalized YAML,
|
||
canonical digest input, semantic field comparison, and source reports. Map
|
||
keys must be sorted for effective output even when source declaration order
|
||
is retained separately.
|
||
6. Keep these APIs internal to the configuration owner. Do not accept a
|
||
`composition` field in `PipelineConfig` yet.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Package-level table tests cover explicit `false`, zero, empty maps/lists,
|
||
disjoint recursive maps, every duplicate class, list atomicity/replacement,
|
||
kind conflicts, null deletion attempts, duplicate YAML keys, non-mapping
|
||
documents, and trailing documents.
|
||
- Diagnostics identify the full field path and both or all relevant sources.
|
||
- Deterministic rendering and digest input are stable across map insertion and
|
||
source traversal order where semantics are equal.
|
||
- Existing `LoadPipeline` behavior and all monolithic examples remain
|
||
unchanged.
|
||
|
||
## Stage 2 — Explicit Additive Imports
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Make root-owned `composition.imports` usable while preserving monolithic
|
||
configuration behavior and one root-relative path base.
|
||
|
||
### Required Work
|
||
|
||
1. Add the root-only composition envelope and ordered `imports` field. Strip
|
||
the envelope before strict `PipelineConfig` decoding. Root ordinary fields
|
||
participate as one additive base source alongside imported partial files.
|
||
2. Resolve import paths relative to the root pipeline directory. Accept only
|
||
non-empty relative `.yml` or `.yaml` paths that remain beneath that directory
|
||
and resolve through `Lstat` to regular files. Reject absolute paths,
|
||
traversal, symlinks/non-regular files, missing files, duplicate imports, and
|
||
an import of the root itself.
|
||
3. Parse every imported document through Stage 1. Reject `composition` in any
|
||
imported document, thereby preventing recursion, nested imports, profiles,
|
||
and cycles.
|
||
4. Additively merge root ordinary fields and imports, strict-decode the result,
|
||
apply existing defaults once, and resolve every ordinary relative pipeline
|
||
path against the root pipeline directory regardless of the declaring file.
|
||
5. Retain root/import paths and base field ownership on runtime-only resolution
|
||
metadata without exposing them as YAML fields or secrets.
|
||
6. Route `LoadPipeline` through the composed loader so monolithic and imported
|
||
roots cannot drift. Do not add an automatic `conf.d` scan.
|
||
7. Update `docs/config.md` and focused internal configuration documentation for
|
||
the implemented import schema, additive conflict contract, confinement, and
|
||
stable root-relative path semantics.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover monolithic compatibility; root-plus-import fields; disjoint
|
||
contributions beneath `scriptorium.artifacts`; duplicate root/import and
|
||
import/import leaves; identical duplicate values; lists; type conflicts;
|
||
missing/absolute/traversing/symlink/non-regular/unsupported-extension paths;
|
||
repeated imports; imported composition; root self-import; source-aware
|
||
errors; and deterministic source ordering.
|
||
- Moving the same relative path field between root and import resolves to the
|
||
same absolute runtime value.
|
||
- `TestExamplesLoadAndValidate` and existing strict loader tests pass without
|
||
fixture-wide changes.
|
||
|
||
## Stage 3 — Versioned Stage Semantic Resume Framework
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Create one typed runner/manifest mechanism for result-affecting stage
|
||
configuration before selectable profiles can change those values.
|
||
|
||
### Required Work
|
||
|
||
1. Add an optional typed semantic-configuration fingerprint record to session
|
||
`StageRecord` and invocation `RunStageRecord`, containing a positive schema
|
||
version and lowercase SHA-256 digest. Use `omitempty` and preserve reads of
|
||
existing manifests without the field.
|
||
2. Add a narrow optional stage interface that returns the current versioned
|
||
semantic fingerprint from resolved `stage.Env` configuration. Provide one
|
||
helper that hashes deterministic JSON from stage-owned typed structs; reject
|
||
invalid versions and never hash raw secret values, arbitrary `map[string]any`
|
||
iteration, or the complete effective configuration digest.
|
||
3. In both execution and read-only planning, compare current evidence whenever
|
||
a terminal stage record would otherwise be reused. Missing legacy evidence
|
||
or a version/digest mismatch makes the stage non-resumable with a bounded,
|
||
actionable reason and follows the existing stale/dependent invalidation
|
||
path.
|
||
4. If the semantic comparison succeeds, continue to call the existing
|
||
stage-specific `ResumeValidator`; both checks are required. Forced execution
|
||
retains its current behavior and does not need a comparison to decide to
|
||
run.
|
||
5. Compute the fingerprint immediately before the stage decision, persist the
|
||
same value only with a successful or intentional skipped result, and copy it
|
||
to the run-stage record. Never promote fingerprint evidence from a failed or
|
||
interrupted execution.
|
||
6. Ensure a run-manifest skip records the reused semantic fingerprint for
|
||
invocation provenance without rewriting the authoritative session stage
|
||
result.
|
||
7. Update focused manifest, runner, plan, and resume internals. Do not claim
|
||
stage coverage until Stages 4–6 add it.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Manifest round-trip tests prove old records remain readable and new records
|
||
preserve version/digest in session and run state.
|
||
- Runner and plan tests cover matching, missing, mismatched, malformed, forced,
|
||
skipped, failed, and existing-resume-validator combinations.
|
||
- A mismatch marks the stage and only its fixed transitive dependents stale;
|
||
read-only plan predicts the same decision without mutation.
|
||
- Fingerprint tests use typed values and prove deterministic hashes without
|
||
asserting implementation-private serialization beyond the versioned
|
||
contract.
|
||
|
||
## Stage 4 — Prepare, Transcribe, And Merge Semantic Contracts
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Protect reuse of the pipeline's input preparation and initial transcript work
|
||
with stage-specific semantic configuration evidence.
|
||
|
||
### Required Work
|
||
|
||
1. Implement Stage 3 fingerprint providers for `prepare`, `transcribe`, and
|
||
`merge`, each with an independently versioned private payload type.
|
||
2. The `prepare` payload must include resolved campaign/session input selection
|
||
semantics, local versus S3 audio selection, previous-session identity and
|
||
selected previous-artifact requirements, and any option that changes
|
||
prepared canonical bytes or names. Exclude absolute source/workspace paths
|
||
when their logical values and bytes are equivalent, cache/spool placement,
|
||
transfer tuning, and the complete profile/digest. Continue to rely on
|
||
prepared-input checksums and current source validation for content identity;
|
||
do not hash large audio a second time solely for configuration evidence.
|
||
3. The `transcribe` payload must include Narratio-visible recognition language
|
||
and service/model identity exposed by its configured adapter contract. It
|
||
must exclude retry, concurrency, timeout, credential, and diagnostic values.
|
||
4. The `merge` payload must include Seriatim operation/output schema and every
|
||
configured merge transformation that can change canonical transcript bytes.
|
||
Exclude executable, timeout, report, and retention settings.
|
||
5. Keep payload construction in the owning stage or a narrowly shared stage
|
||
helper; do not add a reflection-based whole-config hasher.
|
||
6. Update the focused prepare, transcribe, merge, operations, and resume
|
||
documentation for the evidence that is now observable and the external
|
||
private inputs that still require force.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Each stage has table-driven tests proving a representative semantic change
|
||
changes its fingerprint and representative operational/path-only changes do
|
||
not.
|
||
- Runner tests prove a changed prepare value stales its fixed descendants, a
|
||
changed transcribe value reuses prepare but stales transcribe descendants,
|
||
and a changed merge value reuses prepare/transcribe.
|
||
- Existing successful records without semantic evidence receive a safe one-time
|
||
rerun when the corresponding stage is selected.
|
||
- No test invokes live storage or transcription services.
|
||
|
||
## Stage 5 — Polish, Normalize, Trim, And Render Semantic Contracts
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Make transcript refinement and rendering safe across profile-selected model and
|
||
prompt changes without invalidating unaffected upstream transcript work.
|
||
|
||
### Required Work
|
||
|
||
1. Implement independently versioned semantic payloads for `polish`,
|
||
`normalize`, `trim`, and `render`.
|
||
2. The `polish` payload must include every Narratio-visible Audita value that
|
||
can change canonical output, including model, validation model, module set,
|
||
transcript description, output schema, and explicitly selected external
|
||
configuration identity. Include a configured service endpoint when it can
|
||
select different semantics. Exclude executable path, timeout, concurrency,
|
||
report/debug paths, work retention, and credential environment names.
|
||
3. The `normalize` payload must include the Seriatim normalization operation,
|
||
schema, and transformation settings, while excluding operational runner
|
||
settings.
|
||
4. The `trim` payload must include its enablement, prompt/profile identifiers,
|
||
canonical input/output identity, Scriptorium variables and render policy
|
||
that affect canonical output, and relevant Seriatim transformation settings.
|
||
Exclude diagnostic render output, timeout, and executable path.
|
||
5. The `render` payload must include final format, title, timestamp/segment ID/
|
||
metadata inclusion, output identity, and any other canonical render choice.
|
||
6. Document that Narratio cannot observe private prompt/module/model/config file
|
||
contents behind a stable identifier; changes to those contents still require
|
||
force.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- A production/testing Audita model change makes `polish` non-resumable while
|
||
leaving `prepare`, `transcribe`, and `merge` reusable.
|
||
- Each stage's semantic and operational exclusions are covered at its package
|
||
boundary, with one runner-level invalidation test per distinct dependency
|
||
branch rather than duplicated exhaustive cases.
|
||
- Render-only changes do not stale `extract`; trim changes stale both render and
|
||
extract through the existing fixed invalidation relation.
|
||
- Current stage, operations, and manifest documentation accurately describes
|
||
the implemented behavior.
|
||
|
||
## Stage 6 — Extract, Analyze, Publish, And Notify Resume Audit
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Complete semantic resume coverage for downstream artifact work without
|
||
weakening extract validation or analyze's per-artifact granularity.
|
||
|
||
### Required Work
|
||
|
||
1. Give `extract` a versioned semantic payload containing enablement, Notarius
|
||
pipeline identity, declared output contracts, reference-slot/source mapping,
|
||
and canonical output identities. Combine it with the existing reference and
|
||
output resume validator; do not duplicate checksum logic in the payload.
|
||
Exclude binary, timeout, working directory, and private Notarius config file
|
||
contents.
|
||
2. Audit the existing analyze artifact fingerprint schema. Ensure it includes
|
||
all Narratio-visible effective prompt/profile identifiers, static and
|
||
generated variables, dependencies, input source identities/content
|
||
evidence, render-debug policy that affects canonical output, and output
|
||
identity. Bump its schema version only if its payload changes. Keep
|
||
reconciliation and partial selection artifact-granular.
|
||
3. Do not implement the Stage 3 aggregate fingerprint interface for `analyze`.
|
||
Adapt plan/runner integration only as necessary so its existing
|
||
`ResumeValidator` remains the authority and later family members participate
|
||
as ordinary artifacts.
|
||
4. Give `publish` semantic evidence for enabled behavior, normalized concrete
|
||
source/destination/required rules, upload-run policy, static lock policy,
|
||
and remote destination identity (backend, bucket, region/endpoint identity,
|
||
and root prefix). Exclude credentials, timeout/retry tuning, local workspace,
|
||
and run IDs. Retain immediate lock revalidation and commit safety.
|
||
5. Give `notify` semantic evidence for its configured delivery mode and any
|
||
Narratio-visible message-shaping option. Never hash credentials or remote
|
||
response data.
|
||
6. Publish a concise implemented coverage table in the focused resume/manifest
|
||
internals, linking to stage and integration owners rather than duplicating
|
||
their schemas.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Extract tests prove semantic changes and reference/output corruption are
|
||
independently non-resumable and operational changes are reusable.
|
||
- Analyze tests prove a model, variable, dependency, or input change affects
|
||
only the relevant artifact and its artifact dependencies, not every analysis
|
||
record or upstream transcript stage.
|
||
- Publish tests prove a destination/rule/target change reruns publication while
|
||
credential names and operational tuning do not; existing lock and commit
|
||
tests remain authoritative for destructive behavior.
|
||
- All canonical stages now have explicit semantic reuse coverage or the
|
||
documented analyze artifact-level equivalent before named profiles become
|
||
selectable.
|
||
|
||
## Stage 7 — Named Profile Composition And Effective Digest
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Add strict single-profile overlay resolution and complete effective
|
||
configuration provenance within `internal/config`.
|
||
|
||
### Required Work
|
||
|
||
1. Extend the root-only composition envelope with `default_profile` and a
|
||
profile map whose only field is `overlay`. Reject unknown fields, empty
|
||
profile names, invalid defaults, nested composition, imports in overlays,
|
||
profile inheritance, and any attempt to stack profiles.
|
||
2. Add option-aware pipeline loading that distinguishes an explicitly supplied
|
||
profile from omission. Apply the selection rules in Settled Decisions and
|
||
return errors before stage or adapter composition.
|
||
3. Resolve every declared overlay path under the same confinement and regular
|
||
YAML-file rules as imports. Structurally parse every declared overlay to
|
||
catch missing files, malformed YAML, duplicate keys, trailing documents, or
|
||
forbidden composition; apply only the selected overlay to the base.
|
||
4. Use Stage 1 overlay semantics. Preserve explicit `false`, zero, empty-list,
|
||
and keyed-map additions; replace lists completely; reject kind changes and
|
||
null/deletion syntax.
|
||
5. Strict-decode and default once after overlay. Retain selected profile name
|
||
and source (`default` or `cli`), ordered root/import/overlay sources, and
|
||
leaf ownership as runtime-only provenance.
|
||
6. Generate a deterministic effective digest from the normalized secret-free
|
||
runtime pipeline mapping. Provide one recomputation hook for later
|
||
party/family expansion rather than inventing a second digest.
|
||
7. Keep `LoadPipeline` as the omission wrapper and update `LoadWithSession*`
|
||
option types so profile presence can flow through without API duplication.
|
||
8. Update `docs/config.md` and internal configuration documentation for the
|
||
implemented schema, exact selection/overlay rules, digest meaning, and
|
||
absence of inheritance, stacking, deletion, or environment selection. Do
|
||
not document `--profile` until Stage 9 exposes it.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover default and explicit selection; explicit-over-default; omitted
|
||
selection with profiles; profile-free imports; explicit profile against no
|
||
profiles; unknown/empty profiles; invalid defaults; malformed/unselected
|
||
overlay sources; map recursion; false/zero overrides; list replacement;
|
||
keyed artifact addition/disablement; kind/null conflicts; and nested
|
||
composition.
|
||
- Equal normalized results produce equal digests regardless of source split;
|
||
a semantic value change changes the digest; no raw secret value is loaded or
|
||
represented.
|
||
- Monolithic and import-only configurations keep their current behavior and
|
||
source-relative paths.
|
||
|
||
## Stage 8 — One Production Configuration Loading Path
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Refactor application configuration loading so a composed pipeline/profile is
|
||
loaded once and carried unchanged through campaign, local/remote session, and
|
||
command-specific resolution before CLI profile selection is exposed.
|
||
|
||
### Required Work
|
||
|
||
1. Refactor `internal/app/config_loader.go` around one loaded pipeline/campaign
|
||
context that retains the option-aware pipeline result and provenance. The
|
||
subsequent session resolver must consume that loaded value rather than call
|
||
`LoadPipeline` or reread the root path.
|
||
2. Provide one configuration-package resolution entry point that can combine
|
||
an already loaded pipeline with campaign and optional session data. Keep
|
||
compatibility wrappers thin and route them through that owner.
|
||
3. Move pipeline and campaign discovery, explicit `--campaign-file` handling,
|
||
registry selection, and mutual-exclusion rules into shared application
|
||
helpers used by runtime and later inspection commands. Do not silently pick
|
||
a campaign when no existing command rule authorizes it.
|
||
4. Route local sessions, remote session loading, restore, plan, run, status,
|
||
helper commands, single-stage commands, session init/validate, locks,
|
||
artifact listing, session cleanup, and `clean --all` through the shared
|
||
pipeline load where applicable. Preserve their existing behavior and
|
||
mutation boundaries.
|
||
5. Ensure remote-session download changes only the session source and cannot
|
||
discard the already selected pipeline/profile. Closing temporary session
|
||
resources must not invalidate retained configuration/provenance.
|
||
6. Remove or make private any alternate production loader that could bypass
|
||
import/profile selection. Keep narrow pure test helpers only when they call
|
||
the same configuration package APIs.
|
||
7. Update focused application configuration-loader internals. This is a
|
||
behavior-preserving refactor; do not add public flags or user documentation.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Existing command, discovery, campaign registry, remote session, restore,
|
||
clean, and session-init tests pass unchanged except where fixtures must call
|
||
the new shared API.
|
||
- Add a counting/in-memory loader seam or equivalent behavioral evidence proving
|
||
one command invocation does not reread/reselect the root pipeline while
|
||
resolving its session.
|
||
- A test mutation of the pipeline file between base and session resolution
|
||
cannot create a mixed invocation; the initially loaded value is retained.
|
||
- No user-visible command behavior or runtime state format changes in this
|
||
stage.
|
||
|
||
## Stage 9 — Profile CLI Plumbing, Reporting, And Manifest Provenance
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Expose profile selection consistently through every configuration-consuming
|
||
command and retain bounded invocation provenance without using profile identity
|
||
as a cache key.
|
||
|
||
### Required Work
|
||
|
||
1. Add an explicit-presence `--profile <name>` flag to the shared configuration
|
||
flags. Reject duplicates and explicitly empty values under the same
|
||
structural parsing standard as other singleton options.
|
||
2. Thread the profile selection through the Stage 8 loader for `run`, `session
|
||
plan`, `regenerate-artifacts`, `run-stage`, `analyze`, `publish`, status,
|
||
restore, session init/validate, artifacts, locks, cleanup, and every other
|
||
command that loads a pipeline. Convenience commands and aliases must not own
|
||
an independent profile rule.
|
||
3. Add typed optional selected-profile name/source and effective-config digest
|
||
fields to session and run manifests. Preserve old manifest reads. The run
|
||
manifest records the exact invocation; the session manifest records the most
|
||
recently resolved invocation provenance without changing stage reuse
|
||
decisions.
|
||
4. Apply current provenance before the first persistent run mutation and copy
|
||
it into terminal run state. A failed run still retains which effective
|
||
configuration was attempted. Read-only plan must report it without writing
|
||
any manifest.
|
||
5. Include concise profile (or `none`) and digest reporting in run/plan/status
|
||
output. When status also shows persisted provenance, label current resolved
|
||
versus last persisted values so a profile switch is not ambiguous.
|
||
6. Verify secret loading remains after configuration resolution and that raw
|
||
environment/file secret values never enter the digest, manifests, logs, or
|
||
output.
|
||
7. Update `docs/cli.md`, `docs/operations.md`, troubleshooting where useful,
|
||
and focused manifest/command internals for selection, reporting, resume
|
||
implications, and the force requirement for private external-tool changes.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Representative shared parser tests and command tests cover omitted/default,
|
||
explicit, unknown, duplicate, and explicitly empty profile values, including
|
||
the `regenerate-artifacts` alias path.
|
||
- Local, remote-session, restore, plan, and one helper command prove they retain
|
||
the same selected profile through final resolution; do not repeat the same
|
||
assertion for every wrapper.
|
||
- Manifest round trips preserve new provenance and old fixtures remain valid.
|
||
- An assembled runner test switches only the Audita model: prepare through
|
||
merge remain reusable, polish and its fixed dependents become stale, and the
|
||
profile name itself causes no unrelated invalidation.
|
||
|
||
## Stage 10 — Canonical Party Domain And Players Projection
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Implement the strict versioned party contract and deterministic derived players
|
||
document as a pure configuration/domain boundary before wiring it into campaign
|
||
resolution.
|
||
|
||
### Required Work
|
||
|
||
1. Add typed canonical party structures for `narratio.party.v1`, keyed
|
||
characters, nested player/character values, optional singular `alias` list,
|
||
and required class entries with optional pointer levels. Retain stable
|
||
character IDs and declaration order where the public contract requires it.
|
||
2. Parse canonical mode as strict single-document YAML with known fields only.
|
||
A top-level `schema_version` selects canonical parsing; a wrong value,
|
||
malformed value, or otherwise malformed canonical document is an error.
|
||
Classify a document with no version as legacy without decoding it into the
|
||
canonical structures.
|
||
3. Validate non-empty characters; the existing configured-artifact key grammar
|
||
for IDs; exact trimmed, non-control display strings; required player and
|
||
character names; optional non-empty aliases; global case-insensitive
|
||
ambiguity across all character primary names and aliases; required non-empty
|
||
classes; case-insensitive duplicate classes per character; and positive
|
||
levels when present. Player display names may repeat.
|
||
4. Derive class summaries by preserving declared class order and joining
|
||
entries as `<name>` or `<name> <level>` with ` / `. Derive alias summaries by
|
||
joining declared aliases with `, `. Use Unicode-aware case-insensitive
|
||
comparison for ambiguity while retaining original spelling in outputs.
|
||
5. Produce the exact `narratio.players.v1` projection from canonical data, one
|
||
entry per character sorted by stable ID, with player name, character ID,
|
||
character name, and an optional alias list in declared order. Serialize it
|
||
deterministically with one trailing newline.
|
||
6. Return raw canonical bytes separately from normalized domain values so
|
||
prepare can later copy `party.yml` unchanged. Do not expose a general
|
||
campaign metadata extension map.
|
||
7. Keep legacy classification in a small, clearly named compatibility file or
|
||
boundary with a removal comment. It must not grow canonical transformation
|
||
behavior.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Table tests cover every schema rule, alias/primary collisions across and
|
||
within characters, Unicode case folding, repeated player names, invalid IDs,
|
||
whitespace/control characters, duplicate classes, absent/zero/negative
|
||
levels, unknown fields, trailing documents, unsupported versions, and
|
||
unversioned legacy classification.
|
||
- Multiclass and alias summaries preserve declaration order.
|
||
- Projection tests prove stable-ID sorting, repeated-player behavior, optional
|
||
alias omission, deterministic bytes, and no class leakage into the
|
||
players-only contract.
|
||
- Tests target exported/package domain behavior rather than each private YAML
|
||
walk helper.
|
||
|
||
## Stage 11 — Campaign Party Resolution And Isolated Legacy Mode
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Make combined pipeline/campaign resolution own canonical party loading and
|
||
enforce one unambiguous canonical versus legacy input mode.
|
||
|
||
### Required Work
|
||
|
||
1. Resolve the campaign-owned `inputs.party_file` relative to the selected
|
||
`campaign.yml`, require a regular readable file, load/classify it through
|
||
Stage 10, and retain mode, canonical domain data/raw bytes, source path, and
|
||
source identity on runtime-only resolved configuration.
|
||
2. Split campaign validation into syntax/path-independent validation and final
|
||
combined input-mode validation. Do not require `players_file` before party
|
||
mode is known.
|
||
3. In canonical mode, reject a campaign or session `players_file` and reject a
|
||
session `party_file` override. Create a virtual resolved players input whose
|
||
source is the canonical party projection; it has no external source path.
|
||
4. In legacy mode, require the existing effective `players_file`, preserve
|
||
current campaign/session stable-input override semantics for both opaque
|
||
party and players files, and retain the exact prepared-input behavior. Add
|
||
explicit comments and names marking this compatibility surface for removal
|
||
after migration.
|
||
5. Make `Resolve`, `LoadWithSessionOptions`, plan/run loading, and any
|
||
pipeline-plus-campaign resolution entry point share this logic. A caller
|
||
cannot obtain a fully resolved canonical configuration while bypassing party
|
||
validation.
|
||
6. Add typed party/campaign provenance to the resolution metadata for later
|
||
family and `config sources` use. Do not place raw party contents in
|
||
manifests or logs.
|
||
7. Update `docs/config.md` and add/extend the canonical party integration
|
||
document under `docs/integrations/`. Describe legacy mode only as a bounded
|
||
migration path and link rather than duplicate the full schema elsewhere.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Combined-resolution tests cover canonical success, canonical separate
|
||
players rejection from campaign and session, canonical session party
|
||
override rejection, legacy success, missing legacy players, campaign/session
|
||
legacy overrides, missing/non-regular party files, and source-relative paths.
|
||
- The same canonical and legacy decision is observed through direct config
|
||
resolution, local command loading, and remote session loading with only one
|
||
representative application-level test.
|
||
- Existing legacy fixtures continue to load; new canonical fixtures do not
|
||
carry a separate players file.
|
||
- No party-driven family behavior is added yet.
|
||
|
||
## Stage 12 — Canonical Party And Derived Players Preparation
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Materialize one canonical campaign roster and its deterministic players
|
||
projection through the existing prepared-input contracts consumed by Notarius
|
||
and Scriptorium.
|
||
|
||
### Required Work
|
||
|
||
1. Refactor `prepare` input materialization by party mode. In canonical mode,
|
||
copy the validated campaign party source bytes unchanged to
|
||
`inputs/party.yml` and atomically write Stage 10's projection to
|
||
`inputs/players.yml`. In legacy mode, keep the current two-file copy path in
|
||
the isolated compatibility owner.
|
||
2. Record separate `party` and `players` input records with their own content
|
||
checksums. Mark the players record with a stable source identifier such as
|
||
`derived_from_party`; do not pretend it came from a user `players_file`.
|
||
Retain party source/config provenance without recording its full content.
|
||
3. Use existing confined atomic/copy-if-changed filesystem helpers and current
|
||
group-readable permission policy. Remove an obsolete prior players output
|
||
safely when switching modes only through the normal prepared overwrite path.
|
||
4. Preserve `narratio.input.party` and `narratio.input.players` as the two
|
||
runtime source IDs. Ensure extract reference composition passes the prepared
|
||
canonical party unchanged to Notarius's `party` slot and the derived
|
||
projection to its `players` slot; do not add adapter-specific roster logic.
|
||
5. Ensure Scriptorium configured inputs resolve the derived players record by
|
||
the existing manifest-authoritative prepared-input lookup. Missing or
|
||
checksum-invalid generated bytes must fail at that owner.
|
||
6. Include party mode and projection schema version in prepare semantic
|
||
evidence so a legacy-to-canonical migration cannot reuse old prepared
|
||
inputs.
|
||
7. Complete the durable players integration document and update prepare,
|
||
artifact, Notarius, manifest, and operations documentation without copying
|
||
full maintained examples into prose.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Prepare tests compare canonical party bytes exactly, compare deterministic
|
||
projection bytes, and verify separate manifest checksums/source identity.
|
||
- Legacy prepare tests prove current opaque party and explicit players bytes
|
||
remain unchanged through the compatibility path.
|
||
- Extract/reference and analyze-input tests prove both source IDs resolve from
|
||
prepared manifest authority; missing/corrupt projection evidence fails
|
||
clearly without invoking live tools.
|
||
- Re-running prepare with identical inputs is byte-stable and does not perform
|
||
an unsafe partial write.
|
||
|
||
## Stage 13 — Basic Party-Driven Artifact Family Expansion
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Expand a shared character family declaration into deterministic ordinary
|
||
Scriptorium artifacts with member-specific identity, paths, and variables.
|
||
|
||
### Required Work
|
||
|
||
1. Add strict `scriptorium.artifact_families` structures supporting the shared
|
||
concrete-artifact fields named by the feature roadmap plus
|
||
`for_each`, `output_path_pattern`, `member_vars`,
|
||
`member_dependencies`, and the later typed publish block. Reject unknown
|
||
fields through the existing strict pipeline decode.
|
||
2. Require the exact `for_each: party.characters`, a canonical party, a valid
|
||
family key, and exactly one literal `{character_id}` token in
|
||
`output_path_pattern`. Reject all other brace syntax before substitution.
|
||
3. For every family and canonical character sorted by family key then stable
|
||
character ID, create concrete key `<family>_<character_id>` and copy shared
|
||
enablement, prompt/profile, timeout, render-debug, ordinary dependencies,
|
||
inputs, and static variables into an ordinary
|
||
`ScriptoriumArtifactConfig`.
|
||
4. Resolve member variables only from the closed selectors in the roadmap and
|
||
merge their string values into concrete `vars`. Reject invalid selectors,
|
||
invalid destination variable names under the existing Scriptorium variable
|
||
rules, and collisions with static variables. Preserve the existing reserved
|
||
sticky-session variable ownership after expansion.
|
||
5. Substitute the stable character ID into output paths and then run the
|
||
existing configured-key, safe relative path, duplicate output, selected
|
||
executable-field, and Scriptorium variable validation. Do not create a new
|
||
runtime artifact type or loop.
|
||
6. Reject collisions among family keys, explicit concrete keys, generated
|
||
keys, and generated output paths. Expand disabled families too; ordinary
|
||
default effective selection will still omit their disabled members.
|
||
7. Store a runtime-only family catalog mapping family to sorted members and
|
||
each generated member to family/character origins and source ownership.
|
||
Remove resolution-only family declarations from the concrete runtime
|
||
Scriptorium configuration before adapters receive it.
|
||
8. Recompute the normalized effective digest after expansion so party member
|
||
values and generated concrete configuration are represented. Update
|
||
configuration and Scriptorium internal documentation for the implemented
|
||
fields and exact substitution/member-variable limits.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover two families over multiple characters, stable ordering, all
|
||
member selectors, multiclass/alias summaries, static vars, disabled families,
|
||
invalid iteration, legacy/no party, token errors, variable conflicts,
|
||
key/output collisions, unsafe paths, and ordinary post-expansion validation.
|
||
- Reordering source maps without changing party order semantics produces the
|
||
same concrete map, catalog, digest, and normalized output.
|
||
- Adapters and analyze planning receive only ordinary concrete artifacts and
|
||
current `map[string]any` string/bool variables.
|
||
- Adding or removing a character changes the generated member set
|
||
deterministically; reconciliation behavior is deferred to Stage 15.
|
||
|
||
## Stage 14 — Same-Member Dependencies And Member Artifact Sources
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Support several coordinated character artifact families while resolving all
|
||
family-specific dependency syntax before runtime validation.
|
||
|
||
### Required Work
|
||
|
||
1. Validate `member_dependencies` as unique family keys. Each referenced family
|
||
must exist, use the exact same canonical party iteration source, and produce
|
||
the same character IDs. Reject self-dependency early with family/member
|
||
context.
|
||
2. For each generated member, append dependencies on the corresponding
|
||
`<dependency-family>_<character_id>` artifacts. Preserve separately declared
|
||
ordinary dependencies on shared concrete artifacts, normalize duplicates,
|
||
and let the existing concrete dependency planner detect transitive cycles.
|
||
3. Recognize `narratio.member_artifact.<family>` only while expanding family
|
||
inputs. Require the referenced family to appear in that declaration's
|
||
`member_dependencies`, then rewrite it to
|
||
`narratio.artifact.<family>_<character_id>` before artifact-policy and
|
||
analyze validation.
|
||
4. Reject member-artifact syntax in explicit concrete artifacts, references to
|
||
missing/non-member families, malformed suffixes, and any unresolved member
|
||
source reaching the concrete pipeline.
|
||
5. Attribute generated dependency/input paths to both the declaring family and
|
||
relevant party member in provenance. Do not register
|
||
`narratio.member_artifact.*` as a runtime artifact-policy source.
|
||
6. Update configuration, analyze, and artifact internals for the implemented
|
||
resolution boundary and examples no larger than needed to show the syntax.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover valid meta-to-items corresponding dependencies for every member,
|
||
shared concrete dependencies, missing families/members, self and transitive
|
||
cycles, duplicate dependencies, missing `member_dependencies` declarations,
|
||
malformed/member syntax in concrete artifacts, and deterministic rewritten
|
||
sources.
|
||
- Existing concrete dependency ordering, failure propagation, fingerprints,
|
||
and source validation work unchanged after expansion.
|
||
- A repository search/test assertion confirms no unresolved
|
||
`narratio.member_artifact.` source can reach stage or adapter configuration.
|
||
|
||
## Stage 15 — Family Selection, Origin Reporting, And Reconciliation
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Make family declarations ergonomic at command boundaries while preserving
|
||
concrete execution, artifact-granular fingerprints, and manifest authority.
|
||
|
||
### Required Work
|
||
|
||
1. Extend application artifact selection normalization with the Stage 13 family
|
||
catalog. Expand an exact family key to all sorted member keys; retain an
|
||
exact generated/concrete key as one target; reject unknown values; and sort/
|
||
deduplicate mixed family/member/concrete selections before calling the
|
||
existing effective-artifact resolver.
|
||
2. Explicit family selection must pass every generated member through the
|
||
existing explicitly-selected-disabled validation. Default selection still
|
||
includes only enabled concrete artifacts. An empty canonical party is
|
||
already invalid and therefore never turns a family selection into a silent
|
||
no-op.
|
||
3. Extend effective-artifact metadata with optional typed `family` and
|
||
`character_id` origins while keeping `Keys()` and concrete lookup behavior
|
||
stable for existing callers.
|
||
4. Add backward-compatible optional family/character fields to the authoritative
|
||
per-artifact analyze session and run records. Populate them from resolved
|
||
configuration, not by splitting concrete names. Explicit non-family
|
||
artifacts leave them absent.
|
||
5. Include origin plus concrete identity in `session plan`, run/analyze
|
||
summaries, and `artifacts list` output. Family headings may summarize, but
|
||
every executable/reused/failed artifact must remain identifiable by concrete
|
||
key.
|
||
6. Ensure the existing analyze reconciliation treats a newly added party member
|
||
as one or more new configured artifacts and a removed member as removed
|
||
records. Preserve unrelated current members and archive/remove outputs only
|
||
through existing safe reconciliation policy.
|
||
7. Ensure member-specific resolved variables and dependency sources enter the
|
||
existing artifact fingerprint so changing a class or alias stales only
|
||
consumers of that value and their artifact dependents.
|
||
8. Update CLI, analyze, artifact, manifest, and operations documentation for
|
||
family versus member selection and concrete runtime identity.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Selection tests cover family, member, explicit concrete, mixed/duplicate,
|
||
unknown, enabled/disabled, and invalid executable member cases.
|
||
- Plan and runner tests prove family selection and its normalized concrete
|
||
selection are behaviorally equivalent; no adapter receives a family key.
|
||
- Reconciliation tests cover adding, changing, and removing a character while
|
||
preserving unrelated artifact records and respecting member dependencies.
|
||
- Manifest compatibility tests read old analyze records and round-trip new
|
||
optional origin fields without deriving origin from names.
|
||
|
||
## Stage 16 — Family Publish Rule Expansion
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Turn one optional family publish declaration into existing concrete publish
|
||
rules before ordinary validation and publication.
|
||
|
||
### Required Work
|
||
|
||
1. Implement the strict family `publish` block with `enabled`, `required`, and
|
||
optional `dest_pattern`. An absent or disabled block emits no rules. An
|
||
enabled block requires a top-level publish configuration but does not change
|
||
top-level publish enablement or storage policy.
|
||
2. For each generated family member, create one ordinary publish rule whose
|
||
source is `narratio.artifact.<concrete-key>`, whose required bit is copied,
|
||
and whose destination is either the exactly-once substituted destination
|
||
pattern or the generated artifact output path.
|
||
3. Reject unknown/missing/repeated brace tokens, unsafe/escaping destinations,
|
||
conflicts with explicit rules for the same source, duplicate normalized
|
||
destinations, and sources unavailable under existing artifact policy.
|
||
4. Merge generated and explicit rules in deterministic destination/source order
|
||
before existing publish and static-lock validation. Runtime publish code
|
||
continues to receive a concrete list and performs no wildcard/family match.
|
||
5. Include generated rules in normalized effective configuration, effective
|
||
digest, source provenance, publish semantic fingerprint, plan output, and
|
||
static/remote lock applicability.
|
||
6. Update configuration, publish, artifact-policy, operations, and manifest
|
||
internals for the implemented expansion boundary.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover explicit destination patterns, omitted destination derivation,
|
||
disabled policy, missing top-level publish config, explicit/generated source
|
||
conflicts, duplicate destinations, token/path failures, required propagation,
|
||
static locks, and deterministic order.
|
||
- Publish stage tests use existing fake object storage to prove generated rules
|
||
enter the same staging/commit/lock path as explicit rules.
|
||
- No runtime source matcher or adapter accepts family wildcards.
|
||
|
||
## Stage 17 — Read-Only `config validate` And `config show`
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Let operators validate and inspect the complete selected effective pipeline
|
||
without creating a session, workspace, run, or external adapter.
|
||
|
||
### Required Work
|
||
|
||
1. Add the top-level `config` command dispatcher and `validate`/`show`
|
||
subcommands. Reuse the shared explicit-presence profile flag and existing
|
||
pipeline/campaign selection parsers; reject session-only, stage-range,
|
||
force, and artifact-execution flags.
|
||
2. Add a shared read-only inspection resolver that uses the Stage 8 production
|
||
pipeline/campaign path but does not discover/load a session. If the selected
|
||
pipeline contains any party-driven family, require an explicit/unambiguous
|
||
campaign through the existing mechanisms and perform canonical party and
|
||
all family/publish expansion. Otherwise permit pipeline-only resolution.
|
||
3. `config validate` must run complete strict composition, selection, defaults,
|
||
path resolution, campaign/party loading when needed, expansion, concrete
|
||
artifact/publish validation, and effective digesting. On success, print a
|
||
concise root/profile/digest summary; on failure, retain source/field context.
|
||
4. `config show` must perform the same validation and emit one deterministic
|
||
normalized effective YAML document. Remove composition metadata,
|
||
resolution-only families, runtime provenance, and raw secret values; include
|
||
defaulted fields, expanded concrete artifacts, and generated publish rules.
|
||
5. Define a stable serializer over the logical effective representation rather
|
||
than marshaling runtime-only fields or relying on nondeterministic map
|
||
traversal. Use documented YAML scalar types and one trailing newline.
|
||
6. Both commands must be side-effect free: no workspace layout, manifest,
|
||
session lock, secret read, subprocess, network, object storage, or cleanup.
|
||
7. Register help/usage and update `docs/cli.md`, `docs/config.md`, and focused
|
||
command/config internals. `show` owns effective output, not source tracing or
|
||
profile comparison yet.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- CLI tests cover help, unknown subcommands, default/explicit profiles,
|
||
pipeline-only success, campaign registry/file selection, family-without-
|
||
campaign errors, canonical party errors, and imported source diagnostics.
|
||
- `show` golden/semantic tests cover deterministic ordering and expanded
|
||
concrete values without overcoupling to incidental whitespace. Golden updates
|
||
require the repository's explicit-review convention.
|
||
- Side-effect tests prove no workspace/manifest is created and external fake
|
||
adapters/object stores are not invoked.
|
||
- `validate` and `show` resolve the same digest as plan/run for an equivalent
|
||
fully resolved pipeline and campaign.
|
||
|
||
## Stage 18 — Read-Only `config sources`
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Expose enough deterministic source ownership to explain effective values and
|
||
generated configuration without leaking secrets or internal YAML machinery.
|
||
|
||
### Required Work
|
||
|
||
1. Add `config sources` on the Stage 17 resolver and flags. It must validate the
|
||
same complete effective configuration before reporting ownership.
|
||
2. Complete ownership propagation through defaults, overlay replacements,
|
||
canonical campaign/party resolution, derived players, concrete family
|
||
expansion, member dependencies/sources, and generated publish rules.
|
||
3. Emit sorted records with at least effective field path, source role, and
|
||
source path/identifier. Use repeat records when a generated value has both a
|
||
family declaration and party source. Mark centralized defaults explicitly
|
||
rather than assigning them to whichever source happened to be traversed.
|
||
4. Report the root, ordered imports, selected profile/selection source/overlay,
|
||
selected campaign, canonical or legacy party mode, party source, and
|
||
effective digest in a concise header. Legacy players provenance remains
|
||
visible only as a legacy source; canonical derived players point to party.
|
||
5. Normalize paths consistently, but never print raw file contents,
|
||
environment-resolved credential values, secrets directory contents, or
|
||
private adapter configuration contents.
|
||
6. Keep source reporting as a projection over configuration provenance. Do not
|
||
reparse files in the CLI or add annotations to runtime YAML structs solely
|
||
for formatting.
|
||
7. Update the CLI/config internal documentation and troubleshooting guidance
|
||
for diagnosing duplicate ownership and unexpected profile values.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover root/import ownership, profile replacement versus inherited
|
||
leaves, list ownership, defaults, campaign/party, derived players, family
|
||
variables, member sources/dependencies, generated publish rules, and stable
|
||
sorting.
|
||
- A source-aware conflict diagnostic and successful `config sources` output use
|
||
the same logical path convention.
|
||
- Secret sentinel values placed in environment/files never appear in output;
|
||
only configured indirect identifiers may appear where part of effective
|
||
configuration.
|
||
- The command remains side-effect free under the Stage 17 test harness.
|
||
|
||
## Stage 19 — Semantic `config diff`
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Compare two fully resolved profiles by effective meaning rather than raw file
|
||
layout or formatting.
|
||
|
||
### Required Work
|
||
|
||
1. Add `config diff <left-profile> <right-profile>` with exactly two non-empty
|
||
positional profile names plus `--config`, `--campaign`, and
|
||
`--campaign-file`. Reject a separate `--profile`, duplicate singleton flags,
|
||
extra/missing profiles, and identical unknown selections through shared
|
||
profile rules.
|
||
2. Resolve the left and right independently from the same already parsed root
|
||
source set and the same explicitly selected campaign/party. Do not let one
|
||
profile's mutable structs or expansion metadata contaminate the other.
|
||
3. Flatten the two normalized secret-free effective mappings to logical field
|
||
paths. Emit sorted `added`, `removed`, and `changed` records with deterministic
|
||
compact representations of the relevant value(s). Treat an atomic list
|
||
replacement as one changed path unless semantic child paths are independently
|
||
addressable by the normalized model.
|
||
4. Compare expanded concrete artifacts and publish rules, not family source
|
||
text. Profile-only source movement with equal effective values produces no
|
||
semantic difference; differing complete digests with no emitted semantic
|
||
difference is an internal error.
|
||
5. Return success with an explicit `no differences` result when equal. Use
|
||
normal command errors for invalid resolution; do not use a non-zero exit
|
||
merely because differences exist unless Narratio already has a documented
|
||
CLI convention for that behavior.
|
||
6. Reuse Stage 17's side-effect-free resolver and Stage 1's canonical traversal.
|
||
Do not invoke an external `diff` process or add a raw-text diff dependency.
|
||
7. Update `docs/cli.md`, `docs/operations.md`, and focused config command
|
||
internals with the semantic output and migration/review use case.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover scalar changes, additions/removals, explicit false, list
|
||
replacement, keyed artifact additions, disabled artifacts, party-expanded
|
||
model/variable changes, generated publish differences, equal effective
|
||
profiles, invalid profile/campaign inputs, and deterministic order.
|
||
- Reformatting or moving an equal value between eligible base imports produces
|
||
no diff after successful composition.
|
||
- Profile resolution does not mutate cached/shared base nodes and remains
|
||
deterministic when left/right argument order is reversed.
|
||
- The command performs no runtime state or external side effects.
|
||
|
||
## Stage 20 — Maintained Split Configuration Bundle
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Provide a copyable, validated production/testing bundle and move maintained
|
||
example data to the new party source of truth.
|
||
|
||
### Required Work
|
||
|
||
1. Add or migrate one maintained example bundle with root `pipeline.yml`,
|
||
explicit `conf.d` imports for stable concerns, production/testing overlays,
|
||
`production` as the default, and no auto-loaded fragment assumptions.
|
||
2. Use testing/production model selections that are obvious placeholders and
|
||
secret-free. Demonstrate at least one testing-only or testing-disabled
|
||
artifact without relying on deletion syntax or profile inheritance.
|
||
3. Migrate the maintained campaign to `narratio.party.v1` with stable IDs,
|
||
aliases, repeated-safe player semantics, and multiclass data. Remove its
|
||
separate `players_file` and provide at least two character families (meta
|
||
analysis and item tracking), a same-member dependency/source, and one family
|
||
publish policy.
|
||
4. Extend `TestExamplesLoadAndValidate` or its focused helpers to discover each
|
||
maintained root, validate every declared profile intentionally, select the
|
||
required campaign for families, and assert that examples remain offline and
|
||
secret-free. Retain at least one monolithic/explicit-concrete compatibility
|
||
fixture if the maintained set no longer naturally provides it.
|
||
5. Update `examples/README.md` and only the minimum canonical configuration or
|
||
integration links needed to make the new bundle discoverable. Do not defer
|
||
field/command behavior documentation from the stages that implemented it.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- The maintained example test loads monolithic compatibility and every profile
|
||
of the split canonical bundle with its campaign, expands expected concrete
|
||
members/rules, and rejects no unknown fields.
|
||
- Documentation checks pass; complete examples exist only under `examples/`;
|
||
and no secret/private infrastructure values are added.
|
||
- A new operator can validate, show, source-trace, and diff the example profiles
|
||
using the documented commands without creating runtime state.
|
||
|
||
## Stage 21 — Canonical Documentation And Migration Convergence
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Make current user, operator, integration, policy, and internal documentation
|
||
describe the completed feature once, at the correct canonical owners.
|
||
|
||
### Required Work
|
||
|
||
1. Review every document changed in earlier stages against the implementation
|
||
and maintained Stage 20 bundle. Correct commands, flags, fields, defaults,
|
||
schemas, paths, output conventions, compatibility limits, and resume claims;
|
||
do not repeat complete examples outside `examples/`.
|
||
2. Complete `docs/config.md`, `docs/cli.md`, `docs/operations.md`, and integration
|
||
links with a concise migration path: split additively, define/select
|
||
profiles, convert party, remove players, add families, inspect sources/show,
|
||
compare profiles, then run. Link to the maintained bundle rather than copy
|
||
it.
|
||
3. Revise `docs/policy/architecture.md` only for the durable implemented
|
||
invariants: root-owned deterministic composition, campaign-owned canonical
|
||
party, pre-runtime family expansion, one loading path, and semantic
|
||
stage/artifact resume evidence. Do not move field syntax or implementation
|
||
inventory into policy.
|
||
4. Update `docs/internal/overview.md` and focused configuration, manifest,
|
||
prepare, analyze, artifact, adapter, and publish documents so ownership and
|
||
cross-links describe the final implementation without parallel contracts.
|
||
5. Review troubleshooting and README orientation for only stable summaries and
|
||
links. Remove stale players-file guidance from canonical workflows while
|
||
retaining clearly labeled legacy migration documentation at its canonical
|
||
owner.
|
||
6. Check that documentation distinguishes Narratio-observable semantic
|
||
fingerprints from private external-tool content changes that still require
|
||
force, and that it never presents the profile name/effective digest as a
|
||
blanket resume key.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Documentation checks and maintained-example validation pass.
|
||
- Each contract has one canonical owner under the documentation policy; other
|
||
documents contain only a short stable summary and link.
|
||
- Current-behavior documents contain no pending implementation language, and
|
||
roadmap documents do not masquerade as current reference material.
|
||
- No commands, flags, schema fragments, secret values, private infrastructure,
|
||
or complete example bundles are duplicated into the wrong owner.
|
||
|
||
## Stage 22 — Assembled Workflow Regression And Final Validation
|
||
|
||
**Status: Pending**
|
||
|
||
### Goal
|
||
|
||
Prove the completed feature reaches existing pipeline boundaries correctly and
|
||
remove redundant implementation/test scaffolding before release review.
|
||
|
||
### Required Work
|
||
|
||
1. Add a small number of assembled offline workflow tests using real config,
|
||
filesystem, manifest, planner, and reconciliation owners plus fakes only for
|
||
external adapters/storage. Do not reproduce every lower-level parser case.
|
||
2. Prove one representative split bundle and canonical campaign can:
|
||
- select the production default and explicit testing profile;
|
||
- prepare unchanged party plus derived players;
|
||
- pass both prepared references to Notarius;
|
||
- expand two families with member variables, dependencies, and member input;
|
||
- select a full family or one concrete member;
|
||
- reconcile an added, changed, and removed party character; and
|
||
- expand and execute family publication through concrete rules.
|
||
3. Prove profile switching reuses semantically unchanged transcript stages,
|
||
stales an Audita polish result when its model changes, and limits analyze
|
||
regeneration to artifacts whose effective fingerprint changed. Verify
|
||
session/run provenance reports the selected profile/digest throughout.
|
||
4. Exercise `config validate/show/sources/diff`, `session plan`, and a bounded
|
||
run against the same fixtures and assert consistent effective digest,
|
||
concrete artifact identity, and family origin. Inspection remains
|
||
side-effect free.
|
||
5. Audit the final implementation for alternate production config loaders,
|
||
unresolved family/member syntax, reflection/whole-config cache keys,
|
||
compatibility code outside its marked boundary, duplicated merge or
|
||
projection helpers, stale documentation, and redundant tests. Consolidate
|
||
only where behavior remains unchanged.
|
||
6. Re-run formatting and the complete validation set from
|
||
`docs/development.md`:
|
||
- `go test ./...`
|
||
- `go test -race ./...`
|
||
- `go vet ./...`
|
||
- `go build ./...`
|
||
- `go test ./internal/doccheck`
|
||
- `go test ./internal/config -run '^TestExamplesLoadAndValidate$'`
|
||
7. Record any platform limitation exactly as current development policy does;
|
||
do not claim native macOS/Windows runtime evidence from cross-compilation.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- The assembled tests protect cross-package wiring and meaningful state
|
||
transitions while lower-level tests retain ownership of detailed parse and
|
||
validation matrices.
|
||
- All repository validation commands pass offline and without real
|
||
credentials, paid APIs, mutable services, or live Notarius/Scriptorium calls.
|
||
- The implemented code, examples, manifests, CLI output, current documentation,
|
||
feature roadmap target state, and this plan agree on the final behavior.
|
||
- The worktree contains no generated test artifacts or obsolete compatibility
|
||
fixtures outside the intentionally retained legacy boundary.
|
||
|
||
## Open Questions
|
||
|
||
None. The accepted feature roadmap and the settled decisions above are
|
||
sufficient to implement every stage without another product or architecture
|
||
choice. If implementation reveals a genuinely new decision that would change
|
||
the accepted target state, stop that stage and revise the roadmap/plan with the
|
||
user rather than choosing a broader behavior implicitly.
|