Plan pipeline configuration improvements
This commit is contained in:
739
docs/roadmap/pipeline-configuration-ergonomics.md
Normal file
739
docs/roadmap/pipeline-configuration-ergonomics.md
Normal file
@@ -0,0 +1,739 @@
|
||||
# Pipeline Configuration Ergonomics
|
||||
|
||||
## Status
|
||||
|
||||
Accepted target state. This document owns the intended feature scope, user
|
||||
intent, policy choices, and target end state until the behavior is implemented.
|
||||
|
||||
## Goal
|
||||
|
||||
Make Narratio's pipeline configuration easier to organize, vary, inspect, and
|
||||
reuse without weakening its strict configuration contract or turning it into a
|
||||
general-purpose configuration or workflow language.
|
||||
|
||||
The feature should let operators:
|
||||
|
||||
- split a large pipeline configuration into explicitly imported partial files;
|
||||
- maintain one production-default pipeline plus a selectable testing profile;
|
||||
- keep stable settings separate from frequently edited models and artifacts;
|
||||
- define a character-oriented artifact workflow once and expand it for every
|
||||
character in the selected campaign;
|
||||
- reuse one campaign-owned party roster across multiple artifact families and
|
||||
the Notarius integration; and
|
||||
- inspect and compare the fully resolved configuration before running a
|
||||
session.
|
||||
|
||||
All composition and expansion must produce the same concrete, validated
|
||||
`PipelineConfig` and concrete configured-artifact map consumed by the existing
|
||||
stage, planning, manifest, artifact, and adapter boundaries.
|
||||
|
||||
## User Intent
|
||||
|
||||
Pipeline configuration currently combines long-lived platform and transcript
|
||||
settings with model choices, prompt-backed artifact definitions, and publish
|
||||
rules that change frequently during development. Keeping all of those concerns
|
||||
in one file makes edits noisy and makes production/testing variation difficult
|
||||
to review.
|
||||
|
||||
Character-specific artifacts introduce a second kind of repetition. Several
|
||||
artifact families, including character meta-analysis and character-specific
|
||||
item tracking, should execute once per campaign character. Their prompt,
|
||||
inputs, dependencies, and output shape are shared, while player, character,
|
||||
class, alias, and output identity vary by party member. Those campaign facts do
|
||||
not belong in a global pipeline definition.
|
||||
|
||||
The desired model is therefore:
|
||||
|
||||
```text
|
||||
pipeline.yml + explicit imports + one selected profile
|
||||
│
|
||||
campaign.yml + party.yml ─┤
|
||||
▼
|
||||
resolved effective configuration
|
||||
│
|
||||
expand party-driven families
|
||||
│
|
||||
defaults and strict validation
|
||||
▼
|
||||
existing concrete pipeline and artifact model
|
||||
```
|
||||
|
||||
## Architectural Constraints
|
||||
|
||||
The feature must preserve the configuration and orchestration policies in
|
||||
[`docs/policy/architecture.md`](../policy/architecture.md):
|
||||
|
||||
- YAML remains strict and rejects unknown fields.
|
||||
- Defaults remain centralized and are applied exactly once.
|
||||
- Composition and expansion are deterministic and testable.
|
||||
- Session templating remains narrow; this feature does not introduce arbitrary
|
||||
expressions, scripting, or a general template language.
|
||||
- The canonical stage sequence and fixed invalidation relation remain
|
||||
application-owned and non-configurable.
|
||||
- Stages receive resolved Narratio-owned configuration and do not implement
|
||||
import, profile, or family semantics independently.
|
||||
- External tools continue to own their private runtime configuration and
|
||||
defaults.
|
||||
- Raw secrets remain indirect and are never introduced into composed YAML,
|
||||
effective-configuration output, manifests, or diagnostics.
|
||||
|
||||
Configuration composition belongs to `internal/config`. CLI selection and
|
||||
production loading belong to `internal/app`. Party parsing must be a
|
||||
Narratio-owned campaign-input contract. Artifact-family expansion must finish
|
||||
before existing configured-artifact validation, catalog construction,
|
||||
planning, reconciliation, or execution begins.
|
||||
|
||||
## Configuration Imports
|
||||
|
||||
### Root document
|
||||
|
||||
Keep `pipeline.yml` as the selected pipeline entry point. Add an optional
|
||||
top-level composition envelope with explicitly ordered imports:
|
||||
|
||||
```yaml
|
||||
composition:
|
||||
imports:
|
||||
- conf.d/platform.yml
|
||||
- conf.d/transcripts.yml
|
||||
- conf.d/extraction.yml
|
||||
- conf.d/artifacts.yml
|
||||
- conf.d/publish.yml
|
||||
|
||||
default_profile: production
|
||||
|
||||
profiles:
|
||||
production:
|
||||
overlay: profiles/production.yml
|
||||
testing:
|
||||
overlay: profiles/testing.yml
|
||||
```
|
||||
|
||||
An existing monolithic `pipeline.yml` without `composition` remains valid and
|
||||
retains its current behavior. The root document may contain ordinary pipeline
|
||||
fields alongside `composition`, allowing incremental migration.
|
||||
|
||||
`imports` is the public term. Imported files are partial pipeline
|
||||
configuration documents rather than independently runnable pipelines.
|
||||
|
||||
### Import boundaries
|
||||
|
||||
- Only the root pipeline document may declare `composition`, imports, or
|
||||
profiles. Imported and overlay documents cannot import other documents.
|
||||
- Every import is explicit. Narratio does not automatically scan `conf.d` or
|
||||
interpret unlisted files.
|
||||
- Import paths are relative to the root pipeline file's directory, must remain
|
||||
confined beneath that directory, and must resolve to supported regular YAML
|
||||
files. Absolute paths, traversal, non-regular files, and cycles are rejected.
|
||||
- All ordinary relative pipeline paths retain one stable base: the root
|
||||
pipeline file's directory. Moving a field between imported files must not
|
||||
silently change the meaning of its relative path.
|
||||
- The declared import order is retained for diagnostics and provenance, but it
|
||||
is not an implicit precedence mechanism between base imports.
|
||||
- Composition metadata is removed before the effective mapping is decoded as
|
||||
strict `PipelineConfig` data.
|
||||
|
||||
### Additive merge contract
|
||||
|
||||
Base configuration from the root document and its imports is additive:
|
||||
|
||||
- mappings merge recursively when their child fields are disjoint;
|
||||
- two files may therefore contribute different entries beneath a shared map
|
||||
such as `scriptorium.artifacts`;
|
||||
- defining the same final field or keyed entry in more than one base source is
|
||||
an error, even when the values are identical;
|
||||
- lists are atomic values and cannot be contributed more than once at the same
|
||||
base configuration path;
|
||||
- mapping/scalar, mapping/list, and other incompatible type collisions are
|
||||
errors; and
|
||||
- duplicate YAML keys within one source remain errors.
|
||||
|
||||
Errors must identify the complete configuration path and every source file
|
||||
that claims it. This contract prevents an import reorder or newly added file
|
||||
from silently changing production behavior.
|
||||
|
||||
Composition must operate on a presence-aware YAML representation before Go
|
||||
struct decoding. Merging already decoded structs is not sufficient because it
|
||||
cannot reliably distinguish omission from explicitly configured `false`, zero,
|
||||
an empty collection, or another meaningful zero value.
|
||||
|
||||
## Named Profiles
|
||||
|
||||
### Selection and precedence
|
||||
|
||||
Exactly one named profile may be active for an invocation. A composition root
|
||||
may define `default_profile`; the maintained production-shaped configuration
|
||||
uses `production`. Add `--profile <name>` to the common configuration flags for
|
||||
every command that loads pipeline configuration:
|
||||
|
||||
```bash
|
||||
narratio run SESSION
|
||||
narratio run SESSION --profile testing
|
||||
narratio session plan SESSION --profile testing
|
||||
narratio regenerate-artifacts SESSION --profile testing
|
||||
```
|
||||
|
||||
The explicit CLI value overrides `default_profile`. An unknown profile, a
|
||||
profile selection against a configuration that defines no such profile, an
|
||||
empty profile name, or a default naming an undefined profile is an error before
|
||||
stage composition. Do not add an environment-variable selector in this
|
||||
feature; profile selection should remain visible in the command or root
|
||||
configuration.
|
||||
|
||||
Only one profile can be selected. Profiles cannot extend other profiles, and
|
||||
callers cannot stack several profiles. A single overlay provides the required
|
||||
production/testing variation without creating user-programmable precedence.
|
||||
|
||||
### Overlay semantics
|
||||
|
||||
The selected profile is the sole intentional override layer:
|
||||
|
||||
- mappings recursively merge with the assembled base mapping;
|
||||
- scalar and boolean leaves replace base leaves;
|
||||
- lists replace the complete base list rather than concatenate implicitly;
|
||||
- keyed maps such as configured artifacts merge by key, allowing a testing
|
||||
profile to add experimental artifacts;
|
||||
- an existing artifact can be disabled explicitly with `enabled: false`;
|
||||
- an overlay cannot change a value's YAML kind incompatibly; and
|
||||
- the fully overlaid result undergoes the same strict decode, defaults, path
|
||||
resolution, expansion, and validation as a monolithic configuration.
|
||||
|
||||
Generic deletion syntax is outside this feature. Shared definitions should
|
||||
live in base imports, while production-only and testing-only definitions live
|
||||
in their corresponding overlays. Existing enable/disable fields and complete
|
||||
list replacement cover the required cases without a YAML patch language.
|
||||
|
||||
### Effective identity and provenance
|
||||
|
||||
Every loaded configuration must retain bounded, non-secret provenance:
|
||||
|
||||
- the root pipeline path;
|
||||
- the selected profile name, including whether it came from the default or
|
||||
CLI;
|
||||
- the ordered imported and overlay source paths;
|
||||
- a deterministic digest of the normalized effective pipeline configuration;
|
||||
and
|
||||
- enough source ownership information to explain composition errors and
|
||||
effective values.
|
||||
|
||||
Run reporting and invocation state should identify the selected profile and
|
||||
effective configuration digest. The profile name is provenance, not a blanket
|
||||
cache key: changing profiles must invalidate only work whose result-affecting
|
||||
effective configuration changed.
|
||||
|
||||
## Canonical Campaign Party Contract
|
||||
|
||||
### Ownership and schema
|
||||
|
||||
Make `party.yml` a strict, versioned Narratio campaign-input contract. A
|
||||
canonical file has this shape:
|
||||
|
||||
```yaml
|
||||
schema_version: narratio.party.v1
|
||||
|
||||
characters:
|
||||
arannis:
|
||||
player:
|
||||
name: Eric
|
||||
character:
|
||||
name: Arannis
|
||||
alias:
|
||||
- Ari
|
||||
- The Grey Owl
|
||||
classes:
|
||||
- name: wizard
|
||||
level: 8
|
||||
|
||||
brenna:
|
||||
player:
|
||||
name: Jane
|
||||
character:
|
||||
name: Brenna
|
||||
classes:
|
||||
- name: paladin
|
||||
level: 6
|
||||
- name: warlock
|
||||
level: 2
|
||||
```
|
||||
|
||||
The contract has these semantics:
|
||||
|
||||
- `schema_version` is required and must equal `narratio.party.v1`.
|
||||
- `characters` is a non-empty mapping.
|
||||
- Each mapping key is the stable `character_id` and must satisfy the existing
|
||||
configured-artifact key grammar.
|
||||
- `player.name` and `character.name` are required, non-empty display strings.
|
||||
- `character.alias` is optional and accepts a list of zero or more non-empty
|
||||
alias strings. The singular field spelling `alias` is intentional.
|
||||
- Leading or trailing whitespace and control characters in names, aliases, and
|
||||
class names are rejected rather than silently normalized.
|
||||
- A character alias cannot equal that character's primary name, another alias,
|
||||
or another character's primary name or alias under case-insensitive
|
||||
comparison. This keeps party-member grounding unambiguous. Player names may
|
||||
repeat because one player may control more than one character.
|
||||
- `character.classes` is required and non-empty. Every entry has a required
|
||||
free-form `name` and an optional positive integer `level`.
|
||||
- Class names are not restricted to a Narratio-owned D&D enumeration.
|
||||
- Duplicate class names for one character are rejected case-insensitively.
|
||||
- Class order and alias order are preserved.
|
||||
- Narratio derives a deterministic class summary such as
|
||||
`paladin 6 / warlock 2` and an alias summary that joins declared aliases with
|
||||
`, ` in their declared order.
|
||||
- Unknown fields and trailing YAML documents are rejected.
|
||||
|
||||
The stable `character_id`, rather than a display name, owns generated artifact
|
||||
identity. Changing a display name, alias, class, or level retains that identity
|
||||
and changes the relevant semantic inputs. Changing the mapping key is an
|
||||
intentional remove-and-add operation.
|
||||
|
||||
### Load and prepare behavior
|
||||
|
||||
The selected campaign's effective `party_file` resolves relative to
|
||||
`campaign.yml`. Narratio must load and validate canonical party data during
|
||||
combined pipeline/campaign resolution so `session plan`, configuration
|
||||
inspection, and artifact-family expansion see the same roster before stage
|
||||
execution.
|
||||
|
||||
`prepare` continues to materialize the canonical document as
|
||||
`inputs/party.yml`, record its source and content identity, and expose it as
|
||||
`narratio.input.party`. The same canonical file is passed unchanged to
|
||||
Notarius's `party` reference slot, which currently accepts YAML reference
|
||||
material without imposing a competing roster schema.
|
||||
|
||||
### Players projection and compatibility
|
||||
|
||||
The canonical party document becomes the eventual single source of truth for
|
||||
player-to-character relationships. Narratio derives a deterministic,
|
||||
documented players-only YAML projection during prepare, materializes it at
|
||||
`inputs/players.yml`, records its content identity, and exposes it through the
|
||||
existing `narratio.input.players` source. This preserves the separate Notarius
|
||||
`players` reference and existing Scriptorium source without requiring every
|
||||
campaign to maintain duplicate facts.
|
||||
|
||||
The projection uses this versioned shape, with entries sorted by stable
|
||||
character ID:
|
||||
|
||||
```yaml
|
||||
schema_version: narratio.players.v1
|
||||
players:
|
||||
- name: Eric
|
||||
character:
|
||||
id: arannis
|
||||
name: Arannis
|
||||
alias:
|
||||
- Ari
|
||||
- The Grey Owl
|
||||
```
|
||||
|
||||
There is one projection entry per character rather than one grouped entry per
|
||||
display player name. This permits one player to control several characters and
|
||||
does not conflate distinct players who happen to share a display name.
|
||||
|
||||
Backward compatibility must be narrow and removable:
|
||||
|
||||
- an unversioned legacy party file continues to be treated as opaque reference
|
||||
material when the existing `players_file` is also configured;
|
||||
- legacy mode preserves the current prepared `party` and `players` behavior
|
||||
but cannot drive artifact-family expansion;
|
||||
- configuring a party-driven artifact family with a legacy roster produces a
|
||||
clear migration error;
|
||||
- canonical `narratio.party.v1` mode derives players and rejects a separate
|
||||
`players_file`, preventing contradictory authorities;
|
||||
- parsing, validation, documentation, and tests for legacy mode live behind a
|
||||
clearly identified compatibility boundary with comments stating that it is
|
||||
intended for removal after migration; and
|
||||
- the maintained examples migrate to canonical mode and demonstrate no
|
||||
separate players file.
|
||||
|
||||
The derived players projection is a durable integration contract and must be
|
||||
documented under `docs/integrations/` rather than left as incidental generated
|
||||
YAML.
|
||||
|
||||
## Party-Driven Artifact Families
|
||||
|
||||
### Configuration model
|
||||
|
||||
Add `scriptorium.artifact_families`, separate from the existing concrete
|
||||
`scriptorium.artifacts` map:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifact_families:
|
||||
character_meta:
|
||||
enabled: true
|
||||
for_each: party.characters
|
||||
prompt_id: dnd.character_meta
|
||||
profile_id: production
|
||||
output_path_pattern: artifacts/characters/{character_id}/meta.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
member_vars:
|
||||
player_name: player.name
|
||||
character_name: character.name
|
||||
character_class: character.class_summary
|
||||
character_aliases: character.alias_summary
|
||||
|
||||
character_items:
|
||||
enabled: true
|
||||
for_each: party.characters
|
||||
prompt_id: dnd.character_items
|
||||
profile_id: production
|
||||
output_path_pattern: artifacts/characters/{character_id}/items.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
item_occurrences:
|
||||
source: narratio.extraction.item_occurrences
|
||||
required: true
|
||||
member_vars:
|
||||
character_name: character.name
|
||||
```
|
||||
|
||||
Families support the ordinary result-affecting fields shared by concrete
|
||||
Scriptorium artifacts: enablement, prompt and profile IDs, timeout,
|
||||
render-debug policy, ordinary dependencies, inputs, and static variables.
|
||||
`output_path_pattern` replaces concrete `output_path` at the family level.
|
||||
|
||||
`for_each` is not an expression language. This feature accepts only the exact
|
||||
source `party.characters`. `member_vars` maps a Scriptorium variable name to
|
||||
one of a closed set of canonical values:
|
||||
|
||||
- `character_id`;
|
||||
- `player.name`;
|
||||
- `character.name`;
|
||||
- `character.class_summary`; and
|
||||
- `character.alias_summary`.
|
||||
|
||||
Static `vars` and resolved member variables merge deterministically. Duplicate
|
||||
variable names across the two maps are rejected rather than assigned implicit
|
||||
precedence. Narratio's existing reserved sticky-session variable remains
|
||||
application-owned and is applied after expansion under its current rules.
|
||||
|
||||
### Narrow substitution
|
||||
|
||||
`{character_id}` is the only family substitution supported in this feature.
|
||||
It is required in `output_path_pattern` and may appear exactly where documented
|
||||
for family-owned publish destinations. Unknown, repeated in an invalid
|
||||
position, unresolved, or malformed substitutions are errors. Names, aliases,
|
||||
classes, environment values, and arbitrary YAML paths cannot be interpolated
|
||||
into configuration strings.
|
||||
|
||||
### Concrete expansion
|
||||
|
||||
For each family and party character, resolution creates one ordinary concrete
|
||||
artifact key by joining the normalized family key, an underscore, and the
|
||||
stable character ID:
|
||||
|
||||
```text
|
||||
character_meta_arannis
|
||||
character_items_arannis
|
||||
```
|
||||
|
||||
Expansion must be sorted, deterministic, and completed before existing
|
||||
configured-artifact defaults and validation. Every generated artifact then
|
||||
uses the existing catalog, dependency planner, fingerprinting, manifest,
|
||||
reconciliation, execution, materialization, and publish boundaries.
|
||||
|
||||
Validation rejects:
|
||||
|
||||
- invalid family keys or character IDs;
|
||||
- generated keys that fail the existing configured-artifact grammar;
|
||||
- collisions between generated artifacts, explicit concrete artifacts, or
|
||||
family names;
|
||||
- duplicate or escaping output paths;
|
||||
- a family with no canonical party source;
|
||||
- invalid member-variable selectors;
|
||||
- a selected profile that leaves an executable family incomplete; and
|
||||
- any expanded artifact that fails ordinary concrete-artifact validation.
|
||||
|
||||
An added party character creates missing concrete artifacts. Removing a
|
||||
character removes those artifacts from effective configuration so existing
|
||||
analyze reconciliation classifies their old records as removed. A change to a
|
||||
member field changes only fingerprints that actually consume its resolved
|
||||
value or the party document as an input.
|
||||
|
||||
### Dependencies between families
|
||||
|
||||
Ordinary `depends_on` continues to name shared concrete artifacts such as
|
||||
`session_recap`. Add a typed `member_dependencies` list for corresponding
|
||||
members of another family:
|
||||
|
||||
```yaml
|
||||
character_items:
|
||||
member_dependencies:
|
||||
- character_meta
|
||||
```
|
||||
|
||||
For `arannis`, this expands to a dependency on
|
||||
`character_meta_arannis`. The referenced family must use the same party source
|
||||
and generate the same character ID. Cycles and missing members are rejected by
|
||||
the existing concrete dependency validation after expansion.
|
||||
|
||||
When a family input consumes the corresponding member output, provide a narrow
|
||||
pre-expansion member-artifact source form owned by configuration resolution.
|
||||
The syntax is:
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
prior_meta:
|
||||
source: narratio.member_artifact.character_meta
|
||||
required: true
|
||||
```
|
||||
|
||||
For the `arannis` member, this resolves to
|
||||
`narratio.artifact.character_meta_arannis` before ordinary runtime validation.
|
||||
The referenced family must also appear in `member_dependencies`. The
|
||||
`narratio.member_artifact.*` form must not survive into stage or adapter
|
||||
configuration as a new runtime artifact kind.
|
||||
|
||||
### Selection and reporting
|
||||
|
||||
Extend artifact selection so an exact family key selects all of that family's
|
||||
concrete members. An exact generated key selects only that member. Explicitly
|
||||
selecting a disabled family follows the existing rule for explicitly selected
|
||||
disabled concrete artifacts: all generated targets must still have valid
|
||||
executable fields. Mixed family and concrete selections are normalized and
|
||||
deduplicated after expansion.
|
||||
|
||||
Planning, run summaries, artifact listing, and effective-configuration output
|
||||
must show both family origin and concrete artifact identity. Execution remains
|
||||
entirely concrete; a family is not a dynamic pipeline stage or runtime loop.
|
||||
|
||||
### Family publish rules
|
||||
|
||||
Allow a family to declare one typed publish policy so operators need not repeat
|
||||
one publish output per character:
|
||||
|
||||
```yaml
|
||||
publish:
|
||||
enabled: true
|
||||
required: false
|
||||
dest_pattern: artifacts/characters/{character_id}/meta.md
|
||||
```
|
||||
|
||||
Resolution expands this into ordinary concrete publish rules. When
|
||||
`dest_pattern` is omitted, the generated artifact's output path is used where
|
||||
the existing publish contract permits derivation. Conflicts with explicit
|
||||
publish rules, duplicate destinations, unsafe paths, and generated sources
|
||||
that are unavailable are rejected by the existing publish policy after
|
||||
expansion. Family publish behavior must not introduce wildcard source matching
|
||||
at runtime.
|
||||
|
||||
## Profile-Safe Resume And Invalidation
|
||||
|
||||
Named profiles are safe only if result-affecting model and configuration
|
||||
changes cannot silently reuse incompatible successful stage results. Artifact
|
||||
analysis already has a versioned semantic fingerprint that includes effective
|
||||
prompt/profile identifiers, variables, dependencies, inputs, and output
|
||||
identity. The broader pipeline requires the same principle at every applicable
|
||||
stage boundary.
|
||||
|
||||
Audit each stage's resume contract and add a versioned semantic configuration
|
||||
fingerprint wherever current evidence does not already cover all
|
||||
Narratio-observable result-affecting settings. In particular, switching an
|
||||
Audita model through a profile must not reuse polish output produced by the
|
||||
testing model. Conversely, changing only an operational timeout, binary path,
|
||||
workspace path, diagnostic location, or profile name must not invalidate
|
||||
byte-equivalent semantic work unless that value genuinely affects the stage's
|
||||
canonical result.
|
||||
|
||||
The selected profile name and complete effective-config digest are retained as
|
||||
provenance. Resume decisions use stage-specific effective semantics rather
|
||||
than the profile name as a blanket invalidator. This permits production and
|
||||
testing profiles to share unaffected transcript work while correctly staling
|
||||
changed stages and their fixed transitive dependents.
|
||||
|
||||
Private configuration, prompt, model, executable, and module contents loaded
|
||||
inside an external tool remain outside Narratio's observable fingerprint
|
||||
boundary unless an existing integration contract exposes their identity.
|
||||
Changing such private inputs continues to require explicit force, and the
|
||||
documentation must not claim otherwise.
|
||||
|
||||
## Configuration Inspection
|
||||
|
||||
Add a `config` command family that performs composition without running a
|
||||
session:
|
||||
|
||||
```bash
|
||||
narratio config validate [--config <pipeline.yml>] [--profile <name>]
|
||||
narratio config show [--config <pipeline.yml>] [--profile <name>]
|
||||
narratio config sources [--config <pipeline.yml>] [--profile <name>]
|
||||
narratio config diff <left-profile> <right-profile> [--config <pipeline.yml>]
|
||||
```
|
||||
|
||||
Required behavior:
|
||||
|
||||
- `validate` composes imports and the selected profile, resolves campaign data
|
||||
when required for party-driven families, expands artifacts and publish rules,
|
||||
applies defaults, and runs strict validation without creating session or run
|
||||
state.
|
||||
- `show` emits deterministic normalized effective YAML, including generated
|
||||
concrete artifacts, without raw credentials or environment values.
|
||||
- `sources` reports root, import, profile, campaign, and party provenance and
|
||||
identifies which source owns effective configuration paths.
|
||||
- `diff` compares normalized effective configurations rather than raw files
|
||||
and clearly identifies additions, removals, and value changes.
|
||||
- output order is deterministic and suitable for review, but output formatting
|
||||
is not a substitute for the effective configuration digest.
|
||||
|
||||
If family expansion requires a campaign, these commands accept the existing
|
||||
campaign selection mechanisms or report clearly that pipeline-only validation
|
||||
cannot complete the party-dependent portion. They must not silently choose an
|
||||
unintended campaign.
|
||||
|
||||
## Required Application Changes
|
||||
|
||||
The implemented feature will require coordinated changes across existing
|
||||
owners rather than a parallel configuration subsystem:
|
||||
|
||||
- `internal/config`: presence-aware YAML composition, import confinement,
|
||||
profile overlays, provenance, effective digesting, party parsing and
|
||||
validation, players projection, family expansion, defaults, and final strict
|
||||
validation;
|
||||
- `internal/app`: `--profile` plumbing through every configuration-consuming
|
||||
command, `config` inspection commands, campaign-aware inspection, profile
|
||||
reporting, and one shared loading path;
|
||||
- `internal/artifactpolicy` and `internal/artifacts`: generated concrete source
|
||||
registration, family-aware selection metadata, derived-player identity, and
|
||||
reuse of the existing configured-artifact grammar and path policy;
|
||||
- `internal/stage`: consumption only of expanded concrete artifacts, party and
|
||||
derived-player preparation, and stage-specific semantic resume evidence;
|
||||
- `internal/manifest`: bounded profile/effective-config provenance and any
|
||||
versioned stage semantic fingerprints required for safe reuse;
|
||||
- publish planning: expansion of family publish declarations into existing
|
||||
concrete rules before ordinary validation and execution;
|
||||
- Notarius composition: continue passing prepared `party` and `players`
|
||||
sources, with canonical mode sourcing both from the party-owned contract;
|
||||
and
|
||||
- Scriptorium composition: receive only resolved concrete artifacts and
|
||||
string/bool variables under the existing adapter contract.
|
||||
|
||||
There must be one production configuration-loading path. Remote session
|
||||
loading, restore, status, planning, helper commands, and execution must not
|
||||
independently reimplement import, profile, campaign, or family selection.
|
||||
|
||||
## Documentation And Examples
|
||||
|
||||
When behavior is implemented, update the canonical owners in the same change:
|
||||
|
||||
- [`docs/config.md`](../config.md): import, merge, profile, party selection,
|
||||
family fields, defaults, and validation contracts;
|
||||
- [`docs/cli.md`](../cli.md): `--profile` and `config` command syntax and output
|
||||
behavior;
|
||||
- [`docs/operations.md`](../operations.md): recommended production/testing
|
||||
bundle layout, migration, inspection, and profile-switching workflow;
|
||||
- [`docs/policy/architecture.md`](../policy/architecture.md): only the durable
|
||||
composition, campaign-data ownership, and semantic-resume invariants;
|
||||
- `docs/integrations/`: the canonical party and derived players YAML contracts;
|
||||
- [`docs/internal/overview.md`](../internal/overview.md) and focused internal
|
||||
configuration, manifest, analyze, prepare, and publish documents: implemented
|
||||
ownership and mechanics; and
|
||||
- `examples/`: a validated split configuration bundle, production/testing
|
||||
overlays, canonical party roster with aliases and multiclass data, at least
|
||||
two party-driven artifact families, and representative family publishing.
|
||||
|
||||
Do not copy complete example bundles into reference prose. Maintained examples
|
||||
must be secret-free and loaded by the repository's example validation test.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Testing should protect the public contracts and high-risk composition and
|
||||
state-reuse behavior without coupling to private merge helpers. Required
|
||||
confidence includes:
|
||||
|
||||
- monolithic configuration compatibility;
|
||||
- recursive mapping composition and all duplicate/type/list conflict classes;
|
||||
- import confinement, missing files, non-regular files, unsupported nesting,
|
||||
source-aware diagnostics, deterministic ordering, and duplicate YAML keys;
|
||||
- explicit zero-value profile overrides, list replacement, keyed-map additions,
|
||||
unknown profiles, invalid defaults, and absence of profile stacking;
|
||||
- canonical party schema validation, aliases, alias/name ambiguity, multiclass
|
||||
summaries, stable IDs, malformed data, and legacy/canonical mode separation;
|
||||
- deterministic players projection and prepared-input provenance;
|
||||
- deterministic multi-family expansion, generated identities and paths,
|
||||
variable binding, same-member dependencies, collisions, additions,
|
||||
removals, and family selection;
|
||||
- expansion into ordinary analyze planning, fingerprints, manifests, partial
|
||||
selection, failure handling, and publish rules;
|
||||
- model/profile changes staling only stages whose semantic configuration
|
||||
changed while unaffected transcript work remains reusable;
|
||||
- representative CLI validation, show, sources, diff, plan, and run behavior;
|
||||
and
|
||||
- maintained monolithic, imported, production, and testing examples loading
|
||||
and validating offline.
|
||||
|
||||
Use package-level behavioral tests through the narrowest stable owner. A small
|
||||
number of assembled workflow tests should prove that composed profiles and
|
||||
party-driven families reach the existing pipeline correctly; higher-level
|
||||
tests should not repeat every parser and validation case.
|
||||
|
||||
## Compatibility And Migration
|
||||
|
||||
- Existing monolithic pipeline files remain supported.
|
||||
- Omitting `--profile` retains existing behavior for configurations without
|
||||
profiles and selects the declared production default for composed bundles.
|
||||
- Existing explicit concrete Scriptorium artifacts and publish rules remain
|
||||
valid and may coexist with non-colliding families.
|
||||
- Legacy opaque party plus players files remain temporarily usable only through
|
||||
the isolated compatibility mode and cannot power party-driven families.
|
||||
- Migration consists of converting the campaign party document to
|
||||
`narratio.party.v1`, removing the separate players file, and then enabling
|
||||
party-driven families.
|
||||
- Configuration inspection must make the migration result reviewable before a
|
||||
session run.
|
||||
- Compatibility code and documentation must clearly identify the eventual
|
||||
removal boundary; new features must target canonical party mode rather than
|
||||
extend legacy semantics.
|
||||
|
||||
## Non-Goals And Deferred Work
|
||||
|
||||
This feature does not introduce:
|
||||
|
||||
- automatic loading of every file in `conf.d`;
|
||||
- recursive imports, imports in profile overlays, or profile inheritance;
|
||||
- multiple simultaneously active profiles;
|
||||
- environment-selected profiles;
|
||||
- generic YAML patch/delete operations;
|
||||
- arbitrary string interpolation, expressions, scripting, or loops;
|
||||
- artifact expansion from arbitrary files, Notarius outputs, or runtime data;
|
||||
- configurable pipeline stages or a workflow DAG;
|
||||
- runtime wildcard artifact or publish sources;
|
||||
- automatic hashing of private external-tool configuration, prompts, modules,
|
||||
models, or executables; or
|
||||
- immediate removal of the isolated legacy party/player compatibility path.
|
||||
|
||||
Named reusable parameter sets beyond the canonical party roster may be
|
||||
considered later if a second non-party use case demonstrates a need. Additional
|
||||
party-member types, campaign-domain fields, or family iteration sources should
|
||||
be added only through a versioned schema decision rather than an untyped
|
||||
extension map.
|
||||
|
||||
## Target End State
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
- operators can organize a pipeline as a root file plus explicit additive
|
||||
imports with deterministic, source-aware conflict errors;
|
||||
- one production-default or explicitly selected testing profile applies a
|
||||
deliberate override without duplicating stable configuration;
|
||||
- every command uses the same resolved profile and reports its identity;
|
||||
- a campaign owns one strict, versioned party roster containing stable
|
||||
character IDs, players, character names, optional alias lists, and one or
|
||||
more classes;
|
||||
- Narratio derives the existing players input from that roster and supplies the
|
||||
canonical campaign context to Notarius;
|
||||
- multiple Scriptorium artifact families expand once per campaign character
|
||||
into ordinary concrete artifacts, dependencies, variables, output paths, and
|
||||
optional publish rules;
|
||||
- adding, changing, or removing a character produces deterministic and
|
||||
manifest-correct artifact reconciliation;
|
||||
- profile model changes cannot silently reuse semantically incompatible stage
|
||||
outputs and do not unnecessarily invalidate unaffected transcript work;
|
||||
- operators can validate, display, trace, and compare effective configuration
|
||||
without executing a session;
|
||||
- existing monolithic and concrete-artifact configurations remain compatible;
|
||||
and
|
||||
- current user, operator, integration, architecture, internal, and example
|
||||
documentation accurately owns the implemented contracts.
|
||||
Reference in New Issue
Block a user