From c128970f58ea738c1a92d6e1986fc770df8aea60 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 19 May 2026 21:21:06 -0500 Subject: [PATCH] Updated documentation to remove the completed runtime artifacts roadmap and add a new restore subcommand roadmap --- docs/roadmap/restore.md | 715 ++++++++++++++++++++++++++++ docs/roadmap/runtime-artifacts.md | 762 ------------------------------ 2 files changed, 715 insertions(+), 762 deletions(-) create mode 100644 docs/roadmap/restore.md delete mode 100644 docs/roadmap/runtime-artifacts.md diff --git a/docs/roadmap/restore.md b/docs/roadmap/restore.md new file mode 100644 index 0000000..0807d15 --- /dev/null +++ b/docs/roadmap/restore.md @@ -0,0 +1,715 @@ +# Roadmap: `narratio restore` Subcommand + +## Status + +Planned. This document is implementation guidance for 5.3-Codex. + +## Summary + +Add a new `narratio restore` subcommand that hydrates a local session workspace from the current committed remote archive state. + +The primary operator workflow is: + +```bash +narratio restore --session-id 2026-04-04 +narratio run-stage --force analyze +``` + +This should allow a new machine with no local workspace state to restore the durable session manifest, transcripts, and generated artifacts from S3, then generate new Scriptorium artifacts without re-running transcription, merge, normalize, polish, or trim. + +This is intentionally a separate command. Do not fold this behavior into the `prepare` stage. The existing `prepare` stage should remain focused on materializing configured local/S3 inputs for a pipeline run. + +## Goals + +- Add a first-class `narratio restore` command. +- Restore the current committed remote session state into the canonical local session workspace. +- Use the existing object storage adapter boundary. +- Preserve archive commit semantics: only restore from a remote state that has a valid current commit marker. +- Restore durable session-level outputs needed for downstream stages, especially `analyze`. +- Provide safe conflict behavior by default. +- Support `--dry-run`, `--force`, and `--include-audio`. +- Keep the implementation explicit, testable, and narrow. + +## Non-goals + +- Do not make `restore` a pipeline stage. +- Do not change the `prepare` stage behavior as part of this work. +- Do not add implicit restore behavior to `narratio run` in this implementation. +- Do not restore historical run-local sandboxes by default. +- Do not implement a generic remote synchronization engine. +- Do not implement bidirectional sync. +- Do not delete local files merely because they are absent remotely. +- Do not merge remote and local manifests in the first implementation. +- Do not require live S3 for the ordinary unit test suite. + +## Future work explicitly out of scope + +A future change may add: + +```bash +narratio run --restore +``` + +That future flag should run `narratio restore` before starting the normal pipeline. Mention this as future work in roadmap/docs if useful, but do not implement it now. + +## Existing architecture to preserve + +### `prepare` remains input materialization + +The `prepare` stage currently materializes required session inputs into canonical local workspace paths and records input provenance. It owns local copying/materialization of config and audio inputs, including S3 audio download when `session.inputs.audio_s3.prefix` is configured. It does not own transcript generation/processing or archive publish behavior. + +`restore` should not be implemented by expanding `prepare`. It should be an app-level command that reuses shared helpers where appropriate. + +### Workspace model + +The local durable session workspace is campaign-aware: + +```text +{workspace.root}/work/{campaign}/{session_id}/ +``` + +It contains durable session paths such as: + +```text +manifest.json +inputs/ +audio/ +transcripts/ +artifacts/ +reports/ +logs/ +config/ +current/ +runs/ +``` + +Run-local sandboxes live below: + +```text +runs/{run_id}/ +``` + +Restore should target durable session-level paths, not old run-local stage sandboxes. + +### Storage boundary + +The storage adapter owns object-store primitives only: `List`, `Download`, `Upload`, and `Exists`. + +The storage adapter must not infer root prefixes, campaign names, session IDs, run IDs, or archive layout. Restore code must construct full bucket-relative keys before calling storage. + +### Archive commit boundary + +A remote run is current only after the archive stage has uploaded the run record, promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`. + +`current/run_id.txt` is the final remote commit marker and must be written last. + +Restore must not treat incomplete, skipped, failed, or uncommitted archive attempts as current remote state. + +## User-facing command + +Add: + +```bash +narratio restore [flags] +``` + +The command should use the same configuration/session discovery conventions as `run`, `plan`, `resume`, and `run-stage` where practical: + +```bash +narratio restore --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-04-04 +``` + +Required effective inputs: + +- resolved pipeline config; +- resolved session config; +- `session.campaign`; +- `session.session_id`; +- configured remote storage backend. + +Supported flags: + +```text +--config Existing pipeline config path behavior. +--session Existing session config path behavior. +--session-id Existing session template behavior. +--dry-run Plan restore actions without writing local files. +--force Overwrite conflicting local files with remote files. +--include-audio Include archived session-level audio files. +``` + +Do not add `--restore` to `run` in this implementation. + +## Default restore scope + +By default, restore: + +1. Validates and reads the current remote commit marker. +2. Downloads the current remote manifest into the local session manifest path. +3. Downloads durable transcript files. +4. Downloads durable generated artifact files. + +Default included remote/local durable paths: + +```text +manifest.json from remote current manifest +transcripts/** +artifacts/** +``` + +Default excluded paths: + +```text +audio/** unless --include-audio is passed +runs/** always excluded for this implementation +logs/** excluded for this implementation +reports/** excluded for this implementation unless needed for current manifest validation +config/** excluded for this implementation +inputs/** excluded for this implementation +current/** remote control metadata only; do not mirror blindly +``` + +If the existing archive implementation stores promoted files in a different remote layout, use the existing archive/path helpers and current archive semantics rather than inventing a parallel layout. + +## Remote state discovery + +Implement restore around the current committed archive state. + +Expected algorithm: + +1. Resolve pipeline/session config. +2. Ensure storage is configured. +3. Ensure local workspace layout exists. +4. Acquire the session lock. +5. Build the remote session archive prefix using the same helpers/policy used by archive code. +6. Check for the remote `current/run_id.txt` commit marker. +7. Read the committed run ID. +8. Download `current/manifest.json` to a temporary file. +9. Validate that the manifest is parseable and belongs to the requested campaign/session. +10. Build a restore plan from the committed remote state. +11. Execute the restore plan unless `--dry-run` is set. +12. Emit a concise summary. + +Important: `current/run_id.txt` is the commit marker. Do not restore from a remote session prefix merely because files exist under `transcripts/` or `artifacts/`. + +## Restore planning + +Create a planning layer before writing files. + +A restore plan entry should include at least: + +```go +type RestoreAction struct { + Kind RestoreActionKind + RemoteKey string + LocalPath string + Size int64 + ETag string + ExistsLocal bool + SameLocal bool + Conflict bool + Reason string +} +``` + +Suggested action kinds: + +```text +download +skip_same +skip_missing_optional +conflict +``` + +The restore planner should be deterministic: + +- sort remote objects by key; +- sort planned actions by local path or stable restore priority; +- write/report stable output for tests. + +## Conflict and overwrite policy + +Default behavior should be safe. + +For each planned file: + +```text +local absent: + download + +local present and same as remote: + skip + +local present and different: + conflict; fail restore unless --force is set + +--force: + overwrite local conflicting files with remote versions + +--dry-run: + do not write any files; report what would happen +``` + +The first implementation may use size and checksum/hash comparison where available. If remote ETag cannot be treated as a content hash, compare by downloading to a temporary file and hashing locally before deciding whether a local file is the same. Prefer correctness over assuming provider-specific ETag semantics. + +Do not delete local files that are not present remotely. + +## File writing and transactionality + +Restore should avoid partial writes. + +Implementation requirements: + +- download each remote object to a temporary file under the session workspace or OS temp dir; +- validate downloaded content where possible before replacing local files; +- create parent directories as needed; +- atomically rename/copy into place only after successful download; +- do not overwrite local files unless `--force` is set; +- if a later file fails, preserve already-restored files but return a failure summary; +- never corrupt an existing local manifest on failed manifest download/parse. + +Manifest restore is especially sensitive: + +- download remote `current/manifest.json` to a temporary file; +- parse and validate it; +- if no local manifest exists, install it; +- if a local manifest exists and is equivalent, skip; +- if a local manifest exists and differs, fail unless `--force` is set; +- with `--force`, replace the local manifest with the remote manifest after validation; +- do not attempt a manifest merge in the initial implementation. + +## Manifest semantics + +`restore` is not a pipeline run and should not mark stages as running/succeeded/failed. + +The restored remote manifest becomes the local session manifest. That is what allows a subsequent command such as: + +```bash +narratio run-stage --force analyze +``` + +to see existing upstream stage state and canonical durable outputs. + +Do not create a new run manifest for `restore`. + +It is acceptable to write a restore diagnostic report outside the manifest, for example: + +```text +reports/restore-latest.json +``` + +or a timestamped report, if that pattern fits the existing codebase. The report must not contain secrets. + +## Local workspace locking + +`restore` should acquire the same session lock used by ordinary pipeline operations before modifying session workspace state. + +If the lock is held, fail fast with the same lock-conflict behavior used elsewhere. + +`--dry-run` may still acquire the lock for consistency, but it is acceptable to avoid the lock if the codebase already has a clear read-only command pattern. Prefer safety and simplicity. + +## Audio behavior + +By default, do not restore audio. + +If `--include-audio` is passed: + +- restore archived durable session-level audio files only; +- do not use run-scoped spool paths; +- do not mutate or delete spool state; +- do not infer original `session.inputs.audio_s3.prefix` behavior; +- respect the same conflict/force/dry-run behavior used for transcripts/artifacts. + +If the archive does not contain durable audio files, `--include-audio` should report that no archived audio was found rather than failing, unless the final implementation chooses to treat explicit audio restore as required. Prefer non-failure for absent archived audio unless tests or existing archive semantics suggest otherwise. + +## Remote object selection + +Prefer using manifest/artifact metadata when it reliably identifies durable outputs. + +Also support listing committed durable archive prefixes so restore can retrieve all top-level session artifacts that may not yet be fully represented in manifest metadata. + +The implementation should inspect existing archive code before choosing the final object-selection method. Do not duplicate archive path construction. + +Recommended selection priority: + +1. Remote current manifest path. +2. Durable promoted transcript/artifact outputs recorded in the manifest or archive metadata, if available. +3. Objects under committed durable `transcripts/` and `artifacts/` archive prefixes. +4. Objects under durable `audio/` only when `--include-audio` is passed. + +Always exclude: + +```text +runs/** +``` + +for the first implementation. + +## Package and file organization + +Expected areas to inspect and update: + +```text +cmd/narratio/ +internal/app/ +internal/adapters/storage/ +internal/artifacts/ +internal/manifest/ +docs/ +examples/ +``` + +Suggested implementation shape: + +```text +internal/app/restore.go +internal/app/restore_test.go + +internal/archive/restore/ + planner.go + executor.go + report.go + keys.go + *_test.go +``` + +The exact package name may vary. Use whatever best fits the existing repository, but keep these boundaries clear: + +- `internal/app` owns CLI command handling, config/session loading, lock acquisition, and wiring. +- Restore planning/execution owns remote key discovery, conflict detection, downloads, and reporting. +- `internal/adapters/storage` remains a transport boundary only. +- Workspace/path helpers remain centralized; do not scatter string concatenation. + +If the repository already has an `internal/archive` or archive-stage helper package, prefer extending that rather than creating a conflicting package layout. + +## CLI output + +`narratio restore` should print a concise operator summary. + +Example successful output: + +```text +Restored session archive for sample-campaign/2026-04-04 +Remote run: 20260504T031500Z-a1b2c3 +Downloaded: 4 +Skipped unchanged: 2 +Conflicts: 0 +``` + +Example dry run: + +```text +Restore plan for sample-campaign/2026-04-04 +Remote run: 20260504T031500Z-a1b2c3 +Would download: transcripts/processed.json +Would download: transcripts/trimmed.json +Would skip unchanged: artifacts/session_recap.md +``` + +Example conflict: + +```text +restore conflict: local artifacts/session_recap.md differs from remote archive; rerun with --force to overwrite +``` + +Do not print transcript or artifact content. + +## Error behavior + +Fail clearly when: + +- storage backend is not configured; +- S3 bucket/config is missing or invalid; +- remote current commit marker is missing; +- remote current manifest is missing; +- remote manifest is invalid; +- remote manifest does not match requested campaign/session; +- local file differs from remote and `--force` is not set; +- a required remote object download fails; +- a local path would escape the session workspace; +- a remote key maps to an unsafe local path. + +Skip or report non-fatal conditions when: + +- optional audio restore finds no archived audio; +- an included prefix has no objects; +- a local file already matches the remote file. + +## Path safety + +Every restored file must map to a safe path under the session root. + +Validation rules: + +- local restore paths must be relative to the session root; +- reject absolute paths; +- reject `..` traversal; +- reject paths that escape through symlinks if the codebase has symlink-safe path checks; +- do not restore remote keys directly without mapping/classification; +- do not mirror arbitrary remote keys. + +## Testing plan + +Add focused unit tests. Do not require live S3. + +### CLI tests + +Add or update `internal/app` command tests for: + +- `narratio restore --help`; +- restore accepts `--config`, `--session`, and `--session-id`; +- restore accepts `--dry-run`; +- restore accepts `--force`; +- restore accepts `--include-audio`; +- restore fails when storage is not configured; +- restore does not run pipeline stages. + +### Restore planner tests + +Test: + +- missing `current/run_id.txt` fails; +- missing `current/manifest.json` fails; +- invalid manifest fails; +- wrong campaign/session manifest fails; +- default scope includes manifest/transcripts/artifacts; +- default scope excludes audio/logs/reports/config/runs; +- `--include-audio` includes durable audio; +- run-local keys are excluded; +- keys are sorted deterministically; +- unsafe remote-to-local paths are rejected. + +### Conflict policy tests + +Test: + +- absent local file downloads; +- matching local file skips; +- differing local file conflicts by default; +- `--force` overwrites conflicts; +- `--dry-run` writes nothing; +- partial failure does not corrupt an existing local manifest. + +### Storage/fake tests + +Use fake storage to simulate: + +- object listing; +- object download; +- missing objects; +- download failures; +- metadata/ETag behavior. + +### Workspace/lock tests + +Test: + +- session layout is created before restore; +- session lock conflict fails; +- restored files land under the expected campaign/session workspace; +- no files are written outside the session root. + +### Follow-up command workflow test + +Add at least one test that simulates: + +```bash +narratio restore --session-id 2026-04-04 +narratio run-stage --force analyze +``` + +The test does not need to run real Scriptorium. Use existing fake/stub behavior to verify that restored transcripts and manifest state are sufficient for analyze-stage input resolution. + +## Documentation updates when implemented + +When the feature is implemented, update current-behavior docs: + +```text +docs/cli.md +docs/operations.md +docs/internal/storage.md or docs/internal/archive/restore.md +``` + +If the documentation set does not yet have an internal restore document, add one consistent with the existing internal-doc style: + +```text +docs/internal/command-restore.md +``` + +or: + +```text +docs/internal/archive-restore.md +``` + +Do not document future `narratio run --restore` behavior outside `docs/roadmap/` until implemented. + +## Implementation phases + +### Phase 1: Audit existing archive and path helpers + +Before coding behavior, inspect: + +```text +internal/app/ +internal/stage/archive* +internal/adapters/storage/ +internal/artifacts/ +internal/manifest/ +docs/internal/stage-archive.md, if present +``` + +Determine: + +- exact remote archive key layout; +- how root prefix/campaign/session are modeled; +- how current commit marker keys are built; +- how current manifest is uploaded; +- where promoted outputs are uploaded; +- whether helper functions already exist for remote archive keys; +- whether local workspace path helpers can safely map restore destinations. + +Deliverable: + +- small code comments or internal helper selection; +- no large behavior change yet unless required by tests. + +### Phase 2: Add CLI surface and command wiring + +Add `narratio restore` command parsing. + +Wire flags: + +```text +--config +--session +--session-id +--dry-run +--force +--include-audio +``` + +Use the existing config/session load path where practical. + +Deliverable: + +- command exists; +- help output is sensible; +- command validates basic inputs; +- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired; +- CLI tests pass. + +### Phase 3: Implement remote current-state discovery + +Add restore code that: + +- creates an object store from resolved config; +- builds remote current marker key; +- reads `current/run_id.txt`; +- reads/downloads `current/manifest.json`; +- validates manifest identity; +- returns remote current-state metadata. + +Deliverable: + +- fake-storage tests for current-state discovery; +- no local file writes beyond temporary files. + +### Phase 4: Implement restore planning + +Build deterministic restore plans for default scope and `--include-audio`. + +Deliverable: + +- plan lists manifest, transcript, artifact files; +- plan excludes run-local data; +- plan detects local same/conflict/missing states; +- dry-run output works; +- no real file overwrite yet except temp comparisons as needed. + +### Phase 5: Implement restore execution + +Execute the plan safely: + +- create directories; +- download to temporary files; +- validate content where practical; +- atomically install files; +- enforce default conflict failure; +- support `--force`; +- preserve existing manifest unless safe to replace. + +Deliverable: + +- restore works end-to-end against fake storage; +- failures are clear and do not corrupt existing local manifest. + +### Phase 6: Add restore report and operator summary + +Add concise stdout summary and optional JSON restore report if consistent with project diagnostics. + +Deliverable: + +- user-friendly output; +- durable diagnostic report if implemented; +- no content leakage. + +### Phase 7: Workflow integration test + +Add a test for restoring a previous session and then forcing `analyze`. + +Deliverable: + +- restored manifest/transcripts/artifacts are sufficient for analyze input resolution; +- no upstream stages rerun; +- no reliance on live subprocesses or S3. + +### Phase 8: Documentation update + +Once implemented, update current-behavior docs and internal command docs. + +Also leave future `narratio run --restore` in roadmap only. + +## Definition of done + +The feature is complete when: + +- `narratio restore` exists and is documented. +- It uses the same config/session discovery semantics as other commands where practical. +- It requires configured remote storage. +- It restores only from a committed current archive state. +- It restores the current manifest, transcripts, and artifacts by default. +- It restores audio only with `--include-audio`. +- It excludes run-local sandboxes. +- It fails on local/remote conflicts by default. +- `--force` overwrites conflicts. +- `--dry-run` writes nothing. +- It uses fake storage in tests. +- It does not change `prepare` behavior. +- It does not implement `narratio run --restore`. +- It avoids AWS SDK leakage outside the storage adapter. +- It uses centralized path/key helpers rather than scattered string concatenation. +- `go test ./...` passes. + +## Suggested test commands + +Run focused tests first: + +```bash +go test ./internal/app -run TestExecute -v +go test ./internal/adapters/storage -v +go test ./internal/artifacts -v +go test ./internal/manifest -v +``` + +Then run the full suite: + +```bash +go test ./... +``` + +## Suggested commit message + +```text +Add restore subcommand roadmap +``` diff --git a/docs/roadmap/runtime-artifacts.md b/docs/roadmap/runtime-artifacts.md deleted file mode 100644 index 1f8a1fc..0000000 --- a/docs/roadmap/runtime-artifacts.md +++ /dev/null @@ -1,762 +0,0 @@ -# Roadmap: Runtime-Defined Scriptorium Artifacts - -## Status - -Implementation roadmap for a pre-release hard cutover. - -## Purpose - -Narratio currently treats artifact generation as a narrow `analyze` stage that supports a hard-coded `session_recap` artifact. This roadmap describes how to generalize artifact generation so operators can define Scriptorium-backed output artifacts at runtime through `pipeline.yml`. - -The goal is to keep Narratio as a fixed pipeline orchestrator while making the artifact generation step configurable, composable, deterministic, and easy to regenerate selectively. - -## Desired Outcome - -Operators should be able to define artifacts such as session recaps, player handouts, NPC summaries, quest logs, entity maps, or other campaign-specific outputs without changing Narratio code. - -A configured artifact is declared under: - -```text -pipeline.scriptorium.artifacts. -``` - -Each configured artifact becomes a canonical runtime artifact source ID: - -```text -narratio.artifact. -``` - -For example: - -```yaml -scriptorium: - artifacts: - session_recap: - enabled: true - prompt_id: dnd_session.session_recap - output_path: artifacts/session_recap.md - inputs: - transcript: - source: narratio.transcript.trimmed - required: true -``` - -This artifact is addressable by later artifacts as: - -```text -narratio.artifact.session_recap -``` - -A dependent artifact can then consume it explicitly: - -```yaml -scriptorium: - artifacts: - player_handout: - enabled: true - depends_on: - - session_recap - prompt_id: dnd_session.player_handout - output_path: artifacts/player_handout.md - inputs: - recap: - source: narratio.artifact.session_recap - required: true - transcript: - source: narratio.transcript.trimmed - required: true -``` - -## Resolved Design Decisions - -The following decisions are settled for the initial implementation: - -1. Configured artifact outputs must live under Narratio's internal artifact output directory, initially `artifacts/`. -2. The artifact output directory should be defined as an internal default in `internal/config/defaults.go`, but no public configuration knob should be exposed yet. -3. Artifact `output_path` should remain explicit in the initial implementation to avoid guessing file extensions or output formats. -4. A disabled artifact may still be referenced as an input if its declared output already exists on disk and passes basic validation. -5. A disabled artifact is not executable during the current analyze run. -6. Artifact-to-artifact references require an explicit `depends_on` entry. Narratio should fail fast if the dependency declaration is missing. -7. The manifest remains stage-oriented: `analyze` succeeds or fails as a full stage. -8. Analyze-stage metadata may record per-artifact output details for provenance and later resolution, but not for intra-stage resume semantics. -9. `--artifacts` should be added as a CLI filter for selective artifact generation. -10. `--artifacts` does not imply `--force`; it only changes which configured artifacts are treated as executable when `analyze` actually runs. -11. Because Narratio is still pre-release, the hard-coded `session_recap` behavior should be removed immediately rather than deprecated gradually. - -## Scope - -This roadmap covers: - -- introducing a runtime artifact catalog; -- generalizing configured Scriptorium artifact execution; -- supporting `narratio.artifact.` source IDs; -- adding explicit artifact dependencies; -- supporting disabled-but-resolvable artifact inputs; -- adding selective artifact execution via `--artifacts`; -- recording generated artifacts in analyze-stage metadata and/or manifest outputs; -- removing hard-coded `session_recap` behavior; -- updating tests and documentation. - -## Non-Goals - -This feature should not turn Narratio into a general workflow engine. - -The initial implementation should not add: - -- arbitrary shell-command artifacts; -- arbitrary user-defined stages; -- loops or conditional branching; -- automatic archive promotion of generated artifacts; -- semantic knowledge of particular artifact types; -- per-artifact resume semantics within a successful or failed analyze stage; -- automatic dependency inference without `depends_on`. - -Narratio should continue to orchestrate a fixed pipeline. The configurable part is the set of Scriptorium artifact invocations performed during the `analyze` stage. - -## Current State - -Narratio already has several relevant pieces in place: - -- `pipeline.scriptorium.artifacts` is modeled as a map of artifact definitions. -- The Scriptorium adapter already accepts generic run/render requests. -- The artifact resolver already understands canonical artifact source IDs. -- The `analyze` stage already resolves inputs, optionally runs render-debug, invokes Scriptorium, verifies output, and records metadata. - -The main limitation is that `analyze` currently treats `session_recap` as the only executable artifact and rejects other enabled artifact definitions. - -## Target Architecture - -### Runtime Artifact Catalog - -Introduce a per-run artifact catalog that tracks built-in artifacts and configured artifacts. - -Conceptually: - -```text -ArtifactCatalog -├── built-in artifacts -│ ├── narratio.transcript.merged -│ ├── narratio.transcript.polished -│ ├── narratio.transcript.full -│ ├── narratio.transcript.trimmed -│ └── narratio.bounds.session -│ -└── configured artifacts - ├── narratio.artifact.session_recap - ├── narratio.artifact.player_handout - └── narratio.artifact.npc_summary -``` - -The catalog should distinguish between three states: - -```text -planned valid configured or built-in artifact known to Narratio -available artifact has been produced or otherwise resolved -executable configured artifact selected for execution in this analyze run -``` - -Configured artifacts can be planned without being executable. This distinction is important for disabled artifacts and for `--artifacts` filtering. - -### Configured Artifact Source IDs - -Configured artifact keys map directly to source IDs: - -```text -pipeline.scriptorium.artifacts. -→ narratio.artifact. -``` - -`session_recap` should no longer be a special built-in analyze artifact. Instead, it is just a conventional configured artifact key: - -```yaml -scriptorium: - artifacts: - session_recap: - enabled: true - prompt_id: dnd_session.session_recap - output_path: artifacts/session_recap.md -``` - -`narratio.artifact.session_recap` remains valid only because `session_recap` is configured. - -### Artifact Output Directory - -Add an internal default artifact output directory, initially: - -```text -artifacts -``` - -This default should live in `internal/config/defaults.go` or the existing equivalent defaults location. - -For the initial implementation: - -- expose no public config knob for the artifact output directory; -- require each configured artifact to provide an explicit `output_path`; -- validate that each configured artifact `output_path` is run-relative; -- validate that each configured artifact `output_path` is under the internal artifact output directory; -- reject output paths that escape the run workspace or use path traversal. - -This preserves future configurability without forcing Narratio to guess output extensions or formats now. - -### Enabled, Disabled, and Selected Artifacts - -Configured artifacts should have three distinct execution states: - -```text -enabled by config artifact has enabled: true -selected for execution artifact remains executable after --artifacts filtering -disabled for execution artifact is not executable, but may be resolvable from disk -``` - -Without `--artifacts`, all configured artifacts with `enabled: true` are selected for execution. - -With `--artifacts`, only the named artifacts are selected for execution. All other configured artifacts are treated as disabled for the current analyze invocation, regardless of their configured `enabled` value. - -Disabled artifacts may still be resolved as inputs if their configured `output_path` exists on disk and passes validation. - -### Disabled Artifact Resolution - -If artifact `B` references artifact `A`, and `A` is disabled for execution, Narratio should attempt to resolve `A` from disk. - -This should succeed only when: - -1. `A` is defined in `pipeline.scriptorium.artifacts`; -2. `A` has a valid `output_path`; -3. the output path exists in the current run workspace; -4. the output is non-empty, or otherwise passes any available artifact-specific validation. - -The resolved provenance should make the source clear, for example: - -```text -filesystem.disabled_artifact_output -``` - -If the file does not exist or fails validation, the dependent artifact should fail before invoking Scriptorium. - -Example error wording: - -```text -artifact player_handout requires narratio.artifact.session_recap, but session_recap is disabled for execution and artifacts/session_recap.md does not exist -``` - -### Explicit Dependencies - -Artifact-to-artifact references require explicit `depends_on` entries. - -If artifact `B` has an input source of `narratio.artifact.A`, then `B.depends_on` must include `A`. - -This should fail: - -```yaml -scriptorium: - artifacts: - player_handout: - enabled: true - prompt_id: dnd_session.player_handout - output_path: artifacts/player_handout.md - inputs: - recap: - source: narratio.artifact.session_recap - required: true -``` - -This should pass: - -```yaml -scriptorium: - artifacts: - player_handout: - enabled: true - depends_on: - - session_recap - prompt_id: dnd_session.player_handout - output_path: artifacts/player_handout.md - inputs: - recap: - source: narratio.artifact.session_recap - required: true -``` - -`depends_on` values refer to configured artifact keys, not full source IDs. - -Dependency validation should fail on: - -- references to unknown artifact keys; -- missing `depends_on` entries for artifact-to-artifact input references; -- self-dependencies; -- dependency cycles among executable artifacts. - -Dependencies on disabled artifacts are permitted, but the disabled dependency must resolve from disk before the dependent artifact runs. - -### Execution Order - -The analyze stage should execute selected artifacts in dependency order. - -Rules: - -- selected artifacts are executable; -- disabled artifacts are never executed; -- selected artifacts may depend on other selected artifacts; -- selected artifacts may depend on disabled artifacts if those disabled artifacts resolve from disk; -- independent selected artifacts run in deterministic sorted-name order. - -Use topological sorting over selected artifacts, while validating dependency references across the full configured artifact set. - -### Input Resolution - -Input resolution should use the artifact catalog and existing artifact resolver behavior. - -For each configured artifact input: - -- built-in sources resolve through existing resolver behavior; -- `previous_session_artifact` preserves existing behavior; -- `narratio.artifact.` resolves through the runtime artifact catalog; -- selected dependencies resolve after being produced earlier in the same analyze execution; -- disabled dependencies resolve from their configured output path on disk; -- optional missing inputs are omitted; -- required missing inputs fail before Scriptorium is invoked. - -### Analyze Stage Generalization - -The `analyze` stage should become the generic Scriptorium artifact stage. - -High-level flow: - -1. Load configured Scriptorium artifacts. -2. Apply the `--artifacts` filter, if present. -3. If no artifacts are selected for execution, return success metadata with `skipped=true`. -4. Build the runtime artifact catalog. -5. Validate artifact names, output paths, source IDs, dependencies, selected artifacts, and required fields. -6. Resolve any disabled dependencies that are required by selected artifacts. -7. Sort selected artifacts by dependency order. -8. For each selected artifact: - - resolve configured inputs; - - build the Scriptorium run request; - - optionally run Scriptorium render-debug; - - run Scriptorium; - - fail on validation-failed result; - - verify the output exists and is non-empty; - - record artifact output metadata; - - register `narratio.artifact.` as available in the catalog. -9. Return aggregate analyze-stage metadata containing all generated and reused artifacts relevant to the run. - -The Scriptorium adapter should remain generic. It should not decide which artifacts run, how dependencies work, or how artifacts are registered. - -### Manifest and Metadata - -The manifest should remain stage-oriented. - -This means: - -- `analyze` succeeds or fails as a full stage; -- if `analyze` has already succeeded and the user does not force it, the runner skips it as a full stage; -- Narratio should not implement per-artifact resume in the first version. - -However, analyze-stage metadata should still record artifact outputs for provenance and future resolution. - -Recommended metadata shape: - -```json -{ - "skipped": false, - "artifacts": [ - { - "name": "session_recap", - "source_id": "narratio.artifact.session_recap", - "output_kind": "scriptorium_artifact", - "path": "artifacts/session_recap.md", - "prompt_id": "dnd_session.session_recap", - "profile_id": "local-gemma-31b", - "provenance": "generated.current_analyze_run" - }, - { - "name": "player_handout", - "source_id": "narratio.artifact.player_handout", - "output_kind": "scriptorium_artifact", - "path": "artifacts/player_handout.md", - "prompt_id": "dnd_session.player_handout", - "profile_id": "local-gemma-31b", - "provenance": "generated.current_analyze_run" - } - ], - "reused_artifacts": [ - { - "name": "session_recap", - "source_id": "narratio.artifact.session_recap", - "path": "artifacts/session_recap.md", - "provenance": "filesystem.disabled_artifact_output" - } - ] -} -``` - -The exact struct can differ from this example, but it should preserve: - -- artifact name; -- canonical source ID; -- output path; -- prompt/profile provenance for generated artifacts; -- reused-vs-generated provenance. - -### Resume and Force Behavior - -Keep resume behavior stage-level. - -Recommended semantics: - -```text -No --force, analyze already succeeded: - runner skips analyze, regardless of --artifacts. - ---force, no --artifacts: - analyze regenerates all configured artifacts with enabled: true. - ---force --artifacts player_handout: - analyze treats only player_handout as executable. - all other configured artifacts are disabled for execution. - disabled dependencies may be reused from disk. - ---artifacts player_handout on a not-yet-completed analyze stage: - analyze runs only player_handout. - disabled dependencies may be reused from disk. -``` - -`--artifacts` should not imply `--force`. It is an execution filter, not a resume override. - -### `--artifacts` CLI Flag - -Add an `--artifacts` flag to commands that can execute or resume the analyze stage. - -The flag should accept one or more configured artifact names. Internally, normalize values to a set of artifact keys. - -Recommended behavior: - -- validate all requested artifact names against `pipeline.scriptorium.artifacts`; -- reject unknown artifact names before running stages; -- treat requested artifacts as the only executable artifacts for the analyze stage; -- treat all other configured artifacts as disabled for execution; -- allow disabled artifacts to satisfy dependencies from disk as described above; -- if `--artifacts` is used while executing a stage other than `analyze`, either reject it or ignore it with a clear validation error. Prefer rejection. - -The exact CLI parsing style can follow Narratio's existing conventions. Both comma-separated and repeatable values are acceptable if the CLI package supports them cleanly, but the internal representation should be a set of artifact keys. - -### Archive Behavior - -Do not automatically archive every generated artifact. - -Artifact generation and archive promotion should remain separate concerns. Operators should continue to use `archive.promote_artifacts` to decide which generated files should be promoted or uploaded. - -Example: - -```yaml -archive: - promote_artifacts: - - from: artifacts/session_recap.md - to: artifacts/session_recap.md - required: true - - from: artifacts/player_handout.md - to: artifacts/player_handout.md - required: false -``` - -A later enhancement may add opt-in automatic promotion of configured artifacts, but explicit promotion should remain the default. - -## Implementation Plan - -### Phase 1: Config Model and Defaults - -Add or update the configured artifact model to include: - -- `enabled`; -- `depends_on`; -- `prompt_id`; -- `profile_id`; -- `output_path`; -- `timeout`; -- `render_debug`; -- `inputs`; -- `vars`. - -Add an internal default artifact output directory in `internal/config/defaults.go`, initially set to `artifacts`. - -Validation rules: - -- artifact names must match a conservative identifier pattern such as `^[a-z][a-z0-9_]*$`; -- selected/executable artifacts require `prompt_id` and `output_path`; -- configured artifacts that may be referenced while disabled require `output_path`; -- configured artifact output paths must be run-relative; -- configured artifact output paths must live under the internal artifact output directory; -- configured artifact output paths must not escape the run workspace; -- `narratio.artifact.` input sources must refer to configured artifact keys; -- any `narratio.artifact.` input source must have a matching `depends_on` entry; -- `depends_on` entries must refer to configured artifact keys; -- dependencies must not contain self-references or executable cycles; -- input names and var names must remain compatible with the Scriptorium adapter's validation rules; -- unknown YAML fields must continue to fail strict decode. - -Tests: - -- valid single configured artifact; -- valid multiple independent artifacts; -- valid artifact-to-artifact dependency; -- valid dependency on disabled artifact with output path; -- invalid artifact name; -- missing required fields; -- output path outside `artifacts/`; -- dependency on missing artifact; -- missing `depends_on` for artifact input source; -- self-dependency; -- cycle detection; -- typo in `narratio.artifact.` source; -- unknown YAML fields still fail strict decode. - -### Phase 2: CLI Filtering - -Add the `--artifacts` flag and carry the selected artifact set into the run execution options. - -Implementation notes: - -- parse values according to existing CLI conventions; -- normalize to artifact key strings; -- validate against configured artifact definitions after config load; -- make the selected set available to the analyze stage; -- reject use with commands or stages where analyze cannot run. - -Tests: - -- no `--artifacts` means all enabled artifacts are selected; -- one requested artifact is selected; -- multiple requested artifacts are selected; -- unknown requested artifact fails; -- `--artifacts` does not imply `--force`; -- `--artifacts` with already-succeeded analyze stage is skipped unless forced; -- `--artifacts` on unsupported stage command fails clearly. - -### Phase 3: Runtime Artifact Catalog - -Introduce an internal artifact catalog abstraction. - -Responsibilities: - -- register built-in artifact definitions; -- register configured artifact definitions; -- map configured artifact keys to `narratio.artifact.` IDs; -- track planned, available, and executable artifact states; -- expose lookup by canonical source ID; -- record generated provenance; -- record disabled-from-disk provenance. - -Keep the catalog narrow. It should not execute Scriptorium and should not understand prompt semantics. - -Tests: - -- built-in source lookup; -- configured source registration; -- duplicate/conflicting source handling; -- planned but unavailable artifact lookup; -- selected artifact state; -- disabled artifact state; -- registering an artifact as available after generation; -- registering a disabled artifact as available from disk; -- resolving a configured artifact from analyze metadata if that behavior is implemented. - -### Phase 4: Resolver Integration - -Update artifact resolution so configured artifact IDs are resolved through the runtime catalog. - -Resolution behavior: - -- built-in sources continue using existing resolver behavior; -- configured artifact sources resolve from catalog availability/provenance; -- selected configured artifacts become available after generation; -- disabled configured artifacts may become available from disk; -- missing optional configured artifact inputs are omitted; -- missing required configured artifact inputs fail clearly. - -Tests: - -- configured artifact consumes a built-in transcript source; -- configured artifact consumes another configured artifact produced earlier in the same analyze run; -- configured artifact consumes a disabled artifact resolved from disk; -- required disabled artifact missing on disk fails; -- required configured artifact missing fails; -- optional missing configured artifact is omitted; -- reused artifact provenance is recorded distinctly from generated artifact provenance. - -### Phase 5: Analyze Stage Generalization - -Refactor `analyze` to execute selected configured artifacts. - -Implementation notes: - -- remove the hard-coded `session_recap` selection path; -- remove the hard-coded rejection of non-`session_recap` artifacts; -- preserve skip behavior when Scriptorium config is absent or no artifacts are selected; -- build the runtime artifact catalog; -- apply `--artifacts` filtering; -- validate selected artifacts and their dependencies; -- pre-resolve disabled dependencies from disk where required; -- compute deterministic dependency order; -- execute selected artifacts one at a time in dependency order; -- keep render-debug behavior at global and artifact levels; -- keep Scriptorium adapter invocation generic; -- after each successful run, register the artifact as available in the catalog; -- aggregate generated and reused artifact metadata. - -Tests: - -- no Scriptorium config skips; -- empty artifact map skips; -- no selected artifacts skips; -- disabled artifacts do not run; -- one selected artifact runs; -- multiple independent artifacts run in deterministic order; -- dependent selected artifact receives prior selected artifact as input; -- dependent selected artifact receives disabled-from-disk artifact as input; -- render-debug works for configured artifacts; -- Scriptorium validation failure fails the stage; -- missing required input fails the stage; -- successful outputs are non-empty and recorded; -- artifact filter executes only requested artifacts. - -### Phase 6: Manifest and Stage Metadata - -Update analyze-stage metadata and manifest output recording to support dynamic configured artifacts. - -Recommended behavior: - -- every generated configured artifact gets `source_id: narratio.artifact.`; -- every generated configured artifact gets a generic output kind such as `scriptorium_artifact`; -- reused disabled artifacts are recorded separately from generated artifacts; -- metadata is sufficient for debugging, provenance, and future resolver support; -- metadata does not create per-artifact resume semantics. - -Because this is a pre-release hard cutover, do not preserve a special legacy `session_recap` output kind unless a current internal test or archive path still requires it temporarily. Prefer updating tests and examples to treat `session_recap` as an ordinary configured artifact. - -Tests: - -- metadata records one generated configured artifact; -- metadata records multiple generated configured artifacts; -- metadata records reused disabled artifact provenance; -- `session_recap` is recorded as a normal configured artifact; -- manifest still treats `analyze` as a single succeeded or failed stage; -- runner skip behavior remains stage-level. - -### Phase 7: Archive and Promotion Review - -Review archive behavior after dynamic artifacts are recorded. - -Implementation notes: - -- do not automatically promote every configured artifact; -- keep `archive.promote_artifacts` explicit; -- update default or example promotion rules to use configured `session_recap` output path; -- ensure required promotion rules fail clearly when selected artifact generation did not produce a required file. - -Tests: - -- generated artifact can be promoted by explicit archive rule; -- required archive promotion fails if selected artifact was not generated and no file exists; -- optional archive promotion skips cleanly if file is absent; -- hard cutover does not rely on hard-coded `session_recap` generation. - -### Phase 8: Documentation and Examples - -Status: complete. - -Update documentation after the implementation is complete. - -Recommended documentation changes: - -- update `docs/config.md` with the generalized artifact configuration model; -- update `docs/internal/artifacts.md` to describe the runtime artifact catalog; -- update `docs/stages/analyze.md` to describe generic Scriptorium artifact generation; -- update Scriptorium integration docs only if the adapter contract changes; -- update full annotated pipeline examples; -- add at least one example with multiple artifacts and one dependency; -- document `--artifacts` behavior and its relationship to `--force`; -- remove documentation stating that only `session_recap` is supported. - -Documentation should make clear that: - -- configured artifact source IDs use `narratio.artifact.`; -- `depends_on` uses artifact keys, not full source IDs; -- artifact-to-artifact source references require explicit `depends_on`; -- disabled artifacts can be reused from disk when required by selected artifacts; -- `--artifacts` filters execution but does not imply `--force`; -- archive promotion remains explicit; -- per-artifact resume is not part of the initial implementation. - -## Migration Strategy - -Because Narratio is pre-release, perform a hard cutover. - -Required changes: - -1. Remove the hard-coded `session_recap` analyze behavior. -2. Require `session_recap` to be declared under `pipeline.scriptorium.artifacts.session_recap` if the operator wants a session recap. -3. Treat `narratio.artifact.session_recap` as valid only when `session_recap` is a configured artifact key. -4. Update config examples to show `session_recap` as a normal configured artifact. -5. Update tests to stop assuming that `session_recap` is a built-in analyze artifact. -6. Keep archive promotion explicit and path-based. - -Example replacement config: - -```yaml -scriptorium: - binary: scriptorium - config_path: /etc/scriptorium/config.yml - timeout: 10m - render_debug: false - artifacts: - session_recap: - enabled: true - prompt_id: dnd_session.session_recap - profile_id: local-gemma-31b - output_path: artifacts/session_recap.md - timeout: 20m - inputs: - transcript: - source: narratio.transcript.trimmed - required: true - prior_recap: - source: previous_session_artifact - artifact: artifacts/session_recap.md - required: false - vars: - artifact_title: Session Recap -``` - -## Acceptance Criteria - -The feature is complete when: - -- operators can define more than one enabled Scriptorium artifact in `pipeline.yml`; -- Narratio runs selected artifacts in deterministic dependency order; -- configured artifacts are addressable as `narratio.artifact.`; -- one configured artifact can consume another configured artifact as an input; -- artifact-to-artifact input references require explicit `depends_on`; -- disabled artifacts can satisfy dependencies from existing on-disk outputs; -- missing required disabled artifacts fail clearly; -- optional missing inputs are omitted; -- `--artifacts` can selectively execute valid configured artifact names; -- `--artifacts` does not imply `--force`; -- render-debug behavior works for all configured artifacts; -- generated and reused artifacts are recorded in analyze-stage metadata; -- `session_recap` is no longer hard-coded and works as a normal configured artifact; -- archive promotion remains explicit; -- tests cover config validation, dependency sorting, disabled artifact resolution, resolver behavior, CLI filtering, analyze execution, archive interactions, and metadata. - -## Suggested Implementation Order - -1. Config model, defaults, and validation. -2. CLI parsing and propagation of `--artifacts` selection. -3. Runtime artifact catalog. -4. Resolver integration for configured artifacts. -5. Analyze stage generalization. -6. Stage metadata and manifest output recording. -7. Archive behavior review. -8. Documentation and examples. - -This order keeps the most static pieces first, then moves into execution behavior once the configuration contract is explicit and well tested.