From ab59bab04451a5cba95a9bb92d98636d1a2ce3a3 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 23 May 2026 07:11:17 -0500 Subject: [PATCH] Initial documentation cleanup pass --- docs/{ => policy}/architecture.md | 0 docs/{ => policy}/development.md | 0 .../policy.md => policy/documentation.md} | 0 docs/roadmap/campaign.md | 231 -------------- docs/roadmap/cleanup.md | 159 ---------- docs/roadmap/cli.md | 255 ---------------- docs/roadmap/publish.md | 287 ------------------ docs/roadmap/transcripts.md | 210 ------------- 8 files changed, 1142 deletions(-) rename docs/{ => policy}/architecture.md (100%) rename docs/{ => policy}/development.md (100%) rename docs/{documentation/policy.md => policy/documentation.md} (100%) delete mode 100644 docs/roadmap/campaign.md delete mode 100644 docs/roadmap/cleanup.md delete mode 100644 docs/roadmap/cli.md delete mode 100644 docs/roadmap/publish.md delete mode 100644 docs/roadmap/transcripts.md diff --git a/docs/architecture.md b/docs/policy/architecture.md similarity index 100% rename from docs/architecture.md rename to docs/policy/architecture.md diff --git a/docs/development.md b/docs/policy/development.md similarity index 100% rename from docs/development.md rename to docs/policy/development.md diff --git a/docs/documentation/policy.md b/docs/policy/documentation.md similarity index 100% rename from docs/documentation/policy.md rename to docs/policy/documentation.md diff --git a/docs/roadmap/campaign.md b/docs/roadmap/campaign.md deleted file mode 100644 index eb94f9f..0000000 --- a/docs/roadmap/campaign.md +++ /dev/null @@ -1,231 +0,0 @@ -# Roadmap: Campaign Registry - -Status: Implemented - -## Problem - -Narratio currently treats campaign configuration as one selected -`campaign.yml` file: - -- command flags use `--campaign `; -- default discovery searches fixed system file locations; -- `campaign.yml` uses `campaign:` as the identity field. - -That model works for a single campaign, but it is awkward for installations -that manage multiple campaigns. Operators need to pass file paths or maintain a -single global campaign config, while the newer session-oriented CLI already -uses concise positional session IDs and remote session lookup. - -The campaign selection model should become ID-based and pipeline-owned. -Pipeline config should describe where campaigns live, commands should select a -campaign by ID, and each campaign directory should contain its stable campaign -materials. - -## Target Model - -`pipeline.yml` owns the campaign registry: - - campaigns: - root: /usr/local/share/narratio/campaigns - default_campaign_id: dilfs - -Campaign files live at the conventional path: - - {campaigns.root}/{campaign_id}/campaign.yml - -The first implementation should use only the conventional path. Recursive -discovery of every `campaign.yml` under `campaigns.root` is deferred to a -future stage. - -Each campaign file uses `campaign_id` as the canonical identity field: - - campaign_id: dilfs - session_template_file: ./session.template.yml - inputs: - speakers_file: ./speakers.yml - autocorrect_file: ./autocorrect.yml - glossary_file: ./glossary.yml - -Campaign-relative files continue to resolve relative to the selected -`campaign.yml`, including stable input files and `session_template_file`. - -The public CLI changes from path-based campaign selection to ID-based campaign -selection: - -- `--campaign ` selects a campaign ID. -- `--campaign-file ` explicitly loads one campaign file for - development, tests, and unusual local workflows. -- `--campaign` and `--campaign-file` are mutually exclusive. - -If neither `--campaign` nor `--campaign-file` is passed, Narratio uses -`pipeline.campaigns.default_campaign_id`. If no campaign can be selected, -commands fail clearly before session loading or stage execution. - -Resolved campaign ID remains the campaign segment used for: - -- workspace paths; -- spool paths; -- S3 session prefixes; -- remote `session.yml` lookup; -- archive locks and promoted output keys; -- session/campaign mismatch validation; -- status, plan, restore, and helper output. - -## Compatibility Policy - -This is a breaking public/config contract change. - -After the cutover: - -- `--campaign` no longer accepts a filesystem path; -- default fixed campaign file discovery is removed; -- `campaign:` is no longer accepted in `campaign.yml`; -- `campaign_id:` is required. - -Keep `--campaign-file` as the only explicit file override. Do not retain hidden -aliases for the old `--campaign ` behavior. - -## Implementation Stages - -### Stage 1: Add Campaign Registry Selection - -Status: Implemented - -Add the registry model and switch command loading to resolve campaigns through -pipeline config. - -Implementation requirements: - -- Add `pipeline.campaigns.root`. -- Add `pipeline.campaigns.default_campaign_id`. -- Add `campaign_id` to campaign config and make it the canonical identity. -- Resolve pipeline config first, then campaign selection. -- Use this selection order: - 1. explicit `--campaign-file `; - 2. explicit `--campaign `; - 3. `pipeline.campaigns.default_campaign_id`; - 4. fail clearly. -- For ID selection, load `{campaigns.root}/{campaign_id}/campaign.yml`. -- Validate that the loaded `campaign_id` matches the selected ID. -- Reject `--campaign` with `--campaign-file`. -- Preserve strict YAML decoding. -- Preserve campaign-relative stable input and session template resolution. -- Keep storage details behind the existing storage adapter and object-store - helper. -- Keep remote session lookup and archive key construction based on the - resolved campaign ID. - -Acceptance criteria: - -- Commands can run with only a pipeline config and the pipeline default - campaign ID. -- Commands can select another campaign with `--campaign `. -- Commands can load a specific file with `--campaign-file `. -- Existing session loading, remote session fallback, prepare materialization, - restore, archive, locks, clean, analyze, and publish behavior continue to use - the same resolved campaign identity. -- No generic config registry framework is introduced. - -### Stage 2: Remove Old Single-File Campaign Behavior - -Status: Implemented - -Remove the old public campaign file model after registry selection is in -place. - -Implementation requirements: - -- Remove fixed default campaign config discovery from command loading. -- Remove `DefaultCampaignConfigSearchPaths` and related path-only resolution if - no current tests or helpers still need them. -- Remove support for `campaign:` from `campaign.yml`. -- Update validation errors to refer to `campaign_id`. -- Update examples to use campaign directories and `campaign_id`. -- Update current-behavior docs to document: - - `pipeline.campaigns.root`; - - `pipeline.campaigns.default_campaign_id`; - - `campaign_id`; - - `--campaign `; - - `--campaign-file `. -- Update troubleshooting examples that currently pass `--campaign `. - -Acceptance criteria: - -- `campaign.yml` files with `campaign:` fail strict decoding. -- `--campaign /path/to/campaign.yml` is treated as a campaign ID and fails - unless that ID exists under `campaigns.root`. -- `--campaign-file /path/to/campaign.yml` is the supported file override. -- User-facing docs no longer describe fixed campaign config discovery. - -## Test Guidance - -Focused tests: - -- `go test ./internal/config -v` -- `go test ./internal/app -v` -- `go test ./internal/stage -run Prepare -v` - -Full validation: - -- `go test ./...` - -Config tests to add or update: - -- strict decode accepts `pipeline.campaigns.root`; -- strict decode accepts `pipeline.campaigns.default_campaign_id`; -- strict decode accepts `campaign_id`; -- selected campaign ID mismatch fails; -- missing campaign root fails when ID selection is needed; -- missing default campaign ID fails when no explicit campaign selector is - passed; -- old `campaign:` fails after Stage 2. - -App tests to add or update: - -- `--campaign ` resolves `{campaigns.root}/{id}/campaign.yml`; -- omitted `--campaign` uses `pipeline.campaigns.default_campaign_id`; -- `--campaign-file` loads an explicit campaign file; -- `--campaign` plus `--campaign-file` fails; -- remote session fallback uses the resolved campaign ID; -- `session init`, `run`, `run-stage`, `resume`, `analyze`, `publish`, `clean`, - and `session` subcommands all use the same campaign selection path; -- path-based `--campaign` examples and tests are removed after Stage 2. - -## Documentation Guidance - -Update current-behavior docs only after implementation lands: - -- `docs/config.md` -- `docs/cli.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- relevant files under `docs/internal/` -- `examples/` - -Planned campaign registry behavior belongs only in this roadmap until the code, -tests, examples, and current-behavior docs are updated. - -## Architecture Guardrails - -- Keep Narratio explicit and stage-driven. -- Do not introduce a generic configuration registry or workflow framework. -- Keep YAML decoding strict. -- Keep defaults centralized and testable. -- Keep campaign-relative path resolution centralized. -- Use centralized S3 and workspace path helpers. -- Keep storage details behind `storage.ObjectStore`. -- Keep secret-backed object-store construction in `internal/app`. -- Preserve manifest-driven resume and restore behavior. -- Do not store raw secrets in campaign configs, manifests, logs, generated - configs, or archive metadata. - -## Assumptions - -- The canonical pipeline schema is grouped under `campaigns`. -- The canonical campaign identity field is `campaign_id`. -- `--campaign` means campaign ID. -- `--campaign-file` is retained as an explicit override. -- Recursive discovery is planned but not part of the first implementation. -- Existing production configs can be migrated from `campaign:` to - `campaign_id:` and from `--campaign ` to `--campaign ` or - `--campaign-file `. diff --git a/docs/roadmap/cleanup.md b/docs/roadmap/cleanup.md deleted file mode 100644 index ce1a8c3..0000000 --- a/docs/roadmap/cleanup.md +++ /dev/null @@ -1,159 +0,0 @@ -# Roadmap: Legacy Config Cleanup - -Status: Implemented - -## Problem - -Narratio's current pipeline config schema still accepts fields that predate the current storage, artifact, and previous-session models: - -- `pipeline.storage.bucket` -- `pipeline.storage.prefix` -- `pipeline.analyzer.*` -- `previous_session_artifact` - -These names make the config reference harder to trust because they suggest supported behavior that operators should no longer use. The modern interface is: - -- `pipeline.storage.s3.*` for remote storage. -- Scriptorium configured artifacts under `pipeline.scriptorium.artifacts`. -- Canonical artifact source IDs such as `narratio.artifact.`. -- Canonical previous-session artifact sources such as `narratio.previous_session.artifact.`. - -Strict YAML decoding should reject removed legacy fields once this cleanup lands. - -## Current State - -`pipeline.storage.bucket` and `pipeline.storage.prefix` were inert compatibility fields and have been removed: - -- They are no longer present on `config.StorageConfig`. -- Strict decoding rejects them. -- Runtime S3 behavior uses `pipeline.storage.s3.bucket` and `pipeline.storage.s3.root_prefix`. -- No current code reads the top-level storage bucket or prefix fields. - -`pipeline.analyzer.*` was legacy code surface and has been removed: - -- `config.PipelineConfig` no longer includes analyzer config. -- Strict decoding rejects `pipeline.analyzer`. -- `stage.Env` no longer exposes an analyzer runner, and `internal/adapters/analyzer` has been deleted. -- Modern analyze execution is Scriptorium-backed; the analyzer adapter is not used by current stage execution. - -`previous_session_artifact` was a live legacy behavior and has been removed: - -- Config validation rejects it as an unsupported Scriptorium input source. -- The analyze stage no longer has path-based previous-artifact resolution through `inputs..path`. -- Tests cover canonical previous-session sources and the rejection of the legacy source. -- The canonical replacement is `narratio.previous_session.artifact.`, resolved through the previous-session cache/catalog model. - -## Target Model - -The pipeline config schema should expose only current behavior: - -- Remote storage is configured only through `pipeline.storage.s3.*`. -- Generated artifacts are configured only through `pipeline.scriptorium.artifacts`. -- Scriptorium artifact inputs use canonical source IDs. -- Previous-session artifact inputs use `narratio.previous_session.artifact.`. -- Unknown legacy fields fail strict YAML decoding. - -No compatibility aliases should remain unless a future migration requirement explicitly reintroduces them. - -## Cleanup Order - -### Stage 1: Remove Inert Storage Compatibility Fields - -Status: Implemented - -Remove `pipeline.storage.bucket` and `pipeline.storage.prefix`. - -Implementation requirements: - -- Delete `StorageConfig.Bucket` and `StorageConfig.Prefix`. -- Keep `StorageConfig.Backend` and `StorageConfig.S3`. -- Confirm all runtime storage paths continue to use `storage.s3.bucket` and `storage.s3.root_prefix`. -- Update examples and docs to remove top-level storage `bucket` and `prefix`. -- Add or update strict-decode tests proving `pipeline.storage.bucket` and `pipeline.storage.prefix` are rejected. - -Acceptance criteria: - -- Existing S3 workflows still pass with `pipeline.storage.s3.bucket`. -- Pipeline configs containing top-level `storage.bucket` or `storage.prefix` fail to load. -- No docs or examples present those fields as available. - -### Stage 2: Remove Legacy Analyzer Schema and Adapter Surface - -Status: Implemented - -Remove the unused analyzer configuration and adapter contract. - -Implementation requirements: - -- Delete `PipelineConfig.Analyzer`. -- Delete `AnalyzerConfig` and `ArtifactSettings`. -- Remove analyzer timeout validation. -- Remove `stage.Env.Analyzer`. -- Delete `internal/adapters/analyzer` if no remaining code imports it. -- Remove `pipeline.analyzer.*` from tests, examples, and docs. -- Add or update strict-decode tests proving `pipeline.analyzer` is rejected. - -Acceptance criteria: - -- Analyze behavior remains fully Scriptorium-backed. -- No runtime code imports `internal/adapters/analyzer`. -- Pipeline configs containing `pipeline.analyzer` fail to load. -- Contributor and internal adapter docs no longer list the analyzer adapter. - -### Stage 3: Remove Path-Based Previous Session Artifact Source - -Status: Implemented - -Remove `previous_session_artifact` and require canonical previous-session artifact sources. - -Implementation requirements: - -- Remove `previous_session_artifact` from supported Scriptorium input sources. -- Remove analyze-stage special-case handling that resolves `inputs..path` for previous artifacts. -- Keep canonical handling for `narratio.previous_session.artifact.`. -- Rewrite tests that use `previous_session_artifact` to use canonical sources and prepared previous-cache fixtures. -- Add validation tests proving `previous_session_artifact` is rejected. -- Update docs to remove the legacy path-based source and document only canonical previous-session sources. - -Acceptance criteria: - -- `pipeline.scriptorium.artifacts.*.inputs.*.source: previous_session_artifact` fails validation. -- Canonical previous-session sources continue to work for required and optional inputs. -- Prepare/restore previous-cache behavior remains unchanged. -- No docs or examples mention `previous_session_artifact` as supported. - -## Test Guidance - -Run focused tests after each stage: - -- `go test ./internal/config -v` -- `go test ./internal/stage -run Analyze -v` -- `go test ./internal/app -v` -- `go test ./...` - -For Stage 1, focus on config load/strict-decode and S3 workflow regression tests. - -For Stage 2, focus on compile-time removal, config strict-decode tests, and full app/stage tests to catch stale adapter references. - -For Stage 3, focus on Scriptorium config validation, analyze-stage input resolution, previous-cache behavior, and restore/analyze workflows. - -## Documentation Updates - -Update current-behavior docs only after the corresponding code removal lands: - -- `docs/config.md` -- `docs/cli.md`, only if command behavior text references removed fields. -- `docs/operations.md`, only if operator workflow text references removed fields. -- `docs/internal/stage-analyze.md` -- `docs/internal/adapters.md` -- `examples/pipeline.full.annotated.yml` -- `examples/pipeline.production.yml` - -Do not preserve removed fields in examples as compatibility notes. The goal is to make strict config behavior and documentation line up. - -## Assumptions - -- This is a hard cleanup; no backward-compatible aliases are retained. -- Current production configs can be migrated to `storage.s3.*`, Scriptorium artifacts, and canonical previous-session sources before this lands. -- Removing the unused analyzer adapter does not block any active stage behavior. -- The cleanup should be implemented in the listed order so inert schema removal is separated from behavior removal. diff --git a/docs/roadmap/cli.md b/docs/roadmap/cli.md deleted file mode 100644 index 245c41d..0000000 --- a/docs/roadmap/cli.md +++ /dev/null @@ -1,255 +0,0 @@ -# Roadmap: Session-Oriented CLI Cleanup - -Status: Implemented - -## Problem - -Narratio's public CLI has accumulated too many top-level commands. Several -commands are session-scoped operator helpers, but they currently appear as -independent top-level verbs: - -- `plan` -- `status` -- `restore` -- `artifacts list` -- `locks` -- `session validate` -- `session init` - -This makes the command surface harder to learn because the CLI does not clearly -separate primary workflow actions from session inspection, initialization, -restore, and helper operations. - -## Target Model - -Keep primary workflow commands at top level: - -- `run` -- `run-stage` -- `resume` -- `analyze` -- `publish` -- `clean` -- `session` - -Keep `clean` top-level because it can operate on one session or all local -sessions and is a workspace maintenance command, not only a session helper. - -Move session-scoped helper commands under `narratio session` and use positional -session identifiers: - -- `narratio session init [--remote|--output ] [--flags]` -- `narratio session validate [--flags]` -- `narratio session status [--flags]` -- `narratio session plan [--flags]` -- `narratio session restore [--flags]` -- `narratio session artifacts [--remote] [--flags]` -- `narratio session locks [--flags]` -- `narratio session locks add [--reason ] [--force] [--flags]` -- `narratio session locks remove [--flags]` - -Update top-level workflow commands to use positional session identifiers: - -- `narratio run [--flags]` -- `narratio resume [--flags]` -- `narratio analyze [--flags]` -- `narratio publish [--flags]` -- `narratio run-stage [--flags]` - -The positional session ID replaces `--session-id` as the primary public -interface. Existing `--config`, `--campaign`, `--session`, and -`--previous-session-id` flags remain available where they are meaningful. - -## Command Mapping - -| Current command | Target command | -| --- | --- | -| `narratio run --session-id ` | `narratio run ` | -| `narratio resume --session-id ` | `narratio resume ` | -| `narratio analyze --session-id ` | `narratio analyze ` | -| `narratio publish --session-id ` | `narratio publish ` | -| `narratio run-stage [flags] --session-id ` | `narratio run-stage [flags]` | -| `narratio plan --session-id ` | `narratio session plan ` | -| `narratio status --session-id ` | `narratio session status ` | -| `narratio restore --session-id ` | `narratio session restore ` | -| `narratio artifacts list --session-id ` | `narratio session artifacts ` | -| `narratio locks --session-id ` | `narratio session locks ` | -| `narratio locks add --session-id ` | `narratio session locks add ` | -| `narratio locks remove --session-id ` | `narratio session locks remove ` | -| `narratio session validate --session-id ` | `narratio session validate ` | -| `narratio session init --session-id ` | `narratio session init ` | -| `narratio clean --session-id ` | `narratio clean ` | -| `narratio clean --all` | unchanged | - -`clean` remains top-level, but its session-scoped form should also move from -`--session-id` to positional `` for consistency. - -## Compatibility Policy - -This is a hard public CLI cleanup after the migration step lands. - -During Step 1, old forms may remain as compatibility aliases to keep the -implementation reviewable. During Step 2, remove the old forms from command -dispatch, tests, docs, and examples: - -- remove top-level `plan`; -- remove top-level `status`; -- remove top-level `restore`; -- remove top-level `artifacts`; -- remove top-level `locks`; -- remove `--session-id` from the public command syntax for session-aware - commands. - -Do not keep long-term deprecated aliases unless a later roadmap explicitly -chooses a compatibility window. - -`status --manifest` does not fit the session-oriented command shape. Remove it -from the public CLI in this cleanup. If direct manifest inspection is needed -later, add a separate diagnostic command in a future roadmap rather than keeping -it as a special case in `session status`. - -## Implementation Step 1: Add New Session-Oriented Interface - -Status: Implemented - -Add the target command forms while preserving current behavior internally. - -Implementation requirements: - -- Add positional session ID parsing helpers in `internal/app`. -- Keep the existing `loadCommandConfig` behavior and populate - `config.SessionLoadOptions.SessionID` from the positional ID. -- Add or update command wrappers: - - `Run(ctx, args, out)` parses `run `. - - `Resume(ctx, args, out)` parses `resume `. - - `Analyze(ctx, args, out)` parses `analyze `. - - `Publish(ctx, args, out)` parses `publish `. - - `RunStage(ctx, args, out)` parses `run-stage `. - - `Clean(ctx, args, out)` parses `clean ` and keeps - `clean --all`. -- Extend `Session(ctx, args, out)` dispatch to support: - - `init ` - - `validate ` - - `status ` - - `plan ` - - `restore ` - - `artifacts ` - - `locks ` - - `locks add ` - - `locks remove ` -- Keep storage access through the existing app-level object-store helper. -- Keep AWS SDK details behind storage adapters. -- Keep the runner, stages, manifest behavior, archive behavior, restore - planning, lock semantics, and artifact catalog behavior unchanged. - -Acceptance criteria: - -- New forms execute the same code paths and produce equivalent results. -- Positional session ID mismatch with concrete local or remote `session.yml` - fails through existing session identity checks. -- Remote session fallback still uses the positional session ID as the lookup - value. -- Current command tests cover the new forms before old forms are removed. - -## Implementation Step 2: Remove Old Public Forms - -Status: Implemented - -Remove compatibility aliases and make the session-oriented interface the only -documented and supported public CLI. - -Implementation requirements: - -- Remove top-level dispatch for: - - `plan` - - `status` - - `restore` - - `artifacts` - - `locks` -- Remove `--session-id` flags from public session-aware commands. -- Keep `--previous-session-id` as an expected previous-session identity flag. -- Keep explicit `--session ` for loading a local concrete session file, - but still require the positional session ID for commands that operate on a - session. -- Remove `status --manifest`. -- Update usage text and invalid-command errors. -- Update `docs/cli.md` and `docs/operations.md` to use only the new forms. -- Update any roadmap docs that mention old helper command names. -- Update tests to expect old top-level helper commands and `--session-id` forms - to fail. - -Acceptance criteria: - -- Top-level command list is exactly: - - `run` - - `run-stage` - - `resume` - - `analyze` - - `publish` - - `clean` - - `session` -- All session-oriented commands use `narratio session - [--flags]`, except nested lock mutation forms, which use - `narratio session locks add|remove [--flags]`. -- `clean ` and `clean --all` remain top-level. -- Current-behavior docs and tests no longer advertise `--session-id`. - -## Test Guidance - -Focused tests: - -- `go test ./internal/app -run TestExecute -v` -- `go test ./internal/app -run 'Session|Status|Restore|Clean|Locks|Artifacts|Plan|RunStage|Analyze|Publish' -v` -- `go test ./internal/config -v` - -Full validation: - -- `go test ./...` - -Test cases to add or update: - -- `run ` loads local and remote sessions through the existing - config path. -- `resume `, `analyze `, and `publish ` - preserve current behavior. -- `run-stage ` preserves current run-stage output and - force/artifact-selection behavior. -- `session plan ` replaces top-level `plan`. -- `session status ` replaces top-level session status. -- `session validate ` replaces `session validate --session-id`. -- `session init ` writes the same local or remote concrete - `session.yml`. -- `session restore ` preserves restore planning/execution. -- `session artifacts --remote` preserves promoted-output - availability reporting. -- `session locks `, `session locks add `, and - `session locks remove ` preserve static/remote lock - semantics. -- `clean ` preserves session cleanup behavior, while `clean --all` - remains unchanged. -- Old top-level helper commands fail after Step 2. -- `--session-id` fails after Step 2. -- `status --manifest` fails after Step 2. - -## Documentation Guidance - -Update only after implementation lands: - -- `docs/cli.md` -- `docs/operations.md` -- any internal docs that list command names or examples - -Keep planned behavior only in this roadmap until the command refactor is -implemented. - -## Architecture Guardrails - -- Keep Narratio explicit and stage-driven. -- Do not introduce a generic workflow or command framework abstraction. -- Reuse existing app command helpers where practical. -- Keep config loading strict and centralized. -- Keep storage details behind `storage.ObjectStore`. -- Keep secret-backed object-store construction in `internal/app`. -- Preserve manifest-driven resume and restore behavior. -- Treat command renaming as a public CLI contract change, not a runtime stage - behavior change. diff --git a/docs/roadmap/publish.md b/docs/roadmap/publish.md deleted file mode 100644 index 35bffcb..0000000 --- a/docs/roadmap/publish.md +++ /dev/null @@ -1,287 +0,0 @@ -# Roadmap: Publish Contract - -Status: Planned - -## Problem - -Narratio currently uses several terms for one operator-facing concept: - -- `archive` is the stage that uploads run state and commits remote current - state. -- `publish` is the convenience command that force-runs the archive stage. -- `promote`, `promoted`, and `promote_artifacts` describe configured top-level - remote output writes. - -This mixed vocabulary makes the public contract harder to explain. Operators -should not need to distinguish "archive the run", "publish the run", and -"promote artifacts" when these are all part of the same publish action. - -The public model should use: - -- `publish` for the stage, command, config section, and action; -- `published` for an expected remote output that exists at its top-level - current destination; -- `publish rules` for the configured source-to-destination output rules; -- `locked` for sources whose top-level published destination must not be - overwritten; -- `run history` for immutable per-run records under `runs//`. - -## Target Model - -The public stage is `publish`. - -The convenience command: - - narratio publish - -is equivalent to: - - narratio run-stage publish --force - -Pipeline configuration uses `publish`: - - publish: - enabled: true - upload_run: true - outputs: - - source: narratio.transcript.final_trimmed - - source: narratio.artifact.session_recap - locks: - - source: narratio.artifact.session_recap - reason: Final recap was manually edited. - -Publish output rules are source-based. Each rule writes one artifact source to -a top-level remote destination. If `dest` is omitted, Narratio derives the -destination from the artifact registry or configured artifact output path. - -The mutable remote lock store remains: - - {session_prefix}/locks.yml - -Remote availability output uses `published`: - - Published: - - narratio.transcript.final_trimmed remote=published - - narratio.artifact.session_recap locked remote=published - -The remote key layout is otherwise unchanged: - -- immutable run history stays under `{session_prefix}/runs/{run_id}/`; -- current state stays under `{session_prefix}/current/manifest.json`; -- the final commit marker stays `{session_prefix}/current/run_id.txt`; -- `current/run_id.txt` is still written last. - -## Compatibility Policy - -This is a hard cutover. - -After implementation: - -- `pipeline.archive` is rejected by strict YAML decoding. -- `pipeline.archive.promote_artifacts` is rejected. -- `pipeline.workspace.cleanup_after_archive` is rejected. -- `pipeline.spool.delete_audio_after_archive` is rejected. -- `narratio run-stage archive ` is an unknown stage. -- manifests that record an `archive` stage are not migrated. -- old archive/promotion metadata keys are not read as compatibility fallbacks. - -Existing remote objects are not moved or renamed. Remote layout remains stable; -the rename changes configuration, stage names, status output, metadata, helper -names, tests, examples, and documentation. - -## Implementation Stages - -### Stage 1: Public Schema and Stage Cutover - -Status: Planned - -Switch the public config and stage contract to publish terminology. - -Implementation requirements: - -- Replace `pipeline.archive` with `pipeline.publish`. -- Replace `archive.promote_artifacts` with `publish.outputs`. -- Keep output rule fields: - - `source` - - `dest` - - `required` -- Replace `pipeline.archive.locks` with `pipeline.publish.locks`. -- Rename post-publish cleanup fields: - - `pipeline.workspace.cleanup_after_publish` - - `pipeline.spool.delete_audio_after_publish` -- Rename the registered stage from `archive` to `publish`. -- Update stage order so `publish` runs after `analyze` and before `notify`. -- Update top-level `narratio publish` to target stage `publish`. -- Keep `run-stage --artifacts publish` support. -- Reject `run-stage --artifacts ` for stages other than `analyze` and - `publish`. -- Preserve the remote commit ordering and storage adapter boundaries. - -Acceptance criteria: - -- `narratio run-stage publish ` executes the publish stage. -- `narratio publish ` force-runs the publish stage. -- `narratio run-stage archive ` fails clearly as an unknown stage. -- Old archive config fields fail strict decoding. -- New publish config fields load, default, and validate. - -### Stage 2: Runtime Terminology and Metadata Cutover - -Status: Planned - -Rename implementation concepts and runtime output to publish terminology. - -Implementation requirements: - -- Rename archive/promotion config and runtime types conceptually to - publish/output terms. -- Rename the remote key helper intent from promoted artifact to published - output while keeping generated keys unchanged. -- Change helper output: - - `Promoted:` becomes `Published:` - - `remote=promoted` becomes `remote=published` - - lock output uses `published` / `not-published` -- Rename publish-stage metadata, including: - - `promoted_paths` to `published_paths` - - `promoted_files_uploaded` to `published_files_uploaded` - - `skipped_optional_promotions` to `skipped_optional_outputs` - - `skipped_unselected_promotions` to `skipped_unselected_outputs` - - `locked_promotion_count` to `locked_output_count` - - `locked_promotions` to `locked_outputs` -- Update previous-cache and restore logic to use the `publish` stage and - `published_paths` metadata only. -- Keep run-local stage output materialization separate from remote publish - terminology. If local helper names are confusing, rename them to - materialization-oriented names rather than publish names. - -Acceptance criteria: - -- Status and artifact helper output use `Published:` and `remote=published`. -- Publish metadata contains only publish/output terminology. -- Previous-cache and restore behavior works with publish metadata and does not - depend on old archive metadata. -- Storage adapters still receive explicit keys and no AWS SDK details leak into - app or stage logic. - -### Stage 3: Documentation, Examples, and Final Cleanup - -Status: Planned - -Update implemented-behavior docs and remove stale public terminology after the -runtime cutover lands. - -Implementation requirements: - -- Update current-behavior docs: - - `docs/config.md` - - `docs/cli.md` - - `docs/operations.md` - - `docs/troubleshooting.md` - - `docs/architecture.md` - - relevant files under `docs/internal/` -- Rename `docs/internal/stage-archive.md` to - `docs/internal/stage-publish.md`. -- Update internal documentation links and references. -- Update examples to use: - - `publish.outputs` - - `publish.locks` - - `cleanup_after_publish` - - `delete_audio_after_publish` -- Update tests and final searches so old terminology remains only in this - roadmap as historical context. - -Acceptance criteria: - -- Maintained examples load and validate. -- Current-behavior docs describe only implemented publish terminology. -- Internal docs describe run history, published outputs, locks, and current - commit ordering clearly. -- Old user-facing archive/promote wording is removed except where discussing - historical behavior in this roadmap. - -## Test Guidance - -Focused tests: - -- `go test ./internal/config -v` -- `go test ./internal/app -v` -- `go test ./internal/stage -v` -- `go test ./internal/artifacts -v` - -Full validation: - -- `go test ./...` - -Config tests to add or update: - -- `publish.outputs` defaults and validates. -- `publish.outputs[].dest` derives from the artifact registry when omitted. -- `publish.locks` validates with the same source rules as publish outputs. -- old `archive` fails strict decode. -- old `promote_artifacts` fails strict decode. -- old cleanup fields fail strict decode. - -App and stage tests to add or update: - -- stage order uses `publish` before `notify`. -- `run-stage publish` succeeds. -- `run-stage archive` fails clearly. -- `narratio publish` force-runs the `publish` stage. -- `--artifacts` is accepted for `run-stage publish`. -- `--artifacts` error text names `analyze` and `publish`. -- status and artifact list output show `Published:` and `remote=published`. -- lock output says `published` or `not-published`. -- previous-cache and restore use `publish` stage metadata. - -Final searches: - -- Config/stage names: - - `pipeline.archive` - - `archive:` - - `promote_artifacts` - - `cleanup_after_archive` - - `delete_audio_after_archive` -- User-facing output: - - `Promoted:` - - `remote=promoted` - - `not-promoted` -- Runtime symbols and metadata: - - `ArchiveConfig` - - `ArchivePromotionRule` - - `S3PromotedArtifactKey` - - `promoted_paths` - - `promoted_files_uploaded` - - `locked_promotions` - -Expected remaining matches should be limited to this roadmap and narrowly -justified historical references until the roadmap is fully retired. - -## Architecture Guardrails - -- Keep Narratio explicit and stage-driven. -- Do not introduce a generic workflow or DAG abstraction. -- Keep strict YAML decoding. -- Keep remote path construction centralized. -- Keep storage details behind `storage.ObjectStore`. -- Keep AWS SDK types inside storage adapters. -- Preserve manifest-driven resume and restore behavior. -- Preserve current-state commit ordering with `current/run_id.txt` written - last. -- Keep raw secrets out of configs, manifests, logs, generated configs, and - publish metadata. -- Keep planned behavior only in this roadmap until implementation lands. - -## Assumptions - -- This is a breaking public/config/stage contract change. -- No compatibility aliases are retained. -- No migration logic is needed for in-progress local manifests. -- No migration logic is needed for old remote manifests. -- Existing remote objects are not moved or renamed. -- `publish` means uploading run history, writing configured published outputs, - and committing current state. -- `run history` is the preferred term for immutable per-run records under - `runs//`. -- `archive` remains acceptable only as a generic English concept in historical - roadmap context, not as a public Narratio command, config field, stage name, - or metadata term after implementation. diff --git a/docs/roadmap/transcripts.md b/docs/roadmap/transcripts.md deleted file mode 100644 index 4f0e6a1..0000000 --- a/docs/roadmap/transcripts.md +++ /dev/null @@ -1,210 +0,0 @@ -# Roadmap: Transcript Artifact Naming - -Status: Implemented - -## Problem - -Narratio's built-in transcript artifact names and canonical paths currently mix -operator-facing artifact meaning with historical stage and tool terminology: - -- `narratio.transcript.merged` maps to `transcripts/merged.json`. -- `narratio.transcript.polished` maps to `transcripts/processed.json`. -- `narratio.transcript.full` maps to `transcripts/normalized.json`. -- `narratio.transcript.trimmed` maps to `transcripts/trimmed.json`. - -This makes the public artifact surface harder to reason about. Operators see -`full`, `normalized`, `processed`, `polished`, `merged`, and `trimmed` used in -different places for the same transcript lineage. - -The transcript source IDs, canonical paths, and manifest output kinds should -use one vocabulary based on each transcript's role in the session artifact -model. - -## Target Model - -Built-in transcript artifacts should use these public source IDs, canonical -paths, and manifest output kinds: - -| Source ID | Canonical path | Output kind | Meaning | -| --- | --- | --- | --- | -| `narratio.transcript.base` | `transcripts/base.json` | `transcript_base` | First unified transcript produced by merging per-speaker raw transcripts. | -| `narratio.transcript.polished` | `transcripts/polished.json` | `transcript_polished` | Audita-polished transcript. | -| `narratio.transcript.final` | `transcripts/final.json` | `transcript_final` | Full final transcript after normalization. | -| `narratio.transcript.final_trimmed` | `transcripts/final.trimmed.json` | `transcript_final_trimmed` | Trimmed version of the final transcript. | - -Stage names remain process-oriented and unchanged: - -- `merge` -- `polish` -- `normalize` -- `trim` - -Downstream adapter contracts also remain process-oriented. The rename changes -Narratio's artifact model, canonical paths, config examples, archive promotion -sources, lock sources, status output, and documentation. It should not rename -the stages themselves or move external integration details into stage logic. - -## Compatibility Policy - -This is a hard cutover. - -After implementation, these old source IDs should be rejected: - -- `narratio.transcript.merged` -- `narratio.transcript.full` -- `narratio.transcript.trimmed` - -These old canonical paths should not be compatibility fallbacks: - -- `transcripts/merged.json` -- `transcripts/processed.json` -- `transcripts/normalized.json` -- `transcripts/trimmed.json` - -Existing remote archives are not migrated automatically. Operators who want -new promoted keys for old sessions should republish those sessions after -updating configuration. - -## Implementation Stages - -### Stage 1: Centralize Transcript Artifact Naming - -Status: Implemented - -Consolidate transcript artifact source IDs, canonical paths, and output kinds -in the artifact/path layer before changing runtime behavior. - -Implementation requirements: - -- Add or consolidate constants/helpers for built-in transcript source IDs. -- Add or consolidate constants/helpers for canonical transcript paths. -- Add or consolidate constants/helpers for transcript manifest output kinds. -- Keep source ID, path, and output-kind mappings in one registry or one - obviously shared artifact model. -- Update artifact registry tests to prove the target mapping. -- Avoid changing stage output behavior in this stage unless the implementation - is simpler and still reviewable. - -Acceptance criteria: - -- There is one clear source of truth for built-in transcript artifact names, - paths, and output kinds. -- Tests prove the new target mapping in the artifact layer. -- No generic workflow abstraction is introduced. - -### Stage 2: Rename Runtime Outputs and Defaults - -Status: Implemented - -Switch runtime behavior to the new transcript artifact model. - -Implementation requirements: - -- Update `merge` to write and record `transcripts/base.json` with - `transcript_base`. -- Update `polish` to write and record `transcripts/polished.json` with - `transcript_polished`. -- Update `normalize` to write and record `transcripts/final.json` with - `transcript_final`. -- Update `trim` to write and record `transcripts/final.trimmed.json` with - `transcript_final_trimmed`. -- Update normalize and trim defaults to: - - `pipeline.normalize.output_path: transcripts/final.json` - - `pipeline.trim.output_path: transcripts/final.trimmed.json` -- Update built-in artifact resolution, archive promotion destination - derivation, archive locks, status output, artifact catalog output, - previous-cache resolution, restore planning, and restore execution to use - the new registry values. -- Ensure old source IDs fail config validation. - -Acceptance criteria: - -- New runs produce the target canonical transcript files. -- Manifest outputs use the target output kinds. -- Archive promotion and lock validation accept new source IDs and reject old - source IDs. -- Status and artifact listing display new source IDs. -- Restore uses the new canonical paths and does not restore old transcript - paths as canonical outputs. - -### Stage 3: Update Tests, Examples, and Current Documentation - -Status: Implemented - -Update all implemented-behavior references after the runtime cutover lands. - -Implementation requirements: - -- Update examples to use `narratio.transcript.final_trimmed` and - `transcripts/final.trimmed.json` where trimmed final transcript is intended. -- Update examples that refer to full final transcripts to use - `narratio.transcript.final` and `transcripts/final.json`. -- Update `docs/config.md`, `docs/internal/artifacts.md`, stage docs, - CLI examples, operations examples, archive examples, lock examples, and - status/artifact-list examples. -- Add strict validation tests proving old source IDs are rejected. -- Mark roadmap stages implemented only after code, tests, examples, and - current-behavior docs agree. - -Acceptance criteria: - -- Maintained examples load and validate. -- Current-behavior docs describe only implemented new names. -- Old names remain only in this roadmap as historical/planning context until - this roadmap is retired or archived. - -## Test Guidance - -Run focused tests while implementing: - -- `go test ./internal/artifacts -v` -- `go test ./internal/config -v` -- `go test ./internal/stage -v` -- `go test ./internal/app -v` - -Run full validation before finishing: - -- `go test ./...` - -Run final searches: - -- Old source IDs: - - `narratio.transcript.merged` - - `narratio.transcript.full` - - `narratio.transcript.trimmed` -- Old paths: - - `transcripts/merged.json` - - `transcripts/processed.json` - - `transcripts/normalized.json` - - `transcripts/trimmed.json` -- Old output kinds: - - `transcript_merged` - - `transcript_processed` - - `transcript_normalized` - - `transcript_trimmed` - -Expected remaining matches should be limited to this roadmap's -historical/planning references until the roadmap is fully completed. - -## Architecture Guardrails - -- Keep Narratio explicit and stage-driven; do not introduce a generic workflow - or DAG abstraction. -- Keep path and artifact naming in centralized helpers rather than scattered - string concatenation. -- Preserve manifest-driven resume behavior. -- Keep storage details behind storage adapters. -- Do not move Seriatim, Audita, or Scriptorium command details out of their - adapter boundaries. -- Keep current-behavior documentation in sync only after implementation lands; - planned behavior belongs in this roadmap until then. - -## Assumptions - -- The cutover is intentionally not backward-compatible. -- Existing remote archive objects are not renamed or migrated automatically. -- Stage names and downstream adapter request field names remain unchanged. -- The term `base` is preferred over `merged` for the first unified transcript. -- The term `final` is preferred over `full` or `normalized` for the full final - transcript. -- The trimmed final path is `transcripts/final.trimmed.json`.