From e7319ea0160464dc418c884c406fbb81fce09839 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 9 Aug 2026 21:50:21 +0000 Subject: [PATCH] Align internal documentation and maintained examples --- README.md | 4 +- docs/config.md | 10 +-- docs/development.md | 2 +- docs/integrations/audita.md | 5 +- docs/integrations/scriptorium.md | 5 +- docs/integrations/seriatim.md | 5 +- docs/internal/adapters.md | 28 ++++++- docs/internal/artifacts.md | 48 +++++++++--- docs/internal/command-restore.md | 61 ++++++++-------- docs/internal/manifest.md | 44 ++++++++--- docs/internal/overview.md | 8 +- docs/internal/stage-analyze.md | 24 +++++- docs/internal/stage-merge.md | 12 +++ docs/internal/stage-normalize.md | 12 +++ docs/internal/stage-polish.md | 13 ++++ docs/internal/stage-prepare.md | 26 +++++-- docs/internal/stage-publish.md | 41 ++++++++--- docs/internal/stage-render.md | 13 ++++ docs/internal/stage-transcribe.md | 16 +++- docs/internal/stage-trim.md | 17 ++++- docs/internal/storage.md | 28 +++++-- docs/internal/workspace.md | 73 ++++++++++--------- docs/troubleshooting.md | 3 +- examples/README.md | 46 ++++++++++++ .../campaigns/sample-campaign/speakers.yml | 6 +- examples/speakers.yml | 6 +- 26 files changed, 416 insertions(+), 140 deletions(-) create mode 100644 examples/README.md diff --git a/README.md b/README.md index e573e55..8aa9d85 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,8 @@ This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete safe remedies. - [Integration contracts](docs/integrations/) — external tools, formats, and compatibility expectations. -- [Maintained examples](examples/) — complete copyable configuration and input - files. +- [Maintained examples](examples/README.md) — complete copyable configuration + and input files. ## Maintainer Documentation diff --git a/docs/config.md b/docs/config.md index 3289901..bd39941 100644 --- a/docs/config.md +++ b/docs/config.md @@ -274,10 +274,6 @@ Audio rules: ## Maintained Examples -- `examples/pipeline.minimal.yml` -- `examples/pipeline.production.yml` -- `examples/pipeline.full.annotated.yml` -- `examples/campaigns/sample-campaign/campaign.yml` -- `examples/session.local-audio.yml` -- `examples/session.s3-audio.yml` -- `examples/session.template.yml` +See the [maintained examples index](../examples/README.md) for complete pipeline, +campaign, session, template, and input fixtures. Keep complete copyable files +there rather than duplicating them in this reference. diff --git a/docs/development.md b/docs/development.md index 8b0e38b..3d355a2 100644 --- a/docs/development.md +++ b/docs/development.md @@ -24,7 +24,7 @@ polished transcripts and generated artifacts. Start with the | Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. | | Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. | | Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. | -| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/) | The reference owns field meanings; the examples directory owns complete copyable files. | +| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/README.md) | The reference owns field meanings; the examples directory owns complete copyable files. | | Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. | For an existing subsystem, also inspect its focused tests and package-level diff --git a/docs/integrations/audita.md b/docs/integrations/audita.md index 3f95a65..94b7889 100644 --- a/docs/integrations/audita.md +++ b/docs/integrations/audita.md @@ -54,5 +54,6 @@ Operator-selected values are defined under `pipeline.audita.*` in the [configuration reference](../config.md#pipeline). Maintained example with Audita config: -- `examples/pipeline.full.annotated.yml` -- `examples/pipeline.production.yml` + +- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml) +- [Production-shaped pipeline](../../examples/pipeline.production.yml) diff --git a/docs/integrations/scriptorium.md b/docs/integrations/scriptorium.md index 36cd645..c1473fd 100644 --- a/docs/integrations/scriptorium.md +++ b/docs/integrations/scriptorium.md @@ -63,5 +63,6 @@ per-artifact settings under `pipeline.scriptorium.artifacts.*`, in the [configuration reference](../config.md#pipeline). Maintained examples with Scriptorium config: -- `examples/pipeline.full.annotated.yml` -- `examples/pipeline.production.yml` + +- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml) +- [Production-shaped pipeline](../../examples/pipeline.production.yml) diff --git a/docs/integrations/seriatim.md b/docs/integrations/seriatim.md index cd9e26e..fc82fad 100644 --- a/docs/integrations/seriatim.md +++ b/docs/integrations/seriatim.md @@ -55,5 +55,6 @@ Operator-selected values are defined under `pipeline.seriatim.*` and [configuration reference](../config.md#pipeline). Maintained examples with Seriatim config: -- `examples/pipeline.full.annotated.yml` -- `examples/pipeline.production.yml` + +- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml) +- [Production-shaped pipeline](../../examples/pipeline.production.yml) diff --git a/docs/internal/adapters.md b/docs/internal/adapters.md index e28b198..d65d5c0 100644 --- a/docs/internal/adapters.md +++ b/docs/internal/adapters.md @@ -1,12 +1,17 @@ # Internal: Adapters ## Purpose -Define external integration boundaries and default adapter wiring used by app/stage orchestration. + +Explain the adapter interfaces and production composition used by application +and stage orchestration. Externally observable protocols and formats belong in +the [integration contracts](../integrations/). ## Adapter Boundaries + Narratio stage logic depends on adapter interfaces, not transport-specific details. Primary adapters: + - `whisperx.Client` - `seriatim.Runner` - `audita.Runner` @@ -15,17 +20,22 @@ Primary adapters: - `notify.Sender` ## Ownership + Adapters own: + - HTTP/subprocess/SDK argument and transport details. - Backend-specific request/response mapping. Adapters do not own: + - stage ordering/skip/force logic; - manifest transitions; - canonical path policy. ## Default Wiring + `internal/app/runner.go` initializes default adapters when not injected: + - WhisperX HTTP client from pipeline config. - Seriatim subprocess runner. - Audita subprocess runner. @@ -33,17 +43,29 @@ Adapters do not own: - Noop notifier (`notify.NoopSender`). - Object store only when required by selected stages/config. -Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization. +Object-store construction goes through `newCommandObjectStore`, which loads +configured filesystem secrets before adapter initialization. ## Failure Semantics + - Constructor errors fail stage execution setup early. - Runtime adapter errors propagate to stage code and then manifest failure handling. - Subprocess adapters persist stage logs/generated configs through stage-managed paths. -## Test Surfaces +## Implementation And Tests + +- Composition: `internal/app/runner.go`, `internal/app/object_store.go` +- Shared subprocess mechanics: `internal/adapters/subprocess` +- Focused adapters: `internal/adapters/{whisperx,seriatim,audita,scriptorium,storage,notify}` - `internal/adapters/whisperx/http_test.go` - `internal/adapters/seriatim/subprocess_test.go` - `internal/adapters/audita/subprocess_test.go` - `internal/adapters/scriptorium/subprocess_test.go` - `internal/adapters/storage/*_test.go` - `internal/app/runner_test.go` + +See the [WhisperX](../integrations/whisperx.md), +[Seriatim](../integrations/seriatim.md), [Audita](../integrations/audita.md), +and [Scriptorium](../integrations/scriptorium.md) contracts before changing an +externally visible boundary. Operator-selected values belong in +[Configuration](../config.md). diff --git a/docs/internal/artifacts.md b/docs/internal/artifacts.md index fe32855..3e215a4 100644 --- a/docs/internal/artifacts.md +++ b/docs/internal/artifacts.md @@ -1,17 +1,28 @@ # Internal: Artifacts ## Purpose -Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code. + +Explain the artifact registry, runtime catalog, resolver, previous-input +requirements, and shared remote current-state mechanics implemented by +`internal/artifacts`. Configuration fields that accept source IDs belong in +[Configuration](../config.md); physical placement belongs in +[Operations](../operations.md). ## Built-in Source IDs -- `narratio.transcript.base` -> `transcripts/base.json` (`merge`) -- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`) -- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`) -- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`) -- `narratio.transcript.final_markdown` -> `transcripts/final.md` (`render`) -- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` (`render`) -- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`) +The internal registry recognizes these stable built-in source IDs: + +- `narratio.transcript.base` +- `narratio.transcript.polished` +- `narratio.transcript.final` +- `narratio.transcript.final_trimmed` +- `narratio.transcript.final_markdown` +- `narratio.transcript.final_trimmed_markdown` +- `narratio.bounds.session` + +Registry entries bind each ID to its producer, output kind, canonical fallback, +and content validator. The focused stage documents own their input/output flow; +[Configuration](../config.md) owns where operators may select these IDs. ## Configured and Previous-Session Sources @@ -72,7 +83,8 @@ Validation by content type: ## Current-State Helpers -Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning. +Artifacts package owns shared remote current-state loading mechanics used by +restore, status and validation checks, and previous-cache planning. Core helpers: @@ -107,9 +119,27 @@ Caller policy is intentionally outside artifacts helpers: - spool/cache paths; - S3 session/run/current-state key layout. +See [Workspace Internals](workspace.md) for how callers consume local helpers +and [Operations](../operations.md#local-state-layout) for the authoritative +physical layout. + ## Invariants - source ID formats are stable contracts; - artifact resolution is deterministic and manifest-aware; - previous-session source resolution in `analyze` is local-only; - remote current-state key construction remains centralized in artifacts helpers. + +## Implementation And Tests + +- Registry and resolution: `internal/artifacts/artifact_resolver.go`, + `internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go` +- Current state: `internal/artifacts/current_state.go` +- Paths and keys: `internal/artifacts/paths.go`, + `internal/artifacts/s3_keys.go` +- Previous requirements: `internal/artifacts/previous_requirements.go` +- Tests: `internal/artifacts/artifact_resolver_test.go`, + `internal/artifacts/catalog_test.go`, + `internal/artifacts/current_state_test.go`, + `internal/artifacts/paths_model_test.go`, + `internal/artifacts/previous_requirements_test.go` diff --git a/docs/internal/command-restore.md b/docs/internal/command-restore.md index 6b20bf5..2befcb4 100644 --- a/docs/internal/command-restore.md +++ b/docs/internal/command-restore.md @@ -1,21 +1,20 @@ # Internal: Command Restore ## Purpose -Define the implemented `narratio session restore` command contract: -- committed remote current-state discovery; -- deterministic restore planning; -- safe local install semantics; -- durable restore reporting. +Explain the implemented restore discovery, planning, installation, and +reporting flow in `internal/app`. User invocation belongs in +[CLI](../cli.md#session-restore), and the operator recovery procedure and +physical restore scope belong in +[Operations](../operations.md#restore-workflow). + +Restore is split into explicit phases so remote authority, local conflict +policy, and filesystem mutation can be tested independently. ## Discovery Contract -Restore resolves remote committed state from the session publish current pointers: - -- `current/run_id.txt` (required, non-empty); -- `current/manifest.json` (required, decodable). - -Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config: +Discovery delegates current-state pointer and manifest loading to +`internal/artifacts`, then validates the result against the resolved request: - campaign must match; - session ID must match. @@ -34,26 +33,11 @@ Planner behavior: - remote list scope is the resolved session prefix; - remote-to-local mapping is traversal-safe; -- actions are sorted deterministically by local relative path. +- actions are sorted by local relative path and then remote key; +- force converts differing local targets from conflicts to downloads. -Restore scope from current remote state: - -- include `manifest.json`; -- include `transcripts/**`; -- include `artifacts/**`; -- include `audio/**` only with `--include-audio`. - -Explicit exclusions from current remote state mapping: - -- `current/**`; -- `runs/**`; -- `logs/**`; -- `reports/**`; -- `config/**`; -- `inputs/**`; -- `previous/**`. - -Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist. +Previous-cache files are planned separately through `previouscache.BuildPlan` +when configured previous-session requirements exist. ## Execution Contract @@ -73,8 +57,9 @@ Audio restore path: ## Reporting Contract -- `--dry-run`: prints summary only; no local writes. -- non-dry-run: writes `reports/restore-latest.json`. +- dry-run mode prints a summary and performs no local writes; +- execution mode persists the canonical restore report described in + [Operations](../operations.md#restore-workflow); - report includes plan counts, per-action status, and execution failures. ## Invariants @@ -82,3 +67,15 @@ Audio restore path: - restore uses committed remote current state as authority; - `current/run_id.txt` is the remote publish commit marker; - restore does not execute pipeline stages. + +## Implementation And Tests + +- Discovery: `internal/app/restore_discovery.go` +- Planning: `internal/app/restore_plan.go`, `internal/previouscache` +- Execution: `internal/app/restore_execute.go` +- Reporting and command coordination: `internal/app/restore_report.go`, + `internal/app/restore.go` +- Tests: `internal/app/restore_discovery_test.go`, + `internal/app/restore_plan_test.go`, + `internal/app/restore_execution_test.go`, + `internal/app/restore_workflow_test.go` diff --git a/docs/internal/manifest.md b/docs/internal/manifest.md index 9354f45..1506362 100644 --- a/docs/internal/manifest.md +++ b/docs/internal/manifest.md @@ -1,13 +1,15 @@ # Internal: Manifest ## Purpose -Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts. + +Explain the session-progress and invocation-audit models implemented by +`internal/manifest`. Physical manifest placement belongs in +[Operations](../operations.md#local-state-layout). ## Session Manifest -Path: -- `{workspace.root}/work/{campaign}/{session_id}/manifest.json` -Primary model (`manifest.Manifest`): +`manifest.Manifest` records: + - identity (`session_id`, `campaign`, `run_id`) - local path metadata (`local_workdir`, `local_spool_dir`) - remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`) @@ -15,7 +17,8 @@ Primary model (`manifest.Manifest`): - durable `artifacts` records - per-stage `stages` map -Stage status enum: +The model admits these stage states: + - `pending` - `running` - `succeeded` @@ -25,10 +28,9 @@ Stage status enum: - `interrupted` ## Run Manifest -Path: -- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json` -Run model (`manifest.RunManifest`): +`manifest.RunManifest` is created for each invocation and records: + - invocation identity and `force` flag - requested stages - per-stage action (`run` or `skip`) @@ -36,22 +38,40 @@ Run model (`manifest.RunManifest`): - overall run status (`running`, `succeeded`, `failed`) ## Persistence Semantics + `manifest.LocalStore`: + - validates loaded documents; - normalizes missing maps/stage records; - writes atomically via temp file + rename; - updates `updated_at` on save. ## Execution Semantics -Runner updates both manifests per stage transition: -- mark running -- mark succeeded/failed/skipped -- persist logs/generated config refs and metadata + +The application runner marks an executing stage running and then succeeded or +failed in both manifests, persisting each transition. On success it records +outputs, logs, generated configuration references, and metadata. A successful +forced rerun marks only succeeded downstream session-stage records stale. + +When an already-succeeded stage is skipped, the invocation run manifest records +the `skip` action and reason. The session manifest deliberately retains its +existing succeeded record because it remains the cross-invocation progress +authority. Session manifest is the authoritative stage-progress ledger across invocations. Run manifest is invocation-scoped audit state. ## Invariants + - stage resume/skip decisions are session-manifest driven. - force reruns stale downstream succeeded stages. - run manifest does not replace session manifest as progress authority. + +## Implementation And Tests + +- Models and transitions: `internal/manifest/manifest.go`, + `internal/manifest/run_manifest.go` +- Persistence and validation: `internal/manifest/store.go` +- Package tests: `internal/manifest/*_test.go` +- Assembled execution behavior: `internal/app/runner_test.go`, + `internal/app/run_stage_test.go` diff --git a/docs/internal/overview.md b/docs/internal/overview.md index c216410..c83c7b1 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -59,7 +59,8 @@ The implemented canonical order is: 10. `notify` (placeholder) `notify` currently has optional notifier call behavior and no persisted pipeline -outputs. The focused stage documents own implementation mechanics. The +outputs; its default collaborator is a no-op sender. The focused stage +documents own implementation mechanics. The [CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation and execution semantics. @@ -88,3 +89,8 @@ and execution semantics. Use this map to find an owner, then read its focused documentation and tests before changing behavior. + +The stage registry is implemented in `internal/stage/placeholders.go` and its +ordering is protected by `internal/app/planner_test.go`. Cross-invocation skip, +force, failure, and invalidation behavior is exercised in +`internal/app/runner_test.go` and `internal/app/run_stage_test.go`. diff --git a/docs/internal/stage-analyze.md b/docs/internal/stage-analyze.md index d12d1ff..001ba3d 100644 --- a/docs/internal/stage-analyze.md +++ b/docs/internal/stage-analyze.md @@ -1,24 +1,29 @@ # Stage: analyze ## Purpose + Execute selected configured Scriptorium artifacts in dependency order and materialize outputs. ## Inputs + - configured artifacts from `pipeline.scriptorium.artifacts` -- optional selected artifact filter (`--artifacts`) +- optional selected artifact keys supplied through the stage environment - built-in/configured/previous-session source references in artifact inputs Supported source families: - built-ins: `narratio.transcript.*`, `narratio.bounds.session` -- prepared stable inputs: `narratio.input.players`, `narratio.input.party`, `narratio.input.glossary` +- prepared stable inputs: `narratio.input.players`, `narratio.input.party`, + `narratio.input.glossary` - configured artifacts: `narratio.artifact.` - previous-session cache: `narratio.previous_session.artifact.` ## Outputs + - one materialized output per executed configured artifact (`output_path`) - stage metadata describing selected/generated/reused artifacts ## Key Behavior + - skips with metadata when Scriptorium config is missing or no executable artifacts remain. - builds runtime artifact catalog (built-ins + configured artifacts). - marks non-executable configured artifacts as reusable when output files already exist. @@ -30,13 +35,26 @@ Supported source families: - validates non-empty output files and materializes canonical outputs. ## Failure Semantics + - required missing configured/previous-session inputs fail. - missing required prepared stable input source includes prepare rerun guidance. - missing required previous-session source includes prepare rerun guidance. -- missing required `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` inputs includes render rerun guidance. +- missing required `narratio.transcript.final_markdown` or + `narratio.transcript.final_trimmed_markdown` inputs includes render rerun + guidance. - dependency cycles or unavailable required dependencies fail. - adapter validation failures fail stage. ## Invariants + - `analyze` performs no remote storage calls for previous-session source resolution. - output provenance and metadata are deterministic per execution. + +## Related Contracts And Tests + +- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact + fields and source-selection rules. +- [CLI](../cli.md) owns user-visible artifact selection. +- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract. +- Implementation and tests: `internal/stage/analyze.go`, + `internal/stage/analyze_test.go` diff --git a/docs/internal/stage-merge.md b/docs/internal/stage-merge.md index f05ab12..22b86d5 100644 --- a/docs/internal/stage-merge.md +++ b/docs/internal/stage-merge.md @@ -1,18 +1,22 @@ # Stage: merge ## Purpose + Normalize raw transcript inputs and merge into base transcript via Seriatim. ## Inputs + - `transcripts/raw/*.json` - `inputs/speakers.yml` - `inputs/autocorrect.yml` ## Outputs + - `transcripts/base.json` - optional `artifacts/seriatim.report.json` ## Key Behavior + - discovers and validates raw transcript inputs. - normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output. - merges normalized inputs (`seriatim.Run`) into base transcript. @@ -20,6 +24,14 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim. - materializes canonical outputs and records stage logs/generated configs. ## Invariants + - merge always consumes normalized forms of raw inputs. - base transcript must validate before stage success. - report output is config-gated. + +## Related Contracts And Tests + +- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics. +- [Configuration](../config.md#pipeline) owns operator-selected Seriatim values. +- Implementation and tests: `internal/stage/merge.go`, + `internal/stage/merge_test.go` diff --git a/docs/internal/stage-normalize.md b/docs/internal/stage-normalize.md index cd9a048..fa8b799 100644 --- a/docs/internal/stage-normalize.md +++ b/docs/internal/stage-normalize.md @@ -1,16 +1,20 @@ # Stage: normalize ## Purpose + Normalize polished transcript into final transcript using Seriatim. ## Inputs + - `transcripts/polished.json` ## Outputs + - `transcripts/final.json` (or configured normalize output path) - optional `artifacts/seriatim.normalize.report.json` ## Key Behavior + - resolves polished transcript from manifest outputs/canonical fallback. - applies `pipeline.normalize` config or default normalize config. - runs Seriatim normalize with configured timeout/binary. @@ -18,5 +22,13 @@ Normalize polished transcript into final transcript using Seriatim. - materializes canonical outputs and records logs/generated configs. ## Invariants + - final transcript must validate as processed transcript JSON (`segments` array). - normalize defaults are applied when `pipeline.normalize` is unset. + +## Related Contracts And Tests + +- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics. +- [Configuration](../config.md#pipeline) owns normalize fields and defaults. +- Implementation and tests: `internal/stage/normalize.go`, + `internal/stage/normalize_test.go` diff --git a/docs/internal/stage-polish.md b/docs/internal/stage-polish.md index 3125563..294a929 100644 --- a/docs/internal/stage-polish.md +++ b/docs/internal/stage-polish.md @@ -1,17 +1,21 @@ # Stage: polish ## Purpose + Run Audita polishing on base transcript and produce polished transcript. ## Inputs + - `transcripts/base.json` - `inputs/glossary.yml` ## Outputs + - `transcripts/polished.json` - optional `artifacts/audita.report.json` ## Key Behavior + - resolves base transcript from merge outputs/canonical fallback. - invokes Audita with configured model/module/runtime options. - validates processed transcript structure (`segments` array required). @@ -19,5 +23,14 @@ Run Audita polishing on base transcript and produce polished transcript. - materializes canonical outputs; records logs/generated config and adapter metadata. ## Invariants + - polished transcript schema validation is mandatory. - report output is config-gated. + +## Related Contracts And Tests + +- [Audita](../integrations/audita.md) owns subprocess, validation, and failure + semantics. +- [Configuration](../config.md#pipeline) owns operator-selected Audita values. +- Implementation and tests: `internal/stage/polish.go`, + `internal/stage/polish_test.go` diff --git a/docs/internal/stage-prepare.md b/docs/internal/stage-prepare.md index 2cc048f..4439b33 100644 --- a/docs/internal/stage-prepare.md +++ b/docs/internal/stage-prepare.md @@ -1,17 +1,18 @@ # Stage: prepare ## Purpose + Materialize canonical current-session inputs before processing stages. ## Inputs -- resolved `campaign.yml`, `session.yml`, and pipeline config + +- resolved campaign, session, and pipeline configuration - stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`) -- audio source: - - local `audio_dir`/`audio_files`, or - - S3 `audio_s3.prefix` +- one resolved local or S3 audio source - enabled configured artifact input requirements for previous-session sources ## Outputs + - `inputs/campaign.yml` - `inputs/session.yml` - `inputs/pipeline.resolved.yml` @@ -26,6 +27,7 @@ Materialize canonical current-session inputs before processing stages. - deterministic `manifest.inputs` entries (checksums + provenance) ## Key Behavior + - validates required config/store state. - enforces local audio vs S3 audio mutual exclusivity. - materializes S3 audio through spool/cache-aware logic. @@ -39,6 +41,20 @@ Materialize canonical current-session inputs before processing stages. Required previous-session inputs fail when unavailable; optional missing inputs are skipped. ## Invariants + - only `prepare` hydrates canonical `previous/` cache state. -- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting. +- managed previous artifacts are stored under `previous/artifacts/**` without + duplicate `artifacts/artifacts/` nesting. - `manifest.inputs` ordering is deterministic (`kind`, `path`). + +## Related Contracts And Tests + +- [Configuration](../config.md) owns audio selection, stable input fields, and + previous-session settings. +- [Operations](../operations.md) owns physical input, audio, spool, cache, and + previous-state layout. +- [Storage Internals](storage.md) and [Artifact Internals](artifacts.md) explain + the internal collaborators. +- Implementation and tests: `internal/stage/prepare.go`, + `internal/stage/prepare_test.go`, `internal/audio/s3_audio_test.go`, + `internal/previouscache/*_test.go` diff --git a/docs/internal/stage-publish.md b/docs/internal/stage-publish.md index d5e4f19..ceeaeea 100644 --- a/docs/internal/stage-publish.md +++ b/docs/internal/stage-publish.md @@ -1,23 +1,29 @@ # Stage: publish ## Purpose + Upload run/session outputs to object storage and atomically advance remote current state. ## Inputs -- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze` -- run root `runs/{run_id}/**` -- publish output rules (`pipeline.publish.outputs`) + +- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set) +- invocation-scoped run files +- resolved publish output rules - effective publish locks (static + remote merged lock set) -- local `previous/**` files when present +- durable previous-session cache files when present ## Outputs -- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`) -- uploaded selected publish outputs under session prefix -- uploaded `previous/**` files under session prefix when present -- uploaded `current/manifest.json` -- uploaded `current/run_id.txt` written last + +- uploaded invocation record and selected publish outputs; +- uploaded durable previous-session cache files when present; +- updated remote current manifest; and +- remote current-run commit marker, written last. + +Exact remote placement and the operator workflow belong in +[Operations](../operations.md#publish-workflow). ## Key Behavior + - stage can self-skip when publish disabled or run upload disabled. - validates prerequisite stage success and object-store availability. - collects deterministic run file list plus run `manifest.json`. @@ -28,6 +34,7 @@ Upload run/session outputs to object storage and atomically advance remote curre - writes remote current manifest before current run pointer. ## Metadata Signals + Includes counts/lists for: - run uploads - published output uploads @@ -39,6 +46,22 @@ Includes counts/lists for: - `current_pointer_written` ## Invariants + - `current/run_id.txt` is the remote commit marker and is written last. - run upload excludes `audio/**`. - publish locks are not overridden by `--force`. + +The commit boundary and cleanup gate are normative architecture invariants; see +[Architecture](../policy/architecture.md#publish-commit-boundary). + +## Related Contracts And Tests + +- [Configuration](../config.md#publish-configuration-summary) owns output and + static-lock fields. +- [Operations](../operations.md#publish-locks) owns remote lock lifecycle and + physical remote state. +- [Artifact Internals](artifacts.md) explains source resolution and current-state + helpers. +- Implementation and tests: `internal/stage/publish.go`, + `internal/stage/publish_test.go`, `internal/app/operator_helpers_test.go`, + `internal/app/post_publish_cleanup_test.go` diff --git a/docs/internal/stage-render.md b/docs/internal/stage-render.md index e9ba806..07cb964 100644 --- a/docs/internal/stage-render.md +++ b/docs/internal/stage-render.md @@ -1,17 +1,21 @@ # Stage: render ## Purpose + Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim. ## Inputs + - `narratio.transcript.final` (`transcripts/final.json`) - `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`) ## Outputs + - `narratio.transcript.final_markdown` -> `transcripts/final.md` - `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` ## Key Behavior + - uses `pipeline.render` settings (enabled/format/title/booleans). - resolves inputs manifest-first, then canonical fallback. - writes run-local outputs first, then materializes canonical session outputs. @@ -19,11 +23,20 @@ Render Markdown transcript artifacts from normalized JSON transcripts via Seriat - skips with stage metadata when `pipeline.render.enabled=false`. ## Failure Semantics + - missing normalized input fails with normalize rerun guidance. - missing trimmed input fails with trim rerun guidance. - adapter/subprocess failure fails stage. - empty render output files fail validation. ## Invariants + - only `format: markdown` is supported. - render stage owns production of built-in Markdown transcript sources. + +## Related Contracts And Tests + +- [Seriatim](../integrations/seriatim.md) owns render subprocess behavior. +- [Configuration](../config.md#pipeline) owns render fields and defaults. +- Implementation and tests: `internal/stage/render.go`, + `internal/stage/render_test.go` diff --git a/docs/internal/stage-transcribe.md b/docs/internal/stage-transcribe.md index 6d90ff9..c896526 100644 --- a/docs/internal/stage-transcribe.md +++ b/docs/internal/stage-transcribe.md @@ -1,22 +1,36 @@ # Stage: transcribe ## Purpose + Generate raw per-speaker transcripts from prepared audio using WhisperX. ## Inputs + - `audio/*.flac` from `prepare` ## Outputs + - `transcripts/raw/.json` ## Key Behavior + - discovers prepared audio from manifest inputs or canonical audio directory. - derives speaker ID from `.flac` basename. -- runs WhisperX with configured concurrency/retry settings. +- dispatches WhisperX requests through a bounded worker pool. - validates each output as JSON. - writes run-local outputs then materializes canonical transcript outputs. ## Invariants + - speaker basenames must be unique. - output path returned by adapter must match requested output path. - each successful output is validated before stage success. + +## Related Contracts And Tests + +- [WhisperX](../integrations/whisperx.md) owns HTTP, retry, timeout, and + cancellation semantics. +- [Configuration](../config.md#pipeline) owns concurrency and other + operator-selected values. +- Implementation and tests: `internal/stage/transcribe.go`, + `internal/stage/transcribe_test.go` diff --git a/docs/internal/stage-trim.md b/docs/internal/stage-trim.md index 8e2e430..c7ea5ca 100644 --- a/docs/internal/stage-trim.md +++ b/docs/internal/stage-trim.md @@ -1,16 +1,21 @@ # Stage: trim ## Purpose -Produce a final-trimmed transcript. By default, the stage generates bounds and applies a bounds-driven trim. + +Produce a final-trimmed transcript. By default, the stage generates bounds and +applies a bounds-driven trim. ## Inputs + - `transcripts/final.json` ## Outputs + - `transcripts/final.trimmed.json` (or configured trim output path) - when trim enabled: `artifacts/session_bounds.json` ## Key Behavior + When `trim.enabled=true`: - runs Scriptorium bounds artifact generation; - optionally runs render-debug output generation; @@ -23,6 +28,16 @@ When `trim.enabled=false`: - copies normalized transcript to trimmed output. ## Invariants + - normalized transcript is required input. - bounds output exists only in enabled trim path. - render-debug output is diagnostic and not a declared stage output. + +## Related Contracts And Tests + +- [Scriptorium](../integrations/scriptorium.md) owns bounds generation and + debug-render subprocess behavior. +- [Seriatim](../integrations/seriatim.md) owns transcript trimming behavior. +- [Configuration](../config.md#pipeline) owns trim fields and defaults. +- Implementation and tests: `internal/stage/trim.go`, + `internal/stage/trim_test.go` diff --git a/docs/internal/storage.md b/docs/internal/storage.md index c979093..640a960 100644 --- a/docs/internal/storage.md +++ b/docs/internal/storage.md @@ -1,10 +1,16 @@ # Internal: Storage ## Purpose -Document remote object-store contracts and S3 implementation behavior. + +Explain the object-store interface and S3 implementation used by Narratio. +Remote key layout and lifecycle belong in [Operations](../operations.md), while +operator-selected storage fields and credential mechanisms belong in +[Configuration](../config.md). ## Primary Contract + `storage.ObjectStore` interface: + - `List(ctx, prefix)` - `Download(ctx, key, localPath)` - `Upload(ctx, localPath, key, opts)` @@ -14,15 +20,15 @@ Key invariant: - callers pass full bucket-relative keys; - storage implementations do not infer campaign/session/run prefixes. -## Configuration -`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config. +## Composition -S3 constructor behavior: -- requires configured bucket; -- uses region/endpoint/path-style options when set; -- resolves credentials from configured env var names (with defaults). +`NewObjectStoreFromConfig` constructs the S3-backed implementation from +resolved configuration. The application loads configured filesystem secrets +before calling it. The storage adapter consumes already-resolved values; it does +not own discovery, defaults, or configuration validation. ## S3 Backend Behavior + - normalizes object keys. - `List` paginates and returns normalized `ObjectInfo`. - `Download` writes local files with parent directory creation. @@ -30,5 +36,13 @@ S3 constructor behavior: - `Exists` maps not-found responses to `false`. ## Invariants + - storage layer is stateless regarding manifest/stage progression. - publish ordering semantics are owned by stage/app code, not storage adapters. + +## Implementation And Tests + +- Contract and S3 adapter: `internal/adapters/storage` +- Composition: `internal/app/object_store.go` +- Tests: `internal/adapters/storage/*_test.go`, + `internal/app/object_store_test.go` diff --git a/docs/internal/workspace.md b/docs/internal/workspace.md index 19cf398..87882e7 100644 --- a/docs/internal/workspace.md +++ b/docs/internal/workspace.md @@ -1,57 +1,62 @@ # Internal: Workspace ## Purpose -Define local session layout, run-local stage layout, and cleanup guardrails. -## Canonical Session Layout -Session root: -- `{workspace.root}/work/{campaign}/{session_id}` +Explain the helpers that construct local session and run paths, coordinate +single-writer access, and confine cleanup. The authoritative physical layout and +retention workflow belong in [Operations](../operations.md#local-state-layout). -Core directories/files: -- `inputs/` -- `audio/` -- `transcripts/` -- `artifacts/` -- `reports/` -- `logs/` -- `config/` -- `current/` -- `runs/` -- `previous/` -- `manifest.json` -- `.lock` +## Path Ownership -`previous/` reserved files: -- `previous/manifest.json` -- `previous/artifacts/**` +`internal/artifacts` owns canonical session, run, spool, cache, and +previous-cache path construction. `SessionPathsFor` provides the session-scoped +path model, and layout creation goes through `EnsureLayoutFor`. Callers should +consume those helpers instead of rebuilding relative paths. + +`internal/pathsafe` and application cleanup helpers enforce confinement for +relative destinations and deletion targets. ## Run-Local Stage Layout -When run context is available, stages use: -- `runs/{run_id}/{stage}/outputs/` -- `runs/{run_id}/{stage}/logs/` -- `runs/{run_id}/{stage}/reports/` -- `runs/{run_id}/{stage}/config/` -- `runs/{run_id}/{stage}/scratch/` -Run-local outputs are materialized back into canonical session paths before stage success. -`previous/**` writes are never redirected to run-local output paths. +`internal/stage/run_local.go` maps stage outputs and diagnostics into an +invocation-scoped layout. Successful outputs are validated and atomically +materialized into canonical session paths before stage success. Managed +previous-session cache paths remain session-durable and are never redirected +into run-local output space. ## Locking -`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention). + +`artifacts.LocalStore` enforces the single-writer session lock via `.lock` +(`ErrLockConflict` on contention). ## Cleanup Semantics + Automatic post-publish cleanup: + - only runs when publish actually executed and succeeded; - requires `uploaded=true` and `current_pointer_written=true` metadata; -- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`; +- consumes the resolved cleanup policy described in + [Configuration](../config.md); - refuses unsafe deletes (root delete, out-of-root delete, symlink paths). -Manual clean command: -- `clean ` removes session work and spool subtree. -- `clean --all` removes all workspace work and spool children. -- durable cache is preserved unless `--clear-cache` is requested. +Manual cleanup uses the same scoped-target checks. Invocation syntax and exact +deletion scope belong in [CLI](../cli.md#clean) and +[Operations](../operations.md#cleanup). ## Invariants + - campaign-aware session root is mandatory. - manifest-driven stage state is durable across runs. - cleanup guardrails prevent destructive root/out-of-scope deletion. + +## Implementation And Tests + +- Path model and local store: `internal/artifacts/paths.go`, + `internal/artifacts/local.go` +- Run-local materialization: `internal/stage/run_local.go` +- Cleanup confinement: `internal/app/cleanup_targets.go`, + `internal/app/post_publish_cleanup.go` +- Tests: `internal/artifacts/paths_model_test.go`, + `internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`, + `internal/app/cleanup_targets_test.go`, + `internal/app/post_publish_cleanup_test.go` diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 8efd254..8c98caf 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -66,7 +66,8 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa Safe fix: -- align config with [docs/config.md](./config.md) and maintained files under `examples/`. +- align config with [Configuration](./config.md) and the + [maintained examples](../examples/README.md). Relevant reference: [Configuration](./config.md). diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..ec2b52f --- /dev/null +++ b/examples/README.md @@ -0,0 +1,46 @@ +# Maintained Examples + +These files are safe, copyable starting points for Narratio configuration and +input structure. Replace placeholder identifiers, storage names, integration +URLs, and paths for the target environment. Field meanings and defaults belong +in the [configuration reference](../docs/config.md). + +## Pipeline Configuration + +- [Minimal pipeline](pipeline.minimal.yml): campaign discovery plus the required + WhisperX URL. +- [Production-shaped pipeline](pipeline.production.yml): S3 storage, publish, + external tools, and configured Scriptorium artifacts. +- [Full annotated pipeline](pipeline.full.annotated.yml): every implemented + pipeline section with explanatory comments. + +The existing `internal/config` example test loads and validates each pipeline +with the sample campaign and a compatible local- or S3-audio session. + +## Campaign And Session Configuration + +- [Sample campaign](campaigns/sample-campaign/campaign.yml), its + [session template](campaigns/sample-campaign/session.template.yml), and its + adjacent stable inputs provide a complete campaign directory shape. +- [Local-audio session](session.local-audio.yml) and + [S3-audio session](session.s3-audio.yml) are concrete session files. +- [Session template](session.template.yml) and the campaign-local equivalent + demonstrate the narrow placeholder syntax consumed by `session init`; they + are templates, not runtime session files. + +## Input Fixtures + +- [Speakers](speakers.yml), [autocorrect](autocorrect.yml), and + [glossary](glossary.yml) show the standalone input shapes. +- The sample campaign references its local + [speakers](campaigns/sample-campaign/speakers.yml), + [autocorrect](campaigns/sample-campaign/autocorrect.yml), + [glossary](campaigns/sample-campaign/glossary.yml), + [players](campaigns/sample-campaign/players.yml), and + [party](campaigns/sample-campaign/party.yml) fixtures. +- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that + reserves the expected filename and directory shape. Replace it with a real + FLAC file before running transcription. + +The examples contain environment-variable names but no credential values. They +use fictional campaign content and reserved example domains. diff --git a/examples/campaigns/sample-campaign/speakers.yml b/examples/campaigns/sample-campaign/speakers.yml index 7357b02..b0100ab 100644 --- a/examples/campaigns/sample-campaign/speakers.yml +++ b/examples/campaigns/sample-campaign/speakers.yml @@ -1,5 +1,5 @@ match: - - speaker: "Eric Rakestraw" + - speaker: "Example Speaker" match: - - "Eric_Rakestraw" - - "Eric" + - "Example_Speaker" + - "Example" diff --git a/examples/speakers.yml b/examples/speakers.yml index 7357b02..b0100ab 100644 --- a/examples/speakers.yml +++ b/examples/speakers.yml @@ -1,5 +1,5 @@ match: - - speaker: "Eric Rakestraw" + - speaker: "Example Speaker" match: - - "Eric_Rakestraw" - - "Eric" + - "Example_Speaker" + - "Example"