diff --git a/docs/internal/README.md b/docs/internal/README.md index 3c2f02b..0c0f34f 100644 --- a/docs/internal/README.md +++ b/docs/internal/README.md @@ -1,29 +1,43 @@ # Internal Documentation Index ## Audience -Developers and LLM coding agents changing Narratio internals. +Developers and coding agents changing Narratio internals. ## Scope -Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior. +`docs/internal/` documents implemented internal contracts: stage boundaries, manifest/state behavior, artifact resolution, restore behavior, storage boundaries, and workspace invariants. -## Component Docs -- `adapters.md`: external adapter map, runtime wiring, and boundary ownership. -- `storage.md`: remote storage backend contracts and object-store invariants. -- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics. -- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior. -- `workspace.md`: local state model, manifests, run-local layout, materialization, and cleanup invariants. -- `command-restore.md`: restore command discovery/planning/execution/reporting contract. -- `stage-prepare.md`: input materialization and provenance capture. -- `stage-transcribe.md`: WhisperX transcript generation. -- `stage-merge.md`: Seriatim normalization + merge. -- `stage-polish.md`: Audita transcript polishing. -- `stage-normalize.md`: post-polish normalization. -- `stage-trim.md`: bounds-driven transcript trimming. -- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts. -- `stage-publish.md`: publish upload and current-pointer commit contract. +User and operator behavior belongs in: +- `docs/cli.md` +- `docs/config.md` +- `docs/operations.md` +- `docs/troubleshooting.md` -## External Integration Notes -- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`). +## Pipeline Stage Set +Canonical stage order from `internal/stage.All()`: +1. `prepare` +2. `transcribe` +3. `merge` +4. `polish` +5. `normalize` +6. `trim` +7. `analyze` +8. `publish` +9. `notify` (placeholder) -## Canonical Owner -`docs/internal/` is the canonical home for implemented internals per `docs/policy/documentation.md`. +`notify` is currently a placeholder stage with optional notifier call behavior; it has no persisted pipeline outputs. + +## Internal Component Docs +- `adapters.md`: external adapter boundaries and default runtime wiring. +- `artifacts.md`: canonical source IDs, runtime catalog behavior, and resolution rules. +- `manifest.md`: session and run manifest contracts. +- `storage.md`: object-store interface and S3 implementation behavior. +- `workspace.md`: local session layout, run-local layout, and cleanup guardrails. +- `command-restore.md`: restore discovery, planning, execution, and reporting. +- `stage-prepare.md` +- `stage-transcribe.md` +- `stage-merge.md` +- `stage-polish.md` +- `stage-normalize.md` +- `stage-trim.md` +- `stage-analyze.md` +- `stage-publish.md` diff --git a/docs/internal/adapters.md b/docs/internal/adapters.md index 8355093..4f9eff7 100644 --- a/docs/internal/adapters.md +++ b/docs/internal/adapters.md @@ -1,38 +1,12 @@ # Internal: Adapters ## Purpose -Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring. +Define external integration boundaries and default adapter wiring used by app/stage orchestration. -## Inputs and outputs -Inputs: -- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications). -- Resolved config values used to construct default adapters. +## Adapter Boundaries +Narratio stage logic depends on adapter interfaces, not transport-specific details. -Outputs: -- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details). -- Adapter errors returned to stage/app orchestration. - -## Boundaries -Owns: -- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls). -- Request/response contracts in `internal/adapters/*` packages. - -Does not own: -- Stage sequencing, skip/force/resume decisions. -- Manifest transition logic. -- Canonical workspace path policy. - -## Config fields used -Default wiring and adapter calls consume: -- `pipeline.whisperx.*` -- `pipeline.seriatim.*` -- `pipeline.audita.*` -- `pipeline.scriptorium.*` -- `pipeline.storage.*` and `pipeline.publish.*` (object-store construction/gating) -- `pipeline.notification.*` (sender boundary exists; placeholder behavior today) - -## External adapters used -Runtime env boundary fields (`internal/stage.Env`): +Primary adapters: - `whisperx.Client` - `seriatim.Runner` - `audita.Runner` @@ -40,39 +14,39 @@ Runtime env boundary fields (`internal/stage.Env`): - `storage.ObjectStore` - `notify.Sender` -Current execution usage: -- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`. -- Present but not used by implemented stage set: legacy `storage.Backend`. +Legacy compatibility boundary: +- `storage.Backend` remains in the storage adapter package and defaults to `NoopBackend`; current pipeline stages use `storage.ObjectStore`. -Default construction in app runner: -- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`. -- Object-store construction goes through app command orchestration so configured filesystem secrets are loaded before the storage adapter is initialized. -- Callers can inject test/fake implementations through `app.RunOptions.Env`. +## Ownership +Adapters own: +- HTTP/subprocess/SDK argument and transport details. +- Backend-specific request/response mapping. -## State and manifest behavior -- Adapters do not directly mutate session/run manifests. -- Stages and runner own manifest writes and stage status transitions. -- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata). +Adapters do not own: +- stage ordering/skip/force logic; +- manifest transitions; +- canonical path policy. -## Skip and resume behavior -- No adapter-level skip/resume semantics. -- Skip/resume/force behavior is decided by app runner using manifest stage state. +## Default Wiring +`internal/app/runner.go` initializes default adapters when not injected: +- WhisperX HTTP client from pipeline config. +- Seriatim subprocess runner. +- Audita subprocess runner. +- Scriptorium subprocess runner. +- Noop notifier (`notify.NoopSender`). +- Object store only when required by selected stages/config. -## Failure behavior -- Adapter constructors validate config-derived values and fail early on invalid required inputs. -- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic. -- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis. +Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization. -## Tests to inspect before changing +## Failure Semantics +- Constructor errors fail stage execution setup early. +- Runtime adapter errors propagate to stage code and then manifest failure handling. +- Subprocess adapters persist stage logs/generated configs through stage-managed paths. + +## Test Surfaces - `internal/adapters/whisperx/http_test.go` - `internal/adapters/seriatim/subprocess_test.go` - `internal/adapters/audita/subprocess_test.go` - `internal/adapters/scriptorium/subprocess_test.go` - `internal/adapters/storage/*_test.go` -- `internal/adapters/notify/fake_test.go` - `internal/app/runner_test.go` - -## Architectural invariants -- Stage code depends on adapter interfaces, not transport-specific implementation types. -- External SDK-specific types remain inside adapter implementations. -- Default app wiring must remain deterministic and overrideable via injected env dependencies. diff --git a/docs/internal/artifacts.md b/docs/internal/artifacts.md index d13dd75..387803a 100644 --- a/docs/internal/artifacts.md +++ b/docs/internal/artifacts.md @@ -1,106 +1,68 @@ # Internal: Artifacts ## Purpose -Define Narratio artifact identity, catalog, and source-resolution behavior for: -- built-in session artifacts; -- configured analyze artifacts; -- canonical previous-session artifact sources. +Define canonical artifact IDs, runtime catalog behavior, and source resolution rules for stage execution and publish output selection. -## Inputs and outputs -Inputs: -- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`); -- session paths and manifest inputs/outputs; -- runtime catalog state. +## Built-in Source IDs +- `narratio.transcript.base` -> `transcripts/base.json` (`merge`) +- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`) +- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`) +- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`) +- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`) -Outputs: -- resolved artifact path + provenance (`ResolvedSessionArtifact`); -- runtime catalog entries for built-ins and configured artifacts; -- requirement sets for canonical previous-session inputs; -- canonical S3 session, run, current, session config, session locks, audio, and published output keys. +## Configured and Previous-Session Sources +- Configured artifact source ID: `narratio.artifact.` +- Previous-session source ID: `narratio.previous_session.artifact.` -## Boundaries -Owns: -- built-in source registry and validation; -- configured artifact catalog identity (`narratio.artifact.`); -- canonical previous-session source parsing and resolution; -- previous-session requirement collection (`CollectPreviousArtifactRequirements`). +Configured and previous-session source IDs are validated by strict regex rules. -Does not own: -- prepare-stage remote hydration; -- stage success/skip transitions; -- publish upload orchestration. +## Runtime Catalog +`ArtifactCatalog` tracks: +- `planned`: source registered for run context. +- `executable`: selected and enabled for analyze execution. +- `available`: local file exists and validated. +- `provenance`: availability source. -## Built-in IDs -| Artifact ID | Canonical file | Producer stage | Output kind | -| --- | --- | --- | --- | -| `narratio.transcript.base` | `transcripts/base.json` | `merge` | `transcript_base` | -| `narratio.transcript.polished` | `transcripts/polished.json` | `polish` | `transcript_polished` | -| `narratio.transcript.final` | `transcripts/final.json` | `normalize` | `transcript_final` | -| `narratio.transcript.final_trimmed` | `transcripts/final.trimmed.json` | `trim` | `transcript_final_trimmed` | -| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | - -## Source families -- built-in: `narratio.transcript.*`, `narratio.bounds.session` -- configured artifact: `narratio.artifact.` -- canonical previous-session artifact: `narratio.previous_session.artifact.` - -## S3 key helpers -- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/` -- session config: `{session_prefix}/session.yml` -- session lock store: `{session_prefix}/locks.yml` -- run prefix: `{session_prefix}/runs/{run_id}/` -- audio prefix: `{session_prefix}/{session.inputs.audio_s3.prefix}` -- current manifest: `{session_prefix}/current/manifest.json` -- current run pointer: `{session_prefix}/current/run_id.txt` - -## Runtime catalog model -Catalog entries track: -- `planned`: source is registered for this run; -- `executable`: configured artifact is selected for analyze execution; -- `available`: usable local file exists (generated this run or reused from disk). - -Configured artifact provenance values include: +Current provenance values: - `generated.current_analyze_run` - `filesystem.disabled_artifact_output` - -Previous-session canonical provenance values include: - `manifest.inputs.previous_cache` - `current_session.previous_cache` -## Resolution behavior -- Built-ins resolve via manifest producer outputs first, then canonical fallback paths. -- Configured `narratio.artifact.` sources resolve through catalog availability. -- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths. -- Publish-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment. -- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`. +## Resolution Rules +Built-ins: +1. manifest producer outputs (when present) +2. canonical session path fallback -## Previous-session requirement scanning +Configured sources (`narratio.artifact.*`): +- resolve only through runtime catalog availability. + +Previous-session sources (`narratio.previous_session.artifact.*`): +- resolve only from local `previous/` cache state. +- prefer manifest-backed previous input paths. +- fallback to existing previous-cache filesystem paths. + +Validation by content type: +- transcript built-ins: JSON with top-level `segments` array. +- bounds built-in: valid JSON. +- configured/previous-session artifact files: non-empty text file. + +## Previous Requirement Collection `CollectPreviousArtifactRequirements`: - scans enabled configured artifacts only; -- includes canonical previous-session sources only; +- extracts only canonical previous-session sources; - deduplicates by artifact key; -- merges required/optional references (`required` wins); -- records deterministic sorted source locations for diagnostics. +- merges required/optional (required wins); +- returns deterministic ordering and source locations. -## Validation behavior -- transcript built-ins: JSON with top-level `segments` array; -- bounds built-in: valid JSON; -- configured and previous-session artifact files: non-empty text content. +## Key Path Helpers +`internal/artifacts/paths.go` defines canonical helpers for: +- session/work/run paths; +- previous-cache paths; +- spool/cache paths; +- S3 key layout helpers for session/run/current pointers. -## Failure behavior -- unsupported source or malformed canonical previous source: validation/resolution error; -- known source unavailable: `ErrSessionArtifactNotFound`; -- configured/previous canonical source without catalog: error; -- resolved invalid file content: validation error. - -## Tests to inspect before changing -- `internal/artifacts/artifact_resolver_test.go` -- `internal/artifacts/catalog_test.go` -- `internal/artifacts/previous_requirements_test.go` -- `internal/stage/prepare_previous_test.go` -- `internal/stage/analyze_test.go` - -## Architectural invariants -- Built-in source IDs are static. -- Configured and previous-session source IDs are artifact-key based and validation-gated. -- Resolution behavior remains deterministic and manifest-aware. +## Invariants +- Source ID formats are stable contracts. +- Resolution is deterministic and manifest-aware. +- Previous-session source resolution does not call remote storage in `analyze`; remote hydration is `prepare` responsibility. diff --git a/docs/internal/command-restore.md b/docs/internal/command-restore.md index 511a25a..66f4d1a 100644 --- a/docs/internal/command-restore.md +++ b/docs/internal/command-restore.md @@ -1,106 +1,65 @@ # Internal: Command Restore ## Purpose -Define the implemented `narratio session restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting. +Document the implemented `narratio session restore` command contract: +- committed remote current-state discovery; +- deterministic restore plan classification; +- safe local install semantics; +- durable restore reporting. -## Inputs and outputs -Inputs: -- CLI syntax: `narratio session restore `. -- CLI flags: `--config`, `--campaign`, `--campaign-file`, `--session`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`. -- Resolved/validated `pipeline.yml` and `session.yml`. -- Configured remote object store. -- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`). +## Discovery Contract +Restore discovers remote committed state using: +- `current/run_id.txt` (required, non-empty) +- `current/manifest.json` (required, decodable) -Outputs: -- Dry-run summary to stdout (plan + counts). -- Non-dry-run completion summary to stdout. -- Local durable session files restored under canonical session root. -- Non-dry-run restore report at `reports/restore-latest.json`. +Discovered manifest identity must match requested `session_id` and `campaign`. -## Boundaries -Owns: -- Restore command flag parsing and command wiring. -- Remote current-state discovery and identity validation. -- Restore plan construction and conflict classification. -- Restore execution for planned downloads. -- Restore report model and persistence. +## Plan Contract +Planner actions: +- `download` +- `skip_same` +- `conflict` -Does not own: -- Stage execution orchestration (`run`, `resume`, `run-stage`). -- Publish-stage behavior. -- Storage transport implementation details (owned by storage adapters). +Plan behavior: +- remote list scope is the resolved session prefix; +- mapping to local paths is traversal-safe; +- actions are sorted deterministically by local relative path. -## Config fields used -- Config/session discovery and templating fields consumed by all commands. -- `pipeline.workspace.root` (local restore target root). -- `pipeline.storage.*` (remote backend + publish identity derivation). -- `pipeline.storage.s3.*` identity components used by session-prefix helpers. -- `pipeline.spool.root` for active audio downloads. -- `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache. -- `session.session_id` -- `session.campaign` +Restore scope from current remote state: +- include `manifest.json` +- include `transcripts/**` +- include `artifacts/**` +- include `audio/**` only with `--include-audio` -## External adapters used -- `storage.ObjectStore` for `Exists`, `List`, `Download`. -- `artifacts.Store` (`LocalStore`) for layout and session lock management. -- `manifest.LocalStore` for manifest decode/validation and identity checks. +Explicit exclusions from current remote state mapping: +- `current/**` +- `runs/**` +- `logs/**` +- `reports/**` +- `config/**` +- `inputs/**` +- `previous/**` -## State and manifest behavior -- Restore is not a pipeline run and does not create a run manifest. -- Restore uses committed remote current state only: - - `current/run_id.txt` must exist and be non-empty. - - `current/manifest.json` must decode and match requested session/campaign. -- Non-dry-run writes restore files to canonical session paths. -- With `--include-audio`, restore uses the shared S3 audio cache for `audio/**` objects. Cache hits avoid object downloads; cache misses download through spool, install the work file, and populate cache. -- Manifest install behavior: - - validated before replacement. - - installed last among download actions. - - existing local manifest is preserved if restored manifest validation/install fails. -- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`. +Previous-cache restore files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist. -Restore path scope: -- includes: - - `manifest.json` - - `transcripts/**` - - `artifacts/**` - - `previous/**` - - `audio/**` only when `--include-audio` is set -- excludes: - - `runs/**` - - `logs/**` - - `reports/**` - - `config/**` - - `inputs/**` - - remote `current/**` pointer files as local restore targets +## Execution Contract +- non-manifest downloads happen before manifest install; +- `manifest.json` is installed last; +- downloads use sibling temp files + atomic rename; +- manifest replacement is validated before rename; +- failed installs do not roll back previously written files. -## Skip and resume behavior -- Restore does not participate in stage skip/resume decisions. -- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state. -- Audio cache is outside the workspace and is reused across restore and prepare invocations. -- Dry-run is read-only and returns plan output only. +Audio restore path: +- uses `audio.MaterializeS3Audio`; +- integrates spool and S3 audio cache paths; +- supports cache hit reuse without object redownload. -## Failure behavior -- Fails when storage backend is unavailable or publish identity cannot be resolved. -- Fails when remote current pointer/manifest is missing or invalid. -- Fails when remote manifest identity mismatches requested campaign/session. -- Fails on local conflicts unless `--force` is set. -- Fails fast on session lock acquisition conflict for non-dry-run execution. -- On execution failure, previously installed files remain; no rollback is performed. +## Reporting Contract +- dry-run: summary only (no writes). +- non-dry-run: writes `reports/restore-latest.json`. +- report captures plan counts, action status, and execution failures. -## Tests to inspect before changing -- `internal/app/restore_test.go` -- `internal/app/restore_discovery_test.go` -- `internal/app/restore_plan_test.go` -- `internal/app/restore_execution_test.go` -- `internal/app/restore_workflow_test.go` -- `internal/artifacts/archive_identity_test.go` - -## Architectural invariants -- Restore relies on centralized path/key helpers (`internal/artifacts`) rather than ad hoc key building. -- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files. -- Local path mapping is traversal-safe and constrained to session root. -- Restore scope is deterministic and path-classified: - - include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**` - - include `audio/**` only with `--include-audio` - - exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**` -- Command remains standalone; no implicit `run --restore` behavior. +## Invariants +- restore uses only committed remote current state as authority. +- `current/run_id.txt` is the remote commit marker. +- restore is a standalone command and does not run stages. diff --git a/docs/internal/manifest.md b/docs/internal/manifest.md index a8c3a69..9354f45 100644 --- a/docs/internal/manifest.md +++ b/docs/internal/manifest.md @@ -1,81 +1,57 @@ # Internal: Manifest ## Purpose -Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior. +Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts. -## Inputs and outputs -Inputs: -- Session identity and run identity from app orchestration. -- Stage transition events and stage result payloads. +## Session Manifest +Path: +- `{workspace.root}/work/{campaign}/{session_id}/manifest.json` -Outputs: -- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`. -- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`. +Primary model (`manifest.Manifest`): +- identity (`session_id`, `campaign`, `run_id`) +- local path metadata (`local_workdir`, `local_spool_dir`) +- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`) +- `inputs` records +- durable `artifacts` records +- per-stage `stages` map -## Boundaries -Owns: -- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records). -- Stage status/action transition methods. -- Persistent store contract (`manifest.Store`) and local JSON store implementation. +Stage status enum: +- `pending` +- `running` +- `succeeded` +- `failed` +- `skipped` +- `stale` +- `interrupted` -Does not own: -- Stage implementation details. -- Path construction policy outside manifest file persistence calls. -- CLI command behavior. +## Run Manifest +Path: +- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json` -## Config fields used -Manifest package itself does not read config directly. +Run model (`manifest.RunManifest`): +- invocation identity and `force` flag +- requested stages +- per-stage action (`run` or `skip`) +- per-stage status +- overall run status (`running`, `succeeded`, `failed`) -Manifest identity fields are populated by app/stage orchestration from: -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.storage.s3.*` (when publish/S3 identity is set) +## Persistence Semantics +`manifest.LocalStore`: +- validates loaded documents; +- normalizes missing maps/stage records; +- writes atomically via temp file + rename; +- updates `updated_at` on save. -## External adapters used -- No external service adapters. -- Uses local filesystem for persistence via `manifest.LocalStore`. +## Execution Semantics +Runner updates both manifests per stage transition: +- mark running +- mark succeeded/failed/skipped +- persist logs/generated config refs and metadata -## State and manifest behavior -Session manifest model: -- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`). -- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata. +Session manifest is the authoritative stage-progress ledger across invocations. +Run manifest is invocation-scoped audit state. -Run manifest model: -- Tracks one invocation (`run_id`) with requested stages and force mode. -- Tracks per-stage action (`run` or `skip`) and per-stage status. -- Tracks overall run status (`running`, `succeeded`, `failed`). - -Persistence behavior: -- Load validates required identity/timestamp fields and normalizes maps/records. -- Save updates `updated_at` and writes JSON atomically (temp file + rename). -- Session and run manifests are saved incrementally before/after stage transitions. - -Relationship during execution: -- Runner updates both manifests for every stage transition. -- Session manifest is the durable pipeline-progress ledger. -- Run manifest is invocation history and audit record. -- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.` for configured artifact identity. - -## Skip and resume behavior -- Resume and skip decisions are based on session-manifest stage statuses. -- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest. -- Run manifest records whether each stage was executed or skipped in that invocation. - -## Failure behavior -- Stage failure marks both manifests failed for that stage and records error messages/timestamps. -- Save failures are returned immediately and fail the command. -- Invalid/malformed manifest files fail load with explicit validation/decode errors. - -## Tests to inspect before changing -- `internal/manifest/manifest_test.go` -- `internal/manifest/run_manifest_test.go` -- `internal/manifest/store_test.go` -- `internal/app/runner_test.go` -- `internal/app/run_control_test.go` -- `internal/app/resume_run_stage_test.go` - -## Architectural invariants -- Session manifest is authoritative for stage progression across invocations. -- Run manifest is invocation-scoped and never replaces session manifest as progress authority. -- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern). +## Invariants +- stage resume/skip decisions are session-manifest driven. +- force reruns stale downstream succeeded stages. +- run manifest does not replace session manifest as progress authority. diff --git a/docs/internal/stage-analyze.md b/docs/internal/stage-analyze.md index a78b863..1181d23 100644 --- a/docs/internal/stage-analyze.md +++ b/docs/internal/stage-analyze.md @@ -1,80 +1,38 @@ # Stage: analyze ## Purpose -Execute selected configured Scriptorium artifacts in deterministic dependency order and materialize successful outputs to canonical session artifact paths. +Execute selected configured Scriptorium artifacts in dependency order and materialize outputs. -## Inputs and outputs -Inputs: -- configured artifact definitions from `pipeline.scriptorium.artifacts`; -- selected artifact filter (`--artifacts`) when provided; -- resolved artifact sources from resolver/catalog. +## Inputs +- configured artifacts from `pipeline.scriptorium.artifacts` +- optional selected artifact filter (`--artifacts`) +- built-in/configured/previous-session source references in artifact inputs -Source types used by analyze: -- built-ins: `narratio.transcript.*`, `narratio.bounds.session`; -- configured artifacts: `narratio.artifact.`; -- canonical previous-session artifacts: `narratio.previous_session.artifact.`. +Supported source families: +- built-ins: `narratio.transcript.*`, `narratio.bounds.session` +- configured artifacts: `narratio.artifact.` +- previous-session cache: `narratio.previous_session.artifact.` -Outputs: -- materialized configured artifact files at each configured `output_path`; -- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info). +## Outputs +- one materialized output per executed configured artifact (`output_path`) +- stage metadata describing selected/generated/reused artifacts -## Boundaries -Owns: -- runtime artifact catalog construction; -- selected-artifact planning and dependency ordering; -- per-input resolution and required/optional handling; -- Scriptorium render/run invocation; -- run-local output generation and canonical materialization. +## Key Behavior +- skips with metadata when Scriptorium config is missing or no executable artifacts remain. +- builds runtime artifact catalog (built-ins + configured artifacts). +- marks non-executable configured artifacts as reusable when output files already exist. +- validates selected artifact dependency order (cycle-safe topo ordering). +- resolves required/optional inputs per artifact source definition. +- resolves previous-session sources from local `previous/` cache only. +- runs optional render-debug, then artifact execution. +- validates non-empty output files and materializes canonical outputs. -Does not own: -- prepare-time previous-session hydration; -- object-store access for previous-session sources; -- publish output rule behavior. +## Failure Semantics +- required missing configured/previous-session inputs fail. +- missing required previous-session source includes prepare rerun guidance. +- dependency cycles or unavailable required dependencies fail. +- adapter validation failures fail stage. -## Config fields used -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.scriptorium.binary` -- `pipeline.scriptorium.config_path` -- `pipeline.scriptorium.timeout` -- `pipeline.scriptorium.render_debug` -- `pipeline.scriptorium.artifacts..*` - -## External adapters used -- Scriptorium adapter: - - optional `RenderArtifact` when render-debug is enabled; - - `RunArtifact` for artifact generation. - -## State and manifest behavior -- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`. -- Builds runtime catalog with built-ins and configured `narratio.artifact.` entries. -- Non-executable configured artifacts may still be marked available from existing canonical output files. -- Resolves canonical previous-session sources from local prepared `previous/` cache: - - prefers manifest-backed previous input paths when present; - - may fall back to current-session `previous/` filesystem paths. -- Analyze does not call object storage for canonical previous-session source resolution. -- Required canonical previous-session input missing: - - fails with guidance to run `narratio run-stage --force prepare`. -- Optional missing sources are omitted from adapter input paths. - -## Skip and resume behavior -- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set. -- Analyze is stage-scoped for resume; no per-artifact manifest resume state. -- `--artifacts` filters executable artifacts but does not imply force rerun. - -## Failure behavior -- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs. -- Required unavailable configured artifact source (`narratio.artifact.`) fails before invocation. -- Required canonical previous-session source fails with prepare-rerun guidance. - -## Tests to inspect before changing -- `internal/stage/analyze_test.go` -- `internal/artifacts/catalog_test.go` -- `internal/artifacts/artifact_resolver_test.go` -- `internal/app/restore_workflow_test.go` - -## Architectural invariants -- Canonical previous-session behavior is local-cache only during analyze. -- Generated outputs are validated and materialized before stage success is recorded. -- Resolver/catalog decisions stay deterministic and validation-gated. +## Invariants +- `analyze` performs no remote storage calls for previous-session source resolution. +- output provenance and metadata are deterministic per execution. diff --git a/docs/internal/stage-merge.md b/docs/internal/stage-merge.md index 21f92b6..f05ab12 100644 --- a/docs/internal/stage-merge.md +++ b/docs/internal/stage-merge.md @@ -1,63 +1,25 @@ # Stage: merge ## Purpose -Normalize per-speaker raw transcripts and merge them into the base transcript via Seriatim. +Normalize raw transcript inputs and merge into base transcript via Seriatim. -## Inputs and Outputs -Inputs: +## Inputs - `transcripts/raw/*.json` - `inputs/speakers.yml` - `inputs/autocorrect.yml` -Outputs: +## Outputs - `transcripts/base.json` -- optional `artifacts/seriatim.report.json` (when report enabled) +- optional `artifacts/seriatim.report.json` -## Boundaries -Owns: -- Raw transcript discovery/validation -- Per-input normalize calls to Seriatim -- Final merge call to Seriatim -- Run-local log/config/report path wiring -- Materialization of base/report outputs to canonical paths +## Key Behavior +- discovers and validates raw transcript inputs. +- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output. +- merges normalized inputs (`seriatim.Run`) into base transcript. +- validates merged transcript and optional report JSON. +- materializes canonical outputs and records stage logs/generated configs. -Does not own: -- Transcript polishing or downstream artifact generation - -## Config Fields Used -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.seriatim.binary` -- `pipeline.seriatim.timeout` -- `pipeline.seriatim.output_schema` -- `pipeline.seriatim.coalesce_gap` -- `pipeline.seriatim.report` -- `pipeline.seriatim.env.*` - -## External Adapters Used -- Seriatim adapter: - - `Normalize` for each raw input - - `Run` for final merge - -## State and Manifest Behavior -- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory. -- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled. -- Materializes canonical base transcript and optional report. -- Records normalized-input provenance and adapter metadata in stage metadata. - -## Skip and Resume Behavior -- Runner-level skip applies when already succeeded and not forced. -- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation. - -## Failure Behavior -- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid base output JSON, or invalid report JSON when enabled. - -## Tests to Inspect Before Changing -- `internal/stage/merge_test.go` -- `internal/adapters/seriatim/subprocess_test.go` - -## Architectural Invariants -- Merge consumes normalized forms of each raw transcript. -- Base transcript must validate before materialization. -- Report output is optional and gated by config. +## Invariants +- merge always consumes normalized forms of raw inputs. +- base transcript must validate before stage success. +- report output is config-gated. diff --git a/docs/internal/stage-normalize.md b/docs/internal/stage-normalize.md index 325e469..cd9a048 100644 --- a/docs/internal/stage-normalize.md +++ b/docs/internal/stage-normalize.md @@ -1,56 +1,22 @@ # Stage: normalize ## Purpose -Normalize the polished transcript into the full final transcript and optionally emit a normalize report. +Normalize polished transcript into final transcript using Seriatim. -## Inputs and Outputs -Inputs: +## Inputs - `transcripts/polished.json` -Outputs: +## Outputs - `transcripts/final.json` (or configured normalize output path) - optional `artifacts/seriatim.normalize.report.json` -## Boundaries -Owns: -- Polished transcript discovery/validation -- Normalize request construction and invocation -- Optional normalize report wiring -- Promotion of final transcript and optional report +## Key Behavior +- resolves polished transcript from manifest outputs/canonical fallback. +- applies `pipeline.normalize` config or default normalize config. +- runs Seriatim normalize with configured timeout/binary. +- validates normalized transcript and optional report. +- materializes canonical outputs and records logs/generated configs. -Does not own: -- Bounds detection or segment trimming - -## Config Fields Used -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.normalize.output_path` -- `pipeline.normalize.output_schema` -- `pipeline.normalize.report` -- `pipeline.seriatim.binary` -- `pipeline.seriatim.timeout` - -## External Adapters Used -- Seriatim adapter (`Normalize`). - -## State and Manifest Behavior -- Reads polished transcript from polish outputs in manifest when present; falls back to canonical path. -- Uses run-local output/report/log/config paths when run layout is enabled. -- Promotes canonical final transcript and optional normalize report. -- Records adapter/result metadata including source path selection. - -## Skip and Resume Behavior -- Runner-level skip applies when already succeeded and not forced. -- Forced reruns can stale downstream succeeded stages. - -## Failure Behavior -- Fails on missing/invalid polished transcript, adapter error, invalid final output, or invalid report output when report enabled. - -## Tests to Inspect Before Changing -- `internal/stage/normalize_test.go` -- `internal/adapters/seriatim/subprocess_test.go` - -## Architectural Invariants -- Final output must validate as transcript-compatible JSON (`segments` array required). -- Default normalize config is applied when `pipeline.normalize` is unset. +## Invariants +- final transcript must validate as processed transcript JSON (`segments` array). +- normalize defaults are applied when `pipeline.normalize` is unset. diff --git a/docs/internal/stage-polish.md b/docs/internal/stage-polish.md index a9f54bc..3125563 100644 --- a/docs/internal/stage-polish.md +++ b/docs/internal/stage-polish.md @@ -1,69 +1,23 @@ # Stage: polish ## Purpose -Polish the base transcript with Audita and produce a polished transcript for downstream normalization/analyze. +Run Audita polishing on base transcript and produce polished transcript. -## Inputs and Outputs -Inputs: +## Inputs - `transcripts/base.json` - `inputs/glossary.yml` -Outputs: +## Outputs - `transcripts/polished.json` -- optional `artifacts/audita.report.json` (when report enabled) +- optional `artifacts/audita.report.json` -## Boundaries -Owns: -- Base transcript discovery/validation -- Audita invocation request construction -- Run-local logs/config/work-dir/report wiring -- Promotion of polished transcript and optional report +## Key Behavior +- resolves base transcript from merge outputs/canonical fallback. +- invokes Audita with configured model/module/runtime options. +- validates processed transcript structure (`segments` array required). +- validates optional report JSON. +- materializes canonical outputs; records logs/generated config and adapter metadata. -Does not own: -- Upstream merge normalization -- Downstream normalize/trim/analyze logic - -## Config Fields Used -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.audita.binary` -- `pipeline.audita.timeout` -- `pipeline.audita.llm_api_key_env` -- `pipeline.audita.modules` -- `pipeline.audita.base_url` -- `pipeline.audita.model` -- `pipeline.audita.transcript_description` -- `pipeline.audita.config_path` -- `pipeline.audita.output_schema` -- `pipeline.audita.work_dir_retention` -- `pipeline.audita.total_llm_concurrency` -- `pipeline.audita.proposal_llm_concurrency` -- `pipeline.audita.validation_model` -- `pipeline.audita.validation_llm_concurrency` -- `pipeline.audita.report` - -## External Adapters Used -- Audita adapter (`env.Audita.Run`). - -## State and Manifest Behavior -- Reads base transcript from merge manifest outputs when available; falls back to canonical base path. -- Uses run-local output/report/log/config/scratch paths when run layout is enabled. -- Promotes canonical `transcripts/polished.json` and optional report. -- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata. - -## Skip and Resume Behavior -- Runner-level skip applies when already succeeded and not forced. -- Forced rerun can stale downstream succeeded stages via runner invalidation. - -## Failure Behavior -- Fails on missing/invalid base transcript, missing glossary, adapter error, invalid polished output shape (`segments` array required), or invalid report JSON when enabled. - -## Tests to Inspect Before Changing -- `internal/stage/polish_test.go` -- `internal/adapters/audita/subprocess_test.go` - -## Architectural Invariants -- Polished transcript must contain a top-level `segments` array. -- Report behavior is strictly config-gated. -- Stage output canonicalization always ends at `transcripts/polished.json`. +## Invariants +- polished transcript schema validation is mandatory. +- report output is config-gated. diff --git a/docs/internal/stage-prepare.md b/docs/internal/stage-prepare.md index 31b8eae..517893f 100644 --- a/docs/internal/stage-prepare.md +++ b/docs/internal/stage-prepare.md @@ -1,124 +1,42 @@ # Stage: prepare ## Purpose -Materialize canonical current-session input state and provenance before downstream stages run. +Materialize canonical current-session inputs before processing stages. -Prepare owns: -- local input file materialization (`inputs/**`); -- audio input materialization (`audio/**`); -- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources. - -## Inputs and outputs -Inputs: -- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`); -- remote session provenance when `session.yml` was loaded from S3; -- campaign or session input files (`speakers`, `autocorrect`, `glossary`); +## Inputs +- resolved `campaign.yml`, `session.yml`, and pipeline config +- stable input files (`speakers`, `autocorrect`, `glossary`) - audio source: - - local: `session.inputs.audio_dir` or `session.inputs.audio_files`; - - S3: `session.inputs.audio_s3.prefix`; -- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning); -- remote previous-session current publish state when previous hydration is required. + - local `audio_dir`/`audio_files`, or + - S3 `audio_s3.prefix` +- enabled configured artifact input requirements for previous-session sources -Outputs: -- `inputs/campaign.yml`; -- `inputs/session.yml`; -- `inputs/pipeline.resolved.yml`; -- `inputs/speakers.yml`; -- `inputs/autocorrect.yml`; -- `inputs/glossary.yml`; -- `audio/*.flac` in canonical session `audio/`; -- optional `previous/manifest.json`; -- optional `previous/artifacts/**`; -- deterministic `manifest.Inputs` records with checksums and provenance metadata. +## Outputs +- `inputs/campaign.yml` +- `inputs/session.yml` +- `inputs/pipeline.resolved.yml` +- `inputs/speakers.yml` +- `inputs/autocorrect.yml` +- `inputs/glossary.yml` +- `audio/*.flac` +- optional `previous/manifest.json` +- optional `previous/artifacts/**` +- deterministic `manifest.inputs` entries (checksums + provenance) -## Boundaries -Owns: -- input path resolution and materialization; -- S3 audio list/download/copy flow; -- previous-session artifact requirement collection from enabled configured artifacts; -- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state). - -Does not own: -- transcript or artifact generation; -- analyze-stage source resolution; -- publish commit behavior. - -## Config fields used -- `session.session_id` -- `session.previous_session_id` -- `session.campaign` -- `session.inputs.speakers_file` -- `session.inputs.autocorrect_file` -- `session.inputs.glossary_file` -- `session.inputs.audio_dir` -- `session.inputs.audio_files` -- `session.inputs.audio_s3.prefix` -- `pipeline.workspace.root` -- `pipeline.spool.root` -- `pipeline.cache.root` -- `pipeline.cache.s3_audio` -- `pipeline.storage.s3.bucket` -- `pipeline.storage.s3.root_prefix` -- `pipeline.scriptorium.artifacts..enabled` -- `pipeline.scriptorium.artifacts..inputs..source` -- `pipeline.scriptorium.artifacts..inputs..required` -- `campaign.campaign_id` -- `campaign.inputs.speakers_file` -- `campaign.inputs.autocorrect_file` -- `campaign.inputs.glossary_file` - -## External adapters used -- `storage.ObjectStore` for: - - S3 audio listing/downloads; - - previous-session current pointer/manifest/artifact object checks and downloads. - -## State and manifest behavior -- Ensures workspace layout exists. -- Materializes canonical input files and audio files. -- For S3 audio, uses run-scoped spool for active downloads and durable cache for reusable audio files; cache hits copy directly to work audio without downloading the object again. -- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`. -- Resolves campaign-provided stable input paths relative to `campaign.yml`. -- Resolves session-provided stable input overrides relative to `session.yml`. -- Scans enabled configured artifact inputs for canonical sources: - - `narratio.previous_session.artifact.` -- If one or more canonical previous-session requirements exist: +## Key Behavior +- validates required config/store state. +- enforces local audio vs S3 audio mutual exclusivity. +- materializes S3 audio through spool/cache-aware logic. +- scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements. +- when previous requirements exist: - clears managed `previous/` state; - - hydrates required/optional previous artifacts from the configured previous session’s committed publish current state; - - writes `previous/manifest.json` and hydrated `previous/artifacts/**`; - - stores publish-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`; - - records hydrated previous inputs in `manifest.Inputs` with source `previous_session_publish.current`. -- If no canonical previous-session requirements exist, prepare does not manage `previous/`. -- `manifest.Inputs` is sorted deterministically by `(kind, path)`. -- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file. + - builds previous-cache remote plan; + - downloads previous manifest/artifacts; + - records previous inputs in `manifest.inputs`. -## Required and optional previous-session behavior -- `previous_session_id` unset: - - if any referenced previous artifact is required: fail; - - if all referenced previous artifacts are optional: continue and omit them. -- Previous session publish current pointer or manifest missing: - - if any referenced previous artifact is required: fail; - - if all referenced previous artifacts are optional: continue and omit missing ones. -- Missing required previous artifact object: fail. -- Missing optional previous artifact object: omit. -- Downloaded previous artifacts must validate as non-empty files. +Required previous-session inputs fail when unavailable; optional missing inputs are skipped. -## Skip and resume behavior -- Runner-level skip remains authoritative: - - if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs. -- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs. - -## Failure behavior -- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors. -- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun: - - `narratio run-stage --force prepare` - -## Tests to inspect before changing -- `internal/stage/prepare_test.go` -- `internal/stage/prepare_previous_test.go` -- `internal/artifacts/previous_requirements_test.go` -- `internal/app/runner_test.go` - -## Architectural invariants -- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive. -- Storage keys are computed by callers using path helpers; storage adapter receives explicit keys. -- `prepare` is the only stage that hydrates canonical previous-session cache state. +## Invariants +- only `prepare` hydrates canonical `previous/` cache state. +- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting. +- `manifest.inputs` ordering is deterministic (`kind`, `path`). diff --git a/docs/internal/stage-publish.md b/docs/internal/stage-publish.md index 8231473..208bd0c 100644 --- a/docs/internal/stage-publish.md +++ b/docs/internal/stage-publish.md @@ -1,88 +1,44 @@ # Stage: publish ## Purpose -Publish durable run/session state to object storage, then atomically advance remote current state. +Upload run/session outputs to object storage and atomically advance remote current state. -## Inputs and Outputs -Inputs: -- session manifest and prerequisite stage records -- run root contents under `runs/{run_id}/` -- publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`) -- effective source-based publish locks from static config and remote session lock store -- session-level `previous/**` cache files when present +## Inputs +- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze` +- run root `runs/{run_id}/**` +- publish output rules (`pipeline.publish.outputs`) +- effective publish locks (static + remote merged lock set) +- local `previous/**` files when present -Outputs: -- uploaded run files under `{session_prefix}/runs/{run_id}/...` -- uploaded published outputs under `{session_prefix}/...` -- uploaded session previous-cache files under `{session_prefix}/previous/...` when present -- `{session_prefix}/current/manifest.json` -- `{session_prefix}/current/run_id.txt` written last +## Outputs +- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`) +- uploaded selected publish outputs under session prefix +- uploaded `previous/**` files under session prefix when present +- uploaded `current/manifest.json` +- uploaded `current/run_id.txt` written last -## Boundaries -Owns: -- publish enable/disable gate behavior -- prerequisite stage success enforcement -- run file collection and upload (excluding `audio/`) -- publish output rule resolution and upload -- publish lock enforcement -- session previous-cache file collection/upload -- commit pointer publish order +## Key Behavior +- stage can self-skip when publish disabled or run upload disabled. +- validates prerequisite stage success and object-store availability. +- collects deterministic run file list plus run `manifest.json`. +- resolves publish output sources through runtime artifact catalog and manifest-aware resolution. +- selected artifact filter applies to configured artifact sources only. +- locked outputs are skipped intentionally (including required ones). +- optional missing outputs are skipped; required missing unlocked outputs fail. +- writes remote current manifest before current run pointer. -Does not own: -- stage execution before publish -- post-publish local cleanup policy execution (handled by app cleanup logic) +## Metadata Signals +Includes counts/lists for: +- run uploads +- published output uploads +- previous uploads +- skipped optional outputs +- skipped unselected outputs +- locked outputs +- current-state key paths +- `current_pointer_written` -## Config Fields Used -- `pipeline.publish.enabled` -- `pipeline.publish.upload_run` -- `pipeline.publish.outputs` -- `pipeline.publish.locks` -- `{session_prefix}/locks.yml` loaded by app orchestration before publish execution -- `pipeline.storage.s3.bucket` -- `pipeline.storage.s3.root_prefix` -- `pipeline.workspace.root` -- `session.campaign` -- `session.session_id` - -## External Adapters Used -- Object storage backend (`env.ObjectStore`) for upload/list primitives. - -## State and Manifest Behavior -- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`. -- Resolves bucket/prefix from manifest identity first, then config fallback. -- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists. -- Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged. -- When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.` sources; built-in transcript and bounds outputs still publish. -- Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources. -- Writes metadata including: - - upload counts/paths - - `previous_files_uploaded` and `previous_uploaded_paths` - - `published_files_uploaded` and `published_paths` - - `skipped_optional_outputs` - - `skipped_unselected_outputs` - - `locked_output_count` and `locked_outputs` - - `current_manifest_key` - - `current_run_id_key` - - `current_pointer_written` -- On skipped publish path, returns metadata with `skipped=true` and pointer not written. - -## Skip and Resume Behavior -- Stage may self-skip (metadata skip) when publish disabled or run upload disabled. -- Runner-level skip also applies for previously succeeded stage unless forced. - -## Failure Behavior -- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures. -- Locked required outputs are intentional skips and do not fail publish. -- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail. - -## Tests to Inspect Before Changing -- `internal/stage/archive_test.go` -- `internal/app/post_archive_cleanup_test.go` - -## Architectural Invariants -- Run upload excludes `audio/` subtree. -- Session `previous/**` is publishable durable input/provenance state, not run-local output. -- Ordinary `--force` does not override publish locks. -- Malformed or unreadable remote lock store fails publish-capable execution before output uploads. -- `current/manifest.json` uploads before `current/run_id.txt`. -- `current/run_id.txt` is the remote publish commit marker. +## Invariants +- `current/run_id.txt` is the remote commit marker and is written last. +- run upload excludes `audio/**`. +- publish locks are not overridden by `--force`. diff --git a/docs/internal/stage-transcribe.md b/docs/internal/stage-transcribe.md index 8b7ce96..6d90ff9 100644 --- a/docs/internal/stage-transcribe.md +++ b/docs/internal/stage-transcribe.md @@ -1,58 +1,22 @@ # Stage: transcribe ## Purpose -Generate per-speaker raw transcripts from prepared audio using WhisperX. +Generate raw per-speaker transcripts from prepared audio using WhisperX. -## Inputs and Outputs -Inputs: -- `audio/*.flac` prepared by `prepare` +## Inputs +- `audio/*.flac` from `prepare` -Outputs: -- `transcripts/raw/.json` for each input audio file +## Outputs +- `transcripts/raw/.json` -## Boundaries -Owns: -- Discovering prepared audio inputs -- Deriving speaker ids from audio basenames -- Parallel WhisperX invocation with bounded concurrency -- Validating produced JSON and materializing run-local outputs +## Key Behavior +- discovers prepared audio from manifest inputs or canonical audio directory. +- derives speaker ID from `.flac` basename. +- runs WhisperX with configured concurrency/retry settings. +- validates each output as JSON. +- writes run-local outputs then materializes canonical transcript outputs. -Does not own: -- Transcript merge/polish/normalize/trim/analyze - -## Config Fields Used -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.whisperx.transcribe_url` -- `pipeline.whisperx.language` -- `pipeline.whisperx.timeout` -- `pipeline.whisperx.retries` -- `pipeline.whisperx.retry_delay` -- `pipeline.whisperx.concurrency` - -## External Adapters Used -- WhisperX adapter (`env.WhisperX.Transcribe`). - -## State and Manifest Behavior -- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled. -- Validates each generated transcript JSON before materialization. -- Materializes canonical outputs to `transcripts/raw/*.json`. -- Records per-file metadata (attempts/status/duration/output path) in stage metadata. - -## Skip and Resume Behavior -- Runner-level skip applies for previously succeeded stage unless forced. -- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic. - -## Failure Behavior -- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails. -- Cancels in-flight workers after first terminal error. - -## Tests to Inspect Before Changing -- `internal/stage/transcribe_test.go` -- `internal/app/whisperx_wiring_test.go` - -## Architectural Invariants -- Speaker identity is derived from `.flac` basename and must be unique. -- Every successful speaker output must be valid JSON before materialization. -- Canonical raw transcript set is the only supported merge input surface. +## Invariants +- speaker basenames must be unique. +- output path returned by adapter must match requested output path. +- each successful output is validated before stage success. diff --git a/docs/internal/stage-trim.md b/docs/internal/stage-trim.md index d520e14..16b66bc 100644 --- a/docs/internal/stage-trim.md +++ b/docs/internal/stage-trim.md @@ -1,75 +1,28 @@ # Stage: trim ## Purpose -Optionally trim the final transcript to session bounds; always produce a durable final-trimmed transcript. +Produce a final-trimmed transcript; optionally generate bounds-driven trim. -## Inputs and Outputs -Inputs: +## Inputs - `transcripts/final.json` -Outputs: +## Outputs - `transcripts/final.trimmed.json` (or configured trim output path) - when trim enabled: `artifacts/session_bounds.json` -## Boundaries -Owns: -- Trim-enabled switch behavior -- Bounds generation via Scriptorium artifact run -- Bounds validation against final transcript -- Keep-selector derivation and Seriatim trim invocation -- Copy-through behavior when disabled or bounds indicate unchanged transcript +## Key Behavior +When `trim.enabled=false`: +- copies normalized transcript to trimmed output. -Does not own: -- Upstream normalization -- Downstream artifact analysis +When `trim.enabled=true`: +- runs Scriptorium bounds artifact generation; +- optionally runs render-debug output generation; +- validates bounds payload against transcript; +- derives keep selector; +- either copies unchanged transcript or runs Seriatim trim; +- validates trimmed transcript and materializes bounds output. -## Config Fields Used -- `session.session_id` -- `session.campaign` -- `pipeline.workspace.root` -- `pipeline.trim.enabled` -- `pipeline.trim.output_path` -- `pipeline.trim.bounds.prompt_id` -- `pipeline.trim.bounds.profile_id` -- `pipeline.trim.bounds.timeout` -- `pipeline.trim.bounds.output_path` -- `pipeline.trim.bounds.transcript_input_name` -- `pipeline.trim.bounds.render_debug` -- `pipeline.trim.bounds.render_output_path` -- `pipeline.seriatim.binary` -- `pipeline.seriatim.timeout` -- `pipeline.scriptorium.binary` -- `pipeline.scriptorium.config_path` -- `pipeline.scriptorium.timeout` - -## External Adapters Used -- Scriptorium adapter: - - optional `RenderArtifact` for bounds debug render - - `RunArtifact` for bounds output -- Seriatim adapter: - - `Trim` when bounds indicate trimming is required - -## State and Manifest Behavior -- Reads final transcript from normalize manifest outputs when available; falls back to canonical path. -- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled. -- Materializes canonical final-trimmed transcript and session bounds when trim is enabled. -- Records bounds diagnostics, trim action, keep selector, and adapter metadata. - -## Skip and Resume Behavior -- Runner-level skip applies when already succeeded and not forced. -- Forced reruns can stale downstream succeeded stages. -- When `trim.enabled=false`, stage still succeeds by copying final to final-trimmed output. - -## Failure Behavior -- Fails on missing/invalid final transcript. -- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid final-trimmed output. - -## Tests to Inspect Before Changing -- `internal/stage/trim_test.go` -- `internal/adapters/scriptorium/subprocess_test.go` -- `internal/adapters/seriatim/subprocess_test.go` - -## Architectural Invariants -- Trim never falls back to polished transcript; final transcript is required input. -- `session_bounds` output exists only for enabled trim path. -- Render-debug artifacts are diagnostics and not declared stage outputs. +## Invariants +- normalized transcript is required input. +- bounds output exists only in enabled trim path. +- render-debug output is diagnostic and not a declared stage output. diff --git a/docs/internal/storage.md b/docs/internal/storage.md index 610f7a4..0c8422a 100644 --- a/docs/internal/storage.md +++ b/docs/internal/storage.md @@ -1,75 +1,37 @@ # Internal: Storage ## Purpose -Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`. +Document remote object-store contracts and S3 implementation behavior. -## Inputs and outputs -Inputs: -- Resolved storage config (`pipeline.storage.*`). -- Already-loaded environment variables for configured S3 credentials. -- Bucket-relative object keys and local file paths from app/stage orchestration. +## Primary Contract +`storage.ObjectStore` interface: +- `List(ctx, prefix)` +- `Download(ctx, key, localPath)` +- `Upload(ctx, localPath, key, opts)` +- `Exists(ctx, key)` -Outputs: -- Listed/downloaded/uploaded object metadata (`ObjectInfo`). -- Existence checks and storage-layer errors. +Key invariant: +- callers pass full bucket-relative keys; +- storage implementations do not infer campaign/session/run prefixes. -## Boundaries -Owns: -- Remote object-store interface and implementation details. -- S3 client wiring and API calls. -- Object key normalization and upload/download/list primitives. +## Configuration +`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config. -Does not own: -- Session/run prefix semantics. -- Archive commit order semantics. -- Manifest updates. -- Filesystem secret loading from `pipeline.secrets.env_dir`. +S3 constructor behavior: +- requires configured bucket; +- uses region/endpoint/path-style options when set; +- resolves credentials from configured env var names (with defaults). -## Config fields used -- `pipeline.storage.backend` -- `pipeline.storage.s3.bucket` -- `pipeline.storage.s3.region` -- `pipeline.storage.s3.endpoint` -- `pipeline.storage.s3.force_path_style` -- `pipeline.storage.s3.access_key_id_env` -- `pipeline.storage.s3.secret_access_key_env` +## S3 Backend Behavior +- normalizes object keys. +- `List` paginates and returns normalized `ObjectInfo`. +- `Download` writes local files with parent directory creation. +- `Upload` streams local file and returns remote metadata. +- `Exists` maps not-found responses to `false`. -## External adapters used -Storage package contracts: -- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`. -- `Backend` (legacy compatibility boundary): currently implemented with `NoopBackend` only. +## Legacy Compatibility Interface +`storage.Backend` (with `ArchiveRequest`) remains as compatibility surface with `NoopBackend`; it is not used by current stage execution. -Implementations: -- `S3Backend`: AWS SDK-backed `ObjectStore` implementation. -- `FakeBackend`: deterministic test `ObjectStore` and compatibility backend. -- `NoopBackend`: deterministic no-op compatibility backend for wiring/tests. - -## State and manifest behavior -- Storage implementations are stateless with respect to manifest/session lifecycle. -- Caller supplies fully-qualified bucket-relative keys. -- Storage layer does not infer campaign/session/run/root-prefix semantics. -- Caller controls publish ordering; storage layer executes individual operations in the order invoked. - -## Skip and resume behavior -- No storage-level skip/resume behavior. -- Skip/resume decisions are made by stage/app logic before storage calls occur. - -## Failure behavior -- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing. -- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails. -- App command orchestration loads configured filesystem secrets before calling the object-store factory. -- CRUD operations return contextual errors (including not-found behavior via `Exists`). -- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed). -- Remote session loading uses `List` to find the exact `session.yml` key and `Download` to materialize it to a local temp file. - -## Tests to inspect before changing -- `internal/adapters/storage/factory_test.go` -- `internal/adapters/storage/s3_backend_test.go` -- `internal/adapters/storage/fake_test.go` -- `internal/adapters/storage/keys_test.go` -- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `publish`) - -## Architectural invariants -- Callers pass full bucket-relative keys. -- Storage backends must not prepend or infer narratio prefixes. -- Remote transport details remain isolated to storage adapter implementations. +## Invariants +- storage layer is stateless regarding manifest/stage progression. +- publish ordering semantics are owned by stage/app code, not storage adapters. diff --git a/docs/internal/workspace.md b/docs/internal/workspace.md index ca97482..e8863d5 100644 --- a/docs/internal/workspace.md +++ b/docs/internal/workspace.md @@ -1,78 +1,57 @@ -# Workspace internals +# Internal: Workspace ## Purpose -Define the local durable and run-local workspace model used by stages, manifests, resume, and publish. +Define local session layout, run-local stage layout, and cleanup guardrails. -## Inputs and Outputs -Inputs: -- `pipeline.workspace.root` -- `session.campaign` -- `session.session_id` -- generated `run_id` +## Canonical Session Layout +Session root: +- `{workspace.root}/work/{campaign}/{session_id}` -Outputs: -- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json` -- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json` -- Canonical durable session directories and run-local stage trees +Core directories/files: +- `inputs/` +- `audio/` +- `transcripts/` +- `artifacts/` +- `reports/` +- `logs/` +- `config/` +- `current/` +- `runs/` +- `previous/` +- `manifest.json` +- `.lock` -## Boundaries -Owns: -- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`, `previous/`) - - `previous/manifest.json` and `previous/artifacts/**` are reserved for previous-session cache state materialized by `prepare` or `restore` -- Run-local stage sandbox layout under `runs/{run_id}/{stage}/` -- Session lock acquisition/release (`.lock`) +`previous/` reserved files: +- `previous/manifest.json` +- `previous/artifacts/**` -Does not own: -- Stage business logic -- Remote publish semantics (documented in `stage-publish.md`) -- CLI argument parsing +## Run-Local Stage Layout +When run context is available, stages use: +- `runs/{run_id}/{stage}/outputs/` +- `runs/{run_id}/{stage}/logs/` +- `runs/{run_id}/{stage}/reports/` +- `runs/{run_id}/{stage}/config/` +- `runs/{run_id}/{stage}/scratch/` -## Config Fields Used -- `pipeline.workspace.root` -- `pipeline.workspace.cleanup_after_publish` -- `pipeline.spool.root` -- `pipeline.spool.delete_audio_after_publish` -- `pipeline.cache.root` -- `pipeline.cache.s3_audio` -- `session.campaign` -- `session.session_id` +Run-local outputs are materialized back into canonical session paths before stage success. +`previous/**` writes are never redirected to run-local output paths. -## External Adapters Used -None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout. +## Locking +`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention). -## State and Manifest Behavior -- Session state is persisted in the session manifest (`manifest.Manifest`). -- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`. -- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and then materialized to canonical session paths after stage success. -- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs. -- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object. -- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current publish state rather than treating current-session stored `previous/**` as authoritative. -- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`. -- `narratio clean ` removes the session work root and session spool root. -- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`. -- `narratio clean --clear-cache` is the explicit opt-in for deleting matching S3 audio cache entries. +## Cleanup Semantics +Automatic post-publish cleanup (`runPostArchiveCleanup`): +- only runs when publish actually executed and succeeded; +- requires `uploaded=true` and `current_pointer_written=true` metadata; +- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`; +- refuses unsafe deletes (root delete, out-of-root delete, symlink paths). -## Skip and Resume Behavior -- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest. -- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`. -- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs. +Manual clean command: +- `clean ` removes session work and spool subtree. +- `clean --all` removes all workspace work and spool children. +- durable cache is preserved unless `--clear-cache` is requested. -## Failure Behavior -- Failures preserve manifests and run-local files for inspection. -- Lock conflicts fail fast via `ErrLockConflict`. -- Cleanup can fail post-publish; failure is recorded in publish stage metadata and returned by the run. - -## Tests to Inspect Before Changing -- `internal/artifacts/local_test.go` -- `internal/stage/run_local_test.go` -- `internal/app/run_control_test.go` -- `internal/app/resume_run_stage_test.go` -- `internal/app/post_archive_cleanup_test.go` - -## Architectural Invariants -- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`. -- Run roots are always nested: `runs/{run_id}` under the session root. -- Run-local output materialization must end in canonical session paths. -- `previous/**` is session-durable state and must not be treated as run-local output scratch state. -- Automatic post-publish cleanup only targets run-scoped directories and must never delete configured root directories. -- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets. +## Invariants +- campaign-aware session root is mandatory. +- manifest-driven stage state is durable across runs. +- cleanup guardrails prevent destructive root/out-of-scope deletion.