Align internal documentation and maintained examples

This commit is contained in:
2026-08-09 21:50:21 +00:00
parent bd2d5e2496
commit e7319ea016
26 changed files with 416 additions and 140 deletions

View File

@@ -25,8 +25,8 @@ This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete
safe remedies.
- [Integration contracts](docs/integrations/) — external tools, formats, and
compatibility expectations.
- [Maintained examples](examples/) — complete copyable configuration and input
files.
- [Maintained examples](examples/README.md) — complete copyable configuration
and input files.
## Maintainer Documentation

View File

@@ -274,10 +274,6 @@ Audio rules:
## Maintained Examples
- `examples/pipeline.minimal.yml`
- `examples/pipeline.production.yml`
- `examples/pipeline.full.annotated.yml`
- `examples/campaigns/sample-campaign/campaign.yml`
- `examples/session.local-audio.yml`
- `examples/session.s3-audio.yml`
- `examples/session.template.yml`
See the [maintained examples index](../examples/README.md) for complete pipeline,
campaign, session, template, and input fixtures. Keep complete copyable files
there rather than duplicating them in this reference.

View File

@@ -24,7 +24,7 @@ polished transcripts and generated artifacts. Start with the
| Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. |
| Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. |
| Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. |
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/) | The reference owns field meanings; the examples directory owns complete copyable files. |
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/README.md) | The reference owns field meanings; the examples directory owns complete copyable files. |
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
For an existing subsystem, also inspect its focused tests and package-level

View File

@@ -54,5 +54,6 @@ Operator-selected values are defined under `pipeline.audita.*` in the
[configuration reference](../config.md#pipeline).
Maintained example with Audita config:
- `examples/pipeline.full.annotated.yml`
- `examples/pipeline.production.yml`
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
- [Production-shaped pipeline](../../examples/pipeline.production.yml)

View File

@@ -63,5 +63,6 @@ per-artifact settings under `pipeline.scriptorium.artifacts.*`, in the
[configuration reference](../config.md#pipeline).
Maintained examples with Scriptorium config:
- `examples/pipeline.full.annotated.yml`
- `examples/pipeline.production.yml`
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
- [Production-shaped pipeline](../../examples/pipeline.production.yml)

View File

@@ -55,5 +55,6 @@ Operator-selected values are defined under `pipeline.seriatim.*` and
[configuration reference](../config.md#pipeline).
Maintained examples with Seriatim config:
- `examples/pipeline.full.annotated.yml`
- `examples/pipeline.production.yml`
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
- [Production-shaped pipeline](../../examples/pipeline.production.yml)

View File

@@ -1,12 +1,17 @@
# Internal: Adapters
## Purpose
Define external integration boundaries and default adapter wiring used by app/stage orchestration.
Explain the adapter interfaces and production composition used by application
and stage orchestration. Externally observable protocols and formats belong in
the [integration contracts](../integrations/).
## Adapter Boundaries
Narratio stage logic depends on adapter interfaces, not transport-specific details.
Primary adapters:
- `whisperx.Client`
- `seriatim.Runner`
- `audita.Runner`
@@ -15,17 +20,22 @@ Primary adapters:
- `notify.Sender`
## Ownership
Adapters own:
- HTTP/subprocess/SDK argument and transport details.
- Backend-specific request/response mapping.
Adapters do not own:
- stage ordering/skip/force logic;
- manifest transitions;
- canonical path policy.
## Default Wiring
`internal/app/runner.go` initializes default adapters when not injected:
- WhisperX HTTP client from pipeline config.
- Seriatim subprocess runner.
- Audita subprocess runner.
@@ -33,17 +43,29 @@ Adapters do not own:
- Noop notifier (`notify.NoopSender`).
- Object store only when required by selected stages/config.
Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization.
Object-store construction goes through `newCommandObjectStore`, which loads
configured filesystem secrets before adapter initialization.
## Failure Semantics
- Constructor errors fail stage execution setup early.
- Runtime adapter errors propagate to stage code and then manifest failure handling.
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
## Test Surfaces
## Implementation And Tests
- Composition: `internal/app/runner.go`, `internal/app/object_store.go`
- Shared subprocess mechanics: `internal/adapters/subprocess`
- Focused adapters: `internal/adapters/{whisperx,seriatim,audita,scriptorium,storage,notify}`
- `internal/adapters/whisperx/http_test.go`
- `internal/adapters/seriatim/subprocess_test.go`
- `internal/adapters/audita/subprocess_test.go`
- `internal/adapters/scriptorium/subprocess_test.go`
- `internal/adapters/storage/*_test.go`
- `internal/app/runner_test.go`
See the [WhisperX](../integrations/whisperx.md),
[Seriatim](../integrations/seriatim.md), [Audita](../integrations/audita.md),
and [Scriptorium](../integrations/scriptorium.md) contracts before changing an
externally visible boundary. Operator-selected values belong in
[Configuration](../config.md).

View File

@@ -1,17 +1,28 @@
# Internal: Artifacts
## Purpose
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
Explain the artifact registry, runtime catalog, resolver, previous-input
requirements, and shared remote current-state mechanics implemented by
`internal/artifacts`. Configuration fields that accept source IDs belong in
[Configuration](../config.md); physical placement belongs in
[Operations](../operations.md).
## Built-in Source IDs
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`)
- `narratio.transcript.final_markdown` -> `transcripts/final.md` (`render`)
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` (`render`)
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
The internal registry recognizes these stable built-in source IDs:
- `narratio.transcript.base`
- `narratio.transcript.polished`
- `narratio.transcript.final`
- `narratio.transcript.final_trimmed`
- `narratio.transcript.final_markdown`
- `narratio.transcript.final_trimmed_markdown`
- `narratio.bounds.session`
Registry entries bind each ID to its producer, output kind, canonical fallback,
and content validator. The focused stage documents own their input/output flow;
[Configuration](../config.md) owns where operators may select these IDs.
## Configured and Previous-Session Sources
@@ -72,7 +83,8 @@ Validation by content type:
## Current-State Helpers
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
Artifacts package owns shared remote current-state loading mechanics used by
restore, status and validation checks, and previous-cache planning.
Core helpers:
@@ -107,9 +119,27 @@ Caller policy is intentionally outside artifacts helpers:
- spool/cache paths;
- S3 session/run/current-state key layout.
See [Workspace Internals](workspace.md) for how callers consume local helpers
and [Operations](../operations.md#local-state-layout) for the authoritative
physical layout.
## Invariants
- source ID formats are stable contracts;
- artifact resolution is deterministic and manifest-aware;
- previous-session source resolution in `analyze` is local-only;
- remote current-state key construction remains centralized in artifacts helpers.
## Implementation And Tests
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`
- Current state: `internal/artifacts/current_state.go`
- Paths and keys: `internal/artifacts/paths.go`,
`internal/artifacts/s3_keys.go`
- Previous requirements: `internal/artifacts/previous_requirements.go`
- Tests: `internal/artifacts/artifact_resolver_test.go`,
`internal/artifacts/catalog_test.go`,
`internal/artifacts/current_state_test.go`,
`internal/artifacts/paths_model_test.go`,
`internal/artifacts/previous_requirements_test.go`

View File

@@ -1,21 +1,20 @@
# Internal: Command Restore
## Purpose
Define the implemented `narratio session restore` command contract:
- committed remote current-state discovery;
- deterministic restore planning;
- safe local install semantics;
- durable restore reporting.
Explain the implemented restore discovery, planning, installation, and
reporting flow in `internal/app`. User invocation belongs in
[CLI](../cli.md#session-restore), and the operator recovery procedure and
physical restore scope belong in
[Operations](../operations.md#restore-workflow).
Restore is split into explicit phases so remote authority, local conflict
policy, and filesystem mutation can be tested independently.
## Discovery Contract
Restore resolves remote committed state from the session publish current pointers:
- `current/run_id.txt` (required, non-empty);
- `current/manifest.json` (required, decodable).
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
Discovery delegates current-state pointer and manifest loading to
`internal/artifacts`, then validates the result against the resolved request:
- campaign must match;
- session ID must match.
@@ -34,26 +33,11 @@ Planner behavior:
- remote list scope is the resolved session prefix;
- remote-to-local mapping is traversal-safe;
- actions are sorted deterministically by local relative path.
- actions are sorted by local relative path and then remote key;
- force converts differing local targets from conflicts to downloads.
Restore scope from current remote state:
- include `manifest.json`;
- include `transcripts/**`;
- include `artifacts/**`;
- include `audio/**` only with `--include-audio`.
Explicit exclusions from current remote state mapping:
- `current/**`;
- `runs/**`;
- `logs/**`;
- `reports/**`;
- `config/**`;
- `inputs/**`;
- `previous/**`.
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
Previous-cache files are planned separately through `previouscache.BuildPlan`
when configured previous-session requirements exist.
## Execution Contract
@@ -73,8 +57,9 @@ Audio restore path:
## Reporting Contract
- `--dry-run`: prints summary only; no local writes.
- non-dry-run: writes `reports/restore-latest.json`.
- dry-run mode prints a summary and performs no local writes;
- execution mode persists the canonical restore report described in
[Operations](../operations.md#restore-workflow);
- report includes plan counts, per-action status, and execution failures.
## Invariants
@@ -82,3 +67,15 @@ Audio restore path:
- restore uses committed remote current state as authority;
- `current/run_id.txt` is the remote publish commit marker;
- restore does not execute pipeline stages.
## Implementation And Tests
- Discovery: `internal/app/restore_discovery.go`
- Planning: `internal/app/restore_plan.go`, `internal/previouscache`
- Execution: `internal/app/restore_execute.go`
- Reporting and command coordination: `internal/app/restore_report.go`,
`internal/app/restore.go`
- Tests: `internal/app/restore_discovery_test.go`,
`internal/app/restore_plan_test.go`,
`internal/app/restore_execution_test.go`,
`internal/app/restore_workflow_test.go`

View File

@@ -1,13 +1,15 @@
# Internal: Manifest
## Purpose
Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts.
Explain the session-progress and invocation-audit models implemented by
`internal/manifest`. Physical manifest placement belongs in
[Operations](../operations.md#local-state-layout).
## Session Manifest
Path:
- `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
Primary model (`manifest.Manifest`):
`manifest.Manifest` records:
- identity (`session_id`, `campaign`, `run_id`)
- local path metadata (`local_workdir`, `local_spool_dir`)
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
@@ -15,7 +17,8 @@ Primary model (`manifest.Manifest`):
- durable `artifacts` records
- per-stage `stages` map
Stage status enum:
The model admits these stage states:
- `pending`
- `running`
- `succeeded`
@@ -25,10 +28,9 @@ Stage status enum:
- `interrupted`
## Run Manifest
Path:
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
Run model (`manifest.RunManifest`):
`manifest.RunManifest` is created for each invocation and records:
- invocation identity and `force` flag
- requested stages
- per-stage action (`run` or `skip`)
@@ -36,22 +38,40 @@ Run model (`manifest.RunManifest`):
- overall run status (`running`, `succeeded`, `failed`)
## Persistence Semantics
`manifest.LocalStore`:
- validates loaded documents;
- normalizes missing maps/stage records;
- writes atomically via temp file + rename;
- updates `updated_at` on save.
## Execution Semantics
Runner updates both manifests per stage transition:
- mark running
- mark succeeded/failed/skipped
- persist logs/generated config refs and metadata
The application runner marks an executing stage running and then succeeded or
failed in both manifests, persisting each transition. On success it records
outputs, logs, generated configuration references, and metadata. A successful
forced rerun marks only succeeded downstream session-stage records stale.
When an already-succeeded stage is skipped, the invocation run manifest records
the `skip` action and reason. The session manifest deliberately retains its
existing succeeded record because it remains the cross-invocation progress
authority.
Session manifest is the authoritative stage-progress ledger across invocations.
Run manifest is invocation-scoped audit state.
## Invariants
- stage resume/skip decisions are session-manifest driven.
- force reruns stale downstream succeeded stages.
- run manifest does not replace session manifest as progress authority.
## Implementation And Tests
- Models and transitions: `internal/manifest/manifest.go`,
`internal/manifest/run_manifest.go`
- Persistence and validation: `internal/manifest/store.go`
- Package tests: `internal/manifest/*_test.go`
- Assembled execution behavior: `internal/app/runner_test.go`,
`internal/app/run_stage_test.go`

View File

@@ -59,7 +59,8 @@ The implemented canonical order is:
10. `notify` (placeholder)
`notify` currently has optional notifier call behavior and no persisted pipeline
outputs. The focused stage documents own implementation mechanics. The
outputs; its default collaborator is a no-op sender. The focused stage
documents own implementation mechanics. The
[CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation
and execution semantics.
@@ -88,3 +89,8 @@ and execution semantics.
Use this map to find an owner, then read its focused documentation and tests
before changing behavior.
The stage registry is implemented in `internal/stage/placeholders.go` and its
ordering is protected by `internal/app/planner_test.go`. Cross-invocation skip,
force, failure, and invalidation behavior is exercised in
`internal/app/runner_test.go` and `internal/app/run_stage_test.go`.

View File

@@ -1,24 +1,29 @@
# Stage: analyze
## Purpose
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
## Inputs
- configured artifacts from `pipeline.scriptorium.artifacts`
- optional selected artifact filter (`--artifacts`)
- optional selected artifact keys supplied through the stage environment
- built-in/configured/previous-session source references in artifact inputs
Supported source families:
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`, `narratio.input.glossary`
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`,
`narratio.input.glossary`
- configured artifacts: `narratio.artifact.<key>`
- previous-session cache: `narratio.previous_session.artifact.<key>`
## Outputs
- one materialized output per executed configured artifact (`output_path`)
- stage metadata describing selected/generated/reused artifacts
## Key Behavior
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
- builds runtime artifact catalog (built-ins + configured artifacts).
- marks non-executable configured artifacts as reusable when output files already exist.
@@ -30,13 +35,26 @@ Supported source families:
- validates non-empty output files and materializes canonical outputs.
## Failure Semantics
- required missing configured/previous-session inputs fail.
- missing required prepared stable input source includes prepare rerun guidance.
- missing required previous-session source includes prepare rerun guidance.
- missing required `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` inputs includes render rerun guidance.
- missing required `narratio.transcript.final_markdown` or
`narratio.transcript.final_trimmed_markdown` inputs includes render rerun
guidance.
- dependency cycles or unavailable required dependencies fail.
- adapter validation failures fail stage.
## Invariants
- `analyze` performs no remote storage calls for previous-session source resolution.
- output provenance and metadata are deterministic per execution.
## Related Contracts And Tests
- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact
fields and source-selection rules.
- [CLI](../cli.md) owns user-visible artifact selection.
- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract.
- Implementation and tests: `internal/stage/analyze.go`,
`internal/stage/analyze_test.go`

View File

@@ -1,18 +1,22 @@
# Stage: merge
## Purpose
Normalize raw transcript inputs and merge into base transcript via Seriatim.
## Inputs
- `transcripts/raw/*.json`
- `inputs/speakers.yml`
- `inputs/autocorrect.yml`
## Outputs
- `transcripts/base.json`
- optional `artifacts/seriatim.report.json`
## Key Behavior
- discovers and validates raw transcript inputs.
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
- merges normalized inputs (`seriatim.Run`) into base transcript.
@@ -20,6 +24,14 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim.
- materializes canonical outputs and records stage logs/generated configs.
## Invariants
- merge always consumes normalized forms of raw inputs.
- base transcript must validate before stage success.
- report output is config-gated.
## Related Contracts And Tests
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
- [Configuration](../config.md#pipeline) owns operator-selected Seriatim values.
- Implementation and tests: `internal/stage/merge.go`,
`internal/stage/merge_test.go`

View File

@@ -1,16 +1,20 @@
# Stage: normalize
## Purpose
Normalize polished transcript into final transcript using Seriatim.
## Inputs
- `transcripts/polished.json`
## Outputs
- `transcripts/final.json` (or configured normalize output path)
- optional `artifacts/seriatim.normalize.report.json`
## Key Behavior
- resolves polished transcript from manifest outputs/canonical fallback.
- applies `pipeline.normalize` config or default normalize config.
- runs Seriatim normalize with configured timeout/binary.
@@ -18,5 +22,13 @@ Normalize polished transcript into final transcript using Seriatim.
- materializes canonical outputs and records logs/generated configs.
## Invariants
- final transcript must validate as processed transcript JSON (`segments` array).
- normalize defaults are applied when `pipeline.normalize` is unset.
## Related Contracts And Tests
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
- [Configuration](../config.md#pipeline) owns normalize fields and defaults.
- Implementation and tests: `internal/stage/normalize.go`,
`internal/stage/normalize_test.go`

View File

@@ -1,17 +1,21 @@
# Stage: polish
## Purpose
Run Audita polishing on base transcript and produce polished transcript.
## Inputs
- `transcripts/base.json`
- `inputs/glossary.yml`
## Outputs
- `transcripts/polished.json`
- optional `artifacts/audita.report.json`
## Key Behavior
- resolves base transcript from merge outputs/canonical fallback.
- invokes Audita with configured model/module/runtime options.
- validates processed transcript structure (`segments` array required).
@@ -19,5 +23,14 @@ Run Audita polishing on base transcript and produce polished transcript.
- materializes canonical outputs; records logs/generated config and adapter metadata.
## Invariants
- polished transcript schema validation is mandatory.
- report output is config-gated.
## Related Contracts And Tests
- [Audita](../integrations/audita.md) owns subprocess, validation, and failure
semantics.
- [Configuration](../config.md#pipeline) owns operator-selected Audita values.
- Implementation and tests: `internal/stage/polish.go`,
`internal/stage/polish_test.go`

View File

@@ -1,17 +1,18 @@
# Stage: prepare
## Purpose
Materialize canonical current-session inputs before processing stages.
## Inputs
- resolved `campaign.yml`, `session.yml`, and pipeline config
- resolved campaign, session, and pipeline configuration
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
- audio source:
- local `audio_dir`/`audio_files`, or
- S3 `audio_s3.prefix`
- one resolved local or S3 audio source
- enabled configured artifact input requirements for previous-session sources
## Outputs
- `inputs/campaign.yml`
- `inputs/session.yml`
- `inputs/pipeline.resolved.yml`
@@ -26,6 +27,7 @@ Materialize canonical current-session inputs before processing stages.
- deterministic `manifest.inputs` entries (checksums + provenance)
## Key Behavior
- validates required config/store state.
- enforces local audio vs S3 audio mutual exclusivity.
- materializes S3 audio through spool/cache-aware logic.
@@ -39,6 +41,20 @@ Materialize canonical current-session inputs before processing stages.
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
## Invariants
- only `prepare` hydrates canonical `previous/` cache state.
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
- managed previous artifacts are stored under `previous/artifacts/**` without
duplicate `artifacts/artifacts/` nesting.
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
## Related Contracts And Tests
- [Configuration](../config.md) owns audio selection, stable input fields, and
previous-session settings.
- [Operations](../operations.md) owns physical input, audio, spool, cache, and
previous-state layout.
- [Storage Internals](storage.md) and [Artifact Internals](artifacts.md) explain
the internal collaborators.
- Implementation and tests: `internal/stage/prepare.go`,
`internal/stage/prepare_test.go`, `internal/audio/s3_audio_test.go`,
`internal/previouscache/*_test.go`

View File

@@ -1,23 +1,29 @@
# Stage: publish
## Purpose
Upload run/session outputs to object storage and atomically advance remote current state.
## Inputs
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`
- run root `runs/{run_id}/**`
- publish output rules (`pipeline.publish.outputs`)
- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set)
- invocation-scoped run files
- resolved publish output rules
- effective publish locks (static + remote merged lock set)
- local `previous/**` files when present
- durable previous-session cache files when present
## Outputs
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
- uploaded selected publish outputs under session prefix
- uploaded `previous/**` files under session prefix when present
- uploaded `current/manifest.json`
- uploaded `current/run_id.txt` written last
- uploaded invocation record and selected publish outputs;
- uploaded durable previous-session cache files when present;
- updated remote current manifest; and
- remote current-run commit marker, written last.
Exact remote placement and the operator workflow belong in
[Operations](../operations.md#publish-workflow).
## Key Behavior
- stage can self-skip when publish disabled or run upload disabled.
- validates prerequisite stage success and object-store availability.
- collects deterministic run file list plus run `manifest.json`.
@@ -28,6 +34,7 @@ Upload run/session outputs to object storage and atomically advance remote curre
- writes remote current manifest before current run pointer.
## Metadata Signals
Includes counts/lists for:
- run uploads
- published output uploads
@@ -39,6 +46,22 @@ Includes counts/lists for:
- `current_pointer_written`
## Invariants
- `current/run_id.txt` is the remote commit marker and is written last.
- run upload excludes `audio/**`.
- publish locks are not overridden by `--force`.
The commit boundary and cleanup gate are normative architecture invariants; see
[Architecture](../policy/architecture.md#publish-commit-boundary).
## Related Contracts And Tests
- [Configuration](../config.md#publish-configuration-summary) owns output and
static-lock fields.
- [Operations](../operations.md#publish-locks) owns remote lock lifecycle and
physical remote state.
- [Artifact Internals](artifacts.md) explains source resolution and current-state
helpers.
- Implementation and tests: `internal/stage/publish.go`,
`internal/stage/publish_test.go`, `internal/app/operator_helpers_test.go`,
`internal/app/post_publish_cleanup_test.go`

View File

@@ -1,17 +1,21 @@
# Stage: render
## Purpose
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
## Inputs
- `narratio.transcript.final` (`transcripts/final.json`)
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
## Outputs
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
## Key Behavior
- uses `pipeline.render` settings (enabled/format/title/booleans).
- resolves inputs manifest-first, then canonical fallback.
- writes run-local outputs first, then materializes canonical session outputs.
@@ -19,11 +23,20 @@ Render Markdown transcript artifacts from normalized JSON transcripts via Seriat
- skips with stage metadata when `pipeline.render.enabled=false`.
## Failure Semantics
- missing normalized input fails with normalize rerun guidance.
- missing trimmed input fails with trim rerun guidance.
- adapter/subprocess failure fails stage.
- empty render output files fail validation.
## Invariants
- only `format: markdown` is supported.
- render stage owns production of built-in Markdown transcript sources.
## Related Contracts And Tests
- [Seriatim](../integrations/seriatim.md) owns render subprocess behavior.
- [Configuration](../config.md#pipeline) owns render fields and defaults.
- Implementation and tests: `internal/stage/render.go`,
`internal/stage/render_test.go`

View File

@@ -1,22 +1,36 @@
# Stage: transcribe
## Purpose
Generate raw per-speaker transcripts from prepared audio using WhisperX.
## Inputs
- `audio/*.flac` from `prepare`
## Outputs
- `transcripts/raw/<speaker>.json`
## Key Behavior
- discovers prepared audio from manifest inputs or canonical audio directory.
- derives speaker ID from `.flac` basename.
- runs WhisperX with configured concurrency/retry settings.
- dispatches WhisperX requests through a bounded worker pool.
- validates each output as JSON.
- writes run-local outputs then materializes canonical transcript outputs.
## Invariants
- speaker basenames must be unique.
- output path returned by adapter must match requested output path.
- each successful output is validated before stage success.
## Related Contracts And Tests
- [WhisperX](../integrations/whisperx.md) owns HTTP, retry, timeout, and
cancellation semantics.
- [Configuration](../config.md#pipeline) owns concurrency and other
operator-selected values.
- Implementation and tests: `internal/stage/transcribe.go`,
`internal/stage/transcribe_test.go`

View File

@@ -1,16 +1,21 @@
# Stage: trim
## Purpose
Produce a final-trimmed transcript. By default, the stage generates bounds and applies a bounds-driven trim.
Produce a final-trimmed transcript. By default, the stage generates bounds and
applies a bounds-driven trim.
## Inputs
- `transcripts/final.json`
## Outputs
- `transcripts/final.trimmed.json` (or configured trim output path)
- when trim enabled: `artifacts/session_bounds.json`
## Key Behavior
When `trim.enabled=true`:
- runs Scriptorium bounds artifact generation;
- optionally runs render-debug output generation;
@@ -23,6 +28,16 @@ When `trim.enabled=false`:
- copies normalized transcript to trimmed output.
## Invariants
- normalized transcript is required input.
- bounds output exists only in enabled trim path.
- render-debug output is diagnostic and not a declared stage output.
## Related Contracts And Tests
- [Scriptorium](../integrations/scriptorium.md) owns bounds generation and
debug-render subprocess behavior.
- [Seriatim](../integrations/seriatim.md) owns transcript trimming behavior.
- [Configuration](../config.md#pipeline) owns trim fields and defaults.
- Implementation and tests: `internal/stage/trim.go`,
`internal/stage/trim_test.go`

View File

@@ -1,10 +1,16 @@
# Internal: Storage
## Purpose
Document remote object-store contracts and S3 implementation behavior.
Explain the object-store interface and S3 implementation used by Narratio.
Remote key layout and lifecycle belong in [Operations](../operations.md), while
operator-selected storage fields and credential mechanisms belong in
[Configuration](../config.md).
## Primary Contract
`storage.ObjectStore` interface:
- `List(ctx, prefix)`
- `Download(ctx, key, localPath)`
- `Upload(ctx, localPath, key, opts)`
@@ -14,15 +20,15 @@ Key invariant:
- callers pass full bucket-relative keys;
- storage implementations do not infer campaign/session/run prefixes.
## Configuration
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
## Composition
S3 constructor behavior:
- requires configured bucket;
- uses region/endpoint/path-style options when set;
- resolves credentials from configured env var names (with defaults).
`NewObjectStoreFromConfig` constructs the S3-backed implementation from
resolved configuration. The application loads configured filesystem secrets
before calling it. The storage adapter consumes already-resolved values; it does
not own discovery, defaults, or configuration validation.
## S3 Backend Behavior
- normalizes object keys.
- `List` paginates and returns normalized `ObjectInfo`.
- `Download` writes local files with parent directory creation.
@@ -30,5 +36,13 @@ S3 constructor behavior:
- `Exists` maps not-found responses to `false`.
## Invariants
- storage layer is stateless regarding manifest/stage progression.
- publish ordering semantics are owned by stage/app code, not storage adapters.
## Implementation And Tests
- Contract and S3 adapter: `internal/adapters/storage`
- Composition: `internal/app/object_store.go`
- Tests: `internal/adapters/storage/*_test.go`,
`internal/app/object_store_test.go`

View File

@@ -1,57 +1,62 @@
# Internal: Workspace
## Purpose
Define local session layout, run-local stage layout, and cleanup guardrails.
## Canonical Session Layout
Session root:
- `{workspace.root}/work/{campaign}/{session_id}`
Explain the helpers that construct local session and run paths, coordinate
single-writer access, and confine cleanup. The authoritative physical layout and
retention workflow belong in [Operations](../operations.md#local-state-layout).
Core directories/files:
- `inputs/`
- `audio/`
- `transcripts/`
- `artifacts/`
- `reports/`
- `logs/`
- `config/`
- `current/`
- `runs/`
- `previous/`
- `manifest.json`
- `.lock`
## Path Ownership
`previous/` reserved files:
- `previous/manifest.json`
- `previous/artifacts/**`
`internal/artifacts` owns canonical session, run, spool, cache, and
previous-cache path construction. `SessionPathsFor` provides the session-scoped
path model, and layout creation goes through `EnsureLayoutFor`. Callers should
consume those helpers instead of rebuilding relative paths.
`internal/pathsafe` and application cleanup helpers enforce confinement for
relative destinations and deletion targets.
## Run-Local Stage Layout
When run context is available, stages use:
- `runs/{run_id}/{stage}/outputs/`
- `runs/{run_id}/{stage}/logs/`
- `runs/{run_id}/{stage}/reports/`
- `runs/{run_id}/{stage}/config/`
- `runs/{run_id}/{stage}/scratch/`
Run-local outputs are materialized back into canonical session paths before stage success.
`previous/**` writes are never redirected to run-local output paths.
`internal/stage/run_local.go` maps stage outputs and diagnostics into an
invocation-scoped layout. Successful outputs are validated and atomically
materialized into canonical session paths before stage success. Managed
previous-session cache paths remain session-durable and are never redirected
into run-local output space.
## Locking
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
`artifacts.LocalStore` enforces the single-writer session lock via `.lock`
(`ErrLockConflict` on contention).
## Cleanup Semantics
Automatic post-publish cleanup:
- only runs when publish actually executed and succeeded;
- requires `uploaded=true` and `current_pointer_written=true` metadata;
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
- consumes the resolved cleanup policy described in
[Configuration](../config.md);
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
Manual clean command:
- `clean <session_id>` removes session work and spool subtree.
- `clean --all` removes all workspace work and spool children.
- durable cache is preserved unless `--clear-cache` is requested.
Manual cleanup uses the same scoped-target checks. Invocation syntax and exact
deletion scope belong in [CLI](../cli.md#clean) and
[Operations](../operations.md#cleanup).
## Invariants
- campaign-aware session root is mandatory.
- manifest-driven stage state is durable across runs.
- cleanup guardrails prevent destructive root/out-of-scope deletion.
## Implementation And Tests
- Path model and local store: `internal/artifacts/paths.go`,
`internal/artifacts/local.go`
- Run-local materialization: `internal/stage/run_local.go`
- Cleanup confinement: `internal/app/cleanup_targets.go`,
`internal/app/post_publish_cleanup.go`
- Tests: `internal/artifacts/paths_model_test.go`,
`internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`,
`internal/app/cleanup_targets_test.go`,
`internal/app/post_publish_cleanup_test.go`

View File

@@ -66,7 +66,8 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa
Safe fix:
- align config with [docs/config.md](./config.md) and maintained files under `examples/`.
- align config with [Configuration](./config.md) and the
[maintained examples](../examples/README.md).
Relevant reference: [Configuration](./config.md).

46
examples/README.md Normal file
View File

@@ -0,0 +1,46 @@
# Maintained Examples
These files are safe, copyable starting points for Narratio configuration and
input structure. Replace placeholder identifiers, storage names, integration
URLs, and paths for the target environment. Field meanings and defaults belong
in the [configuration reference](../docs/config.md).
## Pipeline Configuration
- [Minimal pipeline](pipeline.minimal.yml): campaign discovery plus the required
WhisperX URL.
- [Production-shaped pipeline](pipeline.production.yml): S3 storage, publish,
external tools, and configured Scriptorium artifacts.
- [Full annotated pipeline](pipeline.full.annotated.yml): every implemented
pipeline section with explanatory comments.
The existing `internal/config` example test loads and validates each pipeline
with the sample campaign and a compatible local- or S3-audio session.
## Campaign And Session Configuration
- [Sample campaign](campaigns/sample-campaign/campaign.yml), its
[session template](campaigns/sample-campaign/session.template.yml), and its
adjacent stable inputs provide a complete campaign directory shape.
- [Local-audio session](session.local-audio.yml) and
[S3-audio session](session.s3-audio.yml) are concrete session files.
- [Session template](session.template.yml) and the campaign-local equivalent
demonstrate the narrow placeholder syntax consumed by `session init`; they
are templates, not runtime session files.
## Input Fixtures
- [Speakers](speakers.yml), [autocorrect](autocorrect.yml), and
[glossary](glossary.yml) show the standalone input shapes.
- The sample campaign references its local
[speakers](campaigns/sample-campaign/speakers.yml),
[autocorrect](campaigns/sample-campaign/autocorrect.yml),
[glossary](campaigns/sample-campaign/glossary.yml),
[players](campaigns/sample-campaign/players.yml), and
[party](campaigns/sample-campaign/party.yml) fixtures.
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
reserves the expected filename and directory shape. Replace it with a real
FLAC file before running transcription.
The examples contain environment-variable names but no credential values. They
use fictional campaign content and reserved example domains.

View File

@@ -1,5 +1,5 @@
match:
- speaker: "Eric Rakestraw"
- speaker: "Example Speaker"
match:
- "Eric_Rakestraw"
- "Eric"
- "Example_Speaker"
- "Example"

View File

@@ -1,5 +1,5 @@
match:
- speaker: "Eric Rakestraw"
- speaker: "Example Speaker"
match:
- "Eric_Rakestraw"
- "Eric"
- "Example_Speaker"
- "Example"