Align internal documentation and maintained examples
This commit is contained in:
@@ -25,8 +25,8 @@ This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete
|
|||||||
safe remedies.
|
safe remedies.
|
||||||
- [Integration contracts](docs/integrations/) — external tools, formats, and
|
- [Integration contracts](docs/integrations/) — external tools, formats, and
|
||||||
compatibility expectations.
|
compatibility expectations.
|
||||||
- [Maintained examples](examples/) — complete copyable configuration and input
|
- [Maintained examples](examples/README.md) — complete copyable configuration
|
||||||
files.
|
and input files.
|
||||||
|
|
||||||
## Maintainer Documentation
|
## Maintainer Documentation
|
||||||
|
|
||||||
|
|||||||
@@ -274,10 +274,6 @@ Audio rules:
|
|||||||
|
|
||||||
## Maintained Examples
|
## Maintained Examples
|
||||||
|
|
||||||
- `examples/pipeline.minimal.yml`
|
See the [maintained examples index](../examples/README.md) for complete pipeline,
|
||||||
- `examples/pipeline.production.yml`
|
campaign, session, template, and input fixtures. Keep complete copyable files
|
||||||
- `examples/pipeline.full.annotated.yml`
|
there rather than duplicating them in this reference.
|
||||||
- `examples/campaigns/sample-campaign/campaign.yml`
|
|
||||||
- `examples/session.local-audio.yml`
|
|
||||||
- `examples/session.s3-audio.yml`
|
|
||||||
- `examples/session.template.yml`
|
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ polished transcripts and generated artifacts. Start with the
|
|||||||
| Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. |
|
| Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. |
|
||||||
| Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. |
|
| Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. |
|
||||||
| Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. |
|
| Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. |
|
||||||
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/) | The reference owns field meanings; the examples directory owns complete copyable files. |
|
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/README.md) | The reference owns field meanings; the examples directory owns complete copyable files. |
|
||||||
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|
||||||
For an existing subsystem, also inspect its focused tests and package-level
|
For an existing subsystem, also inspect its focused tests and package-level
|
||||||
|
|||||||
@@ -54,5 +54,6 @@ Operator-selected values are defined under `pipeline.audita.*` in the
|
|||||||
[configuration reference](../config.md#pipeline).
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
Maintained example with Audita config:
|
Maintained example with Audita config:
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
@@ -63,5 +63,6 @@ per-artifact settings under `pipeline.scriptorium.artifacts.*`, in the
|
|||||||
[configuration reference](../config.md#pipeline).
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
Maintained examples with Scriptorium config:
|
Maintained examples with Scriptorium config:
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
@@ -55,5 +55,6 @@ Operator-selected values are defined under `pipeline.seriatim.*` and
|
|||||||
[configuration reference](../config.md#pipeline).
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
Maintained examples with Seriatim config:
|
Maintained examples with Seriatim config:
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
@@ -1,12 +1,17 @@
|
|||||||
# Internal: Adapters
|
# Internal: Adapters
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define external integration boundaries and default adapter wiring used by app/stage orchestration.
|
|
||||||
|
Explain the adapter interfaces and production composition used by application
|
||||||
|
and stage orchestration. Externally observable protocols and formats belong in
|
||||||
|
the [integration contracts](../integrations/).
|
||||||
|
|
||||||
## Adapter Boundaries
|
## Adapter Boundaries
|
||||||
|
|
||||||
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
||||||
|
|
||||||
Primary adapters:
|
Primary adapters:
|
||||||
|
|
||||||
- `whisperx.Client`
|
- `whisperx.Client`
|
||||||
- `seriatim.Runner`
|
- `seriatim.Runner`
|
||||||
- `audita.Runner`
|
- `audita.Runner`
|
||||||
@@ -15,17 +20,22 @@ Primary adapters:
|
|||||||
- `notify.Sender`
|
- `notify.Sender`
|
||||||
|
|
||||||
## Ownership
|
## Ownership
|
||||||
|
|
||||||
Adapters own:
|
Adapters own:
|
||||||
|
|
||||||
- HTTP/subprocess/SDK argument and transport details.
|
- HTTP/subprocess/SDK argument and transport details.
|
||||||
- Backend-specific request/response mapping.
|
- Backend-specific request/response mapping.
|
||||||
|
|
||||||
Adapters do not own:
|
Adapters do not own:
|
||||||
|
|
||||||
- stage ordering/skip/force logic;
|
- stage ordering/skip/force logic;
|
||||||
- manifest transitions;
|
- manifest transitions;
|
||||||
- canonical path policy.
|
- canonical path policy.
|
||||||
|
|
||||||
## Default Wiring
|
## Default Wiring
|
||||||
|
|
||||||
`internal/app/runner.go` initializes default adapters when not injected:
|
`internal/app/runner.go` initializes default adapters when not injected:
|
||||||
|
|
||||||
- WhisperX HTTP client from pipeline config.
|
- WhisperX HTTP client from pipeline config.
|
||||||
- Seriatim subprocess runner.
|
- Seriatim subprocess runner.
|
||||||
- Audita subprocess runner.
|
- Audita subprocess runner.
|
||||||
@@ -33,17 +43,29 @@ Adapters do not own:
|
|||||||
- Noop notifier (`notify.NoopSender`).
|
- Noop notifier (`notify.NoopSender`).
|
||||||
- Object store only when required by selected stages/config.
|
- Object store only when required by selected stages/config.
|
||||||
|
|
||||||
Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization.
|
Object-store construction goes through `newCommandObjectStore`, which loads
|
||||||
|
configured filesystem secrets before adapter initialization.
|
||||||
|
|
||||||
## Failure Semantics
|
## Failure Semantics
|
||||||
|
|
||||||
- Constructor errors fail stage execution setup early.
|
- Constructor errors fail stage execution setup early.
|
||||||
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
||||||
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
||||||
|
|
||||||
## Test Surfaces
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Composition: `internal/app/runner.go`, `internal/app/object_store.go`
|
||||||
|
- Shared subprocess mechanics: `internal/adapters/subprocess`
|
||||||
|
- Focused adapters: `internal/adapters/{whisperx,seriatim,audita,scriptorium,storage,notify}`
|
||||||
- `internal/adapters/whisperx/http_test.go`
|
- `internal/adapters/whisperx/http_test.go`
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
- `internal/adapters/seriatim/subprocess_test.go`
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
- `internal/adapters/audita/subprocess_test.go`
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||||
- `internal/adapters/storage/*_test.go`
|
- `internal/adapters/storage/*_test.go`
|
||||||
- `internal/app/runner_test.go`
|
- `internal/app/runner_test.go`
|
||||||
|
|
||||||
|
See the [WhisperX](../integrations/whisperx.md),
|
||||||
|
[Seriatim](../integrations/seriatim.md), [Audita](../integrations/audita.md),
|
||||||
|
and [Scriptorium](../integrations/scriptorium.md) contracts before changing an
|
||||||
|
externally visible boundary. Operator-selected values belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|||||||
@@ -1,17 +1,28 @@
|
|||||||
# Internal: Artifacts
|
# Internal: Artifacts
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
|
|
||||||
|
Explain the artifact registry, runtime catalog, resolver, previous-input
|
||||||
|
requirements, and shared remote current-state mechanics implemented by
|
||||||
|
`internal/artifacts`. Configuration fields that accept source IDs belong in
|
||||||
|
[Configuration](../config.md); physical placement belongs in
|
||||||
|
[Operations](../operations.md).
|
||||||
|
|
||||||
## Built-in Source IDs
|
## Built-in Source IDs
|
||||||
|
|
||||||
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
|
The internal registry recognizes these stable built-in source IDs:
|
||||||
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
|
|
||||||
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
|
- `narratio.transcript.base`
|
||||||
- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`)
|
- `narratio.transcript.polished`
|
||||||
- `narratio.transcript.final_markdown` -> `transcripts/final.md` (`render`)
|
- `narratio.transcript.final`
|
||||||
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` (`render`)
|
- `narratio.transcript.final_trimmed`
|
||||||
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
|
- `narratio.transcript.final_markdown`
|
||||||
|
- `narratio.transcript.final_trimmed_markdown`
|
||||||
|
- `narratio.bounds.session`
|
||||||
|
|
||||||
|
Registry entries bind each ID to its producer, output kind, canonical fallback,
|
||||||
|
and content validator. The focused stage documents own their input/output flow;
|
||||||
|
[Configuration](../config.md) owns where operators may select these IDs.
|
||||||
|
|
||||||
## Configured and Previous-Session Sources
|
## Configured and Previous-Session Sources
|
||||||
|
|
||||||
@@ -72,7 +83,8 @@ Validation by content type:
|
|||||||
|
|
||||||
## Current-State Helpers
|
## Current-State Helpers
|
||||||
|
|
||||||
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
|
Artifacts package owns shared remote current-state loading mechanics used by
|
||||||
|
restore, status and validation checks, and previous-cache planning.
|
||||||
|
|
||||||
Core helpers:
|
Core helpers:
|
||||||
|
|
||||||
@@ -107,9 +119,27 @@ Caller policy is intentionally outside artifacts helpers:
|
|||||||
- spool/cache paths;
|
- spool/cache paths;
|
||||||
- S3 session/run/current-state key layout.
|
- S3 session/run/current-state key layout.
|
||||||
|
|
||||||
|
See [Workspace Internals](workspace.md) for how callers consume local helpers
|
||||||
|
and [Operations](../operations.md#local-state-layout) for the authoritative
|
||||||
|
physical layout.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- source ID formats are stable contracts;
|
- source ID formats are stable contracts;
|
||||||
- artifact resolution is deterministic and manifest-aware;
|
- artifact resolution is deterministic and manifest-aware;
|
||||||
- previous-session source resolution in `analyze` is local-only;
|
- previous-session source resolution in `analyze` is local-only;
|
||||||
- remote current-state key construction remains centralized in artifacts helpers.
|
- remote current-state key construction remains centralized in artifacts helpers.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
|
||||||
|
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`
|
||||||
|
- Current state: `internal/artifacts/current_state.go`
|
||||||
|
- Paths and keys: `internal/artifacts/paths.go`,
|
||||||
|
`internal/artifacts/s3_keys.go`
|
||||||
|
- Previous requirements: `internal/artifacts/previous_requirements.go`
|
||||||
|
- Tests: `internal/artifacts/artifact_resolver_test.go`,
|
||||||
|
`internal/artifacts/catalog_test.go`,
|
||||||
|
`internal/artifacts/current_state_test.go`,
|
||||||
|
`internal/artifacts/paths_model_test.go`,
|
||||||
|
`internal/artifacts/previous_requirements_test.go`
|
||||||
|
|||||||
@@ -1,21 +1,20 @@
|
|||||||
# Internal: Command Restore
|
# Internal: Command Restore
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the implemented `narratio session restore` command contract:
|
|
||||||
|
|
||||||
- committed remote current-state discovery;
|
Explain the implemented restore discovery, planning, installation, and
|
||||||
- deterministic restore planning;
|
reporting flow in `internal/app`. User invocation belongs in
|
||||||
- safe local install semantics;
|
[CLI](../cli.md#session-restore), and the operator recovery procedure and
|
||||||
- durable restore reporting.
|
physical restore scope belong in
|
||||||
|
[Operations](../operations.md#restore-workflow).
|
||||||
|
|
||||||
|
Restore is split into explicit phases so remote authority, local conflict
|
||||||
|
policy, and filesystem mutation can be tested independently.
|
||||||
|
|
||||||
## Discovery Contract
|
## Discovery Contract
|
||||||
|
|
||||||
Restore resolves remote committed state from the session publish current pointers:
|
Discovery delegates current-state pointer and manifest loading to
|
||||||
|
`internal/artifacts`, then validates the result against the resolved request:
|
||||||
- `current/run_id.txt` (required, non-empty);
|
|
||||||
- `current/manifest.json` (required, decodable).
|
|
||||||
|
|
||||||
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
|
|
||||||
|
|
||||||
- campaign must match;
|
- campaign must match;
|
||||||
- session ID must match.
|
- session ID must match.
|
||||||
@@ -34,26 +33,11 @@ Planner behavior:
|
|||||||
|
|
||||||
- remote list scope is the resolved session prefix;
|
- remote list scope is the resolved session prefix;
|
||||||
- remote-to-local mapping is traversal-safe;
|
- remote-to-local mapping is traversal-safe;
|
||||||
- actions are sorted deterministically by local relative path.
|
- actions are sorted by local relative path and then remote key;
|
||||||
|
- force converts differing local targets from conflicts to downloads.
|
||||||
|
|
||||||
Restore scope from current remote state:
|
Previous-cache files are planned separately through `previouscache.BuildPlan`
|
||||||
|
when configured previous-session requirements exist.
|
||||||
- include `manifest.json`;
|
|
||||||
- include `transcripts/**`;
|
|
||||||
- include `artifacts/**`;
|
|
||||||
- include `audio/**` only with `--include-audio`.
|
|
||||||
|
|
||||||
Explicit exclusions from current remote state mapping:
|
|
||||||
|
|
||||||
- `current/**`;
|
|
||||||
- `runs/**`;
|
|
||||||
- `logs/**`;
|
|
||||||
- `reports/**`;
|
|
||||||
- `config/**`;
|
|
||||||
- `inputs/**`;
|
|
||||||
- `previous/**`.
|
|
||||||
|
|
||||||
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
|
|
||||||
|
|
||||||
## Execution Contract
|
## Execution Contract
|
||||||
|
|
||||||
@@ -73,8 +57,9 @@ Audio restore path:
|
|||||||
|
|
||||||
## Reporting Contract
|
## Reporting Contract
|
||||||
|
|
||||||
- `--dry-run`: prints summary only; no local writes.
|
- dry-run mode prints a summary and performs no local writes;
|
||||||
- non-dry-run: writes `reports/restore-latest.json`.
|
- execution mode persists the canonical restore report described in
|
||||||
|
[Operations](../operations.md#restore-workflow);
|
||||||
- report includes plan counts, per-action status, and execution failures.
|
- report includes plan counts, per-action status, and execution failures.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
@@ -82,3 +67,15 @@ Audio restore path:
|
|||||||
- restore uses committed remote current state as authority;
|
- restore uses committed remote current state as authority;
|
||||||
- `current/run_id.txt` is the remote publish commit marker;
|
- `current/run_id.txt` is the remote publish commit marker;
|
||||||
- restore does not execute pipeline stages.
|
- restore does not execute pipeline stages.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Discovery: `internal/app/restore_discovery.go`
|
||||||
|
- Planning: `internal/app/restore_plan.go`, `internal/previouscache`
|
||||||
|
- Execution: `internal/app/restore_execute.go`
|
||||||
|
- Reporting and command coordination: `internal/app/restore_report.go`,
|
||||||
|
`internal/app/restore.go`
|
||||||
|
- Tests: `internal/app/restore_discovery_test.go`,
|
||||||
|
`internal/app/restore_plan_test.go`,
|
||||||
|
`internal/app/restore_execution_test.go`,
|
||||||
|
`internal/app/restore_workflow_test.go`
|
||||||
|
|||||||
@@ -1,13 +1,15 @@
|
|||||||
# Internal: Manifest
|
# Internal: Manifest
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts.
|
|
||||||
|
Explain the session-progress and invocation-audit models implemented by
|
||||||
|
`internal/manifest`. Physical manifest placement belongs in
|
||||||
|
[Operations](../operations.md#local-state-layout).
|
||||||
|
|
||||||
## Session Manifest
|
## Session Manifest
|
||||||
Path:
|
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
|
||||||
|
|
||||||
Primary model (`manifest.Manifest`):
|
`manifest.Manifest` records:
|
||||||
|
|
||||||
- identity (`session_id`, `campaign`, `run_id`)
|
- identity (`session_id`, `campaign`, `run_id`)
|
||||||
- local path metadata (`local_workdir`, `local_spool_dir`)
|
- local path metadata (`local_workdir`, `local_spool_dir`)
|
||||||
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
||||||
@@ -15,7 +17,8 @@ Primary model (`manifest.Manifest`):
|
|||||||
- durable `artifacts` records
|
- durable `artifacts` records
|
||||||
- per-stage `stages` map
|
- per-stage `stages` map
|
||||||
|
|
||||||
Stage status enum:
|
The model admits these stage states:
|
||||||
|
|
||||||
- `pending`
|
- `pending`
|
||||||
- `running`
|
- `running`
|
||||||
- `succeeded`
|
- `succeeded`
|
||||||
@@ -25,10 +28,9 @@ Stage status enum:
|
|||||||
- `interrupted`
|
- `interrupted`
|
||||||
|
|
||||||
## Run Manifest
|
## Run Manifest
|
||||||
Path:
|
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
|
||||||
|
|
||||||
Run model (`manifest.RunManifest`):
|
`manifest.RunManifest` is created for each invocation and records:
|
||||||
|
|
||||||
- invocation identity and `force` flag
|
- invocation identity and `force` flag
|
||||||
- requested stages
|
- requested stages
|
||||||
- per-stage action (`run` or `skip`)
|
- per-stage action (`run` or `skip`)
|
||||||
@@ -36,22 +38,40 @@ Run model (`manifest.RunManifest`):
|
|||||||
- overall run status (`running`, `succeeded`, `failed`)
|
- overall run status (`running`, `succeeded`, `failed`)
|
||||||
|
|
||||||
## Persistence Semantics
|
## Persistence Semantics
|
||||||
|
|
||||||
`manifest.LocalStore`:
|
`manifest.LocalStore`:
|
||||||
|
|
||||||
- validates loaded documents;
|
- validates loaded documents;
|
||||||
- normalizes missing maps/stage records;
|
- normalizes missing maps/stage records;
|
||||||
- writes atomically via temp file + rename;
|
- writes atomically via temp file + rename;
|
||||||
- updates `updated_at` on save.
|
- updates `updated_at` on save.
|
||||||
|
|
||||||
## Execution Semantics
|
## Execution Semantics
|
||||||
Runner updates both manifests per stage transition:
|
|
||||||
- mark running
|
The application runner marks an executing stage running and then succeeded or
|
||||||
- mark succeeded/failed/skipped
|
failed in both manifests, persisting each transition. On success it records
|
||||||
- persist logs/generated config refs and metadata
|
outputs, logs, generated configuration references, and metadata. A successful
|
||||||
|
forced rerun marks only succeeded downstream session-stage records stale.
|
||||||
|
|
||||||
|
When an already-succeeded stage is skipped, the invocation run manifest records
|
||||||
|
the `skip` action and reason. The session manifest deliberately retains its
|
||||||
|
existing succeeded record because it remains the cross-invocation progress
|
||||||
|
authority.
|
||||||
|
|
||||||
Session manifest is the authoritative stage-progress ledger across invocations.
|
Session manifest is the authoritative stage-progress ledger across invocations.
|
||||||
Run manifest is invocation-scoped audit state.
|
Run manifest is invocation-scoped audit state.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- stage resume/skip decisions are session-manifest driven.
|
- stage resume/skip decisions are session-manifest driven.
|
||||||
- force reruns stale downstream succeeded stages.
|
- force reruns stale downstream succeeded stages.
|
||||||
- run manifest does not replace session manifest as progress authority.
|
- run manifest does not replace session manifest as progress authority.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Models and transitions: `internal/manifest/manifest.go`,
|
||||||
|
`internal/manifest/run_manifest.go`
|
||||||
|
- Persistence and validation: `internal/manifest/store.go`
|
||||||
|
- Package tests: `internal/manifest/*_test.go`
|
||||||
|
- Assembled execution behavior: `internal/app/runner_test.go`,
|
||||||
|
`internal/app/run_stage_test.go`
|
||||||
|
|||||||
@@ -59,7 +59,8 @@ The implemented canonical order is:
|
|||||||
10. `notify` (placeholder)
|
10. `notify` (placeholder)
|
||||||
|
|
||||||
`notify` currently has optional notifier call behavior and no persisted pipeline
|
`notify` currently has optional notifier call behavior and no persisted pipeline
|
||||||
outputs. The focused stage documents own implementation mechanics. The
|
outputs; its default collaborator is a no-op sender. The focused stage
|
||||||
|
documents own implementation mechanics. The
|
||||||
[CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation
|
[CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation
|
||||||
and execution semantics.
|
and execution semantics.
|
||||||
|
|
||||||
@@ -88,3 +89,8 @@ and execution semantics.
|
|||||||
|
|
||||||
Use this map to find an owner, then read its focused documentation and tests
|
Use this map to find an owner, then read its focused documentation and tests
|
||||||
before changing behavior.
|
before changing behavior.
|
||||||
|
|
||||||
|
The stage registry is implemented in `internal/stage/placeholders.go` and its
|
||||||
|
ordering is protected by `internal/app/planner_test.go`. Cross-invocation skip,
|
||||||
|
force, failure, and invalidation behavior is exercised in
|
||||||
|
`internal/app/runner_test.go` and `internal/app/run_stage_test.go`.
|
||||||
|
|||||||
@@ -1,24 +1,29 @@
|
|||||||
# Stage: analyze
|
# Stage: analyze
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- configured artifacts from `pipeline.scriptorium.artifacts`
|
- configured artifacts from `pipeline.scriptorium.artifacts`
|
||||||
- optional selected artifact filter (`--artifacts`)
|
- optional selected artifact keys supplied through the stage environment
|
||||||
- built-in/configured/previous-session source references in artifact inputs
|
- built-in/configured/previous-session source references in artifact inputs
|
||||||
|
|
||||||
Supported source families:
|
Supported source families:
|
||||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
||||||
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`, `narratio.input.glossary`
|
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`,
|
||||||
|
`narratio.input.glossary`
|
||||||
- configured artifacts: `narratio.artifact.<key>`
|
- configured artifacts: `narratio.artifact.<key>`
|
||||||
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- one materialized output per executed configured artifact (`output_path`)
|
- one materialized output per executed configured artifact (`output_path`)
|
||||||
- stage metadata describing selected/generated/reused artifacts
|
- stage metadata describing selected/generated/reused artifacts
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
||||||
- builds runtime artifact catalog (built-ins + configured artifacts).
|
- builds runtime artifact catalog (built-ins + configured artifacts).
|
||||||
- marks non-executable configured artifacts as reusable when output files already exist.
|
- marks non-executable configured artifacts as reusable when output files already exist.
|
||||||
@@ -30,13 +35,26 @@ Supported source families:
|
|||||||
- validates non-empty output files and materializes canonical outputs.
|
- validates non-empty output files and materializes canonical outputs.
|
||||||
|
|
||||||
## Failure Semantics
|
## Failure Semantics
|
||||||
|
|
||||||
- required missing configured/previous-session inputs fail.
|
- required missing configured/previous-session inputs fail.
|
||||||
- missing required prepared stable input source includes prepare rerun guidance.
|
- missing required prepared stable input source includes prepare rerun guidance.
|
||||||
- missing required previous-session source includes prepare rerun guidance.
|
- missing required previous-session source includes prepare rerun guidance.
|
||||||
- missing required `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` inputs includes render rerun guidance.
|
- missing required `narratio.transcript.final_markdown` or
|
||||||
|
`narratio.transcript.final_trimmed_markdown` inputs includes render rerun
|
||||||
|
guidance.
|
||||||
- dependency cycles or unavailable required dependencies fail.
|
- dependency cycles or unavailable required dependencies fail.
|
||||||
- adapter validation failures fail stage.
|
- adapter validation failures fail stage.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- `analyze` performs no remote storage calls for previous-session source resolution.
|
- `analyze` performs no remote storage calls for previous-session source resolution.
|
||||||
- output provenance and metadata are deterministic per execution.
|
- output provenance and metadata are deterministic per execution.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact
|
||||||
|
fields and source-selection rules.
|
||||||
|
- [CLI](../cli.md) owns user-visible artifact selection.
|
||||||
|
- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract.
|
||||||
|
- Implementation and tests: `internal/stage/analyze.go`,
|
||||||
|
`internal/stage/analyze_test.go`
|
||||||
|
|||||||
@@ -1,18 +1,22 @@
|
|||||||
# Stage: merge
|
# Stage: merge
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/raw/*.json`
|
- `transcripts/raw/*.json`
|
||||||
- `inputs/speakers.yml`
|
- `inputs/speakers.yml`
|
||||||
- `inputs/autocorrect.yml`
|
- `inputs/autocorrect.yml`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- optional `artifacts/seriatim.report.json`
|
- optional `artifacts/seriatim.report.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- discovers and validates raw transcript inputs.
|
- discovers and validates raw transcript inputs.
|
||||||
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
||||||
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
||||||
@@ -20,6 +24,14 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
|||||||
- materializes canonical outputs and records stage logs/generated configs.
|
- materializes canonical outputs and records stage logs/generated configs.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- merge always consumes normalized forms of raw inputs.
|
- merge always consumes normalized forms of raw inputs.
|
||||||
- base transcript must validate before stage success.
|
- base transcript must validate before stage success.
|
||||||
- report output is config-gated.
|
- report output is config-gated.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns operator-selected Seriatim values.
|
||||||
|
- Implementation and tests: `internal/stage/merge.go`,
|
||||||
|
`internal/stage/merge_test.go`
|
||||||
|
|||||||
@@ -1,16 +1,20 @@
|
|||||||
# Stage: normalize
|
# Stage: normalize
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Normalize polished transcript into final transcript using Seriatim.
|
Normalize polished transcript into final transcript using Seriatim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/final.json` (or configured normalize output path)
|
- `transcripts/final.json` (or configured normalize output path)
|
||||||
- optional `artifacts/seriatim.normalize.report.json`
|
- optional `artifacts/seriatim.normalize.report.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- resolves polished transcript from manifest outputs/canonical fallback.
|
- resolves polished transcript from manifest outputs/canonical fallback.
|
||||||
- applies `pipeline.normalize` config or default normalize config.
|
- applies `pipeline.normalize` config or default normalize config.
|
||||||
- runs Seriatim normalize with configured timeout/binary.
|
- runs Seriatim normalize with configured timeout/binary.
|
||||||
@@ -18,5 +22,13 @@ Normalize polished transcript into final transcript using Seriatim.
|
|||||||
- materializes canonical outputs and records logs/generated configs.
|
- materializes canonical outputs and records logs/generated configs.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- final transcript must validate as processed transcript JSON (`segments` array).
|
- final transcript must validate as processed transcript JSON (`segments` array).
|
||||||
- normalize defaults are applied when `pipeline.normalize` is unset.
|
- normalize defaults are applied when `pipeline.normalize` is unset.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns normalize fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/normalize.go`,
|
||||||
|
`internal/stage/normalize_test.go`
|
||||||
|
|||||||
@@ -1,17 +1,21 @@
|
|||||||
# Stage: polish
|
# Stage: polish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Run Audita polishing on base transcript and produce polished transcript.
|
Run Audita polishing on base transcript and produce polished transcript.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- `inputs/glossary.yml`
|
- `inputs/glossary.yml`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
- optional `artifacts/audita.report.json`
|
- optional `artifacts/audita.report.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- resolves base transcript from merge outputs/canonical fallback.
|
- resolves base transcript from merge outputs/canonical fallback.
|
||||||
- invokes Audita with configured model/module/runtime options.
|
- invokes Audita with configured model/module/runtime options.
|
||||||
- validates processed transcript structure (`segments` array required).
|
- validates processed transcript structure (`segments` array required).
|
||||||
@@ -19,5 +23,14 @@ Run Audita polishing on base transcript and produce polished transcript.
|
|||||||
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- polished transcript schema validation is mandatory.
|
- polished transcript schema validation is mandatory.
|
||||||
- report output is config-gated.
|
- report output is config-gated.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Audita](../integrations/audita.md) owns subprocess, validation, and failure
|
||||||
|
semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns operator-selected Audita values.
|
||||||
|
- Implementation and tests: `internal/stage/polish.go`,
|
||||||
|
`internal/stage/polish_test.go`
|
||||||
|
|||||||
@@ -1,17 +1,18 @@
|
|||||||
# Stage: prepare
|
# Stage: prepare
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Materialize canonical current-session inputs before processing stages.
|
Materialize canonical current-session inputs before processing stages.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
- resolved `campaign.yml`, `session.yml`, and pipeline config
|
|
||||||
|
- resolved campaign, session, and pipeline configuration
|
||||||
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
|
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
|
||||||
- audio source:
|
- one resolved local or S3 audio source
|
||||||
- local `audio_dir`/`audio_files`, or
|
|
||||||
- S3 `audio_s3.prefix`
|
|
||||||
- enabled configured artifact input requirements for previous-session sources
|
- enabled configured artifact input requirements for previous-session sources
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `inputs/campaign.yml`
|
- `inputs/campaign.yml`
|
||||||
- `inputs/session.yml`
|
- `inputs/session.yml`
|
||||||
- `inputs/pipeline.resolved.yml`
|
- `inputs/pipeline.resolved.yml`
|
||||||
@@ -26,6 +27,7 @@ Materialize canonical current-session inputs before processing stages.
|
|||||||
- deterministic `manifest.inputs` entries (checksums + provenance)
|
- deterministic `manifest.inputs` entries (checksums + provenance)
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- validates required config/store state.
|
- validates required config/store state.
|
||||||
- enforces local audio vs S3 audio mutual exclusivity.
|
- enforces local audio vs S3 audio mutual exclusivity.
|
||||||
- materializes S3 audio through spool/cache-aware logic.
|
- materializes S3 audio through spool/cache-aware logic.
|
||||||
@@ -39,6 +41,20 @@ Materialize canonical current-session inputs before processing stages.
|
|||||||
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- only `prepare` hydrates canonical `previous/` cache state.
|
- only `prepare` hydrates canonical `previous/` cache state.
|
||||||
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
|
- managed previous artifacts are stored under `previous/artifacts/**` without
|
||||||
|
duplicate `artifacts/artifacts/` nesting.
|
||||||
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md) owns audio selection, stable input fields, and
|
||||||
|
previous-session settings.
|
||||||
|
- [Operations](../operations.md) owns physical input, audio, spool, cache, and
|
||||||
|
previous-state layout.
|
||||||
|
- [Storage Internals](storage.md) and [Artifact Internals](artifacts.md) explain
|
||||||
|
the internal collaborators.
|
||||||
|
- Implementation and tests: `internal/stage/prepare.go`,
|
||||||
|
`internal/stage/prepare_test.go`, `internal/audio/s3_audio_test.go`,
|
||||||
|
`internal/previouscache/*_test.go`
|
||||||
|
|||||||
@@ -1,23 +1,29 @@
|
|||||||
# Stage: publish
|
# Stage: publish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Upload run/session outputs to object storage and atomically advance remote current state.
|
Upload run/session outputs to object storage and atomically advance remote current state.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`
|
|
||||||
- run root `runs/{run_id}/**`
|
- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set)
|
||||||
- publish output rules (`pipeline.publish.outputs`)
|
- invocation-scoped run files
|
||||||
|
- resolved publish output rules
|
||||||
- effective publish locks (static + remote merged lock set)
|
- effective publish locks (static + remote merged lock set)
|
||||||
- local `previous/**` files when present
|
- durable previous-session cache files when present
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
|
|
||||||
- uploaded selected publish outputs under session prefix
|
- uploaded invocation record and selected publish outputs;
|
||||||
- uploaded `previous/**` files under session prefix when present
|
- uploaded durable previous-session cache files when present;
|
||||||
- uploaded `current/manifest.json`
|
- updated remote current manifest; and
|
||||||
- uploaded `current/run_id.txt` written last
|
- remote current-run commit marker, written last.
|
||||||
|
|
||||||
|
Exact remote placement and the operator workflow belong in
|
||||||
|
[Operations](../operations.md#publish-workflow).
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- stage can self-skip when publish disabled or run upload disabled.
|
- stage can self-skip when publish disabled or run upload disabled.
|
||||||
- validates prerequisite stage success and object-store availability.
|
- validates prerequisite stage success and object-store availability.
|
||||||
- collects deterministic run file list plus run `manifest.json`.
|
- collects deterministic run file list plus run `manifest.json`.
|
||||||
@@ -28,6 +34,7 @@ Upload run/session outputs to object storage and atomically advance remote curre
|
|||||||
- writes remote current manifest before current run pointer.
|
- writes remote current manifest before current run pointer.
|
||||||
|
|
||||||
## Metadata Signals
|
## Metadata Signals
|
||||||
|
|
||||||
Includes counts/lists for:
|
Includes counts/lists for:
|
||||||
- run uploads
|
- run uploads
|
||||||
- published output uploads
|
- published output uploads
|
||||||
@@ -39,6 +46,22 @@ Includes counts/lists for:
|
|||||||
- `current_pointer_written`
|
- `current_pointer_written`
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- `current/run_id.txt` is the remote commit marker and is written last.
|
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||||
- run upload excludes `audio/**`.
|
- run upload excludes `audio/**`.
|
||||||
- publish locks are not overridden by `--force`.
|
- publish locks are not overridden by `--force`.
|
||||||
|
|
||||||
|
The commit boundary and cleanup gate are normative architecture invariants; see
|
||||||
|
[Architecture](../policy/architecture.md#publish-commit-boundary).
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md#publish-configuration-summary) owns output and
|
||||||
|
static-lock fields.
|
||||||
|
- [Operations](../operations.md#publish-locks) owns remote lock lifecycle and
|
||||||
|
physical remote state.
|
||||||
|
- [Artifact Internals](artifacts.md) explains source resolution and current-state
|
||||||
|
helpers.
|
||||||
|
- Implementation and tests: `internal/stage/publish.go`,
|
||||||
|
`internal/stage/publish_test.go`, `internal/app/operator_helpers_test.go`,
|
||||||
|
`internal/app/post_publish_cleanup_test.go`
|
||||||
|
|||||||
@@ -1,17 +1,21 @@
|
|||||||
# Stage: render
|
# Stage: render
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
|
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `narratio.transcript.final` (`transcripts/final.json`)
|
- `narratio.transcript.final` (`transcripts/final.json`)
|
||||||
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
|
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
|
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
|
||||||
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
|
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- uses `pipeline.render` settings (enabled/format/title/booleans).
|
- uses `pipeline.render` settings (enabled/format/title/booleans).
|
||||||
- resolves inputs manifest-first, then canonical fallback.
|
- resolves inputs manifest-first, then canonical fallback.
|
||||||
- writes run-local outputs first, then materializes canonical session outputs.
|
- writes run-local outputs first, then materializes canonical session outputs.
|
||||||
@@ -19,11 +23,20 @@ Render Markdown transcript artifacts from normalized JSON transcripts via Seriat
|
|||||||
- skips with stage metadata when `pipeline.render.enabled=false`.
|
- skips with stage metadata when `pipeline.render.enabled=false`.
|
||||||
|
|
||||||
## Failure Semantics
|
## Failure Semantics
|
||||||
|
|
||||||
- missing normalized input fails with normalize rerun guidance.
|
- missing normalized input fails with normalize rerun guidance.
|
||||||
- missing trimmed input fails with trim rerun guidance.
|
- missing trimmed input fails with trim rerun guidance.
|
||||||
- adapter/subprocess failure fails stage.
|
- adapter/subprocess failure fails stage.
|
||||||
- empty render output files fail validation.
|
- empty render output files fail validation.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- only `format: markdown` is supported.
|
- only `format: markdown` is supported.
|
||||||
- render stage owns production of built-in Markdown transcript sources.
|
- render stage owns production of built-in Markdown transcript sources.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns render subprocess behavior.
|
||||||
|
- [Configuration](../config.md#pipeline) owns render fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/render.go`,
|
||||||
|
`internal/stage/render_test.go`
|
||||||
|
|||||||
@@ -1,22 +1,36 @@
|
|||||||
# Stage: transcribe
|
# Stage: transcribe
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `audio/*.flac` from `prepare`
|
- `audio/*.flac` from `prepare`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/raw/<speaker>.json`
|
- `transcripts/raw/<speaker>.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- discovers prepared audio from manifest inputs or canonical audio directory.
|
- discovers prepared audio from manifest inputs or canonical audio directory.
|
||||||
- derives speaker ID from `.flac` basename.
|
- derives speaker ID from `.flac` basename.
|
||||||
- runs WhisperX with configured concurrency/retry settings.
|
- dispatches WhisperX requests through a bounded worker pool.
|
||||||
- validates each output as JSON.
|
- validates each output as JSON.
|
||||||
- writes run-local outputs then materializes canonical transcript outputs.
|
- writes run-local outputs then materializes canonical transcript outputs.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- speaker basenames must be unique.
|
- speaker basenames must be unique.
|
||||||
- output path returned by adapter must match requested output path.
|
- output path returned by adapter must match requested output path.
|
||||||
- each successful output is validated before stage success.
|
- each successful output is validated before stage success.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [WhisperX](../integrations/whisperx.md) owns HTTP, retry, timeout, and
|
||||||
|
cancellation semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns concurrency and other
|
||||||
|
operator-selected values.
|
||||||
|
- Implementation and tests: `internal/stage/transcribe.go`,
|
||||||
|
`internal/stage/transcribe_test.go`
|
||||||
|
|||||||
@@ -1,16 +1,21 @@
|
|||||||
# Stage: trim
|
# Stage: trim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Produce a final-trimmed transcript. By default, the stage generates bounds and applies a bounds-driven trim.
|
|
||||||
|
Produce a final-trimmed transcript. By default, the stage generates bounds and
|
||||||
|
applies a bounds-driven trim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/final.json`
|
- `transcripts/final.json`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/final.trimmed.json` (or configured trim output path)
|
- `transcripts/final.trimmed.json` (or configured trim output path)
|
||||||
- when trim enabled: `artifacts/session_bounds.json`
|
- when trim enabled: `artifacts/session_bounds.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
When `trim.enabled=true`:
|
When `trim.enabled=true`:
|
||||||
- runs Scriptorium bounds artifact generation;
|
- runs Scriptorium bounds artifact generation;
|
||||||
- optionally runs render-debug output generation;
|
- optionally runs render-debug output generation;
|
||||||
@@ -23,6 +28,16 @@ When `trim.enabled=false`:
|
|||||||
- copies normalized transcript to trimmed output.
|
- copies normalized transcript to trimmed output.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- normalized transcript is required input.
|
- normalized transcript is required input.
|
||||||
- bounds output exists only in enabled trim path.
|
- bounds output exists only in enabled trim path.
|
||||||
- render-debug output is diagnostic and not a declared stage output.
|
- render-debug output is diagnostic and not a declared stage output.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Scriptorium](../integrations/scriptorium.md) owns bounds generation and
|
||||||
|
debug-render subprocess behavior.
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns transcript trimming behavior.
|
||||||
|
- [Configuration](../config.md#pipeline) owns trim fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/trim.go`,
|
||||||
|
`internal/stage/trim_test.go`
|
||||||
|
|||||||
@@ -1,10 +1,16 @@
|
|||||||
# Internal: Storage
|
# Internal: Storage
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Document remote object-store contracts and S3 implementation behavior.
|
|
||||||
|
Explain the object-store interface and S3 implementation used by Narratio.
|
||||||
|
Remote key layout and lifecycle belong in [Operations](../operations.md), while
|
||||||
|
operator-selected storage fields and credential mechanisms belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|
||||||
## Primary Contract
|
## Primary Contract
|
||||||
|
|
||||||
`storage.ObjectStore` interface:
|
`storage.ObjectStore` interface:
|
||||||
|
|
||||||
- `List(ctx, prefix)`
|
- `List(ctx, prefix)`
|
||||||
- `Download(ctx, key, localPath)`
|
- `Download(ctx, key, localPath)`
|
||||||
- `Upload(ctx, localPath, key, opts)`
|
- `Upload(ctx, localPath, key, opts)`
|
||||||
@@ -14,15 +20,15 @@ Key invariant:
|
|||||||
- callers pass full bucket-relative keys;
|
- callers pass full bucket-relative keys;
|
||||||
- storage implementations do not infer campaign/session/run prefixes.
|
- storage implementations do not infer campaign/session/run prefixes.
|
||||||
|
|
||||||
## Configuration
|
## Composition
|
||||||
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
|
|
||||||
|
|
||||||
S3 constructor behavior:
|
`NewObjectStoreFromConfig` constructs the S3-backed implementation from
|
||||||
- requires configured bucket;
|
resolved configuration. The application loads configured filesystem secrets
|
||||||
- uses region/endpoint/path-style options when set;
|
before calling it. The storage adapter consumes already-resolved values; it does
|
||||||
- resolves credentials from configured env var names (with defaults).
|
not own discovery, defaults, or configuration validation.
|
||||||
|
|
||||||
## S3 Backend Behavior
|
## S3 Backend Behavior
|
||||||
|
|
||||||
- normalizes object keys.
|
- normalizes object keys.
|
||||||
- `List` paginates and returns normalized `ObjectInfo`.
|
- `List` paginates and returns normalized `ObjectInfo`.
|
||||||
- `Download` writes local files with parent directory creation.
|
- `Download` writes local files with parent directory creation.
|
||||||
@@ -30,5 +36,13 @@ S3 constructor behavior:
|
|||||||
- `Exists` maps not-found responses to `false`.
|
- `Exists` maps not-found responses to `false`.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- storage layer is stateless regarding manifest/stage progression.
|
- storage layer is stateless regarding manifest/stage progression.
|
||||||
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Contract and S3 adapter: `internal/adapters/storage`
|
||||||
|
- Composition: `internal/app/object_store.go`
|
||||||
|
- Tests: `internal/adapters/storage/*_test.go`,
|
||||||
|
`internal/app/object_store_test.go`
|
||||||
|
|||||||
@@ -1,57 +1,62 @@
|
|||||||
# Internal: Workspace
|
# Internal: Workspace
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define local session layout, run-local stage layout, and cleanup guardrails.
|
|
||||||
|
|
||||||
## Canonical Session Layout
|
Explain the helpers that construct local session and run paths, coordinate
|
||||||
Session root:
|
single-writer access, and confine cleanup. The authoritative physical layout and
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
retention workflow belong in [Operations](../operations.md#local-state-layout).
|
||||||
|
|
||||||
Core directories/files:
|
## Path Ownership
|
||||||
- `inputs/`
|
|
||||||
- `audio/`
|
|
||||||
- `transcripts/`
|
|
||||||
- `artifacts/`
|
|
||||||
- `reports/`
|
|
||||||
- `logs/`
|
|
||||||
- `config/`
|
|
||||||
- `current/`
|
|
||||||
- `runs/`
|
|
||||||
- `previous/`
|
|
||||||
- `manifest.json`
|
|
||||||
- `.lock`
|
|
||||||
|
|
||||||
`previous/` reserved files:
|
`internal/artifacts` owns canonical session, run, spool, cache, and
|
||||||
- `previous/manifest.json`
|
previous-cache path construction. `SessionPathsFor` provides the session-scoped
|
||||||
- `previous/artifacts/**`
|
path model, and layout creation goes through `EnsureLayoutFor`. Callers should
|
||||||
|
consume those helpers instead of rebuilding relative paths.
|
||||||
|
|
||||||
|
`internal/pathsafe` and application cleanup helpers enforce confinement for
|
||||||
|
relative destinations and deletion targets.
|
||||||
|
|
||||||
## Run-Local Stage Layout
|
## Run-Local Stage Layout
|
||||||
When run context is available, stages use:
|
|
||||||
- `runs/{run_id}/{stage}/outputs/`
|
|
||||||
- `runs/{run_id}/{stage}/logs/`
|
|
||||||
- `runs/{run_id}/{stage}/reports/`
|
|
||||||
- `runs/{run_id}/{stage}/config/`
|
|
||||||
- `runs/{run_id}/{stage}/scratch/`
|
|
||||||
|
|
||||||
Run-local outputs are materialized back into canonical session paths before stage success.
|
`internal/stage/run_local.go` maps stage outputs and diagnostics into an
|
||||||
`previous/**` writes are never redirected to run-local output paths.
|
invocation-scoped layout. Successful outputs are validated and atomically
|
||||||
|
materialized into canonical session paths before stage success. Managed
|
||||||
|
previous-session cache paths remain session-durable and are never redirected
|
||||||
|
into run-local output space.
|
||||||
|
|
||||||
## Locking
|
## Locking
|
||||||
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
|
||||||
|
`artifacts.LocalStore` enforces the single-writer session lock via `.lock`
|
||||||
|
(`ErrLockConflict` on contention).
|
||||||
|
|
||||||
## Cleanup Semantics
|
## Cleanup Semantics
|
||||||
|
|
||||||
Automatic post-publish cleanup:
|
Automatic post-publish cleanup:
|
||||||
|
|
||||||
- only runs when publish actually executed and succeeded;
|
- only runs when publish actually executed and succeeded;
|
||||||
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||||
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
- consumes the resolved cleanup policy described in
|
||||||
|
[Configuration](../config.md);
|
||||||
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
||||||
|
|
||||||
Manual clean command:
|
Manual cleanup uses the same scoped-target checks. Invocation syntax and exact
|
||||||
- `clean <session_id>` removes session work and spool subtree.
|
deletion scope belong in [CLI](../cli.md#clean) and
|
||||||
- `clean --all` removes all workspace work and spool children.
|
[Operations](../operations.md#cleanup).
|
||||||
- durable cache is preserved unless `--clear-cache` is requested.
|
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- campaign-aware session root is mandatory.
|
- campaign-aware session root is mandatory.
|
||||||
- manifest-driven stage state is durable across runs.
|
- manifest-driven stage state is durable across runs.
|
||||||
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Path model and local store: `internal/artifacts/paths.go`,
|
||||||
|
`internal/artifacts/local.go`
|
||||||
|
- Run-local materialization: `internal/stage/run_local.go`
|
||||||
|
- Cleanup confinement: `internal/app/cleanup_targets.go`,
|
||||||
|
`internal/app/post_publish_cleanup.go`
|
||||||
|
- Tests: `internal/artifacts/paths_model_test.go`,
|
||||||
|
`internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`,
|
||||||
|
`internal/app/cleanup_targets_test.go`,
|
||||||
|
`internal/app/post_publish_cleanup_test.go`
|
||||||
|
|||||||
@@ -66,7 +66,8 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa
|
|||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
|
|
||||||
- align config with [docs/config.md](./config.md) and maintained files under `examples/`.
|
- align config with [Configuration](./config.md) and the
|
||||||
|
[maintained examples](../examples/README.md).
|
||||||
|
|
||||||
Relevant reference: [Configuration](./config.md).
|
Relevant reference: [Configuration](./config.md).
|
||||||
|
|
||||||
|
|||||||
46
examples/README.md
Normal file
46
examples/README.md
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
# Maintained Examples
|
||||||
|
|
||||||
|
These files are safe, copyable starting points for Narratio configuration and
|
||||||
|
input structure. Replace placeholder identifiers, storage names, integration
|
||||||
|
URLs, and paths for the target environment. Field meanings and defaults belong
|
||||||
|
in the [configuration reference](../docs/config.md).
|
||||||
|
|
||||||
|
## Pipeline Configuration
|
||||||
|
|
||||||
|
- [Minimal pipeline](pipeline.minimal.yml): campaign discovery plus the required
|
||||||
|
WhisperX URL.
|
||||||
|
- [Production-shaped pipeline](pipeline.production.yml): S3 storage, publish,
|
||||||
|
external tools, and configured Scriptorium artifacts.
|
||||||
|
- [Full annotated pipeline](pipeline.full.annotated.yml): every implemented
|
||||||
|
pipeline section with explanatory comments.
|
||||||
|
|
||||||
|
The existing `internal/config` example test loads and validates each pipeline
|
||||||
|
with the sample campaign and a compatible local- or S3-audio session.
|
||||||
|
|
||||||
|
## Campaign And Session Configuration
|
||||||
|
|
||||||
|
- [Sample campaign](campaigns/sample-campaign/campaign.yml), its
|
||||||
|
[session template](campaigns/sample-campaign/session.template.yml), and its
|
||||||
|
adjacent stable inputs provide a complete campaign directory shape.
|
||||||
|
- [Local-audio session](session.local-audio.yml) and
|
||||||
|
[S3-audio session](session.s3-audio.yml) are concrete session files.
|
||||||
|
- [Session template](session.template.yml) and the campaign-local equivalent
|
||||||
|
demonstrate the narrow placeholder syntax consumed by `session init`; they
|
||||||
|
are templates, not runtime session files.
|
||||||
|
|
||||||
|
## Input Fixtures
|
||||||
|
|
||||||
|
- [Speakers](speakers.yml), [autocorrect](autocorrect.yml), and
|
||||||
|
[glossary](glossary.yml) show the standalone input shapes.
|
||||||
|
- The sample campaign references its local
|
||||||
|
[speakers](campaigns/sample-campaign/speakers.yml),
|
||||||
|
[autocorrect](campaigns/sample-campaign/autocorrect.yml),
|
||||||
|
[glossary](campaigns/sample-campaign/glossary.yml),
|
||||||
|
[players](campaigns/sample-campaign/players.yml), and
|
||||||
|
[party](campaigns/sample-campaign/party.yml) fixtures.
|
||||||
|
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
|
||||||
|
reserves the expected filename and directory shape. Replace it with a real
|
||||||
|
FLAC file before running transcription.
|
||||||
|
|
||||||
|
The examples contain environment-variable names but no credential values. They
|
||||||
|
use fictional campaign content and reserved example domains.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
match:
|
match:
|
||||||
- speaker: "Eric Rakestraw"
|
- speaker: "Example Speaker"
|
||||||
match:
|
match:
|
||||||
- "Eric_Rakestraw"
|
- "Example_Speaker"
|
||||||
- "Eric"
|
- "Example"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
match:
|
match:
|
||||||
- speaker: "Eric Rakestraw"
|
- speaker: "Example Speaker"
|
||||||
match:
|
match:
|
||||||
- "Eric_Rakestraw"
|
- "Example_Speaker"
|
||||||
- "Eric"
|
- "Example"
|
||||||
|
|||||||
Reference in New Issue
Block a user