From 571a289296d491ac93bf3917d9bcfa9432c8db9b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 19 May 2026 08:47:45 -0500 Subject: [PATCH] Added new internal documentation --- docs/internal/README.md | 4 ++ docs/internal/adapters.md | 79 +++++++++++++++++++++++++++++++++++ docs/internal/artifacts.md | 85 ++++++++++++++++++++++++++++++++++++++ docs/internal/manifest.md | 80 +++++++++++++++++++++++++++++++++++ docs/internal/storage.md | 71 +++++++++++++++++++++++++++++++ go.mod | 2 +- 6 files changed, 320 insertions(+), 1 deletion(-) create mode 100644 docs/internal/adapters.md create mode 100644 docs/internal/artifacts.md create mode 100644 docs/internal/manifest.md create mode 100644 docs/internal/storage.md diff --git a/docs/internal/README.md b/docs/internal/README.md index fa0fa99..26a7136 100644 --- a/docs/internal/README.md +++ b/docs/internal/README.md @@ -7,6 +7,10 @@ Developers and LLM coding agents changing Narratio internals. Implementation-accurate contracts for workspace/state, stages, and external adapter boundaries. ## Component Docs +- `adapters.md`: external adapter map, runtime wiring, and boundary ownership. +- `storage.md`: remote storage backend contracts and object-store invariants. +- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics. +- `artifacts.md`: supported artifact IDs, transcript tiers, and source-resolution behavior. - `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants. - `stage-prepare.md`: input materialization and provenance capture. - `stage-transcribe.md`: WhisperX transcript generation. diff --git a/docs/internal/adapters.md b/docs/internal/adapters.md new file mode 100644 index 0000000..797c3eb --- /dev/null +++ b/docs/internal/adapters.md @@ -0,0 +1,79 @@ +# Internal: Adapters + +## Purpose +Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring. + +## Inputs and outputs +Inputs: +- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications). +- Resolved config values used to construct default adapters. + +Outputs: +- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details). +- Adapter errors returned to stage/app orchestration. + +## Boundaries +Owns: +- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls). +- Request/response contracts in `internal/adapters/*` packages. + +Does not own: +- Stage sequencing, skip/force/resume decisions. +- Manifest transition logic. +- Canonical workspace path policy. + +## Config fields used +Default wiring and adapter calls consume: +- `pipeline.whisperx.*` +- `pipeline.seriatim.*` +- `pipeline.audita.*` +- `pipeline.scriptorium.*` +- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating) +- `pipeline.notification.*` (sender boundary exists; placeholder behavior today) + +## External adapters used +Runtime env boundary fields (`internal/stage.Env`): +- `whisperx.Client` +- `seriatim.Runner` +- `audita.Runner` +- `scriptorium.Runner` +- `storage.ObjectStore` +- `notify.Sender` +- `analyzer.Runner` + +Current execution usage: +- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`. +- Present but not used by implemented stage set: `Analyzer`, legacy `storage.Backend`. + +Default construction in app runner: +- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`. +- Callers can inject test/fake implementations through `app.RunOptions.Env`. + +## State and manifest behavior +- Adapters do not directly mutate session/run manifests. +- Stages and runner own manifest writes and stage status transitions. +- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata). + +## Skip and resume behavior +- No adapter-level skip/resume semantics. +- Skip/resume/force behavior is decided by app runner using manifest stage state. + +## Failure behavior +- Adapter constructors validate config-derived values and fail early on invalid required inputs. +- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic. +- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis. + +## Tests to inspect before changing +- `internal/adapters/whisperx/http_test.go` +- `internal/adapters/seriatim/subprocess_test.go` +- `internal/adapters/audita/subprocess_test.go` +- `internal/adapters/scriptorium/subprocess_test.go` +- `internal/adapters/storage/*_test.go` +- `internal/adapters/notify/fake_test.go` +- `internal/adapters/analyzer/fake_test.go` +- `internal/app/runner_test.go` + +## Architectural invariants +- Stage code depends on adapter interfaces, not transport-specific implementation types. +- External SDK-specific types remain inside adapter implementations. +- Default app wiring must remain deterministic and overrideable via injected env dependencies. diff --git a/docs/internal/artifacts.md b/docs/internal/artifacts.md new file mode 100644 index 0000000..4d04137 --- /dev/null +++ b/docs/internal/artifacts.md @@ -0,0 +1,85 @@ +# Internal: Artifacts + +## Purpose +Describe supported session artifact IDs, transcript tiers, and artifact resolution/provenance behavior used by stage logic and Scriptorium input configuration. + +## Inputs and outputs +Inputs: +- Artifact source identifiers from stage config/runtime (for example `pipeline.scriptorium.artifacts.*.inputs.*.source`). +- Session paths and optional session manifest stage outputs. + +Outputs: +- Resolved local artifact path + provenance (`ResolvedSessionArtifact`). +- Validation errors for unsupported or unreadable artifact sources. + +## Boundaries +Owns: +- Canonical artifact ID registry and metadata (`internal/artifacts/artifact_resolver.go`). +- Alias normalization for legacy source names. +- Resolution order and artifact content validation. + +Does not own: +- Artifact generation (stages produce files). +- Manifest transition policy. +- Remote archive publishing behavior. + +## Config fields used +Artifact source usage is driven by: +- `pipeline.scriptorium.artifacts..inputs..source` +- Optional source-specific fields for previous artifact input (`artifact`, `path`, `required`). + +## External adapters used +- No external service adapters. +- Resolver relies on local filesystem checks + session manifest state. + +## State and manifest behavior +Supported canonical IDs and current mappings: + +| Artifact ID | Canonical file | Producer stage | Output kind | +| --- | --- | --- | --- | +| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | +| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | +| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | +| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | +| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | +| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `session_recap` | + +Legacy aliases normalized by resolver: +- `processed_transcript` -> `narratio.transcript.polished` +- `normalized_transcript` -> `narratio.transcript.full` +- `trimmed_transcript` -> `narratio.transcript.trimmed` + +Resolution order: +1. Session manifest producer-stage outputs (if readable/valid). +2. Canonical session path fallback. + +Provenance fields: +- `ProducerStage` +- `OutputKind` +- `ProducerRunID` (when resolved from manifest output) +- `Provenance` (`manifest..outputs` or `fallback.canonical_path`) + +Content validation by artifact type: +- Transcript artifacts: JSON with top-level `segments` array. +- `narratio.bounds.session`: valid JSON. +- `narratio.artifact.session_recap`: non-empty text. + +## Skip and resume behavior +- Resolver has no direct skip/resume logic. +- Resolver output influences stage behavior (for example analyze input resolution and required-input failures). + +## Failure behavior +- Unsupported or empty artifact source -> normalization error. +- Known source not found/readable -> `ErrSessionArtifactNotFound` wrapped error. +- Found but invalid content -> validation error. + +## Tests to inspect before changing +- `internal/artifacts/artifact_resolver_test.go` +- `internal/artifacts/resolve_test.go` +- `internal/stage/analyze_test.go` +- `internal/config/scriptorium_test.go` + +## Architectural invariants +- Artifact IDs are canonical interface values for stage/config integration. +- Alias support is compatibility behavior layered on top of canonical IDs. +- Manifest producer outputs are preferred over canonical fallback when both exist. diff --git a/docs/internal/manifest.md b/docs/internal/manifest.md new file mode 100644 index 0000000..39261c2 --- /dev/null +++ b/docs/internal/manifest.md @@ -0,0 +1,80 @@ +# Internal: Manifest + +## Purpose +Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior. + +## Inputs and outputs +Inputs: +- Session identity and run identity from app orchestration. +- Stage transition events and stage result payloads. + +Outputs: +- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`. +- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`. + +## Boundaries +Owns: +- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records). +- Stage status/action transition methods. +- Persistent store contract (`manifest.Store`) and local JSON store implementation. + +Does not own: +- Stage implementation details. +- Path construction policy outside manifest file persistence calls. +- CLI command behavior. + +## Config fields used +Manifest package itself does not read config directly. + +Manifest identity fields are populated by app/stage orchestration from: +- `session.session_id` +- `session.campaign` +- `pipeline.workspace.root` +- `pipeline.storage.s3.*` (when archive/S3 identity is set) + +## External adapters used +- No external service adapters. +- Uses local filesystem for persistence via `manifest.LocalStore`. + +## State and manifest behavior +Session manifest model: +- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`). +- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata. + +Run manifest model: +- Tracks one invocation (`run_id`) with requested stages and force mode. +- Tracks per-stage action (`run` or `skip`) and per-stage status. +- Tracks overall run status (`running`, `succeeded`, `failed`). + +Persistence behavior: +- Load validates required identity/timestamp fields and normalizes maps/records. +- Save updates `updated_at` and writes JSON atomically (temp file + rename). +- Session and run manifests are saved incrementally before/after stage transitions. + +Relationship during execution: +- Runner updates both manifests for every stage transition. +- Session manifest is the durable pipeline-progress ledger. +- Run manifest is invocation history and audit record. + +## Skip and resume behavior +- Resume and skip decisions are based on session-manifest stage statuses. +- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest. +- Run manifest records whether each stage was executed or skipped in that invocation. + +## Failure behavior +- Stage failure marks both manifests failed for that stage and records error messages/timestamps. +- Save failures are returned immediately and fail the command. +- Invalid/malformed manifest files fail load with explicit validation/decode errors. + +## Tests to inspect before changing +- `internal/manifest/manifest_test.go` +- `internal/manifest/run_manifest_test.go` +- `internal/manifest/store_test.go` +- `internal/app/runner_test.go` +- `internal/app/run_control_test.go` +- `internal/app/resume_run_stage_test.go` + +## Architectural invariants +- Session manifest is authoritative for stage progression across invocations. +- Run manifest is invocation-scoped and never replaces session manifest as progress authority. +- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern). diff --git a/docs/internal/storage.md b/docs/internal/storage.md new file mode 100644 index 0000000..fc777c3 --- /dev/null +++ b/docs/internal/storage.md @@ -0,0 +1,71 @@ +# Internal: Storage + +## Purpose +Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`. + +## Inputs and outputs +Inputs: +- Resolved storage config (`pipeline.storage.*`). +- Bucket-relative object keys and local file paths from stage/app orchestration. + +Outputs: +- Listed/downloaded/uploaded object metadata (`ObjectInfo`). +- Existence checks and storage-layer errors. + +## Boundaries +Owns: +- Remote object-store interface and implementation details. +- S3 client wiring and API calls. +- Object key normalization and upload/download/list primitives. + +Does not own: +- Session/run prefix semantics. +- Archive commit order semantics. +- Manifest updates. + +## Config fields used +- `pipeline.storage.backend` +- `pipeline.storage.s3.bucket` +- `pipeline.storage.s3.region` +- `pipeline.storage.s3.endpoint` +- `pipeline.storage.s3.force_path_style` +- `pipeline.storage.s3.access_key_id_env` +- `pipeline.storage.s3.secret_access_key_env` + +## External adapters used +Storage package contracts: +- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`. +- `Backend` (archive request boundary): currently implemented with `NoopBackend` only. + +Implementations: +- `S3Backend`: AWS SDK-backed `ObjectStore` implementation. +- `FakeBackend`: deterministic test `ObjectStore` and archive backend. +- `NoopBackend`: deterministic no-op archive backend for compatibility wiring. + +## State and manifest behavior +- Storage implementations are stateless with respect to manifest/session lifecycle. +- Caller supplies fully-qualified bucket-relative keys. +- Storage layer does not infer campaign/session/run/root-prefix semantics. +- Caller controls publish ordering; storage layer executes individual operations in the order invoked. + +## Skip and resume behavior +- No storage-level skip/resume behavior. +- Skip/resume decisions are made by stage/app logic before storage calls occur. + +## Failure behavior +- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing. +- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails. +- CRUD operations return contextual errors (including not-found behavior via `Exists`). +- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed). + +## Tests to inspect before changing +- `internal/adapters/storage/factory_test.go` +- `internal/adapters/storage/s3_backend_test.go` +- `internal/adapters/storage/fake_test.go` +- `internal/adapters/storage/keys_test.go` +- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`) + +## Architectural invariants +- Callers pass full bucket-relative keys. +- Storage backends must not prepend or infer narratio prefixes. +- Remote transport details remain isolated to storage adapter implementations. diff --git a/go.mod b/go.mod index 17fc325..542124f 100644 --- a/go.mod +++ b/go.mod @@ -4,6 +4,7 @@ go 1.25.0 require ( github.com/aws/aws-sdk-go-v2/config v1.32.17 + github.com/aws/aws-sdk-go-v2/credentials v1.19.16 github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 github.com/aws/smithy-go v1.25.1 gopkg.in/yaml.v3 v3.0.1 @@ -12,7 +13,6 @@ require ( require ( github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect - github.com/aws/aws-sdk-go-v2/credentials v1.19.16 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect