diff --git a/.gitignore b/.gitignore index 072e8ab..82450fc 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,8 @@ .codex AGENTS.md +.DS_Store + # ---> Go # If you prefer the allow list template instead of the deny list, see community template: # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore diff --git a/docs/development/workspace.md b/docs/development/workspace.md new file mode 100644 index 0000000..5c44788 --- /dev/null +++ b/docs/development/workspace.md @@ -0,0 +1,753 @@ +# Narratio Workspace, Run History, and Artifact Resolution Architecture + +## 1. Purpose + +This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract. + +Narratio is an idempotent session orchestrator. The command: + +```bash +narratio run --session-id 2026-05-07 +``` + +means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state." + +This distinction drives the architecture: + +* A **session** is the durable domain object and idempotency boundary. +* A **run** is an execution attempt that may update the session's durable state. +* Durable outputs live at the session level. +* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`. +* Successful stage outputs are promoted from run-local locations into canonical session-level locations. +* The session manifest records current durable state. +* Run manifests record execution history and debugging/provenance details. + +This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history. + +## 2. Core Concepts + +### 2.1 Session + +A session is the stable unit of work identified by `campaign_id` and `session_id`. + +Examples: + +```text +campaign_id = dilfs +session_id = 2026-05-07 +``` + +The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected. + +### 2.2 Run + +A run is a particular execution attempt identified by a generated `run_id`, for example: + +```text +20260517T174748Z-abcd1234 +``` + +A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command. + +Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout. + +### 2.3 Durable Output + +A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions. + +Examples: + +```text +transcripts/merged.json +transcripts/processed.json +transcripts/normalized.json +transcripts/trimmed.json +artifacts/session_recap.md +``` + +Durable outputs live directly under the session directory, not under a particular run directory. + +### 2.4 Run-Local Output + +A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths. + +Run-local outputs, logs, generated configs, reports, and scratch files should remain under: + +```text +runs/{run_id}/{stage}/... +``` + +## 3. Local Workspace Layout + +The canonical local workspace layout is: + +```text +{workspace.root}/work/{campaign_id}/{session_id}/ + manifest.json + current/ + manifest.json + run_id.txt + inputs/ + transcripts/ + artifacts/ + reports/ + logs/ + config/ + runs/ + {run_id}/ + manifest.json + prepare/ + transcribe/ + merge/ + polish/ + normalize/ + trim/ + analyze/ + archive/ + notify/ +``` + +Not every directory must exist at all times. Directories should be created idempotently when needed. + +### 3.1 Session Root + +The session root is: + +```text +{workspace.root}/work/{campaign_id}/{session_id}/ +``` + +The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths. + +The only files that should live directly in the session root are core session-state files, primarily: + +```text +manifest.json +``` + +Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts. + +### 3.2 Session-Level Canonical Directories + +The following directories contain current durable session state: + +```text +inputs/ +transcripts/ +artifacts/ +reports/ +logs/ +config/ +current/ +``` + +Recommended meanings: + +| Directory | Purpose | +| -------------- | ----------------------------------------------------------------------------- | +| `inputs/` | Materialized or copied input files used by the current durable session state. | +| `transcripts/` | Canonical transcript tiers. | +| `artifacts/` | User-facing and machine-readable generated artifacts. | +| `reports/` | Canonical stage reports worth preserving at the session level. | +| `logs/` | Optional session-level logs or promoted/latest logs. | +| `config/` | Optional session-level generated config snapshots or promoted/latest configs. | +| `current/` | Current published session pointers, mirroring the archive backend. | + +Canonical durable outputs should use stable paths under these directories. + +### 3.3 Run History Directory + +Run history lives under: + +```text +{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/ +``` + +Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset. + +Example full run: + +```text +runs/20260517T174748Z-abcd1234/ + manifest.json + prepare/ + transcribe/ + merge/ + polish/ + normalize/ + trim/ + analyze/ + archive/ + notify/ +``` + +Example sparse forced analyze run: + +```text +runs/20260518T030000Z-efgh5678/ + manifest.json + analyze/ +``` + +Example sparse polish-through-analyze rerun: + +```text +runs/20260518T041500Z-a1b2c3d4/ + manifest.json + polish/ + normalize/ + trim/ + analyze/ +``` + +Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so. + +### 3.4 Stage Run-Local Directories + +Each stage receives a run-local directory: + +```text +runs/{run_id}/{stage}/ +``` + +Within that stage directory, the stage may use subdirectories such as: + +```text +outputs/ +logs/ +reports/ +config/ +scratch/ +``` + +For example: + +```text +runs/{run_id}/polish/ + outputs/transcripts/processed.json + reports/audita.polish.report.json + logs/stdout.log + logs/stderr.log + config/audita.polish.generated.yml + scratch/ +``` + +The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins. + +## 4. Promotion Model + +Narratio uses stage-level promotion with immediate promotion after successful validation. + +The stage lifecycle is: + +1. Resolve required inputs from the current session state and/or run-local context. +2. Create the run-local stage directory. +3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`. +4. Validate run-local outputs. +5. Promote durable outputs into session-level canonical paths. +6. Update the session manifest. +7. Update the run manifest. + +Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path. + +Example: + +```text +runs/{run_id}/polish/outputs/transcripts/processed.json +``` + +is promoted to: + +```text +transcripts/processed.json +``` + +Promotion should be safe and deterministic: + +* Validate before promotion. +* Write promoted files atomically where possible. +* Never leave partially written durable outputs. +* Record the producing `run_id` in the session manifest. +* Preserve run-local files for debugging unless retention policy deletes them. + +## 5. Promotion Policy: Option A + +Narratio uses immediate stage-level promotion. + +If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails. + +Example: + +```bash +narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze +``` + +If `polish` succeeds and `normalize` fails: + +* `transcripts/processed.json` may be updated from the new run. +* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state. +* Downstream outputs may now be stale relative to the newly promoted polished transcript. + +This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling. + +## 6. Stale and Invalidation Semantics + +Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns. + +When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation. + +Example stage order: + +```text +prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify +``` + +If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully: + +```text +normalize +trim +analyze +archive +notify +``` + +A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration. + +Minimum manifest state model: + +```text +pending +running +succeeded +failed +skipped +stale +``` + +If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs. + +## 7. Manifest Responsibilities + +Narratio should distinguish between session manifests and run manifests. + +The same underlying Go types may be reused where practical, but the concepts should remain separate. + +### 7.1 Session Manifest + +Path: + +```text +{workspace.root}/work/{campaign_id}/{session_id}/manifest.json +``` + +The session manifest answers: + +```text +What is the current durable state of this session? +``` + +It should record: + +* campaign ID +* session ID +* current or latest run ID +* current stage states +* canonical durable output refs +* artifact IDs and paths +* producing run ID for each current stage output +* relevant input/config checksums when available +* stale/invalidated stage information +* archive/current publication metadata + +A session's durable state may be a composite of multiple runs. + +For example: + +```text +transcripts/merged.json produced by run A +transcripts/processed.json produced by run B +transcripts/normalized.json produced by run B +transcripts/trimmed.json produced by run B +artifacts/session_recap.md produced by run C +``` + +This is valid and expected. + +### 7.2 Run Manifest + +Path: + +```text +{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json +``` + +The run manifest answers: + +```text +What happened during this specific execution attempt? +``` + +It should record: + +* run ID +* campaign ID +* session ID +* command mode and selected stages +* force flags or stage selection flags +* stages considered during this run +* stages executed during this run +* stages skipped during this run and reasons +* run-local output paths +* promoted output paths +* logs +* reports +* generated configs +* timings +* errors +* non-secret subprocess invocation metadata + +Run manifests are primarily for debugging, auditability, and archive history. + +## 8. Idempotency and Resume Behavior + +The idempotency boundary is the session, not the run. + +By default: + +```bash +narratio run --session-id 2026-05-07 +``` + +should consult the session manifest and skip stages that are already succeeded and not stale. + +If all stages are already complete, the command should execute zero stages and report that the session is already complete. + +Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs. + +Examples: + +```bash +narratio run --session-id 2026-05-07 --force +``` + +Creates a new run and attempts to re-execute the selected/default stage set. + +```bash +narratio run-stage --session-id 2026-05-07 analyze --force +``` + +Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful. + +```bash +narratio resume --session-id 2026-05-07 +``` + +Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run. + +## 9. Artifact Resolution Contract + +Narratio should provide a first-class artifact registry and resolver. + +The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds. + +Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID. + +### 9.1 Canonical Artifact IDs + +Preferred artifact IDs should be namespaced: + +```text +narratio.transcript.merged +narratio.transcript.polished +narratio.transcript.full +narratio.transcript.trimmed +narratio.bounds.session +narratio.artifact.session_recap +``` + +Recommended initial registry: + +| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning | +| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- | +| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. | +| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. | +| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. | +| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. | +| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. | +| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. | + +### 9.2 Backward-Compatible Aliases + +Existing source names should remain supported: + +| Legacy Source | Preferred Artifact ID | +| ----------------------- | ------------------------------ | +| `processed_transcript` | `narratio.transcript.polished` | +| `normalized_transcript` | `narratio.transcript.full` | +| `trimmed_transcript` | `narratio.transcript.trimmed` | + +These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs. + +### 9.3 Resolver Behavior + +Artifact resolution should follow this order: + +1. Normalize aliases to canonical artifact IDs. +2. Look for a current output reference in the session manifest. +3. Fall back to the canonical session-level path. +4. If the artifact is required, fail clearly if missing. +5. If the artifact is optional and missing, omit it from the downstream invocation. +6. Validate the artifact using the expected content validator. +7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance. + +Example conceptual result: + +```json +{ + "id": "narratio.transcript.trimmed", + "path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json", + "producer_stage": "trim", + "producer_run_id": "20260517T174748Z-abcd1234", + "output_kind": "transcript_trimmed", + "content_type": "application/json" +} +``` + +### 9.4 Artifact Validation + +Transcript artifacts must be valid JSON with a top-level `segments` array. + +Markdown/text artifacts must exist and be non-empty when required. + +Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage. + +Validation should happen before a resolved artifact is passed to another stage or external subprocess. + +## 10. Analyze Stage Implications + +The analyze stage should consume artifacts through the artifact resolver. + +Preferred Scriptorium config shape: + +```yaml +scriptorium: + artifacts: + session_recap: + enabled: true + prompt_id: "dnd.session_recap" + output_path: "artifacts/session_recap.md" + inputs: + transcript: + source: "narratio.transcript.trimmed" + required: true +``` + +Additional artifacts can choose different transcript tiers: + +```yaml +scriptorium: + artifacts: + table_summary: + enabled: true + prompt_id: "dnd.table_summary" + output_path: "artifacts/table_summary.md" + inputs: + transcript: + source: "narratio.transcript.full" + required: true +``` + +For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous. + +Rules: + +* Artifact inputs resolve from current session-level durable state. +* Outputs are first written run-locally. +* Successful analyze outputs are promoted to session-level `artifacts/` paths. +* Manifest output refs record the producing run ID. +* Optional inputs are omitted when unavailable. +* Required missing inputs fail before invoking Scriptorium. + +## 11. Archive Alignment + +Local workspace semantics should mirror archive semantics. + +Local session-level durable paths: + +```text +work/{campaign}/{session}/transcripts/trimmed.json +work/{campaign}/{session}/artifacts/session_recap.md +work/{campaign}/{session}/current/manifest.json +work/{campaign}/{session}/current/run_id.txt +work/{campaign}/{session}/runs/{run_id}/... +``` + +should map naturally to remote archive paths: + +```text +{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json +{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md +{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json +{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt +{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/... +``` + +The archive stage should publish run records and promoted current artifacts consistently with the local model. + +`current/run_id.txt` remains the effective commit marker for the archived current session state. + +## 12. Path Helper Requirements + +All code should use centralized path helpers for workspace paths. + +Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model. + +Recommended helper surface: + +```text +SessionRoot(campaignID, sessionID) +SessionManifestPath(campaignID, sessionID) +SessionCurrentDir(campaignID, sessionID) +SessionTranscriptsDir(campaignID, sessionID) +SessionArtifactsDir(campaignID, sessionID) +SessionReportsDir(campaignID, sessionID) +SessionLogsDir(campaignID, sessionID) +SessionConfigDir(campaignID, sessionID) +RunsDir(campaignID, sessionID) +RunRoot(campaignID, sessionID, runID) +RunManifestPath(campaignID, sessionID, runID) +RunStageDir(campaignID, sessionID, runID, stage) +RunStageOutputsDir(campaignID, sessionID, runID, stage) +RunStageLogsDir(campaignID, sessionID, runID, stage) +RunStageReportsDir(campaignID, sessionID, runID, stage) +RunStageConfigDir(campaignID, sessionID, runID, stage) +CanonicalArtifactPath(campaignID, sessionID, artifactID) +``` + +Path helpers should enforce safe relative paths for configured output paths: + +* reject absolute paths unless explicitly allowed for a particular config field +* reject `..` traversal +* normalize separators +* preserve deterministic output paths + +## 13. Directory Creation Policy + +Directory creation should be centralized and idempotent. + +Recommended policy: + +* `prepare` ensures the baseline session directory structure exists. +* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing. +* `run-stage` should not depend on a prior `prepare` invocation merely to create folders. +* Missing directories should be created with appropriate permissions. +* Directory creation should not imply stage success. + +This provides consistent layout while keeping direct stage execution robust. + +## 14. Cleanup and Retention + +Cleanup must preserve the distinction between durable session state and run history. + +Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration. + +Potential retention policies: + +```text +keep_all_runs +keep_failed_runs +keep_last_n_runs +delete_run_after_success +``` + +For v1.0, conservative retention is preferred: + +* Do not delete durable session-level outputs unless explicitly requested. +* Do not delete failed run directories by default. +* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths. +* Local development audio inputs must never be deleted by workspace cleanup. + +## 15. Migration From Existing Layout + +Existing installations may currently use a simpler path such as: + +```text +{workspace.root}/work/{session_id}/manifest.json +``` + +The v1.0 layout introduces campaign-aware session roots: + +```text +{workspace.root}/work/{campaign_id}/{session_id}/manifest.json +``` + +Migration options: + +1. Best-effort automatic discovery of legacy session manifests. +2. A one-time migration command. +3. Clear release notes requiring users to move or regenerate workspace state. + +For v1.0, it is acceptable to require explicit migration if the user base is small and the archive contains the authoritative durable outputs. However, the application should fail clearly when it detects an ambiguous legacy layout rather than silently creating duplicate state. + +## 16. Documentation Updates Required + +The following documentation should be updated to reflect this architecture: + +* `README.md` +* `docs/architecture.md` +* a dedicated workspace/run-history document, such as this file +* S3/archive documentation +* analyze/artifact configuration documentation +* example pipeline files + +Documentation should consistently use the following terms: + +| Term | Meaning | +| ---------------- | ---------------------------------------------------------------------- | +| Session | Durable domain object and idempotency boundary. | +| Run | Execution attempt that may update session state. | +| Durable output | Canonical current session-level output. | +| Run-local output | Output produced inside a specific run directory before promotion. | +| Promotion | Validated copy/rename from run-local output to durable session output. | +| Session manifest | Current durable state of the session. | +| Run manifest | Execution record for a particular run. | +| Artifact ID | Symbolic source name resolved by the artifact registry. | + +## 17. Architectural Invariants + +The following invariants should hold after implementation: + +1. `session_id` remains the idempotency boundary for normal operator commands. +2. `run_id` identifies an execution attempt, not the primary durable workspace. +3. Session-level canonical artifacts are the default inputs for downstream stages. +4. Run-local outputs are promoted only after validation. +5. A session's current durable state may be composed of outputs from multiple runs. +6. Sparse run directories are valid and expected. +7. The session manifest records current stage/artifact state and producer run IDs. +8. The run manifest records what happened during one invocation. +9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver. +10. Local workspace semantics mirror S3 archive semantics. +11. Directory creation is centralized and idempotent. +12. Stage code uses path helpers rather than ad hoc path construction. +13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully. +14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so. + +## 18. Implementation Guidance + +A practical implementation sequence is: + +1. Add this architecture document. +2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths. +3. Introduce session manifest versus run manifest concepts. +4. Route stage outputs through run-local directories. +5. Add promotion helpers with validation and atomic writes. +6. Update existing stages to promote durable outputs to session-level canonical paths. +7. Add artifact registry and resolver. +8. Update analyze to use artifact IDs and aliases. +9. Add simple downstream stale invalidation for forced upstream reruns. +10. Align archive/local path behavior and documentation. +11. Update examples and README. +12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution. + +This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history. diff --git a/docs/roadmap/narratio-s3-archive-implementation-plan.md b/docs/roadmap/narratio-s3-archive-implementation-plan.md deleted file mode 100644 index 5696a56..0000000 --- a/docs/roadmap/narratio-s3-archive-implementation-plan.md +++ /dev/null @@ -1,1063 +0,0 @@ -# Narratio S3 Input and Archive Implementation Plan - -## 1. Purpose - -This document defines the implementation plan for adding S3-based audio input and S3 archival/promotion to `narratio`. - -The feature has two related responsibilities: - -1. **Input acquisition**: load source audio files from an S3 bucket into local working storage before transcription. -2. **Archival and promotion**: after a successful run, upload the complete run record to S3 and promote selected outputs to stable session-level paths. - -The design preserves the existing stage-based architecture: - -```text -prepare -transcribe -merge -polish -normalize -trim -analyze -archive -notify -``` - -The new S3 behavior should fit into the existing modular design: - -- `prepare` acquires input audio. -- intermediate stages operate on the local workdir. -- `archive` uploads successful run outputs and promotes configured artifacts. -- external storage details remain behind a storage backend abstraction. -- failed runs remain local for diagnostics and are not uploaded to S3. - -## 2. Finalized Design Decisions - -The following design choices are settled: - -```text -S3 session prefix: - {root_prefix}/campaigns/{campaign}/sessions/{session_id}/ - -Example: - dnd/campaigns/forsaken/sessions/2026-04-19/ - -run_id format: - 20260515T031522Z-a1b2c3d4 - -local work path: - /var/lib/narratio/work/{campaign}/{session_id}/{run_id}/ - -local spool path: - /var/spool/narratio/{campaign}/{session_id}/{run_id}/audio/ - -S3 audio source: - audio already exists in S3 before narratio runs - -failed runs: - retained locally only; not uploaded to S3 - -archive: - real final pipeline stage - -promotion: - performed during archive stage after all prior required stages succeed - -default promoted outputs: - transcripts/trimmed.json - artifacts/session_recap.md - current/manifest.json - current/run_id.txt - -raw WhisperX transcripts: - uploaded under runs/{run_id}/transcripts/raw/ - -audio re-upload: - original source audio is not re-uploaded by archive by default - -current/run_id.txt: - written last as the effective S3 commit pointer -``` - -### 2.1 Implementation Status (2026-05-16) - -Implemented in repository: - -- storage/archive configuration and validation foundations: - - `storage.s3` - - `spool` - - `archive` - - promotion-rule safety checks - - `inputs.audio_s3` modeling -- run and path-model foundations: - - run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`) - - S3 key builders for session/run/current/promoted destinations - - campaign/session/run local work and spool path helpers - - manifest run/path identity fields -- examples and tests for the above foundations -- session template operator UX: - - default session config discovery (`./session.yml`, `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`) - - `--session-id` template injection for `session_id` - - session template rendering before strict YAML decode - - unresolved template placeholders and `session_id` mismatches fail clearly -- remote storage backend layer: - - object-store abstraction with `List`, `Download`, `Upload`, and `Exists` - - fake storage backend for deterministic, no-network testing - - S3-compatible backend built from `storage.s3` config - - backend construction helper from resolved config -- archive run upload behavior: - - archive validates required prior stage success before uploading - - archive uploads successful run records under `runs/{run_id}/` - - upload set includes run-record files (`inputs`, `transcripts`, `artifacts`, optional `reports`, `config`, `logs`, `manifest.json`) - - local audio is not uploaded by default - - upload uses storage backend abstraction and deterministic ordering - - archive skips cleanly when `archive.enabled` or `archive.upload_run` is false -- archive promotion and current publish behavior: - - promotion rules upload configured outputs to session-level destinations - - required missing promotion sources fail archive - - optional missing promotion sources are skipped and recorded - - `current/manifest.json` is uploaded after run upload and promotions - - `current/run_id.txt` is uploaded last as the effective commit marker - - current pointer content is `{run_id}` plus trailing newline - - if promotion/current manifest upload fails, current pointer is not written -- post-archive local cleanup behavior: - - `pipeline.spool.delete_audio_after_archive` removes run-scoped spool audio only after successful archive commit - - `pipeline.workspace.cleanup_after_archive` removes run-scoped workdir only after successful archive commit - - cleanup is skipped for failed/incomplete/skipped/unarchived runs - -Not implemented yet: - -- `notify` stage behavior -- generic stale detection based on input/config checksums -- optional future mode for uploading source audio from local workspace/spool -- additional artifact generation beyond current implemented set - -## 3. S3 Layout - -The canonical S3 layout should be: - -```text -s3://{bucket}/{root_prefix}/campaigns/{campaign}/sessions/{session_id}/ - audio/ - speaker-1.flac - speaker-2.flac - - transcripts/ - trimmed.json - - artifacts/ - session_recap.md - - current/ - manifest.json - run_id.txt - - runs/ - {run_id}/ - inputs/ - session.yml - speakers.yml - glossary.yml - autocorrect.yml - pipeline.resolved.yml - - transcripts/ - raw/ - speaker-1.json - speaker-2.json - merged.json - processed.json - normalized.json - trimmed.json - - artifacts/ - session_bounds.json - session_recap.md - - reports/ - seriatim.merge.report.json - audita.report.json - seriatim.normalize.report.json - seriatim.trim.report.json - - config/ - seriatim.generated.yml - seriatim.normalize.generated.yml - seriatim.trim.generated.yml - audita.generated.yml - scriptorium.bounds.generated.yml - scriptorium.session_recap.generated.yml - - logs/ - whisperx.*.log - seriatim.*.log - audita.*.log - scriptorium.*.log - - manifest.json -``` - -### 3.1 Session Root - -The session root is: - -```text -{root_prefix}/campaigns/{campaign}/sessions/{session_id}/ -``` - -For example: - -```text -dnd/campaigns/forsaken/sessions/2026-04-19/ -``` - -The `campaigns/` path segment is intentional. It leaves room for future campaign-level material: - -```text -dnd/campaigns/{campaign}/campaign.yml -dnd/campaigns/{campaign}/glossary.yml -dnd/campaigns/{campaign}/characters/ -dnd/campaigns/{campaign}/sessions/ -``` - -### 3.2 Session-Level Paths - -The session-level root contains the durable, promoted, current view of the session: - -```text -audio/ -transcripts/ -artifacts/ -current/ -runs/ -``` - -The top-level `audio/` directory is the source of truth for original session audio. Audio is assumed already present in S3 and should not be re-uploaded by `archive` by default. - -The top-level `transcripts/` and `artifacts/` directories should contain only configured promoted outputs. - -### 3.3 Run-Specific Paths - -Each successful run is uploaded under: - -```text -runs/{run_id}/ -``` - -This contains the full run record: - -- materialized inputs -- intermediate transcripts -- raw WhisperX transcripts -- tool reports -- generated configs -- logs -- generated artifacts -- manifest - -Inputs belong under `runs/{run_id}/inputs/`, not at the session root, because inputs are run-specific. A rerun may use different `speakers.yml`, `glossary.yml`, `autocorrect.yml`, `pipeline.resolved.yml`, Scriptorium prompts, trim settings, models, or runtime configuration. - -### 3.4 Current Pointer - -The effective commit pointer is: - -```text -current/run_id.txt -``` - -This file should be written last during archive. - -`current/manifest.json` should also be written during promotion so consumers can inspect the current promoted run without first resolving the run directory. - -The archive stage should upload in this order: - -1. run record under `runs/{run_id}/` -2. configured promoted outputs under top-level `transcripts/` and `artifacts/` -3. `current/manifest.json` -4. `current/run_id.txt` last - -This makes `current/run_id.txt` the closest practical S3 equivalent of an atomic session commit marker. - -## 4. Local Filesystem Layout - -The production local layout should be: - -```text -/var/lib/narratio/ - work/ - {campaign}/ - {session_id}/ - {run_id}/ - inputs/ - audio/ - transcripts/ - artifacts/ - reports/ - config/ - logs/ - manifest.json - .lock - -/var/spool/narratio/ - {campaign}/ - {session_id}/ - {run_id}/ - audio/ - speaker-1.flac - speaker-2.flac -``` - -For local development, these roots should be configurable. For example: - -```yaml -workspace: - root: ./workspace - -spool: - root: ./spool -``` - -Resulting in: - -```text -./workspace/work/{campaign}/{session_id}/{run_id}/ -./spool/{campaign}/{session_id}/{run_id}/audio/ -``` - -## 5. Local Audio Handling - -The intended local flow is: - -```text -S3 audio - ↓ -spool/audio - ↓ -work/audio - ↓ -transcribe -``` - -Implemented `prepare` behavior: - -1. list `.flac` objects under the configured S3 audio prefix, -2. fail clearly if none are found, -3. download audio files into the spool directory, -4. copy or materialize them into the run workdir `audio/`, -5. record provenance in the manifest. - -The rest of the pipeline should use `work/audio/`, not S3 paths directly and not spool paths. - -Audio cleanup should be conservative in the first implementation. It is acceptable to add a config knob such as: - -```yaml -spool: - delete_audio_after_archive: true -``` - -Current boundary: - -- downloaded audio is retained in spool/workdir -- spool cleanup policy remains future work -- archive does not upload source audio by default - -## 6. Configuration Design - -### 6.1 Storage Config - -Add or refine a storage section: - -```yaml -storage: - s3: - bucket: "my-dnd-archive" - root_prefix: "dnd" - region: "us-east-1" - endpoint: "" - force_path_style: false -``` - -Requirements: - -- `bucket` is required when S3 input/archive is enabled. -- `root_prefix` defaults to `dnd`. -- `region` may be optional depending on SDK behavior. -- `endpoint` is optional for S3-compatible storage. -- `force_path_style` is useful for MinIO/Garage/S3-compatible backends. -- Credentials must not be stored in config. Use standard AWS environment/profile/instance-role mechanisms. - -### 6.2 Workspace and Spool Config - -Use: - -```yaml -workspace: - root: "/var/lib/narratio" - -spool: - root: "/var/spool/narratio" - delete_audio_after_archive: false -``` - -If existing `workspace.root` currently points directly to a work root, the implementation should either preserve the existing semantics or migrate carefully with documentation. The new layout should include campaign/session/run path segments. - -### 6.3 Session Config - -The session config should include campaign and session ID: - -```yaml -session: - id: "2026-04-19" - campaign: "forsaken" -``` - -If the current config shape uses top-level `session_id`, either migrate to the nested shape with compatibility or maintain the current shape while ensuring both campaign and session ID are available to the path builder. - -### 6.4 S3 Audio Input Config - -Use relative audio prefix resolution: - -```yaml -inputs: - audio_s3: - prefix: "audio/" -``` - -This resolves relative to: - -```text -{root_prefix}/campaigns/{campaign}/sessions/{session_id}/ -``` - -For example: - -```text -dnd/campaigns/forsaken/sessions/2026-04-19/audio/ -``` - -The prepare stage should fail if no `.flac` files are found under this prefix. - -Local audio input should continue working for development unless intentionally deprecated later. - -### 6.5 Archive Config - -Add an archive config section: - -```yaml -archive: - enabled: true - upload_run: true - - promote_artifacts: - - from: "transcripts/trimmed.json" - to: "transcripts/trimmed.json" - required: true - - - from: "artifacts/session_recap.md" - to: "artifacts/session_recap.md" - required: true -``` - -If `promote_artifacts` is omitted, use the built-in default list: - -```yaml -archive: - promote_artifacts: - - from: "transcripts/trimmed.json" - to: "transcripts/trimmed.json" - required: true - - - from: "artifacts/session_recap.md" - to: "artifacts/session_recap.md" - required: true -``` - -Additionally, archive should always write: - -```text -current/manifest.json -current/run_id.txt -``` - -Those current-pointer artifacts are part of archive semantics and should not need to be listed in `promote_artifacts`. - -## 7. Run Identity - -`run_id` should be first-class. - -Use the format: - -```text -YYYYMMDDTHHMMSSZ-xxxxxxxx -``` - -Example: - -```text -20260515T031522Z-a1b2c3d4 -``` - -Properties: - -- sortable by timestamp -- human-readable -- collision-resistant via short random suffix -- safe for file paths and S3 keys - -The manifest should include: - -```json -{ - "campaign": "forsaken", - "session_id": "2026-04-19", - "run_id": "20260515T031522Z-a1b2c3d4", - "local_workdir": "/var/lib/narratio/work/forsaken/2026-04-19/20260515T031522Z-a1b2c3d4", - "s3_session_prefix": "dnd/campaigns/forsaken/sessions/2026-04-19/", - "s3_run_prefix": "dnd/campaigns/forsaken/sessions/2026-04-19/runs/20260515T031522Z-a1b2c3d4/" -} -``` - -### 7.1 CLI Behavior - -Recommended behavior: - -```text -run: - creates a new run_id unless --run-id is supplied - -resume: - uses --run-id when supplied - otherwise may discover latest local run for the campaign/session - -run-stage: - uses --run-id when supplied - otherwise may discover latest local run for the campaign/session -``` - -For safety, implementation may choose to require `--run-id` for `resume` and `run-stage` when multiple local runs exist. - -The exact CLI behavior should be documented. - -## 8. Manifest Changes - -Extend manifest data to include: - -- `campaign` -- `session_id` -- `run_id` -- `local_workdir` -- `local_spool_dir` -- `s3_bucket` -- `s3_session_prefix` -- `s3_run_prefix` -- archive status and promoted outputs -- S3 source provenance for audio inputs -- S3 destination records for archived outputs - -Audio input records should include: - -```text -source = s3 -s3_bucket -s3_key -local_path -size -etag -sha256 if computed -``` - -Use ETag as S3 metadata only, not as a reliable checksum. - -SHA-256 should be computed after download if practical. - -## 9. Storage Backend Abstraction - -S3 code should not leak into stages. - -Add or extend a storage backend interface with operations like: - -```text -List(ctx, prefix) ([]ObjectInfo, error) -Download(ctx, key, localPath) error -Upload(ctx, localPath, key, metadata) error -Exists(ctx, key) (bool, error) -``` - -Potential object metadata: - -```text -key -size -etag -last_modified -``` - -A future copy method may be useful, but v1 can upload from local paths. - -The `prepare` stage should use the storage backend to list/download audio. - -The `archive` stage should use the storage backend to upload run records and promoted outputs. - -Tests should use a fake storage backend, not real S3. - -## 10. Prepare Stage Changes - -The `prepare` stage should support both existing local audio workflows and the new S3 audio source. - -### 10.1 S3 Audio Flow - -When `inputs.audio_s3.prefix` is configured: - -1. compute the session root: - ```text - {root_prefix}/campaigns/{campaign}/sessions/{session_id}/ - ``` - -2. resolve the audio prefix: - ```text - {session_root}/{inputs.audio_s3.prefix} - ``` - -3. list objects under that prefix - -4. filter to `.flac` - -5. fail clearly if no `.flac` files are found - -6. download each file to: - ```text - {spool.root}/{campaign}/{session_id}/{run_id}/audio/ - ``` - -7. copy or materialize each file to: - ```text - {workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/ - ``` - -8. compute local checksums if practical - -9. record audio input provenance in manifest - -### 10.2 Local Audio Flow - -Existing local audio flow should continue to work unless intentionally changed later. - -If both local audio and S3 audio are configured, fail clearly unless a precedence rule is explicitly documented. Prefer requiring exactly one audio input source. - -## 11. Archive Stage - -The archive stage becomes a real stage. - -It should run after `analyze` and before `notify`. - -Archive should only upload successful runs. - -Since the normal runner is sequential, if any prior stage fails, archive will not run. If the user invokes `run-stage archive` manually, the archive stage should validate prerequisite stages before uploading. - -### 11.1 Archive Prerequisites - -For v1, require these stages to have succeeded before archive: - -```text -prepare -transcribe -merge -polish -normalize -trim -analyze -``` - -If a stage is optional in a future config, this prerequisite list may become configurable. For now, hardcoded prerequisites are acceptable. - -### 11.2 Run Upload - -Upload the local workdir record to: - -```text -{session_root}/runs/{run_id}/ -``` - -Suggested mapping: - -```text -workdir/inputs/ → runs/{run_id}/inputs/ -workdir/transcripts/ → runs/{run_id}/transcripts/ -workdir/artifacts/ → runs/{run_id}/artifacts/ -workdir/reports/ → runs/{run_id}/reports/ -workdir/config/ → runs/{run_id}/config/ -workdir/logs/ → runs/{run_id}/logs/ -workdir/manifest.json → runs/{run_id}/manifest.json -``` - -If reports currently live under `artifacts/`, implementation may either: - -1. keep that local layout and upload them under `runs/{run_id}/artifacts/`, or -2. add a logical archive mapping into `runs/{run_id}/reports/`. - -Avoid disruptive local layout changes unless they are already easy and well-tested. - -### 11.3 Promotion - -For each configured promotion rule: - -```yaml -- from: "transcripts/trimmed.json" - to: "transcripts/trimmed.json" - required: true -``` - -Upload: - -```text -local workdir/transcripts/trimmed.json - → s3://bucket/{session_root}/transcripts/trimmed.json -``` - -Rules: - -- `from` is local workdir-relative. -- `to` is session-root-relative. -- if `required: true` and the source is missing, archive fails. -- if `required: false` and the source is missing, archive records a skipped promotion. - -### 11.4 Commit Pointer - -Write these last: - -```text -current/manifest.json -current/run_id.txt -``` - -`current/run_id.txt` should contain exactly the run ID plus a trailing newline. - -Writing `current/run_id.txt` last is the effective S3 commit marker. - -### 11.5 Failed Runs - -Failed runs should not be uploaded to S3. - -Failed workdirs should remain local for diagnostics. - -The archive stage should never upload a run that does not satisfy its prerequisite success checks. - -## 12. Audio Upload Policy - -Audio is assumed already present under: - -```text -{session_root}/audio/ -``` - -Archive should not re-upload source audio by default. - -The archive stage may record audio input provenance in manifest, but should avoid duplicating large FLAC files under `runs/{run_id}/`. - -A future option may support uploading local audio into S3, but that is not part of this implementation. - -## 13. Promotion Defaults - -Default promoted outputs: - -```text -transcripts/trimmed.json -artifacts/session_recap.md -``` - -Always write current pointers: - -```text -current/manifest.json -current/run_id.txt -``` - -Do not promote `session_bounds.json` by default. - -Do not promote raw transcripts, logs, tool reports, generated configs, or full intermediate transcript tiers by default. They remain available under `runs/{run_id}/`. - -## 14. Testing Strategy - -Tests should not require real S3. - -Use a fake storage backend for: - -- listing audio objects -- downloading objects -- uploading objects -- recording upload order -- simulating missing objects -- simulating upload failures - -### 14.1 Config Tests - -Test: - -- valid S3 storage config -- missing bucket when S3 mode enabled -- default root prefix -- invalid archive promotion rules -- default promotion list -- audio_s3 prefix validation -- local audio config still works -- conflict when both S3 audio and local audio are configured, if that rule is implemented - -### 14.2 Path Builder Tests - -Test S3 key construction: - -```text -dnd/campaigns/forsaken/sessions/2026-04-19/audio/ -dnd/campaigns/forsaken/sessions/2026-04-19/runs/{run_id}/... -dnd/campaigns/forsaken/sessions/2026-04-19/current/run_id.txt -``` - -Test local paths: - -```text -/var/lib/narratio/work/forsaken/2026-04-19/{run_id}/ -/var/spool/narratio/forsaken/2026-04-19/{run_id}/audio/ -``` - -### 14.3 Prepare Tests - -Test: - -- S3 audio prefix with `.flac` objects downloads files -- no `.flac` objects fails clearly -- non-FLAC objects are ignored -- downloaded audio is materialized in workdir audio -- manifest records S3 provenance -- local audio mode still works -- fake storage errors fail the stage clearly - -### 14.4 Archive Tests - -Test: - -- archive refuses to run if prerequisites are missing or failed -- successful archive uploads run record -- promotion rules upload configured outputs -- default promotion list applies -- optional missing promotion is skipped -- required missing promotion fails -- `current/manifest.json` is uploaded near the end -- `current/run_id.txt` is uploaded last -- failed runs are not uploaded -- audio files are not uploaded by default -- manifest records archive metadata -- fake storage upload failure fails the stage clearly - -### 14.5 CLI/Run-ID Tests - -Test: - -- `run` creates a run ID -- supplied `--run-id` is honored -- `resume` can find or require a run ID according to final CLI policy -- `run-stage` can find or require a run ID according to final CLI policy -- multiple local runs are handled deterministically - -## 15. Implementation Sequence - -### Config and Path Model (Implemented) - -Implement: - -- storage.s3 config -- spool config -- archive config -- promotion rules -- run ID generator -- campaign-aware local work/spool path builder -- S3 session/run key builder - -No real S3 calls yet. - -Expected commit: - -```text -Add archive storage path configuration -``` - -### Storage Backend Interface and S3 Backend (Implemented) - -Implemented: - -- storage backend interface -- object metadata type -- fake backend -- real S3 backend using AWS SDK or existing project dependency policy -- backend construction from config - -No prepare/archive stage behavior yet. - -Expected commit: - -```text -Add S3 storage backend abstraction -``` - -### Prepare Stage S3 Audio Download - -Implement: - -- S3 audio source support -- list/download `.flac` files -- fail on empty audio prefix -- materialize audio into workdir -- manifest S3 provenance -- preserve local audio mode - -Expected commit: - -```text -Download S3 audio during prepare" -``` - -### Real Archive Stage Run Upload - -Implement: - -- archive prerequisites -- upload successful run workdir to `runs/{run_id}/` -- manifest archive metadata -- no promotion yet, or minimal internal scaffolding only - -Expected commit: - -```text -Upload successful run records to S3 -``` - -### Promotion Rules and Current Pointer - -Implement: - -- default promotion rules -- configurable promotion rules -- required/optional behavior -- top-level promoted uploads -- `current/manifest.json` -- `current/run_id.txt` written last - -Expected commit: - -```text -Promote current session artifacts to S3 -``` - -### Documentation and Examples - -Update: - -- architecture.md -- README.md -- examples -- runbook instructions -- config samples -- S3 layout documentation - -Expected commit: - -```text -Document S3 archive workflow -``` - -### Architectural Review - -Review: - -- storage code isolation -- prepare/archive stage boundaries -- no S3 leakage into unrelated stages -- no failed-run upload -- current pointer semantics -- promotion config -- tests -- docs - -Expected commit: - -```text -Review S3 archive architecture -``` - -## 16. Operational Workflow - -Expected production flow: - -1. Upload source audio to: - ```text - s3://{bucket}/dnd/campaigns/{campaign}/sessions/{session_id}/audio/ - ``` - -2. Run narratio: - ```text - narratio run --config pipeline.yml --session sessions/{session_id}/session.yml - ``` - -3. `prepare` downloads audio into spool/workdir. - -4. Pipeline runs locally. - -5. `archive` uploads the successful run record. - -6. `archive` promotes configured current outputs. - -7. `archive` writes `current/manifest.json`. - -8. `archive` writes `current/run_id.txt` last. - -Consumers can then read: - -```text -s3://{bucket}/dnd/campaigns/{campaign}/sessions/{session_id}/current/run_id.txt -s3://{bucket}/dnd/campaigns/{campaign}/sessions/{session_id}/transcripts/trimmed.json -s3://{bucket}/dnd/campaigns/{campaign}/sessions/{session_id}/artifacts/session_recap.md -``` - -## 17. Security and Privacy - -Rules: - -- Do not store AWS credentials in config. -- Use standard AWS credential mechanisms. -- Do not log full environment variables. -- Do not store secrets in manifests or generated configs. -- Treat transcripts and artifacts as potentially sensitive. -- Do not upload failed runs to S3. -- Preserve local failed workdirs for diagnostics. -- Do not re-upload source audio by default. -- Be careful not to log transcript contents during archive. - -## 18. Non-Goals - -Do not implement in this feature: - -- uploading local audio to S3 -- uploading failed runs to S3 -- remote deletion or cleanup policies -- S3 object lifecycle configuration -- remote locking -- multi-user concurrency control -- a database-backed run registry -- a generic artifact publishing framework beyond the configured promotion list -- checksum-based stale detection, except where checksums are recorded as metadata -- archive-time redaction of manifests; secrets should not enter manifests in the first place - -## 19. Open Follow-Up Ideas - -Potential future improvements: - -- local retention policy for successful workdirs -- optional cleanup of spool audio after successful archive -- optional upload of human-readable transcript exports -- optional promotion of `session_bounds.json` -- S3-side run index by date/model/prompt version -- S3 object metadata for checksums and content types -- remote run discovery for `resume` -- support for local-audio-to-S3 ingestion mode -- optional failed-run diagnostic upload behind an explicit flag -- checksum-based stale detection and stage invalidation -- archive verification pass after upload diff --git a/docs/runbooks/s3-archive-foundations.md b/docs/runbooks/s3-archive-foundations.md deleted file mode 100644 index 73babaa..0000000 --- a/docs/runbooks/s3-archive-foundations.md +++ /dev/null @@ -1,35 +0,0 @@ -# S3 Archive Foundations Runbook - -This runbook documents the currently implemented storage/archive foundations and the boundaries of current behavior. - -## Implemented Now - -- config modeling for: - - `pipeline.storage.s3` - - `pipeline.spool` - - `pipeline.archive` - - `session.inputs.audio_s3` -- promotion rule validation for safe relative paths -- run ID generation and path/key helper functions -- manifest run/path identity fields -- remote storage backend layer: - - object-store interface (`List`, `Download`, `Upload`, `Exists`) - - fake backend for deterministic tests - - S3-compatible backend using AWS SDK v2 - - config-based backend construction helper - -## Not Implemented Yet - -- prepare-stage S3 object listing or download -- archive-stage S3 upload or promotion writes -- writing `current/manifest.json` or `current/run_id.txt` in S3 - -## Operational Notes - -- local audio workflows remain the active development path (`audio_dir` or `audio_files`) -- `audio_s3` and local audio config are mutually exclusive -- do not place AWS credentials in Narratio config files - -## Next Implementation Target - -Use the storage backend layer in prepare-stage session audio discovery/download flow, while preserving local audio input support. diff --git a/docs/archive-storage.md b/docs/stages/archive.md similarity index 100% rename from docs/archive-storage.md rename to docs/stages/archive.md diff --git a/docs/s3-audio-input.md b/docs/stages/prepare.md similarity index 100% rename from docs/s3-audio-input.md rename to docs/stages/prepare.md diff --git a/docs/storage-backends.md b/docs/storage-backends.md deleted file mode 100644 index 9201551..0000000 --- a/docs/storage-backends.md +++ /dev/null @@ -1,62 +0,0 @@ -# Storage Backends - -This document describes the currently implemented remote object storage backend layer used by Narratio, and its intended role in later prepare/archive work. - -## Implemented - -Remote object store abstraction: - -- `List(ctx, prefix)` -- `Download(ctx, key, localPath)` -- `Upload(ctx, localPath, key, opts)` -- `Exists(ctx, key)` - -Object metadata model includes: - -- key -- size -- ETag (provider metadata only) -- last modified time when available - -Backends: - -- fake storage backend for deterministic tests -- S3-compatible backend implemented with AWS SDK for Go v2 - -Construction: - -- config-based constructor builds S3 backend from `pipeline.storage.s3` values: - - bucket - - region - - endpoint - - force_path_style - - access_key_id_env - - secret_access_key_env - -## Key Invariant - -- callers pass full bucket-relative object keys -- storage backends do not prepend `root_prefix` -- storage backends do not infer campaign/session/run paths - -S3 session/run key builders remain separate and continue to live outside backend implementations. - -## Security Boundary - -- do not store AWS credentials in Narratio config -- Narratio first checks configured env-var names (`access_key_id_env`, `secret_access_key_env`); - when both are present and non-empty, it uses static credentials from those values -- when either configured credential value is missing, Narratio falls back to the standard AWS SDK credential chain -- AWS SDK-specific types remain isolated to the storage adapter package - -## Testing - -- fake storage tests cover list/download/upload/exists and error paths -- S3 backend tests use injected fake S3 API clients -- tests do not require live S3 services, AWS credentials, or network access - -## Not Implemented Yet - -- prepare-stage S3 object listing or downloads -- archive-stage S3 uploads or promotion writes -- writing `current/manifest.json` or `current/run_id.txt` to S3