Documentation update and reorganization

docs/stages/archive.md

@@ -0,0 +1,119 @@
+# Archive Storage
+
+This document describes implemented archive-stage publish behavior.
+
+## S3 Paths
+
+Session root:
+
+`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
+
+Run prefix:
+
+`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
+
+## Scope
+
+Implemented:
+
+- archive uploads successful run records to remote object storage through the storage backend abstraction.
+- archive uploads configured promoted outputs to session-level keys.
+- archive uploads `current/manifest.json`.
+- archive uploads `current/run_id.txt` last as the effective commit marker.
+- optional post-archive local cleanup:
+  - `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
+  - `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
+- tests use fake storage and do not require live S3.
+
+Future work:
+
+- `notify` stage behavior
+- stale detection
+- optional future source-audio upload mode
+- additional artifact generation beyond current implemented set
+
+## Prerequisites
+
+Archive verifies these stages succeeded before upload:
+
+- `prepare`
+- `transcribe`
+- `merge`
+- `polish`
+- `normalize`
+- `trim`
+- `analyze`
+
+If any prerequisite is missing or not succeeded, archive fails and does not upload.
+Failed or incomplete runs remain local only.
+
+## Run Upload
+
+Archive uploads existing files from the run workdir when present:
+
+- `inputs/`
+- `transcripts/`
+- `artifacts/`
+- `reports/` (optional)
+- `config/`
+- `logs/`
+- `manifest.json`
+
+Relative paths are preserved under `runs/{run_id}/`.
+
+## Promotion Rules
+
+Archive applies `archive.promote_artifacts` in config order.
+
+Rule behavior:
+
+- `from`: local workdir-relative source path
+- `to`: session-root-relative destination key
+- `required: true`: missing source fails archive
+- `required: false`: missing source is skipped and recorded
+
+Default promoted outputs:
+
+- `transcripts/trimmed.json`
+- `artifacts/session_recap.md`
+
+## Current Pointers
+
+Archive writes:
+
+1. `current/manifest.json` (after run upload + promotions)
+2. `current/run_id.txt` last
+
+`current/run_id.txt` contains exactly:
+
+- `{run_id}` plus trailing newline
+
+Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
+
+If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
+Cleanup runs only after this commit-marker write has succeeded.
+
+## Audio Upload Policy
+
+Archive does not upload local `audio/` by default.
+Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
+
+## Config Controls
+
+- `archive.enabled: false` skips archive cleanly.
+- `archive.upload_run: false` skips run upload cleanly.
+- both skip cases also skip post-archive local cleanup.
+
+## Metadata
+
+Archive stage metadata includes non-secret upload context (for example):
+
+- `s3_bucket`
+- `s3_run_prefix`
+- run upload counts/paths
+- promoted upload counts/paths
+- skipped optional promotions
+- `current_manifest_key`
+- `current_run_id_key`
+- `current_pointer_written`
+- `audio_upload_skipped`

docs/stages/prepare.md

@@ -0,0 +1,84 @@
+# S3 Audio Input
+
+This document describes implemented S3 audio input behavior in `prepare`.
+
+## Scope
+
+Implemented:
+
+- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
+- object listing and download go through the storage backend abstraction.
+- tests use fake storage; no live S3 service is required for test runs.
+
+Not implemented:
+
+- uploads of failed runs
+
+## Required Configuration
+
+`pipeline.yml`:
+
+- `storage.s3.bucket` must be set when S3 audio input is used.
+- `storage.s3.root_prefix` defaults to `dnd`.
+- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
+- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
+- `spool.root` defaults to `/var/spool/narratio`.
+
+`session.yml`:
+
+- configure `session.campaign` and `session.session_id`.
+- configure `session.inputs.audio_s3.prefix` for S3 audio input.
+- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
+
+## Prefix Shape
+
+Session S3 root:
+
+`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
+
+Audio prefix:
+
+`{session_root}/{audio_s3.prefix}`
+
+Example:
+
+`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
+
+Audio files must already exist in S3 before running Narratio.
+
+## Prepare Behavior
+
+When `inputs.audio_s3.prefix` is configured, `prepare`:
+
+1. lists objects under the computed S3 audio prefix
+2. filters to `.flac` objects
+3. fails when no `.flac` objects are found
+4. downloads selected objects to spool audio:
+   - `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
+5. materializes audio into workdir audio:
+   - `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
+6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
+
+Notes:
+
+- `.flac` filtering is case-insensitive.
+- ETag is recorded as provider metadata only and is not treated as a checksum.
+
+## Local Audio Development
+
+Local audio workflows remain supported:
+
+- `inputs.audio_dir`
+- `inputs.audio_files`
+
+These options are mutually exclusive with `inputs.audio_s3`.
+
+## Archive Boundary
+
+Current archive behavior relevant to S3 audio input:
+
+- successful runs are uploaded by archive under `runs/{run_id}/`
+- configured promotions are uploaded to session-level destinations
+- `current/manifest.json` and `current/run_id.txt` are published
+- local source audio is not re-uploaded by default
+- failed or incomplete runs are not uploaded