Documentation update and reorganization

docs/stages/archive.md

@@ -0,0 +1,119 @@
+# Archive Storage
+
+This document describes implemented archive-stage publish behavior.
+
+## S3 Paths
+
+Session root:
+
+`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
+
+Run prefix:
+
+`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
+
+## Scope
+
+Implemented:
+
+- archive uploads successful run records to remote object storage through the storage backend abstraction.
+- archive uploads configured promoted outputs to session-level keys.
+- archive uploads `current/manifest.json`.
+- archive uploads `current/run_id.txt` last as the effective commit marker.
+- optional post-archive local cleanup:
+  - `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
+  - `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
+- tests use fake storage and do not require live S3.
+
+Future work:
+
+- `notify` stage behavior
+- stale detection
+- optional future source-audio upload mode
+- additional artifact generation beyond current implemented set
+
+## Prerequisites
+
+Archive verifies these stages succeeded before upload:
+
+- `prepare`
+- `transcribe`
+- `merge`
+- `polish`
+- `normalize`
+- `trim`
+- `analyze`
+
+If any prerequisite is missing or not succeeded, archive fails and does not upload.
+Failed or incomplete runs remain local only.
+
+## Run Upload
+
+Archive uploads existing files from the run workdir when present:
+
+- `inputs/`
+- `transcripts/`
+- `artifacts/`
+- `reports/` (optional)
+- `config/`
+- `logs/`
+- `manifest.json`
+
+Relative paths are preserved under `runs/{run_id}/`.
+
+## Promotion Rules
+
+Archive applies `archive.promote_artifacts` in config order.
+
+Rule behavior:
+
+- `from`: local workdir-relative source path
+- `to`: session-root-relative destination key
+- `required: true`: missing source fails archive
+- `required: false`: missing source is skipped and recorded
+
+Default promoted outputs:
+
+- `transcripts/trimmed.json`
+- `artifacts/session_recap.md`
+
+## Current Pointers
+
+Archive writes:
+
+1. `current/manifest.json` (after run upload + promotions)
+2. `current/run_id.txt` last
+
+`current/run_id.txt` contains exactly:
+
+- `{run_id}` plus trailing newline
+
+Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
+
+If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
+Cleanup runs only after this commit-marker write has succeeded.
+
+## Audio Upload Policy
+
+Archive does not upload local `audio/` by default.
+Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
+
+## Config Controls
+
+- `archive.enabled: false` skips archive cleanly.
+- `archive.upload_run: false` skips run upload cleanly.
+- both skip cases also skip post-archive local cleanup.
+
+## Metadata
+
+Archive stage metadata includes non-secret upload context (for example):
+
+- `s3_bucket`
+- `s3_run_prefix`
+- run upload counts/paths
+- promoted upload counts/paths
+- skipped optional promotions
+- `current_manifest_key`
+- `current_run_id_key`
+- `current_pointer_written`
+- `audio_upload_skipped`