Initial documentation cleanup pass

docs/policy/architecture.md

@@ -0,0 +1,202 @@
+# Narratio Architecture
+
+## Purpose
+
+`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
+
+This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
+
+Implemented component details belong under `docs/internal/`.
+
+## Project Shape
+
+Narratio is a modular, stage-driven orchestrator.
+
+It coordinates specialized downstream systems rather than reimplementing their domains:
+
+- WhisperX handles transcription.
+- Seriatim handles deterministic transcript merge/normalization/trim behavior.
+- Audita handles transcript correction and polishing.
+- Scriptorium handles prompt execution and generated artifacts.
+
+Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
+
+Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
+
+## Core Principles
+
+### Modular and composable
+
+Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
+
+### Hexagonal boundaries
+
+External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
+
+### Standard library preference
+
+Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
+
+Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
+
+### Explicit orchestration
+
+The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
+
+## Stage Design
+
+Each stage should have a clear scope of responsibility.
+
+A stage should define:
+
+- its purpose;
+- required input state;
+- produced output state;
+- config fields it consumes;
+- external adapters it uses;
+- manifest refs it reads or writes;
+- skip, force, and resume behavior;
+- failure behavior;
+- tests that protect its contract.
+
+Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
+
+## Transactionality and Resume
+
+A stage should behave transactionally.
+
+A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
+
+A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
+
+## Manifest Model
+
+The manifest is the durable local ledger for a run.
+
+It should record:
+
+- run identity;
+- stage status;
+- input and output refs;
+- logs and generated config refs;
+- checksums or provenance where useful;
+- non-secret adapter and publish metadata.
+
+Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
+
+## Adapter Boundaries
+
+Adapters own external integration details.
+
+Expected boundaries:
+
+- WhisperX HTTP details stay in the WhisperX adapter.
+- Seriatim CLI construction stays in the Seriatim adapter.
+- Audita CLI construction stays in the Audita adapter.
+- Scriptorium CLI construction stays in the Scriptorium adapter.
+- Object-storage details stay behind the storage adapter interface.
+- AWS SDK types stay inside the S3 storage implementation.
+
+Stage code should express intent in Narratio terms and call adapters through narrow contracts.
+
+## Configuration Philosophy
+
+Configuration should be strict, explicit, and operator-friendly.
+
+Principles:
+
+- YAML decoding should reject unknown fields.
+- Defaults should be centralized and testable.
+- Empty configured values should not silently override meaningful defaults.
+- Session templating should remain narrow and deterministic.
+- Template support should serve operator convenience, not become a general configuration language.
+
+Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
+
+## Path and Storage Discipline
+
+Local and remote paths are part of Narratio’s application contract.
+
+Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
+
+Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
+
+## Publish Invariants
+
+Publish behavior must preserve a clear commit boundary.
+
+A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
+
+`current/run_id.txt` is the final remote commit marker and must be written last.
+
+Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
+
+## Security and Privacy
+
+Narratio handles private campaign material.
+
+Rules:
+
+- Do not store raw secrets in pipeline or session YAML.
+- Use environment variable names or secret-file references for secret handling.
+- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
+- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
+- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
+
+## Diagnostics
+
+Diagnostics should be durable and discoverable, but distinct from canonical outputs.
+
+Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
+
+Manifest refs should preserve that distinction.
+
+## Determinism
+
+Where practical, Narratio should prefer deterministic behavior:
+
+- stable local path layout;
+- stable remote key layout;
+- sorted upload order;
+- predictable generated config files;
+- repeatable command construction;
+- tests that do not depend on live external services.
+
+Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
+
+## Testing Expectations
+
+Core behavior should be testable without live external services.
+
+Tests should cover:
+
+- config loading, defaults, and validation;
+- CLI parsing and command construction;
+- path helpers;
+- manifest transitions;
+- stage success, failure, skip, and resume behavior;
+- adapter command construction;
+- fake storage behavior;
+- publish commit ordering;
+- example config validity where practical.
+
+Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
+
+## Documentation Expectations
+
+Documentation must follow `docs/documentation/policy.md`.
+
+Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
+
+`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
+
+## Non-Goals
+
+Narratio is not:
+
+- a generic DAG or workflow engine;
+- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
+- a storage backend abstraction beyond the needs of this pipeline;
+- a place to embed raw secrets;
+- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
+- a prompt-authoring system.