diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 923458e..74af1ff 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -1,202 +1,230 @@ -# Narratio Architecture +# Architecture -## Purpose +This document defines Narratio's intended high-level architecture and the +invariants that changes must preserve. Implemented component details belong in +the [Internal Overview](../internal/overview.md) and its linked documents. +Significant architectural decision history belongs under `docs/adr/` when such +records exist. -`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts. +## System Shape -This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference. +Narratio is a small Go application that turns D&D session audio into polished +transcripts and generated session artifacts. It is an explicit, stage-driven +orchestrator, not a general workflow engine. -Implemented component details belong under `docs/internal/`. +Narratio coordinates specialized external systems rather than reimplementing +their domains: -## Project Shape +- WhisperX performs transcription; +- Seriatim performs deterministic transcript processing and rendering; +- Audita performs transcript correction and polishing; and +- Scriptorium executes prompts and produces configured artifacts. -Narratio is a modular, stage-driven orchestrator. +Narratio owns orchestration, configuration resolution, session and run state, +artifact and path modeling, manifest persistence, stage sequencing, resume, +restore, cleanup gates, and publish semantics. External contracts are defined +in the [integration documentation](../integrations/). -It coordinates specialized downstream systems rather than reimplementing their domains: +The pipeline has one canonical ordered stage set. Configuration may enable, +disable, or parameterize supported behavior, but it must not turn that sequence +into an arbitrary DAG or hide orchestration in generic workflow abstractions. +The implemented stage inventory belongs in the +[Internal Overview](../internal/overview.md). -- WhisperX handles transcription. -- Seriatim handles deterministic transcript merge/normalization/trim behavior. -- Audita handles transcript correction and polishing. -- Scriptorium handles prompt execution and generated artifacts. +Narratio is contract-first without being abstraction-heavy. Interfaces and +extension points should protect demonstrated boundaries. New abstraction is not +itself an architectural goal. -Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics. +## Ownership And Dependency Direction -Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine. +The application boundary owns command dispatch, configuration selection, +production composition, session locking, and top-level lifecycle. It may depend +on concrete implementations to assemble a run. -## Core Principles +Stage orchestration expresses intent in Narratio-level data and interfaces. +Stages may depend on configuration, manifest, artifact, path, and adapter +contracts, but they must not depend on transport-specific request types, +subprocess argument construction, cloud SDK types, or downstream tool internals. -### Modular and composable +Adapters translate between Narratio contracts and external systems. They own +HTTP, subprocess, notification, and object-storage mechanics, including command +construction, transport behavior, provider response handling, and external +error adaptation. External dependency types must remain inside the adapter that +owns them unless that dependency is the adapter's explicit public contract. +WhisperX HTTP behavior, Seriatim, Audita, and Scriptorium command construction, +notification transport, and object-storage SDK details remain behind these +boundaries. -Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable. +State and path services must not infer stage policy. Storage implementations +receive explicit bucket-relative keys and do not infer campaign, session, run, +or root-prefix semantics. Manifest persistence records transitions but does not +choose orchestration policy. Artifact resolution identifies and validates +artifacts but does not execute producers. -### Hexagonal boundaries +Dependencies should remain narrow and point toward Narratio-owned contracts. +Prefer the Go standard library. Add an external dependency only when it provides +a clear correctness, security, interoperability, or complexity benefit, and +confine it to the boundary that needs it. -External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details. +## Stage Boundaries -### Standard library preference +Each stage has one explicit responsibility and declares: -Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard. - -Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage. - -### Explicit orchestration - -The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction. - -## Stage Design - -Each stage should have a clear scope of responsibility. - -A stage should define: - -- its purpose; - required input state; - produced output state; -- config fields it consumes; +- configuration it consumes; - external adapters it uses; -- manifest refs it reads or writes; -- skip, force, and resume behavior; -- failure behavior; -- tests that protect its contract. +- manifest references and metadata it reads or writes; +- skip, force, invalidation, and resume behavior; and +- failure behavior. -Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages. +Stages write and validate run-local results before materializing canonical +outputs where that distinction applies. A stage is complete only after its +required outputs have been written, validated, and recorded in durable manifest +state. Later stages depend on recorded success and artifact resolution, not +merely on incidental files existing on disk. -## Transactionality and Resume +A failed or interrupted stage must not be presented as successful. Failure +should preserve enough local state and diagnostics for inspection, recovery, +and resume. Forcing an upstream stage invalidates succeeded downstream work +according to the canonical stage order. -A stage should behave transactionally. +Shared behavior should live behind a narrow service or helper with one clear +owner. Stages must not reach across boundaries or reproduce adapter, manifest, +artifact, or path policy ad hoc. -A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume. +## Manifest, Resume, And Restore -A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk. +The session manifest is the durable ledger for progress across invocations. It +records session and run identity, stage state, input and output references, +diagnostic references, checksums or provenance where useful, and non-secret +adapter and publish metadata. -## Manifest Model +Resume and skip decisions are manifest-driven. Filesystem state may be +inspected and validated, but file presence alone does not replace recorded +stage state. Invocation-scoped run records provide an audit of one execution; +they do not replace the session manifest as progress authority. -The manifest is the durable local ledger for a run. +Restore treats committed remote current state as its authority. It must plan +deterministically, confine remote-to-local paths, protect local conflicts, and +install the validated session manifest after other restored durable files. The +physical workflow and recovery procedures belong in +[Operations](../operations.md). -It should record: +## Configuration -- run identity; -- stage status; -- input and output refs; -- logs and generated config refs; -- checksums or provenance where useful; -- non-secret adapter and publish metadata. +Configuration is strict, explicit, centralized, and operator-oriented. -Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress. +- YAML decoding rejects unknown fields. +- Defaults are centralized and testable. +- Empty configured values do not silently replace meaningful defaults. +- Validation rejects invalid composition before stage execution where + practical. +- Session templating remains narrow and deterministic rather than becoming a + general configuration language. +- Secret values are supplied indirectly and are not persisted in ordinary + configuration. -## Adapter Boundaries +Narratio must not become a second configuration system for downstream tools. +External systems own their runtime defaults wherever practical; Narratio passes +the paths required by its stage contracts and explicit operator overrides. The +field-level contract and credential-supply mechanisms belong in +[Configuration](../config.md). -Adapters own external integration details. +## Artifacts, Paths, And Storage -Expected boundaries: +Artifact identities and local and remote paths are application contracts. +Canonical helpers own workspace, spool, cache, session, run, input, transcript, +artifact, log, report, configuration, and publish-current paths. Callers must +not reconstruct canonical paths through scattered string concatenation. -- WhisperX HTTP details stay in the WhisperX adapter. -- Seriatim CLI construction stays in the Seriatim adapter. -- Audita CLI construction stays in the Audita adapter. -- Scriptorium CLI construction stays in the Scriptorium adapter. -- Object-storage details stay behind the storage adapter interface. -- AWS SDK types stay inside the S3 storage implementation. +Artifact resolution is deterministic and manifest-aware. Producers materialize +canonical outputs before reporting success, and consumers resolve declared +artifact identities rather than infer files from unrelated directory contents. -Stage code should express intent in Narratio terms and call adapters through narrow contracts. +Writes, moves, replacements, and deletions must use narrow, explicit, +root-confined destinations. Symlinks, traversal, broad roots, and ambiguous +relative destinations must not expand the scope of an operation. Cleanup is +permitted only through explicit operator action or configured post-publish +gates, and it must preserve durable cache unless cache removal is explicitly +requested. -## Configuration Philosophy +Physical layout, retention, and operational lifecycle belong in +[Operations](../operations.md). Logical external formats and durable integration +contracts belong under [Integrations](../integrations/). -Configuration should be strict, explicit, and operator-friendly. +## Publish Commit Boundary -Principles: +Publish has one explicit remote commit boundary. A remote run becomes current +only after Narratio has successfully uploaded the run record, required published +outputs, `current/manifest.json`, and finally `current/run_id.txt`. -- YAML decoding should reject unknown fields. -- Defaults should be centralized and testable. -- Empty configured values should not silently override meaningful defaults. -- Session templating should remain narrow and deterministic. -- Template support should serve operator convenience, not become a general configuration language. +`current/run_id.txt` is the commit marker and must be written last. Failed, +incomplete, skipped, or uncommitted publish attempts must not be presented as +current remote state. Publish locks remain authoritative and are not bypassed by +a forced run. -Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides. +Automatic local cleanup is permitted only after a successful publish commit, +only when explicitly configured, and only through the path-safety guardrails. -## Path and Storage Discipline +## Security, Privacy, And Diagnostics -Local and remote paths are part of Narratio’s application contract. +Narratio handles private campaign material. Transcripts, prompts, generated +artifacts, reports, logs, manifests, and diagnostic files are potentially +sensitive. -Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation. +Raw secrets must not be stored in pipeline, campaign, or session YAML or written +to manifests, logs, generated configuration, reports, publish metadata, +documentation, or examples. Secrets enter through configured environment +variable names or secret-file references. Diagnostics should avoid transcript +and prompt content unless a deliberate, bounded inspection mechanism requires +it. -Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics. +Logs, reports, generated invocation files, generated configuration, and render +debug files are diagnostics, not canonical pipeline products. They should be +durable and discoverable where configured, and manifest references must preserve +the distinction between diagnostics and artifacts. -## Publish Invariants +Documentation security rules belong in the +[Documentation Policy](documentation.md). Credential supply belongs in +[Configuration](../config.md), while permissions, sensitive runtime-artifact +handling, and recovery belong in [Operations](../operations.md). -Publish behavior must preserve a clear commit boundary. +## Determinism And Testability -A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`. +Narratio prefers deterministic behavior where practical, including stable local +and remote layouts, sorted operation order, predictable generated +configuration, repeatable command construction, deterministic artifact +resolution, and reproducible planning. -`current/run_id.txt` is the final remote commit marker and must be written last. +Run IDs and timestamps may be intentionally variable, but surrounding behavior +must remain controllable in tests. Core behavior should be testable without live +external services; expensive, nondeterministic, destructive, or external +boundaries should be replaceable with focused test doubles. General testing +philosophy and sufficiency rules belong in the [Testing Policy](testing.md). -Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured. +## Documentation And Decision Records -## Security and Privacy +Documentation follows the [Documentation Policy](documentation.md). Current +behavior belongs in its canonical user, operator, integration, architecture, or +internal owner. Proposed behavior and implementation status belong under +`docs/roadmap/`. -Narratio handles private campaign material. +Significant architectural decisions may be recorded under `docs/adr/` using the +format and lifecycle defined by the documentation policy. ADR acceptance does +not establish that a decision has been implemented. -Rules: +## Architectural Non-Goals -- Do not store raw secrets in pipeline or session YAML. -- Use environment variable names or secret-file references for secret handling. -- Do not write raw secret values to manifests, logs, generated configs, or publish metadata. -- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive. -- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason. - -## Diagnostics - -Diagnostics should be durable and discoverable, but distinct from canonical outputs. - -Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products. - -Manifest refs should preserve that distinction. - -## Determinism - -Where practical, Narratio should prefer deterministic behavior: - -- stable local path layout; -- stable remote key layout; -- sorted upload order; -- predictable generated config files; -- repeatable command construction; -- tests that do not depend on live external services. - -Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable. - -## Testing Expectations - -Core behavior should be testable without live external services. - -Tests should cover: - -- config loading, defaults, and validation; -- CLI parsing and command construction; -- path helpers; -- manifest transitions; -- stage success, failure, skip, and resume behavior; -- adapter command construction; -- fake storage behavior; -- publish commit ordering; -- example config validity where practical. - -Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs. - -## Documentation Expectations - -Documentation must follow `docs/documentation/policy.md`. - -Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`. - -`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation. - -## Non-Goals - -Narratio is not: +Narratio does not aim to provide: - a generic DAG or workflow engine; -- a replacement configuration layer for Seriatim, Audita, or Scriptorium; -- a storage backend abstraction beyond the needs of this pipeline; -- a place to embed raw secrets; -- a place for stage logic to depend directly on AWS SDK types or downstream tool internals; +- a replacement configuration layer for WhisperX, Seriatim, Audita, + Scriptorium, or other downstream tools; +- a storage abstraction broader than the needs of this pipeline; +- stage logic coupled directly to cloud SDKs, transports, subprocess details, + or downstream implementation internals; +- raw-secret persistence; +- implicit cross-stage behavior that bypasses manifest and artifact contracts; + or - a prompt-authoring system.