# Architecture This document defines Notarius development policy. It is inward-facing: developers and LLM coding agents should use it to preserve the project's shape, boundaries, and invariants as the code evolves. Keep this document concise. It should describe durable architectural rules, not CLI syntax, configuration reference material, module catalogs, or roadmap items. ## Project Shape Notarius is a small, explicit, dependency-light Go application for extracting structured artifacts from source material using modular pipeline stages. The application is contract-first but not abstraction-heavy. Add interfaces and extension points when they protect a real boundary: - external source formats; - pipeline stage modules; - validators; - LLM providers and runtime plumbing; - output schemas and embedded assets. Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow contracts that can be exercised by tests and real modules. ## Core Invariants The framework must remain source-agnostic and domain-agnostic. Source-format details belong in input modules. Transcript-specific concepts such as segments, speakers, timestamps, and transcript schemas must not spread into runner, extractor, validator, or LLM framework code. Extraction-domain details belong in domain modules. D&D-specific concepts such as spells, NPCs, items, combat turns, and encounters must not spread into core source, runner, or LLM framework packages. Extracted facts should be grounded with source references. Source references should point to generic source units, not transcript-only structures. Framework code should preserve source-reference ranges exactly and should not merge or rewrite overlapping ranges unless a module explicitly owns that behavior. The application workflow is fixed: ```text input -> chunk -> extract -> merge -> normalize -> output ``` These stages should remain explicit in the architecture. Chunking, merging, and normalization must not be hidden inside domain extractors when they represent general pipeline behavior. Pipelines are fixed-shape templates for this workflow, not arbitrary DAGs or a general workflow language. Module selection should be configuration- and registry-driven, not scattered through conditionals. ## Package Boundaries Prefer fewer, larger framework packages until a boundary proves itself through import direction, ownership, test seams, or substantial file size. Core packages should contain deterministic models and policy. Framework packages should contain reusable orchestration and provider plumbing. Concrete business logic should live under stage-oriented module packages: ```text internal/modules/input/... internal/modules/chunk/... internal/modules/extract/... internal/modules/merge/... internal/modules/normalize/... internal/modules/output/... ``` Use short, lowercase, idiomatic Go package names. Avoid package names that repeat parent-stage context. Input modules translate external source formats into the core source model. They may know about external schema details, source-specific metadata, and format-specific validation rules. They should not own extraction-domain decisions. Extract modules own artifact semantics, prompt usage, structured response schema selection, and domain-specific interpretation. They should depend on framework contracts and core source/artifact types, not concrete input module packages. Production validation defaults are central catalog policy, not behavior owned by module packages. Merge modules combine extracted candidates. Normalize modules reconcile merged candidates for semantic consistency. Generic behavior may exist for simple artifact types, but domain-specific behavior belongs in modules for the relevant stage. Output modules serialize final artifacts and may report warnings out of band. CLI, diagnostics, and reporting layers are responsible for surfacing those warnings. ## Validation Validators should be independently testable and composable. Validators evaluate immutable module outputs returned by `chunk`, `extract`, `merge`, and `normalize` stages. Validator decision semantics should be explicit: each validator call approves, rejects, or approves with warnings for the whole module output it receives. Validator rejection records rejected raw output; validator execution errors are framework errors. Default validator chains belong in central production catalog mappings keyed by stage and module key. Pipeline configuration may override those mappings at the stage-local module binding. Empty chains are valid and approve by default. Deterministic validators should run before LLM-backed validators in production defaults when both are present. Configured validator order is authoritative and must not be silently reordered. Shared validator runtime mechanics belong in framework code. Concrete validator behavior belongs in module or validator implementation packages. ## LLM Runtime LLM provider details belong behind transport-neutral framework contracts. Provider-specific HTTP request and response types should stay inside the LLM runtime package. Prompt construction should stay in extractors, validators, or shared prompt helpers; provider adapters should not own domain prompt logic. Errors, diagnostics, reports, manifests, and redacted configuration must not expose secrets. ## Configuration Configuration should make pipeline composition explicit and discoverable. Centralize configuration loading, precedence, defaults, and validation. Structural pipeline choices should come from named pipeline definitions, not ad hoc command flags. Operational overrides may be handled separately when they do not obscure the configured pipeline structure. Module registries should expose module metadata and capabilities without requiring module construction. Configuration validation should fail fast when a pipeline binds incompatible or unknown modules. Run manifests should record enough resolved pipeline provenance to make a run auditable after named configuration changes over time. ## Dependencies Prefer the Go standard library where practical. Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include widely used file formats, complex validation behavior, or secure transport handling. Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit contract of that package. ## State, Files, and Safety If the application writes durable state, writes should be atomic where practical. Multi-step workflows should preserve enough diagnostics to support inspection after failure. Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in. ## Errors and Logging Errors should be actionable and preserve context. Wrap errors with operation and path or resource context. CLI code should convert internal errors into concise user-facing messages. Errors and logs must not expose secrets. Logs should describe operations, external calls, retries, and failure causes, but should not include large source or artifact payloads by default. Long-running operations should accept `context.Context`. External calls, subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage workflows should respect cancellation and timeouts. ## Testing Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for input modules, extract modules, validators, and LLM clients where practical. Contract-first work should include fake implementations that prove interfaces compose before real modules depend on them. Maintain a fixture-driven walking skeleton that exercises the full pipeline with fake modules and fake external clients. This protects stage composition as real modules evolve. Important CLI and configuration workflows should have tests. Adapter, extractor, validator, and stage contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional. ## Documentation Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`. Core documentation should use generic terms such as source document, source unit, source reference, input adapter, extractor, chunker, merger, normalizer, artifact, validator, and run manifest. Source-format details belong in input module or integration docs. Domain-specific extraction details belong in extract module or artifact docs. When changing architecture, config, CLI behavior, stage modules, extractor contracts, validator contracts, LLM runtime behavior, or artifact schemas, update the relevant docs and examples in the same change.