notarius/docs/policy/architecture.md

Architecture

This document defines the development principles for Notarius. It is inward-facing: developers and LLM coding agents should use it to preserve the project's shape, boundaries, and invariants as the code evolves.

Project Shape

Notarius is a small, explicit, dependency-light Go application for extracting structured artifacts from source material using modular extractors.

The application should be contract-first but not abstraction-heavy. Add interfaces and extension points when they protect a real boundary:

• external source formats;
• extractor modules;
• validators;
• LLM providers and runtime plumbing;
• output schemas and embedded assets.

Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow contracts that can be exercised by tests and real modules.

Core Invariants

The core framework must remain source-agnostic and domain-agnostic.

Source-format details belong in input modules. Transcript-specific concepts such as segments, speakers, timestamps, and transcript schemas must not spread into runner, extractor, or validator framework code.

Extraction-domain details belong in extract modules. D&D-specific concepts such as spells, NPCs, items, combat turns, and encounters must not spread into core source, runner, or LLM framework packages.

Extracted facts should be grounded with source references. Source references should point to generic source units, not to transcript-only structures.

The application workflow is:

input -> chunk -> extract -> merge -> normalize -> output

These stages should remain explicit in the architecture. Chunking, merging, and normalization must not be hidden inside domain extract modules when they represent general pipeline behavior.

Dependency Policy

Prefer the Go standard library where practical.

Use external dependencies only when justified by correctness, security, interoperability, or substantial complexity reduction. Good reasons include widely used file formats, complex validation behavior, or secure transport handling.

Avoid dependencies for small conveniences. Do not let external dependency types leak across internal package boundaries unless the dependency is itself the explicit public contract of that package.

Package Layout

Use this layout unless a change documents a better project-specific reason.

CLI and executable entrypoint:

• cmd/notarius: executable entrypoint.
• internal/cli: CLI command definitions, flags, argument parsing, and command wiring.

Core deterministic model and policy:

• internal/core/config: configuration structs, defaults, loading, precedence, and validation.
• internal/core/source: source document, source unit, and source reference types.
• internal/core/sourcechunking: deterministic chunking of ordered source units.
• internal/core/artifacts: artifact envelope, artifact candidates, rejected artifacts, and manifests.
• internal/core/diagnostics: run directories and diagnostics artifact paths.
• internal/core/reporting: process reports and report serialization.
• internal/core/inputcatalog: known input adapter keys and metadata.
• internal/core/extractorcatalog: known extractor keys and metadata.

Reusable framework plumbing:

• internal/framework/contracts: core interfaces and transport-neutral request/response contracts.
• internal/framework/runner: orchestration across adapters, extractors, validators, and artifact output.
• internal/framework/pipeline: shared pipeline-stage orchestration types, when needed.
• internal/framework/extraction: shared extraction helper code.
• internal/framework/merge: shared merge-stage behavior.
• internal/framework/normalize: shared normalization-stage behavior.
• internal/framework/output: output encoding contracts and shared helpers.
• internal/framework/validators: shared validator runtime behavior and decision checks.
• internal/framework/llm: LLM runtime, scheduling, and provider adapters.
• internal/framework/responseschema: embedded structured-output schema registry.
• internal/framework/structuredoutput: structured-output parsing and malformed-response handling.
• internal/framework/promptcontext: source-document prompt rendering helpers.
• internal/framework/warnings: shared warning records.

Domain implementations:

• internal/modules/input/<name>: input-stage modules that parse external input into core source documents.
• internal/modules/chunk/<name>: chunk-stage modules.
• internal/modules/extract/<domain>/<name>: extract-stage extractor modules.
• internal/modules/merge/<name> or internal/modules/merge/<domain>/<name>: merge-stage modules.
• internal/modules/normalize/<name> or internal/modules/normalize/<domain>/<name>: normalize-stage modules.
• internal/modules/output/<name>: output-stage modules.
• internal/validators/<validator>: built-in validator implementations.
• internal/prompts: embedded prompt assets and prompt metadata registry.
• internal/transport/http: shared HTTP client code, if needed by provider integrations.

Package-private implementation constants may live near the package that owns them, preferably in constants.go when useful.

Stage Modules

Concrete business logic should live under internal/modules/<stage>/....

Stage-oriented module layout is preferred because it makes the application workflow visible in the filesystem:

internal/modules/input/...
internal/modules/chunk/...
internal/modules/extract/...
internal/modules/merge/...
internal/modules/normalize/...
internal/modules/output/...

Use short, lowercase, idiomatic Go package names. Prefer names such as dndtranscript, appendorder, and spells over names like dnd_transcript, serial_merge, or spell_extractor that repeat parent-stage context.

Input Modules

Use a hexagonal architecture style for source input.

Input modules translate external source formats into the core source model. They may know about external schema details, source-specific metadata, and format-specific validation rules. They should not own extraction-domain decisions.

Other packages should interact with source input through adapter contracts and core source types. Input module implementation details and external dependency types must not leak into framework or extract module packages.

Input module metadata may preserve source-specific facts such as transcript speaker, timestamps, Markdown heading path, page number, or block ID. Framework code may carry metadata through, but should not require a specific adapter's metadata shape.

Extractors

Extractors are independent modules that process source chunks or whole source documents and produce one kind of structured artifact candidate. Each extract module owns:

• its artifact semantics;
• its prompt usage;
• its structured response schema selection;
• its validator chain;
• any domain-specific mapping or interpretation.

Extract modules should depend on framework contracts and core source/artifact types. They should not depend on concrete input module packages.

Extractors should not be the only place where chunking, merging, or normalization happens. They may choose processing mode or provide domain-specific merge/normalization behavior when generic behavior is insufficient, but the pipeline stages themselves are framework concepts.

The runner should be able to compose, skip, resume, or run individual extractors when their prerequisites are satisfied. Ordering should be explicit through configuration, a default sequence, or documented orchestration rules.

Extractor selection must go through a registry or equivalent mechanism rather than scattered conditionals.

Pipeline Stages

The pipeline has six conceptual stages:

1. input: external source material becomes a SourceDocument;
2. chunk: a SourceDocument becomes ordered source chunks;
3. extract: extractors produce artifact candidates from chunks or whole documents;
4. merge: per-chunk candidates become a merged candidate collection;
5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
6. output: final artifacts are serialized.

Chunking is first-class because source documents may be too large for a single LLM pass. Chunkers should preserve source-unit order and produce stable chunk metadata.

Merge and normalize are separate concerns. Merge combines per-chunk results into a deterministic collection. Normalize performs semantic reconciliation after merge. Generic append-in-chunk-order merge and no-op normalization should be available for simple artifact types, while domain-specific behavior can be provided where needed.

The framework should allow serial and parallel chunk processing. The first implementation may execute chunks serially for determinism, but contracts should not prevent later parallel execution.

Validators

Validators should be independently testable and composable.

Deterministic validators should run before LLM-backed validators when both are present. Validator decision semantics should be explicit: each candidate artifact should receive exactly one decision from each validator that evaluates it.

Shared validator runtime mechanics belong under internal/framework/validators. Concrete validator behavior belongs under internal/validators/<validator>.

LLM Runtime

LLM provider details belong behind transport-neutral framework contracts.

Provider-specific HTTP request and response types should stay inside the LLM runtime package. Prompt construction should stay in extractors, validators, or shared prompt-context helpers; provider adapters should not own domain prompt logic.

Errors, diagnostics, reports, and redacted config must not expose secrets.

Configuration

Centralize configuration loading, processing, precedence, defaults, and validation in internal/core/config.

The goal is to make configuration discoverable and avoid implicit or hidden operational values. User-visible defaults and cross-package operational defaults should be defined in config code.

Unless documented otherwise, precedence is:

1. CLI flags
2. environment variables
3. configuration file
4. built-in defaults

Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discoverable at /usr/local/etc/notarius/config.yml, with a CLI override via --config.

Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets.

Stage-module-specific configuration should remain grouped by the module that owns it.

Embedded Assets

Store embedded JSON schemas, Markdown prompts, templates, and similar assets as separate files, not inline string literals, unless there is a strong reason otherwise.

Embedded prompts and response schemas should have stable IDs, versions, source metadata, and hashes suitable for diagnostics and run manifests.

Errors and Logging

Errors should be actionable and preserve context. Wrap errors with operation and path/resource context. CLI code should convert internal errors into concise user-facing messages.

Errors and logs must not expose secrets.

Use structured logging where practical. Logs should describe operations, paths, external calls, retries, and failure causes, but should not include large source or artifact payloads by default.

Context, Timeouts, and Cancellation

Long-running operations should accept context.Context. External calls, subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage workflows should respect cancellation and timeouts.

State, Files, and Safety

If the application writes durable state, writes should be atomic where practical. Multi-step workflows should preserve enough state to support inspection, retry, or resume after failure.

Code that deletes, moves, or overwrites files must use narrow, explicit paths. Avoid broad parent-directory operations. Cleanup that can cause data loss must be opt-in.

Testing

Core logic should be testable without real external services. Use fakes, fixtures, or local test doubles for input modules, extract modules, validators, and LLM clients where practical.

Contract-first work should include fake implementations that prove interfaces compose before real input modules or extract modules depend on them.

Config examples should be load-tested once config files exist. Important CLI workflows should have parser or command tests. Adapter, extractor, and validator contracts should have focused tests that do not require running the full application unless end-to-end coverage is intentional.

Documentation

Documentation should follow the project documentation policy. Keep user docs focused on implemented behavior. Put future, planned, or aspirational work only under docs/roadmap/.

Core documentation should use generic terms such as source document, source unit, source reference, input adapter, extractor, chunker, merger, normalizer, artifact, validator, and run manifest.

Source-format details belong in input module or integration docs. Domain-specific extraction details belong in extract module or artifact docs.

When changing architecture, config, CLI behavior, stage modules, extractor contracts, validator contracts, LLM runtime behavior, or artifact schemas, update the relevant docs and examples in the same change.