8.7 KiB
Architecture
This document defines the intended high-level architecture of Notarius and the invariants that changes must preserve. Implemented component details belong in Internal Overview and its linked documents. The reasoning behind significant architectural choices belongs in ADRs.
System Shape
Notarius is a small, dependency-light Go application for extracting structured artifacts from source material. It is a general extraction platform whose source formats, extraction domains, validation policies, LLM providers, and output formats are isolated behind explicit boundaries.
The application has one fixed pipeline shape:
input -> chunk -> extract -> merge -> normalize -> output
Pipelines are configured compositions of this shape. They are not arbitrary DAGs or a general workflow language. Every stage remains explicit; general chunking, merging, or normalization behavior must not be hidden inside an extractor.
Input and chunking are pipeline-wide. Each selected artifact lane owns its extract, merge, and normalize stages, and the output stage aggregates the run's lane outcomes.
Notarius is contract-first without being abstraction-heavy. Interfaces and extension points should protect demonstrated boundaries. New abstraction is not itself an architectural goal.
Layers And Dependency Direction
The application boundary is the composition root and may depend on concrete implementations. Domain-neutral model and framework layers provide reusable policy, contracts, and orchestration. Concrete input, pipeline, output, and validation extensions depend inward on those generic layers.
Generic layers must not depend on production extensions. Concrete extensions must not compose the application or take ownership of process behavior. The current packages implementing these layers are inventoried in Internal Overview.
The following dependency boundaries are mandatory:
- extractors and validators do not depend on concrete input adapters;
- provider-specific types do not cross the LLM runtime boundary;
- external dependency types do not leak across internal package boundaries unless that dependency is the package's explicit contract.
Shared helpers may support demonstrated common needs, but must not move source-format or extraction-domain knowledge into generic framework packages. External dependencies require a clear correctness, security, interoperability, or complexity benefit.
Source And Domain Boundaries
Input modules translate external source formats into the generic source model. Format-specific schemas, fields, and validation remain with the input module and its integration contract.
Framework stages operate on source documents, source units, and source references rather than format-specific structures. A source reference identifies an ordered range of generic source units. Framework code preserves those ranges and does not merge or rewrite them unless a stage module explicitly owns that behavior. Every source unit carries a validated self-reference to its containing document and its own unit ID.
Extract modules own artifact semantics, prompt use, response schemas, and domain interpretation. Domain-specific concepts remain in the relevant module, validator, shared domain helper, and artifact contract.
Typed artifact registrations declare one stable artifact kind and exact Go type from extraction through merge, normalization, and semantic validation. Pipeline resolution requires a compatible codec and matching kind-specific variants before a typed lane can be accepted. Framework-owned erasure remains private and must report type incompatibility as an error rather than a panic.
Auxiliary references provide context or disambiguation. They are not source evidence and must not be converted into source references.
Pipeline Composition And Ownership
Module selection is configuration- and registry-driven. The framework resolves named pipeline definitions, applies explicit defaults and runtime overrides, and verifies module availability and capabilities before execution. Structural pipeline choices must not be scattered through conditionals or hidden behind ad hoc command flags.
Resolution validates every selected module and validator option set. A separate preparation boundary then constructs the complete input, chunk, lane, validation, and output implementation set in pipeline order. The runner accepts only that prepared set, so construction and dependency failures occur before source parsing or any other module operation.
Stage ownership is explicit:
- input modules convert external material into the generic source model;
- chunk modules partition source material for extraction;
- extract modules produce domain artifacts from chunks;
- merge modules combine accepted extraction outputs;
- normalize modules reconcile merged output;
- output modules encode accepted results and run outcomes into logical files.
The framework owns orchestration and handoff provenance. Modules return logical results and warnings; they do not own CLI reporting, workspace paths, durable file placement, checkpoints, or diagnostics.
Validation
Validation is a framework-managed boundary around raw outputs from chunk, extract, merge, and normalize stages. Validators receive immutable stage output and make an explicit whole-output decision: approve, approve with warnings, or reject.
Typed artifact validators receive the domain value directly. Chunk validators receive source-zone chunks, while serialized validators receive immutable representation bytes and declared schema metadata. A validator registered for one target or artifact kind cannot satisfy an incompatible selection.
Rejection is a recorded pipeline outcome, not a framework execution error. Validator execution failures are framework errors. Rejected output does not advance to the next stage.
Default validator chains are production composition policy and are registered centrally by stage and module. Configuration may replace a stage-local default, including with an explicitly empty chain. Configured validator order is authoritative; the framework must not silently reorder it.
LLM Boundary
Modules and validators use transport-neutral structured completion contracts. Provider request and response types, authentication, transport behavior, and provider error adaptation remain inside the LLM runtime.
The caller of the LLM owns prompt selection, prompt inputs, response schema, and interpretation of structured output. Provider adapters do not own source- or domain-specific prompt logic.
LLM calls and other external operations accept cancellation and respect timeouts. Concurrency control belongs in shared runtime plumbing rather than in individual modules.
Configuration And Provenance
Configuration loading, precedence, defaults, environment overrides, redaction, and validation are centralized. Named pipeline definitions make structural composition explicit and discoverable. Operational overrides are permitted when they do not obscure the configured pipeline structure.
Run preparation fails before stage execution when statically discoverable modules, capabilities, reference bindings, or explicitly selected profiles are invalid or incompatible.
Run manifests record enough resolved pipeline, module, source, reference, and LLM provenance to make a run auditable after configuration changes. Manifests record identities and summaries rather than secret or large payload content.
State, Output, And Safety
Durable output, diagnostics, checkpoints, and debug artifacts are separate surfaces with separate ownership:
- output modules define logical durable output; the application boundary owns filesystem placement;
- diagnostics provide redacted run inspection and are not the durable output contract;
- checkpoints support validated stage reuse and are not diagnostics;
- debug artifacts are opt-in inspection data and may contain sensitive source, prompt, reference, and model-output content.
Writes of durable state are atomic where practical. Paths for writes, moves, overwrites, and deletion must be narrow and explicit. Cleanup that can lose data is opt-in.
Secrets must not appear in errors, logs, diagnostics, manifests, documentation, examples, or redacted configuration. Default logs and diagnostics must not include large source, prompt, reference, or artifact payloads.
Architectural Non-Goals
Notarius does not aim to provide:
- an arbitrary workflow graph or general workflow language;
- source-format or extraction-domain behavior in generic framework packages;
- provider-specific contracts exposed to modules;
- structural pipeline composition through ad hoc CLI flags;
- implicit cross-stage behavior that bypasses the fixed pipeline;
- abstractions introduced solely for hypothetical future complexity.