176 lines
7.6 KiB
Markdown
176 lines
7.6 KiB
Markdown
# Architecture
|
|
|
|
This document defines the intended high-level architecture of Notarius and the
|
|
invariants that changes must preserve. Implemented component details belong in
|
|
[Internal Overview](../internal/overview.md) and its linked documents. The
|
|
reasoning behind significant architectural choices belongs in
|
|
[ADRs](../adr/).
|
|
|
|
## System Shape
|
|
|
|
Notarius is a small, dependency-light Go application for extracting structured
|
|
artifacts from source material. It is a general extraction platform whose
|
|
source formats, extraction domains, validation policies, LLM providers, and
|
|
output formats are isolated behind explicit boundaries.
|
|
|
|
The application has one fixed pipeline shape:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
Pipelines are configured compositions of this shape. They are not arbitrary
|
|
DAGs or a general workflow language. Every stage remains explicit; general
|
|
chunking, merging, or normalization behavior must not be hidden inside an
|
|
extractor.
|
|
|
|
Input and chunking are pipeline-wide. Each selected artifact lane owns its
|
|
extract, merge, and normalize stages, and the output stage aggregates the run's
|
|
lane outcomes.
|
|
|
|
Notarius is contract-first without being abstraction-heavy. Interfaces and
|
|
extension points should protect demonstrated boundaries. New abstraction is not
|
|
itself an architectural goal.
|
|
|
|
## Layers And Dependency Direction
|
|
|
|
The application boundary is the composition root and may depend on concrete
|
|
implementations. Domain-neutral model and framework layers provide reusable
|
|
policy, contracts, and orchestration. Concrete input, pipeline, output, and
|
|
validation extensions depend inward on those generic layers.
|
|
|
|
Generic layers must not depend on production extensions. Concrete extensions
|
|
must not compose the application or take ownership of process behavior. The
|
|
current packages implementing these layers are inventoried in
|
|
[Internal Overview](../internal/overview.md).
|
|
|
|
The following dependency boundaries are mandatory:
|
|
|
|
- extractors and validators do not depend on concrete input adapters;
|
|
- provider-specific types do not cross the LLM runtime boundary;
|
|
- external dependency types do not leak across internal package boundaries
|
|
unless that dependency is the package's explicit contract.
|
|
|
|
Shared helpers may support demonstrated common needs, but must not move
|
|
source-format or extraction-domain knowledge into generic framework packages.
|
|
External dependencies require a clear correctness, security, interoperability,
|
|
or complexity benefit.
|
|
|
|
## Source And Domain Boundaries
|
|
|
|
Input modules translate external source formats into the generic source model.
|
|
Format-specific schemas, fields, and validation remain with the input module
|
|
and its integration contract.
|
|
|
|
Framework stages operate on source documents, source units, and source
|
|
references rather than format-specific structures. A source reference identifies
|
|
an ordered range of generic source units. Framework code preserves those ranges
|
|
and does not merge or rewrite them unless a stage module explicitly owns that
|
|
behavior.
|
|
|
|
Extract modules own artifact semantics, prompt use, response schemas, and
|
|
domain interpretation. Domain-specific concepts remain in the relevant module,
|
|
validator, shared domain helper, and artifact contract.
|
|
|
|
Auxiliary references provide context or disambiguation. They are not source
|
|
evidence and must not be converted into source references.
|
|
|
|
## Pipeline Composition And Ownership
|
|
|
|
Module selection is configuration- and registry-driven. The framework resolves
|
|
named pipeline definitions, applies explicit defaults and runtime overrides,
|
|
and verifies module availability and capabilities before execution. Structural
|
|
pipeline choices must not be scattered through conditionals or hidden behind
|
|
ad hoc command flags.
|
|
|
|
Stage ownership is explicit:
|
|
|
|
- input modules convert external material into the generic source model;
|
|
- chunk modules partition source material for extraction;
|
|
- extract modules produce domain artifacts from chunks;
|
|
- merge modules combine accepted extraction outputs;
|
|
- normalize modules reconcile merged output;
|
|
- output modules encode accepted results and run outcomes into logical files.
|
|
|
|
The framework owns orchestration and handoff provenance. Modules return logical
|
|
results and warnings; they do not own CLI reporting, workspace paths, durable
|
|
file placement, checkpoints, or diagnostics.
|
|
|
|
## Validation
|
|
|
|
Validation is a framework-managed boundary around raw outputs from chunk,
|
|
extract, merge, and normalize stages. Validators receive immutable stage output
|
|
and make an explicit whole-output decision: approve, approve with warnings, or
|
|
reject.
|
|
|
|
Rejection is a recorded pipeline outcome, not a framework execution error.
|
|
Validator execution failures are framework errors. Rejected output does not
|
|
advance to the next stage.
|
|
|
|
Default validator chains are production composition policy and are registered
|
|
centrally by stage and module. Configuration may replace a stage-local default,
|
|
including with an explicitly empty chain. Configured validator order is
|
|
authoritative; the framework must not silently reorder it.
|
|
|
|
## LLM Boundary
|
|
|
|
Modules and validators use transport-neutral structured completion contracts.
|
|
Provider request and response types, authentication, transport behavior, and
|
|
provider error adaptation remain inside the LLM runtime.
|
|
|
|
The caller of the LLM owns prompt selection, prompt inputs, response schema,
|
|
and interpretation of structured output. Provider adapters do not own source-
|
|
or domain-specific prompt logic.
|
|
|
|
LLM calls and other external operations accept cancellation and respect
|
|
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
|
|
individual modules.
|
|
|
|
## Configuration And Provenance
|
|
|
|
Configuration loading, precedence, defaults, environment overrides, redaction,
|
|
and validation are centralized. Named pipeline definitions make structural
|
|
composition explicit and discoverable. Operational overrides are permitted
|
|
when they do not obscure the configured pipeline structure.
|
|
|
|
Run preparation fails before stage execution when statically discoverable
|
|
modules, capabilities, reference bindings, or explicitly selected profiles are
|
|
invalid or incompatible.
|
|
|
|
Run manifests record enough resolved pipeline, module, source, reference, and
|
|
LLM provenance to make a run auditable after configuration changes. Manifests
|
|
record identities and summaries rather than secret or large payload content.
|
|
|
|
## State, Output, And Safety
|
|
|
|
Durable output, diagnostics, checkpoints, and debug artifacts are separate
|
|
surfaces with separate ownership:
|
|
|
|
- output modules define logical durable output; the application boundary owns
|
|
filesystem placement;
|
|
- diagnostics provide redacted run inspection and are not the durable output
|
|
contract;
|
|
- checkpoints support validated stage reuse and are not diagnostics;
|
|
- debug artifacts are opt-in inspection data and may contain sensitive source,
|
|
prompt, reference, and model-output content.
|
|
|
|
Writes of durable state are atomic where practical. Paths for writes, moves,
|
|
overwrites, and deletion must be narrow and explicit. Cleanup that can lose data
|
|
is opt-in.
|
|
|
|
Secrets must not appear in errors, logs, diagnostics, manifests,
|
|
documentation, examples, or redacted configuration. Default logs and
|
|
diagnostics must not include large source, prompt, reference, or artifact
|
|
payloads.
|
|
|
|
## Architectural Non-Goals
|
|
|
|
Notarius does not aim to provide:
|
|
|
|
- an arbitrary workflow graph or general workflow language;
|
|
- source-format or extraction-domain behavior in generic framework packages;
|
|
- provider-specific contracts exposed to modules;
|
|
- structural pipeline composition through ad hoc CLI flags;
|
|
- implicit cross-stage behavior that bypasses the fixed pipeline;
|
|
- abstractions introduced solely for hypothetical future complexity.
|