238 lines
11 KiB
Markdown
238 lines
11 KiB
Markdown
# Architecture
|
|
|
|
This document defines the intended high-level architecture of Notarius and the
|
|
invariants that changes must preserve. Implemented component details belong in
|
|
[Internal Overview](../internal/overview.md) and its linked documents. The
|
|
reasoning behind significant architectural choices belongs in
|
|
[ADRs](../adr/).
|
|
|
|
## System Shape
|
|
|
|
Notarius is a small, dependency-light Go application for extracting structured
|
|
artifacts from source material. It is a general extraction platform whose
|
|
source formats, extraction domains, validation policies, LLM providers, and
|
|
output formats are isolated behind explicit boundaries.
|
|
|
|
The application has one fixed pipeline shape:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
Pipelines are configured compositions of this shape. They are not arbitrary
|
|
DAGs or a general workflow language. Every stage remains explicit; general
|
|
chunking, merging, or normalization behavior must not be hidden inside an
|
|
extractor.
|
|
|
|
Input and chunking are pipeline-wide. Each selected artifact lane owns its
|
|
extract, merge, and normalize stages, and the output stage aggregates the run's
|
|
lane outcomes.
|
|
|
|
Notarius is contract-first without being abstraction-heavy. Interfaces and
|
|
extension points should protect demonstrated boundaries. New abstraction is not
|
|
itself an architectural goal.
|
|
|
|
## Layers And Dependency Direction
|
|
|
|
The application boundary is the composition root and may depend on concrete
|
|
implementations. Domain-neutral model and framework layers provide reusable
|
|
policy, contracts, and orchestration. Concrete input, pipeline, output, and
|
|
validation extensions depend inward on those generic layers.
|
|
|
|
Generic layers must not depend on production extensions. Concrete extensions
|
|
must not compose the application or take ownership of process behavior. The
|
|
current packages implementing these layers are inventoried in
|
|
[Internal Overview](../internal/overview.md).
|
|
|
|
The following dependency boundaries are mandatory:
|
|
|
|
- extractors and validators do not depend on concrete input adapters;
|
|
- provider-specific types do not cross the LLM runtime boundary;
|
|
- external dependency types do not leak across internal package boundaries
|
|
unless that dependency is the package's explicit contract.
|
|
|
|
Shared helpers may support demonstrated common needs, but must not move
|
|
source-format or extraction-domain knowledge into generic framework packages.
|
|
External dependencies require a clear correctness, security, interoperability,
|
|
or complexity benefit.
|
|
|
|
## Source And Domain Boundaries
|
|
|
|
Input modules translate external source formats into the generic source model.
|
|
Format-specific schemas, fields, and validation remain with the input module
|
|
and its integration contract.
|
|
|
|
Framework stages operate on source documents, source units, and source
|
|
references rather than format-specific structures. A source reference identifies
|
|
an ordered range of generic source units. Framework code preserves those ranges
|
|
and does not merge or rewrite them unless a stage module explicitly owns that
|
|
behavior. Every source unit carries a validated self-reference to its containing
|
|
document and its own unit ID.
|
|
|
|
Extract modules own artifact semantics, prompt use, response schemas, and
|
|
domain interpretation. Domain-specific concepts remain in the relevant module,
|
|
validator, shared domain helper, and artifact contract.
|
|
|
|
Typed artifact registrations declare one stable artifact kind and exact Go
|
|
type from extraction through merge, normalization, and semantic validation.
|
|
Pipeline resolution requires a compatible codec and matching kind-specific
|
|
variants before a typed lane can be accepted. Framework-owned erasure remains
|
|
private and must report type incompatibility as an error rather than a panic.
|
|
|
|
An artifact kind may additionally provide a typed evidence projection that
|
|
copies its direct generic source references. Preparation proves that projection
|
|
matches the artifact codec's exact Go type before retaining it for an output
|
|
policy. The runner reconstructs evidence only from accepted serialized
|
|
normalized artifacts, and the output boundary owns any resulting publication.
|
|
Generic framework code never infers evidence by inspecting domain JSON or
|
|
depends on domain artifact types.
|
|
|
|
Auxiliary references provide context or disambiguation. They are not source
|
|
evidence and must not be converted into source references.
|
|
|
|
## Pipeline Composition And Ownership
|
|
|
|
Module selection is configuration- and registry-driven. The framework resolves
|
|
named pipeline definitions, applies explicit defaults and runtime overrides,
|
|
and verifies module availability and capabilities before execution. Structural
|
|
pipeline choices must not be scattered through conditionals or hidden behind
|
|
ad hoc command flags.
|
|
|
|
Resolution validates every selected module and validator option set. A separate
|
|
preparation boundary then constructs the complete input, chunk, lane,
|
|
validation, and output implementation set in pipeline order. The runner accepts
|
|
only that prepared set, so construction and dependency failures occur before
|
|
source parsing or any other module operation.
|
|
|
|
Stage ownership is explicit:
|
|
|
|
- input modules convert external material into the generic source model;
|
|
- chunk modules partition source material for extraction;
|
|
- extract modules produce domain artifacts from chunks;
|
|
- merge modules combine accepted extraction outputs;
|
|
- normalize modules reconcile merged output;
|
|
- output modules encode accepted results and run outcomes into logical files.
|
|
|
|
Chunk modules produce source-addressed chunk plans rather than materialized
|
|
chunks. The framework validates and materializes those plans into the generic
|
|
chunk representation before chunk validation and lane execution. Plan reuse is
|
|
therefore independent of the configured pipeline, module options, references,
|
|
lanes, validators, and LLM profile: the canonical source digest selects the
|
|
plan, while the current run still applies its configured chunk validators to
|
|
the materialized chunks.
|
|
|
|
The framework owns orchestration and handoff provenance. Modules return logical
|
|
results and warnings; they do not own CLI reporting, physical output, cache, or
|
|
debug roots, durable file placement, or checkpoint and debug lifecycle.
|
|
|
|
After pipeline-wide chunking, extraction uses bounded framework concurrency.
|
|
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
|
|
chunk-first, lane-second order. A lane may begin its merge and normalize
|
|
continuation only after all of its extract jobs are terminal; that continuation
|
|
remains serial within the lane, while bounded continuations for different lanes
|
|
may overlap. The framework must not create unbounded goroutines per lane or
|
|
chunk.
|
|
|
|
Completion timing does not choose public ordering or errors. The coordinator
|
|
orders accepted artifacts, warnings, rejections, checkpoint events, and
|
|
framework errors by stable pipeline scope. Rejections do not cancel unrelated
|
|
work. A framework error cancels derived work, prevents undispatched work from
|
|
starting, waits for started work, and prevents output encoding.
|
|
|
|
## Validation
|
|
|
|
Validation is a framework-managed boundary around outputs from chunk, extract,
|
|
merge, and normalize stages. Validators receive immutable stage output
|
|
and make an explicit whole-output decision: approve, approve with warnings, or
|
|
reject.
|
|
|
|
Typed artifact validators receive the domain value directly. Chunk validators
|
|
receive source-zone chunks, while serialized validators receive immutable
|
|
representation bytes and declared schema metadata. A validator registered for
|
|
one target or artifact kind cannot satisfy an incompatible selection.
|
|
|
|
Rejection is a recorded pipeline outcome, not a framework execution error.
|
|
Validator execution failures are framework errors. Rejected output does not
|
|
advance to the next stage.
|
|
|
|
Default validator chains are production composition policy and are registered
|
|
centrally by stage and module. Configuration may replace a stage-local default,
|
|
including with an explicitly empty chain. Configured validator order is
|
|
authoritative; the framework must not silently reorder it.
|
|
|
|
## LLM Boundary
|
|
|
|
Modules and validators use transport-neutral structured completion contracts.
|
|
Provider request and response types, authentication, transport behavior, and
|
|
provider error adaptation remain inside the LLM runtime.
|
|
|
|
The caller of the LLM owns prompt selection, prompt inputs, response schema,
|
|
and interpretation of structured output. Provider adapters do not own source-
|
|
or domain-specific prompt logic.
|
|
|
|
LLM calls and other external operations accept cancellation and respect
|
|
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
|
|
individual modules.
|
|
|
|
The application-wide LLM scheduler bounds actual provider calls independently
|
|
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
|
the single injected scheduled client, including work performed by overlapping
|
|
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
|
|
beneath this mandatory application-wide scheduler.
|
|
|
|
## Configuration And Provenance
|
|
|
|
Configuration loading, precedence, defaults, environment overrides, redaction,
|
|
and validation are centralized. Named pipeline definitions make structural
|
|
composition explicit and discoverable. Operational overrides are permitted
|
|
when they do not obscure the configured pipeline structure.
|
|
|
|
Run preparation fails before stage execution when statically discoverable
|
|
modules, capabilities, reference bindings, or explicitly selected profiles are
|
|
invalid or incompatible.
|
|
|
|
Run manifests record enough resolved pipeline, module, source, reference, and
|
|
LLM provenance to make a run auditable after configuration changes. Manifests
|
|
record identities and summaries rather than secret or large payload content.
|
|
|
|
## State, Output, And Safety
|
|
|
|
Notarius exposes three filesystem surfaces with independent roots and
|
|
lifecycle:
|
|
|
|
- output is durable user data; output modules define logical files and the CLI
|
|
owns their placement;
|
|
- cache is reconstructible state, with separate chunk-plan and checkpoint
|
|
families; and
|
|
- debug is explicitly requested inspection data, combining a redacted summary
|
|
with a detailed trace.
|
|
|
|
Chunk plans are keyed only by canonical source digest. Configured checkpoint
|
|
recording is independent of checkpoint reuse; checkpoints are loaded only for
|
|
an invocation that explicitly requests resume. Debug is never a cache input and
|
|
is never created without an explicit request. Pipeline modules receive
|
|
collaborator interfaces and never physical roots.
|
|
|
|
Writes are atomic where practical. Paths for writes, moves, overwrites, and
|
|
deletion must be narrow and explicit. Notarius never automatically deletes
|
|
output or requested debug bundles; cache cleanup is explicit and recoverable.
|
|
|
|
Secrets must not appear in errors, logs, output, cache, debug summaries,
|
|
traces, manifests, documentation, examples, or redacted configuration. Debug
|
|
collection is allowlisted to application-owned payloads and must not capture
|
|
unrelated process environment values or filesystem content. Trace data may
|
|
contain application data and therefore inherits its sensitivity; operators own
|
|
access controls and retention. Physical layout and operation are defined in
|
|
[Operations](../operations.md).
|
|
|
|
## Architectural Non-Goals
|
|
|
|
Notarius does not aim to provide:
|
|
|
|
- an arbitrary workflow graph or general workflow language;
|
|
- source-format or extraction-domain behavior in generic framework packages;
|
|
- provider-specific contracts exposed to modules;
|
|
- structural pipeline composition through ad hoc CLI flags;
|
|
- implicit cross-stage behavior that bypasses the fixed pipeline;
|
|
- abstractions introduced solely for hypothetical future complexity.
|