# Architecture

This document defines the intended high-level architecture of Notarius and the
invariants that changes must preserve. Implemented component details belong in
[Internal Overview](../internal/overview.md) and its linked documents. The
reasoning behind significant architectural choices belongs in
[ADRs](../adr/).

## System Shape

Notarius is a small, dependency-light Go application for extracting structured
artifacts from source material. It is a general extraction platform whose
source formats, extraction domains, validation policies, LLM providers, and
output formats are isolated behind explicit boundaries.

The application has one fixed pipeline shape:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

Pipelines are configured compositions of this shape. They are not arbitrary
DAGs or a general workflow language. Every stage remains explicit; general
chunking, merging, or normalization behavior must not be hidden inside an
extractor.

Input and chunking are pipeline-wide. Each selected artifact lane owns its
extract, merge, and normalize stages, and the output stage aggregates the run's
lane outcomes.

Notarius is contract-first without being abstraction-heavy. Interfaces and
extension points should protect demonstrated boundaries. New abstraction is not
itself an architectural goal.

## Layers And Dependency Direction

The application boundary is the composition root and may depend on concrete
implementations. Domain-neutral model and framework layers provide reusable
policy, contracts, and orchestration. Concrete input, pipeline, output, and
validation extensions depend inward on those generic layers.

Generic layers must not depend on production extensions. Concrete extensions
must not compose the application or take ownership of process behavior. The
current packages implementing these layers are inventoried in
[Internal Overview](../internal/overview.md).

The root `assets` package is a content-only dependency leaf. It may expose a
read-only embedded filesystem, but it must contain no business logic and must
not depend on `internal` packages or PromptKit. Consumers scope that filesystem
to the content they own; the root package is not a behavioral registry or a
public extension contract. The rationale and compatibility consequence are
recorded in [ADR-0011](../adr/0011-centralize-llm-assets.md).

The following dependency boundaries are mandatory:

- extractors and validators do not depend on concrete input adapters;
- provider-specific types do not cross the LLM runtime boundary;
- external dependency types do not leak across internal package boundaries
  unless that dependency is the package's explicit contract.

Shared helpers may support demonstrated common needs, but must not move
source-format or extraction-domain knowledge into generic framework packages.
External dependencies require a clear correctness, security, interoperability,
or complexity benefit.

## Source And Domain Boundaries

Input modules translate external source formats into the generic source model.
Format-specific schemas, fields, and validation remain with the input module
and its integration contract.

Framework stages operate on source documents, source units, and source
references rather than format-specific structures. A source reference identifies
an ordered range of generic source units. Framework code preserves those ranges
and does not merge or rewrite them unless a stage module explicitly owns that
behavior. Every source unit carries a validated self-reference to its containing
document and its own unit ID.

Extract modules own artifact semantics, prompt use, response schemas, and
domain interpretation. Domain-specific concepts remain in the relevant module,
validator, shared domain helper, and artifact contract.

Physical centralization of LLM-facing content does not transfer semantic
ownership from those modules. Modules retain their manifests, response-schema
identities, prompt ordering, and registration, while reading only their scoped
content subtree. Generic framework code remains domain-neutral when it reads
its own scoped generic assets from the shared content container.

Typed artifact registrations declare one stable artifact kind and exact Go
type from extraction through merge, normalization, and semantic validation.
Pipeline resolution requires a compatible codec and matching kind-specific
variants before a typed lane can be accepted. Framework-owned erasure remains
private and must report type incompatibility as an error rather than a panic.

An artifact kind may additionally provide a typed evidence projection that
copies its direct generic source references. Preparation proves that projection
matches the artifact codec's exact Go type before retaining it for an output
policy. The runner reconstructs evidence only from accepted serialized
normalized artifacts, and the output boundary owns any resulting publication.
Generic framework code never infers evidence by inspecting domain JSON or
depends on domain artifact types.

Auxiliary references provide context or disambiguation. They are not source
evidence and must not be converted into source references.

## Pipeline Composition And Ownership

Module selection is configuration- and registry-driven. The framework resolves
named pipeline definitions, applies explicit defaults and runtime overrides,
and verifies module availability and capabilities before execution. Structural
pipeline choices must not be scattered through conditionals or hidden behind
ad hoc command flags.

Resolution validates every selected module and validator option set. A separate
preparation boundary then constructs the complete input, chunk, lane,
validation, and output implementation set in pipeline order. The runner accepts
only that prepared set, so construction and dependency failures occur before
source parsing or any other module operation.

Stage ownership is explicit:

- input modules convert external material into the generic source model;
- chunk modules partition source material for extraction;
- extract modules produce domain artifacts from chunks;
- merge modules combine accepted extraction outputs;
- normalize modules reconcile merged output;
- output modules encode accepted results and run outcomes into logical files.

Chunk modules produce source-addressed chunk plans rather than materialized
chunks. The framework validates and materializes those plans into the generic
chunk representation before chunk validation and lane execution. Plan reuse is
therefore independent of the configured pipeline, module options, references,
lanes, validators, and LLM profile: the canonical source digest selects the
plan, while the current run still applies its configured chunk validators to
the materialized chunks.

The framework owns orchestration and handoff provenance. Modules return logical
results and warnings; they do not own CLI reporting, physical output, cache, or
debug roots, durable file placement, or checkpoint and debug lifecycle.

After pipeline-wide chunking, extraction uses bounded framework concurrency.
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
chunk-first, lane-second order. A lane may begin its merge and normalize
continuation only after all of its extract jobs are terminal; that continuation
remains serial within the lane, while bounded continuations for different lanes
may overlap. The framework must not create unbounded goroutines per lane or
chunk.

Completion timing does not choose public ordering or errors. The coordinator
orders accepted artifacts, warnings, rejections, checkpoint events, and
framework errors by stable pipeline scope. Rejections do not cancel unrelated
work. A framework error cancels derived work, prevents undispatched work from
starting, waits for started work, and prevents output encoding.

## Validation

Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output
and make an explicit whole-output decision: approve, approve with warnings, or
reject.

Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection.

Rejection is a recorded pipeline outcome, not a framework execution error.
Validator execution failures are framework errors. Rejected output does not
advance to the next stage.

Default validator chains are production composition policy and are registered
centrally by stage and module. Configuration may replace a stage-local default,
including with an explicitly empty chain. Configured validator order is
authoritative; the framework must not silently reorder it.

## LLM Boundary

Modules and validators use transport-neutral structured completion contracts.
Provider request and response types, authentication, transport behavior, and
provider error adaptation remain inside the LLM runtime.

The caller of the LLM owns prompt selection, prompt inputs, response schema,
and interpretation of structured output. Provider adapters do not own source-
or domain-specific prompt logic.

LLM calls and other external operations accept cancellation and respect
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
individual modules.

The application-wide LLM scheduler bounds actual provider calls independently
of framework worker limits. Every LLM-backed module, retry, and validator uses
the single injected scheduled client, including work performed by overlapping
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
beneath this mandatory application-wide scheduler.

## Configuration And Provenance

Configuration loading, precedence, defaults, environment overrides, redaction,
and validation are centralized. Named pipeline definitions make structural
composition explicit and discoverable. Operational overrides are permitted
when they do not obscure the configured pipeline structure.

Run preparation fails before stage execution when statically discoverable
modules, capabilities, reference bindings, or explicitly selected profiles are
invalid or incompatible.

Run manifests record enough resolved pipeline, module, source, reference, and
LLM provenance to make a run auditable after configuration changes. Manifests
record identities and summaries rather than secret or large payload content.

## State, Output, And Safety

Notarius exposes three filesystem surfaces with independent roots and
lifecycle:

- output is durable user data; output modules define logical files and the CLI
  owns their placement;
- cache is reconstructible state, with separate chunk-plan and checkpoint
  families; and
- debug is explicitly requested inspection data, combining a redacted summary
  with a detailed trace.

Chunk plans are keyed only by canonical source digest. Configured checkpoint
recording is independent of checkpoint reuse; checkpoints are loaded only for
an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots.

Writes are atomic where practical. Paths for writes, moves, overwrites, and
deletion must be narrow and explicit. Notarius never automatically deletes
output or requested debug bundles; cache cleanup is explicit and recoverable.

Secrets must not appear in errors, logs, output, cache, debug summaries,
traces, manifests, documentation, examples, or redacted configuration. Debug
collection is allowlisted to application-owned payloads and must not capture
unrelated process environment values or filesystem content. Trace data may
contain application data and therefore inherits its sensitivity; operators own
access controls and retention. Physical layout and operation are defined in
[Operations](../operations.md).

## Architectural Non-Goals

Notarius does not aim to provide:

- an arbitrary workflow graph or general workflow language;
- source-format or extraction-domain behavior in generic framework packages;
- provider-specific contracts exposed to modules;
- structural pipeline composition through ad hoc CLI flags;
- implicit cross-stage behavior that bypasses the fixed pipeline;
- abstractions introduced solely for hypothetical future complexity.
