# Architecture

This document defines the intended high-level architecture of Notarius and the
invariants that changes must preserve. Implemented component details belong in
[Internal Overview](../internal/overview.md) and its linked documents. The
reasoning behind significant architectural choices belongs in
[ADRs](../adr/).

## System Shape

Notarius is a small, dependency-light Go application for extracting structured
artifacts from source material. It is a general extraction platform whose
source formats, extraction domains, validation policies, LLM providers, and
output formats are isolated behind explicit boundaries.

The application has one fixed pipeline shape:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

Pipelines are configured compositions of this shape. They are not arbitrary
DAGs or a general workflow language. Every stage remains explicit; general
chunking, merging, or normalization behavior must not be hidden inside an
extractor.

A stage module is one configured implementation of one pipeline stage. An
artifact family is the cohesive domain feature that owns an artifact across
the explicit stages and supporting codecs, validators, prompts, identity
rules, and reference projections. Artifact-family ownership does not combine
stages or alter the fixed pipeline.

Input and chunking are pipeline-wide. Each selected artifact lane owns its
extract, merge, and normalize stages, and the output stage aggregates the run's
lane outcomes.

Notarius is contract-first without being abstraction-heavy. Interfaces and
extension points should protect demonstrated boundaries. New abstraction is not
itself an architectural goal.

## Layers And Dependency Direction

The application boundary is the composition root and may depend on concrete
implementations. Domain-neutral model and framework layers provide reusable
policy, contracts, and orchestration. Concrete input, pipeline, output, and
validation extensions depend inward on those generic layers.

Semantic reconciliation is one such domain-neutral framework mechanism. It
prepares bounded source context, invokes a shared model-judgment protocol,
validates proposals, and applies safe plans through typed policies supplied by
the consuming artifact family. It does not own domain identity, durable IDs,
warning semantics, or artifact construction rules.

Generic layers must not depend on production extensions. Concrete extensions
must not compose the application or take ownership of process behavior. The
current packages implementing these layers are inventoried in
[Internal Overview](../internal/overview.md).

The root `assets` package is a content-only dependency leaf. It may expose a
read-only embedded filesystem, but it must contain no business logic and must
not depend on `internal` packages or PromptKit. Consumers scope that filesystem
to the content they own; the root package is not a behavioral registry or a
public extension contract. The rationale and compatibility consequence are
recorded in [ADR-0011](../adr/0011-centralize-llm-assets.md).

The following dependency boundaries are mandatory:

- extractors and validators do not depend on concrete input adapters;
- provider-specific types do not cross the LLM runtime boundary;
- external dependency types do not leak across internal package boundaries
  unless that dependency is the package's explicit contract.

Shared helpers may support demonstrated common needs, but must not move
source-format or extraction-domain knowledge into generic framework packages.
External dependencies require a clear correctness, security, interoperability,
or complexity benefit.

## Source And Domain Boundaries

Input modules translate external source formats into the generic source model.
Format-specific schemas, fields, and validation remain with the input module
and its integration contract.

Framework stages operate on source documents, source units, and source
references rather than format-specific structures. A source reference identifies
an ordered range of generic source units. Framework code preserves those ranges
and does not merge or rewrite them unless a stage module explicitly owns that
behavior. Every source unit carries a validated self-reference to its containing
document and its own unit ID.

Extract modules own artifact semantics, prompt use, response schemas, and
domain interpretation. Domain-specific concepts remain in the relevant module,
validator, shared domain helper, and artifact contract.

Physical centralization of LLM-facing content does not transfer semantic
ownership from those modules. Modules retain their manifests, response-schema
identities, prompt ordering, and registration, while reading only their scoped
content subtree. Generic framework code remains domain-neutral when it reads
its own scoped generic assets from the shared content container.

Typed artifact registrations declare one stable artifact kind and exact Go
type from extraction through merge, normalization, and semantic validation.
Pipeline resolution requires a compatible codec and matching kind-specific
variants before a typed lane can be accepted. Framework-owned erasure remains
private and must report type incompatibility as an error rather than a panic.

An artifact kind may additionally provide a typed evidence projection that
copies its direct generic source references. Preparation proves that projection
matches the artifact codec's exact Go type before retaining it for an output
policy. The runner reconstructs evidence only from accepted serialized
normalized artifacts, and the output boundary owns any resulting publication.
Generic framework code never infers evidence by inspecting domain JSON or
depends on domain artifact types.

Auxiliary references provide context or disambiguation. They are not source
evidence and must not be converted into source references.

## Pipeline Composition And Ownership

Module selection is configuration- and registry-driven. The framework resolves
named pipeline definitions, applies explicit defaults and runtime overrides,
and verifies module availability and capabilities before execution. Structural
pipeline choices must not be scattered through conditionals or hidden behind
ad hoc command flags.

Resolution validates every selected module and validator option set. A separate
preparation boundary then constructs the complete input, chunk, lane,
validation, and output implementation set in pipeline order. The runner accepts
only that prepared set, so construction and dependency failures occur before
source parsing or any other module operation.

Stage ownership is explicit:

- input modules convert external material into the generic source model;
- chunk modules partition source material for extraction;
- extract modules produce domain artifacts from chunks;
- merge modules combine accepted extraction outputs;
- normalize modules reconcile merged output;
- output modules encode accepted results and run outcomes into logical files.

Chunk modules produce source-addressed chunk plans rather than materialized
chunks. The framework validates and materializes those plans into the generic
chunk representation before chunk validation and lane execution. Plan reuse is
therefore independent of the configured pipeline, module options, references,
lanes, validators, and LLM profile: the canonical source digest selects the
plan, while the current run still applies its configured chunk validators to
the materialized chunks.

The framework owns orchestration, origin enrichment, aggregation, and handoff
provenance. Modules return logical results and classified diagnostics; they do
not own CLI reporting, physical output, cache, or
debug roots, durable file placement, or checkpoint and debug lifecycle.

After pipeline-wide chunking, extraction uses bounded framework concurrency.
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
chunk-first, lane-second order. A lane may begin its merge and normalize
continuation only after all of its extract jobs are terminal; that continuation
remains serial within the lane, while bounded continuations for different lanes
may overlap. The framework must not create unbounded goroutines per lane or
chunk.

Completion timing does not choose public ordering or errors. The coordinator
orders accepted artifacts, grouped diagnostics, rejections, checkpoint events, and
framework errors by stable pipeline scope. Rejections do not cancel unrelated
work. A framework error cancels derived work, prevents undispatched work from
starting, waits for started work, and prevents output encoding.

Warnings are process-only signals: configuration degradation, approved fallback,
or incomplete configured validation. Quality uncertainty and grounding findings
are advisories; successful canonicalization and cleanup are observations.
Modules choose that semantic classification, while the framework attaches
origin, aggregates groups, enforces bounds, and presents final collections.
An ordinary successful run therefore has zero warnings. See
[ADR-0015](../adr/0015-separate-process-warnings-from-quality-diagnostics.md)
for the decision rationale.

## Validation

Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output and
make an explicit whole-output decision: approve, reject, fail, or skip when a
runtime prerequisite is unavailable.

Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection.

The framework runs every applicable validator sequentially in configured order.
It aggregates rejections, exhausted validator failures, and skips before the
producer policy chooses a disposition. A completed rejection never advances.
With no rejection, an exhausted validator failure may fail the run or, under
the configured `warn_continue` policy, advance a structurally valid candidate
with explicit incomplete-validation provenance. Validators report findings;
they do not choose candidate disposition.

A completed rejection supplies a stable reason code for internal provenance
and bounded actionable correction guidance for the candidate producer. Reason
codes, validator keys, and operator-facing messages remain diagnostic data;
they are not model instructions. The framework constructs model-facing retry
text only from the semantic guidance and fails the contract rather than
inventing or truncating missing guidance.

An LLM-backed module may also request a feedback-aware retry when its own
deterministic translation or typed safety policy rejects a structurally valid
model response. It must supply model-facing guidance separately from its
reason code, operator message, and fallback diagnostics, together with the
exact `single_response_v1` candidate. A feedback-free module retry remains
valid when no exact candidate exists.

Default validator chains are production composition policy and are registered
centrally by stage and module. Configuration may replace a stage-local default,
including with an explicitly empty chain. Configured validator order is
authoritative; the framework must not silently reorder it.

## LLM Boundary

Modules and validators use transport-neutral structured completion contracts.
Provider request and response types, authentication, transport behavior, and
provider error adaptation remain inside the LLM runtime.

The caller of the LLM owns prompt selection, prompt inputs, response schema,
and interpretation of structured output. Provider adapters do not own source-
or domain-specific prompt logic.

PromptKit owns bounded structural correction within one structured completion.
Notarius owns outer stage attempts, semantic validation, and acceptance policy;
the two budgets must remain separate.

An LLM-backed producer can participate in semantic correction only when it
declares `single_response_v1` and returns the exact one response that directly
controlled its candidate. On an actionable validator rejection or
feedback-capable module retry, the framework rebuilds the ordinary request and
appends only the latest defective response as an `assistant` message plus one
aggregated `user` correction message. This is a fresh replacement request, not
a growing conversation. The retry budgets, terminal policy, and sensitive-data
rationale are recorded in
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).

When a model selects an application entity, callers must supply a contextual
selection and deterministically attach the opaque application identity whenever
the selection resolves exactly. Models do not receive or reproduce opaque
application identifiers. Semantic reconciliation may instead expose
contiguous, one-based candidate handles that exist only for one request;
deterministic code resolves them before typed application, and they never
become durable identity. This is the approved request-local-label application
of [ADR-0012](../adr/0012-resolve-opaque-entity-identifiers-deterministically.md)
recorded by
[ADR-0013](../adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md).

LLM calls and other external operations accept cancellation and respect
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
individual modules.

The application-wide LLM scheduler bounds actual provider calls independently
of framework worker limits. Every LLM-backed module, retry, and validator uses
the single injected scheduled client, including work performed by overlapping
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
beneath this mandatory application-wide scheduler.

## Configuration And Provenance

Configuration loading, precedence, defaults, environment overrides, redaction,
and validation are centralized. Named pipeline definitions make structural
composition explicit and discoverable. Operational overrides are permitted
when they do not obscure the configured pipeline structure.

Run preparation fails before stage execution when statically discoverable
modules, capabilities, reference bindings, or explicitly selected profiles are
invalid or incompatible.

Run manifests record enough resolved pipeline, module, source, reference, and
LLM provenance to make a run auditable after configuration changes. Manifests
record identities and bounded validation summaries rather than secret, raw
model, correction, or large payload content.

## State, Output, And Safety

Notarius exposes three filesystem surfaces with independent roots and
lifecycle:

- output is durable user data; output modules define logical files and the CLI
  owns their placement;
- cache is reconstructible state, with separate chunk-plan and checkpoint
  families; and
- debug is explicitly requested inspection data, combining a redacted summary
  with a detailed trace.

Chunk plans are keyed only by canonical source digest. Configured checkpoint
recording is independent of checkpoint reuse; checkpoints are loaded only for
an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots.

Only accepted, completely validated producer output is reusable checkpoint or
chunk-plan state. Rejected, structurally invalid, and validation-incomplete
results cannot become cache or checkpoint inputs, even when a
`warn_continue` result is allowed to advance in the current run. This
ineligibility follows derived merge and normalize results and generated
references for the remainder of the run: current-run handoff remains allowed,
but no dependent cache or checkpoint may be loaded or published.

Writes are atomic where practical. Paths for writes, moves, overwrites, and
deletion must be narrow and explicit. Notarius never automatically deletes
output or requested debug bundles; cache cleanup is explicit and recoverable.

Secrets must not appear in errors, logs, output, cache, debug summaries,
traces, manifests, documentation, examples, or redacted configuration. Raw
assistant responses and complete correction messages are attempt-local and are
excluded from ordinary durable records and summaries; the requested detailed
debug trace is the sole diagnostic surface allowed to retain them. Debug
collection is allowlisted to application-owned payloads and must not capture
unrelated process environment values or filesystem content. Trace data may
contain application data and therefore inherits its sensitivity; operators own
access controls and retention. Physical layout and operation are defined in
[Operations](../operations.md).

## Platform And Distribution

Linux is the supported deployment platform. macOS is supported only as a
best-effort development and compilation environment, while Windows is
unsupported. Notarius distributes source releases only: an immutable source
tag and its checked-in release note identify a release. The project does not
publish executable binaries, archives, installers, container images,
checksums, signatures, or package-manager entries. Maintainer release commands
and tag guards belong to [Source Releases](../release.md).

## Architectural Non-Goals

Notarius does not aim to provide:

- an arbitrary workflow graph or general workflow language;
- source-format or extraction-domain behavior in generic framework packages;
- provider-specific contracts exposed to modules;
- structural pipeline composition through ad hoc CLI flags;
- implicit cross-stage behavior that bypasses the fixed pipeline;
- abstractions introduced solely for hypothetical future complexity.
