notarius/docs/adr/0003-typed-interfaces-with-two-zone-data-model.md

ADR-0003: Strongly typed stage interfaces with a two-zone data model

Status: Accepted Date: 2026-07-13

Context

Pipeline stages must exchange source data and extracted artifacts. Universal source data has one engine-wide meaning, while extracted artifacts have domain-specific shapes. Passing opaque bytes or any between all stages would make invalid wiring and merge behavior runtime concerns. Requiring JSON at every handoff would preserve interoperability but discard useful Go type safety while all modules are in-process.

The framework must also support multiple configured artifact domains, durable checkpoints, diagnostics, and output encoders without making those consumers depend on every domain's Go types.

Decision

Notarius uses two typed data zones followed by one serialized boundary.

Source zone

Input and chunk stages use conservative, engine-owned document, segment, chunk, and source-reference types. Their exact Go names are implementation details. Every segment carries engine-owned source provenance identifying the source location from which it was produced. Chunks preserve the ordered provenance of their segments.

Source-format-specific fields remain in input modules or explicitly namespaced metadata; they do not become framework contracts.

Domain artifact zone

Each artifact lane has one domain-owned Go artifact type T. Its extract, merge, normalize, and domain-aware validation implementations use generic, strongly typed contracts over the same T. Raw JSON, opaque bytes, and any are not stage-handoff contracts within a lane.

Each registered domain artifact type supplies a codec for T. The codec owns:

• stable schema identity and an explicit schema version;
• JSON serialization and deserialization;
• the media type and schema metadata required at serialized boundaries; and
• rejection of data that cannot be represented by the declared artifact schema.

An artifact type's JSON representation is a maintained domain contract. Changing it incompatibly requires a new schema version.

Extract, merge, and normalize may change the contents of T, but they do not change the lane's canonical Go artifact type or artifact schema identity. An extractor maps any provider- or prompt-specific response type into T before returning. A future lane that requires different artifact types at different stages requires a new architectural decision.

Serialized boundary

After normalization, each typed artifact is converted into an engine-owned serialized artifact containing bytes, media type, and schema metadata. Output aggregation and output encoders consume this type-erased form. Intermediate checkpoint and debug encodings do not become stage-handoff contracts.

LLM transport, checkpoints, and opt-in debug recording are also explicit serialization boundaries. They may encode or decode a typed artifact through its domain codec, but they do not change the in-memory type used between extract, merge, normalize, and typed validators. Checkpoint reuse requires a compatible schema identity and version.

An LLM structured-response schema is a module transport contract and may differ from the domain artifact schema. The calling module owns the response type and maps it into the canonical T; the artifact codec remains authoritative for artifact checkpoints and output serialization.

The framework may use private type-erased adapters to store heterogeneous lane registrations and execute configured domains. Such an adapter must assemble a type-consistent lane before execution and must not expose any or raw payloads as module-facing handoffs inside the domain artifact zone.

Construction and dependencies

Every module operation accepts context.Context. Modules receive stable runtime collaborators through an injected dependency set at construction time. In particular, LLM-using modules receive the application-provided structured LLM client and do not construct provider clients or bypass shared scheduling.

The application boundary enforces one configurable global upper bound on in-flight LLM calls across all stages, lanes, retries, and validators.

Configuration options are parsed and validated while a module is constructed, before that module executes. Per-run data such as source material, references, session identity, and lane identity remains operation input rather than a construction dependency.

Alternatives considered

• Pass raw bytes between stages. This maximizes decoupling but moves wiring, parsing, and merge errors to runtime and prevents domain types from being the canonical in-process contract.
• Require JSON plus schemas at every stage boundary. This is appropriate for an out-of-process boundary, but adds serialization and parsing inside the current in-process pipeline. The stable codec contract preserves this upgrade path if remote plugins are introduced.
• Use a uniform Process(any) (any, error) contract. This simplifies a fully dynamic engine but turns incompatible module composition into type assertions and runtime failures. The fixed topology does not require that tradeoff.

Consequences

Domain pipelines gain compile-time handoff safety and explicit merge semantics. Serialization, schema compatibility, checkpoint decoding, and output erasure have named owners. Dynamic registration requires a small erased adapter around each typed lane, and generic stage implementations must be instantiated for a specific artifact type or behavior rather than manipulating arbitrary JSON.

The engine-owned source model becomes a long-lived contract and must evolve conservatively. Domain authors must maintain a codec and versioned schema in addition to their Go artifact type.