notarius/docs/roadmap/domain.md

Domain-Typed Pipeline Feature Roadmap

Status

Decision-complete; implementation pending. This roadmap defines the desired end state for ADR-0002, ADR-0003, and ADR-0004. The work needed to reach that state is owned by the domain pipeline implementation plan.

Until that plan is complete, current behavior remains defined by the architecture, configuration, integration, operations, and internal documentation outside docs/roadmap/.

User Intent

Notarius should remain a small, explicit pipes-and-filters application while making domain extensions safe to compose and straightforward to maintain. A configured pipeline should fail before execution when its modules are incompatible, should carry typed domain values rather than reparsed JSON between artifact stages, and should preserve provenance and durable output contracts.

The application must also enforce one configurable, process-wide ceiling on in-flight LLM calls. Pipeline scheduling may impose stricter limits, but no stage, lane, retry, validator, or future LLM-backed extension may bypass that global ceiling.

Target Architecture

Pipeline and outcome model

The topology remains:

input -> chunk -> extract -> merge -> normalize -> output

Input and chunk are pipeline-wide. Extract, merge, normalize, and their validators operate per artifact lane. Output aggregates the terminal artifacts from all lanes. The resolved pipeline retains explicit fields for those roles; it does not become a general DAG or a heterogeneous ordered-stage list.

Framework errors abort the run. Validator rejection is a recorded domain outcome and does not abort unrelated work. Accepted extract results reach merge in source-chunk order, regardless of execution completion order. Warnings, rejections, artifacts, and reported errors are likewise ordered by stable pipeline scope rather than goroutine completion time.

Engine-owned source model

The engine owns source.SourceDocument, source.SourceUnit, source.SourceRef, and source.Chunk. Domain modules may consume these types but must not redefine their provenance semantics.

• Every source unit has a canonical self-reference identifying its source and unit range.
• A chunk contains ordered source units and one canonical reference spanning its first through last unit.
• Chunk and unit references are validated for source identity, order, and containment.
• Auxiliary reference material remains distinct from source provenance.
• Cloning, canonicalization, checkpointing, debugging, and digest computation preserve the source references exactly.

source.Chunk.Ref replaces duplicate start/end boundary fields. Because that changes persisted workspace state, the workspace checkpoint schema advances to notarius.workspace.v2. Existing v1 checkpoints are left intact but treated as incompatible and recomputed; no in-place migration or deletion is required.

Typed artifact lanes

Each lane has one canonical artifact type T from extraction through merge, normalization, and typed validation. Module-facing Zone-B contracts are generic:

• Extractor[T] produces typed per-chunk values plus framework-owned provenance and diagnostics;
• Merger[T] combines accepted values in source-chunk order;
• Normalizer[T] canonicalizes the merged value; and
• TypedValidator[T] applies semantic checks at its configured artifact stage.

The framework may use private erased adapters to keep heterogeneous lanes in one resolved pipeline, but any, raw JSON, and a generic Process(any) API are not module-facing handoffs. The extractor selected for a lane establishes its artifact kind. Resolution uses that kind to select compatible merger, normalizer, validator, and codec variants and rejects an incompatible lane before any stage executes.

Generic strategies remain reusable without knowing concrete domains. In particular, append-order merge is parameterized by a typed combine function provided during domain registration, and no-op normalization is instantiated for the lane's concrete type.

Artifact identity and codecs

Every typed artifact kind has exactly one registered ArtifactCodec[T]. An artifact kind is a stable logical identifier, separate from a Go type name. A codec owns:

• artifact kind;
• schema identifier, name, and version;
• media type and JSON Schema bytes; and
• strict, deterministic encoding and decoding between T and the serialized representation.

Equal canonical values must encode to equal bytes. Those bytes are the basis for artifact digests. Codec decoding rejects malformed or schema-incompatible content. Domain validators continue to own semantic validity; codecs do not replace them.

SerializedArtifact is the Zone-C representation and includes the artifact kind, schema metadata, media type, encoded content, and framework metadata. Type erasure occurs through the codec after normalization for final output. Intermediate checkpointing and opt-in debug recording may also use the codec, but serialization for those side effects is not a stage handoff.

Checkpoint metadata records artifact kind, schema identifier, schema version, and schema digest. Reuse requires an exact compatible registered codec; otherwise the checkpoint is safely invalidated. Output remains domain-neutral and consumes serialized artifacts.

Generic serialized validators remain supported for representation-level checks such as valid JSON and JSON Schema validation. The framework encodes T through its registered codec before invoking them. Domain validators receive T directly. Chunk-stage validators remain in the source zone: semantic chunk validators receive engine-owned chunks, while representation-level validators receive the framework's canonical serialized chunk view. They do not force source-zone values through a domain artifact codec.

Registration and resolution

Registries expose typed registration helpers while privately retaining the Go type identity needed to assemble erased lane executors.

• Codecs are keyed by artifact kind, with exactly one codec per kind.
• Extractors are keyed by their existing module key and declare an artifact kind.
• Mergers, normalizers, and validators are keyed by (module key, artifact kind), allowing stable generic keys such as appendorder and noop to have multiple typed specializations.
• Resolved pipeline identity and dependency fingerprints include artifact kind and schema identity, version, and digest.
• Duplicate or incompatible registrations and selections fail deterministically during composition or resolution.

The framework's public typed registration surface uses free generic functions, because Go methods cannot declare their own type parameters. Private reflection may verify and erase registered types, but it is not exposed to module authors.

Preparation, options, and dependencies

Pipeline execution is split into resolution, preparation, and running. Preparation constructs every selected module and validator before source input begins and returns a prepared pipeline with explicit input, chunk, lane, and output fields.

• Construction receives framework-owned dependencies, including the one shared scheduled structured-LLM client.
• Raw configured options are decoded once into implementation-owned option structs during preparation.
• Missing dependencies, malformed options, unknown options, and incompatible typed selections fail before stage execution.
• Configuration validation uses the same option decoders without requiring live provider dependencies.
• Per-run data such as sources, chunks, references, session identity, lane identity, and metadata remains in operation requests.
• Constructed implementations that can be scheduled concurrently are immutable after preparation or otherwise explicitly concurrency-safe.

Modules and validators must not construct provider clients, wrap their own independent schedulers, or bypass the injected scheduled client.

D&D artifact model

The D&D package root owns canonical dnd.SpellList, dnd.SpellCast, and related evidence types, using engine-owned source.SourceRef values. The spell extractor keeps its LLM response DTO and response schema private and maps the canonicalized response to the domain model.

The D&D spell codec separately owns the existing durable spell artifact schema. The LLM response schema and durable artifact schema remain distinct contracts even if their current JSON shapes are similar. Shape, source-reference, and source-relatedness validators operate on the canonical typed model. The validator-only duplicate spell model and inter-stage JSON reparsing disappear.

The migration preserves the existing D&D spell payload, logical output bundle, module and validator keys, prompt/schema identities, default validator chains, warnings, rejection semantics, and manifest provenance unless a separate compatibility decision explicitly changes one of those contracts.

Package Ownership

The target production extension layout is:

internal/modules/dnd/
  types.go
  codec/spells/
  chunk/scenes/
  extract/spells/
  validate/spells/shape/
  validate/spells/source_refs/
  validate/spells/source_relatedness/
  shared/
  register/

internal/modules/generic/
  chunk/units/
  merge/appendorder/
  normalize/noop/
  validate/always_accept/
  validate/always_reject/
  validate/valid_json/
  validate/valid_json_schema/
  output/json/
  register/

internal/modules/seriatim/
  input/transcript/
  register/

internal/framework/promptfs/

Shared domain types live at the domain root. Registration lives in a sibling register package so that the root never imports child implementations. Each registrar exposes one composition entry point accepting the pipeline registry set and LLM asset registry. The CLI composition root creates those registries and invokes the generic, Seriatim, and D&D registrars.

Concrete domain implementations do not import peer domains. Generic extensions never import a concrete domain. A domain registrar may import generic packages to register typed specializations for its domain. The application composition root and designated black-box integration tests may compose multiple registrars. Domain-neutral embedded prompt-asset filesystem support belongs to the framework rather than a domain package.

Concurrency Policy

Configuration

The existing concurrency.total_llm setting remains the application-wide ceiling on actual provider calls. Version-2 configuration gains an extensible stage-worker map:

concurrency:
  total_llm: 4
  stage_workers:
    extract: 4

Initially, extract is the only recognized key. Unknown stage keys are rejected so misspellings cannot silently alter scheduling. If omitted, the effective extract worker count equals total_llm. Its valid range is 1..concurrency.total_llm. The environment override is NOTARIUS_STAGE_WORKERS_EXTRACT; future stage overrides receive similarly explicit names that map to the extensible file representation.

The worker setting bounds framework jobs, not provider calls. Only an actual LLM call consumes a permit from the shared scheduled client. The global scheduled client remains authoritative even if future stages gain worker limits.

Scheduling

After pipeline-wide chunking, all lanes may run concurrently. A central dispatcher submits (lane, chunk) jobs to one run-wide extract worker pool in round-robin order: source chunk first, then resolved lane order. This avoids one unbounded goroutine per job and prevents an early lane from monopolizing the queue.

One job contains extraction, its stage-local retry behavior, and extract-stage validation for that lane and chunk. Each lane begins its serial merge then normalize continuation when all of its extract jobs reach a terminal state. Different lanes' continuations may overlap, and any LLM-backed continuation or validator still shares the global scheduled client.

Workers publish immutable task results to a coordinator. Only the coordinator mutates aggregate results, manifests, checkpoint indexes, warnings, and rejections. Debug artifacts use attempt-specific paths and do not rely on concurrent writes to shared files.

Rejections do not cancel work. A framework error cancels the derived run context, stops undispatched jobs, and waits for started jobs to finish or observe cancellation. If the parent context was canceled, its error is returned. Otherwise, internal cancellation errors are ignored when at least one real framework error exists, and the primary returned error is selected from all started-task framework errors by this stable ordering:

1. stage order: extract, merge, then normalize;
2. resolved lane order;
3. source chunk index for chunk-scoped work; and
4. configured validator or operation order within that scope.

The full per-task errors may be retained in opt-in diagnostics, but completion timing never chooses the public error. Output runs only after every lane reaches a successful or rejection-only terminal state and no framework error exists.

Extractors and any validator instance callable by multiple workers must be safe for concurrent use. Production implementations should normally satisfy this by being immutable after preparation.

Compatibility and Safety

• Existing production module keys, validator keys, profiles, default chains, and maintained configurations continue to resolve.
• Existing durable D&D JSON content and logical output paths remain unchanged.
• Framework errors, validator rejections, retries, checkpoints, diagnostics, and debug behavior retain their current semantics except for the explicitly documented workspace-v2 compatibility boundary and deterministic concurrent ordering.
• All provider calls pass through the shared global scheduler, across lanes, stages, retries, and validators.
• Source text and LLM payloads remain subject to the existing opt-in debug and sensitive-data handling policies.
• Package moves do not create user-visible key changes or concrete cross-domain dependencies.

Non-Goals

This feature does not introduce:

• a general DAG or configurable stage topology;
• out-of-process plugins or an RPC extension protocol;
• cross-lane normalization;
• a new durable D&D spell schema merely to mirror internal Go types;
• per-stage LLM permit pools that could exceed or partition the global ceiling; or
• concurrent work implemented through an unbounded goroutine per lane or chunk.

Completion Criteria

The target state is reached when:

• all production lanes use one typed artifact from extract through normalize and typed validation;
• codecs own schema-aware serialization at every type-erasure, checkpoint, and debug boundary;
• incompatible lane composition and invalid options fail before source work;
• source units and chunks carry validated canonical provenance;
• production extensions follow the domain-first package and registrar rules;
• extract scheduling is bounded, deterministic, concurrent across lanes, and race-free;
• instrumented tests prove actual concurrent LLM calls never exceed concurrency.total_llm across all callers;
• the maintained D&D example and compatibility baselines retain their durable contracts; and
• current-behavior documentation is updated as each implemented boundary lands.