notarius/docs/roadmap/domain.md

# Domain-Typed Pipeline Feature Roadmap

## Status

Decision-complete; implementation pending. This roadmap defines the desired end
state for [ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
[ADR-0004](../adr/0004-package-modules-by-domain.md). The work needed to reach
that state is owned by the
[domain pipeline implementation plan](implementation.md).

Until that plan is complete, current behavior remains defined by the
architecture, configuration, integration, operations, and internal
documentation outside `docs/roadmap/`.

## User Intent

Notarius should remain a small, explicit pipes-and-filters application while
making domain extensions safe to compose and straightforward to maintain. A
configured pipeline should fail before execution when its modules are
incompatible, should carry typed domain values rather than reparsed JSON between
artifact stages, and should preserve provenance and durable output contracts.

The application must also enforce one configurable, process-wide ceiling on
in-flight LLM calls. Pipeline scheduling may impose stricter limits, but no
stage, lane, retry, validator, or future LLM-backed extension may bypass that
global ceiling.

## Target Architecture

### Pipeline and outcome model

The topology remains:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

Input and chunk are pipeline-wide. Extract, merge, normalize, and their
validators operate per artifact lane. Output aggregates the terminal artifacts
from all lanes. The resolved pipeline retains explicit fields for those roles;
it does not become a general DAG or a heterogeneous ordered-stage list.

Framework errors abort the run. Validator rejection is a recorded domain
outcome and does not abort unrelated work. Accepted extract results reach merge
in source-chunk order, regardless of execution completion order. Warnings,
rejections, artifacts, and reported errors are likewise ordered by stable
pipeline scope rather than goroutine completion time.

### Engine-owned source model

The engine owns `source.SourceDocument`, `source.SourceUnit`, `source.SourceRef`,
and `source.Chunk`. Domain modules may consume these types but must not redefine
their provenance semantics.

- Every source unit has a canonical self-reference identifying its source and
  unit range.
- A chunk contains ordered source units and one canonical reference spanning
  its first through last unit.
- Chunk and unit references are validated for source identity, order, and
  containment.
- Auxiliary reference material remains distinct from source provenance.
- Cloning, canonicalization, checkpointing, debugging, and digest computation
  preserve the source references exactly.

`source.Chunk.Ref` replaces duplicate start/end boundary fields. Because that
changes persisted workspace state, the workspace checkpoint schema advances to
`notarius.workspace.v2`. Existing v1 checkpoints are left intact but treated as
incompatible and recomputed; no in-place migration or deletion is required.

### Typed artifact lanes

Each lane has one canonical artifact type `T` from extraction through merge,
normalization, and typed validation. Module-facing Zone-B contracts are generic:

- `Extractor[T]` produces typed per-chunk values plus framework-owned
  provenance and diagnostics;
- `Merger[T]` combines accepted values in source-chunk order;
- `Normalizer[T]` canonicalizes the merged value; and
- `TypedValidator[T]` applies semantic checks at its configured artifact stage.

The framework may use private erased adapters to keep heterogeneous lanes in
one resolved pipeline, but `any`, raw JSON, and a generic `Process(any)` API are
not module-facing handoffs. The extractor selected for a lane establishes its
artifact kind. Resolution uses that kind to select compatible merger,
normalizer, validator, and codec variants and rejects an incompatible lane
before any stage executes.

Generic strategies remain reusable without knowing concrete domains. In
particular, append-order merge is parameterized by a typed combine function
provided during domain registration, and no-op normalization is instantiated
for the lane's concrete type.

### Artifact identity and codecs

Every typed artifact kind has exactly one registered `ArtifactCodec[T]`. An
artifact kind is a stable logical identifier, separate from a Go type name. A
codec owns:

- artifact kind;
- schema identifier, name, and version;
- media type and JSON Schema bytes; and
- strict, deterministic encoding and decoding between `T` and the serialized
  representation.

Equal canonical values must encode to equal bytes. Those bytes are the basis
for artifact digests. Codec decoding rejects malformed or schema-incompatible
content. Domain validators continue to own semantic validity; codecs do not
replace them.

`SerializedArtifact` is the Zone-C representation and includes the artifact
kind, schema metadata, media type, encoded content, and framework metadata.
Type erasure occurs through the codec after normalization for final output.
Intermediate checkpointing and opt-in debug recording may also use the codec,
but serialization for those side effects is not a stage handoff.

Checkpoint metadata records artifact kind, schema identifier, schema version,
and schema digest. Reuse requires an exact compatible registered codec;
otherwise the checkpoint is safely invalidated. Output remains domain-neutral
and consumes serialized artifacts.

Generic serialized validators remain supported for representation-level checks
such as valid JSON and JSON Schema validation. The framework encodes `T` through
its registered codec before invoking them. Domain validators receive `T`
directly. Chunk-stage validators remain in the source zone: semantic chunk
validators receive engine-owned chunks, while representation-level validators
receive the framework's canonical serialized chunk view. They do not force
source-zone values through a domain artifact codec.

### Registration and resolution

Registries expose typed registration helpers while privately retaining the Go
type identity needed to assemble erased lane executors.

- Codecs are keyed by artifact kind, with exactly one codec per kind.
- Extractors are keyed by their existing module key and declare an artifact
  kind.
- Mergers, normalizers, and validators are keyed by `(module key, artifact
  kind)`, allowing stable generic keys such as `appendorder` and `noop` to have
  multiple typed specializations.
- Resolved pipeline identity and dependency fingerprints include artifact kind
  and schema identity, version, and digest.
- Duplicate or incompatible registrations and selections fail deterministically
  during composition or resolution.

The framework's public typed registration surface uses free generic functions,
because Go methods cannot declare their own type parameters. Private reflection
may verify and erase registered types, but it is not exposed to module authors.

### Preparation, options, and dependencies

Pipeline execution is split into resolution, preparation, and running.
Preparation constructs every selected module and validator before source input
begins and returns a prepared pipeline with explicit input, chunk, lane, and
output fields.

- Construction receives framework-owned dependencies, including the one shared
  scheduled structured-LLM client.
- Raw configured options are decoded once into implementation-owned option
  structs during preparation.
- Missing dependencies, malformed options, unknown options, and incompatible
  typed selections fail before stage execution.
- Configuration validation uses the same option decoders without requiring live
  provider dependencies.
- Per-run data such as sources, chunks, references, session identity, lane
  identity, and metadata remains in operation requests.
- Constructed implementations that can be scheduled concurrently are immutable
  after preparation or otherwise explicitly concurrency-safe.

Modules and validators must not construct provider clients, wrap their own
independent schedulers, or bypass the injected scheduled client.

### D&D artifact model

The D&D package root owns canonical `dnd.SpellList`, `dnd.SpellCast`, and
related evidence types, using engine-owned `source.SourceRef` values. The spell
extractor keeps its LLM response DTO and response schema private and maps the
canonicalized response to the domain model.

The D&D spell codec separately owns the existing durable spell artifact schema.
The LLM response schema and durable artifact schema remain distinct contracts
even if their current JSON shapes are similar. Shape, source-reference, and
source-relatedness validators operate on the canonical typed model. The
validator-only duplicate spell model and inter-stage JSON reparsing disappear.

The migration preserves the existing D&D spell payload, logical output bundle,
module and validator keys, prompt/schema identities, default validator chains,
warnings, rejection semantics, and manifest provenance unless a separate
compatibility decision explicitly changes one of those contracts.

## Package Ownership

The target production extension layout is:

```text
internal/modules/dnd/
  types.go
  codec/spells/
  chunk/scenes/
  extract/spells/
  validate/spells/shape/
  validate/spells/source_refs/
  validate/spells/source_relatedness/
  shared/
  register/

internal/modules/generic/
  chunk/units/
  merge/appendorder/
  normalize/noop/
  validate/always_accept/
  validate/always_reject/
  validate/valid_json/
  validate/valid_json_schema/
  output/json/
  register/

internal/modules/seriatim/
  input/transcript/
  register/

internal/framework/promptfs/
```

Shared domain types live at the domain root. Registration lives in a sibling
`register` package so that the root never imports child implementations. Each
registrar exposes one composition entry point accepting the pipeline registry
set and LLM asset registry. The CLI composition root creates those registries
and invokes the generic, Seriatim, and D&D registrars.

Concrete domain implementations do not import peer domains. Generic extensions
never import a concrete domain. A domain registrar may import generic packages
to register typed specializations for its domain. The application composition
root and designated black-box integration tests may compose multiple
registrars. Domain-neutral embedded prompt-asset filesystem support belongs to
the framework rather than a domain package.

## Concurrency Policy

### Configuration

The existing `concurrency.total_llm` setting remains the application-wide
ceiling on actual provider calls. Version-2 configuration gains an extensible
stage-worker map:

```yaml
concurrency:
  total_llm: 4
  stage_workers:
    extract: 4
```

Initially, `extract` is the only recognized key. Unknown stage keys are rejected
so misspellings cannot silently alter scheduling. If omitted, the effective
extract worker count equals `total_llm`. Its valid range is
`1..concurrency.total_llm`. The environment override is
`NOTARIUS_STAGE_WORKERS_EXTRACT`; future stage overrides receive similarly
explicit names that map to the extensible file representation.

The worker setting bounds framework jobs, not provider calls. Only an actual
LLM call consumes a permit from the shared scheduled client. The global
scheduled client remains authoritative even if future stages gain worker limits.

### Scheduling

After pipeline-wide chunking, all lanes may run concurrently. A central
dispatcher submits `(lane, chunk)` jobs to one run-wide extract worker pool in
round-robin order: source chunk first, then resolved lane order. This avoids one
unbounded goroutine per job and prevents an early lane from monopolizing the
queue.

One job contains extraction, its stage-local retry behavior, and extract-stage
validation for that lane and chunk. Each lane begins its serial merge then
normalize continuation when all of its extract jobs reach a terminal state.
Different lanes' continuations may overlap, and any LLM-backed continuation or
validator still shares the global scheduled client.

Workers publish immutable task results to a coordinator. Only the coordinator
mutates aggregate results, manifests, checkpoint indexes, warnings, and
rejections. Debug artifacts use attempt-specific paths and do not rely on
concurrent writes to shared files.

Rejections do not cancel work. A framework error cancels the derived run
context, stops undispatched jobs, and waits for started jobs to finish or
observe cancellation. If the parent context was canceled, its error is
returned. Otherwise, internal cancellation errors are ignored when at least one
real framework error exists, and the primary returned error is selected from
all started-task framework errors by this stable ordering:

1. stage order: extract, merge, then normalize;
2. resolved lane order;
3. source chunk index for chunk-scoped work; and
4. configured validator or operation order within that scope.

The full per-task errors may be retained in opt-in diagnostics, but completion
timing never chooses the public error. Output runs only after every lane reaches
a successful or rejection-only terminal state and no framework error exists.

Extractors and any validator instance callable by multiple workers must be safe
for concurrent use. Production implementations should normally satisfy this by
being immutable after preparation.

## Compatibility and Safety

- Existing production module keys, validator keys, profiles, default chains,
  and maintained configurations continue to resolve.
- Existing durable D&D JSON content and logical output paths remain unchanged.
- Framework errors, validator rejections, retries, checkpoints, diagnostics,
  and debug behavior retain their current semantics except for the explicitly
  documented workspace-v2 compatibility boundary and deterministic concurrent
  ordering.
- All provider calls pass through the shared global scheduler, across lanes,
  stages, retries, and validators.
- Source text and LLM payloads remain subject to the existing opt-in debug and
  sensitive-data handling policies.
- Package moves do not create user-visible key changes or concrete cross-domain
  dependencies.

## Non-Goals

This feature does not introduce:

- a general DAG or configurable stage topology;
- out-of-process plugins or an RPC extension protocol;
- cross-lane normalization;
- a new durable D&D spell schema merely to mirror internal Go types;
- per-stage LLM permit pools that could exceed or partition the global ceiling;
  or
- concurrent work implemented through an unbounded goroutine per lane or chunk.

## Completion Criteria

The target state is reached when:

- all production lanes use one typed artifact from extract through normalize
  and typed validation;
- codecs own schema-aware serialization at every type-erasure, checkpoint, and
  debug boundary;
- incompatible lane composition and invalid options fail before source work;
- source units and chunks carry validated canonical provenance;
- production extensions follow the domain-first package and registrar rules;
- extract scheduling is bounded, deterministic, concurrent across lanes, and
  race-free;
- instrumented tests prove actual concurrent LLM calls never exceed
  `concurrency.total_llm` across all callers;
- the maintained D&D example and compatibility baselines retain their durable
  contracts; and
- current-behavior documentation is updated as each implemented boundary lands.