Add documentation and roadmap for a significant refactor around domain-focused module packages

2026-07-16 23:36:46 -05:00
parent 21888d625f
commit 35f9446ed8
6 changed files with 1440 additions and 1 deletions
--- a/docs/roadmap/domain.md
+++ b/docs/roadmap/domain.md
@@ -0,0 +1,348 @@
+# Domain-Typed Pipeline Feature Roadmap
+
+## Status
+
+Decision-complete; implementation pending. This roadmap defines the desired end
+state for [ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
+[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
+[ADR-0004](../adr/0004-package-modules-by-domain.md). The work needed to reach
+that state is owned by the
+[domain pipeline implementation plan](implementation.md).
+
+Until that plan is complete, current behavior remains defined by the
+architecture, configuration, integration, operations, and internal
+documentation outside `docs/roadmap/`.
+
+## User Intent
+
+Notarius should remain a small, explicit pipes-and-filters application while
+making domain extensions safe to compose and straightforward to maintain. A
+configured pipeline should fail before execution when its modules are
+incompatible, should carry typed domain values rather than reparsed JSON between
+artifact stages, and should preserve provenance and durable output contracts.
+
+The application must also enforce one configurable, process-wide ceiling on
+in-flight LLM calls. Pipeline scheduling may impose stricter limits, but no
+stage, lane, retry, validator, or future LLM-backed extension may bypass that
+global ceiling.
+
+## Target Architecture
+
+### Pipeline and outcome model
+
+The topology remains:
+
+```text
+input -> chunk -> extract -> merge -> normalize -> output
+```
+
+Input and chunk are pipeline-wide. Extract, merge, normalize, and their
+validators operate per artifact lane. Output aggregates the terminal artifacts
+from all lanes. The resolved pipeline retains explicit fields for those roles;
+it does not become a general DAG or a heterogeneous ordered-stage list.
+
+Framework errors abort the run. Validator rejection is a recorded domain
+outcome and does not abort unrelated work. Accepted extract results reach merge
+in source-chunk order, regardless of execution completion order. Warnings,
+rejections, artifacts, and reported errors are likewise ordered by stable
+pipeline scope rather than goroutine completion time.
+
+### Engine-owned source model
+
+The engine owns `source.SourceDocument`, `source.SourceUnit`, `source.SourceRef`,
+and `source.Chunk`. Domain modules may consume these types but must not redefine
+their provenance semantics.
+
+- Every source unit has a canonical self-reference identifying its source and
+  unit range.
+- A chunk contains ordered source units and one canonical reference spanning
+  its first through last unit.
+- Chunk and unit references are validated for source identity, order, and
+  containment.
+- Auxiliary reference material remains distinct from source provenance.
+- Cloning, canonicalization, checkpointing, debugging, and digest computation
+  preserve the source references exactly.
+
+`source.Chunk.Ref` replaces duplicate start/end boundary fields. Because that
+changes persisted workspace state, the workspace checkpoint schema advances to
+`notarius.workspace.v2`. Existing v1 checkpoints are left intact but treated as
+incompatible and recomputed; no in-place migration or deletion is required.
+
+### Typed artifact lanes
+
+Each lane has one canonical artifact type `T` from extraction through merge,
+normalization, and typed validation. Module-facing Zone-B contracts are generic:
+
+- `Extractor[T]` produces typed per-chunk values plus framework-owned
+  provenance and diagnostics;
+- `Merger[T]` combines accepted values in source-chunk order;
+- `Normalizer[T]` canonicalizes the merged value; and
+- `TypedValidator[T]` applies semantic checks at its configured artifact stage.
+
+The framework may use private erased adapters to keep heterogeneous lanes in
+one resolved pipeline, but `any`, raw JSON, and a generic `Process(any)` API are
+not module-facing handoffs. The extractor selected for a lane establishes its
+artifact kind. Resolution uses that kind to select compatible merger,
+normalizer, validator, and codec variants and rejects an incompatible lane
+before any stage executes.
+
+Generic strategies remain reusable without knowing concrete domains. In
+particular, append-order merge is parameterized by a typed combine function
+provided during domain registration, and no-op normalization is instantiated
+for the lane's concrete type.
+
+### Artifact identity and codecs
+
+Every typed artifact kind has exactly one registered `ArtifactCodec[T]`. An
+artifact kind is a stable logical identifier, separate from a Go type name. A
+codec owns:
+
+- artifact kind;
+- schema identifier, name, and version;
+- media type and JSON Schema bytes; and
+- strict, deterministic encoding and decoding between `T` and the serialized
+  representation.
+
+Equal canonical values must encode to equal bytes. Those bytes are the basis
+for artifact digests. Codec decoding rejects malformed or schema-incompatible
+content. Domain validators continue to own semantic validity; codecs do not
+replace them.
+
+`SerializedArtifact` is the Zone-C representation and includes the artifact
+kind, schema metadata, media type, encoded content, and framework metadata.
+Type erasure occurs through the codec after normalization for final output.
+Intermediate checkpointing and opt-in debug recording may also use the codec,
+but serialization for those side effects is not a stage handoff.
+
+Checkpoint metadata records artifact kind, schema identifier, schema version,
+and schema digest. Reuse requires an exact compatible registered codec;
+otherwise the checkpoint is safely invalidated. Output remains domain-neutral
+and consumes serialized artifacts.
+
+Generic serialized validators remain supported for representation-level checks
+such as valid JSON and JSON Schema validation. The framework encodes `T` through
+its registered codec before invoking them. Domain validators receive `T`
+directly. Chunk-stage validators remain in the source zone: semantic chunk
+validators receive engine-owned chunks, while representation-level validators
+receive the framework's canonical serialized chunk view. They do not force
+source-zone values through a domain artifact codec.
+
+### Registration and resolution
+
+Registries expose typed registration helpers while privately retaining the Go
+type identity needed to assemble erased lane executors.
+
+- Codecs are keyed by artifact kind, with exactly one codec per kind.
+- Extractors are keyed by their existing module key and declare an artifact
+  kind.
+- Mergers, normalizers, and validators are keyed by `(module key, artifact
+  kind)`, allowing stable generic keys such as `appendorder` and `noop` to have
+  multiple typed specializations.
+- Resolved pipeline identity and dependency fingerprints include artifact kind
+  and schema identity, version, and digest.
+- Duplicate or incompatible registrations and selections fail deterministically
+  during composition or resolution.
+
+The framework's public typed registration surface uses free generic functions,
+because Go methods cannot declare their own type parameters. Private reflection
+may verify and erase registered types, but it is not exposed to module authors.
+
+### Preparation, options, and dependencies
+
+Pipeline execution is split into resolution, preparation, and running.
+Preparation constructs every selected module and validator before source input
+begins and returns a prepared pipeline with explicit input, chunk, lane, and
+output fields.
+
+- Construction receives framework-owned dependencies, including the one shared
+  scheduled structured-LLM client.
+- Raw configured options are decoded once into implementation-owned option
+  structs during preparation.
+- Missing dependencies, malformed options, unknown options, and incompatible
+  typed selections fail before stage execution.
+- Configuration validation uses the same option decoders without requiring live
+  provider dependencies.
+- Per-run data such as sources, chunks, references, session identity, lane
+  identity, and metadata remains in operation requests.
+- Constructed implementations that can be scheduled concurrently are immutable
+  after preparation or otherwise explicitly concurrency-safe.
+
+Modules and validators must not construct provider clients, wrap their own
+independent schedulers, or bypass the injected scheduled client.
+
+### D&D artifact model
+
+The D&D package root owns canonical `dnd.SpellList`, `dnd.SpellCast`, and
+related evidence types, using engine-owned `source.SourceRef` values. The spell
+extractor keeps its LLM response DTO and response schema private and maps the
+canonicalized response to the domain model.
+
+The D&D spell codec separately owns the existing durable spell artifact schema.
+The LLM response schema and durable artifact schema remain distinct contracts
+even if their current JSON shapes are similar. Shape, source-reference, and
+source-relatedness validators operate on the canonical typed model. The
+validator-only duplicate spell model and inter-stage JSON reparsing disappear.
+
+The migration preserves the existing D&D spell payload, logical output bundle,
+module and validator keys, prompt/schema identities, default validator chains,
+warnings, rejection semantics, and manifest provenance unless a separate
+compatibility decision explicitly changes one of those contracts.
+
+## Package Ownership
+
+The target production extension layout is:
+
+```text
+internal/modules/dnd/
+  types.go
+  codec/spells/
+  chunk/scenes/
+  extract/spells/
+  validate/spells/shape/
+  validate/spells/source_refs/
+  validate/spells/source_relatedness/
+  shared/
+  register/
+
+internal/modules/generic/
+  chunk/units/
+  merge/appendorder/
+  normalize/noop/
+  validate/always_accept/
+  validate/always_reject/
+  validate/valid_json/
+  validate/valid_json_schema/
+  output/json/
+  register/
+
+internal/modules/seriatim/
+  input/transcript/
+  register/
+
+internal/framework/promptfs/
+```
+
+Shared domain types live at the domain root. Registration lives in a sibling
+`register` package so that the root never imports child implementations. Each
+registrar exposes one composition entry point accepting the pipeline registry
+set and LLM asset registry. The CLI composition root creates those registries
+and invokes the generic, Seriatim, and D&D registrars.
+
+Concrete domain implementations do not import peer domains. Generic extensions
+never import a concrete domain. A domain registrar may import generic packages
+to register typed specializations for its domain. The application composition
+root and designated black-box integration tests may compose multiple
+registrars. Domain-neutral embedded prompt-asset filesystem support belongs to
+the framework rather than a domain package.
+
+## Concurrency Policy
+
+### Configuration
+
+The existing `concurrency.total_llm` setting remains the application-wide
+ceiling on actual provider calls. Version-2 configuration gains an extensible
+stage-worker map:
+
+```yaml
+concurrency:
+  total_llm: 4
+  stage_workers:
+    extract: 4
+```
+
+Initially, `extract` is the only recognized key. Unknown stage keys are rejected
+so misspellings cannot silently alter scheduling. If omitted, the effective
+extract worker count equals `total_llm`. Its valid range is
+`1..concurrency.total_llm`. The environment override is
+`NOTARIUS_STAGE_WORKERS_EXTRACT`; future stage overrides receive similarly
+explicit names that map to the extensible file representation.
+
+The worker setting bounds framework jobs, not provider calls. Only an actual
+LLM call consumes a permit from the shared scheduled client. The global
+scheduled client remains authoritative even if future stages gain worker limits.
+
+### Scheduling
+
+After pipeline-wide chunking, all lanes may run concurrently. A central
+dispatcher submits `(lane, chunk)` jobs to one run-wide extract worker pool in
+round-robin order: source chunk first, then resolved lane order. This avoids one
+unbounded goroutine per job and prevents an early lane from monopolizing the
+queue.
+
+One job contains extraction, its stage-local retry behavior, and extract-stage
+validation for that lane and chunk. Each lane begins its serial merge then
+normalize continuation when all of its extract jobs reach a terminal state.
+Different lanes' continuations may overlap, and any LLM-backed continuation or
+validator still shares the global scheduled client.
+
+Workers publish immutable task results to a coordinator. Only the coordinator
+mutates aggregate results, manifests, checkpoint indexes, warnings, and
+rejections. Debug artifacts use attempt-specific paths and do not rely on
+concurrent writes to shared files.
+
+Rejections do not cancel work. A framework error cancels the derived run
+context, stops undispatched jobs, and waits for started jobs to finish or
+observe cancellation. If the parent context was canceled, its error is
+returned. Otherwise, internal cancellation errors are ignored when at least one
+real framework error exists, and the primary returned error is selected from
+all started-task framework errors by this stable ordering:
+
+1. stage order: extract, merge, then normalize;
+2. resolved lane order;
+3. source chunk index for chunk-scoped work; and
+4. configured validator or operation order within that scope.
+
+The full per-task errors may be retained in opt-in diagnostics, but completion
+timing never chooses the public error. Output runs only after every lane reaches
+a successful or rejection-only terminal state and no framework error exists.
+
+Extractors and any validator instance callable by multiple workers must be safe
+for concurrent use. Production implementations should normally satisfy this by
+being immutable after preparation.
+
+## Compatibility and Safety
+
+- Existing production module keys, validator keys, profiles, default chains,
+  and maintained configurations continue to resolve.
+- Existing durable D&D JSON content and logical output paths remain unchanged.
+- Framework errors, validator rejections, retries, checkpoints, diagnostics,
+  and debug behavior retain their current semantics except for the explicitly
+  documented workspace-v2 compatibility boundary and deterministic concurrent
+  ordering.
+- All provider calls pass through the shared global scheduler, across lanes,
+  stages, retries, and validators.
+- Source text and LLM payloads remain subject to the existing opt-in debug and
+  sensitive-data handling policies.
+- Package moves do not create user-visible key changes or concrete cross-domain
+  dependencies.
+
+## Non-Goals
+
+This feature does not introduce:
+
+- a general DAG or configurable stage topology;
+- out-of-process plugins or an RPC extension protocol;
+- cross-lane normalization;
+- a new durable D&D spell schema merely to mirror internal Go types;
+- per-stage LLM permit pools that could exceed or partition the global ceiling;
+  or
+- concurrent work implemented through an unbounded goroutine per lane or chunk.
+
+## Completion Criteria
+
+The target state is reached when:
+
+- all production lanes use one typed artifact from extract through normalize
+  and typed validation;
+- codecs own schema-aware serialization at every type-erasure, checkpoint, and
+  debug boundary;
+- incompatible lane composition and invalid options fail before source work;
+- source units and chunks carry validated canonical provenance;
+- production extensions follow the domain-first package and registrar rules;
+- extract scheduling is bounded, deterministic, concurrent across lanes, and
+  race-free;
+- instrumented tests prove actual concurrent LLM calls never exceed
+  `concurrency.total_llm` across all callers;
+- the maintained D&D example and compatibility baselines retain their durable
+  contracts; and
+- current-behavior documentation is updated as each implemented boundary lands.