diff --git a/docs/adr/0002-linear-pipes-and-filters-pipeline.md b/docs/adr/0002-linear-pipes-and-filters-pipeline.md new file mode 100644 index 0000000..4b39563 --- /dev/null +++ b/docs/adr/0002-linear-pipes-and-filters-pipeline.md @@ -0,0 +1,51 @@ +# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG + +**Status:** Proposed +**Date:** 2026-07-13 + +## Context + +Notarius processes source material through one known workflow: + +```text +input -> chunk -> extract -> merge -> normalize -> output +``` + +Input and chunking apply to the source as a whole. Each selected artifact lane +then performs extract, merge, and normalize, after which output aggregates the +lane outcomes. Chunk extraction has a natural scatter-gather shape, but no +current use case requires arbitrary branches, joins, or user-defined stage +topology. + +## Decision + +Notarius implements a fixed six-stage pipes-and-filters pipeline. Configuration +selects implementations for these stages but cannot add stages, reorder them, +or define an arbitrary graph. + +The framework owns stage sequencing and the scatter-gather boundary between +chunk, extract, and merge. Extract results are handed to merge in deterministic +source-chunk order regardless of execution strategy. Each artifact lane remains +logically linear. Output runs after every selected lane has either produced an +accepted normalized artifact or reached a recorded rejection. A framework +execution failure aborts the pipeline. + +The runner's concrete internal representation and stage-specific scheduling +policies are implementation details. Concurrency must preserve the pipeline's +deterministic handoffs, validation behavior, and provenance, and all execution +strategies must continue to honor context cancellation. + +## Alternatives considered + +- Build a general DAG engine now. This would support hypothetical branching + topologies, but would add scheduling, topology validation, configuration, and + state-management complexity without a current consumer. Revisit this choice + only when a concrete workflow requires a topology the fixed pipeline cannot + express. + +## Consequences + +The runner, configuration model, and operator mental model remain small. Stage +ownership stays visible, and general chunking, merging, or normalization cannot +be hidden inside extractors. A future DAG requirement will require an explicit +architectural change rather than incremental exceptions to the fixed pipeline. diff --git a/docs/adr/0003-typed-interfaces-with-two-zone-data-model.md b/docs/adr/0003-typed-interfaces-with-two-zone-data-model.md new file mode 100644 index 0000000..273b789 --- /dev/null +++ b/docs/adr/0003-typed-interfaces-with-two-zone-data-model.md @@ -0,0 +1,119 @@ +# ADR-0003: Strongly typed stage interfaces with a two-zone data model + +**Status:** Proposed +**Date:** 2026-07-13 + +## Context + +Pipeline stages must exchange source data and extracted artifacts. Universal +source data has one engine-wide meaning, while extracted artifacts have +domain-specific shapes. Passing opaque bytes or `any` between all stages would +make invalid wiring and merge behavior runtime concerns. Requiring JSON at +every handoff would preserve interoperability but discard useful Go type safety +while all modules are in-process. + +The framework must also support multiple configured artifact domains, durable +checkpoints, diagnostics, and output encoders without making those consumers +depend on every domain's Go types. + +## Decision + +Notarius uses two typed data zones followed by one serialized boundary. + +### Source zone + +Input and chunk stages use conservative, engine-owned document, segment, chunk, +and source-reference types. Their exact Go names are implementation details. +Every segment carries engine-owned source provenance identifying the source +location from which it was produced. Chunks preserve the ordered provenance of +their segments. + +Source-format-specific fields remain in input modules or explicitly namespaced +metadata; they do not become framework contracts. + +### Domain artifact zone + +Each artifact lane has one domain-owned Go artifact type `T`. Its extract, +merge, normalize, and domain-aware validation implementations use generic, +strongly typed contracts over the same `T`. Raw JSON, opaque bytes, and `any` +are not stage-handoff contracts within a lane. + +Each registered domain artifact type supplies a codec for `T`. The codec owns: + +- stable schema identity and an explicit schema version; +- JSON serialization and deserialization; +- the media type and schema metadata required at serialized boundaries; and +- rejection of data that cannot be represented by the declared artifact + schema. + +An artifact type's JSON representation is a maintained domain contract. +Changing it incompatibly requires a new schema version. + +Extract, merge, and normalize may change the contents of `T`, but they do not +change the lane's canonical Go artifact type or artifact schema identity. An +extractor maps any provider- or prompt-specific response type into `T` before +returning. A future lane that requires different artifact types at different +stages requires a new architectural decision. + +### Serialized boundary + +After normalization, each typed artifact is converted into an engine-owned +serialized artifact containing bytes, media type, and schema metadata. Output +aggregation and output encoders consume this type-erased form. Intermediate +checkpoint and debug encodings do not become stage-handoff contracts. + +LLM transport, checkpoints, and opt-in debug recording are also explicit +serialization boundaries. They may encode or decode a typed artifact through +its domain codec, but they do not change the in-memory type used between +extract, merge, normalize, and typed validators. Checkpoint reuse requires a +compatible schema identity and version. + +An LLM structured-response schema is a module transport contract and may differ +from the domain artifact schema. The calling module owns the response type and +maps it into the canonical `T`; the artifact codec remains authoritative for +artifact checkpoints and output serialization. + +The framework may use private type-erased adapters to store heterogeneous lane +registrations and execute configured domains. Such an adapter must assemble a +type-consistent lane before execution and must not expose `any` or raw payloads +as module-facing handoffs inside the domain artifact zone. + +### Construction and dependencies + +Every module operation accepts `context.Context`. Modules receive stable runtime +collaborators through an injected dependency set at construction time. In +particular, LLM-using modules receive the application-provided structured LLM +client and do not construct provider clients or bypass shared scheduling. + +The application boundary enforces one configurable global upper bound on +in-flight LLM calls across all stages, lanes, retries, and validators. + +Configuration options are parsed and validated while a module is constructed, +before that module executes. Per-run data such as source material, references, +session identity, and lane identity remains operation input rather than a +construction dependency. + +## Alternatives considered + +- Pass raw bytes between stages. This maximizes decoupling but moves wiring, + parsing, and merge errors to runtime and prevents domain types from being the + canonical in-process contract. +- Require JSON plus schemas at every stage boundary. This is appropriate for an + out-of-process boundary, but adds serialization and parsing inside the current + in-process pipeline. The stable codec contract preserves this upgrade path if + remote plugins are introduced. +- Use a uniform `Process(any) (any, error)` contract. This simplifies a fully + dynamic engine but turns incompatible module composition into type assertions + and runtime failures. The fixed topology does not require that tradeoff. + +## Consequences + +Domain pipelines gain compile-time handoff safety and explicit merge semantics. +Serialization, schema compatibility, checkpoint decoding, and output erasure +have named owners. Dynamic registration requires a small erased adapter around +each typed lane, and generic stage implementations must be instantiated for a +specific artifact type or behavior rather than manipulating arbitrary JSON. + +The engine-owned source model becomes a long-lived contract and must evolve +conservatively. Domain authors must maintain a codec and versioned schema in +addition to their Go artifact type. diff --git a/docs/adr/0004-package-modules-by-domain.md b/docs/adr/0004-package-modules-by-domain.md new file mode 100644 index 0000000..b5f157a --- /dev/null +++ b/docs/adr/0004-package-modules-by-domain.md @@ -0,0 +1,80 @@ +# ADR-0004: Package modules by domain, not by stage + +**Status:** Proposed +**Date:** 2026-07-13 + +## Context + +Module packages can be grouped first by pipeline stage, such as +`modules/chunk/dnd/scenes`, or first by domain, such as +`modules/dnd/chunk/scenes`. A domain's extract, merge, normalize, validation, +schema, prompt, and artifact-codec implementations collaborate around the same +artifact types and are likely to evolve together. + +Go package dependencies also constrain registration. If shared types live in a +domain root package, that package cannot import child implementation packages +to register them because the children already import the root types. + +## Decision + +Production extensions are grouped by domain under: + +```text +internal/modules/// +``` + +Shared artifact types live at the domain root, for example +`internal/modules/dnd/types.go`. Domain-specific validators, prompt fragments, +schemas, reference helpers, and codecs also live within that domain tree. + +Each domain exposes one production registration entry point from a sibling +registrar package, for example `internal/modules/dnd/register`. The registrar +may import the domain root and its child implementations; the domain root does +not import its registrar or child packages. This keeps shared types available +as `dnd.SpellList` without creating a Go import cycle. + +The `generic` tree is a peer extension family for reusable implementations that +contain no concrete source-format or artifact-domain knowledge. Source-format +and output-format families, such as Seriatim and JSON output, follow the same +domain-first organization even when they do not define a Zone-B artifact type. + +Concrete domain implementation packages do not import another concrete domain. +Generic extension packages never import concrete domains. A domain registrar +may import domain-neutral generic extension packages to instantiate a reusable +strategy for that domain's artifact type; the generic implementation remains +unaware of the concrete type's domain semantics. Reuse needed directly by a +domain implementation lives in a domain-neutral framework or helper package, +not in a peer extension package. + +The application composition root may import multiple registrar packages, and +black-box integration tests may compose multiple domains. Other cross-domain +reuse occurs through engine contracts and composition-time registration rather +than concrete peer-domain imports. + +A domain registrar owns registration of that domain's modules, validators, +default validator chains, artifact codecs, schemas, and prompt assets. It does +not take ownership of application execution or process behavior. + +## Alternatives considered + +- Group modules by stage. This keeps interchangeable strategies side by side, + but scatters a domain's shared artifact model and collaborating extensions + across the repository. It is preferable when generic strategy libraries + dominate or when the project is primarily a stage-extension framework rather + than an application composed from domain suites. +- Put both shared types and `Register` in the domain root. This gives the + shortest import path but creates an import cycle once child implementations + import the root artifact types. + +## Consequences + +The repository layout makes supported domains immediately visible, and adding +or extracting a domain affects one cohesive subtree. The CLI composition root +depends on a small set of domain registrars instead of every leaf package. + +Package moves must preserve user-visible module and validator keys unless a +separate compatibility decision changes them. Shared behavior that cannot be +expressed through framework contracts may need to move into a domain-neutral +framework package rather than creating a concrete peer-domain import. Registrar +packages become explicit composition points for instantiating generic typed +strategies, in addition to registering domain-owned implementations. diff --git a/docs/roadmap/domain.md b/docs/roadmap/domain.md new file mode 100644 index 0000000..d65344d --- /dev/null +++ b/docs/roadmap/domain.md @@ -0,0 +1,348 @@ +# Domain-Typed Pipeline Feature Roadmap + +## Status + +Decision-complete; implementation pending. This roadmap defines the desired end +state for [ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md), +[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and +[ADR-0004](../adr/0004-package-modules-by-domain.md). The work needed to reach +that state is owned by the +[domain pipeline implementation plan](implementation.md). + +Until that plan is complete, current behavior remains defined by the +architecture, configuration, integration, operations, and internal +documentation outside `docs/roadmap/`. + +## User Intent + +Notarius should remain a small, explicit pipes-and-filters application while +making domain extensions safe to compose and straightforward to maintain. A +configured pipeline should fail before execution when its modules are +incompatible, should carry typed domain values rather than reparsed JSON between +artifact stages, and should preserve provenance and durable output contracts. + +The application must also enforce one configurable, process-wide ceiling on +in-flight LLM calls. Pipeline scheduling may impose stricter limits, but no +stage, lane, retry, validator, or future LLM-backed extension may bypass that +global ceiling. + +## Target Architecture + +### Pipeline and outcome model + +The topology remains: + +```text +input -> chunk -> extract -> merge -> normalize -> output +``` + +Input and chunk are pipeline-wide. Extract, merge, normalize, and their +validators operate per artifact lane. Output aggregates the terminal artifacts +from all lanes. The resolved pipeline retains explicit fields for those roles; +it does not become a general DAG or a heterogeneous ordered-stage list. + +Framework errors abort the run. Validator rejection is a recorded domain +outcome and does not abort unrelated work. Accepted extract results reach merge +in source-chunk order, regardless of execution completion order. Warnings, +rejections, artifacts, and reported errors are likewise ordered by stable +pipeline scope rather than goroutine completion time. + +### Engine-owned source model + +The engine owns `source.SourceDocument`, `source.SourceUnit`, `source.SourceRef`, +and `source.Chunk`. Domain modules may consume these types but must not redefine +their provenance semantics. + +- Every source unit has a canonical self-reference identifying its source and + unit range. +- A chunk contains ordered source units and one canonical reference spanning + its first through last unit. +- Chunk and unit references are validated for source identity, order, and + containment. +- Auxiliary reference material remains distinct from source provenance. +- Cloning, canonicalization, checkpointing, debugging, and digest computation + preserve the source references exactly. + +`source.Chunk.Ref` replaces duplicate start/end boundary fields. Because that +changes persisted workspace state, the workspace checkpoint schema advances to +`notarius.workspace.v2`. Existing v1 checkpoints are left intact but treated as +incompatible and recomputed; no in-place migration or deletion is required. + +### Typed artifact lanes + +Each lane has one canonical artifact type `T` from extraction through merge, +normalization, and typed validation. Module-facing Zone-B contracts are generic: + +- `Extractor[T]` produces typed per-chunk values plus framework-owned + provenance and diagnostics; +- `Merger[T]` combines accepted values in source-chunk order; +- `Normalizer[T]` canonicalizes the merged value; and +- `TypedValidator[T]` applies semantic checks at its configured artifact stage. + +The framework may use private erased adapters to keep heterogeneous lanes in +one resolved pipeline, but `any`, raw JSON, and a generic `Process(any)` API are +not module-facing handoffs. The extractor selected for a lane establishes its +artifact kind. Resolution uses that kind to select compatible merger, +normalizer, validator, and codec variants and rejects an incompatible lane +before any stage executes. + +Generic strategies remain reusable without knowing concrete domains. In +particular, append-order merge is parameterized by a typed combine function +provided during domain registration, and no-op normalization is instantiated +for the lane's concrete type. + +### Artifact identity and codecs + +Every typed artifact kind has exactly one registered `ArtifactCodec[T]`. An +artifact kind is a stable logical identifier, separate from a Go type name. A +codec owns: + +- artifact kind; +- schema identifier, name, and version; +- media type and JSON Schema bytes; and +- strict, deterministic encoding and decoding between `T` and the serialized + representation. + +Equal canonical values must encode to equal bytes. Those bytes are the basis +for artifact digests. Codec decoding rejects malformed or schema-incompatible +content. Domain validators continue to own semantic validity; codecs do not +replace them. + +`SerializedArtifact` is the Zone-C representation and includes the artifact +kind, schema metadata, media type, encoded content, and framework metadata. +Type erasure occurs through the codec after normalization for final output. +Intermediate checkpointing and opt-in debug recording may also use the codec, +but serialization for those side effects is not a stage handoff. + +Checkpoint metadata records artifact kind, schema identifier, schema version, +and schema digest. Reuse requires an exact compatible registered codec; +otherwise the checkpoint is safely invalidated. Output remains domain-neutral +and consumes serialized artifacts. + +Generic serialized validators remain supported for representation-level checks +such as valid JSON and JSON Schema validation. The framework encodes `T` through +its registered codec before invoking them. Domain validators receive `T` +directly. Chunk-stage validators remain in the source zone: semantic chunk +validators receive engine-owned chunks, while representation-level validators +receive the framework's canonical serialized chunk view. They do not force +source-zone values through a domain artifact codec. + +### Registration and resolution + +Registries expose typed registration helpers while privately retaining the Go +type identity needed to assemble erased lane executors. + +- Codecs are keyed by artifact kind, with exactly one codec per kind. +- Extractors are keyed by their existing module key and declare an artifact + kind. +- Mergers, normalizers, and validators are keyed by `(module key, artifact + kind)`, allowing stable generic keys such as `appendorder` and `noop` to have + multiple typed specializations. +- Resolved pipeline identity and dependency fingerprints include artifact kind + and schema identity, version, and digest. +- Duplicate or incompatible registrations and selections fail deterministically + during composition or resolution. + +The framework's public typed registration surface uses free generic functions, +because Go methods cannot declare their own type parameters. Private reflection +may verify and erase registered types, but it is not exposed to module authors. + +### Preparation, options, and dependencies + +Pipeline execution is split into resolution, preparation, and running. +Preparation constructs every selected module and validator before source input +begins and returns a prepared pipeline with explicit input, chunk, lane, and +output fields. + +- Construction receives framework-owned dependencies, including the one shared + scheduled structured-LLM client. +- Raw configured options are decoded once into implementation-owned option + structs during preparation. +- Missing dependencies, malformed options, unknown options, and incompatible + typed selections fail before stage execution. +- Configuration validation uses the same option decoders without requiring live + provider dependencies. +- Per-run data such as sources, chunks, references, session identity, lane + identity, and metadata remains in operation requests. +- Constructed implementations that can be scheduled concurrently are immutable + after preparation or otherwise explicitly concurrency-safe. + +Modules and validators must not construct provider clients, wrap their own +independent schedulers, or bypass the injected scheduled client. + +### D&D artifact model + +The D&D package root owns canonical `dnd.SpellList`, `dnd.SpellCast`, and +related evidence types, using engine-owned `source.SourceRef` values. The spell +extractor keeps its LLM response DTO and response schema private and maps the +canonicalized response to the domain model. + +The D&D spell codec separately owns the existing durable spell artifact schema. +The LLM response schema and durable artifact schema remain distinct contracts +even if their current JSON shapes are similar. Shape, source-reference, and +source-relatedness validators operate on the canonical typed model. The +validator-only duplicate spell model and inter-stage JSON reparsing disappear. + +The migration preserves the existing D&D spell payload, logical output bundle, +module and validator keys, prompt/schema identities, default validator chains, +warnings, rejection semantics, and manifest provenance unless a separate +compatibility decision explicitly changes one of those contracts. + +## Package Ownership + +The target production extension layout is: + +```text +internal/modules/dnd/ + types.go + codec/spells/ + chunk/scenes/ + extract/spells/ + validate/spells/shape/ + validate/spells/source_refs/ + validate/spells/source_relatedness/ + shared/ + register/ + +internal/modules/generic/ + chunk/units/ + merge/appendorder/ + normalize/noop/ + validate/always_accept/ + validate/always_reject/ + validate/valid_json/ + validate/valid_json_schema/ + output/json/ + register/ + +internal/modules/seriatim/ + input/transcript/ + register/ + +internal/framework/promptfs/ +``` + +Shared domain types live at the domain root. Registration lives in a sibling +`register` package so that the root never imports child implementations. Each +registrar exposes one composition entry point accepting the pipeline registry +set and LLM asset registry. The CLI composition root creates those registries +and invokes the generic, Seriatim, and D&D registrars. + +Concrete domain implementations do not import peer domains. Generic extensions +never import a concrete domain. A domain registrar may import generic packages +to register typed specializations for its domain. The application composition +root and designated black-box integration tests may compose multiple +registrars. Domain-neutral embedded prompt-asset filesystem support belongs to +the framework rather than a domain package. + +## Concurrency Policy + +### Configuration + +The existing `concurrency.total_llm` setting remains the application-wide +ceiling on actual provider calls. Version-2 configuration gains an extensible +stage-worker map: + +```yaml +concurrency: + total_llm: 4 + stage_workers: + extract: 4 +``` + +Initially, `extract` is the only recognized key. Unknown stage keys are rejected +so misspellings cannot silently alter scheduling. If omitted, the effective +extract worker count equals `total_llm`. Its valid range is +`1..concurrency.total_llm`. The environment override is +`NOTARIUS_STAGE_WORKERS_EXTRACT`; future stage overrides receive similarly +explicit names that map to the extensible file representation. + +The worker setting bounds framework jobs, not provider calls. Only an actual +LLM call consumes a permit from the shared scheduled client. The global +scheduled client remains authoritative even if future stages gain worker limits. + +### Scheduling + +After pipeline-wide chunking, all lanes may run concurrently. A central +dispatcher submits `(lane, chunk)` jobs to one run-wide extract worker pool in +round-robin order: source chunk first, then resolved lane order. This avoids one +unbounded goroutine per job and prevents an early lane from monopolizing the +queue. + +One job contains extraction, its stage-local retry behavior, and extract-stage +validation for that lane and chunk. Each lane begins its serial merge then +normalize continuation when all of its extract jobs reach a terminal state. +Different lanes' continuations may overlap, and any LLM-backed continuation or +validator still shares the global scheduled client. + +Workers publish immutable task results to a coordinator. Only the coordinator +mutates aggregate results, manifests, checkpoint indexes, warnings, and +rejections. Debug artifacts use attempt-specific paths and do not rely on +concurrent writes to shared files. + +Rejections do not cancel work. A framework error cancels the derived run +context, stops undispatched jobs, and waits for started jobs to finish or +observe cancellation. If the parent context was canceled, its error is +returned. Otherwise, internal cancellation errors are ignored when at least one +real framework error exists, and the primary returned error is selected from +all started-task framework errors by this stable ordering: + +1. stage order: extract, merge, then normalize; +2. resolved lane order; +3. source chunk index for chunk-scoped work; and +4. configured validator or operation order within that scope. + +The full per-task errors may be retained in opt-in diagnostics, but completion +timing never chooses the public error. Output runs only after every lane reaches +a successful or rejection-only terminal state and no framework error exists. + +Extractors and any validator instance callable by multiple workers must be safe +for concurrent use. Production implementations should normally satisfy this by +being immutable after preparation. + +## Compatibility and Safety + +- Existing production module keys, validator keys, profiles, default chains, + and maintained configurations continue to resolve. +- Existing durable D&D JSON content and logical output paths remain unchanged. +- Framework errors, validator rejections, retries, checkpoints, diagnostics, + and debug behavior retain their current semantics except for the explicitly + documented workspace-v2 compatibility boundary and deterministic concurrent + ordering. +- All provider calls pass through the shared global scheduler, across lanes, + stages, retries, and validators. +- Source text and LLM payloads remain subject to the existing opt-in debug and + sensitive-data handling policies. +- Package moves do not create user-visible key changes or concrete cross-domain + dependencies. + +## Non-Goals + +This feature does not introduce: + +- a general DAG or configurable stage topology; +- out-of-process plugins or an RPC extension protocol; +- cross-lane normalization; +- a new durable D&D spell schema merely to mirror internal Go types; +- per-stage LLM permit pools that could exceed or partition the global ceiling; + or +- concurrent work implemented through an unbounded goroutine per lane or chunk. + +## Completion Criteria + +The target state is reached when: + +- all production lanes use one typed artifact from extract through normalize + and typed validation; +- codecs own schema-aware serialization at every type-erasure, checkpoint, and + debug boundary; +- incompatible lane composition and invalid options fail before source work; +- source units and chunks carry validated canonical provenance; +- production extensions follow the domain-first package and registrar rules; +- extract scheduling is bounded, deterministic, concurrent across lanes, and + race-free; +- instrumented tests prove actual concurrent LLM calls never exceed + `concurrency.total_llm` across all callers; +- the maintained D&D example and compatibility baselines retain their durable + contracts; and +- current-behavior documentation is updated as each implemented boundary lands. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index d2e6d04..4f9802a 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -4,6 +4,12 @@ Current Notarius behavior is documented in the canonical README, CLI, configuration, operations, internal, and integration docs. This roadmap records future work only. +## Focused Roadmaps + +- [Domain-Typed Pipeline Implementation](domain.md): proposed migration to + domain-owned typed artifact lanes, domain-first packages, explicit + serialization boundaries, and bounded deterministic extract execution. + ## Candidate Product Work - Additional input adapters, such as Markdown or note-export formats. @@ -32,7 +38,6 @@ future work only. enforcement that a validator is suitable for a specific stage or module. - Batching or context-window controls for LLM-backed validators if validator inputs become large enough to require them. -- Parallel execution where it preserves deterministic manifests and diagnostics. - Additional output encoders. ## Candidate Operational Work diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..8b2dc8f --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,836 @@ +# Domain-Typed Pipeline Implementation Plan + +## Purpose + +This document is the executable implementation plan for the target state in the +[domain-typed pipeline feature roadmap](domain.md). It assumes the decisions in +[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md), +[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and +[ADR-0004](../adr/0004-package-modules-by-domain.md). + +The intended operator is an LLM coding agent working through one stage per +implementation prompt. Complete the stages in order. Each stage must leave the +repository buildable and tested; do not defer a broken intermediate state to a +later stage. + +## Implementation Rules + +For every stage: + +1. Read `docs/development.md` and follow its task-specific reading guide. Read + the current implementation and focused tests for every touched subsystem. +2. Treat the feature roadmap as the canonical owner of desired behavior and + this document as the canonical owner of task sequencing. Do not restate + future behavior in current-behavior documentation before it exists. +3. Preserve unrelated user changes. Use mechanical moves where possible so file + history and test intent remain legible. +4. Add focused tests with the change. Run those tests while iterating, then run + `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` before ending + the stage. +5. Run `go test -race ./...` in stages that introduce or change concurrency and + in the final stage. +6. Update the canonical current-behavior documents in the same stage in which + behavior changes. At minimum, reconsider `docs/policy/architecture.md`, + `docs/internal/overview.md`, `docs/internal/pipeline.md`, + `docs/internal/modules.md`, `docs/internal/llm.md`, `docs/config.md`, + `docs/operations.md`, and `docs/integrations/` according to the documentation + policy; edit only the documents whose owned facts changed. +7. Do not change user-visible module keys, validator keys, output paths, durable + D&D JSON, prompt/schema identities, default chains, or rejection semantics + unless this plan explicitly requires it. +8. Stop after a stage if an exit criterion cannot be met. Record the concrete + blocker rather than implementing a second architecture alongside this one. + +## Fixed Technical Decisions + +The following choices are inputs to implementation, not questions to reopen in +individual stages. + +### Source types + +Keep the existing names `source.SourceDocument`, `source.SourceUnit`, and +`source.SourceRef`. Add `Ref source.SourceRef` to `SourceUnit`. Move +`contracts.SourceChunk` to `internal/core/source` as `source.Chunk`, with this +logical shape: + +```go +type Chunk struct { + ID string + SourceID string + Index int + Ref SourceRef + Content []byte + MediaType string + Units []SourceUnit + Metadata map[string]any +} +``` + +Remove `StartUnitID` and `EndUnitID`; `Ref` is the only chunk-boundary +representation. A Seriatim unit's self-reference is +`{SourceID: document ID, StartUnitID: unit ID, EndUnitID: unit ID}`. A chunk +reference spans its first and last included units. Advance persisted workspace +state from `notarius.workspace.v1` to `notarius.workspace.v2`; v1 state is +incompatible and must be recomputed, but never deleted automatically. + +### Artifact contracts + +Place engine-owned artifact primitives with the other universal contracts under +`internal/framework/contracts`: + +```go +type ArtifactKind string + +type ArtifactSchema struct { + ID string + Name string + Version string + JSONSchema []byte +} + +type SerializedArtifact struct { + Kind ArtifactKind + Schema ArtifactSchema + MediaType string + Content []byte + Metadata map[string]any +} + +type ArtifactCodec[T any] interface { + Kind() ArtifactKind + Schema() ArtifactSchema + MediaType() string + Encode(T) ([]byte, error) + Decode([]byte) (T, error) +} +``` + +Use strict JSON decoding for JSON codecs: reject unknown fields and trailing +tokens. Encoding must be deterministic for equal canonical values. Clone byte +slices and maps at framework ownership boundaries. Validate non-empty artifact +kind, schema ID, schema name, schema version, media type, and JSON Schema during +registration. Compute and retain a SHA-256 digest of the JSON Schema bytes. + +Use generic `Extractor[T]`, `Merger[T]`, `Normalizer[T]`, and +`TypedValidator[T]` contracts. The exact request/result structs may retain +existing names where that reduces churn, but they must satisfy these rules: + +- the extractor returns `T`, warnings, and framework-owned chunk provenance; +- merge receives accepted per-chunk typed values carrying lane ID, source ID, + chunk ID, chunk index, and chunk reference, already sorted by chunk index; +- normalize receives and returns `T`; +- typed validators receive `T` plus the relevant universal source, chunk, + reference, lane, stage, profile, and metadata context; +- the LLM client and decoded module options are held by constructed + implementations, not passed in operation requests; and +- no module-facing Zone-B request or result contains `RawPayload`, `any`, or + serialized JSON as its artifact value. + +Retain separate framework wrappers around `T` for extract, merge, and normalize +provenance. Do not put lane IDs, module keys, or framework warnings into the D&D +domain value itself. + +Support a second `SerializedValidator` contract for representation-level +generic validators. Its request contains immutable bytes, media type, and +optional schema metadata. For a Zone-B value, the framework produces that +request with the lane codec. Keep a separate non-generic `ChunkValidator` +contract for semantic validation of immutable `[]source.Chunk`; when a +representation validator is selected at chunk, the framework instead supplies +its canonical JSON chunk encoding. `valid_json` and `valid_json_schema` use the +serialized path, domain validators use `TypedValidator[T]`, and generic +approve/reject validators register explicit chunk and typed-artifact variants. + +### Typed registry model + +Because Go methods cannot introduce type parameters, expose free generic +registration functions in `internal/framework/pipeline`, backed by private +non-generic registry entries. Use `reflect.TypeFor[T]()` only inside registration +and framework assembly to prove exact type equality. + +- Codec registry key: artifact kind. Exactly one codec may be registered per + kind. +- Extractor registry key: existing module key. Each entry declares one artifact + kind and exact Go type. +- Merger and normalizer registry key: `(existing module key, artifact kind)`. +- Typed validator registry key: `(existing validator key, artifact kind)`. +- Chunk-validator registry key: existing validator key in the distinct chunk + target namespace. +- Serialized validators retain their existing validator key and declare whether + they support chunk values, artifact values, or both. +- Duplicate keys/variants, missing codecs, Go-type mismatches, and incompatible + selected variants are errors. + +The extractor selected for a lane establishes the lane artifact kind. During +resolution, look up merger, normalizer, and validators against that kind. +Record artifact kind, schema ID, schema version, and schema digest on the +resolved lane and in the resolved-pipeline digest. A resolved pipeline with an +incompatible lane must fail before preparation or source execution. + +The private erased lane entry owns closures for construction and execution of +one concrete `T`. It may store a value as `any` internally, but it must verify +the exact registered `reflect.Type` at every erased boundary and return a +descriptive framework error rather than panic. + +### Construction and preparation + +Use one uniform construction context: + +```go +type ModuleDependencies struct { + LLM contracts.StructuredLLMClient +} + +type BuildRequest struct { + Dependencies ModuleDependencies + Options map[string]any +} +``` + +Each registry entry stores both an option-validation closure and a constructor. +Implementations own concrete option structs and one decoder used by both +closures. Resolution/configuration validation calls the decoder and discards +the value; preparation calls it once and supplies the decoded value to the +constructor. Reject unknown option fields. Empty options produce the +implementation's explicit defaults. + +Add: + +```go +func Prepare( + resolved ResolvedPipeline, + registries Registries, + deps ModuleDependencies, +) (*PreparedPipeline, error) +``` + +`PreparedPipeline` retains explicit resolved input, chunk, artifact-lane, and +output fields and all constructed validators. Construct in stable pipeline +order: input; chunk and its validators; each resolved lane in order with extract, +merge, normalize, and their validator chains in stage order; then output. On the +first failure, return an error identifying stage, lane if any, module or +validator key, and cause. No operation method may have run. + +Create the one scheduled production LLM client first, inject it into +preparation, and then run only the prepared pipeline. Test-only deterministic +modules may accept a nil LLM dependency; any implementation that declares or +uses LLM-backed execution must reject a nil client at preparation. Remove LLM +clients and raw option maps from operation requests after every production +implementation has migrated. + +Constructed modules and validators are reused for a run. Anything callable from +parallel extract workers must be concurrency-safe; production implementations +should be immutable after construction. + +### Package registration + +Each package-family registrar exposes: + +```go +func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error +``` + +The generic and Seriatim registrars ignore the asset argument until they need +it. Registrars validate the registry pointers they use and return contextual +errors. The CLI creates one complete registry set and one asset registry, then +calls registrars in this order: generic, Seriatim, D&D. The D&D registrar owns +D&D codecs, implementations, typed generic specializations, prompt/schema +assets, and default validator chains. + +`internal/modules/dnd` owns D&D shared types. Its `register` sibling may import +children and generic strategies; the root package must not import its children. +`internal/modules/generic` never imports D&D. `internal/modules/seriatim` does not +import D&D. Move domain-neutral embedded prompt-filesystem helpers to +`internal/framework/promptfs`. + +### D&D typed model + +Define `dnd.SpellList`, `dnd.SpellCast`, and evidence/source-reference fields at +the D&D package root. Use `source.SourceRef`; do not create another D&D unit-ref +type for artifact provenance. The stable artifact kind is +`dnd/spell-list`. + +The spell extractor owns a private LLM DTO and the existing +`dnd_spells_llm.v1.json` response schema. It canonicalizes and maps that DTO to +`dnd.SpellList`. The codec package owns `dnd_spells.v1.json` and the durable +encoding. Keep those schemas separate. Remove the validator-only +`spellpayload` model after all three D&D validators consume `dnd.SpellList`. + +Make `appendorder` a generic strategy that accepts a typed combine function at +registration/construction. The D&D registrar supplies a function that appends +spell casts in already-sorted source-chunk order. Make `noop` a generic typed +strategy. Neither generic package imports D&D. + +### Serialized boundaries + +After normalize, encode `T` once to `SerializedArtifact` for final output. +Output remains domain-neutral and receives serialized artifacts. Preserve the +existing output bundle and index contract. + +Extract, merge, and normalize checkpoints encode and decode through the same +lane codec. Checkpoint identity includes artifact kind, schema ID, schema +version, and schema digest. A missing codec or any mismatch invalidates reuse +and recomputes the stage; it is not a fatal run error by itself. Codec decode +failure also invalidates that checkpoint and records the reason. Never pass +serialized checkpoint content directly to the next typed stage. + +Debug recording uses the codec for typed artifact values and preserves existing +opt-in/sensitive-content rules. Artifact/checkpoint digests use the stable codec +bytes. Aggregate manifests, checkpoint indexes, warning slices, and rejection +slices have one coordinator writer. + +### Concurrency + +Version-2 configuration gains: + +```yaml +concurrency: + total_llm: 4 + stage_workers: + extract: 4 +``` + +Represent effective stage worker limits as `map[string]int`. Initially accept +only `extract`; reject unknown keys. Missing extract defaults to `total_llm` and +its valid range is `1..total_llm`. Add the environment override +`NOTARIUS_STAGE_WORKERS_EXTRACT`. Preserve precedence rules and keep the file +configuration version at 2. + +Use one run-wide fixed extract worker pool. Dispatch jobs in round-robin order +with source chunk as the outer loop and resolved lane as the inner loop. Use one +job channel whose capacity equals the effective extract worker count, so the +dispatcher applies bounded backpressure. Do not create one goroutine per job. A +job includes extract retries and extract-stage validators. Store results by +lane index and chunk index. + +When all extract jobs for a lane are terminal, run that lane's merge and then +normalize serially. Lane continuations may overlap. All LLM calls at all stages, +including retries and validators, use the single injected scheduled client, so +`total_llm` remains the authoritative process-wide provider-call ceiling. + +Rejections are terminal results and do not cancel other work. A framework error +cancels a derived run context, stops dispatching jobs not yet started, and waits +for started tasks to finish or observe cancellation. Choose the returned error +as follows: + +1. if the parent context is canceled, return its error; +2. otherwise discard internal `context.Canceled`/`DeadlineExceeded` errors when + at least one non-context framework error exists; and +3. choose the earliest remaining error by stage order (`extract`, `merge`, + `normalize`), resolved lane index, chunk index for chunk-scoped work, and + configured validator/operation index. + +Use a sentinel chunk index after all real chunks for lane-scoped merge and +normalize errors. Retain other started-task errors only in opt-in diagnostics. +Sort accepted artifacts, warnings, and rejections by resolved lane index, source +chunk index where applicable, stage order, validator order, and original +within-result order. Completion timing must not affect public output. Run output +only if every lane has a successful or rejection-only terminal outcome and no +framework error occurred. + +## Staged Implementation + +### Stage 1: Compatibility Baselines and ADR Acceptance + +Goal: lock down behavior that subsequent internal migrations must preserve and +record the architectural decisions as accepted. + +Tasks: + +- Add semantic or golden compatibility tests for the maintained Seriatim-to-D&D + path: durable output files and JSON, output index, manifest provenance, + warnings, rejections, and stable lane/chunk ordering. +- Snapshot production module keys, validator keys, default validator chains, + prompt/schema identities, and maintained example/profile resolution in tests. +- Strengthen runner tests for fixed topology, validator rejection as a nonfatal + outcome, framework-error abort, retries, parent cancellation, checkpoint reuse + and invalidation, diagnostics, and opt-in debug behavior. +- Add an instrumented scheduled-client test showing that all existing production + LLM callers share `concurrency.total_llm`. It need not demonstrate parallel + lanes yet. +- Review the three ADRs against this decision-complete plan, set their status to + `Accepted`, and update their dates only if ADR policy requires an acceptance + date. Do not rewrite accepted decision text after this stage. + +Exit criteria: + +- compatibility tests fail on an unintended durable-output, key, chain, + provenance, or outcome-semantics change; and +- ADR-0002, ADR-0003, and ADR-0004 are accepted. + +### Stage 2: Registrar Composition Without Package Moves + +Goal: replace CLI leaf-by-leaf registration with package-family composition +before changing imports. + +Tasks: + +- Add `internal/modules/generic/register`, `internal/modules/seriatim/register`, + and `internal/modules/dnd/register` using the fixed registrar signature. +- Initially let those registrars import the existing stage-oriented packages. + Move ownership of production validators, default chains, and prompt assets out + of `internal/cli/catalog.go` and into the appropriate registrar. +- Have the CLI allocate complete registries and the asset registry once, invoke + generic, Seriatim, then D&D registration, and retain its existing test + injection paths. +- Test nil registry handling, duplicate registration errors, stable registered + keys, default chains, and asset identities. + +Exit criteria: + +- the CLI composition root names only the three registrar packages, framework + registry types, and asset registry; and +- no production key, chain, prompt, schema, or runtime behavior changes. + +### Stage 3: Mechanical Generic and Seriatim Package Moves + +Goal: establish the domain-first generic and source-format trees without +changing contracts. + +Tasks: + +- Move the Seriatim adapter and tests to + `internal/modules/seriatim/input/transcript`. +- Move the generic chunker to `internal/modules/generic/chunk/units`, retaining + the configured key `generic`. +- Move append-order merge, no-op normalize, JSON output, and all generic + validators to their target paths under `internal/modules/generic`. +- Update only registrar imports and affected black-box tests. Preserve package + behavior and all public registry keys. +- Remove the emptied old directories. + +Exit criteria: + +- generic and Seriatim production implementations exist only under their target + trees; and +- compatibility baselines remain green. + +### Stage 4: Mechanical D&D Package Move and Import Guard + +Goal: establish the D&D tree and enforce ADR-0004 dependency direction while +legacy contracts are still intact. + +Tasks: + +- Move the D&D scenes chunker, spell extractor, validators, schemas, prompt + assets, and D&D shared helpers into the target D&D tree. Do not create the + typed root model or codec yet. +- Move domain-neutral prompt filesystem helpers from + `internal/modules/sharedassets` to `internal/framework/promptfs`. +- Move tests with their owning implementation. Relocate tests that intentionally + compose domains to a black-box integration-test package rather than creating + peer-domain production imports. +- Add a Go-parser-based import-boundary test. It must reject concrete + D&D-to-Seriatim and Seriatim-to-D&D imports, all generic-to-D&D imports, and + root-domain imports of child implementations. Allow domain registrars, the CLI + composition root, and designated external integration tests to compose + packages. +- Remove old empty stage-oriented and validator directories. + +Exit criteria: + +- all production extensions use the target domain-first package layout except + the not-yet-created typed codec/model pieces; +- the import guard detects a deliberate fixture violation; and +- behavior and keys remain unchanged. + +### Stage 5: Source-Unit Provenance + +Goal: add canonical provenance to engine-owned source units without yet changing +the chunk type. + +Tasks: + +- Add `Ref source.SourceRef` to `source.SourceUnit`, including clone and debug + representations. +- Make the Seriatim adapter assign the fixed self-reference for every unit. +- Extend `source.ValidateDocument` to require the unit reference's source ID to + match the document, require start and end IDs to equal the unit ID, reject + missing/invalid/reversed references, and preserve the existing unit-order and + uniqueness checks. +- Make source digests and source checkpoint serialization include the new + reference deterministically. +- Add focused tests for valid refs and missing, foreign, non-self, and reversed + refs, plus source checkpoint/debug round trips. + +Exit criteria: + +- every produced source unit has a validated self-reference; and +- source state preserves it through clone, debug, digest, and checkpoint paths. + +### Stage 6: Engine-Owned Chunks and Workspace v2 + +Goal: finish the Zone-A source model and make its persisted compatibility break +explicit. + +Tasks: + +- Add `source.Chunk` with the fixed shape and update universal chunk contracts, + modules, validators, runner code, checkpoints, debug envelopes, and tests to + use it. +- Derive `Chunk.Ref` from the first and last included unit references. Validate + source identity, non-empty ordered units, contiguous boundary agreement, and + exact correspondence between the chunk ref and first/last unit refs. +- Remove `contracts.SourceChunk`, `StartUnitID`, and `EndUnitID` after all + consumers migrate. Do not keep aliases. +- Advance `workspace.WorkspaceSchemaVersion` to `notarius.workspace.v2`. Make + loader behavior explicitly classify v1 as incompatible and recompute while + leaving files untouched. +- Update checkpoint identity/digest tests and operations documentation for the + one-time v1 resume miss. + +Exit criteria: + +- no production code imports a framework-owned chunk type; +- chunk provenance round-trips exactly; and +- v1 workspaces are safely ignored while v2 workspaces reuse successfully. + +### Stage 7: Artifact and Codec Foundation + +Goal: add the typed primitives and prove strict serialization independently of +production lanes. + +Tasks: + +- Add the fixed artifact types, codec interface, schema digest helper, and clone + helpers under `internal/framework/contracts`. +- Add `ArtifactCodecRegistry` to `pipeline.Registries` and `ModuleCatalog`. +- Implement generic codec registration and private erasure/type tracking. +- Validate registration metadata and duplicates. Ensure erased encode/decode + returns typed errors, never reflection panics. +- Use two small test artifact types to cover registration, exact type identity, + deterministic encoding, strict decoding, cloning, duplicate kind rejection, + and schema metadata/digest behavior. +- Wire the new empty registry through CLI/test registry constructors without + changing production lane execution. + +Exit criteria: + +- codecs for heterogeneous test types can coexist and safely round-trip through + erased framework storage; and +- current production behavior remains on the legacy raw path and unchanged. + +### Stage 8: Typed Contracts, Variants, and Resolution + +Goal: resolve a complete type-compatible lane before executing it. + +Tasks: + +- Add the typed stage, typed validator, chunk validator, serialized validator, + and provenance wrapper contracts from the fixed decisions. +- Extend extractor specs with artifact kind/type. Convert merger, normalizer, + and typed validator registries to artifact-kind variants while retaining + serialized-validator registration by key. +- Implement free generic registration helpers and private erased entries. +- Extend lane resolution to derive kind from extractor, require its codec, + select exact merger/normalizer/validator variants, and include artifact/schema + identity in resolved lanes and pipeline digest. +- Keep legacy registration helpers only as explicitly named transitional APIs; + do not let a raw registration satisfy a typed lane. +- Add composition tests with two artifact types and heterogeneous lanes. Cover + missing codec, missing variant, Go-type mismatch, duplicate variant, wrong + validator kind, stable resolution order, and digest changes on schema identity + or schema digest changes. + +Exit criteria: + +- heterogeneous typed test lanes resolve without module-facing erasure; +- every incompatible selection fails before execution; and +- existing raw production lanes continue to resolve only through their visible + transitional path. + +### Stage 9: Preparation and Construction Foundation + +Goal: construct and validate an entire resolved pipeline before source work. + +Tasks: + +- Add `ModuleDependencies`, `BuildRequest`, `PreparedPipeline`, and `Prepare` as + specified. +- Extend registry entries/specs with option validation and construction + functions. Supply adapters for legacy zero-argument constructors during the + migration. +- Call option validation for every selected module and validator during + resolution/config validation. Reject unknown fields and contextualize errors. +- Have preparation construct all selected components in fixed order and retain + immutable prepared lane executors. +- Update the runner API so `Run` receives a prepared pipeline. At the CLI, create + the shared scheduled LLM client, prepare, and only then invoke the runner. +- Prove with fakes that malformed options, missing required LLM dependencies, + and late-component construction failures occur before the input adapter's + `Parse` method. + +Exit criteria: + +- all components are constructed before source work; +- preparation errors identify exact scope and perform no operations; and +- legacy production modules still run through temporary construction adapters. + +### Stage 10: Migrate Universal Modules to Construction + +Goal: remove legacy option/dependency handling from input, chunk, and output. + +Tasks: + +- Give Seriatim input, generic units chunking, D&D scenes chunking, generic JSON + output, and all applicable chunk validators implementation-owned option + structs and strict decoders. +- Build those implementations with decoded options and injected dependencies. + Require the LLM client for D&D scenes; keep deterministic implementations + independent of it. +- Remove `Options` and `LLMClient` from the corresponding operation requests. + Retain per-run source, reference, profile, session, and metadata fields. +- Update registrars and focused tests. Verify options are decoded once during + preparation and operation methods do not inspect raw maps. + +Exit criteria: + +- no universal production module parses raw options during execution; and +- every universal LLM call uses the injected shared client. + +### Stage 11: Canonical D&D Model, Codec, and Typed Extractor + +Goal: establish the first production `T` and its extraction boundary. + +Tasks: + +- Add canonical spell types at the D&D root using engine-owned source refs. +- Add `internal/modules/dnd/codec/spells`, register kind `dnd/spell-list`, and + make it own the existing durable `dnd_spells.v1.json` schema and strict stable + encoding. +- Keep the private spell-extraction LLM DTO and + `dnd_spells_llm.v1.json` in the extractor package. Map canonicalized DTO values + to `dnd.SpellList` and do not expose the DTO to validators or the codec. +- Convert the extractor to `Extractor[dnd.SpellList]`, construction-time options + and dependency injection. Preserve prompt assets, retries, warnings, evidence, + and LLM response validation. +- Register the codec and typed extractor from the D&D registrar. +- Add codec compatibility tests against existing durable fixtures and tests + proving LLM schema ownership is separate from durable schema ownership. + +Exit criteria: + +- the typed extractor returns the canonical domain model; +- codec output is semantically identical to the maintained durable spell JSON; + and +- no downstream production consumer is switched until the next stages. + +### Stage 12: Typed Validators and Generic Typed Strategies + +Goal: complete all typed components required by the D&D lane. + +Tasks: + +- Convert D&D spell shape, source-reference, and source-relatedness validators + to `TypedValidator[dnd.SpellList]` with construction-time options/dependencies. +- Remove JSON reparsing from those validators. Remove the duplicate + `spellpayload` package after its final consumer migrates. +- Convert `valid_json` and `valid_json_schema` to serialized validators. Register + them so the framework uses the D&D codec when they occur in the spell chain. +- Implement generic typed append-order merge and no-op normalize. In the D&D + registrar, register D&D variants using a spell-list append function and + `noop[dnd.SpellList]`. +- Convert always-accept/reject into explicit chunk and typed variants and + register the D&D variants without changing their keys. Verify serialized + validators operate on the framework encoding at chunk and the codec encoding + at artifact stages. +- Test typed validator requests, source refs, relatedness LLM injection, generic + strategy reuse with a second test type, default chain order, and rejection + behavior. + +Exit criteria: + +- every selected spell-lane component has a compatible D&D typed variant; +- generic packages import no D&D code; and +- the duplicate validator payload model and inter-validator JSON parsing are + gone. + +### Stage 13: Typed D&D Runner Vertical Slice + +Goal: execute one complete production lane as `dnd.SpellList` while retaining +temporary raw output/checkpoint adapters. + +Tasks: + +- Implement the erased typed lane executor and runner path for extract, stage + retries, typed/serialized validation, merge, and normalize. +- Keep accepted extract values indexed by chunk and pass them to merge in source + order. Preserve warnings and rejections in stable scope order. +- Add narrow transitional adapters from typed stage outputs to the existing raw + checkpoint/debug/output envelopes. These adapters must use the registered + codec and be named/commented as migration-only. +- Route the production D&D lane through the typed path; leave no production raw + D&D stage module registered in parallel. +- Add end-to-end and checkpoint-disabled tests proving the maintained D&D output + and outcome semantics are unchanged. Add incompatible-lane tests proving + failure occurs before input. + +Exit criteria: + +- D&D values remain typed from extractor through normalize and typed validation; +- the runner's only D&D erasure is its private lane adapter and explicit codec + boundary; and +- durable output remains unchanged through the transitional adapter. + +### Stage 14: Typed Checkpoints, Debugging, and Output + +Goal: move every serialization side effect and the final Zone-C boundary to the +codec model. + +Tasks: + +- Change runner/output contracts so final normalized results are + `SerializedArtifact` values. Update generic JSON output without importing D&D. +- Preserve logical file names, index fields, media types, manifest contents, and + durable spell JSON. +- Change extract, merge, and normalize checkpoints to store codec bytes plus + artifact kind, schema ID, version, and schema digest. Decode reused values + back to `T` before the next stage. +- Implement safe invalidation for missing/mismatched codecs and decode failure, + with explicit checkpoint-event reasons. +- Change typed debug envelopes to serialize through the codec, preserving opt-in + and redaction behavior. Use stable codec bytes for artifact digests. +- Remove the transitional raw output/checkpoint/debug adapters introduced in + Stage 13 after all tests use the typed boundaries. + +Exit criteria: + +- typed values round-trip at every stage checkpoint; +- incompatible checkpoint artifacts recompute safely; +- output and debug code are domain-neutral; and +- no D&D typed lane depends on a raw-boundary adapter. + +### Stage 15: Finish Construction Migration and Remove Raw Contracts + +Goal: leave one production extension system rather than parallel raw and typed +models. + +Tasks: + +- Migrate any remaining merge, normalize, extract, and validator implementations + to construction-time option decoding and dependency injection. +- Remove LLM clients and raw option maps from all remaining operation requests. +- Remove legacy raw extractor/merger/normalizer/validator contracts, + constructors, registry entries, `RawPayload`, `ResponseSchema` if superseded, + raw clone helpers, raw checkpoint envelopes, and migration-only adapters. +- Remove dead duplicate models and compatibility helpers. Search for production + references to old stage-oriented paths and raw Zone-B types. +- Keep serialized artifacts only at codec, checkpoint/debug, and output + boundaries. + +Exit criteria: + +- no production Zone-B handoff uses JSON bytes, `RawPayload`, or `any`; +- all production configuration options are validated before execution; +- all production LLM users receive the one injected client; and +- there is no legacy production registration path. + +### Stage 16: Stage-Worker Configuration + +Goal: add the decided scheduling control without changing runner execution yet. + +Tasks: + +- Add `StageWorkers map[string]int` to effective concurrency configuration and + `stage_workers` to the version-2 YAML shape. Deep-clone the map. +- Accept only `extract`, reject unknown or empty keys, default missing extract to + effective `total_llm`, and validate the inclusive range + `1..total_llm` after file/environment precedence resolves. +- Add `NOTARIUS_STAGE_WORKERS_EXTRACT` with the existing environment precedence + and integer error style. +- Preserve redaction/effective-config diagnostics and version 2. +- Update `docs/config.md` and maintained examples that intentionally demonstrate + concurrency. Do not add the field to every example when the default conveys + the intended behavior. +- Add file, default, merge/precedence, environment, unknown-key, boundary, and + redacted-effective-config tests. + +Exit criteria: + +- every run has a validated effective extract worker count; +- omitted configuration preserves current effective behavior at the default + `total_llm: 1`; and +- current configuration documentation owns the implemented contract. + +### Stage 17: Concurrent Lane and Extract Scheduling + +Goal: implement bounded concurrent lanes while preserving deterministic public +behavior and the global LLM invariant. + +Tasks: + +- Add the fixed-size run-wide worker pool and central round-robin dispatcher. + Bound queued work so dispatch applies backpressure; do not enqueue the whole + run into unbounded memory. +- Treat extract, retries, and extract validators as one job. Publish immutable + results to a coordinator indexed by lane and chunk. +- Start each lane's serial merge/normalize continuation only after all its + extract jobs are terminal. Permit different lane continuations to overlap. +- Make the coordinator the sole writer of aggregate output, manifest, + checkpoint-event collection, warnings, and rejections. Use attempt-specific + debug paths and synchronize any recorder state that remains shared. +- Implement rejection, cancellation, stable sorting, deterministic primary + error selection, and output gating exactly as specified under Concurrency. +- Audit every concurrently reused extractor, validator, codec, LLM/debug wrapper, + checkpoint loader/recorder, and manifest metadata provider. Make production + implementations immutable or add narrowly scoped synchronization. +- Add deterministic barrier-controlled tests that force reverse completion + order, simultaneous failures, parent cancellation, rejections mixed with + successes, lane continuation overlap, and undispatched-job cancellation. +- Add instrumented integration tests issuing calls from multiple lanes, retries, + and LLM-backed validators. Assert provider calls never exceed + `total_llm`, extract jobs never exceed the effective extract worker count, and + both limits are exercised independently. +- Run `go test -race ./...` and eliminate races rather than weakening tests. + +Exit criteria: + +- lanes and extract jobs actually overlap when configured above one; +- job and provider-call limits are independently enforced; +- public results and primary errors are identical across forced completion + orders; and +- full race testing passes. + +### Stage 18: Documentation, Cleanup, and Final Verification + +Goal: make the implemented repository and its canonical documentation agree, +then close the focused roadmap. + +Tasks: + +- Review every current-behavior document routed by `docs/development.md` and + update only its owned facts: architecture and dependency direction, package + inventory, pipeline resolution/preparation/execution, typed module contracts, + LLM scheduling, configuration, checkpoint compatibility, diagnostics, + operations, and durable integration contracts. +- Verify maintained examples and copyable files against the implementation. +- Remove stale old package paths, raw-contract terminology, and superseded + future-work entries. Validate all changed documentation links. +- Update ADR consequences only in ways permitted for accepted ADR metadata; do + not edit accepted decision text. Record a new superseding ADR if final code + required an architectural change. +- Mark the feature roadmap implemented and reduce this implementation plan to a + concise completion record, or move it to the repository's established + completed-roadmap location if one exists. Do not let this file become a second + current-behavior reference. +- Run final checks: + + ```sh + go test ./... + go test -race ./... + go vet ./... + go build ./cmd/notarius + ``` + +Exit criteria: + +- all feature-roadmap completion criteria are met; +- no stale production paths or legacy typed/raw bridge remain; +- current documentation and examples describe only implemented behavior; and +- all final validation commands pass. + +## Open Questions + +None. The feature-policy decisions and the implementation choices required to +begin each stage are resolved above. If implementation evidence contradicts one +of the accepted architectural decisions, stop and handle that as an ADR change +or supersession rather than treating it as an implicit implementation choice.