Add ADR, roadmap, and staged implementation plan for input chunk caching

docs/adr/0005-cache-canonical-chunk-plans-by-source.md

@@ -0,0 +1,138 @@
+# ADR-0005: Cache one canonical chunk plan per source
+
+**Status:** Proposed
+**Date:** 2026-07-17
+
+## Context
+
+Notarius may run several extraction passes over the same source. A D&D
+transcript, for example, may first produce NPC artifacts and later produce
+spell or combat artifacts, with output from an earlier pass supplied as a
+reference to a later pass.
+
+An LLM-backed chunker may process an entire, potentially large source in one
+expensive request. Recomputing boundaries for every pipeline or pass repeats
+that cost and can make otherwise comparable extraction runs use different
+source partitions. Stable chunk material also gives later extraction requests
+a better opportunity to benefit from provider-side prompt caching.
+
+Chunk boundaries can affect extraction quality. Evidence may span a boundary,
+overlap may produce duplicates, and different partitions may change the context
+available to a model. Merge and normalization should remove structural signs
+of chunking from durable output, but they cannot guarantee recovery of evidence
+that an extractor did not receive.
+
+Notarius therefore needs an explicit policy for choosing between automatically
+applying the latest chunking configuration and preserving one stable partition
+for repeated work on the same source.
+
+## Decision
+
+Notarius assigns one active canonical chunk plan to a source and reuses that
+plan by default across pipelines and invocations.
+
+The canonical source identity is derived from the validated generic source
+document and covers the source-unit identity, order, and content needed to
+interpret plan boundaries. Input-adapter and chunk-producer identities are
+recorded as provenance, but the active-plan lookup does not vary with:
+
+- pipeline identity or selected artifact lanes;
+- the configured chunk module or its options;
+- references;
+- LLM provider, model, profile, prompt, or response schema; or
+- configuration for later pipeline stages.
+
+When an active plan exists, Notarius uses it even if the current pipeline
+configures a different chunk module or different chunk-module settings. The
+configured chunk module generates a plan only when none exists or when the
+operator explicitly requests recomputation.
+
+The framework-owned minimum plan contract is an ordered, non-empty set of
+source-unit ranges. Each range identifies the inclusive start and end unit for
+one chunk. A chunk module may also provide namespaced, domain-specific
+annotations at plan or range scope. Those annotations are stored with the plan
+and passed through the pipeline when present, but they remain optional.
+Downstream stages must not assume that annotations associated with the
+currently configured chunk module are present on a reused plan produced by a
+different module.
+
+The cache stores the plan rather than fully materialized chunks. The framework
+validates a reused plan against the current source and deterministically
+materializes its ranges into chunks. The same source and plan must produce
+byte-stable chunk input for later stages.
+
+Canonical plan storage is a distinct cache surface with an independently
+configurable location. It is not coupled to the roots or lifecycles of
+invocation checkpoints, diagnostics, debug artifacts, or durable output. This
+allows per-user and system-service deployments to apply cache-specific
+ownership, permissions, placement, and cleanup policy without relocating other
+Notarius state.
+
+One mutable active plan is stored under the canonical source identity and
+retains provenance for the module and relevant runtime inputs that produced it.
+The effective plan producer is reported separately from the chunk module
+requested by the current pipeline; reuse must not attribute cached boundaries
+or annotations to a module that did not produce them.
+
+Reuse is enabled by default. Operators can explicitly:
+
+- bypass cached plans for an invocation without changing the active plan; or
+- recompute a plan with the configured chunk module and make it active for
+  later work.
+
+Exact storage layout, configuration fields, CLI syntax, publication mechanics,
+recovery behavior, and diagnostics are implementation and operational
+contracts rather than part of this decision.
+
+## Alternatives considered
+
+- Recompute chunks on every invocation. This always applies the current
+  chunking configuration, but repeats the most expensive stage and weakens
+  provider-side caching and cross-pass comparability.
+- Cache every distinct chunking request by including module options,
+  references, prompts, profiles, and other runtime inputs in its identity. This
+  closely associates a cached result with its producing request, but reduces
+  reuse and permits boundary drift across operationally different passes.
+- Key plans by source plus chunk module and options. This shares plans across
+  pipelines using the same strategy, but changing the configured strategy
+  silently selects a different partition rather than preserving one canonical
+  partition for the source.
+- Require operators to name or supply a plan for every run. Explicit selection
+  is reproducible and may be useful as an advanced operation, but adds friction
+  to the default workflow and does not provide automatic reuse.
+- Store fully materialized chunks. This simplifies loading, but duplicates
+  source content and couples durable state to the current chunk representation
+  rather than the stable boundary decision.
+- Store canonical plans beneath the general workspace root. This would reuse an
+  existing location setting, but it couples a reusable application cache to
+  checkpoint, diagnostic, and debug state that have different ownership,
+  sensitivity, retention, and deployment requirements.
+
+## Consequences
+
+Independent pipelines and passes over the same source use stable boundaries by
+default. This reduces repeated LLM work, improves cross-pass comparability, and
+increases the opportunity for cached provider reads.
+
+The configured chunk module may not execute, and its settings may have no
+effect, when an active plan already exists. Domain-specific annotations reflect
+the plan's original producer and may be absent or differ from those the current
+module would produce. User-visible provenance must make the effective plan
+clear.
+
+A poor or outdated partition remains active until an operator replaces it.
+This can preserve suboptimal context boundaries and affect extraction recall or
+duplication even when merge and normalization hide the partition structure in
+durable output. Stable reuse is an intentional priority over automatically
+incorporating later chunk-strategy changes.
+
+The framework gains a durable minimal chunk-plan contract and deterministic
+materialization responsibility. Chunk modules must separate required boundary
+output from optional annotations, and downstream modules may rely only on the
+minimal boundary contract unless a future decision introduces an explicit plan
+compatibility mechanism.
+
+Operators must configure and secure canonical plan storage independently from
+other workspace state when the per-user default is not appropriate. Removing
+that cache remains recoverable because Notarius can regenerate it from the
+source, but doing so may repeat an expensive LLM operation.

docs/internal/modules.md

@@ -196,12 +196,15 @@ point for specializing reusable generic implementations, while the generic
 registrar composes only generic children.
 
 Core and framework production packages do not import production extensions.
-CLI production code imports only exact family registrar packages. Compatibility
-tests in the CLI, core, and framework trees may import roots and implementation
-leaves directly. White-box tests within module families retain the production
-family boundaries. `internal/modules/integration` is test infrastructure: its
-black-box tests may compose multiple families, but it is not a production
-module family or production dependency target.
+CLI production code is the sole application composition root for extensions
+and imports only exact family registrar packages. Other production packages,
+including commands and newly introduced package trees, do not import module
+packages directly. Compatibility tests in the CLI, core, and framework trees
+may import roots and implementation leaves directly. Other non-module tests do
+not receive that exemption. White-box tests within module families retain the
+production family boundaries. `internal/modules/integration` is test
+infrastructure: its black-box tests may compose multiple families, but it is
+not a production module family or production dependency target.
 
 ## Adding An Extension
 

docs/internal/pipeline.md

@@ -236,17 +236,17 @@ and terminal error text; failures before a candidate exists omit that payload.
 Only LLM calls made by the module operation belong to the module attempt.
 Validator calls retain independent scopes under `validate/` and are not
 duplicated into the module envelope. A failed terminal-envelope write is a
-framework error and is joined with any primary attempt error. Debug data is
-never used as a checkpoint source. Typed artifact debug envelopes are
-domain-neutral, redact sensitive metadata and bytes through the common debug
-policy, and record codec identity plus schema and content digests.
+non-retryable framework error and is joined with any primary attempt error.
+Debug data is never used as a checkpoint source. Typed artifact debug envelopes
+are domain-neutral, redact sensitive metadata and bytes through the common
+debug policy, and record codec identity plus schema and content digests.
 
 Merge and normalize attempts serialize their in-memory candidate with the
-codec's candidate encoder before typed validation. Serialized validators and
-attempt debug use that candidate representation, which carries the codec media
-type and schema identity but is never checkpointed or passed downstream. Only
-a validator-approved value is encoded through the strict final codec and made
-eligible for a checkpoint or stage output.
+codec's required candidate encoder before typed validation. Serialized
+validators and attempt debug use that candidate representation, which carries
+the codec media type and schema identity but is never checkpointed or passed
+downstream. Only a validator-approved value is encoded through the strict final
+codec and made eligible for a checkpoint or stage output.
 
 Checkpoint identity, physical layout, reuse behavior, and debug artifact
 handling are operator contracts in [Operations](../operations.md). Serialization

docs/operations.md

@@ -155,8 +155,8 @@ envelope recording acceptance, validator rejection, or a module, validator,
 candidate-serialization, or final-serialization error as applicable. It
 includes attempt-local warnings and any available candidate or rejection. A
 failure before a candidate exists has no candidate payload. If the envelope
-cannot be persisted, the run reports that debug failure together with any
-primary attempt error.
+cannot be persisted, the run does not retry that module attempt and reports the
+debug failure together with any primary attempt error.
 
 Checkpoint-reused chunk, extract, merge, and normalize work retains the
 stage-level input and output artifacts but has no retry-attempt artifacts

docs/roadmap/adr0005.md

@@ -0,0 +1,166 @@
+# ADR-0005 Feature Roadmap
+
+This roadmap defines the intended end state for
+[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature
+is not implemented. The ordered coding work belongs in the
+[implementation plan](implementation.md).
+
+## Intent
+
+Notarius may run multiple extraction passes over the same source. Chunking,
+especially LLM-backed chunking over a large transcript, can be substantially
+more expensive than later per-chunk operations. Stable chunk material also
+improves the opportunity for provider-side prompt-cache reads across passes.
+
+Notarius therefore prioritizes stable, aggressively reused source partitions
+over automatically applying later chunk-module, reference, prompt, model, or
+pipeline changes. An operator must explicitly request repartitioning.
+
+## Target State
+
+### Canonical source plan
+
+Each validated generic source document has at most one stored chunk plan. The
+source document's canonical digest selects that plan. Pipeline identity,
+selected lanes, configured chunk module and options, references, validators,
+prompts, schemas, and LLM settings do not participate in lookup.
+
+In the default mode, the first accepted plan for a source is reused by later
+pipelines and invocations. A configured chunk module runs only when no valid
+stored plan exists, when reuse is bypassed for one invocation, or when the
+operator requests refresh.
+
+### Plan and annotation contract
+
+A plan contains an ordered, non-empty collection of inclusive source-unit
+ranges. The framework validates these boundaries and deterministically
+materializes runtime chunks from the current source document.
+
+Chunk modules may attach optional domain annotations at plan or range scope.
+Annotations are namespaced, validated JSON values. They are preserved through
+storage and materialization but are never a hard cross-domain capability.
+Downstream modules may rely on the generic chunk contract only.
+
+The store contains boundaries, annotations, warnings, and provenance. It does
+not contain fully materialized chunks or duplicated source-unit content.
+
+### Persistence
+
+Chunk plans use a dedicated cache root that is independent of
+`workspace.directory` and the locations of checkpoints, diagnostics, debug
+artifacts, and durable output. The default is the platform-appropriate per-user
+cache directory: on Linux, `$XDG_CACHE_HOME/notarius/chunk-plans` when
+`XDG_CACHE_HOME` is set to a valid absolute path, otherwise
+`$HOME/.cache/notarius/chunk-plans` when it is unset, through the platform
+cache-directory resolver. An invalid relative `XDG_CACHE_HOME` is an error, not
+a fallback.
+
+Operators may set `workspace.chunk_cache.directory` to replace that root. The
+recommended system-wide setting for a dedicated Linux service account is
+`/var/cache/notarius/chunk-plans`. This is an operational recommendation, not
+the application default; the service account must own the directory and it must
+not be shared across mutually untrusted users.
+
+One mutable, versioned plan file is stored beneath the chunk-plan root under the
+full canonical source digest. Normal publication and refresh replace that file
+atomically; history, rollback, multiple variants, and content deduplication are
+outside this feature.
+
+Plan state is potentially sensitive. Paths are confined, directories and files
+use restrictive permissions, default diagnostics omit annotation and source
+payloads, and debug output remains explicitly opt-in.
+
+### Runtime policy
+
+The effective chunk-cache mode is one of:
+
+- `auto`: load a valid stored plan; otherwise generate, validate, and publish
+  one;
+- `bypass`: do not read or write plan state for this invocation; or
+- `refresh`: generate and validate a plan, then atomically replace stored state.
+
+`auto` is the default. Invalid or incompatible stored state is a reported miss
+in `auto`; it is replaced only after a newly generated plan is accepted. A
+failed generation never overwrites prior state. Concurrent writes must never
+expose partial data and use last-successful-atomic-write semantics.
+
+The public CLI override is
+`--chunk_cache <auto|bypass|refresh>`. Configuration and environment values use
+the same three modes, with CLI taking highest precedence.
+
+### Validation and execution
+
+Framework plan validation and deterministic materialization run for generated
+and reused plans. The current pipeline's configured chunk-validator chain then
+validates the materialized chunks.
+
+On a cache hit, the configured chunk module is constructed during normal
+pipeline preparation but its operation is not invoked and it makes no LLM call.
+A validator rejection of a structurally valid reused plan is a run outcome; it
+does not implicitly authorize rechunking.
+
+Only the generic `chunks` capability is hard. Domain annotations such as D&D
+scene information are opportunistic and cannot be required solely because the
+current pipeline selected the module that normally produces them.
+
+Canonical plans are the sole owner of chunk reuse. Invocation-scoped
+checkpoints no longer load or record chunk outputs. Downstream checkpoint
+identity continues to depend on the digest of the effective materialized
+chunks, so refresh invalidates affected downstream work.
+
+### Provenance
+
+Every run distinguishes the chunk module requested by the resolved pipeline
+from the producer of the effective stored plan. Durable provenance records the
+source and plan digests, plan schema version, effective cache mode and action,
+producer module, producer references and LLM profile where applicable, and
+non-sensitive producer metadata.
+
+Default diagnostics record lookup, validation, generation, and publication
+decisions without source or annotation payloads. Opt-in debug artifacts may
+contain complete plan and annotation material and are treated as sensitive.
+
+### Deployment documentation
+
+The implemented configuration reference identifies the dedicated directory
+field, its environment override, and the per-user default. The operations guide
+documents the resulting filesystem layout and permissions, and recommends
+`/var/cache/notarius/chunk-plans` for a system-wide Linux deployment running as
+a dedicated service account. It also explains that cache deletion is
+recoverable but may repeat expensive chunk generation. Neither document
+presents the system-wide path as the default for an ordinary unprivileged
+invocation.
+
+## Compatibility Policy
+
+Existing fully materialized chunk checkpoint files are not migrated or promoted
+to canonical plans. They are ignored for chunk reuse after this feature lands.
+The first `auto` run generates the source's plan through the configured chunk
+module.
+
+The existing source digest is the canonical lookup identity. Any source change
+that alters that digest creates a separate plan. Earlier plan schema versions,
+malformed files, and digest mismatches are incompatible and follow the invalid
+`auto`-miss behavior.
+
+## Non-Goals
+
+This feature does not provide:
+
+- plan editing, comparison, history, rollback, or garbage collection;
+- remote or shared plan storage;
+- multiple active or automatically selected plan variants for one source;
+- mandatory domain annotation contracts;
+- automatic rechunking because configuration or model inputs changed; or
+- a guarantee that an LLM provider will report cache hits.
+
+## Completion Outcomes
+
+The feature is complete when independent runs over the same canonical source
+reuse byte-stable materialized chunks across pipeline, lane, reference,
+chunk-module, and LLM configuration changes; bypass and refresh obey their
+documented state semantics; provenance identifies the effective producer;
+legacy chunk checkpoints cannot compete with plan reuse; and all focused,
+integration, compatibility, CLI, and repository-wide validation passes. The
+configuration and operations references must also document both the per-user
+default and the recommended system-wide Linux setting.

docs/roadmap/domain.md

@@ -1,348 +0,0 @@
-# Domain-Typed Pipeline Feature Roadmap
-
-## Status
-
-Implemented on 2026-07-17. This roadmap records the design delivered for
-[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
-[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
-[ADR-0004](../adr/0004-package-modules-by-domain.md). Its implementation history
-is summarized in the [completion record](implementation.md).
-
-This file is historical design context, not a current-behavior reference.
-Implemented contracts and mechanics are defined by the architecture,
-configuration, integration, operations, and internal documentation outside
-`docs/roadmap/`.
-
-## User Intent
-
-Notarius should remain a small, explicit pipes-and-filters application while
-making domain extensions safe to compose and straightforward to maintain. A
-configured pipeline should fail before execution when its modules are
-incompatible, should carry typed domain values rather than reparsed JSON between
-artifact stages, and should preserve provenance and durable output contracts.
-
-The application must also enforce one configurable, process-wide ceiling on
-in-flight LLM calls. Pipeline scheduling may impose stricter limits, but no
-stage, lane, retry, validator, or future LLM-backed extension may bypass that
-global ceiling.
-
-## Target Architecture
-
-### Pipeline and outcome model
-
-The topology remains:
-
-```text
-input -> chunk -> extract -> merge -> normalize -> output
-```
-
-Input and chunk are pipeline-wide. Extract, merge, normalize, and their
-validators operate per artifact lane. Output aggregates the terminal artifacts
-from all lanes. The resolved pipeline retains explicit fields for those roles;
-it does not become a general DAG or a heterogeneous ordered-stage list.
-
-Framework errors abort the run. Validator rejection is a recorded domain
-outcome and does not abort unrelated work. Accepted extract results reach merge
-in source-chunk order, regardless of execution completion order. Warnings,
-rejections, artifacts, and reported errors are likewise ordered by stable
-pipeline scope rather than goroutine completion time.
-
-### Engine-owned source model
-
-The engine owns `source.SourceDocument`, `source.SourceUnit`, `source.SourceRef`,
-and `source.Chunk`. Domain modules may consume these types but must not redefine
-their provenance semantics.
-
-- Every source unit has a canonical self-reference identifying its source and
-  unit range.
-- A chunk contains ordered source units and one canonical reference spanning
-  its first through last unit.
-- Chunk and unit references are validated for source identity, order, and
-  containment.
-- Auxiliary reference material remains distinct from source provenance.
-- Cloning, canonicalization, checkpointing, debugging, and digest computation
-  preserve the source references exactly.
-
-`source.Chunk.Ref` replaces duplicate start/end boundary fields. Because that
-changes persisted workspace state, the workspace checkpoint schema advances to
-`notarius.workspace.v2`. Existing v1 checkpoints are left intact but treated as
-incompatible and recomputed; no in-place migration or deletion is required.
-
-### Typed artifact lanes
-
-Each lane has one canonical artifact type `T` from extraction through merge,
-normalization, and typed validation. Module-facing Zone-B contracts are generic:
-
-- `Extractor[T]` produces typed per-chunk values plus framework-owned
-  provenance and diagnostics;
-- `Merger[T]` combines accepted values in source-chunk order;
-- `Normalizer[T]` canonicalizes the merged value; and
-- `TypedValidator[T]` applies semantic checks at its configured artifact stage.
-
-The framework may use private erased adapters to keep heterogeneous lanes in
-one resolved pipeline, but `any`, raw JSON, and a generic `Process(any)` API are
-not module-facing handoffs. The extractor selected for a lane establishes its
-artifact kind. Resolution uses that kind to select compatible merger,
-normalizer, validator, and codec variants and rejects an incompatible lane
-before any stage executes.
-
-Generic strategies remain reusable without knowing concrete domains. In
-particular, append-order merge is parameterized by a typed combine function
-provided during domain registration, and no-op normalization is instantiated
-for the lane's concrete type.
-
-### Artifact identity and codecs
-
-Every typed artifact kind has exactly one registered `ArtifactCodec[T]`. An
-artifact kind is a stable logical identifier, separate from a Go type name. A
-codec owns:
-
-- artifact kind;
-- schema identifier, name, and version;
-- media type and JSON Schema bytes; and
-- strict, deterministic encoding and decoding between `T` and the serialized
-  representation.
-
-Equal canonical values must encode to equal bytes. Those bytes are the basis
-for artifact digests. Codec decoding rejects malformed or schema-incompatible
-content. Domain validators continue to own semantic validity; codecs do not
-replace them.
-
-`SerializedArtifact` is the Zone-C representation and includes the artifact
-kind, schema metadata, media type, encoded content, and framework metadata.
-Type erasure occurs through the codec after normalization for final output.
-Intermediate checkpointing and opt-in debug recording may also use the codec,
-but serialization for those side effects is not a stage handoff.
-
-Checkpoint metadata records artifact kind, schema identifier, schema version,
-and schema digest. Reuse requires an exact compatible registered codec;
-otherwise the checkpoint is safely invalidated. Output remains domain-neutral
-and consumes serialized artifacts.
-
-Generic serialized validators remain supported for representation-level checks
-such as valid JSON and JSON Schema validation. The framework encodes `T` through
-its registered codec before invoking them. Domain validators receive `T`
-directly. Chunk-stage validators remain in the source zone: semantic chunk
-validators receive engine-owned chunks, while representation-level validators
-receive the framework's canonical serialized chunk view. They do not force
-source-zone values through a domain artifact codec.
-
-### Registration and resolution
-
-Registries expose typed registration helpers while privately retaining the Go
-type identity needed to assemble erased lane executors.
-
-- Codecs are keyed by artifact kind, with exactly one codec per kind.
-- Extractors are keyed by their existing module key and declare an artifact
-  kind.
-- Mergers, normalizers, and validators are keyed by `(module key, artifact
-  kind)`, allowing stable generic keys such as `appendorder` and `noop` to have
-  multiple typed specializations.
-- Resolved pipeline identity and dependency fingerprints include artifact kind
-  and schema identity, version, and digest.
-- Duplicate or incompatible registrations and selections fail deterministically
-  during composition or resolution.
-
-The framework's public typed registration surface uses free generic functions,
-because Go methods cannot declare their own type parameters. Private reflection
-may verify and erase registered types, but it is not exposed to module authors.
-
-### Preparation, options, and dependencies
-
-Pipeline execution is split into resolution, preparation, and running.
-Preparation constructs every selected module and validator before source input
-begins and returns a prepared pipeline with explicit input, chunk, lane, and
-output fields.
-
-- Construction receives framework-owned dependencies, including the one shared
-  scheduled structured-LLM client.
-- Raw configured options are decoded once into implementation-owned option
-  structs during preparation.
-- Missing dependencies, malformed options, unknown options, and incompatible
-  typed selections fail before stage execution.
-- Configuration validation uses the same option decoders without requiring live
-  provider dependencies.
-- Per-run data such as sources, chunks, references, session identity, lane
-  identity, and metadata remains in operation requests.
-- Constructed implementations that can be scheduled concurrently are immutable
-  after preparation or otherwise explicitly concurrency-safe.
-
-Modules and validators must not construct provider clients, wrap their own
-independent schedulers, or bypass the injected scheduled client.
-
-### D&D artifact model
-
-The D&D package root owns canonical `dnd.SpellList`, `dnd.SpellCast`, and
-related evidence types, using engine-owned `source.SourceRef` values. The spell
-extractor keeps its LLM response DTO and response schema private and maps the
-canonicalized response to the domain model.
-
-The D&D spell codec separately owns the existing durable spell artifact schema.
-The LLM response schema and durable artifact schema remain distinct contracts
-even if their current JSON shapes are similar. Shape, source-reference, and
-source-relatedness validators operate on the canonical typed model. The
-validator-only duplicate spell model and inter-stage JSON reparsing disappear.
-
-The migration preserves the existing D&D spell payload, logical output bundle,
-module and validator keys, prompt/schema identities, default validator chains,
-warnings, rejection semantics, and manifest provenance unless a separate
-compatibility decision explicitly changes one of those contracts.
-
-## Package Ownership
-
-The target production extension layout is:
-
-```text
-internal/modules/dnd/
-  types.go
-  codec/spells/
-  chunk/scenes/
-  extract/spells/
-  validate/spells/shape/
-  validate/spells/source_refs/
-  validate/spells/source_relatedness/
-  shared/
-  register/
-
-internal/modules/generic/
-  chunk/units/
-  merge/appendorder/
-  normalize/noop/
-  validate/always_accept/
-  validate/always_reject/
-  validate/valid_json/
-  validate/valid_json_schema/
-  output/json/
-  register/
-
-internal/modules/seriatim/
-  input/transcript/
-  register/
-
-internal/framework/promptfs/
-```
-
-Shared domain types live at the domain root. Registration lives in a sibling
-`register` package so that the root never imports child implementations. Each
-registrar exposes one composition entry point accepting the pipeline registry
-set and LLM asset registry. The CLI composition root creates those registries
-and invokes the generic, Seriatim, and D&D registrars.
-
-Concrete domain implementations do not import peer domains. Generic extensions
-never import a concrete domain. A domain registrar may import generic packages
-to register typed specializations for its domain. The application composition
-root and designated black-box integration tests may compose multiple
-registrars. Domain-neutral embedded prompt-asset filesystem support belongs to
-the framework rather than a domain package.
-
-## Concurrency Policy
-
-### Configuration
-
-The existing `concurrency.total_llm` setting remains the application-wide
-ceiling on actual provider calls. Version-2 configuration gains an extensible
-stage-worker map:
-
-```yaml
-concurrency:
-  total_llm: 4
-  stage_workers:
-    extract: 4
-```
-
-Initially, `extract` is the only recognized key. Unknown stage keys are rejected
-so misspellings cannot silently alter scheduling. If omitted, the effective
-extract worker count equals `total_llm`. Its valid range is
-`1..concurrency.total_llm`. The environment override is
-`NOTARIUS_STAGE_WORKERS_EXTRACT`; future stage overrides receive similarly
-explicit names that map to the extensible file representation.
-
-The worker setting bounds framework jobs, not provider calls. Only an actual
-LLM call consumes a permit from the shared scheduled client. The global
-scheduled client remains authoritative even if future stages gain worker limits.
-
-### Scheduling
-
-After pipeline-wide chunking, all lanes may run concurrently. A central
-dispatcher submits `(lane, chunk)` jobs to one run-wide extract worker pool in
-round-robin order: source chunk first, then resolved lane order. This avoids one
-unbounded goroutine per job and prevents an early lane from monopolizing the
-queue.
-
-One job contains extraction, its stage-local retry behavior, and extract-stage
-validation for that lane and chunk. Each lane begins its serial merge then
-normalize continuation when all of its extract jobs reach a terminal state.
-Different lanes' continuations may overlap, and any LLM-backed continuation or
-validator still shares the global scheduled client.
-
-Workers publish immutable task results to a coordinator. Only the coordinator
-mutates aggregate results, manifests, checkpoint indexes, warnings, and
-rejections. Debug artifacts use attempt-specific paths and do not rely on
-concurrent writes to shared files.
-
-Rejections do not cancel work. A framework error cancels the derived run
-context, stops undispatched jobs, and waits for started jobs to finish or
-observe cancellation. If the parent context was canceled, its error is
-returned. Otherwise, internal cancellation errors are ignored when at least one
-real framework error exists, and the primary returned error is selected from
-all started-task framework errors by this stable ordering:
-
-1. stage order: extract, merge, then normalize;
-2. resolved lane order;
-3. source chunk index for chunk-scoped work; and
-4. configured validator or operation order within that scope.
-
-The full per-task errors may be retained in opt-in diagnostics, but completion
-timing never chooses the public error. Output runs only after every lane reaches
-a successful or rejection-only terminal state and no framework error exists.
-
-Extractors and any validator instance callable by multiple workers must be safe
-for concurrent use. Production implementations should normally satisfy this by
-being immutable after preparation.
-
-## Compatibility and Safety
-
-- Existing production module keys, validator keys, profiles, default chains,
-  and maintained configurations continue to resolve.
-- Existing durable D&D JSON content and logical output paths remain unchanged.
-- Framework errors, validator rejections, retries, checkpoints, diagnostics,
-  and debug behavior retain their current semantics except for the explicitly
-  documented workspace-v2 compatibility boundary and deterministic concurrent
-  ordering.
-- All provider calls pass through the shared global scheduler, across lanes,
-  stages, retries, and validators.
-- Source text and LLM payloads remain subject to the existing opt-in debug and
-  sensitive-data handling policies.
-- Package moves do not create user-visible key changes or concrete cross-domain
-  dependencies.
-
-## Non-Goals
-
-This feature does not introduce:
-
-- a general DAG or configurable stage topology;
-- out-of-process plugins or an RPC extension protocol;
-- cross-lane normalization;
-- a new durable D&D spell schema merely to mirror internal Go types;
-- per-stage LLM permit pools that could exceed or partition the global ceiling;
-  or
-- concurrent work implemented through an unbounded goroutine per lane or chunk.
-
-## Completion Criteria
-
-The target state is reached when:
-
-- all production lanes use one typed artifact from extract through normalize
-  and typed validation;
-- codecs own schema-aware serialization at every type-erasure, checkpoint, and
-  debug boundary;
-- incompatible lane composition and invalid options fail before source work;
-- source units and chunks carry validated canonical provenance;
-- production extensions follow the domain-first package and registrar rules;
-- extract scheduling is bounded, deterministic, concurrent across lanes, and
-  race-free;
-- instrumented tests prove actual concurrent LLM calls never exceed
-  `concurrency.total_llm` across all callers;
-- the maintained D&D example and compatibility baselines retain their durable
-  contracts; and
-- current-behavior documentation is updated as each implemented boundary lands.

internal/cli/run_test.go

@@ -3788,6 +3788,9 @@ func (fakeRunCodec) Schema() contracts.ArtifactSchema {
 	return contracts.ArtifactSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
 }
 func (fakeRunCodec) MediaType() string { return "application/json" }
+func (fakeRunCodec) EncodeCandidate(v fakeRunArtifact) ([]byte, error) {
+	return json.Marshal(v)
+}
 func (fakeRunCodec) Encode(v fakeRunArtifact) ([]byte, error) { return json.Marshal(v) }
 func (fakeRunCodec) Decode(b []byte) (fakeRunArtifact, error) {
 	var v fakeRunArtifact

internal/core/config/validation_test.go

@@ -643,6 +643,9 @@ func (fakeArtifactCodec) Schema() contracts.ArtifactSchema {
 	return contracts.ArtifactSchema{ID: "urn:notarius:test:artifact", Name: "Test artifact", Version: "1", JSONSchema: []byte(`{"type":"string"}`)}
 }
 func (fakeArtifactCodec) MediaType() string { return "application/json" }
+func (fakeArtifactCodec) EncodeCandidate(value fakeArtifact) ([]byte, error) {
+	return []byte(fmt.Sprintf("%q", value)), nil
+}
 func (fakeArtifactCodec) Encode(value fakeArtifact) ([]byte, error) {
 	return []byte(fmt.Sprintf("%q", value)), nil
 }

internal/framework/contracts/artifact.go

@@ -41,6 +41,10 @@ type ArtifactCodec[T any] interface {
 	Kind() ArtifactKind
 	Schema() ArtifactSchema
 	MediaType() string
+	// EncodeCandidate serializes a stage result before semantic validation. It
+	// must not apply validity checks owned by typed validators; Encode remains
+	// the strict final-artifact boundary used after validation succeeds.
+	EncodeCandidate(T) ([]byte, error)
 	Encode(T) ([]byte, error)
 	Decode([]byte) (T, error)
 }

internal/framework/pipeline/artifact_codec_registry.go

@@ -120,20 +120,17 @@ func RegisterArtifactCodec[T any](registry *ArtifactCodecRegistry, codec contrac
 			return decoded, nil
 		},
 	}
-	entry.encodeCandidate = entry.encode
-	if candidate, ok := any(codec).(interface{ EncodeCandidate(T) ([]byte, error) }); ok {
 	entry.encodeCandidate = func(value any) ([]byte, error) {
 		typed, err := exactTypedValue[T]("encode candidate artifact", value)
 		if err != nil {
 			return nil, err
 		}
-			content, err := candidate.EncodeCandidate(typed)
+		content, err := codec.EncodeCandidate(typed)
 		if err != nil {
-				return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
+			return nil, &ArtifactCodecOperationError{Operation: "encode candidate", Kind: spec.Kind, Err: err}
 		}
 		return append([]byte(nil), content...), nil
 	}
-	}
 	if provider, ok := any(codec).(interface{ Metadata(T) map[string]any }); ok {
 		entry.metadata = func(value any) map[string]any {
 			typed, err := exactTypedValue[T]("artifact metadata", value)

internal/framework/pipeline/artifact_codec_registry_test.go

@@ -28,15 +28,24 @@ type testArtifactCodec[T any] struct {
 	schema        contracts.ArtifactSchema
 	mediaType     string
 	encodeFunc    func(T) ([]byte, error)
+	candidateFunc func(T) ([]byte, error)
 	decodeFunc    func([]byte) (T, error)
 }
 
 func (c testArtifactCodec[T]) Kind() contracts.ArtifactKind     { return c.kind }
 func (c testArtifactCodec[T]) Schema() contracts.ArtifactSchema { return c.schema }
 func (c testArtifactCodec[T]) MediaType() string                { return c.mediaType }
+func (c testArtifactCodec[T]) EncodeCandidate(value T) ([]byte, error) {
+	if c.candidateFunc != nil {
+		return c.candidateFunc(value)
+	}
+	return c.encodeFunc(value)
+}
 func (c testArtifactCodec[T]) Encode(value T) ([]byte, error)   { return c.encodeFunc(value) }
 func (c testArtifactCodec[T]) Decode(content []byte) (T, error) { return c.decodeFunc(content) }
 
+var _ contracts.ArtifactCodec[codecNotes] = testArtifactCodec[codecNotes]{}
+
 func TestArtifactCodecRegistryStoresHeterogeneousExactTypes(t *testing.T) {
 	registry := NewArtifactCodecRegistry()
 	if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
@@ -92,6 +101,43 @@ func TestArtifactCodecRegistryStoresHeterogeneousExactTypes(t *testing.T) {
 	}
 }
 
+func TestArtifactCodecRegistryKeepsCandidateAndFinalEncodingDistinct(t *testing.T) {
+	candidateCalls, finalCalls := 0, 0
+	codec := notesCodec()
+	codec.candidateFunc = func(codecNotes) ([]byte, error) {
+		candidateCalls++
+		return []byte(`{"items":["candidate"]}`), nil
+	}
+	codec.encodeFunc = func(codecNotes) ([]byte, error) {
+		finalCalls++
+		return []byte(`{"items":["final"]}`), nil
+	}
+	registry := NewArtifactCodecRegistry()
+	if err := RegisterArtifactCodec(registry, codec); err != nil {
+		t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
+	}
+	entry, _, err := registry.entry(codec.kind)
+	if err != nil {
+		t.Fatalf("entry() error = %v, want nil", err)
+	}
+
+	candidate, err := serializeArtifact(entry, codecNotes{}, true)
+	if err != nil {
+		t.Fatalf("serialize candidate error = %v, want nil", err)
+	}
+	if string(candidate.Content) != `{"items":["candidate"]}` || candidateCalls != 1 || finalCalls != 0 {
+		t.Fatalf("candidate content = %s, calls = candidate %d, final %d", candidate.Content, candidateCalls, finalCalls)
+	}
+
+	final, err := serializeArtifact(entry, codecNotes{}, false)
+	if err != nil {
+		t.Fatalf("serialize final error = %v, want nil", err)
+	}
+	if string(final.Content) != `{"items":["final"]}` || candidateCalls != 1 || finalCalls != 1 {
+		t.Fatalf("final content = %s, calls = candidate %d, final %d", final.Content, candidateCalls, finalCalls)
+	}
+}
+
 func TestArtifactCodecRegistryStoresValidatedSchemaMetadata(t *testing.T) {
 	registry := NewArtifactCodecRegistry()
 	codec := notesCodec()

internal/framework/pipeline/debug.go

@@ -409,6 +409,17 @@ type attemptTerminalRecorder struct {
 	envelope debugTimedEnvelope
 }
 
+type attemptDebugPersistenceError struct {
+	label string
+	err   error
+}
+
+func (e *attemptDebugPersistenceError) Error() string {
+	return fmt.Sprintf("write %s attempt debug artifact: %v", e.label, e.err)
+}
+
+func (e *attemptDebugPersistenceError) Unwrap() error { return e.err }
+
 func newAttemptTerminalRecorder(recorder DebugRecorder, attemptPath, label string, scope *debugLLMScope, envelope debugTimedEnvelope) attemptTerminalRecorder {
 	return attemptTerminalRecorder{recorder: recorder, path: attemptPath, label: label, scope: scope, envelope: envelope}
 }
@@ -420,7 +431,7 @@ func (r attemptTerminalRecorder) record(payload any, terminalErr error) error {
 		envelope.Error = terminalErr.Error()
 	}
 	if err := writeDebugAttempt(r.recorder, r.path, envelope, r.scope); err != nil {
-		debugErr := fmt.Errorf("write %s attempt debug artifact: %w", r.label, err)
+		debugErr := &attemptDebugPersistenceError{label: r.label, err: err}
 		return errors.Join(terminalErr, debugErr)
 	}
 	return terminalErr

internal/framework/pipeline/default_modules_test.go

@@ -144,6 +144,9 @@ func (defaultArtifactCodec) Schema() contracts.ArtifactSchema {
 	return contracts.ArtifactSchema{ID: "urn:notarius:test:default", Name: "default", Version: "1", JSONSchema: []byte(`{"type":"object"}`)}
 }
 func (defaultArtifactCodec) MediaType() string { return "application/json" }
+func (defaultArtifactCodec) EncodeCandidate(value defaultArtifact) ([]byte, error) {
+	return json.Marshal(value)
+}
 func (defaultArtifactCodec) Encode(value defaultArtifact) ([]byte, error) { return json.Marshal(value) }
 func (defaultArtifactCodec) Decode(content []byte) (defaultArtifact, error) {
 	var value defaultArtifact

internal/framework/pipeline/runner.go

@@ -357,6 +357,10 @@ func runWithRetry(ctx context.Context, retries int, run func(attempt int) (bool,
 		}
 		accepted, rejection, err := run(attempt)
 		if err != nil {
+			var debugErr *attemptDebugPersistenceError
+			if errors.As(err, &debugErr) {
+				return false, nil, fmt.Errorf("failed after %d attempt(s): %w", attempt, err)
+			}
 			if attempt == attempts {
 				return false, nil, fmt.Errorf("failed after %d attempt(s): %w", attempt, err)
 			}

internal/framework/pipeline/runner_terminal_debug_test.go

@@ -18,6 +18,7 @@ type terminalChunker struct {
 	chunks   []source.Chunk
 	warnings []contracts.Warning
 	err      error
+	calls    *int
 }
 
 func (c terminalChunker) Key() string { return c.key }
@@ -25,6 +26,9 @@ func (c terminalChunker) Key() string { return c.key }
 func (terminalChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
 
 func (c terminalChunker) Chunk(context.Context, contracts.ChunkRequest) (contracts.ChunkResult, error) {
+	if c.calls != nil {
+		(*c.calls)++
+	}
 	return contracts.ChunkResult{Chunks: cloneSourceChunks(c.chunks), Warnings: cloneWarnings(c.warnings)}, c.err
 }
 
@@ -211,6 +215,30 @@ func TestRunnerJoinsPrimaryAndAttemptWriteErrors(t *testing.T) {
 	})
 }
 
+func TestRunnerDoesNotRetryAfterTerminalAttemptWriteFailure(t *testing.T) {
+	prepared, chunks := preparedTerminalDebugPipeline(t)
+	prepared.resolved.Chunk.Retries = 1
+	calls := 0
+	prepared.chunker = terminalChunker{key: prepared.resolved.Chunk.Module, chunks: chunks, calls: &calls}
+	prepared.chunkValidators.validators = []preparedValidator{{
+		resolved: ResolvedValidator{Binding: Binding("terminal/chunk-validator"), Target: ValidatorTargetChunk},
+		chunk:    terminalChunkValidator{result: contracts.ValidationResult{Approved: true}},
+	}}
+	debug := newCapturedDebugRecorder()
+	debug.failPath = "chunk/attempt-01.json"
+
+	_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug})
+	if err == nil || !strings.Contains(err.Error(), "write chunk attempt debug artifact") || !strings.Contains(err.Error(), "debug recorder failure") {
+		t.Fatalf("Run() error = %v, want terminal attempt debug failure", err)
+	}
+	if calls != 1 {
+		t.Fatalf("chunk calls = %d, want one attempt without retry", calls)
+	}
+	if debug.has("chunk/attempt-02.json") {
+		t.Fatal("second chunk attempt envelope exists after non-retryable debug persistence failure")
+	}
+}
+
 func TestRunnerKeepsExtractModuleAndValidatorLLMCallsIsolated(t *testing.T) {
 	prepared := preparedAttemptDebugPipeline(t)
 	debug := newCapturedDebugRecorder()

internal/modules/import_boundaries_test.go

@@ -189,6 +189,41 @@ func TestImportBoundaryRules(t *testing.T) {
 			sourcePackage: "cli",
 			importPath:    moduleImportPrefix + "almanac/extract/events",
 		},
+		{
+			name:          "command production cannot import registrar",
+			filename:      "cmd/notarius/main.go",
+			sourcePackage: "main",
+			importPath:    moduleImportPrefix + "almanac/register",
+			wantError:     true,
+		},
+		{
+			name:          "command production cannot import concrete leaf",
+			filename:      "cmd/notarius/main.go",
+			sourcePackage: "main",
+			importPath:    moduleImportPrefix + "almanac/extract/events",
+			wantError:     true,
+		},
+		{
+			name:          "unknown production package cannot import concrete leaf",
+			filename:      "internal/application/bootstrap.go",
+			sourcePackage: "application",
+			importPath:    moduleImportPrefix + "almanac/extract/events",
+			wantError:     true,
+		},
+		{
+			name:          "unknown production package cannot import registrar",
+			filename:      "internal/application/bootstrap.go",
+			sourcePackage: "application",
+			importPath:    moduleImportPrefix + "almanac/register",
+			wantError:     true,
+		},
+		{
+			name:          "unknown test package is not a compatibility root",
+			filename:      "internal/application/bootstrap_test.go",
+			sourcePackage: "application",
+			importPath:    moduleImportPrefix + "almanac/extract/events",
+			wantError:     true,
+		},
 		{
 			name:          "framework production cannot import concrete module",
 			filename:      "internal/framework/pipeline/runner.go",
@@ -324,19 +359,8 @@ func validateImport(filename string, sourcePackage string, importPath string) er
 		}
 		return importBoundaryViolation(filename, importPath, "module integration composition is allowed only in black-box tests")
 	}
-	if !isTest && (strings.HasPrefix(filename, "internal/framework/") || strings.HasPrefix(filename, "internal/core/")) {
-		return importBoundaryViolation(filename, importPath, "core and framework production code must not import module implementations")
-	}
-	if !isTest && strings.HasPrefix(filename, "internal/cli/") {
-		if target.registrar {
-			return nil
-		}
-		return importBoundaryViolation(filename, importPath, "CLI production code may import only exact module family registrar packages")
-	}
 	sourceFamily, sourceRoot, sourceRegistrar := moduleFamilyForFile(filename)
-	if sourceFamily == "" {
-		return nil
-	}
+	if sourceFamily != "" {
 		if sourceRoot && sourceFamily == target.family && target.child {
 			return importBoundaryViolation(filename, importPath, "family root must not import child packages")
 		}
@@ -354,6 +378,23 @@ func validateImport(filename string, sourcePackage string, importPath string) er
 		}
 		return importBoundaryViolation(filename, importPath, fmt.Sprintf("concrete family %q must not import concrete family %q", sourceFamily, target.family))
 	}
+	if isTest {
+		if isCompatibilityTestFile(filename) {
+			return nil
+		}
+		return importBoundaryViolation(filename, importPath, "direct module imports from non-module tests are allowed only in CLI, core, and framework compatibility-test roots")
+	}
+	if strings.HasPrefix(filename, "internal/framework/") || strings.HasPrefix(filename, "internal/core/") {
+		return importBoundaryViolation(filename, importPath, "core and framework production code must not import module implementations")
+	}
+	if strings.HasPrefix(filename, "internal/cli/") {
+		if target.registrar {
+			return nil
+		}
+		return importBoundaryViolation(filename, importPath, "CLI production code may import only exact module family registrar packages")
+	}
+	return importBoundaryViolation(filename, importPath, "production code outside module families may import modules only from the CLI composition root through exact registrar packages")
+}
 
 type moduleImportTarget struct {
 	family      string
@@ -404,6 +445,15 @@ func isBlackBoxIntegrationTest(filename string, sourcePackage string) bool {
 	return isIntegrationFile(filename) && strings.HasSuffix(filename, "_test.go") && sourcePackage == "integration_test"
 }
 
+func isCompatibilityTestFile(filename string) bool {
+	if !strings.HasSuffix(filename, "_test.go") {
+		return false
+	}
+	return strings.HasPrefix(filename, "internal/cli/") ||
+		strings.HasPrefix(filename, "internal/core/") ||
+		strings.HasPrefix(filename, "internal/framework/")
+}
+
 func testRepositoryRoot(t *testing.T) string {
 	t.Helper()
 	_, filename, _, ok := runtime.Caller(0)

internal/modules/seriatim/input/transcript/config_test.go

@@ -293,6 +293,9 @@ func (seriatimArtifactCodec) Schema() contracts.ArtifactSchema {
 	return contracts.ArtifactSchema{ID: "fake.event", Name: "fake_event", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
 }
 func (seriatimArtifactCodec) MediaType() string { return "application/json" }
+func (seriatimArtifactCodec) EncodeCandidate(value seriatimArtifact) ([]byte, error) {
+	return json.Marshal(value)
+}
 func (seriatimArtifactCodec) Encode(value seriatimArtifact) ([]byte, error) {
 	return json.Marshal(value)
 }