Add an implementation plan to address gaps and fixes for the ADR-0006 refactor

.gitignore

@@ -1,3 +1,7 @@
+# build and testing artifacts
+notarius
+notarius-output
+
 # ---> Go
 # If you prefer the allow list template instead of the deny list, see community template:
 # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
@@ -47,7 +51,8 @@ go.work.sum
 .LSOverride
 
 # Icon must end with two \r
-Icon
+Icon
+
 
 # Thumbnails
 ._*

AGENTS.md

@@ -1,3 +1,2 @@
-Please carefully review the documents in `docs/policy` before making any changes to this repository.
+Please review `docs/development.md` for initial orientation in this repository
- - `architecture.md` provides the canonical high-level architecture policy for this repository.
+and follow its task-specific reading guide.
- - `documentation.md` provides the canonical documentation policy for this repository.

README.md

@@ -4,33 +4,30 @@ Notarius is a Go CLI for extracting structured artifacts from source material
 with explicit, configurable pipeline modules.
 
 The current implementation reads Seriatim transcript JSON, chunks the source
-units, extracts D&D spell-cast artifacts with an OpenAI-compatible LLM, and
+units, extracts D&D spell-cast artifacts with a Scriptorium-backed LLM runtime,
-writes JSON output plus diagnostics for each run.
+and writes JSON output. Add `--debug` when a per-run inspection bundle is
+needed.
 
 ```sh
-NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \
+OPENROUTER_API_KEY=... \
-NOTARIUS_LLM_DEFAULT_MODEL=your-model \
 go run ./cmd/notarius run dnd-session \
   --config examples/dnd-spells.config.yml \
   --input examples/seriatim-minimal-transcript.json
 ```
 
-If the provider requires authentication, set
+This invocation uses the maintained example configuration and input. See the
-`NOTARIUS_LLM_DEFAULT_API_KEY` in the environment before running the command.
+configuration and operations references for profile selection, credentials, and
-Outputs are written under `./notarius-output/<run-id>/` unless `--output-dir`
+run artifacts.
-is provided.
 
 Useful references:
 
 - [CLI reference](docs/cli.md)
 - [Configuration reference](docs/config.md)
 - [Operations](docs/operations.md)
-- [Troubleshooting](docs/troubleshooting.md)
 - [Seriatim input contract](docs/integrations/seriatim.md)
-- [OpenAI-compatible provider contract](docs/integrations/openai-compatible.md)
 - [JSON output contract](docs/integrations/json-output.md)
 - [D&D spell artifact contract](docs/integrations/dnd-spell-artifacts.md)
-- [Developer workflow](docs/policy/development.md)
+- [Developer guide](docs/development.md)
-- [Internal architecture docs](docs/internal/overview.md)
+- [Internal implementation docs](docs/internal/overview.md)
 - [Maintained example config](examples/dnd-spells.config.yml)
 - [Maintained example input](examples/seriatim-minimal-transcript.json)

docs/adr/0001-record-architecture-decisions.md

@@ -0,0 +1,23 @@
+# ADR-0001: Record architecture decisions as ADRs
+
+**Status:** Accepted
+**Date:** 2026-07-13
+
+## Context
+Architectural reasoning made during design (pattern choices, rejected
+alternatives, trigger conditions for revisiting) is lost if only the final
+state is documented.
+
+## Decision
+We keep a living overview in docs/policy/architecture.md describing current
+intended state, and immutable, numbered ADRs (Nygard format) in docs/adr/
+recording each significant decision, its alternatives, and its consequences.
+Changed decisions get a new ADR that marks the old one Superseded.
+
+## Alternatives considered
+- Overview doc only: loses the "why" and the rejected options.
+- arc42 / RFC-style design docs: heavier than warranted for a solo repo.
+
+## Consequences
+Small ongoing writing cost; durable reasoning trail; cheap onboarding for
+future contributors (including future-us).

docs/adr/0002-linear-pipes-and-filters-pipeline.md

@@ -0,0 +1,51 @@
+# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG
+
+**Status:** Accepted
+**Date:** 2026-07-13
+
+## Context
+
+Notarius processes source material through one known workflow:
+
+```text
+input -> chunk -> extract -> merge -> normalize -> output
+```
+
+Input and chunking apply to the source as a whole. Each selected artifact lane
+then performs extract, merge, and normalize, after which output aggregates the
+lane outcomes. Chunk extraction has a natural scatter-gather shape, but no
+current use case requires arbitrary branches, joins, or user-defined stage
+topology.
+
+## Decision
+
+Notarius implements a fixed six-stage pipes-and-filters pipeline. Configuration
+selects implementations for these stages but cannot add stages, reorder them,
+or define an arbitrary graph.
+
+The framework owns stage sequencing and the scatter-gather boundary between
+chunk, extract, and merge. Extract results are handed to merge in deterministic
+source-chunk order regardless of execution strategy. Each artifact lane remains
+logically linear. Output runs after every selected lane has either produced an
+accepted normalized artifact or reached a recorded rejection. A framework
+execution failure aborts the pipeline.
+
+The runner's concrete internal representation and stage-specific scheduling
+policies are implementation details. Concurrency must preserve the pipeline's
+deterministic handoffs, validation behavior, and provenance, and all execution
+strategies must continue to honor context cancellation.
+
+## Alternatives considered
+
+- Build a general DAG engine now. This would support hypothetical branching
+  topologies, but would add scheduling, topology validation, configuration, and
+  state-management complexity without a current consumer. Revisit this choice
+  only when a concrete workflow requires a topology the fixed pipeline cannot
+  express.
+
+## Consequences
+
+The runner, configuration model, and operator mental model remain small. Stage
+ownership stays visible, and general chunking, merging, or normalization cannot
+be hidden inside extractors. A future DAG requirement will require an explicit
+architectural change rather than incremental exceptions to the fixed pipeline.

docs/adr/0003-typed-interfaces-with-two-zone-data-model.md

@@ -0,0 +1,119 @@
+# ADR-0003: Strongly typed stage interfaces with a two-zone data model
+
+**Status:** Accepted
+**Date:** 2026-07-13
+
+## Context
+
+Pipeline stages must exchange source data and extracted artifacts. Universal
+source data has one engine-wide meaning, while extracted artifacts have
+domain-specific shapes. Passing opaque bytes or `any` between all stages would
+make invalid wiring and merge behavior runtime concerns. Requiring JSON at
+every handoff would preserve interoperability but discard useful Go type safety
+while all modules are in-process.
+
+The framework must also support multiple configured artifact domains, durable
+checkpoints, diagnostics, and output encoders without making those consumers
+depend on every domain's Go types.
+
+## Decision
+
+Notarius uses two typed data zones followed by one serialized boundary.
+
+### Source zone
+
+Input and chunk stages use conservative, engine-owned document, segment, chunk,
+and source-reference types. Their exact Go names are implementation details.
+Every segment carries engine-owned source provenance identifying the source
+location from which it was produced. Chunks preserve the ordered provenance of
+their segments.
+
+Source-format-specific fields remain in input modules or explicitly namespaced
+metadata; they do not become framework contracts.
+
+### Domain artifact zone
+
+Each artifact lane has one domain-owned Go artifact type `T`. Its extract,
+merge, normalize, and domain-aware validation implementations use generic,
+strongly typed contracts over the same `T`. Raw JSON, opaque bytes, and `any`
+are not stage-handoff contracts within a lane.
+
+Each registered domain artifact type supplies a codec for `T`. The codec owns:
+
+- stable schema identity and an explicit schema version;
+- JSON serialization and deserialization;
+- the media type and schema metadata required at serialized boundaries; and
+- rejection of data that cannot be represented by the declared artifact
+  schema.
+
+An artifact type's JSON representation is a maintained domain contract.
+Changing it incompatibly requires a new schema version.
+
+Extract, merge, and normalize may change the contents of `T`, but they do not
+change the lane's canonical Go artifact type or artifact schema identity. An
+extractor maps any provider- or prompt-specific response type into `T` before
+returning. A future lane that requires different artifact types at different
+stages requires a new architectural decision.
+
+### Serialized boundary
+
+After normalization, each typed artifact is converted into an engine-owned
+serialized artifact containing bytes, media type, and schema metadata. Output
+aggregation and output encoders consume this type-erased form. Intermediate
+checkpoint and debug encodings do not become stage-handoff contracts.
+
+LLM transport, checkpoints, and opt-in debug recording are also explicit
+serialization boundaries. They may encode or decode a typed artifact through
+its domain codec, but they do not change the in-memory type used between
+extract, merge, normalize, and typed validators. Checkpoint reuse requires a
+compatible schema identity and version.
+
+An LLM structured-response schema is a module transport contract and may differ
+from the domain artifact schema. The calling module owns the response type and
+maps it into the canonical `T`; the artifact codec remains authoritative for
+artifact checkpoints and output serialization.
+
+The framework may use private type-erased adapters to store heterogeneous lane
+registrations and execute configured domains. Such an adapter must assemble a
+type-consistent lane before execution and must not expose `any` or raw payloads
+as module-facing handoffs inside the domain artifact zone.
+
+### Construction and dependencies
+
+Every module operation accepts `context.Context`. Modules receive stable runtime
+collaborators through an injected dependency set at construction time. In
+particular, LLM-using modules receive the application-provided structured LLM
+client and do not construct provider clients or bypass shared scheduling.
+
+The application boundary enforces one configurable global upper bound on
+in-flight LLM calls across all stages, lanes, retries, and validators.
+
+Configuration options are parsed and validated while a module is constructed,
+before that module executes. Per-run data such as source material, references,
+session identity, and lane identity remains operation input rather than a
+construction dependency.
+
+## Alternatives considered
+
+- Pass raw bytes between stages. This maximizes decoupling but moves wiring,
+  parsing, and merge errors to runtime and prevents domain types from being the
+  canonical in-process contract.
+- Require JSON plus schemas at every stage boundary. This is appropriate for an
+  out-of-process boundary, but adds serialization and parsing inside the current
+  in-process pipeline. The stable codec contract preserves this upgrade path if
+  remote plugins are introduced.
+- Use a uniform `Process(any) (any, error)` contract. This simplifies a fully
+  dynamic engine but turns incompatible module composition into type assertions
+  and runtime failures. The fixed topology does not require that tradeoff.
+
+## Consequences
+
+Domain pipelines gain compile-time handoff safety and explicit merge semantics.
+Serialization, schema compatibility, checkpoint decoding, and output erasure
+have named owners. Dynamic registration requires a small erased adapter around
+each typed lane, and generic stage implementations must be instantiated for a
+specific artifact type or behavior rather than manipulating arbitrary JSON.
+
+The engine-owned source model becomes a long-lived contract and must evolve
+conservatively. Domain authors must maintain a codec and versioned schema in
+addition to their Go artifact type.

docs/adr/0004-package-modules-by-domain.md

@@ -0,0 +1,81 @@
+# ADR-0004: Package modules by domain, not by stage
+
+**Status:** Accepted
+**Date:** 2026-07-13
+
+## Context
+
+Module packages can be grouped first by pipeline stage, such as
+`modules/chunk/dnd/scenes`, or first by domain, such as
+`modules/dnd/chunk/scenes`. A domain's extract, merge, normalize, validation,
+schema, prompt, and artifact-codec implementations collaborate around the same
+artifact types and are likely to evolve together.
+
+Go package dependencies also constrain registration. If shared types live in a
+domain root package, that package cannot import child implementation packages
+to register them because the children already import the root types.
+
+## Decision
+
+Production extensions are grouped by domain under:
+
+```text
+internal/modules/<domain>/<stage>/<name>
+```
+
+Shared artifact types live at the domain root, for example
+`internal/modules/dnd/types.go`. Domain-specific validators, prompt fragments,
+schemas, reference helpers, and codecs also live within that domain tree.
+
+Each domain exposes one production registration entry point from a sibling
+registrar package, for example `internal/modules/dnd/register`. The registrar
+may import the domain root and its child implementations; the domain root does
+not import its registrar or child packages. This keeps shared types available
+as `dnd.SpellList` without creating a Go import cycle.
+
+The `generic` tree is a peer extension family for reusable implementations that
+contain no concrete source-format or artifact-domain knowledge. Source-format
+and output-format families, such as Seriatim and JSON output, follow the same
+domain-first organization even when they do not define a type in the
+[domain artifact zone](0003-typed-interfaces-with-two-zone-data-model.md#domain-artifact-zone).
+
+Concrete domain implementation packages do not import another concrete domain.
+Generic extension packages never import concrete domains. A domain registrar
+may import domain-neutral generic extension packages to instantiate a reusable
+strategy for that domain's artifact type; the generic implementation remains
+unaware of the concrete type's domain semantics. Reuse needed directly by a
+domain implementation lives in a domain-neutral framework or helper package,
+not in a peer extension package.
+
+The application composition root may import multiple registrar packages, and
+black-box integration tests may compose multiple domains. Other cross-domain
+reuse occurs through engine contracts and composition-time registration rather
+than concrete peer-domain imports.
+
+A domain registrar owns registration of that domain's modules, validators,
+default validator chains, artifact codecs, schemas, and prompt assets. It does
+not take ownership of application execution or process behavior.
+
+## Alternatives considered
+
+- Group modules by stage. This keeps interchangeable strategies side by side,
+  but scatters a domain's shared artifact model and collaborating extensions
+  across the repository. It is preferable when generic strategy libraries
+  dominate or when the project is primarily a stage-extension framework rather
+  than an application composed from domain suites.
+- Put both shared types and `Register` in the domain root. This gives the
+  shortest import path but creates an import cycle once child implementations
+  import the root artifact types.
+
+## Consequences
+
+The repository layout makes supported domains immediately visible, and adding
+or extracting a domain affects one cohesive subtree. The CLI composition root
+depends on a small set of domain registrars instead of every leaf package.
+
+Package moves must preserve user-visible module and validator keys unless a
+separate compatibility decision changes them. Shared behavior that cannot be
+expressed through framework contracts may need to move into a domain-neutral
+framework package rather than creating a concrete peer-domain import. Registrar
+packages become explicit composition points for instantiating generic typed
+strategies, in addition to registering domain-owned implementations.

docs/adr/0005-cache-canonical-chunk-plans-by-source.md

@@ -0,0 +1,141 @@
+# ADR-0005: Cache one canonical chunk plan per source
+
+**Status:** Accepted
+**Date:** 2026-07-17
+
+## Context
+
+Notarius may run several extraction passes over the same source. A D&D
+transcript, for example, may first produce NPC artifacts and later produce
+spell or combat artifacts, with output from an earlier pass supplied as a
+reference to a later pass.
+
+An LLM-backed chunker may process an entire, potentially large source in one
+expensive request. Recomputing boundaries for every pipeline or pass repeats
+that cost and can make otherwise comparable extraction runs use different
+source partitions. Stable chunk material also gives later extraction requests
+a better opportunity to benefit from provider-side prompt caching.
+
+Chunk boundaries can affect extraction quality. Evidence may span a boundary,
+overlap may produce duplicates, and different partitions may change the context
+available to a model. Merge and normalization should remove structural signs
+of chunking from durable output, but they cannot guarantee recovery of evidence
+that an extractor did not receive.
+
+Notarius therefore needs an explicit policy for choosing between automatically
+applying the latest chunking configuration and preserving one stable partition
+for repeated work on the same source.
+
+## Decision
+
+Notarius assigns one active canonical chunk plan to a source and reuses that
+plan by default across pipelines and invocations.
+
+The canonical source identity is derived from the validated generic source
+document and covers the source-unit identity, order, and content needed to
+interpret plan boundaries. Input-adapter and chunk-producer identities are
+recorded as provenance, but the active-plan lookup does not vary with:
+
+- pipeline identity or selected artifact lanes;
+- the configured chunk module or its options;
+- references;
+- LLM provider, model, profile, prompt, or response schema; or
+- configuration for later pipeline stages.
+
+When an active plan exists, Notarius uses it even if the current pipeline
+configures a different chunk module or different chunk-module settings. The
+configured chunk module generates a plan only when none exists or when the
+operator explicitly requests recomputation.
+
+The framework-owned minimum plan contract is an ordered, non-empty set of
+source-unit ranges. Each range identifies the inclusive start and end unit for
+one chunk. A chunk module may also provide namespaced, domain-specific
+annotations at plan or range scope. Those annotations are stored with the plan
+and passed through the pipeline when present, but they remain optional.
+Downstream stages must not assume that annotations associated with the
+currently configured chunk module are present on a reused plan produced by a
+different module.
+
+The cache stores the plan rather than fully materialized chunks. The framework
+validates a reused plan against the current source and deterministically
+materializes its ranges into chunks. The same source and plan must produce
+byte-stable chunk input for later stages.
+
+Canonical plan storage is a distinct cache surface with an independently
+configurable location. It is not coupled to the roots or lifecycles of
+invocation checkpoints, diagnostics, debug artifacts, or durable output. This
+allows per-user and system-service deployments to apply cache-specific
+ownership, permissions, placement, and cleanup policy without relocating other
+Notarius state.
+
+One mutable active plan is stored under the canonical source identity and
+retains provenance for the module and relevant runtime inputs that produced it.
+Refreshing the active plan atomically replaces that one mutable record; readers
+must observe either the previous complete plan or the replacement complete
+plan, never a partial update.
+The effective plan producer is reported separately from the chunk module
+requested by the current pipeline; reuse must not attribute cached boundaries
+or annotations to a module that did not produce them.
+
+Reuse is enabled by default. Operators can explicitly:
+
+- bypass cached plans for an invocation without changing the active plan; or
+- recompute a plan with the configured chunk module and make it active for
+  later work.
+
+Exact storage layout, configuration fields, CLI syntax, publication mechanics,
+recovery behavior, and diagnostics are implementation and operational
+contracts rather than part of this decision.
+
+## Alternatives considered
+
+- Recompute chunks on every invocation. This always applies the current
+  chunking configuration, but repeats the most expensive stage and weakens
+  provider-side caching and cross-pass comparability.
+- Cache every distinct chunking request by including module options,
+  references, prompts, profiles, and other runtime inputs in its identity. This
+  closely associates a cached result with its producing request, but reduces
+  reuse and permits boundary drift across operationally different passes.
+- Key plans by source plus chunk module and options. This shares plans across
+  pipelines using the same strategy, but changing the configured strategy
+  silently selects a different partition rather than preserving one canonical
+  partition for the source.
+- Require operators to name or supply a plan for every run. Explicit selection
+  is reproducible and may be useful as an advanced operation, but adds friction
+  to the default workflow and does not provide automatic reuse.
+- Store fully materialized chunks. This simplifies loading, but duplicates
+  source content and couples durable state to the current chunk representation
+  rather than the stable boundary decision.
+- Store canonical plans beneath the general workspace root. This would reuse an
+  existing location setting, but it couples a reusable application cache to
+  checkpoint, diagnostic, and debug state that have different ownership,
+  sensitivity, retention, and deployment requirements.
+
+## Consequences
+
+Independent pipelines and passes over the same source use stable boundaries by
+default. This reduces repeated LLM work, improves cross-pass comparability, and
+increases the opportunity for cached provider reads.
+
+The configured chunk module may not execute, and its settings may have no
+effect, when an active plan already exists. Domain-specific annotations reflect
+the plan's original producer and may be absent or differ from those the current
+module would produce. User-visible provenance must make the effective plan
+clear.
+
+A poor or outdated partition remains active until an operator replaces it.
+This can preserve suboptimal context boundaries and affect extraction recall or
+duplication even when merge and normalization hide the partition structure in
+durable output. Stable reuse is an intentional priority over automatically
+incorporating later chunk-strategy changes.
+
+The framework gains a durable minimal chunk-plan contract and deterministic
+materialization responsibility. Chunk modules must separate required boundary
+output from optional annotations, and downstream modules may rely only on the
+minimal boundary contract unless a future decision introduces an explicit plan
+compatibility mechanism.
+
+Operators must configure and secure canonical plan storage independently from
+other workspace state when the per-user default is not appropriate. Removing
+that cache remains recoverable because Notarius can regenerate it from the
+source, but doing so may repeat an expensive LLM operation.

docs/adr/0006-separate-output-cache-and-debug-state.md

@@ -0,0 +1,114 @@
+# ADR-0006: Separate output, cache, and debug state
+
+**Status:** Accepted
+**Date:** 2026-07-17
+
+## Context
+
+Notarius currently exposes a workspace as a shared parent for checkpoints,
+debug artifacts, and preferred diagnostics settings. Diagnostics are a second
+inspection surface with their own enablement, directory, retention, and legacy
+configuration. Durable output uses a separate CLI-selected root, while the
+canonical chunk-plan cache introduced by ADR-0005 correctly uses an independent
+cache root.
+
+These concepts reflect implementation history more than operator intent. A user
+must understand differences among workspace state, diagnostics, debug artifacts,
+checkpoints, and chunk plans before deciding where Notarius may write. Some of
+those distinctions are important internally: a redacted run summary has a
+different sensitivity from a trace containing source material, prompts, and
+model responses. They do not require separate public filesystem categories.
+
+Notarius needs a smaller state model that communicates why data exists, how it
+may be treated, and whether it is reconstructible.
+
+## Decision
+
+Notarius exposes three filesystem surfaces: output, cache, and debug. The
+public workspace concept and diagnostics as a separate output surface are
+removed.
+
+### Output
+
+Output is the durable result of a run and the only surface intended for normal
+consumption. It contains the logical files produced by the output stage,
+including the maintained result, manifest, warning, and rejection contracts.
+Output is not cache or inspection state.
+
+### Cache
+
+Cache contains reconstructible state used to avoid repeated work or resume an
+interrupted workflow. Canonical chunk plans and invocation checkpoints are
+distinct cache families with independent identities, compatibility rules,
+enablement policies, locations, and cleanup lifecycles.
+
+ADR-0005 continues to govern canonical chunk-plan selection and reuse. Grouping
+chunk plans and checkpoints under the public cache category does not permit a
+checkpoint to compete with canonical plan reuse or couple their storage roots.
+
+Checkpointing is an invocation policy rather than a prerequisite hidden in
+persistent workspace configuration. An explicit resume invocation may read
+compatible checkpoints and record replacement checkpoint state for work it
+executes. Runs that do not request resume perform no checkpoint I/O.
+
+### Debug
+
+Debug is an explicitly requested per-run inspection bundle intended for
+developers and troubleshooting. It is off by default. When enabled, one bundle
+contains both redacted run summaries and detailed stage and LLM traces. The
+internal distinction between a safe summary and a sensitive trace remains, but
+there is one public enablement and location model.
+
+Debug data is never a cache input and has no automatic retention policy.
+Notarius does not create a debug directory unless debug is requested, and it
+does not automatically delete a requested bundle. Credentials remain redacted
+at every level, while the bundle as a whole is treated as potentially sensitive 
+because traces may contain source, reference, prompt, model-response, and 
+intermediate artifact content.
+
+Concise progress, warnings, and failures continue to use stdout or stderr. A
+run without debug may fail without producing a filesystem inspection record.
+
+Exact configuration fields, CLI flags, default paths, layouts, compatibility
+handling, and migration mechanics are configuration and operational contracts
+rather than part of this decision.
+
+## Alternatives considered
+
+- Keep workspace, diagnostics, checkpoints, debug, and chunk-plan cache as
+  separate public concepts. This preserves compatibility and the current safe
+  default-on failure records, but retains overlapping configuration and asks
+  operators to reason about implementation-specific categories.
+- Keep diagnostics as an always-available redacted operational surface and use
+  debug only for sensitive traces. This distinction is useful for a daemon or
+  managed service with an operational logging contract, but the current CLI can
+  report concise failures on stderr and provide inspection data when explicitly
+  requested.
+- Put all non-output state beneath one physical root. This minimizes path
+  configuration, but couples reconstructible caches to per-run inspection data
+  and couples cache families whose identity, sensitivity, and cleanup policies
+  differ.
+- Treat checkpoints as durable run state rather than cache. This emphasizes
+  resumability, but checkpoints are derived, compatibility-checked data that may
+  be deleted and recomputed. Cache more accurately describes their lifecycle.
+
+## Consequences
+
+The operator model becomes smaller: normal runs produce output and may use
+cache; developers explicitly request debug. Public configuration no longer
+exposes a workspace or overlapping diagnostics and debug systems.
+
+The implementation retains separate collaborators and serializers where their
+security or lifecycle boundaries differ. Redacted summaries remain useful as
+the index to a debug bundle, and chunk plans and checkpoints retain separate
+stores even though both are cache.
+
+Existing configuration, environment variables, flags, examples, and
+documentation require a deliberate compatibility transition. Default-on
+diagnostic directories disappear, so failures without debug are inspectable
+only through stderr and any durable output completed before the failure.
+
+Debug becomes easier to request and substantially more complete, but enabling
+it creates sensitive files that the operator must protect and remove. Cache
+cleanup is recoverable but may repeat expensive work, while deleting output is
+data loss from the user's perspective.

docs/cli.md

@@ -3,26 +3,15 @@
 This is the canonical reference for the implemented Notarius command-line
 interface.
 
-## Quick Run
+For the minimal end-to-end invocation, see the [README](../README.md).
-
-```sh
-NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \
-NOTARIUS_LLM_DEFAULT_MODEL=your-model \
-go run ./cmd/notarius run dnd-session \
-  --config examples/dnd-spells.config.yml \
-  --input examples/seriatim-minimal-transcript.json
-```
-
-Set `NOTARIUS_LLM_DEFAULT_API_KEY` if the OpenAI-compatible provider requires
-a bearer token.
 
 ## Commands
 
 ```text
 notarius help
-notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b]
+notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector]
-notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
+notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
-notarius pipelines list --config path/to/config.yml [--json]
+notarius pipelines list [--config path/to/config.yml] [--json]
 ```
 
 Running `notarius` with no arguments, `notarius help`, `notarius --help`, or
@@ -36,26 +25,147 @@ file.
 Flags:
 
 - `--input path`: required source input file.
-- `--config path`: config file path. If omitted, Notarius checks
+- `--config path`: config file path. If omitted, Notarius uses the discovery
-  `NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`.
+  rules in [Configuration](config.md#discovery).
 - `--only lane-a,lane-b`: run only the named artifact lanes. Values are
   comma-separated and must be non-empty.
-- `--output-dir path`: output root. The run writes to `<path>/<run-id>/`.
+- `--resume`: request checkpoint reuse for this invocation. See
-  Defaults to `./notarius-output`.
+  [Operations](operations.md#checkpoints) for prerequisites and reuse behavior.
-- `--diagnostics-dir path`: diagnostics work directory override for this
+- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
-  invocation.
+  invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
-- `--llm-profile id`: override every effective module binding to use one LLM
+  performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
-  profile.
+  plan only after chunk validation succeeds. See
+  [Configuration](config.md#state-surfaces) for the persistent setting, precedence,
+  and cache-root selection.
+- `--output-dir path`: output root. Defaults to `./notarius-output`.
+- `--debug`: allocate and retain one debug bundle for this invocation.
+- `--debug-dir path`: debug-bundle root override. This flag requires `--debug`.
+- `--llm-profile id`: override every effective LLM-capable pipeline module
+  binding with one Scriptorium profile ID. Validator-specific profiles are not
+  overridden.
+- `--session-id id`: pass a stable prompt session identifier through LLM-backed
+  module calls.
+- `--reference selector=path`: bind a reference path to a chunk, extractor,
+  merger, or normalizer reference slot. Repeatable.
+- `--without-reference selector`: remove a configured optional reference binding.
+  Repeatable. It accepts the same selector forms as `--reference`, without
+  `=path`.
 
-On success, the command prints the completed pipeline ID, approved and rejected
+On success, the command prints the completed pipeline ID, normalized output and
-artifact counts, and the output directory. If the run completes with warnings,
+rejected output counts, and the output directory. A debug-enabled run also
-the warning count is printed to stderr.
+prints `debug=<bundle-path>`. If the run completes with warnings, the warning
+count is printed to stderr.
 
-For durable output, diagnostics, retention, and failure inspection, see
+Reference flags are resolved against selected chunk, extractor, merger, and
-[Operations](operations.md).
+normalizer targets before the run starts. Flat slot names are accepted only
+when exactly one selected target declares that slot. For configured reference
+bindings, precedence, path resolution, and validation, see
+[Configuration](config.md#pipelines).
 
-The current `run` command requires the resolved pipeline to use exactly one
+`--reference` binds or replaces one slot for one selected target. Selectors are:
-distinct LLM profile after defaults and overrides are applied.
+
+- `slot=path`: valid when exactly one selected target declares `slot`;
+- `chunk.slot=path`: target the chunker;
+- `merge.slot=path`: valid when exactly one selected merger declares `slot`;
+- `lane.slot=path`: valid when exactly one selected extractor, merger, or
+  normalizer in that lane declares `slot`;
+- `lane.extract.slot=path`: target a lane extractor;
+- `lane.merge.slot=path`: target a lane merger;
+- `lane.normalize.slot=path`: target a lane normalizer.
+
+Use `slot=path` when the selected targets declare the slot unambiguously:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --reference roster=./campaign-roster.txt
+```
+
+Use an explicit selector when multiple selected targets declare the same slot or
+when you want to target a specific target:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --reference spells.extract.glossary=./campaign-glossary.txt
+```
+
+The same grammar can target chunk, merge, and normalize slots when the configured
+modules declare them:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config path/to/config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --reference chunk.scene_guide=./campaign-scenes.txt \
+  --reference spells.merge.merge_notes=./merge-notes.txt \
+  --reference spells.normalize.normalization_notes=./normalization-notes.txt
+```
+
+Use `--without-reference` to remove a configured optional binding for a run:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --without-reference glossary
+```
+
+Use `--session-id` when an external orchestrator needs all prompt calls from one
+run to share an identifier:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --session-id campaign-17-session-04
+```
+
+The resume flag can be added to an otherwise identical run invocation. It both
+loads compatible checkpoints and records replacements for work executed by that
+invocation; without it, the checkpoint root is not used:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --resume
+```
+
+Use `--debug` to retain the redacted summary and trace bundle for one run. The
+bundle is allocated before pipeline resolution; once allocated, its path is
+also printed to stderr if the command fails. Debug-write failures cause exit
+code `1`.
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --debug --debug-dir ./notarius-debug
+```
+
+Use `refresh` when intentionally replacing the cached plan for the same source:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --chunk_cache refresh
+```
+
+Use `bypass` for a one-off run that must not inspect or create plan-cache state:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --chunk_cache bypass
+```
+
+`--diagnostics-dir` has been removed. For checkpoint behavior, durable output,
+debug-bundle lifecycle, and failure inspection, see [Operations](operations.md).
 
 ## `config validate`
 
@@ -63,8 +173,8 @@ distinct LLM profile after defaults and overrides are applied.
 
 Flags:
 
-- `--config path`: config file path. If omitted, discovery uses
+- `--config path`: config file path. If omitted, Notarius uses the discovery
-  `NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`.
+  rules in [Configuration](config.md#discovery).
 - `--pipeline pipeline-id`: additionally resolve one configured pipeline against
   the production module catalog.
 - `--only lane-a,lane-b`: validate resolution for selected artifact lanes. This
@@ -88,8 +198,8 @@ go run ./cmd/notarius config validate \
 
 Flags:
 
-- `--config path`: config file path. If omitted, discovery uses
+- `--config path`: config file path. If omitted, Notarius uses the discovery
-  `NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`.
+  rules in [Configuration](config.md#discovery).
 - `--json`: print `{"pipelines":[...]}` instead of one ID per line.
 
 Examples:
@@ -107,23 +217,11 @@ go run ./cmd/notarius pipelines list \
 
 - `0`: command succeeded.
 - `1`: command syntax was valid, but loading config, resolving modules, running
-  the pipeline, calling the provider, writing output, or writing diagnostics
+  the pipeline, calling the provider, writing output, or writing a requested
-  failed.
+  debug bundle failed.
 - `2`: command syntax was invalid, a command was unknown, a required argument
   was missing, or a flag value was malformed.
 
-## Implemented Production Pipeline Modules
+For YAML structure, defaults, Scriptorium profile sources, environment
-
+overrides, and selectable module and validator keys, see
-The production CLI currently registers these module keys:
+[Configuration](config.md).
-
-- input: `seriatim`
-- chunk: `generic`
-- extract: `dnd/spells`
-- merge: `appendorder`
-- normalize: `noop`
-- output: `json`
-
-The production CLI does not currently register validator modules.
-
-For YAML structure, defaults, environment overrides, and module binding syntax,
-see [Configuration](config.md).

docs/config.md

@@ -2,117 +2,134 @@
 
 This is the canonical reference for implemented Notarius configuration.
 
-Notarius reads YAML config files with `version: 1`. File config is applied over
+Notarius reads YAML config files with `version: 3`. File configuration is
-built-in defaults, then environment overrides are applied.
+applied over built-in defaults, then environment overrides are applied. Explicit
+CLI overrides are applied last where the command supports them.
 
 ## Discovery
 
-Commands that accept `--config` load configuration in this order:
+Commands that load configuration use this order:
 
-1. the `--config` path, when provided;
+1. an explicit path supplied through the CLI, when provided;
 2. `NOTARIUS_CONFIG`, when set to a non-empty path;
 3. `/usr/local/etc/notarius/config.yml`.
 
 If none is available, the command fails with a config file not found error.
+The explicit-path option is defined in the [CLI reference](cli.md).
 
-## Minimal Example
+## Maintained Examples
 
-```yaml
+- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
-version: 1
+- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
-llm_profiles:
-  default:
-    provider: openai-compatible
-    base_url: http://127.0.0.1:8080/v1
-    model: your-model
-pipelines:
-  dnd-session:
-    input: seriatim
-    chunk:
-      module: generic
-      options:
-        max_units: 50
-    artifacts:
-      spells:
-        extract: dnd/spells
-```
 
-The maintained fixture is [examples/dnd-spells.config.yml](../examples/dnd-spells.config.yml).
+Both are complete version 3 files. The fragments below illustrate individual
+fields and are not alternate complete configurations.
 
 ## Top-Level Fields
 
-- `version`: required. The only supported value is `1`.
+- `version`: required. The only supported value is `3`.
-- `llm_profiles`: optional map of LLM profile IDs to profile settings.
+- `scriptorium`: optional Scriptorium profile source settings.
 - `pipelines`: optional map of pipeline IDs to pipeline definitions.
 - `concurrency`: optional global concurrency settings.
-- `diagnostics`: optional diagnostics settings.
+- `output`: optional durable output placement.
+- `cache`: optional chunk-plan and checkpoint cache placement.
+- `debug`: optional debug-bundle placement. It does not enable debug capture.
 
-Unknown YAML fields are rejected.
+Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
+rejected; execution profiles now come from Scriptorium.
 
 ## Defaults
 
 Built-in defaults:
 
-```yaml
+- `concurrency.total_llm`: `1`
-llm_profiles:
+- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
-  default:
+- `output.directory`: `./notarius-output`
-    provider: openai-compatible
+- `cache.chunk_plans.mode`: `auto`
-    timeout: 600
+- `cache.chunk_plans.directory`: unset, selecting
-    max_retries: 3
+  `<os.UserCacheDir>/notarius/chunk-plans`
-    max_concurrency: 1
+- `cache.checkpoints.directory`: unset, selecting
-concurrency:
+  `<os.UserCacheDir>/notarius/checkpoints`
-  total_llm: 1
+- `debug.directory`: `./notarius-debug`
-diagnostics:
-  work_dir: /tmp/notarius
-  retention: auto
-```
 
 No pipelines are built in. A run requires a configured pipeline.
 
-## LLM Profiles
+If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
+catalog. Prompt definitions may also name default profile IDs. The current D&D
+scene and spell prompts default to the built-in `mistral-small-3` profile when a
+module binding does not set `llm_profile`. That built-in profile reads its
+credential from `OPENROUTER_API_KEY`.
 
-Each `llm_profiles` entry may contain:
+## Scriptorium Profiles
 
-- `provider`: optional provider key. Empty means `openai-compatible`; any other
+`scriptorium` fields:
-  non-empty value must be `openai-compatible`.
-- `base_url`: provider base URL. Required for actual LLM calls.
-- `model`: provider model name. Required for actual LLM calls.
-- `api_key_env`: environment variable name to read for the API key.
-- `timeout`: request timeout as whole seconds or a Go-style duration string such
-  as `10m`.
-- `max_retries`: retry count for provider calls. Must be zero or greater.
-- `max_concurrency`: per-profile LLM concurrency. Must be zero or greater; when
-  zero, Notarius uses `concurrency.total_llm`.
 
-Raw API keys are not accepted as file config fields. Use `api_key_env` or an
+- `profile_dir`: optional directory containing Scriptorium profile YAML files.
-environment override.
+- `profile_file`: optional Scriptorium profile YAML file.
+
+`profile_dir` and `profile_file` are mutually exclusive. Custom profiles
+overlay Scriptorium built-in profiles by profile ID.
+
+Scriptorium profile files use Scriptorium's profile schema. A minimal profile
+looks like:
+
+```yaml
+id: local-fast
+endpoint: http://127.0.0.1:8080/v1
+model: your-model
+api_key_env: SCRIPTORIUM_API_KEY
+timeout_seconds: 180
+```
+
+Notarius does not accept raw API keys in Notarius config. For file-backed
+Scriptorium profiles, store the environment variable name in `api_key_env` and
+set that variable in the run environment. Scriptorium rejects raw `api_key`
+fields in profile YAML.
 
 ## Environment Overrides
 
 These environment variables are applied after the config file:
 
 - `NOTARIUS_CONFIG`: config discovery path.
-- `NOTARIUS_LLM_DEFAULT_API_KEY`: API key for the `default` LLM profile.
-- `NOTARIUS_LLM_DEFAULT_BASE_URL`: base URL for the `default` LLM profile.
-- `NOTARIUS_LLM_DEFAULT_MODEL`: model for the `default` LLM profile.
-- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS`: integer timeout seconds for the
-  `default` LLM profile.
-- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES`: integer retry count for the `default` LLM
-  profile.
-- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY`: integer max concurrency for the
-  `default` LLM profile.
 - `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
-- `NOTARIUS_WORK_DIR`: diagnostics work directory.
+- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
-- `NOTARIUS_DIAGNOSTICS_RETENTION`: diagnostics retention mode.
+- `NOTARIUS_OUTPUT_DIR`: durable output root.
+- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`: chunk-plan cache mode.
+- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`: chunk-plan cache root.
+- `NOTARIUS_CACHE_CHECKPOINTS_DIR`: checkpoint cache root.
+- `NOTARIUS_DEBUG_DIR`: debug-bundle root.
 
-Integer environment values must parse as base-10 integers.
+Integer environment values must parse as base-10 integers. Directory overrides
+must be non-empty after trimming. Cache-directory fields in a file may be
+empty, which deliberately selects the corresponding per-user default.
+
+The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
+endpoint, model, and credential environment variable names through Scriptorium
+profiles.
+
+## Concurrency
+
+`concurrency` fields:
+
+- `total_llm`: positive integer ceiling on concurrent provider calls.
+- `stage_workers`: optional map of framework worker limits. The only supported
+  key is `extract`.
+
+`stage_workers.extract` defaults to the effective `total_llm` value after file
+and environment precedence. It must be between `1` and `total_llm`, inclusive.
+Unknown or empty stage-worker keys are rejected. The environment override
+`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
+`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
+
+The worker value is present in effective and redacted configuration. It bounds
+the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
+submitted by source chunk and then resolved lane; `total_llm` independently
+bounds actual provider calls made by extracts, retries, and validators.
 
 ## Pipelines
 
-A pipeline defines the fixed Notarius workflow:
+A pipeline selects implementations for the fixed workflow defined by
-
+[Architecture](policy/architecture.md#system-shape).
-```text
-input -> chunk -> extract -> merge -> normalize -> output
-```
 
 Pipeline fields:
 
@@ -121,18 +138,88 @@ Pipeline fields:
 - `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
   lane definitions.
 - `output`: optional module binding. Default module is `json`.
+- `references`: optional map of reference slot names to reference paths. These
+  bindings are defaults for eligible pipeline targets that declare the matching
+  slot.
 
 Artifact lane fields:
 
 - `extract`: required module binding.
 - `merge`: optional module binding. Default module is `appendorder`.
 - `normalize`: optional module binding. Default module is `noop`.
-- `validators`: optional list of module bindings. The production CLI currently
+- `validators`: deprecated lane-level validator list. Non-empty lists are
-  does not register validator modules.
+  rejected; use `extract.validators`, `merge.validators`, or
+  `normalize.validators`.
+- `references`: optional compatibility alias for extractor reference bindings.
+  Lane bindings override pipeline-level bindings for the same slot.
 
-`notarius run` and `notarius config validate --pipeline` resolve the pipeline
+Commands that resolve a pipeline fail for unknown or incompatible module keys.
-against the production module catalog and fail fast for unknown or incompatible
+See [CLI Reference](cli.md) for command syntax.
-module keys.
+
+Reference bindings are validated against reference slots declared by eligible
+chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
+must be bound after config defaults, target-local references, lane-level
+compatibility bindings, and command-line reference overrides are applied.
+Config-relative paths are resolved relative to the config file; command-line
+reference paths are resolved relative to the current working directory. Bound
+files must be UTF-8 text. Reference media types are inferred from file
+extensions and checked when a module restricts accepted types; unknown
+extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) for
+command-line selectors and [Operations](operations.md) for recorded provenance
+and sensitive-data handling.
+
+Pipeline-level `references` are defaults. They are valid when at least one
+eligible target in the full configured pipeline declares the slot, including
+chunk, extractor, merger, and normalizer targets. During a run, they apply only
+to the selected targets that declare the slot:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      players: ./campaign/players.txt
+      party: ./campaign/party-roster.txt
+      glossary: ./campaign/glossary.txt
+    artifacts:
+      spells:
+        extract: dnd/spells
+```
+
+Extractor binding `references` are the canonical lane-local location. The
+legacy lane-level `references` field remains supported as an alias; when both
+bind the same slot, `extract.references` wins:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      glossary: ./campaign/glossary.txt
+    artifacts:
+      spells:
+        references:
+          roster: ./campaign/legacy-roster.txt
+        extract:
+          module: dnd/spells
+          references:
+            party: ./campaign/session-party.txt
+```
+
+`chunk.references`, `merge.references`, and `normalize.references` are accepted
+in object-form bindings. They override pipeline-level defaults for slots
+declared by that target module. Extractor-local references apply only to the
+extractor, merger-local references apply only to the merger, and
+normalizer-local references apply only to the normalizer.
+
+Target-local reference fields use the same map shape at:
+
+- `pipelines.<id>.chunk.references`
+- `pipelines.<id>.artifacts.<lane>.extract.references`
+- `pipelines.<id>.artifacts.<lane>.merge.references`
+- `pipelines.<id>.artifacts.<lane>.normalize.references`
+
+Each binding is valid only when that target module declares the slot.
 
 ## Module Bindings
 
@@ -146,20 +233,46 @@ or object form:
 
 ```yaml
 chunk:
-  module: generic
+  module: dnd/scenes
-  llm_profile: default
+  llm_profile: local-fast
-  options:
-    max_units: 50
 ```
 
 Binding fields:
 
 - `module`: module key.
-- `llm_profile`: optional LLM profile ID. Empty means `default`.
+- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
+  Scriptorium prompt default select the profile.
+- `retries`: non-negative retry count for extra runtime attempts after the
+  first attempt. Default: `0`. Supported on `chunk`, `extract`, `merge`, and
+  `normalize` bindings.
 - `options`: optional module-specific settings.
+- `references`: optional reference bindings. Supported only for `chunk`,
+  `extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
+  reject this field during validation.
+- `validators`: optional stage-local validator chain override. Supported only
+  for `chunk`, `extract`, `merge`, and `normalize` bindings. Omit the field to
+  use the production default chain; set `validators: []` to force an empty
+  chain; set a non-empty list to use exactly those validators in configured
+  order.
 
-The `--llm-profile` run flag overrides every effective module binding to use
+During resolution, each selected module's registered option validator runs.
-one configured profile.
+Production input, chunk, and output bindings reject unknown or invalid options
+with the affected binding context.
+
+Validator bindings use the same shorthand or object module-binding form, but
+only these fields are supported:
+
+- `module`: validator key.
+- `llm_profile`: optional Scriptorium profile ID for LLM-backed validators.
+- `options`: optional validator-specific settings.
+
+Validator bindings reject `references`, `retries`, and nested `validators`.
+During resolution, deterministic validators reject explicit `llm_profile`
+values.
+
+Configured LLM-backed validators with explicit `llm_profile` values are
+validated against the configured Scriptorium profile source. Deterministic
+production validators do not call the LLM and must not set `llm_profile`.
 
 ## Implemented Production Modules
 
@@ -167,10 +280,37 @@ one configured profile.
 | --- | --- | --- |
 | input | `seriatim` | Reads Seriatim transcript JSON. |
 | chunk | `generic` | Splits source units into ordered chunks. |
-| extract | `dnd/spells` | Extracts `dnd.spell_cast` artifacts. |
+| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
-| merge | `appendorder` | Keeps candidates in append order. |
+| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
-| normalize | `noop` | Passes merged artifacts through unchanged. |
+| merge | `appendorder` | Combines typed artifacts in chunk order. |
-| output | `json` | Produces JSON output files. |
+| normalize | `noop` | Passes merged typed artifacts through unchanged. |
+| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
+
+## Implemented Production Validators
+
+| Key | Execution | Notes |
+| --- | --- | --- |
+| `generic/always_accept` | deterministic | Accepts returned module output. |
+| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
+| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
+| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
+| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
+| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
+| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
+
+The production default chain for the `dnd/spells` extractor is:
+
+```yaml
+validators:
+  - generic/valid_json
+  - generic/valid_json_schema
+  - extract/dnd/spells/shape
+  - extract/dnd/spells/source_refs
+  - extract/dnd/spells/source_relatedness
+```
+
+No other production module currently has a default validator chain. Empty
+chains approve output by default.
 
 The `generic` chunker accepts:
 
@@ -178,36 +318,155 @@ The `generic` chunker accepts:
 - `overlap_units`: non-negative integer, default `0`, and must be less than
   `max_units`.
 
-## Diagnostics
+The `dnd/scenes` chunker requires transcript source capabilities, calls the
+configured structured LLM runtime, and does not accept module options. It
+declares optional `players`, `party`, and `glossary` references for scene
+disambiguation, and accepts `roster` as a deprecated compatibility alias for
+`party`.
 
-`diagnostics` fields:
+The `dnd/spells` extractor declares optional reference slots:
 
-- `work_dir`: directory for per-run diagnostics. Default: `/tmp/notarius`.
+- `players`
-- `retention`: `auto`, `always`, or `never`. Empty uses `auto`.
+- `party`
+- `glossary`
+- `roster` as a deprecated compatibility alias for `party`
 
-`auto` retains diagnostics for failed runs and successful runs with warnings.
+Both modules accept UTF-8 plain text, Markdown, YAML, or JSON reference files.
-`always` retains diagnostics for every run. `never` removes diagnostics for
+The extractor uses references only as supporting disambiguation material; spell
-successful runs without regard to warnings; failed runs are retained.
+casts still must be present in the source transcript.
 
-The `--diagnostics-dir` run flag overrides `diagnostics.work_dir` for that
+## State Surfaces
-invocation.
+
+The `output`, `cache`, and `debug` top-level fields select independent physical
+roots. Their layout, permissions, lifecycle, and sensitive-data handling are
+defined in [Operations](operations.md).
+
+```yaml
+output:
+  directory: ./notarius-output
+cache:
+  chunk_plans:
+    directory: ""
+    mode: auto
+  checkpoints:
+    directory: ""
+debug:
+  directory: ./notarius-debug
+```
+
+`output.directory` is the durable output root. Its precedence is
+`--output-dir`, `NOTARIUS_OUTPUT_DIR`, the file value, then the default.
+
+`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh`. Its precedence
+is `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then
+`auto`. `auto` reuses a valid source-addressed plan and regenerates missing or
+invalid records; `bypass` performs no plan-cache I/O; `refresh` regenerates and
+publishes a plan after chunk validation.
+
+`cache.chunk_plans.directory` and `cache.checkpoints.directory` each name an
+exact cache-family root. Their precedence is the corresponding environment
+variable, the file value, then the family-specific per-user default. There is
+no CLI cache-root override. The defaults are
+`<os.UserCacheDir>/notarius/chunk-plans` and
+`<os.UserCacheDir>/notarius/checkpoints`; on Unix, `os.UserCacheDir` ordinarily
+uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative
+`XDG_CACHE_HOME` is an error.
+
+Checkpoint I/O occurs only for `notarius run --resume`. That invocation loads
+compatible checkpoints and records work it executes. Without `--resume`,
+Notarius does not resolve, create, load, or record the checkpoint root.
+
+`debug.directory` chooses a root but never enables debug capture. Its precedence
+is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default.
+Only `--debug` requests a bundle; `--debug-dir` is valid only with `--debug`.
+
+Every supplied file, environment, and CLI value is validated even when a
+higher-precedence value wins.
+
+## Version 2 To Version 3 Migration
+
+Version 2 files are rejected. Move each setting to the surface it controls and
+remove obsolete enablement and retention controls. This complete before/after
+example preserves an existing chunk-plan cache and checkpoint directory while
+choosing an output and debug root explicitly.
+
+```yaml
+# Version 2 (no longer accepted)
+version: 2
+workspace:
+  directory: /srv/notarius/state
+  resume:
+    enabled: true
+  debug:
+    enabled: true
+  chunk_cache:
+    directory: /srv/notarius/chunk-plans
+    mode: auto
+  diagnostics:
+    retention: always
+diagnostics:
+  work_dir: /srv/notarius/inspection
+```
+
+```yaml
+# Version 3
+version: 3
+output:
+  directory: /srv/notarius/output
+cache:
+  chunk_plans:
+    directory: /srv/notarius/chunk-plans
+    mode: auto
+  checkpoints:
+    directory: /srv/notarius/state/checkpoints
+debug:
+  directory: /srv/notarius/debug
+```
+
+Run the migrated configuration with `--resume` when checkpoint reuse or
+recording is wanted, and with `--debug` when a debug bundle is wanted.
+
+The removed fields are `workspace.directory`, `workspace.resume.enabled`,
+`workspace.debug.enabled`, `workspace.chunk_cache.mode`,
+`workspace.chunk_cache.directory`, `workspace.diagnostics.enabled`,
+`workspace.diagnostics.retention`, `diagnostics.work_dir`, and
+`diagnostics.retention`. The removed environment variables are `NOTARIUS_WORKSPACE_DIR`,
+`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`,
+`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`,
+`NOTARIUS_WORKSPACE_RESUME_ENABLED`, `NOTARIUS_WORKSPACE_DEBUG_ENABLED`,
+`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`,
+`NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`, `NOTARIUS_WORK_DIR`, and
+`NOTARIUS_DIAGNOSTICS_RETENTION`. The chunk-cache variables are replaced by
+`NOTARIUS_CACHE_CHUNK_PLANS_MODE` and
+`NOTARIUS_CACHE_CHUNK_PLANS_DIR`; the former shared directory has no direct
+replacement.
 
 ## Validation
 
 Configuration validation checks:
 
 - supported config version and known YAML fields;
+- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
 - non-empty, non-duplicated IDs after trimming;
-- supported LLM provider and non-negative profile limits;
 - positive global LLM concurrency;
-- supported diagnostics retention and non-empty work directory;
+- supported stage-worker keys and an effective extract worker count in the
-- module binding LLM profiles refer to configured profiles.
+  inclusive range `1..concurrency.total_llm`;
+- non-empty output and debug directories;
+- a supported chunk-cache mode and state-surface directories without NUL bytes;
+- stale removed fields such as `llm_profiles`.
 
 Pipeline resolution additionally checks:
 
 - the pipeline ID exists;
 - at least one artifact lane is declared and selected;
-- selected lanes exist when `--only` is used;
+- lanes selected through the CLI exist in the resolved pipeline;
 - required module keys are present;
 - module keys are registered for the expected slot;
-- module capability requirements are satisfied.
+- module capability requirements are satisfied;
+- non-empty validator overrides reference registered validator keys;
+- deterministic validators do not set `llm_profile`;
+- LLM-backed validators with explicit `llm_profile` values reference configured
+  Scriptorium profile IDs;
+- bound reference slots are declared by selected chunk, extractor, merger, or
+  normalizer targets;
+- required reference slots are bound for selected targets.

docs/development.md

@@ -0,0 +1,40 @@
+# Development
+
+This is the first-read landing page for people and LLM coding agents working on
+Notarius. It provides a concise repository orientation and routes each kind of
+change to its canonical documentation.
+
+Notarius is a Go CLI for configured structured extraction workflows. Start with
+the [README](../README.md) for product context, [Architecture](policy/architecture.md)
+for system boundaries, and [Internal Overview](internal/overview.md) for the
+implemented component map.
+
+## What To Read
+
+| When working on | Read | Why |
+| --- | --- | --- |
+| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. |
+| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
+| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
+| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
+| Production modules or validators | [Module Internals](internal/modules.md) | It documents implemented module contracts, capabilities, assets, and registration. |
+| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
+| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
+| CLI or user-visible configuration behavior | [CLI Reference](cli.md) and [Configuration](config.md) | These are the canonical user and operator references. |
+| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
+| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
+
+For an existing subsystem, also inspect its focused tests and the package-local
+types and contracts before changing behavior.
+
+## Validation
+
+Use focused package tests while iterating. Run the repository-wide checks when
+a change affects shared contracts, application behavior, or maintained
+documentation examples:
+
+```sh
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```

docs/integrations/dnd-spell-artifacts.md

@@ -1,126 +1,69 @@
-# D&D Spell-Cast Artifacts
+# D&D Spell Artifact
 
-This document is the durable artifact contract for approved
+This document is the durable serialized artifact contract for the production
-`dnd.spell_cast` artifacts produced by the implemented `dnd/spells` extractor.
+D&D spell extractor. Selectable extractor keys are cataloged in
+[Configuration](../config.md#implemented-production-modules).
 
-## Artifact Identity
+## Identity
 
-- Extractor key: `dnd/spells`
+- Artifact kind: `dnd/spell-list`
-- Artifact type: `dnd.spell_cast`
-- Schema version: `v1`
 - Prompt ID: `dnd.spells`
 - Response schema key: `dnd_spells`
 - Response schema ID: `notarius.dnd.spells`
 - Response schema name: `notarius_dnd_spells_v1`
+- Response schema version: `v1`
+- Media type: `application/json`
 
-The extractor requires source chunks and transcript source capability. It
+The durable JSON Schema is owned by the D&D spell artifact codec. The
-returns generic artifact candidates that are serialized by the JSON output
+extractor's private LLM response schema is a separate transport contract: its
-module.
+source-reference objects omit `source_id`, which the extractor assigns while
+mapping the response to the canonical artifact. The LLM DTO and transport
+schema are not part of this durable contract.
 
-## Artifact Envelope
+The output contains canonical spell casts derived from transcript evidence.
+Source IDs are assigned from the input identity; source-unit ranges identify
+the evidence location.
 
-Approved artifacts use the generic artifact envelope documented in
+## Output Shape
-[JSON Output](json-output.md#artifact-files):
 
-```json
+The extractor payload is a JSON object with one required top-level array. Its
-{
+structure is:
-  "extractor_key": "dnd/spells",
+
-  "artifact_type": "dnd.spell_cast",
+```text
-  "schema_version": "v1",
+{"spell_casts": [<spell-cast object>, ...]}
-  "payload": {
-    "caster": "Aria",
-    "spell": "Cure Wounds",
-    "effect": "heals an injured ally",
-    "narrative_description": "Aria raises her holy symbol and casts Cure Wounds."
-  },
-  "source_refs": [
-    {
-      "source_id": "session-alpha",
-      "start_unit_id": "seg-001",
-      "end_unit_id": "seg-001"
-    }
-  ]
-}
 ```
 
-## Payload Fields
+`spell_casts` must be present. It may be empty when no spell casts are found.
+When multiple chunk results are combined, spell casts remain in chunk order.
+When the payload is written as durable output, its logical path is derived from
+the configured artifact lane ID as defined by the
+[JSON output contract](json-output.md#output-payload-files).
 
-The `payload` object contains:
+## Spell-Cast Fields
+
+Each spell cast contains exactly these required fields:
 
 - `caster`: in-world character or creature casting the spell;
 - `spell`: spell name;
 - `effect`: concise spell effect in the scene;
-- `narrative_description`: short description of the spell cast in context.
+- `narrative_description`: short description of the spell cast in context;
+- `source_refs`: transcript source references with extractor-assigned source
+  IDs and evidence unit ranges. It must contain at least one entry.
 
-All payload fields are strings and must be non-empty after trimming.
+All four string fields must be non-empty. `caster` is the in-world caster, not
-
+the transcript speaker. Unknown fields are rejected.
-`caster` is the in-world caster, not the transcript speaker.
 
 ## Source References
 
-Source references live on the artifact envelope as `source_refs`; they are not
+Each source reference contains exactly three required fields: `source_id`,
-duplicated inside the `payload`.
+`start_unit_id`, and `end_unit_id`. The source ID must match the input identity.
+The unit IDs must be positive integers present in the input, and the start unit
+must not appear after the end unit. Unknown fields are rejected.
 
-Each source reference uses the generic source-reference shape:
+Reference slot keys and accepted file types are defined in
-
+[Configuration](../config.md#implemented-production-modules). References are
-- `source_id`
+supporting disambiguation material, not source evidence, and are not
-- `start_unit_id`
+addressable through `source_refs`.
-- `end_unit_id`
-
-Validation requires:
-
-- at least one source reference;
-- non-empty source ID and unit IDs;
-- source ID matching the source document ID;
-- start and end unit IDs existing in the source document;
-- start unit appearing before or at the same position as end unit.
-
-## Structured LLM Response Shape
-
-The extractor asks the LLM for this top-level response shape:
-
-```json
-{
-  "spell_casts": [
-    {
-      "caster": "Aria",
-      "spell": "Cure Wounds",
-      "effect": "heals an injured ally",
-      "narrative_description": "Aria raises her holy symbol and casts Cure Wounds.",
-      "source_refs": [
-        {
-          "source_id": "session-alpha",
-          "start_unit_id": "seg-001",
-          "end_unit_id": "seg-001"
-        }
-      ]
-    }
-  ]
-}
-```
-
-`spell_casts` must be present. It may be empty when no spell casts are found.
-
-The response schema asset is embedded at
-`internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json`.
-
-## Validators
-
-The extractor supplies two deterministic validators by default:
-
-- `dnd/spells/shape`
-- `dnd/spells/source_refs`
-
-Rejection reason codes:
-
-- `invalid_payload`: payload JSON cannot be decoded as a spell-cast payload.
-- `missing_required_field`: `caster`, `spell`, `effect`, or
-  `narrative_description` is blank.
-- `missing_source_ref`: candidate has no source references.
-- `invalid_source_ref`: at least one source reference fails generic source
-  reference validation.
-
-Rejected candidates are written to `rejected.json` by the JSON output module.
 
 ## Manifest Metadata
 
@@ -145,8 +88,3 @@ manifest metadata:
 ```
 
 Raw prompt and schema content are not included in manifest metadata.
-
-## Compatibility Limit
-
-This contract covers only `dnd.spell_cast` artifacts produced by the
-implemented spell-cast extractor.

docs/integrations/json-output.md

@@ -1,30 +1,26 @@
 # JSON Output
 
 This document is the durable JSON output file-format contract produced by the
-implemented `json` output module and written by the CLI.
+production JSON encoder and written by the CLI. Selectable output-encoder keys
+are cataloged in
+[Configuration](../config.md#implemented-production-modules).
 
-## Output Directory
+The output module produces the logical bundle described here. The CLI's
-
+physical placement and lifecycle for that bundle are defined in
-The CLI writes logical output files under:
+[Operations](../operations.md#output-directory).
-
-```text
-<output-root>/<run-id>/
-```
-
-The default output root is `./notarius-output`. Operational behavior is covered
-in [Operations](../operations.md).
 
 ## Files
 
-The `json` output module writes:
+The encoder writes:
 
 - `index.json`
 - `manifest.json`
-- `artifacts/<artifact-type>.json`, one file per approved artifact type
+- `lanes/<lane-id>.json`, one file per normalized serialized artifact
 - `rejected.json`
 - `warnings.json`
 
-Files are pretty-printed JSON with a trailing newline.
+Files are pretty-printed JSON with a trailing newline when the payload is JSON.
+Logical file paths are relative, slash-separated, and may not contain `..`.
 
 ## `index.json`
 
@@ -33,10 +29,15 @@ Shape:
 ```json
 {
   "manifest_file": "manifest.json",
-  "artifact_files": [
+  "output_files": [
     {
-      "artifact_type": "dnd.spell_cast",
+      "lane_id": "spells",
-      "file": "artifacts/dnd.spell_cast.json"
+      "media_type": "application/json",
+      "file": "lanes/spells.json",
+      "module_key": "noop",
+      "schema_id": "notarius.dnd.spells",
+      "schema_name": "notarius_dnd_spells_v1",
+      "schema_version": "v1"
     }
   ],
   "rejected_file": "rejected.json",
@@ -44,25 +45,30 @@ Shape:
 }
 ```
 
-`artifact_files` is sorted by artifact type. It is empty when no artifacts are
+`output_files` is sorted by lane ID. Output file names are produced by
-approved.
+sanitizing the lane ID:
+
+- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
+- repeated `..` sequences are replaced;
+- leading and trailing `.`, `_`, and `-` are trimmed;
+- empty sanitized names are rejected;
+- two lanes that sanitize to the same output file are rejected.
+
+`manifest_file`, `rejected_file`, and `warnings_file` contain the fixed paths
+shown above. Each `output_files` entry requires `lane_id` and `file`. It also
+contains the normalized payload `media_type`, normalizer `module_key`, and
+response `schema_id`, `schema_name`, and `schema_version` when those values are
+available.
 
 ## `manifest.json`
 
-`manifest.json` contains a run manifest:
+`manifest.json` contains a run manifest. This abridged example shows its core
+structure:
 
 ```json
 {
   "run_id": "run-123",
   "pipeline_id": "dnd-session",
-  "pipeline_digest": "sha256:...",
-  "input_module": "seriatim",
-  "chunker": "generic",
-  "source_digests": ["sha256:..."],
-  "extractors": ["dnd/spells"],
-  "merger": "appendorder",
-  "normalizer": "noop",
-  "output_encoder": "json",
   "artifact_lanes": [
     {
       "id": "spells",
@@ -71,13 +77,6 @@ approved.
       "normalizer": "noop"
     }
   ],
-  "llm_profiles": [
-    {
-      "id": "default",
-      "provider": "openai-compatible",
-      "model": "configured-model"
-    }
-  ],
   "validation_status": "approved",
   "started_at": "2026-01-01T00:00:00Z",
   "completed_at": "2026-01-01T00:00:01Z"
@@ -86,52 +85,73 @@ approved.
 
 Fields with empty values may be omitted by JSON encoding.
 
-`validation_status` is `approved` when no candidates were rejected and
+The manifest fields are:
-`rejected` when one or more candidates were rejected.
 
-## Artifact Files
+- `run_id`, `pipeline_id`, and `pipeline_digest`: run and resolved-pipeline
+  identity;
+- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and
+  `output_encoder`: resolved module keys;
+- `chunk_plan`: payload-free provenance for the effective chunk plan. `mode`
+  is the effective cache mode; `action` is `reused`, `generated`,
+  `refreshed`, or `bypassed` when a plan was materialized. `requested_module`
+  is the current pipeline chunker, while `producer_input_module`,
+  `producer_module`, `producer_llm_profile`, `producer_references`,
+  `producer_metadata`, `source_digest`, `plan_digest`, `plan_schema_version`,
+  and `created_at` describe the stored or generated producer when available.
+  A cached plan can therefore identify a producer different from the requested
+  module. This object never embeds ranges, units, annotations, prompts,
+  responses, or reference content;
+- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
+  including prompt and response-schema provenance when provided;
+- `validator_chains`: resolved validation points and validators;
+- `source_digests` and `references`: source and reference provenance;
+- `normalized_outputs` and `rejected_outputs`: payload-free result summaries;
+- `llm_profiles`: selected profile IDs and provider or model names when
+  available;
+- `metadata`: the effective prompt `session_id`;
+- `validation_status`: `approved` or `rejected`;
+- `started_at` and `completed_at`: UTC run timestamps.
 
-Each artifact file has this shape:
+`source_digests` contains source document digests only. Bound references are
+recorded separately under `references`, which contains provenance only: target
+stage, lane ID when present, slot name, origin type and URI, digest, media
+type, byte size, and binding source. Reference content is not written to
+durable output.
 
-```json
+Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is
-{
+omitted for chunk references and present for extract, merge, and normalize
-  "artifact_type": "dnd.spell_cast",
+references.
-  "artifacts": [
-    {
-      "extractor_key": "dnd/spells",
-      "artifact_type": "dnd.spell_cast",
-      "schema_version": "v1",
-      "payload": {},
-      "source_refs": [
-        {
-          "source_id": "session-alpha",
-          "start_unit_id": "seg-001",
-          "end_unit_id": "seg-001"
-        }
-      ]
-    }
-  ]
-}
-```
 
-Artifact envelope fields:
+`validation_status` is `approved` when no outputs were rejected and `rejected`
+when one or more outputs were rejected.
 
-- `extractor_key`: extractor module key.
+Producer warnings and the current run's chunk-validation warnings remain in
-- `artifact_type`: artifact type.
+`warnings.json`. The manifest records only provenance and decision summaries;
-- `schema_version`: artifact schema version.
+empty producer-only values are omitted for compatibility with existing readers.
-- `payload`: artifact-type-specific JSON payload.
-- `source_refs`: optional generic source references.
-- `metadata`: optional artifact metadata.
 
-Artifact file names are produced by sanitizing the artifact type:
+`validator_chains` records the resolved validator chain for each validation
+point. Entries include stage, lane ID when applicable, module key, and validators
+with key and execution class. Empty chains are recorded with an empty
+`validators` array, including chains resolved from explicit empty config
+overrides.
 
-- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
+`normalized_outputs` summarizes each normalized lane output without embedding
-- repeated `..` sequences are replaced;
+payload bytes. Entries include lane ID, normalizer module key, source ID, media
-- leading and trailing `.`, `_`, and `-` are trimmed;
+type, and response schema provenance where available.
-- empty sanitized names are rejected.
 
-For current D&D spell-cast artifacts, the file is
+`rejected_outputs` summarizes rejected module outputs without embedding raw
-`artifacts/dnd.spell_cast.json`.
+payload bytes. Entries include stage, lane, module, chunk, validator or reason,
+message, attempt count, and optional diagnostic artifact path.
+
+## Output Payload Files
+
+Each normalized serialized artifact is written to
+`lanes/<sanitized-lane-id>.json`. The JSON output encoder is domain-neutral and
+accepts only artifacts whose codec media type is `application/json`. The file
+contains the codec-owned JSON bytes pretty-printed.
+
+The schema of each lane payload is owned by that artifact contract. For the
+current D&D spell lane, see [D&D Spell Artifact](dnd-spell-artifacts.md).
 
 ## `rejected.json`
 
@@ -139,25 +159,14 @@ Shape:
 
 ```json
 {
-  "rejected": [
+  "rejected": []
-    {
-      "candidate": {
-        "index": 0,
-        "extractor_key": "dnd/spells",
-        "artifact_type": "dnd.spell_cast",
-        "schema_version": "v1",
-        "payload": {},
-        "source_refs": []
-      },
-      "validator_name": "dnd/spells/source_refs",
-      "reason_code": "missing_source_ref",
-      "message": "spell cast candidate must include at least one source ref"
-    }
-  ]
 }
 ```
 
-`rejected` is an empty array when no candidates are rejected.
+When output validation rejects an output, each entry contains `stage` and
+`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
+`validator_name`, `reason_code`, `attempt_count`, and
+`diagnostic_artifact_path` when applicable.
 
 ## `warnings.json`
 
@@ -167,26 +176,14 @@ Shape:
 {
   "warnings": [
     {
-      "scope": "output",
+      "scope": "extract",
-      "reason_code": "example_warning",
+      "reason_code": "example",
-      "message": "warning message"
+      "message": "human-readable warning"
     }
   ]
 }
 ```
 
 `warnings` is an empty array when no warnings are reported.
-
+Each warning requires `reason_code` and `message`; `scope` is omitted when it is
-## Path Safety
+empty.
-
-The output module returns slash-separated logical paths. The CLI also validates
-logical output names before writing:
-
-- names must be non-empty;
-- names must be relative;
-- names must be clean;
-- names must use `/`, not `\`;
-- names must not contain `..`;
-- resolved paths must stay under the run output directory.
-
-Durable writes are atomic per file.

docs/integrations/openai-compatible.md

@@ -1,128 +0,0 @@
-# OpenAI-Compatible Structured Output
-
-This document describes the external LLM provider contract implemented by the
-production Notarius LLM client.
-
-## Provider
-
-- Provider key: `openai-compatible`
-- HTTP method: `POST`
-- Endpoint: `<base_url>/chat/completions`
-- Request body: JSON
-- Response mode: chat completions with structured JSON schema output
-
-`base_url` is trimmed of trailing slashes before `/chat/completions` is
-appended. Configure provider settings in [Configuration](../config.md).
-
-## Request
-
-The client sends a JSON object with:
-
-```json
-{
-  "model": "configured-model",
-  "messages": [
-    {
-      "role": "system",
-      "content": "..."
-    },
-    {
-      "role": "user",
-      "content": "..."
-    }
-  ],
-  "response_format": {
-    "type": "json_schema",
-    "json_schema": {
-      "name": "schema_name",
-      "strict": true,
-      "schema": {}
-    }
-  }
-}
-```
-
-Implemented request behavior:
-
-- `model` comes from the structured completion request when set, otherwise from
-  the configured LLM profile.
-- `messages` must be non-empty; each role and content must be non-empty after
-  trimming.
-- `response_format.type` is always `json_schema`.
-- `response_format.json_schema.strict` is always `true`.
-- `response_format.json_schema.name` and `schema` come from the extractor or
-  validator making the call.
-
-If an API key is configured, the client sends:
-
-```text
-Authorization: Bearer <api-key>
-```
-
-The client always sends `Content-Type: application/json`.
-
-## Response
-
-The client expects a JSON response with at least one choice:
-
-```json
-{
-  "model": "provider-model",
-  "choices": [
-    {
-      "message": {
-        "content": "{\"field\":\"value\"}"
-      }
-    }
-  ],
-  "usage": {
-    "prompt_tokens": 10,
-    "completion_tokens": 5,
-    "total_tokens": 15
-  }
-}
-```
-
-`choices[0].message.content` may be either:
-
-- a JSON string whose contents are valid JSON; or
-- raw JSON.
-
-The decoded content is unmarshaled into the caller-provided structured output
-target. If `usage` is present, prompt, completion, and total token counts are
-copied into the completion response.
-
-## Errors And Retries
-
-The client validates base URL, model, response schema name, response schema
-JSON, messages, and output target before or during the call.
-
-Retryable failures:
-
-- HTTP request failure;
-- response body read failure;
-- HTTP `429`;
-- HTTP `5xx`;
-- malformed provider response envelope;
-- missing choices;
-- missing, empty, or invalid assistant JSON content;
-- structured-output decode failure.
-
-Non-retryable provider status codes include non-`429` `4xx` responses.
-
-Provider error bodies are parsed for `error.message` or `message` when present.
-Configured API key values and bearer-token values are redacted from returned
-provider errors.
-
-## Timeouts And Concurrency
-
-The configured profile timeout is applied per provider request when greater
-than zero. Context cancellation is respected.
-
-The production CLI wraps the provider client with the LLM scheduler. Effective
-concurrency is described in [LLM runtime internals](../internal/llm.md).
-
-## Limits
-
-This contract documents only the fields the implemented client sends and reads.
-Provider-specific extensions are ignored unless they affect those fields.

docs/integrations/seriatim.md

@@ -1,116 +1,65 @@
 # Seriatim Transcript JSON
 
-This document is the external input contract for the implemented `seriatim`
+This document is the external input contract consumed by the production
-input adapter.
+Seriatim input adapter. Selectable input-adapter keys are cataloged in
+[Configuration](../config.md#implemented-production-modules).
 
 ## Adapter
 
-- Module key: `seriatim`
-- Document kind: `transcript`
-- Unit kind: `transcript_segment`
 - Source format: `application/vnd.seriatim+json`
 
-The adapter parses raw Seriatim JSON into a generic source document. It owns
-transcript-specific JSON parsing and metadata mapping; core source and pipeline
-code stay source-format agnostic.
-
 ## Accepted Shape
 
 The input must be one JSON object with top-level `metadata` and `segments`
 fields. This covers the maintained minimal fixture and Seriatim intermediate
 output that provides the same required segment fields.
 
-```json
-{
-  "metadata": {
-    "id": "session-alpha",
-    "title": "Synthetic D&D spell session"
-  },
-  "segments": [
-    {
-      "id": "seg-001",
-      "start": 0,
-      "end": 4,
-      "speaker": "Aria",
-      "text": "Aria raises her holy symbol and casts Cure Wounds."
-    }
-  ]
-}
-```
-
 The maintained example is
 [examples/seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json).
 
-Top-level metadata entries are preserved. Other segment fields, such as
+Required top-level fields:
-`categories`, are ignored.
+
+- `metadata`: an object. Its entries are accepted as source metadata.
+- `segments`: a non-empty array of segment objects.
+
+Required segment fields:
+
+- `id`: a positive integer JSON number or canonical decimal string without
+  leading zeros or surrounding whitespace;
+- `start`: a finite, non-negative JSON number or numeric string;
+- `end`: a finite, non-negative JSON number or numeric string that is not less
+  than `start`;
+- `speaker`: a non-empty string;
+- `text`: a non-empty string.
+
+Other top-level and segment fields, such as `categories`, are ignored.
 
 Multiple top-level JSON values are rejected.
 
 ## Validation
 
-The adapter rejects:
+The adapter rejects empty input, malformed JSON, multiple top-level JSON values,
-
+non-object segment values, duplicate segment IDs, and any violation of the
-- empty raw input;
+shape or field constraints above.
-- malformed JSON;
-- top-level JSON that is not an object;
-- missing, null, or non-object `metadata`;
-- missing, null, non-array, or empty `segments`;
-- segment values that are not objects;
-- segment `id` values that are neither strings nor numbers;
-- non-string `speaker` or `text`;
-- empty segment IDs;
-- segment IDs with leading or trailing whitespace;
-- duplicate segment IDs;
-- missing or empty `speaker`;
-- missing, empty, invalid, non-finite, or negative `start`;
-- missing, empty, invalid, non-finite, or negative `end`;
-- `end` values before `start`;
-- missing or empty `text`.
 
 Segment text is preserved as provided, but it must not be empty after trimming.
 
-## Source Mapping
+## Derived Identity
 
-The adapter maps input to `SourceDocument`:
+Notarius identifies the parsed source in this order:
 
-- `metadata` becomes `SourceDocument.Metadata`;
+1. `metadata.id`, when it is a non-empty string after trimming;
-- `SourceDocument.Kind` is `transcript`;
+2. `metadata.source_id`, when it is a non-empty string after trimming;
-- `SourceDocument.Format` is `application/vnd.seriatim+json`;
+3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
-- `SourceDocument.Digest` is `sha256:<hex>` of the exact raw input bytes.
 
-`SourceDocument.ID` is selected in this order:
+The exact raw input SHA-256 remains the basis of the fallback source ID. The
-
+source digest recorded in output provenance is instead the SHA-256 of the
-1. the parse request source ID, after trimming;
+canonical generic source document, excluding the digest field itself. It covers
-2. `metadata.id`, when it is a non-empty string after trimming;
+the derived source identity, document kind and format, ordered units and their
-3. `metadata.source_id`, when it is a non-empty string after trimming;
+self-references, and accepted metadata. Segment IDs become the unit IDs used by
-4. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
+artifact source references; each produced unit carries a self-reference whose
-
+source ID is the derived document ID and whose start and end IDs both equal the
-Each segment becomes one `SourceUnit`:
+segment ID.
-
-- `segment.id` becomes `SourceUnit.ID`; numeric IDs are converted to their JSON
-  number text, so `1` becomes `"1"`;
-- `segment.text` becomes `SourceUnit.Text`;
-- `SourceUnit.Kind` is `transcript_segment`;
-- `speaker`, `start`, and `end` are stored in source-unit metadata.
-
-## Metadata Keys
-
-Seriatim unit metadata uses these keys:
-
-- `speaker`: string speaker label;
-- `start`: `json.Number` start value;
-- `end`: `json.Number` end value.
-
-The `internal/modules/input/seriatim` package exposes typed accessors for these
-values.
-
-## Capabilities
-
-The module declares these provided capabilities:
-
-- `source.transcript`
-- `transcript.speaker`
-- `transcript.timestamps`
 
 ## Compatibility Limit
 

docs/internal/diagnostics.md

@@ -1,88 +0,0 @@
-# Diagnostics Internals
-
-Diagnostics internals live in `internal/core/diagnostics`. Operator-facing run
-behavior is documented in [Operations](../operations.md).
-
-## Purpose
-
-Diagnostics provide local inspection artifacts for a run without becoming the
-durable output contract. Durable user output is produced by output modules and
-written by the CLI.
-
-Diagnostics must not expose secrets.
-
-## Run Directory
-
-`NewRunDirectory(workDir, retention)` creates:
-
-```text
-<workDir>/run-<unix-nanoseconds>/
-```
-
-If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults
-to `auto`.
-
-The writer makes the work directory if needed, then attempts to create a unique
-run directory. It retries run ID creation a bounded number of times if a
-collision occurs.
-
-## Artifact Writers
-
-Implemented artifact names:
-
-- `invocation.json`
-- `effective-config.json`
-- `resolved-pipeline.json`
-- `source-document.json`
-- `run-manifest.json`
-- `run-report.json`
-- `warnings.json`
-- `error.log`
-
-JSON artifacts are encoded with indentation and a trailing newline. Writes are
-atomic through a temporary file in the target directory followed by rename.
-
-Artifact names must be single relative file names. Absolute paths, path
-separators, and names resolving outside the run directory are rejected.
-
-## Redacted Effective Config
-
-Diagnostics writers accept payloads that implement
-`RedactedDiagnosticsPayload`. `internal/core/config` uses this to redact API
-keys in effective config diagnostics while preserving resolved pipeline context.
-
-The redaction path clones config data before replacing secret values.
-
-## Retention
-
-Retention is decided by `ShouldRetainRunDirectory`.
-
-- Failed runs are always retained.
-- `always` retains successful runs.
-- `never` removes successful runs.
-- `auto` retains successful runs only when warnings exist.
-- Unknown retention values are treated as retain by the retention decision, but
-  config validation rejects unsupported values before normal runs.
-
-`ApplyRetention` removes only the specific run directory.
-
-## CLI Failure Behavior
-
-The CLI creates the diagnostics run directory after config loading and before
-pipeline resolution. Failures before that point do not have diagnostics.
-
-After diagnostics creation, run failures call `WriteErrorLog` and apply
-retention with `RunSucceeded: false`, so the run directory remains available.
-
-When the pipeline returns a partial manifest on failure, the CLI writes that
-manifest before logging the failure.
-
-## Invariants
-
-- Diagnostics paths must be narrow and run-directory scoped.
-- Writes should be atomic where practical.
-- Secrets must be redacted.
-- Diagnostics write failures are command failures because they can hide the
-  information needed for recovery.
-- Durable output file contracts belong to output modules and integration docs,
-  not to diagnostics.

docs/internal/llm.md

@@ -1,116 +1,147 @@
-# LLM Runtime
+# LLM Runtime Internals
 
-The implemented LLM runtime lives in `internal/framework/llm`. It provides
+`internal/framework/llm` implements Notarius's transport boundary for structured
-transport-neutral structured completion contracts, an OpenAI-compatible HTTP
+completion. It contains the Scriptorium adapter, concurrency scheduler,
-adapter, concurrency scheduling, schema registry helpers, retry behavior, and
+prompt/schema registries, selected-profile recording, and provider-error
-secret redaction.
+redaction.
 
-## Contract
+Provider-neutral ownership rules are defined in
+[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
+credentials, and concurrency settings are defined in
+[Configuration](../config.md).
 
-Modules depend on `contracts.StructuredLLMClient`:
+## Structured Contract
 
-```go
+Modules and LLM-backed validators depend on
-CompleteStructured(ctx, request, out) (response, error)
+`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
-```
+prompt and optional profile/session, supplies named input materials and
+variables, and provides a caller-owned decoding target. A successful response
+contains the validated raw structured bytes plus non-secret provider, model,
+profile, and token metadata.
 
-The request contains messages, optional model override, response schema name,
+The caller owns prompt selection, response-schema selection, and interpretation
-and response schema JSON. The caller supplies a pointer target for decoded
+of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
-structured output.
+their origin metadata so the adapter can pass named artifacts to Scriptorium
+without exposing Scriptorium types through stage contracts.
 
-Extractors own prompts and schemas. Provider adapters should not contain
+## Production Construction
-domain-specific prompt logic.
 
-## Production Client Construction
+`internal/cli` constructs the production runtime by:
 
-`internal/cli` builds the production LLM client from the effective config:
+1. allocating the asset registry populated by the generic, Seriatim, and D&D
+   package-family registrars;
+2. creating a `ScriptoriumClient` from the effective profile source;
+3. attaching an `LLMProfileRecorder`;
+4. creating a scheduler from the effective concurrency limit;
+5. returning a `ScheduledClient` wrapper;
+6. decorating that shared client before preparation when debug recording is
+   enabled; and
+7. injecting that one shared client into complete pipeline preparation before
+   the source file is read or the runner is invoked.
 
-1. find the effective LLM profile;
+The D&D scene chunker and spell extractor retain this injected client and use
-2. build `OpenAICompatibleClientConfig`;
+it for every structured completion. Operation requests do not carry an LLM
-3. create an OpenAI-compatible client;
+client.
-4. create a scheduler from profile or global concurrency;
-5. wrap the client with `NewScheduledClient`;
-6. return non-secret LLM profile manifest metadata.
 
-The current run command requires exactly one distinct effective LLM profile for
+The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
-the resolved pipeline.
+and validator bindings. It prepares a small internal check prompt for each ID so
+missing or invalid profiles fail before pipeline execution. The runtime profile
+override syntax and scope are defined in the
+[CLI reference](../cli.md#run); binding rules are defined in
+[Configuration](../config.md#module-bindings).
 
-## OpenAI-Compatible Adapter
+## Scriptorium Adapter
 
-`OpenAICompatibleClient` posts JSON to:
+`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
+It validates the decoding target and prompt identity, maps named input materials
+to inline artifacts, forwards explicit profile and session context, delegates
+rendering/provider execution/structured validation, and unmarshals successful
+JSON into the caller target.
 
-```text
+Empty optional input material is represented by a single space so Scriptorium
-<base_url>/chat/completions
+retains the named input. The client returns Scriptorium's validated structured
-```
+bytes rather than re-encoding the caller target, allowing modules to preserve
+the runtime result exactly.
 
-It sends:
+Selected profile, provider, model, and token metadata are mapped into the
+Notarius response. The recorder deduplicates profiles by identity and supplies
+manifest-safe profile summaries after actual calls; manifest population does
+not guess the selected prompt default in advance.
 
-- `model`
+Generated-output validation failures and provider failures are wrapped with
-- `messages`
+prompt context. Error strings pass through bearer-token redaction before they
-- `response_format.type = "json_schema"`
+cross the runtime boundary.
-- `response_format.json_schema.name`
-- `response_format.json_schema.strict = true`
-- `response_format.json_schema.schema`
 
-If an API key is configured, the adapter sends an `Authorization: Bearer ...`
+## Scheduling
-header.
 
-The adapter accepts assistant content either as a JSON string containing JSON or
+`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
-as raw JSON content. It then unmarshals that content into the caller-provided
+acquisition increments the in-flight count; queued acquisition waits for a
-target.
+permit or context cancellation. Cancellation removes a queued waiter, while a
+cancelled waiter that has already received a permit releases it.
 
-External wire-contract details belong in the
+`ScheduledClient` acquires a permit around each structured completion and
-[OpenAI-compatible integration doc](../integrations/openai-compatible.md).
+defers release on every result path. The effective limit and default are
+configuration facts in [Configuration](../config.md#defaults).
 
-## Retries And Timeouts
+This provider-call ceiling is independent of the pipeline's extract worker
+limit. Concurrent lanes, retries, and validators all use the same scheduled
+client, so increasing framework workers cannot exceed `total_llm`. Pipeline
+dispatch and cancellation mechanics are documented in
+[Pipeline Internals](pipeline.md#execution-flow).
 
-The adapter retries:
+## Prompt And Schema Assets
 
-- provider request failures;
+`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
-- response read failures;
+and rejects invalid or conflicting registrations. Production module packages
-- HTTP `429`;
+register their own prompt and schema assets; generic framework code contains no
-- HTTP `5xx`;
+D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
-- malformed provider envelopes;
+filesystem composition helper used to combine module-owned files with shared
-- malformed assistant JSON;
+domain prompt fragments.
-- structured-output decode failures.
 
-Non-retryable `4xx` responses are returned without retry. Request timeout comes
+Schema helpers load embedded JSON Schema with identity and digest metadata,
-from the effective LLM profile. Context cancellation is respected.
+return defensive copies, and expose a diagnostics map that omits schema bytes.
+The small framework registry contains only generic test schemas; production
+schemas remain package-owned.
 
-## Scheduler
+## Debug And Redaction Boundaries
 
-`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
+The pipeline may wrap the client with a debug recorder that captures prepared
-FIFO queue of waiters. Cancellation removes queued waiters or releases granted
+prompt/response material for an explicitly requested debug run. Debug summaries
-permits.
+and manifests receive identities, hashes, usage, and selected profile summaries
+rather than prompt, source, reference, schema, or response content.
 
-`NewScheduledClient` wraps any structured LLM client and runs each completion
+The Scriptorium error wrapper removes bearer credential values from surfaced
-inside the scheduler.
+provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
+secret values elsewhere in the runtime. Config summaries use a separate
+clone-and-redact path in `internal/core/config`. These mechanisms implement the
+security invariant in
+[Architecture](../policy/architecture.md#state-output-and-safety); operator
+handling of debug data is defined in [Operations](../operations.md#debug).
 
-Effective concurrency is:
+## Failure Behavior
 
-1. `llm_profiles.<id>.max_concurrency`, when greater than zero;
+- Invalid targets, missing prompt IDs, malformed structured output, and
-2. `concurrency.total_llm`, when greater than zero;
+  Scriptorium failures return contextual errors to the calling module.
-3. `1`.
+- Scheduler construction rejects non-positive limits; acquisition respects
+  context cancellation.
+- Asset registration rejects invalid roots, missing content, and path conflicts.
+- Schema loading distinguishes missing assets, invalid JSON, and invalid
+  metadata.
+- Profile validation errors occur during CLI preparation when an explicit
+  selected ID cannot be prepared.
 
-## Schema Registry
+## Tests To Inspect
 
-The framework schema registry embeds generic test schemas. It also exposes
+- `internal/framework/llm/scriptorium_client_test.go` and
-helpers for caller-owned schemas:
+  `scriptorium_api_test.go`: adapter mapping and local HTTP integration.
-
+- `internal/framework/llm/scheduler_test.go` and
-- `LoadResponseSchema`
+  `scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
-- `LookupResponseSchema`
+  release.
-- `MustLookupResponseSchema`
+- `internal/framework/llm/asset_registry_test.go` and
-- `ResponseSchema.DiagnosticsMap`
+  `schema_registry_test.go`: asset composition, validation, and defensive
-
+  copies.
-`DiagnosticsMap` omits raw schema content and includes metadata such as key,
+- `internal/framework/llm/secrets_test.go`: provider-error redaction.
-ID, version, name, and SHA-256.
+- `internal/cli/run_test.go`: profile validation, production client wiring,
-
+  manifest recording, and debug integration.
-The D&D spell extractor owns and loads its own embedded response schema.
+- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
-
+  asset registration.
-## Secret Redaction
-
-Provider errors are passed through `ErrorWithSecretsRedacted` with the API key
-and bearer-token value. Config diagnostics use redacted effective config
-payloads.
-
-Do not add raw provider request bodies, response bodies, API keys, or prompt
-payloads to diagnostics by default.

docs/internal/modules.md

@@ -1,165 +1,248 @@
-# Modules
+# Module And Validator Internals
 
-Production modules live under `internal/modules`. Each module implements one
+Production module and validator implementations live under their domain-first
-contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
+trees in `internal/modules`.
-registers itself with the matching pipeline registry.
+The selectable keys, configuration options, reference slots, and default
+validator chain are canonical in the
+[module](../config.md#implemented-production-modules) and
+[validator](../config.md#implemented-production-validators) catalogs in
+Configuration.
 
-The CLI production catalog currently registers only the modules listed here.
+## Extension Pattern
 
-## Contract Pattern
+A stage module package provides a stable key, constructor, contract
+implementation, `ModuleSpec`, `Register`, and focused behavior and registration
+tests. A validator package follows the same pattern with `ValidatorSpec` and the
+validator registry. Package-family registrars compose those leaf registrations
+into the production catalog and own family-level policy such as default
+validator chains and prompt asset collection.
 
-A production module package should provide:
+Production input, chunk, output, and D&D spell-extract packages register strict
+option decoders and run-local builders. Preparation decodes their options into
+implementation-owned values and injects dependencies. The spell extractor is
+typed over the canonical D&D model. D&D validators, merge, and normalize use
+typed variants; JSON representation validators use serialized requests; and
+unconditional validators expose separate chunk and typed variants. The D&D
+production registrar registers only the canonical typed spell implementations.
 
-- a stable module key;
+Prepared extractors, extract validators, and codecs may be reused concurrently
-- a constructor such as `New`;
+by the run-wide extract pool. Production implementations are immutable after
-- the relevant contract implementation;
+construction: they retain only typed options, immutable assets, or the shared
-- `ModuleSpec`;
+concurrency-safe LLM client. Implementations that introduce mutable state must
-- `Register`;
+synchronize that state without creating a separate provider scheduler.
-- focused tests for registration, options, contract behavior, and errors.
 
-Module specs should describe capabilities accurately. Resolution uses specs to
+Specs expose capability and execution metadata without constructing an
-reject incompatible pipelines before execution.
+implementation. Registry entries separately expose option validation and
+run-local construction. Chunk, extract, merge, and normalize modules that accept
+auxiliary material declare identical reference slots from both
+`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
+that agreement. Runtime delivery uses the corresponding stage request's
+`References` field.
 
-## `seriatim` Input
+LLM-backed extensions own their prompt definitions and response schemas under
+package-local embedded assets. Shared filesystem composition belongs in
+`internal/framework/promptfs`; reusable D&D prompt fragments, reference
+declarations, prompt-input assembly, and source-unit helpers belong in
+`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
+completion types, not Scriptorium public types.
 
-Package: `internal/modules/input/seriatim`
+Reference material may inform a module or prompt but must not become source
+evidence. The resolver and materializer behavior is described in
+[Pipeline Internals](pipeline.md#reference-materialization).
 
-The `seriatim` adapter parses Seriatim transcript JSON into a generic source
+## Input Adapter
-document. It owns transcript JSON details, source ID selection, source digest
-creation, transcript segment validation, and segment metadata mapping.
 
-Provides:
+### `internal/modules/seriatim/input/transcript`
 
-- `source.transcript`
+The adapter decodes the supported transcript JSON, selects the source identity,
-- `transcript.speaker`
+computes canonical source provenance, validates segments, and maps each segment
-- `transcript.timestamps`
+into a generic source unit with a self-reference plus speaker and timestamp
+metadata. It accepts no module options. Its spec advertises the transcript
+capabilities consumed by D&D modules.
 
-External JSON shape belongs in the Seriatim integration doc.
+Parsing is strict about required values and duplicate unit IDs but deliberately
+ignores unrelated Seriatim fields. The external format and derived-identity
+rules are defined in the
+[Seriatim contract](../integrations/seriatim.md).
 
-## `generic` Chunker
+## Chunkers
 
-Package: `internal/modules/chunk/generic`
+Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
+unit ranges and may carry optional namespaced JSON annotations; it does not
+contain materialized chunk content. The framework canonicalizes annotations,
+validates ranges against the current source, and materializes chunk IDs,
+indexes, references, content, units, and generic metadata. Materialized source
+unit metadata is independently owned. Annotation
+namespaces remain optional data: generic framework code and downstream modules
+must not require D&D scene annotations or import `dnd/scenes`.
 
-The `generic` chunker splits source units into ordered chunks. It validates the
+### `internal/modules/generic/chunk/units`
-source document, clones source units, assigns chunk IDs such as `chunk-000001`,
-and records chunk metadata for start unit, end unit, and unit count.
 
-Options:
+The generic chunker validates the source document and returns ranges over units
+in configured windows. Overlap changes the next window start but never reorders
+units. Framework materialization derives the resulting chunk identity and
+generic metadata from those ranges.
 
-- `max_units`: positive integer, default `50`;
+The accepted options and defaults are defined in
-- `overlap_units`: non-negative integer, default `0`, and less than
+[Configuration](../config.md#implemented-production-modules). Generic
-  `max_units`.
+framework validation canonicalizes the returned unit slices before extraction.
+The chunker decodes its options during construction and retains only the typed
+window settings used by `Plan`.
 
-Provides:
+### `internal/modules/dnd/chunk/scenes`
 
-- `chunks`
+The scene chunker prepares a structured Scriptorium request from the full
+transcript, session, and optional D&D reference inputs. It validates the model's
+scene boundaries against source-unit IDs and converts them into deterministic
+plan ranges with optional scene annotations. Preparation injects the shared
+structured LLM client into the chunker; `Plan`
+supplies only the run-specific profile, session, source, references, and
+metadata.
 
-## `dnd/spells` Extractor
+Scene validation requires sequential, contiguous, non-overlapping coverage from
+the first source unit through the last. Scene descriptions, boundaries,
+confidence, and participants are module-owned annotations. Boundary caveats
+become warnings. Malformed
+structured output is returned as an error; there is no fallback chunker.
 
-Package: `internal/modules/extract/dnd/spells`
+The package embeds its prompt and response schema and reports their non-secret
+identity and hashes through singleton module metadata. Shared D&D assets supply
+reference declarations and prompt inputs; their user-facing keys and accepted
+file types remain canonical in [Configuration](../config.md).
 
-The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It renders
+## Extractor
-embedded prompts, loads the embedded structured response schema, calls the
-structured LLM client, converts spell-cast responses into artifact candidates,
-and supplies deterministic validators.
 
-Requires:
+### `internal/modules/dnd/extract/spells`
 
-- `chunks`
+The spell extractor prepares a structured request from one chunk, the
-- `source.transcript`
+chunk-scoped source input, the session, and optional D&D reference inputs. It
+decodes the model response, assigns the generic source identity to every source
+reference, canonicalizes duplicate references, orders spell casts by their
+earliest cited unit, and returns `dnd.SpellList`.
 
-Provides:
+The extractor owns its private model-response DTO, embedded prompt, LLM response
+schema, strict option decoder, injected shared LLM client, and prompt/schema
+manifest metadata. The separate `internal/modules/dnd/codec/spells` package
+owns the durable schema and stable JSON representation for artifact kind
+`dnd/spell-list`. The runner keeps the result typed through validators and later
+stages, using the codec only for checkpoint, debug, and output boundaries.
+Shared D&D helpers keep prompt input
+names and source-unit reference conversion consistent with the scene chunker.
 
-- `dnd.spell_casts`
+The durable payload and manifest metadata shapes are defined in the
-
-Artifact type and schema version:
-
-- artifact type: `dnd.spell_cast`
-- schema version: `v1`
-
-The extractor adds prompt and response-schema provenance to lane manifest
-metadata. Durable artifact payload details belong in the
 [D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
 
+## Merger And Normalizer
+
+### `internal/modules/generic/merge/appendorder`
+
+The merger passes typed values to an injected combine function in framework
+source-chunk order. The D&D registrar specializes it with a spell-list append
+function.
+
+### `internal/modules/generic/normalize/noop`
+
+The normalizer returns the merged domain value unchanged and is reusable for
+any registered artifact type.
+
+## Output Encoder
+
+### `internal/modules/generic/output/json`
+
+The JSON encoder sorts normalized results by lane, derives collision-checked
+safe logical names, pretty-prints JSON payloads, and assembles the logical index,
+manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
+media types, unsafe names, and sanitized-name collisions are errors.
+
+The encoder returns logical files only. The CLI places them on disk, and the
+[JSON output contract](../integrations/json-output.md) defines their external
+paths and schemas.
+
+## Generic Validators
+
+The generic validator implementations live under
+`internal/modules/generic/validate`.
+
+The unconditional accept and reject validators provide explicit chunk and
+typed-artifact variants used primarily for controlled composition and tests.
+
+The serialized JSON syntax validator uses `encoding/json` to reject malformed
+representation bytes. The serialized JSON Schema validator requires schema
+bytes, parses the instance and schema with `jsonschema`, and distinguishes
+payload rejection from schema loading or compilation errors. The framework
+serialized-validation request carries either canonical chunk bytes or artifact
+codec bytes according to its target context. Neither validator calls the LLM.
+
 ## D&D Spell Validators
 
-The spell extractor returns two built-in validators:
+All three validators receive `dnd.SpellList` directly. The shape validator
+rejects missing or empty spell fields and empty reference lists. The
+source-reference validator applies generic source-reference validation to every
+cited range. The relatedness validator warns when a case-insensitive spell name
+is absent from all cited source text.
 
-- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
+These validators are deterministic. Their selectable keys and production order
-- `dnd/spells/source_refs`: rejects candidates without valid source references.
+are defined in
-
+[Configuration](../config.md#implemented-production-validators); their durable
-Reason codes include:
+payload rules are defined in the
-
+[artifact contract](../integrations/dnd-spell-artifacts.md).
-- `invalid_payload`
-- `missing_required_field`
-- `missing_source_ref`
-- `invalid_source_ref`
-
-These validators are supplied by the extractor when no validators are configured
-for the lane.
-
-## `appendorder` Merger
-
-Package: `internal/modules/merge/appendorder`
-
-The `appendorder` merger clones and appends candidates in chunk order. It does
-not deduplicate or reconcile candidates.
-
-Provides:
-
-- `merged`
-
-## `noop` Normalizer
-
-Package: `internal/modules/normalize/noop`
-
-The `noop` normalizer clones merged candidates and returns them unchanged.
-
-Requires:
-
-- `merged`
-
-Provides:
-
-- `normalized`
-
-## `json` Output
-
-Package: `internal/modules/output/json`
-
-The `json` output encoder converts approved artifacts, rejected artifacts,
-warnings, and the run manifest into logical JSON output files. It groups
-approved artifacts by artifact type and sanitizes artifact-type file names.
-
-Requires:
-
-- `normalized`
-
-Provides:
-
-- `encoded`
-
-Durable output file shapes belong in the
-[JSON output contract](../integrations/json-output.md). Operator behavior
-belongs in [Operations](../operations.md).
 
 ## Production Registration
 
-Production registration is centralized in `internal/cli/catalog.go`.
+Production composition occurs through family registrars. The CLI allocates one
+complete framework registry set and one LLM asset registry. It invokes
+`internal/modules/generic/register`,
+`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
+that order, then exposes the matching catalog for resolution. The generic and
+Seriatim registrars own their production leaf registrations. The D&D registrar
+owns D&D leaf registrations, the spell default-validator chain, and D&D
+prompt/schema asset collection.
 
-Do not make framework code import production modules. The CLI wires production
+Concrete implementation packages do not import generic implementation
-modules at the application boundary; tests may provide fake registries or fake
+packages directly. A concrete family's `register` package is its composition
-catalogs directly.
+point for specializing reusable generic implementations, while the generic
+registrar composes only generic children.
 
-## Adding A Module
+Core and framework production packages do not import production extensions.
+CLI production code is the sole application composition root for extensions
+and imports only exact family registrar packages. Other production packages,
+including commands and newly introduced package trees, do not import module
+packages directly. Compatibility tests in the CLI, core, and framework trees
+may import roots and implementation leaves directly. Other non-module tests do
+not receive that exemption. White-box tests within module families retain the
+production family boundaries. `internal/modules/integration` is test
+infrastructure: its black-box tests may compose multiple families, but it is
+not a production module family or production dependency target.
 
-When adding a module, keep source-format and extraction-domain boundaries clear:
+## Adding An Extension
 
-- input modules may know external source formats;
+When adding a production module or validator:
-- extract modules may know artifact semantics and prompt/schema assets;
-- merge and normalize modules own candidate combination and reconciliation;
-- output modules own serialization, not diagnostics or CLI reporting.
 
-Update [Development](../policy/development.md), [Configuration](../config.md),
+1. implement the stage or validator contract and package-local key;
-internal docs, integration docs, and examples when the new module becomes
+2. expose and test its spec, constructor, and registration function;
-implemented production behavior.
+3. keep format or domain parsing inside the concrete package;
+4. add package-owned prompt/schema assets when the extension is LLM-backed;
+5. register it through its package-family registrar and add a default chain
+   there only when production policy requires one;
+6. add resolution and composition coverage for capabilities, options,
+   references, and validation behavior;
+7. update the selectable-key catalog in [Configuration](../config.md), the
+   relevant external contract, this inventory, and maintained examples when
+   user-visible behavior changes.
+
+Do not add the extension to `docs/development.md`; that file routes by task and
+does not inventory implementations.
+
+## Tests To Inspect
+
+- Package-local `*_test.go` files under the module or validator being changed.
+- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
+  and heterogeneous artifact composition.
+- `internal/framework/pipeline/default_modules_test.go`: framework binding
+  defaults.
+- `internal/cli/run_test.go`: production catalog, config resolution, and
+  end-to-end CLI composition.
+- `internal/framework/promptfs/*_test.go` and
+  `internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
+- `internal/modules/integration/*_test.go`: black-box composition across
+  production extension domains.

docs/internal/overview.md

@@ -1,86 +1,142 @@
 # Internal Overview
 
-This directory documents implemented Notarius internals for developers and LLM
+This document inventories the implemented Notarius components. Normative
-coding agents. It complements [Architecture](../policy/architecture.md), which
+boundaries and dependency direction belong in
-is the durable policy for boundaries and invariants.
+[Architecture](../policy/architecture.md); external behavior belongs in the
+[CLI](../cli.md), [Configuration](../config.md),
+[Operations](../operations.md), and [integration contracts](../integrations/).
 
-## Executable And CLI
+## Execution Path
 
-`cmd/notarius` calls the CLI package. `internal/cli` owns:
+`cmd/notarius` delegates to `internal/cli`, the production composition root.
+The CLI loads configuration, builds the production catalogs and runtime
+collaborators, invokes `internal/framework/pipeline`, and places the logical
+output files returned by the runner. Cache and debug collaborators are supplied
+at this boundary.
 
-- command parsing and usage;
+Resolution produces a fixed ordered workflow and a sorted set of artifact
-- config discovery and loading;
+lanes. Preparation constructs the complete module and validator set before the
-- production module catalog and registry wiring;
+runner receives source bytes. Source parsing and chunking are serial; extraction
-- production LLM client construction;
+uses a bounded run-wide worker pool, followed by serial per-lane merge and
-- run directory creation;
+normalize continuations that may overlap across lanes.
-- durable output writes;
-- user-facing stdout, stderr, and exit codes.
 
-The CLI should stay thin around framework contracts. Domain extraction behavior
+## Application Boundary
-belongs in modules, not in command handlers.
+
+| Package | Implemented responsibility |
+| --- | --- |
+| `cmd/notarius` | Executable entry point and process exit delegation. |
+| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. |
 
 ## Core Packages
 
-- `internal/core/artifacts`: artifact candidates, approved artifacts, rejected
+| Package | Implemented responsibility |
-  artifacts, validation decisions, and run manifests.
+| --- | --- |
-- `internal/core/config`: defaults, YAML config parsing, environment overrides,
+| `internal/core/artifacts` | Run-manifest and provenance models. |
-  validation, redaction, and resolved pipeline config.
+| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
-- `internal/core/diagnostics`: per-run diagnostics directory creation,
+| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. |
-  diagnostics artifact writers, atomic writes, and retention decisions.
+| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. |
-- `internal/core/source`: source documents, source units, source references, and
+| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |
-  validation.
-
-Core packages should remain deterministic and concrete. They should not import
-production modules.
 
 ## Framework Packages
 
-- `internal/framework/contracts`: interfaces and request/result structs for
+| Package | Implemented responsibility |
-  input adapters, chunkers, extractors, mergers, normalizers, validators, output
+| --- | --- |
-  encoders, and structured LLM clients.
+| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
-- `internal/framework/pipeline`: module registries, module specs, profile
+| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
-  resolution, capability checks, run orchestration, warnings, validation, and
+| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
-  manifest population.
+| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
-- `internal/framework/llm`: OpenAI-compatible structured-output client,
+| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
-  scheduler, schema registry, retries, and secret redaction.
+| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
-- `internal/framework/prompt`: embedded prompt registry and template rendering.
+| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
-- `internal/framework/validate`: validator decision helpers and cardinality
+| `internal/framework/debug` | Root-based framework and LLM debug recording. |
-  enforcement.
 
-Framework code should stay source-agnostic and domain-agnostic.
+Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
+serialized-validator, and
+typed-validator interfaces. The runner owns handoff provenance, validation
+sequencing, rejection handling, checkpoint and debug boundaries, and final
+manifest assembly.
 
-## Module Packages
+Artifact registries support heterogeneous typed extraction entries and
+kind-specific merger, normalizer, and validator variants. Resolution derives a
+lane's kind from its extractor, requires the matching codec, verifies exact Go
+type equality across the lane, and records schema identity in the resolved lane
+and pipeline digest. Registry entries carry separate option-validation and
+run-local construction closures. Preparation injects shared dependencies and
+constructs input, chunk, validators, ordered lanes, and output before source
+parsing. Production modules use strict construction-time option decoding, and
+LLM-backed modules retain the injected shared client. The D&D family registers
+the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
+kind-specific generic merge and normalize strategies; generic JSON validators
+use the serialized-validation contract. The runner executes lanes through
+private exact-type-checked closures, coordinates extract results independently
+of completion timing, and serializes artifacts only through their codec at
+checkpoint, debug, and output boundaries.
 
-Production module packages live under `internal/modules`:
+## Production Extensions
 
-- `input/seriatim`
+The canonical catalogs of user-selectable
-- `chunk/generic`
+[module](../config.md#implemented-production-modules) and
-- `extract/dnd/spells`
+[validator](../config.md#implemented-production-validators) keys are in
-- `merge/appendorder`
+Configuration. The implemented module packages are:
-- `normalize/noop`
-- `output/json`
 
-Each module package owns its contract implementation, module spec,
+| Package | Implemented responsibility |
-registration, options, focused tests, and module-specific errors.
+| --- | --- |
+| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
+| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
+| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
+| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
+| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
+| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
+| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
+| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
+| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
 
-## Fixtures And Tests
+`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
+reference declarations, prompt input assembly, and source-unit reference
+helpers. Domain-neutral prompt filesystem composition lives in
+`internal/framework/promptfs`.
 
-The repository uses focused package tests plus a fixture-driven CLI workflow.
+Generic validators under `internal/modules/generic/validate` provide
+unconditional test decisions, JSON syntax validation, and JSON Schema
+validation. D&D spell validators under `internal/modules/dnd/validate/spells`
+consume the canonical spell-list type directly to provide shape,
+source-reference, and source-relatedness decisions.
 
-- CLI acceptance tests cover maintained examples under `examples/`.
+Production composition is grouped behind package-family registrars, and every
-- Pipeline tests cover registry composition and end-to-end framework behavior
+implemented production extension uses its domain-first tree:
-  with fakes.
-- Module tests cover implemented module contracts without requiring real
-  provider calls.
-- LLM tests use local test servers and fakes.
 
-Do not use real external services in tests. Use fakes, fixtures, or local test
+| Package | Implemented responsibility |
-servers.
+| --- | --- |
+| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
+| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
+| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
 
-## Boundary Reminders
+The CLI allocates the framework registries and asset registry, then invokes
+these registrars in generic, Seriatim, and D&D order.
 
-- Source-format details stay in input modules and integration docs.
+Implementation details for all production extensions are in
-- Extraction-domain details stay in extract modules and artifact docs.
+[Module Internals](modules.md).
-- Provider wire details stay in the LLM runtime and provider integration docs.
+
-- Durable output contracts belong in integration docs.
+## Run-State Components
-- Operator procedures belong in `docs/operations.md`, not internal docs.
+
+| Surface | Implemented owners | Internal purpose |
+| --- | --- | --- |
+| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
+| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. |
+| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
+| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
+
+Physical layout, cleanup, recovery, and sensitive-data handling are defined
+in [Operations](../operations.md). Concrete stage modules receive recorder
+interfaces and request data, not physical state roots.
+
+## Focused Documentation
+
+- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
+  checkpoint/debug hooks, and result assembly.
+- [Module Internals](modules.md): production modules, validators, assets,
+  registration, and the contributor recipe for adding an extension.
+- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
+  assets, scheduling, profile recording, and redaction.
+- [Run State Internals](state.md): output, cache, debug collaborator
+  composition, and path safety.

docs/internal/pipeline.md

@@ -1,127 +1,330 @@
 # Pipeline Internals
 
-The implemented pipeline runner lives in `internal/framework/pipeline`. It
+The implemented resolver and runner live in `internal/framework/pipeline`.
-executes the fixed workflow defined by the architecture policy:
+Their fixed workflow and ownership boundaries are defined by
+[Architecture](../policy/architecture.md#system-shape). Configuration fields,
+defaults, and selectable keys are defined in
+[Configuration](../config.md#pipelines).
 
-```text
+Resolution fixes the selected lanes and all stage bindings; preparation
-input -> chunk -> extract -> merge -> normalize -> output
+constructs every selected implementation before the runner begins source work.
-```
+After serial input parsing and plan selection or generation, the runner
+materializes chunks and dispatches extract work to
+one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
+merge and normalize operations remain serial and may overlap other lanes once
+all extracts for that lane are terminal.
 
-Pipeline execution is serial. The runner executes the resolved lanes one after
+## Resolution
-another in the fixed workflow order.
 
-## Profile Resolution
+`internal/core/config.Config.Resolve` validates the loaded configuration,
+selects the named profile, applies the runtime inputs supplied by the CLI, and
+calls `pipeline.ResolvePipeline`.
 
-Config loading produces `pipeline.PipelineProfile` values. Resolution happens
+`ResolvePipeline`:
-before execution:
 
-1. `internal/core/config.Config.Resolve` validates config and finds the named
+1. selects and sorts artifact lanes;
-   pipeline.
+2. completes omitted bindings using the documented configuration defaults;
-2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
+3. looks up each module and validator spec without constructing it;
-3. Module bindings are defaulted:
+4. for a typed extractor, derives its artifact kind, requires the codec, and
-   - chunk: `generic`
+   selects exact-type merger, normalizer, and validator variants;
-   - merge: `appendorder`
+5. checks required and provided capabilities in workflow order;
-   - normalize: `noop`
+6. resolves target-aware reference bindings and validator chains;
-   - output: `json`
+7. validates each selected module and validator option set through its registry
-   - LLM profile: `default`
+   entry; and
-4. The module catalog is checked for each bound module key.
+8. calculates a digest over the resolved structure, including typed artifact
-5. Module capabilities are checked in workflow order.
+   kind and schema identity and the effective validator policy in its resolved
-6. A digest is calculated from the resolved pipeline without the digest field.
+   execution order.
 
-The CLI writes the resolved pipeline and digest to diagnostics.
+Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
+bindings, validator chains, reference targets, and the digest. It does not read
+reference bytes or construct runtime modules. CLI lane and reference selector
+syntax is defined in the [CLI reference](../cli.md#run).
 
-## Registries And Module Specs
+The digest includes each resolved validator chain's stage, lane, owning module,
+ordered validator bindings, execution classes, targets, and artifact kinds.
+Changing a default chain or an explicit override therefore changes pipeline
+identity whenever it changes the effective validator policy.
 
-`pipeline.Registries` holds concrete constructors for execution. A
+## Reference Materialization
-`pipeline.ModuleCatalog` exposes module specs for config validation and
-resolution.
 
-Every production module registers a `ModuleSpec` with:
+The CLI calls `MaterializeReferences` after resolution and before constructing
+the LLM client or running the pipeline. The materializer checks each binding
+against its resolved target declaration, reads and validates the file, and
+builds both a `contracts.ReferenceSet` and provenance-only metadata on the
+corresponding `ResolvedReferenceTarget`.
 
-- `Key`: module key used in config;
+The runner clones the resulting set into the chunk, extract, merge, or normalize
-- `Stage`: module kind such as input, chunk, extract, merge, normalize,
+request that owns the target. LLM-backed extensions may convert those items into
-  validate, or output;
+named prompt inputs. Reference content remains separate from source evidence and
-- `Provides`: capabilities added after that module runs;
+source digests.
-- `Requires`: capabilities that must already be available.
 
-Capability checks prevent incompatible pipeline composition before a run starts.
+Binding precedence, path resolution, accepted content, and media-type behavior
+are configuration contracts; see [Configuration](../config.md#pipelines).
+Durable provenance is defined in the
+[JSON output contract](../integrations/json-output.md#manifestjson), while
+runtime sensitive-data handling belongs in [Operations](../operations.md).
 
-## Runner Input And Output
+## Registries And Specs
 
-`pipeline.RunInput` carries:
+`pipeline.Registries` holds option validators and run-local builders used during
+resolution and preparation.
+`pipeline.ModuleCatalog` exposes their specs during configuration validation and
+resolution. Separate registries exist for every stage and for validators;
+`ValidatorChainRegistry` stores production default-chain mappings. Both
+containers also carry an `ArtifactCodecRegistry`. Generic registration records
+one codec per stable artifact kind, validates its schema metadata and JSON
+Schema, retains the exact schema digest and Go type, and safely encodes or
+decodes framework-erased values with typed errors on incompatibility.
 
-- a `ResolvedPipeline`;
+Typed extractor entries are keyed by module key and declare one artifact kind.
-- optional source ID, input path, and raw input bytes;
+Merger, normalizer, and typed-validator variants are keyed by module or
-- a structured LLM client;
+validator key plus artifact kind. Chunk and serialized validators occupy
-- run ID, start time, LLM profile manifest metadata, and CLI metadata.
+separate target namespaces; serialized registrations declare whether they
+support chunks, artifacts, or both. Duplicate variants and exact Go-type
+mismatches are rejected deterministically.
 
-`pipeline.RunOutput` carries:
+Lane-sensitive merger and normalizer spec discovery always supplies the
+extractor's artifact kind, so variants under one reusable key may declare
+different capabilities and reference slots. Kind-neutral registry inspection
+selects the first registered artifact kind in sorted order.
 
-- run manifest;
+Production composition registers the D&D spell-list codec and typed extractor,
-- approved artifacts;
+matching typed merge, normalize, and semantic-validator variants, and
-- rejected artifacts;
+serialized JSON validators. Every artifact lane resolves through the typed
-- warnings;
+registries and a matching codec.
-- logical output files returned by the output encoder.
 
-The CLI owns durable file writes and diagnostics writes after the runner returns.
+A `ModuleSpec` declares its stage plus required and provided capabilities.
+Chunk, extract, merge, and normalize specs may also declare reference slots.
+Registry implementations defensively copy spec metadata, reject duplicate keys,
+and verify that a constructed implementation reports the registered key.
+Builder registrations accept `ModuleDependencies` and cloned configuration
+options through one `BuildRequest`. Builders decode those options and retain
+typed values or injected dependencies in the constructed implementation.
+Extractors declare their artifact kind, and merger, normalizer, and validator
+resolution selects the matching typed variant.
 
-## Execution
+A `ValidatorSpec` declares a validator key and execution class. Resolution uses
+the execution class to reject incompatible profile bindings before execution.
+The current production catalog and default chain are listed only in
+[Configuration](../config.md#implemented-production-validators).
+
+## Preparation And Runner Boundary
+
+`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
+module dependencies. It constructs input; chunk and its validators; each lane's
+extract, merge, and normalize modules and validator chains in resolved order;
+then output. It stops at the first error with pipeline, stage, lane, module, and
+validator context as applicable. It never invokes an operation method.
+
+`PreparedPipeline` keeps private constructed executors and exposes cloned
+resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
+that prepared pipeline, raw source input, run identity and timing, optional
+session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
+collaborators. The runner
+parses source bytes through the already constructed input adapter. Later stage
+requests receive the generic source model; extract requests receive
+chunk-scoped input material, while chunk, merge, and normalize requests retain
+access to the original source material. Input, chunk, and output operation
+requests do not carry raw module options. The chunk request also does not carry
+an LLM client; an LLM-backed chunker receives the shared client during
+preparation. Their operation requests retain run-specific source, reference,
+profile, session, and metadata context as applicable.
+
+Prepared lanes retain exact-type-checked erased operation closures. The runner
+uses those closures to keep each value typed through extraction, validation,
+merge, and normalization.
+
+Source validation requires every unit to carry a canonical self-reference to
+its containing document and its own unit ID. Explicit clone, checkpoint, and
+debug boundaries retain that reference, and the canonical source digest covers
+it deterministically. Chunks use the same source model and carry one canonical
+reference spanning the first selected unit through the last.
+
+`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
+artifacts with lane and normalizer provenance,
+rejected results, warnings, checkpoint events, and logical files returned by the
+output encoder. The CLI owns debug-summary and durable filesystem writes after
+the runner returns.
+
+## Execution Flow
 
 The runner:
 
-1. validates run input and registries;
+1. validates its prepared input;
-2. builds the input adapter and parses the raw input into a source document;
+2. parses the raw input with the prepared adapter and validates the generic
-3. validates the source document;
+   source document;
-4. builds the chunker and produces source chunks;
+3. selects a stored plan or executes the configured chunker's `Plan` operation;
-5. runs each selected artifact lane in sorted resolved order;
+4. canonicalizes and materializes the plan, then validates the resulting
-6. builds the output encoder and validates logical output file names.
+   chunks;
+5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
+   bounded lane continuation when all extracts for that lane are terminal;
+6. invokes the prepared output encoder and validates its logical file results;
+7. returns the assembled manifest, outcomes, warnings, and files.
 
-Within an artifact lane, the runner:
+Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
+and validators while performing these transitions:
 
-1. builds the extractor, merger, and normalizer;
+1. extract once per accepted chunk and add runner-owned lane, source, and chunk
-2. records module manifest metadata when modules provide it;
+   provenance;
-3. extracts candidates from each chunk;
+2. validate each extract result and omit rejected results from merge input;
-4. normalizes candidate envelope fields such as index, extractor key, artifact
+3. skip the rest of the lane when no extract result is accepted;
-   type, and schema version;
+4. merge accepted extract results in their existing order;
-5. merges candidates;
+5. validate the merge result and skip normalization on rejection;
-6. normalizes merged candidates;
+6. normalize the accepted merge result;
-7. validates candidate envelope consistency;
+7. validate and append the accepted normalized result.
-8. runs validators;
-9. converts approved candidates to artifacts.
 
-## Validators
+Module-provided warnings and payload warnings are promoted only from attempts
+whose results are accepted and used.
 
-If a lane declares validators in config, the runner builds those validators from
+## Chunk Plans And Reuse
-the validator registry. Otherwise it uses validators returned by the extractor.
 
-Each validator must return exactly one decision for each eligible candidate. The
+`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
-runner enforces decision cardinality with `internal/framework/validate`.
+ordered unit-ID ranges, and optional plan or range annotations. The framework
-Rejected candidates are removed before the next validator runs. Approved
+owns plan canonicalization and materialization. It creates the generic chunks
-candidates continue through the chain.
+and therefore owns their IDs, indexes, source references, JSON content, units,
+media type, and generic metadata. Plan and range annotations are independently
+owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
 
-The production CLI currently registers no standalone validator modules. The
+In `auto`, the runner looks up the source digest before invoking the chunker. A
-current D&D spell extractor supplies deterministic shape and source-reference
+valid hit is materialized and sent through the current run's configured chunk
-validators.
+validators; it does not invoke the chunk module, consume its retry budget, or
+make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
+generates a candidate. `refresh` generates without lookup; `bypass` generates
+without cache access. Generated plans are published only after the full chunk
+validator chain approves them. A validator rejection is a regular rejected
+pipeline outcome and never replaces a cached plan.
 
-## Warnings And Failures
+The store is source-addressed, not pipeline-addressed. Changes to pipeline
+configuration, requested chunker, options, references, lanes, validators, or
+LLM profile do not prevent a source-digest hit. The manifest records both the
+currently requested chunker and the effective plan producer. Cache state and
+paths are configured and operated outside the runner; see
+[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
 
-Warnings from chunking, extraction, merging, normalization, validation, and
+The extract job channel has the same capacity as the effective extract worker
-output encoding are accumulated in `RunOutput.Warnings`.
+count, so dispatch applies backpressure. A fixed continuation executor prevents
+ready or checkpoint-reused lanes from creating one goroutine each. Workers and
+continuations publish lane-local results; the coordinator is the only writer of
+aggregate output and merges those results in resolved lane and source-chunk
+order.
 
-Errors wrap the operation and module key or lane context. If execution fails
+## Plan Canonicalization And Chunk Materialization
-after a manifest exists, the returned manifest is marked `failed` and receives a
-completion timestamp.
 
-On successful execution, the manifest validation status is:
+Plan canonicalization requires canonical JSON annotations, a matching source
+digest, at least one range, existing ordered boundaries, and increasing range
+starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
+Materialization deterministically reconstructs each range from the current
+source document, deep-clones JSON-shaped source-unit metadata, and copies
+annotations without interpreting their namespaces. Materialized chunks and
+separate materializations do not share mutable unit metadata; unsupported or
+cyclic metadata fails materialization with context.
 
-- `approved` when no candidates were rejected;
+Before lane execution, generic chunk validation checks the materialized chunks'
-- `rejected` when at least one candidate was rejected.
+identities, order, source references, content, media type, units, and metadata.
+No chunk checkpoint participates in plan selection: plan storage is the only
+chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
+use materialized chunk digests as their dependencies.
 
-## Manifest Population
+## Validation And Retries
 
-The manifest records run ID, pipeline ID, pipeline digest, module keys, artifact
+Chunk, extract, merge, and normalize results pass through the resolved validator
-lanes, LLM profile metadata, source digest, validation status, and timing.
+chain for their stage and module. Chunk validators receive canonical chunks;
+typed validators receive the domain value; and serialized validators receive
+canonical chunk JSON or artifact codec bytes. Validators execute in resolved
+order and stop at the first error or rejection. An empty chain approves the
+result.
 
-Modules can add non-secret manifest metadata by implementing
+`runWithRetry` applies the effective retry policy around module execution and
-`contracts.ManifestMetadataProvider`. The D&D spell extractor uses this for
+its complete validation chain. A module or validator error becomes a framework
-prompt and response-schema provenance.
+error when attempts are exhausted. A rejection becomes a recorded
+`RejectedOutput` when attempts are exhausted. Cancellation stops retry
+processing immediately.
+
+Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
+from discarded attempts are not promoted. Configuration owns retry counts and
+validator overrides; see [Module Bindings](../config.md#module-bindings).
+
+## Checkpoint And Debug Hooks
+
+The runner depends on recorder and loader interfaces, using no-op
+implementations when collaborators are absent. Each checkpointed workflow
+boundary records a running, succeeded, or failed transition. Reuse decisions
+are consulted in workflow order and accepted payloads are cloned before
+entering the normal handoff path. Typed extract, merge, and normalize
+checkpoints store codec bytes with artifact kind, schema ID and version, exact
+schema digest, and media type. Reuse compares that identity with the prepared
+codec and decodes through the codec; missing identity, mismatches, corrupt
+bytes, and decode failures become explicit reuse misses and execute the step
+normally. Dependency fingerprints and debug content digests use the same stable
+codec bytes that cross those boundaries.
+
+Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
+boundaries. Every executed chunk, extract, merge, and normalize attempt writes
+one terminal envelope for acceptance, validator rejection, module or validator
+error, or applicable candidate or final serialization error. The envelope
+contains its attempt-local warnings, any available candidate and rejection,
+and terminal error text; failures before a candidate exists omit that payload.
+Only LLM calls made by the module operation belong to the module attempt.
+Validator calls retain independent scopes under `validate/` and are not
+duplicated into the module envelope. A failed terminal-envelope write is a
+non-retryable framework error and is joined with any primary attempt error.
+Debug data is never used as a checkpoint source. Typed artifact debug envelopes
+are domain-neutral, redact sensitive metadata and bytes through the common
+debug policy, and record codec identity plus schema and content digests.
+
+Merge and normalize attempts serialize their in-memory candidate with the
+codec's required candidate encoder before typed validation. Serialized
+validators and attempt debug use that candidate representation, which carries
+the codec media type and schema identity but is never checkpointed or passed
+downstream. Only a validator-approved value is encoded through the strict final
+codec and made eligible for a checkpoint or stage output.
+
+Checkpoint identity, physical layout, reuse behavior, and debug artifact
+handling are operator contracts in [Operations](../operations.md). Serialization
+and recorder implementation are inventoried in
+[Internal Overview](overview.md#run-state-components).
+
+## Results And Failures
+
+The runner owns manifest assembly and handoff summaries but not the durable JSON
+schema. It records resolved module and lane provenance, validator chains,
+source/reference identities, selected LLM profiles, normalized and rejected
+summaries, status, and timing. Serialized artifact content remains outside the manifest.
+Module metadata providers may add non-secret singleton or lane-scoped metadata.
+
+Execution errors include stage, module, lane, or validator context. Once a
+manifest exists, a failing run returns it with failed status and completion
+time. Successful status reflects whether any result was rejected. The
+durable manifest and logical file schemas are defined in the
+[JSON output contract](../integrations/json-output.md).
+
+On a framework failure, the runner cancels its derived context, stops submitting
+new extract work, drains started tasks, and skips the output encoder. Parent
+cancellation takes precedence. Otherwise context-cancellation fallout is
+discarded when a substantive error exists, and the primary error is selected by
+stage, resolved lane, and source chunk rather than completion time.
+
+## Tests To Inspect
+
+- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
+- `internal/framework/pipeline/profile_test.go`: selection, defaults,
+  capabilities, validator chains, and digest behavior.
+- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
+  metadata, registration, erasure safety, strict decoding, and cloning.
+- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
+  lane resolution and preparation, target-specific validators,
+  incompatibilities, ordering, and schema-sensitive pipeline identity.
+- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
+  continuations, reverse completion, stable errors, rejection, cancellation,
+  retries, and independent provider-call limits.
+- `internal/framework/pipeline/preparation_test.go`: option validation,
+  construction order, dependency failures, and the before-source-work boundary.
+- `internal/framework/pipeline/references_test.go`: target resolution and
+  materialization.
+- `internal/cli/run_test.go`: production stage transitions, retries, rejections,
+  warnings, debug hooks, manifests, and end-to-end composition.
+- `internal/modules/integration/*_test.go` and
+  `internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
+  composition across concrete module families.
+- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
+  collaborators.

docs/internal/state.md

@@ -0,0 +1,56 @@
+# Run State Internals
+
+This document describes the implementation collaborators behind output, cache,
+and debug state. User-visible fields belong in [Configuration](../config.md),
+and layouts and lifecycle belong in [Operations](../operations.md).
+
+## Composition
+
+`internal/cli` is the only physical-path composition root. It resolves the
+effective configuration, selects exact roots, allocates requested debug bundles,
+constructs cache collaborators, writes logical output files, and reports paths.
+Pipeline modules receive interfaces and request data, never output, cache, or
+debug roots.
+
+## Output And Cache
+
+The pipeline runner returns logical output files. The CLI places them beneath
+the selected output root with confined, atomic writes.
+
+`internal/framework/chunkplan` owns source-addressed plan storage, validation,
+and atomic publication. Its store is constructed only when the selected mode is
+not `bypass`.
+
+`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
+codecs, loader, and recorder. The CLI constructs both loader and recorder only
+for a `--resume` invocation. The serialized
+`workspace_schema_version` identifiers are frozen wire-compatibility fields;
+they do not describe a current public state surface.
+
+`internal/core/fileio` provides confined atomic file writes used by state
+collaborators. The chunk-plan store retains its stronger entry validation.
+
+## Debug Bundles
+
+`internal/core/debugbundle` allocates an explicitly requested per-run bundle
+with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
+resolution, run, warning, and failure artifacts. `internal/framework/debug`
+implements the pipeline-facing trace recorder under the trace root.
+
+The CLI allocates a bundle before pipeline resolution and treats requested
+summary or trace persistence failures as command failures. The pipeline's debug
+boundaries redact sensitive metadata and credential-shaped bytes while allowing
+application-owned trace material. Debug data is never a checkpoint source or
+cache input.
+
+## Tests To Inspect
+
+- `internal/cli/state_surfaces_test.go`: debug allocation and configuration
+  boundaries.
+- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
+  permissions, cleanup, and redaction.
+- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes.
+- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and
+  reuse.
+- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement,
+  publication, and permissions.

docs/operations.md

@@ -1,132 +1,170 @@
 # Operations
 
-This is the canonical reference for operating implemented Notarius runs.
+This is the canonical guide to operating Notarius filesystem state. Command
+syntax is in the [CLI reference](cli.md); field definitions and precedence are
+in [Configuration](config.md).
 
-## Normal Run
+## State Model
 
-A run reads one source file, resolves one configured pipeline, calls the
+Notarius uses three independent filesystem surfaces:
-configured OpenAI-compatible LLM profile, writes durable JSON output, and writes
-diagnostics for inspection.
 
-```sh
+- output is durable user data;
-go run ./cmd/notarius run dnd-session \
+- cache is reconstructible chunk-plan and checkpoint state; and
-  --config examples/dnd-spells.config.yml \
+- debug is explicitly requested inspection data.
-  --input examples/seriatim-minimal-transcript.json \
-  --output-dir ./notarius-output \
-  --diagnostics-dir /tmp/notarius
-```
 
-The command prints a success line with the pipeline ID, approved and rejected
+Choose separate roots and access controls for each surface. A normal run writes
-artifact counts, and the output path.
+durable output and may use the chunk-plan cache. It does not create checkpoint
+or debug state unless its invocation includes `--resume` or `--debug`.
 
-## Output Directory
+## Output
 
-Durable output is written to:
+Durable logical files are written under:
 
 ```text
 <output-root>/<run-id>/
 ```
 
-The default output root is `./notarius-output`. Use `--output-dir` to choose a
+Each output file is written atomically. Notarius never automatically removes
-different root.
+output. The [JSON output contract](integrations/json-output.md) owns the
+logical file names, schemas, and media types inside a run directory.
 
-The `json` output module writes these files:
+Remove an output run directory only after its consumer data is no longer
+needed. This is data deletion, not cache cleanup.
 
-- `index.json`: file index with paths to the manifest, artifact files,
+## Chunk-Plan Cache
-  rejected artifacts, and warnings.
-- `manifest.json`: run manifest with resolved pipeline provenance, module keys,
-  validation status, and timing.
-- `artifacts/<artifact-type>.json`: approved artifacts grouped by artifact
-  type. For the current D&D spell extractor, this includes
-  `artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved.
-- `rejected.json`: rejected candidates and validator decisions.
-- `warnings.json`: warnings reported by pipeline modules or the output encoder.
 
-Output writes are atomic per file. Logical output file names must be clean,
+Chunk plans are stored at:
-relative, slash-separated paths and must not contain `..`.
-
-## Diagnostics Directory
-
-Diagnostics are written under:
 
 ```text
-<diagnostics-work-dir>/<run-id>/
+<chunk-plan-root>/<source-sha256-hex>/plan.json
 ```
 
-The default diagnostics work directory is `/tmp/notarius`. It can be set with
+`auto` reuses a complete valid plan or regenerates missing or invalid state.
-`diagnostics.work_dir`, `NOTARIUS_WORK_DIR`, or `--diagnostics-dir`.
+`refresh` regenerates and atomically replaces a plan after chunk validation.
+`bypass` performs no plan-cache I/O and does not resolve or create the root.
+Plan selection is source-addressed and independent of checkpoint and debug
+roots.
 
-Implemented diagnostics artifacts:
+When its directory is empty in configuration, the root is
+`<os.UserCacheDir>/notarius/chunk-plans`. A configured directory is the exact
+root; no suffix is appended. Directories and files created by the store use
+`0700` and `0600` permissions on supported Unix systems. The configured root
+is a trust boundary: do not share it among mutually untrusted users.
 
-- `invocation.json`: command metadata such as operation, config path, input
+Remove an exact digest directory or the configured root only when accepting the
-  path, selected lanes, run ID, and pipeline digest when available.
+cost of recomputing plans and any chunk-stage work. Cache publication is atomic;
-- `effective-config.json`: resolved config with API keys redacted.
+there is no history, locking, garbage collection, or rollback facility.
-- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
-- `run-manifest.json`: the same run manifest written to durable output when it
-  is available.
-- `warnings.json`: warning list.
-- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
-- `error.log`: failure message, written after diagnostics directory creation
-  when a run fails.
 
-`source-document.json` is supported by the diagnostics writer but is not written
+For a Linux service account, provision a dedicated restrictive root such as:
-by the current CLI run workflow.
 
-## Retention
+```yaml
+cache:
+  chunk_plans:
+    directory: /var/cache/notarius/chunk-plans
+```
 
-Diagnostics retention is configured with `diagnostics.retention`,
+## Checkpoint Cache
-`NOTARIUS_DIAGNOSTICS_RETENTION`, or the default `auto`.
 
-- `auto`: keep failed runs and successful runs with warnings; remove successful
+Checkpoint state is used only by an invocation with `--resume`. That invocation
-  warning-free runs.
+loads compatible completed work and records checkpoints for work it executes.
-- `always`: keep every diagnostics run directory.
+Without `--resume`, Notarius neither resolves nor creates the checkpoint root,
-- `never`: remove successful run directories; failed runs are still retained.
+and neither loads nor records checkpoints.
 
-Unknown retention values are rejected during config validation.
+Checkpoints use the selected root and the existing identity hierarchy:
 
-## Failures
+```text
+<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...
+```
 
-Failures before diagnostics directory creation, such as a missing config file or
+An empty configured directory selects
-an unusable diagnostics work directory, are printed to stderr and may not have a
+`<os.UserCacheDir>/notarius/checkpoints`. The root is exact when configured.
-diagnostics run directory.
+Created directories and files use `0700` and `0600` permissions on supported
+Unix systems.
 
-Failures after diagnostics directory creation are printed to stderr and written
+Checkpoint payloads can contain source text, intermediate artifacts, metadata,
-to `error.log`. Depending on where the failure occurred, diagnostics may also
+warnings, and content digests. Treat them as sensitive derived application
-include invocation metadata, redacted effective config, resolved pipeline data,
+data. Compatible files from a former checkpoint root remain reusable when
-the run manifest, warnings, and a run report.
+`cache.checkpoints.directory` names that exact existing root. They are not
+moved, migrated, or deleted automatically. The frozen serialized identifier
+`workspace_schema_version` remains part of checkpoint compatibility; it is not
+a configuration setting.
 
-If durable output writing fails after the pipeline completes, diagnostics are
+For a Linux service account, independently provision:
-retained for inspection and may include `run-manifest.json`, `warnings.json`,
-`run-report.json`, and `error.log`.
 
-## Warnings
+```yaml
+cache:
+  checkpoints:
+    directory: /var/cache/notarius/checkpoints
+```
 
-A successful run with warnings exits with code `0`, prints a warning count to
+Remove an exact checkpoint identity directory or the configured root only when
-stderr, and writes warnings to durable output and diagnostics when retained.
+recomputation is acceptable.
 
-The run manifest `validation_status` indicates whether final artifacts were
+## Debug Bundles
-approved or rejected after validation.
+
+Only `notarius run --debug` enables debug collection. The selected root contains
+one retained bundle per invocation:
+
+```text
+<debug-root>/<run-id>/
+  summary/
+  trace/
+```
+
+`summary/` contains redacted invocation, effective-configuration, resolved
+pipeline and reference provenance, checkpoint and chunk-plan decisions, run
+manifest, warnings, report, and any available error text. It excludes raw
+source, references, annotations, prompts, model responses, credentials, and
+malformed cache bytes.
+
+`trace/` contains application-owned execution detail, including source and
+stage material, plans, chunks, validator attempts, prompts, model responses,
+timing, and serialized artifacts. It may retain application data omitted from
+output. Credentials, credential-shaped values, sensitive metadata, unrelated
+environment values, and unrelated filesystem content are not captured.
+
+Bundles inherit the sensitivity of the application data they capture. Their
+additional risk comes from copying and aggregating that data, so restrict
+access, avoid shared roots between untrusted users, and define retention outside
+Notarius. Created bundle directories use `0700` and files use `0600` on
+supported Unix systems.
+
+Notarius never automatically deletes a requested bundle. If allocation
+succeeds, its path is reported on success and failure. A requested summary or
+trace write failure makes the command fail, preserving whatever bundle data was
+already written for inspection.
+
+## Failures And Warnings
+
+Failures before debug allocation are reported on stderr without a bundle.
+Failures after allocation report the bundle path on stderr and write `error.log`
+when that summary write succeeds. An output-write failure leaves the allocated
+bundle in place. A successful run with warnings exits `0`, reports a warning
+count on stderr, and records warnings in durable output and any requested debug
+summary.
 
 ## Cleanup
 
-It is safe to remove specific old run directories after their output and
+Use exact paths for manual cleanup. Examples:
-diagnostics are no longer needed:
 
 ```sh
-rm -rf /tmp/notarius/run-1234567890
 rm -rf ./notarius-output/run-1234567890
+rm -rf /var/cache/notarius/chunk-plans/0123abcd
+rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab
+rm -rf ./notarius-debug/run-1234567890
 ```
 
-Use exact run-directory paths. Avoid broad cleanup commands against parent
+Avoid broad recursive cleanup against a parent root unless it is an explicit
-directories unless they are part of your own operational policy.
+operator policy. Output deletion is permanent user-data loss. Cache deletion is
+recoverable but can repeat expensive work. Debug deletion removes troubleshooting
+evidence and any retained application-data copy.
 
 ## Operational Limits
 
-There is no command to resume a failed run. Re-run `notarius run` after fixing
+Provider retries and timeouts are handled by Scriptorium according to the
-the cause.
+selected execution profile. Pipeline module retry settings are defined in
-
+[Configuration](config.md#module-bindings). Extract worker concurrency and
-Provider retries are limited to the OpenAI-compatible client retry behavior
+actual provider-call concurrency are separate limits; their fields and
-configured by the effective LLM profile. There is no separate CLI retry command.
+validation are defined in [Configuration](config.md#concurrency). Notarius
-
+writes local files only; remote storage and archive management are outside the
-Notarius writes local files only. Remote storage and archive management are not
+implemented CLI.
-part of the implemented CLI.

docs/policy/architecture.md

@@ -1,210 +1,227 @@
 # Architecture
 
-This document defines Notarius development policy. It is inward-facing:
+This document defines the intended high-level architecture of Notarius and the
-developers and LLM coding agents should use it to preserve the project's shape,
+invariants that changes must preserve. Implemented component details belong in
-boundaries, and invariants as the code evolves.
+[Internal Overview](../internal/overview.md) and its linked documents. The
+reasoning behind significant architectural choices belongs in
+[ADRs](../adr/).
 
-Keep this document concise. It should describe durable architectural rules, not
+## System Shape
-CLI syntax, configuration reference material, module catalogs, or roadmap items.
 
-## Project Shape
+Notarius is a small, dependency-light Go application for extracting structured
+artifacts from source material. It is a general extraction platform whose
+source formats, extraction domains, validation policies, LLM providers, and
+output formats are isolated behind explicit boundaries.
 
-Notarius is a small, explicit, dependency-light Go application for extracting
+The application has one fixed pipeline shape:
-structured artifacts from source material using modular pipeline stages.
-
-The application is contract-first but not abstraction-heavy. Add interfaces and
-extension points when they protect a real boundary:
-
-- external source formats;
-- pipeline stage modules;
-- validators;
-- LLM providers and runtime plumbing;
-- output schemas and embedded assets.
-
-Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow
-contracts that can be exercised by tests and real modules.
-
-## Core Invariants
-
-The framework must remain source-agnostic and domain-agnostic.
-
-Source-format details belong in input modules. Transcript-specific concepts such
-as segments, speakers, timestamps, and transcript schemas must not spread into
-runner, extractor, validator, or LLM framework code.
-
-Extraction-domain details belong in domain modules. D&D-specific concepts such
-as spells, NPCs, items, combat turns, and encounters must not spread into core
-source, runner, or LLM framework packages.
-
-Extracted facts should be grounded with source references. Source references
-should point to generic source units, not transcript-only structures. Framework
-code should preserve source-reference ranges exactly and should not merge or
-rewrite overlapping ranges unless a module explicitly owns that behavior.
-
-The application workflow is fixed:
 
 ```text
 input -> chunk -> extract -> merge -> normalize -> output
 ```
 
-These stages should remain explicit in the architecture. Chunking, merging, and
+Pipelines are configured compositions of this shape. They are not arbitrary
-normalization must not be hidden inside domain extractors when they represent
+DAGs or a general workflow language. Every stage remains explicit; general
-general pipeline behavior.
+chunking, merging, or normalization behavior must not be hidden inside an
+extractor.
 
-Pipelines are fixed-shape templates for this workflow, not arbitrary DAGs or a
+Input and chunking are pipeline-wide. Each selected artifact lane owns its
-general workflow language. Module selection should be configuration- and
+extract, merge, and normalize stages, and the output stage aggregates the run's
-registry-driven, not scattered through conditionals.
+lane outcomes.
 
-## Package Boundaries
+Notarius is contract-first without being abstraction-heavy. Interfaces and
+extension points should protect demonstrated boundaries. New abstraction is not
+itself an architectural goal.
 
-Prefer fewer, larger framework packages until a boundary proves itself through
+## Layers And Dependency Direction
-import direction, ownership, test seams, or substantial file size.
 
-Core packages should contain deterministic models and policy. Framework
+The application boundary is the composition root and may depend on concrete
-packages should contain reusable orchestration and provider plumbing. Concrete
+implementations. Domain-neutral model and framework layers provide reusable
-business logic should live under stage-oriented module packages:
+policy, contracts, and orchestration. Concrete input, pipeline, output, and
+validation extensions depend inward on those generic layers.
 
-```text
+Generic layers must not depend on production extensions. Concrete extensions
-internal/modules/input/...
+must not compose the application or take ownership of process behavior. The
-internal/modules/chunk/...
+current packages implementing these layers are inventoried in
-internal/modules/extract/...
+[Internal Overview](../internal/overview.md).
-internal/modules/merge/...
-internal/modules/normalize/...
-internal/modules/output/...
-```
 
-Use short, lowercase, idiomatic Go package names. Avoid package names that repeat
+The following dependency boundaries are mandatory:
-parent-stage context.
 
-Input modules translate external source formats into the core source model.
+- extractors and validators do not depend on concrete input adapters;
-They may know about external schema details, source-specific metadata, and
+- provider-specific types do not cross the LLM runtime boundary;
-format-specific validation rules. They should not own extraction-domain
+- external dependency types do not leak across internal package boundaries
-decisions.
+  unless that dependency is the package's explicit contract.
 
-Extract modules own artifact semantics, prompt usage, structured response schema
+Shared helpers may support demonstrated common needs, but must not move
-selection, validator defaults, and domain-specific interpretation. They should
+source-format or extraction-domain knowledge into generic framework packages.
-depend on framework contracts and core source/artifact types, not concrete input
+External dependencies require a clear correctness, security, interoperability,
-module packages.
+or complexity benefit.
 
-Merge modules combine extracted candidates. Normalize modules reconcile merged
+## Source And Domain Boundaries
-candidates for semantic consistency. Generic behavior may exist for simple
-artifact types, but domain-specific behavior belongs in modules for the relevant
-stage.
 
-Output modules serialize final artifacts and may report warnings out of band.
+Input modules translate external source formats into the generic source model.
-CLI, diagnostics, and reporting layers are responsible for surfacing those
+Format-specific schemas, fields, and validation remain with the input module
-warnings.
+and its integration contract.
+
+Framework stages operate on source documents, source units, and source
+references rather than format-specific structures. A source reference identifies
+an ordered range of generic source units. Framework code preserves those ranges
+and does not merge or rewrite them unless a stage module explicitly owns that
+behavior. Every source unit carries a validated self-reference to its containing
+document and its own unit ID.
+
+Extract modules own artifact semantics, prompt use, response schemas, and
+domain interpretation. Domain-specific concepts remain in the relevant module,
+validator, shared domain helper, and artifact contract.
+
+Typed artifact registrations declare one stable artifact kind and exact Go
+type from extraction through merge, normalization, and semantic validation.
+Pipeline resolution requires a compatible codec and matching kind-specific
+variants before a typed lane can be accepted. Framework-owned erasure remains
+private and must report type incompatibility as an error rather than a panic.
+
+Auxiliary references provide context or disambiguation. They are not source
+evidence and must not be converted into source references.
+
+## Pipeline Composition And Ownership
+
+Module selection is configuration- and registry-driven. The framework resolves
+named pipeline definitions, applies explicit defaults and runtime overrides,
+and verifies module availability and capabilities before execution. Structural
+pipeline choices must not be scattered through conditionals or hidden behind
+ad hoc command flags.
+
+Resolution validates every selected module and validator option set. A separate
+preparation boundary then constructs the complete input, chunk, lane,
+validation, and output implementation set in pipeline order. The runner accepts
+only that prepared set, so construction and dependency failures occur before
+source parsing or any other module operation.
+
+Stage ownership is explicit:
+
+- input modules convert external material into the generic source model;
+- chunk modules partition source material for extraction;
+- extract modules produce domain artifacts from chunks;
+- merge modules combine accepted extraction outputs;
+- normalize modules reconcile merged output;
+- output modules encode accepted results and run outcomes into logical files.
+
+Chunk modules produce source-addressed chunk plans rather than materialized
+chunks. The framework validates and materializes those plans into the generic
+chunk representation before chunk validation and lane execution. Plan reuse is
+therefore independent of the configured pipeline, module options, references,
+lanes, validators, and LLM profile: the canonical source digest selects the
+plan, while the current run still applies its configured chunk validators to
+the materialized chunks.
+
+The framework owns orchestration and handoff provenance. Modules return logical
+results and warnings; they do not own CLI reporting, physical output, cache, or
+debug roots, durable file placement, or checkpoint and debug lifecycle.
+
+After pipeline-wide chunking, extraction uses bounded framework concurrency.
+One run-wide worker pool receives chunk-scoped lane jobs in deterministic
+chunk-first, lane-second order. A lane may begin its merge and normalize
+continuation only after all of its extract jobs are terminal; that continuation
+remains serial within the lane, while bounded continuations for different lanes
+may overlap. The framework must not create unbounded goroutines per lane or
+chunk.
+
+Completion timing does not choose public ordering or errors. The coordinator
+orders accepted artifacts, warnings, rejections, checkpoint events, and
+framework errors by stable pipeline scope. Rejections do not cancel unrelated
+work. A framework error cancels derived work, prevents undispatched work from
+starting, waits for started work, and prevents output encoding.
 
 ## Validation
 
-Validators should be independently testable and composable.
+Validation is a framework-managed boundary around outputs from chunk, extract,
+merge, and normalize stages. Validators receive immutable stage output
+and make an explicit whole-output decision: approve, approve with warnings, or
+reject.
 
-Deterministic validators should run before LLM-backed validators when both are
+Typed artifact validators receive the domain value directly. Chunk validators
-present. Validator decision semantics should be explicit: each candidate
+receive source-zone chunks, while serialized validators receive immutable
-artifact evaluated by a validator should receive exactly one decision from that
+representation bytes and declared schema metadata. A validator registered for
-validator.
+one target or artifact kind cannot satisfy an incompatible selection.
 
-LLM-backed review belongs in module-owned validator chains, not in an implicit
+Rejection is a recorded pipeline outcome, not a framework execution error.
-global review phase. Extract and normalize modules may both use deterministic
+Validator execution failures are framework errors. Rejected output does not
-and LLM-backed validators.
+advance to the next stage.
 
-Shared validator runtime mechanics belong in framework code. Concrete validator
+Default validator chains are production composition policy and are registered
-behavior belongs in module or validator implementation packages.
+centrally by stage and module. Configuration may replace a stage-local default,
+including with an explicitly empty chain. Configured validator order is
+authoritative; the framework must not silently reorder it.
 
-## LLM Runtime
+## LLM Boundary
 
-LLM provider details belong behind transport-neutral framework contracts.
+Modules and validators use transport-neutral structured completion contracts.
+Provider request and response types, authentication, transport behavior, and
+provider error adaptation remain inside the LLM runtime.
 
-Provider-specific HTTP request and response types should stay inside the LLM
+The caller of the LLM owns prompt selection, prompt inputs, response schema,
-runtime package. Prompt construction should stay in extractors, validators, or
+and interpretation of structured output. Provider adapters do not own source-
-shared prompt helpers; provider adapters should not own domain prompt logic.
+or domain-specific prompt logic.
 
-Errors, diagnostics, reports, manifests, and redacted configuration must not
+LLM calls and other external operations accept cancellation and respect
-expose secrets.
+timeouts. Concurrency control belongs in shared runtime plumbing rather than in
+individual modules.
 
-## Configuration
+The application-wide LLM scheduler bounds actual provider calls independently
+of framework worker limits. Every LLM-backed module, retry, and validator uses
+the single injected scheduled client, including work performed by overlapping
+lanes.
 
-Configuration should make pipeline composition explicit and discoverable.
+## Configuration And Provenance
 
-Centralize configuration loading, precedence, defaults, and validation. Structural
+Configuration loading, precedence, defaults, environment overrides, redaction,
-pipeline choices should come from named pipeline definitions, not ad hoc command
+and validation are centralized. Named pipeline definitions make structural
-flags. Operational overrides may be handled separately when they do not obscure
+composition explicit and discoverable. Operational overrides are permitted
-the configured pipeline structure.
+when they do not obscure the configured pipeline structure.
 
-Module registries should expose module metadata and capabilities without
+Run preparation fails before stage execution when statically discoverable
-requiring module construction. Configuration validation should fail fast when a
+modules, capabilities, reference bindings, or explicitly selected profiles are
-pipeline binds incompatible or unknown modules.
+invalid or incompatible.
 
-Run manifests should record enough resolved pipeline provenance to make a run
+Run manifests record enough resolved pipeline, module, source, reference, and
-auditable after named configuration changes over time.
+LLM provenance to make a run auditable after configuration changes. Manifests
+record identities and summaries rather than secret or large payload content.
 
-## Dependencies
+## State, Output, And Safety
 
-Prefer the Go standard library where practical.
+Notarius exposes three filesystem surfaces with independent roots and
+lifecycle:
 
-Use external dependencies only when justified by correctness, security,
+- output is durable user data; output modules define logical files and the CLI
-interoperability, or substantial complexity reduction. Good reasons include
+  owns their placement;
-widely used file formats, complex validation behavior, or secure transport
+- cache is reconstructible state, with separate chunk-plan and checkpoint
-handling.
+  families; and
+- debug is explicitly requested inspection data, combining a redacted summary
+  with a detailed trace.
 
-Avoid dependencies for small conveniences. Do not let external dependency types
+Chunk plans are keyed only by canonical source digest. Checkpoints are used only
-leak across internal package boundaries unless the dependency is itself the
+for an invocation that explicitly requests resume. Debug is never a cache input
-explicit contract of that package.
+and is never created without an explicit request. Pipeline modules receive
+collaborator interfaces and never physical roots.
 
-## State, Files, and Safety
+Writes are atomic where practical. Paths for writes, moves, overwrites, and
+deletion must be narrow and explicit. Notarius never automatically deletes
+output or requested debug bundles; cache cleanup is explicit and recoverable.
 
-If the application writes durable state, writes should be atomic where
+Secrets must not appear in errors, logs, output, cache, debug summaries,
-practical. Multi-step workflows should preserve enough diagnostics to support
+traces, manifests, documentation, examples, or redacted configuration. Debug
-inspection after failure.
+collection is allowlisted to application-owned payloads and must not capture
+unrelated process environment values or filesystem content. Trace data may
+contain application data and therefore inherits its sensitivity; operators own
+access controls and retention. Physical layout and operation are defined in
+[Operations](../operations.md).
 
-Code that deletes, moves, or overwrites files must use narrow, explicit paths.
+## Architectural Non-Goals
-Avoid broad parent-directory operations. Cleanup that can cause data loss must
-be opt-in.
 
-## Errors and Logging
+Notarius does not aim to provide:
 
-Errors should be actionable and preserve context. Wrap errors with operation and
+- an arbitrary workflow graph or general workflow language;
-path or resource context. CLI code should convert internal errors into concise
+- source-format or extraction-domain behavior in generic framework packages;
-user-facing messages.
+- provider-specific contracts exposed to modules;
-
+- structural pipeline composition through ad hoc CLI flags;
-Errors and logs must not expose secrets. Logs should describe operations,
+- implicit cross-stage behavior that bypasses the fixed pipeline;
-external calls, retries, and failure causes, but should not include large source
+- abstractions introduced solely for hypothetical future complexity.
-or artifact payloads by default.
-
-Long-running operations should accept `context.Context`. External calls,
-subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
-workflows should respect cancellation and timeouts.
-
-## Testing
-
-Core logic should be testable without real external services. Use fakes,
-fixtures, or local test doubles for input modules, extract modules, validators,
-and LLM clients where practical.
-
-Contract-first work should include fake implementations that prove interfaces
-compose before real modules depend on them.
-
-Maintain a fixture-driven walking skeleton that exercises the full pipeline with
-fake modules and fake external clients. This protects stage composition as real
-modules evolve.
-
-Important CLI and configuration workflows should have tests. Adapter, extractor,
-validator, and stage contracts should have focused tests that do not require
-running the full application unless end-to-end coverage is intentional.
-
-## Documentation
-
-Documentation should follow the project documentation policy. Keep user docs
-focused on implemented behavior. Put future, planned, or aspirational work only
-under `docs/roadmap/`.
-
-Core documentation should use generic terms such as source document, source
-unit, source reference, input adapter, extractor, chunker, merger, normalizer,
-artifact, validator, and run manifest.
-
-Source-format details belong in input module or integration docs.
-Domain-specific extraction details belong in extract module or artifact docs.
-
-When changing architecture, config, CLI behavior, stage modules, extractor
-contracts, validator contracts, LLM runtime behavior, or artifact schemas, update
-the relevant docs and examples in the same change.

docs/policy/development.md

@@ -1,139 +0,0 @@
-# Development
-
-This document defines contributor workflow for Notarius. For architectural
-invariants and package boundaries, read [Architecture](architecture.md) first.
-
-## Required Reading
-
-Before changing the repository, review:
-
-- [Architecture](architecture.md)
-- [Documentation Policy](documentation.md)
-
-Keep current-behavior documentation limited to implemented behavior. Put planned
-or deferred behavior under `docs/roadmap/`.
-
-## Repository Layout
-
-- `cmd/notarius`: executable entry point.
-- `internal/cli`: CLI parsing, production catalog wiring, config loading, run
-  command orchestration, output writes, and user-facing errors.
-- `internal/core`: deterministic models and policy for artifacts, source
-  documents, config, and diagnostics.
-- `internal/framework`: reusable contracts, pipeline orchestration, prompt
-  helpers, validation helpers, and LLM runtime plumbing.
-- `internal/modules`: concrete input, chunk, extract, merge, normalize, and
-  output modules.
-- `docs`: policy, user/operator docs, internal docs, integration docs, and
-  roadmap files.
-- `examples`: maintained, secret-free examples covered by tests where practical.
-
-## Validation Commands
-
-Run focused tests for the area changed, then run the broader checks when the
-change affects shared contracts, CLI behavior, or documentation examples.
-
-```sh
-go test ./...
-go vet ./...
-go build ./cmd/notarius
-```
-
-Useful focused checks:
-
-```sh
-go test ./internal/cli
-go test ./internal/core/config
-go test ./internal/framework/pipeline
-go test ./internal/framework/llm
-go test ./internal/modules/input/seriatim
-go test ./internal/modules/extract/dnd/spells
-go test ./internal/modules/output/json
-```
-
-## Go Conventions
-
-- Prefer the standard library unless a dependency is justified by correctness,
-  security, interoperability, or substantial complexity reduction.
-- Keep package names short, lowercase, and idiomatic.
-- Preserve import direction: framework and core code must not depend on concrete
-  production modules.
-- Use `context.Context` for long-running operations and external calls.
-- Return contextual errors that name the operation and relevant module, path, or
-  resource.
-- Do not include secrets in errors, logs, diagnostics, manifests, or docs.
-
-## Adding Config Fields
-
-Config behavior is centralized under `internal/core/config`.
-
-When adding a file config field:
-
-1. Update file config structs and YAML parsing in `file_config.go`.
-2. Apply the field over defaults in config application code.
-3. Add validation in `validation.go` when the field has constraints.
-4. Add environment override support in `env.go` only for operational overrides.
-5. Update redaction if the field can contain secrets.
-6. Add focused config tests.
-7. Update [Configuration](../config.md) and maintained examples when behavior
-   changes.
-
-Pipeline composition should remain config-driven. Do not add command flags that
-silently replace structural pipeline definitions.
-
-## Adding CLI Flags Or Commands
-
-CLI behavior lives in `internal/cli`.
-
-When adding CLI surface:
-
-1. Keep syntax explicit and update usage text.
-2. Validate arguments before running expensive work.
-3. Convert internal errors into concise user-facing messages.
-4. Add CLI tests for success, syntax errors, and failure modes.
-5. Update [CLI Reference](../cli.md), and update
-   [Operations](../operations.md) or [Troubleshooting](../troubleshooting.md)
-   if run behavior changes.
-
-## Adding Modules Or Adapters
-
-Concrete modules live under `internal/modules/<kind>/...` and implement the
-interfaces in `internal/framework/contracts`.
-
-For a new production module:
-
-1. Implement the relevant contract.
-2. Expose a `ModuleSpec` with the correct module key, module kind, provided
-   capabilities, and required capabilities.
-3. Expose a `Register` function that registers the module with its registry.
-4. Add focused module tests for contract behavior, registration, options,
-   validation, and errors.
-5. Register the module in `internal/cli/catalog.go` only when it is production
-   ready.
-6. Update internal docs and user-facing docs only for implemented behavior.
-
-Source-format behavior belongs in input modules and integration docs.
-Extraction-domain behavior belongs in extract modules and artifact docs.
-
-## Updating Examples
-
-Examples must be valid, secret-free, and small.
-
-- Prefer environment-based secret configuration.
-- Keep `examples/dnd-spells.config.yml` loadable by CLI tests.
-- Keep `examples/seriatim-minimal-transcript.json` compatible with the Seriatim
-  adapter.
-- Do not add expected-output fixtures unless they are validated or have a clear
-  regeneration procedure.
-
-## Documentation Updates
-
-Update docs in the same change when behavior changes.
-
-- CLI syntax: `docs/cli.md`
-- Config fields and defaults: `docs/config.md`
-- Output, diagnostics, retention, or recovery: `docs/operations.md`
-- Common user-facing failures: `docs/troubleshooting.md`
-- Internal architecture and contracts: `docs/internal/`
-- External file formats and durable integration contracts: `docs/integrations/`
-- Future or planned work only: `docs/roadmap/`

docs/policy/documentation.md

@@ -1,446 +1,143 @@
-# Go Project Documentation Policy
+# Documentation Policy
 
 ## Purpose
 
-Project documentation must help five audiences:
+This policy assigns each documentation topic to one canonical owner. Its goal is
-
+to keep Notarius documentation accurate, concise, discoverable, and resistant
-1. users who need to run the application;
+to drift for users, operators, developers, integrators, and LLM coding agents.
-2. administrators/operators who need to configure and operate it;
-3. developers who need to understand and change it safely;
-4. LLM coding agents that need clear scope, boundaries, and invariants;
-5. developers and LLM coding agents integrating this project from another codebase.
-
-Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
 
 ## Core Rules
 
-### 1. Keep docs concise
+### One Canonical Owner
 
-Each document should cover a defined scope and only the essentials for that scope.
+Each authoritative fact belongs in one document. A non-owning document may give
+a short, stable summary for orientation, but it must link to the canonical owner
+instead of repeating volatile details.
 
-Avoid:
+Volatile details include commands, flags, configuration fields and defaults,
-- long background explanations;
+module keys, schemas, file names, paths, status codes, retry behavior, and
-- repeated reference material;
+runtime guarantees. If readers could reasonably treat a statement as a
-- implementation detail in user-facing docs;
+contract, maintain it only in the owning document.
-- aspirational language outside roadmap docs;
-- verbose examples where one minimal example is clearer.
 
-### 2. Document only implemented behavior outside roadmap files
+### Current And Future Behavior
 
-Unimplemented, planned, aspirational, experimental, or future work may be described only under:
+Outside `docs/roadmap/`, documentation describes implemented behavior only.
+Partial features may be described only to their implemented boundary.
+
+ADRs are the narrow exception: an ADR may record an accepted architectural
+decision before implementation, but acceptance must not be presented as proof
+that the behavior exists. The roadmap owns implementation status and sequencing
+until the decision is implemented. Current architecture, user, operator,
+integration, and internal documentation are updated when the behavior lands.
 
-- `docs/roadmap/`
+### Audience And Detail
 
-No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
+Write for the document's stated audience and include only the detail needed for
+its owned topic. User and operator docs should not expose implementation detail.
+Developer docs should link to user-facing and external contracts rather than
+restate them.
 
-If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
+### Examples
 
-### 3. Use canonical homes
+Complete copyable files belong in `examples/`. Documentation may use the
-
+smallest illustrative snippet needed to explain its owned topic, but should link
-Each type of information should have one canonical location.
+to maintained examples instead of embedding a second complete copy.
-
+
-Canonical homes:
+Examples must be valid, secret-free, and tested where practical. Commands and
-
+configuration used in documentation should match the application.
-- project purpose and quickstart: `README.md`
+
-- development principles: `docs/policy/architecture.md`
+### Security And Privacy
-- public HTTP API reference: `docs/api.md`
+
-- configuration reference: `docs/config.md`
+Documentation and examples must not contain real credentials, private keys,
-- CLI reference: `docs/cli.md`
+private environment dumps, sensitive source material, or private infrastructure
-- operations and recovery: `docs/operations.md`
+details unless intentionally public. Document secret-handling mechanisms, not
-- troubleshooting: `docs/troubleshooting.md`
+secret values.
-- public API/package consumer guidance: `docs/consumers/`
+
-- implemented internals: `docs/internal/`
+## Canonical Ownership
-- external protocol, service, and file-format contracts: `docs/integrations/`
+
-- future work: `docs/roadmap/`
+| Topic | Canonical owner | Owned content | Content owned elsewhere |
-- contributor workflow: `docs/policy/development.md`
+| --- | --- | --- | --- |
-- copyable examples: `examples/`
+| Product orientation and minimal end-to-end quickstart | `README.md` | What Notarius is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
-
+| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
-Other files should summarize briefly and link to the canonical source.
+| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
-
+| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
-### 4. Keep examples real
+| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
-
+| Configuration contract | `docs/config.md` | Discovery and precedence, file schema, fields, defaults, environment overrides, validation rules, and user-selectable module or validator keys. | Complete example files, CLI syntax, runtime state lifecycle, module implementation details. |
-Examples should be valid, maintained, and free of secrets.
+| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. |
-
+| Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
-Where practical:
+| Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
-- example configs should load successfully;
+| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical output bundle paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
-- example commands should match real CLI syntax;
+| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
-- important examples should be covered by tests.
+| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
-
+| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
-## Documentation Profiles
+| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
-
+| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
-All projects require:
+
-
+Documents that do not exist are required only when the corresponding interface
-- `README.md`
+or responsibility exists. Do not create placeholder API, consumer, integration,
-- `docs/policy/architecture.md`
+or operations documents for behavior the application does not have.
-
+
-Additional docs depend on the project.
+## Boundary Rules
-
+
-### Small library
+### Orientation
-
+
-Recommended:
+The README owns product orientation. The developer guide routes contributors.
-- `docs/policy/development.md`, if contributor conventions are non-obvious
+Architecture owns normative structure. Internal overview owns the current
-
+concrete component map. These documents may link to one another but should not
-### Simple CLI
+maintain parallel package or behavior descriptions.
-
+
-Required:
+### Commands, Configuration, And Operations
-- `docs/cli.md`
+
-
+CLI documentation answers how to invoke the application. Configuration
-Recommended:
+documentation answers what settings mean. Operations answers what happens to
-- `docs/policy/development.md`
+runtime state and how to operate or recover the application. When a workflow
-
+crosses these topics, choose the document that owns the task and link to the
-### Config-driven CLI
+other contracts.
-
+
-Required:
+### Contracts And Implementation
-- `docs/cli.md`
+
-- `docs/config.md`
+Integration and API documents define externally observable shapes and
-
+semantics. Internal documents explain how Notarius implements or consumes those
-Recommended:
+contracts. Internal docs may name a field, file, or protocol to identify a
-- `examples/`
+dependency, but must link to its canonical contract for the definition.
-- `docs/policy/development.md`
+
-
+### Security Topics
-### Stateful or operator-facing application
+
-
+This policy owns what documentation and examples may contain. Architecture owns
-Required:
+application security invariants. Configuration owns credential-supply
-- `docs/cli.md`, if CLI-based
+mechanisms. Operations owns permissions and handling of sensitive runtime
-- `docs/config.md`, if config-driven
+artifacts. Internal docs own implementation mechanisms only.
-- `docs/operations.md`
+
-
+## Architecture Decision Records
-Recommended:
+
-- `docs/troubleshooting.md`
+Use sequentially numbered ADR filenames such as
-- `examples/`
+`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
-- `docs/policy/development.md`
+
-
+1. title;
-### Modular, service-oriented, or orchestration application
+2. status;
-
+3. date;
-Required:
+4. context;
-- `docs/cli.md`, if CLI-based
+5. decision;
-- `docs/config.md`, if config-driven
+6. alternatives considered;
-- `docs/operations.md`
+7. consequences.
-- `docs/internal/`
+
-- `docs/policy/development.md`
+Treat the decision content of an accepted ADR as immutable. When a decision
-
+changes, create a new ADR and update the earlier ADR's status to superseded.
-Recommended:
+Rejected architectural alternatives belong in the ADR; rejected product ideas
-- `docs/troubleshooting.md`
+belong in the roadmap.
-- validated examples under `examples/`
+
-
+## Maintenance
-### Public HTTP API service
+
-
+When behavior changes, update its canonical owner in the same change. If
-Required:
+ownership moves, remove the old definition and replace it with a link where
-- `docs/api.md`
+navigation remains useful.
-- `docs/cli.md`, if CLI-based
+
-- `docs/config.md`, if config-driven
+Before completing documentation work:
-- `docs/operations.md`
+
-- `docs/internal/`
+- verify affected behavior and examples;
-- `docs/policy/development.md`
+- check commands, flags, fields, defaults, schemas, and paths against their
-
+  implementation;
-Recommended:
+- keep unimplemented behavior in the roadmap, subject to the ADR exception;
-- `docs/troubleshooting.md`
+- remove stale references and validate links;
-- `docs/consumers/`, for task-oriented client integration guides
+- confirm that non-owning documents summarize and link rather than redefine;
-- `docs/integrations/`, for upstream/downstream service contracts
+- confirm that no secrets or sensitive private data were added.
-- validated examples under `examples/`
-
-### Project with public packages or consumer APIs
-
-Required:
-- `docs/consumers/api.md`
-- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
-
-Recommended:
-- copyable consumer examples under `examples/`, if practical
-
-## Required Documents
-
-### README.md
-
-**Audience:** users, administrators, operators
-
-The README is the outward-facing project orientation page.
-
-It should include, in order:
-
-1. concise description;
-2. elevator pitch;
-3. shortest useful command or usage example;
-4. links to targeted docs.
-
-The README should be short. It is not a manual.
-
-The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
-
-### docs/policy/architecture.md
-
-**Audience:** developers, LLM coding agents
-
-`docs/policy/architecture.md` is required for every project.
-
-It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
-
-It should include:
-
-- project shape;
-- core design principles;
-- package and boundary philosophy;
-- state/persistence philosophy, if applicable;
-- external integration philosophy, if applicable;
-- error-handling and logging principles;
-- testing expectations;
-- documentation expectations;
-- architectural invariants;
-- explicit non-goals, if useful.
-
-Notably, this file should prescribe a core development *policy* that should remain unchanged as the application evolves. It is not a place for details (e.g., CLI flags) that could change over time.
-
-The contents of `architecture.md` should be trim and concise. LLMs may be directed to review it routinely via AGENTS.md, CLAUDE.md, or similar.
-
-### docs/api.md
-
-**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
-
-Required for projects whose primary public interface is HTTP.
-
-`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
-
-It should include:
-
-1. base URL conventions;
-2. authentication and authorization behavior, if implemented;
-3. response envelope;
-4. supported media types and content negotiation behavior;
-5. shared query parameters;
-6. endpoint reference grouped by route family;
-7. request parameters and validation rules;
-8. response fields, units, nullability, and optionality;
-9. error response shape and status codes;
-10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
-11. compact request and response examples.
-
-It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
-
-For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
-
-### docs/policy/development.md
-
-**Audience:** developers, LLM coding agents
-
-Required for projects maintained by humans and LLM coding agents.
-
-It should include:
-
-- repository layout;
-- build/test commands;
-- coding conventions;
-- dependency policy;
-- how to add config fields;
-- how to add CLI flags;
-- how to add modules or adapters, if applicable;
-- how to update examples;
-- documentation update expectations.
-
-### docs/config.md
-
-**Audience:** administrators, operators, advanced users
-
-Required for applications with configuration files.
-
-It should include, in order:
-
-1. config file locations and discovery precedence;
-2. minimal working config;
-3. production-oriented config;
-4. full configuration reference;
-5. secrets handling, if applicable;
-6. links to maintained examples.
-
-The full configuration reference should be canonical.
-
-### docs/cli.md
-
-**Audience:** users, administrators, operators
-
-Required for CLI applications.
-
-It should include, in order:
-
-1. shortest useful command;
-2. command overview;
-3. complete flag reference;
-4. common workflows;
-5. diagnostic or recovery commands, if applicable.
-
-Explain when commands are useful, not just their syntax.
-
-### docs/operations.md
-
-**Audience:** administrators, operators
-
-Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
-
-It should cover:
-
-- normal workflow;
-- filesystem layout;
-- remote storage layout, if applicable;
-- logs and manifests;
-- resume/retry behavior;
-- cleanup behavior;
-- archive/backup behavior;
-- safe recovery procedures;
-- operational caveats.
-
-### docs/troubleshooting.md
-
-**Audience:** administrators, operators
-
-Recommended once recurring failure modes exist.
-
-Each entry should include:
-
-- symptom;
-- likely cause;
-- diagnostic command or inspection step;
-- safe fix;
-- relevant links.
-
-### docs/consumers/
-
-**Audience:** developers and LLM coding agents integrating this project from another codebase
-
-Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
-
-This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
-
-For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
-
-`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
-
-1. intended consumer audience and use cases;
-2. required inputs supplied by operators or deployment configuration;
-3. recommended public package or API workflow;
-4. minimal copyable example;
-5. consumer responsibilities and boundaries;
-6. retry, idempotency, or status behavior, if applicable;
-7. links to package-specific docs and canonical integration contracts.
-
-Package-specific docs should be named `pkg-<name>.md` and should include:
-
-1. import path;
-2. intended use cases;
-3. primary types and functions needed by consumers;
-4. minimal examples;
-5. validation, error, retry, and boundary behavior;
-6. links to canonical file-format or wire-protocol contracts.
-
-### docs/internal/
-
-**Audience:** developers, LLM coding agents
-
-Required for modular, service-oriented, or orchestration projects.
-
-This directory describes implemented internal components. It is not the roadmap.
-
-Use one file per major component where useful.
-
-Each component doc should include:
-
-1. purpose;
-2. inputs and outputs;
-3. boundaries;
-4. config fields used;
-5. external adapters used;
-6. state or manifest behavior, if applicable;
-7. skip/resume behavior, if applicable;
-8. failure behavior;
-9. tests to inspect before changing;
-10. architectural invariants.
-
-### docs/roadmap/
-
-**Audience:** maintainers, developers, LLM coding agents
-
-This is the only place for planned, future, aspirational, experimental, or unimplemented work.
-
-Roadmap docs should clearly distinguish:
-
-- proposed work;
-- accepted plans;
-- deferred ideas;
-- rejected ideas;
-- implementation prompts or task breakdowns, if useful.
-
-Roadmap docs should not be confused with current behavior.
-
-### docs/integrations/
-
-**Audience:** developers, LLM coding agents
-
-Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
-
-This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
-
-For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
-
-Use one file per integration where useful.
-
-## Examples Directory
-
-Projects with non-trivial configuration or workflows should include `examples/`.
-
-Useful examples include:
-
-- minimal working config;
-- production-oriented config;
-- full annotated config;
-- local development config;
-- remote/object-storage config;
-- minimal session/input file.
-
-Examples should be valid, maintained, tested when practical, and linked from relevant docs.
-
-## Security and Privacy
-
-Docs and examples must not include:
-
-- real API keys;
-- tokens;
-- passwords;
-- private keys;
-- private environment dumps;
-- sensitive user data;
-- raw private transcripts;
-- private infrastructure details unless intentionally public.
-
-Document secret-handling mechanisms, not actual secret values.
-
-## Maintenance Rules
-
-When docs change, verify the affected behavior.
-
-Where practical:
-
-- load example config files in tests;
-- test CLI examples or command parser behavior;
-- validate documented flags against real flags;
-- remove stale references;
-- update links after renames;
-- keep roadmap content out of non-roadmap docs.
-
-If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
-
-Documentation is complete only when it matches the current code.
-
-## Documentation Change Checklist
-
-Before merging documentation changes, verify:
-
-- README is concise and orientation-focused.
-- `docs/policy/architecture.md` describes development principles.
-- `docs/api.md` is the canonical HTTP contract for HTTP API services.
-- Future work appears only under `docs/roadmap/`.
-- User-facing docs avoid unnecessary internals.
-- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
-- Developer-facing docs preserve boundaries and invariants.
-- Config examples match the schema.
-- CLI examples match real commands and flags.
-- Defaults appear in the canonical config reference.
-- No secrets or private data are included.
-- Links are accurate.

docs/roadmap/future.md

@@ -0,0 +1,50 @@
+# Future Work
+
+Current Notarius behavior is documented in the canonical README, CLI,
+configuration, operations, internal, and integration docs. This roadmap records
+future work only.
+
+## Candidate Product Work
+
+- Additional input adapters, such as Markdown or note-export formats.
+- Additional D&D extractors beyond spell casts.
+- Add non-file reference producers, such as prior-run artifacts, derived
+  summaries, or entity registries, without changing module-facing reference
+  item contracts.
+- Add token budgeting and model context-window management for reference content.
+- Add per-slot or per-chunk inclusion policies so modules can avoid repeating
+  large reference content in every prompt when that becomes important.
+- Add structured or parsed references, such as typed roster schemas, when a
+  module has a clear need for more than opaque UTF-8 text.
+- Add reference caching, preprocessing, summarization, embedding, or retrieval
+  if references become large enough to require preprocessing.
+- Cross-lane entity normalization.
+- Cross-chunk semantic deduplication.
+- Additional validator packages and production default chains for future
+  modules.
+- Production LLM-backed validators when there is a concrete review policy that
+  benefits from model judgment.
+- Validator diagnostics and timing summaries if operators need more detail than
+  the current [durable output bundle](../integrations/json-output.md) provides.
+- Media-type validators for non-JSON module outputs when such modules are
+  introduced.
+- Validator compatibility metadata if real deployments need config-time
+  enforcement that a validator is suitable for a specific stage or module.
+- Batching or context-window controls for LLM-backed validators if validator
+  inputs become large enough to require them.
+- Additional output encoders.
+
+## Candidate Operational Work
+
+- Packaged release artifacts for alpha distribution.
+- A documented versioning and release process.
+- Optional generated example output fixtures with a regeneration procedure.
+- Additional diagnostics or reporting views if operator workflows need them.
+
+## Candidate Workspace Work
+
+- Default-idempotent run behavior with an explicit force override.
+- Remote workspace storage.
+- Workspace garbage collection.
+- Workspace archival policy.
+- Cross-machine checkpoint reuse.

docs/roadmap/implementation.md

@@ -0,0 +1,403 @@
+# ADR-0006 Findings 1–3 Remediation Plan
+
+This document is the executable implementation plan for correcting Findings 1,
+2, and 3 from the post-implementation review of ADR-0006:
+
+1. unredacted resolved-pipeline data in the debug summary;
+2. collision-prone output run directories; and
+3. missing failure `run-report.json` artifacts.
+
+Finding 4—the broad loss of CLI and configuration regression coverage—is
+explicitly deferred. The focused regression tests required to prove these three
+fixes are in scope; restoring or redesigning the general test suites is not.
+
+The audience is an LLM coding agent. Implement the stages in order. Each stage
+must finish with a compiling, tested repository.
+
+## Execution Rules
+
+Before every stage:
+
+1. read [Development](../development.md), both documents under `docs/policy/`,
+   [ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md), this plan,
+   and every stage-specific document listed below;
+2. inspect `git status` and preserve user changes;
+3. inspect the focused implementation and tests before editing; and
+4. confirm all earlier stages are complete.
+
+During every stage:
+
+- keep the CLI as the run-identity and physical-path composition root;
+- preserve durable logical output schemas, checkpoint identity and wire
+  compatibility, chunk-plan behavior, debug layout, exit codes, and current
+  configuration contracts unless this plan expressly changes them;
+- never allow a secondary debug persistence error to hide the primary command
+  error;
+- add focused tests for the changed behavior without expanding into Finding 4;
+  and
+- update a canonical current-behavior document only if the implemented contract
+  changes or its existing statement is inaccurate.
+
+At the end of every stage:
+
+1. run the focused tests listed for that stage;
+2. run `go test ./...`;
+3. run `go vet ./...`;
+4. run `go build ./cmd/notarius`;
+5. run `git diff --check`; and
+6. mark the stage complete here only after all checks pass.
+
+If repository reality conflicts with this plan, update the plan before making a
+materially different design choice. Do not silently weaken redaction, output
+exclusivity, or failure reporting.
+
+## Fixed Decisions
+
+### Redacted summary boundary
+
+- Every file under `summary/` is a redacted artifact. The fact that the adjacent
+  `trace/` may contain application data does not relax this boundary.
+- `resolved-pipeline.json` retains its current logical shape and useful module,
+  validator, digest, capability, and reference-provenance data, but all module
+  and validator option maps pass through the same recursive sensitive-key
+  redaction used by `effective-config.json`.
+- The debug-bundle writer must not accept an arbitrary resolved pipeline for a
+  method whose name promises redacted summary output. Introduce a narrow
+  payload interface, analogous to `RedactedSummaryPayload`, that can only be
+  satisfied by a producer responsible for redaction. `config.EffectiveConfig`
+  implements it and returns a deep-cloned, redacted
+  `pipeline.ResolvedPipeline`.
+- The CLI passes the effective configuration to that method, not
+  `effective.ResolvedPipeline` directly. Do not expose the private redaction
+  helper merely to keep the unsafe call shape.
+- Redaction must cover input, chunk, output, lane extract/merge/normalize,
+  resolved validator-chain bindings, deprecated lane validator bindings, and
+  recursively nested option maps and lists. Safe option values remain intact.
+- Reference contents remain excluded through the existing `json:"-"`
+  `ReferenceSet` contract. Reference provenance and configured paths may remain
+  in the summary.
+- Credentials and sensitive values must not be copied into errors while testing
+  this behavior.
+
+### Run identity and output allocation
+
+- The CLI owns one run ID before allocating debug or executing the pipeline.
+  Output, debug, manifests, metadata, stdout, and stderr use that same ID.
+- The default ID format becomes:
+
+  ```text
+  run-<started-at-unix-nanoseconds>-<32-lowercase-hex-characters>
+  ```
+
+  The suffix is 16 bytes from `crypto/rand`. Time preserves useful ordering;
+  randomness makes IDs collision-resistant across processes and hosts sharing a
+  root.
+- Add an injectable `RunIDGenerator func(time.Time) (string, error)` to
+  `internal/cli.Options`. The production default uses `crypto/rand.Reader`.
+  Tests inject deterministic valid IDs. Do not retain package-global clocks or
+  random sources for this path.
+- Generate the ID after configuration and CLI validation but before debug
+  allocation and pipeline resolution. A generation failure is a normal command
+  failure with exit code `1` and cannot have a debug bundle because no run
+  identity exists yet.
+- Change debug allocation to accept the CLI-generated run ID and `startedAt`.
+  It validates that the ID is one safe path component and creates that exact
+  bundle with `os.Mkdir`, failing on an existing entry. Debug no longer creates
+  a different ID from its own clock or retries by changing identity.
+- Durable output allocation is exclusive. Validate every logical output name
+  first, create the output parent as needed, then create
+  `<output-root>/<run-id>` with `os.Mkdir`, not `os.MkdirAll`. An existing run
+  directory is an error; Notarius never writes into it.
+- Nested logical output directories may still use `MkdirAll` after the exclusive
+  run directory is owned by the invocation. File writes remain atomic.
+- If output allocation or a later output write fails, retain any newly created
+  partial run directory for inspection. Never remove or overwrite an existing
+  run directory. Document this only if current Operations wording needs
+  clarification.
+- Keep output directory permissions and logical file permissions unchanged;
+  this remediation addresses identity and exclusivity, not the public output
+  permission contract.
+
+### Terminal debug reports
+
+- Every debug-enabled invocation whose bundle was successfully allocated makes
+  one best-effort attempt to write a terminal `run-report.json`, whether the
+  command succeeds or fails.
+- A failure report uses `Succeeded: false`. Populate fields only from state that
+  is known at the failure point:
+  - `run_id`, requested `pipeline_id`, and `debug_path` are always known after
+    bundle allocation;
+  - `output_path` is included once the effective output root and run ID are
+    known;
+  - output, rejection, warning, and validation fields are populated when a
+    `pipeline.RunOutput` is available; otherwise their zero values are retained.
+- `error.log` remains the human-readable primary error artifact. Do not add the
+  error string to `RunReport`; this avoids a second unstructured error surface
+  and preserves the existing report schema.
+- Introduce one CLI-owned run-state value that accumulates terminal-report data
+  as resolution and execution progress. Success and failure reporting both
+  derive `debugbundle.RunReport` from this value rather than constructing
+  unrelated literals.
+- Add a single summary terminalization operation that attempts
+  `run-report.json` and, on failure, `error.log`. On a successful command it
+  writes only the success report. On a failed command it writes the failure
+  report and error log, attempting both even if one write fails and joining
+  secondary persistence errors for stderr reporting.
+- Terminalization is invoked at most once. If the primary failure is itself a
+  summary persistence failure, do not recursively retry the failed operation;
+  make one terminalization attempt that excludes the artifact already known to
+  have failed. In particular, a failure to write `run-report.json` must never
+  call itself again.
+- Stderr prints the primary error first, then any terminalization error, then
+  the allocated debug path. Exit code remains `1`. A secondary error never
+  replaces or wraps away the primary error.
+- Runs without debug perform no summary work and retain current concise stderr
+  behavior.
+
+## Stage 1: Enforce redaction for resolved-pipeline summaries
+
+**Status:** Not started
+
+### Objective
+
+Close the summary credential leak and make the redacted-payload boundary
+difficult to bypass accidentally.
+
+### Read first
+
+- `internal/core/config/redaction.go`
+- `internal/core/config/effective_config.go`
+- `internal/core/debugbundle/summary.go`
+- the summary-writing section of `internal/cli/run.go`
+- `internal/core/config/v3_test.go`
+- `internal/cli/state_hardening_test.go`
+- [Architecture: State, Output, And Safety](../policy/architecture.md#state-output-and-safety)
+
+### Implement
+
+1. Add the resolved-pipeline redacted payload interface to `debugbundle` and
+   change `SummaryWriter.WriteResolvedPipeline` to accept it rather than `any`.
+2. Implement the interface on `config.EffectiveConfig` using the existing deep
+   clone/redaction path.
+3. Audit recursive option redaction. Extend it only where nested list/map values
+   can currently retain a value under a sensitive key.
+4. Change the CLI call to pass `effective` through the redacted interface.
+5. Keep `effective-config.json` behavior and `resolved-pipeline.json` logical
+   structure unchanged apart from required redaction.
+
+### Focused tests
+
+- Unit-test a resolved pipeline containing distinct sentinel secrets in every
+  module and validator binding listed under Fixed Decisions.
+- Include nested maps and lists, safe neighboring values, and materialized
+  reference content. Assert secrets and reference contents are absent, redaction
+  markers are present, and safe provenance remains.
+- Run a debug-enabled CLI invocation whose accepted fake module options include
+  sensitive keys. Inspect both `effective-config.json` and
+  `resolved-pipeline.json` and prove neither contains the sentinels.
+- Assert mutation of a redacted payload cannot alter the effective configuration
+  or resolved pipeline.
+
+Run at minimum:
+
+```sh
+go test ./internal/core/config ./internal/core/debugbundle ./internal/cli
+```
+
+### Exit criteria
+
+No arbitrary resolved pipeline can be passed to the redacted summary writer;
+all resolved bindings are recursively redacted; focused leak tests fail against
+the old implementation and pass against the new one; and all repository checks
+pass.
+
+## Stage 2: Make run identity collision-resistant and output exclusive
+
+**Status:** Not started
+
+### Objective
+
+Ensure separate invocations can never silently share or overwrite one durable
+output run directory.
+
+### Read first
+
+- run initialization and output writing in `internal/cli/run.go`
+- `internal/core/debugbundle/bundle.go`
+- `internal/core/debugbundle/bundle_test.go`
+- `internal/cli/state_surfaces_test.go`
+- `internal/cli/state_hardening_test.go`
+- [Operations: Output](../operations.md#output)
+
+### Implement
+
+1. Add the default cryptographic run-ID generator and inject it through
+   `cli.Options` as specified above. Validate generated IDs before any path use.
+2. Generate one ID from `startedAt` before debug allocation. Remove the debug
+   allocator's package-global clock and identity generation.
+3. Refactor `debugbundle.Allocate` to accept the run ID and start time and
+   exclusively create that exact bundle.
+4. Refactor output writing so the run directory is created exclusively after
+   all logical file names validate. Return a contextual collision error without
+   touching the existing directory.
+5. Preserve atomic per-file writes, partial-new-directory behavior on later
+   write failures, and the single ID across manifests and reported paths.
+6. Update existing focused tests to inject deterministic run IDs rather than
+   relying on timestamps or the debug allocator's global clock.
+
+### Focused tests
+
+- Verify the production generator's format and validate many generated IDs for
+  uniqueness and path safety without asserting random bytes.
+- Inject a deterministic ID and prove debug and output use the exact same run
+  directory name and manifest run ID.
+- Pre-create the target output run directory with sentinel files. Run the
+  command and assert exit `1`, a collision error, and byte-for-byte preservation
+  of the existing tree.
+- Execute two invocations with the same injected ID and root. The first may
+  succeed; the second must fail without changing the first output.
+- Pre-create the target debug bundle and assert allocation fails without
+  changing it.
+- Prove nested logical output paths still work and unsafe logical paths are
+  rejected before the run directory is created.
+- Prove a later output-file failure retains only the newly allocated partial
+  directory and never affects a sibling run.
+
+Run at minimum:
+
+```sh
+go test ./internal/core/debugbundle ./internal/cli
+```
+
+### Exit criteria
+
+Every invocation has one collision-resistant ID, output and debug allocation
+are exclusive, an existing run directory is never reused or overwritten, and
+all repository checks pass.
+
+## Stage 3: Write terminal run reports for failures
+
+**Status:** Not started
+
+### Objective
+
+Make every allocated debug summary record a structured terminal outcome while
+preserving primary errors and preventing recursive persistence failures.
+
+### Read first
+
+- `internal/core/debugbundle/summary.go`
+- failure and summary paths in `internal/cli/run.go`
+- `internal/cli/state_hardening_test.go`
+- [Run State Internals](../internal/state.md)
+- [Operations: Debug Bundles](../operations.md#debug-bundles)
+- [Operations: Failures And Warnings](../operations.md#failures-and-warnings)
+
+### Implement
+
+1. Add the CLI-owned run-state/report builder described under Fixed Decisions.
+   Initialize it immediately after run-ID generation and update it as pipeline
+   identity, output path, and `RunOutput` become available.
+2. Add a debug-summary terminalization method or focused helper that writes the
+   success report, or the failure report plus error log, exactly once.
+3. Route every post-allocation failure through the common terminalization path:
+   catalog and reference resolution, pipeline resolution, profile validation,
+   reference materialization, registry and LLM construction, preparation, input
+   read, cache/checkpoint construction, pipeline execution, summary writes,
+   output allocation, and output writes.
+4. Replace the existing success-only report literal with the same report
+   builder and terminalization path.
+5. Preserve failures before run-ID generation or debug allocation as stderr-only
+   failures.
+6. Keep explicit guards for failures caused by summary terminalization itself so
+   the command never recursively rewrites `run-report.json` or `error.log`.
+
+### Focused tests
+
+- For resolution, pipeline, and output-write failures, assert both
+  `run-report.json` and `error.log` exist, `succeeded` is false, and every known
+  field is correct.
+- For a pipeline failure with partial `RunOutput`, assert available counts,
+  validation status, manifest, warnings, checkpoint events, and chunk-plan
+  summary are retained.
+- Assert a success report uses `succeeded: true` and contains the same paths and
+  counts printed by the CLI.
+- Inject independent failures for the run-report writer and error-log writer.
+  Assert each operation is attempted no more than once, the primary error is
+  printed first, the secondary error is reported, the debug path is printed,
+  and exit code is `1`.
+- Assert a pre-allocation configuration failure creates no report or debug root.
+- Assert a non-debug failure performs no summary writes.
+
+Run at minimum:
+
+```sh
+go test ./internal/core/debugbundle ./internal/cli
+```
+
+### Exit criteria
+
+Every successfully allocated debug bundle has a best-effort terminal report;
+ordinary failures retain both structured and textual outcomes; summary failures
+cannot recurse or mask the primary error; and all repository checks pass.
+
+## Stage 4: Final audit and close the remediation plan
+
+**Status:** Not started
+
+### Objective
+
+Verify the three findings are closed, align any affected current-behavior
+documentation, and remove this completed planning artifact.
+
+### Read first
+
+- the final code and tests from Stages 1–3
+- [Documentation Policy](../policy/documentation.md)
+- [Architecture Policy](../policy/architecture.md)
+- [CLI Reference](../cli.md)
+- [Operations](../operations.md)
+- [Run State Internals](../internal/state.md)
+
+### Implement
+
+1. Audit every `summary/` writer call and confirm each accepts or constructs a
+   redacted payload. Search for direct serialization of raw effective pipeline
+   bindings into summary files.
+2. Audit every output path and confirm no code writes into a pre-existing run
+   directory. Confirm debug and output use one CLI-owned ID.
+3. Audit every post-allocation return path and confirm it reaches terminal
+   reporting exactly once or is itself an explicitly guarded terminalization
+   failure.
+4. Update Operations or Run State Internals only where the final implementation
+   adds a useful current-behavior clarification about run IDs, exclusive output
+   allocation, partial output after failure, or best-effort failure reports.
+   Do not duplicate volatile CLI or configuration contracts.
+5. Confirm Finding 4 remains deferred. Do not restore unrelated deleted tests,
+   introduce a coverage threshold, or add a legacy-checkpoint fixture in this
+   remediation.
+6. After all validation succeeds and documentation is current, delete this
+   implementation plan. The accepted ADR and canonical current-behavior
+   documents remain authoritative.
+
+### Validation
+
+Run:
+
+```sh
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+git diff --check
+```
+
+Manually inspect the changed debug artifacts from one success, one pipeline
+failure, and one output collision. Confirm no test fixture contains real
+credentials or private data.
+
+### Exit criteria
+
+Findings 1–3 are closed with focused regression tests; durable output cannot be
+overwritten by a run-ID collision; failure bundles have terminal reports;
+summary redaction is enforced by contract; documentation is accurate; Finding
+4 remains deferred; the plan is removed; and all checks pass.

docs/roadmap/mvp.md

@@ -1,29 +0,0 @@
-# Future Work
-
-Current Notarius behavior is documented in the canonical README, CLI,
-configuration, operations, internal, and integration docs. This roadmap records
-future work only.
-
-## Candidate Product Work
-
-- Additional input adapters, such as Markdown or note-export formats.
-- Additional D&D extractors beyond spell casts.
-- Cross-lane entity normalization.
-- Cross-chunk semantic deduplication.
-- Configurable validator chains with production validator modules.
-- Multiple effective LLM profiles in one run.
-- Parallel execution where it preserves deterministic manifests and diagnostics.
-- Additional output encoders.
-
-## Candidate Operational Work
-
-- Packaged release artifacts for alpha distribution.
-- A documented versioning and release process.
-- Optional generated example output fixtures with a regeneration procedure.
-- Additional diagnostics or reporting views if operator workflows need them.
-
-## Non-Goals To Revisit Deliberately
-
-- A general workflow language.
-- Structural module selection through ad hoc run flags.
-- Storing secrets in config files, diagnostics, manifests, or examples.

docs/troubleshooting.md

@@ -1,215 +0,0 @@
-# Troubleshooting
-
-This guide maps common implemented failure modes to inspection steps and fixes.
-For command syntax, see [CLI Reference](cli.md). For YAML fields and
-environment overrides, see [Configuration](config.md). For output and
-diagnostics layout, see [Operations](operations.md).
-
-## Config File Not Found
-
-Symptom:
-
-```text
-notarius: config file not found; pass --config or set NOTARIUS_CONFIG
-```
-
-Fix:
-
-- Pass `--config path/to/config.yml`.
-- Or set `NOTARIUS_CONFIG` to a readable file.
-- Or install a config at `/usr/local/etc/notarius/config.yml`.
-
-If the message says the config path is a directory or is not available, correct
-the path or file permissions.
-
-## Unsupported Or Invalid Config
-
-Symptoms include:
-
-- `unsupported config version`
-- `config version is required`
-- `field <name> not found`
-- `total LLM concurrency must be greater than zero`
-- `diagnostics retention "<value>" is not supported`
-
-Fix:
-
-- Use `version: 1`.
-- Remove unknown YAML fields.
-- Validate with:
-
-```sh
-go run ./cmd/notarius config validate --config path/to/config.yml
-```
-
-## Unknown Pipeline
-
-Symptom:
-
-```text
-notarius: pipeline "..." is not configured
-```
-
-Fix:
-
-- List configured pipeline IDs:
-
-```sh
-go run ./cmd/notarius pipelines list --config path/to/config.yml
-```
-
-- Use one of those IDs in `notarius run <pipeline-id>`.
-- Check indentation under the top-level `pipelines` map.
-
-## Unknown Or Incompatible Module
-
-Symptoms mention a module key, pipeline slot, lane, capability, or `not
-registered`.
-
-Fix:
-
-- Validate the pipeline against the production module catalog:
-
-```sh
-go run ./cmd/notarius config validate \
-  --config path/to/config.yml \
-  --pipeline dnd-session
-```
-
-- Use only implemented production module keys listed in
-  [Configuration](config.md#implemented-production-modules).
-- Check that artifact lanes include an `extract` binding.
-
-## Invalid `--only`
-
-Symptoms include:
-
-- `--only must contain comma-separated non-empty artifact lane IDs`
-- `--only requires --pipeline`
-- `selected artifact lane`
-
-Fix:
-
-- Use comma-separated lane IDs with no empty entries:
-
-```sh
-go run ./cmd/notarius run dnd-session \
-  --config path/to/config.yml \
-  --input path/to/input.json \
-  --only spells
-```
-
-- For `config validate`, include `--pipeline` when using `--only`.
-- Confirm the lane ID exists under `pipelines.<id>.artifacts`.
-
-## Seriatim Input Validation Failure
-
-Symptoms include `seriatim input`, `parse JSON`, `segments must not be empty`,
-or validation errors naming a segment field.
-
-Fix:
-
-- Compare the input to
-  [examples/seriatim-minimal-transcript.json](../examples/seriatim-minimal-transcript.json).
-- Ensure the JSON has a `metadata` object and a non-empty `segments` array.
-- Each segment needs a non-empty `id`, non-empty `speaker`, non-empty `text`,
-  non-negative numeric `start`, and non-negative numeric `end`.
-- Segment IDs must be unique and must not contain leading or trailing
-  whitespace.
-- `end` must be greater than or equal to `start`.
-
-## Missing LLM Base URL Or Model
-
-Symptoms include:
-
-- `LLM profile "default" base URL must not be empty`
-- `LLM profile "default" model must not be empty`
-- `base URL must be valid`
-
-Fix:
-
-- Set `base_url` and `model` in `llm_profiles.default`.
-- Or set `NOTARIUS_LLM_DEFAULT_BASE_URL` and
-  `NOTARIUS_LLM_DEFAULT_MODEL`.
-- If a profile needs authentication, set `api_key_env` in YAML or set
-  `NOTARIUS_LLM_DEFAULT_API_KEY`.
-
-## LLM Profile Override Failure
-
-Symptom:
-
-```text
-notarius: LLM profile override "..." is not configured
-```
-
-Fix:
-
-- Add the profile under `llm_profiles`.
-- Or use an existing profile ID with `--llm-profile`.
-
-Current runs require exactly one distinct effective LLM profile. If a pipeline
-uses several profiles, run with `--llm-profile <id>` or align the bindings in
-configuration.
-
-## Provider HTTP Or Response Failure
-
-Symptoms include:
-
-- `provider request failed`
-- `provider returned status 400`
-- `provider returned status 403`
-- `provider response missing choices`
-- `provider response assistant message content is not valid JSON`
-- `decode structured output`
-
-Fix:
-
-- Confirm the `base_url` points to an OpenAI-compatible endpoint root. Notarius
-  posts to `<base_url>/chat/completions`.
-- Check `model` and provider credentials.
-- Inspect the retained diagnostics `error.log`.
-- For 400 and 403 responses, fix the request configuration or credentials.
-- For 429 and 5xx responses, the client retries according to `max_retries`; if
-  the failure persists, inspect the provider response and adjust capacity,
-  credentials, or model settings.
-- The assistant message content must decode as JSON matching the extractor's
-  structured response schema.
-
-Provider error messages are redacted for configured API key values.
-
-## Output Write Failure
-
-Symptoms include:
-
-- `create output directory`
-- `write output file`
-- `output file name must`
-
-Fix:
-
-- Ensure `--output-dir` points to a directory path or a path that can be
-  created.
-- Check filesystem permissions and available disk space.
-- If diagnostics were retained, inspect `run-report.json`, `run-manifest.json`,
-  and `error.log`.
-
-The CLI rejects unsafe logical output paths before writing files.
-
-## Diagnostics Directory Surprise
-
-Symptom: the diagnostics directory is missing after a successful run.
-
-Fix:
-
-- Check `diagnostics.retention`.
-- With `auto`, successful runs without warnings are removed.
-- Use `diagnostics.retention: always` when every diagnostics run directory
-  should be kept.
-- Use `--diagnostics-dir` to override the configured work directory for a run.
-
-Symptom: diagnostics exist even with `retention: never`.
-
-Explanation:
-
-- Failed runs are retained so that `error.log` and available context can be
-  inspected.

examples/dnd-spells-glossary.txt

@@ -0,0 +1,2 @@
+Cure Wounds: healing spell cast by touch.
+Shield: defensive reaction spell.

examples/dnd-spells-production.config.yml

@@ -0,0 +1,28 @@
+version: 3
+concurrency:
+  total_llm: 1
+  stage_workers:
+    extract: 1
+output:
+  directory: ./notarius-output
+cache:
+  chunk_plans:
+    directory: /var/cache/notarius/chunk-plans
+    mode: auto
+  checkpoints:
+    directory: /var/cache/notarius/checkpoints
+debug:
+  directory: ./notarius-debug
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      party: ./dnd-spells-roster.txt
+      glossary: ./dnd-spells-glossary.txt
+    chunk:
+      module: generic
+      options:
+        max_units: 50
+    artifacts:
+      spells:
+        extract: dnd/spells

examples/dnd-spells-roster.txt

@@ -0,0 +1,3 @@
+Aria: party cleric and recurring healer.
+Borin: fighter ally.
+Bandit mage: hostile spellcaster.

examples/dnd-spells.config.yml

@@ -1,16 +1,16 @@
-version: 1
+version: 3
-llm_profiles:
+output:
-  default:
+  directory: ./notarius-output
-    provider: openai-compatible
+cache:
-    base_url: http://127.0.0.1:1
+  chunk_plans:
-    model: fake-model
+    mode: bypass
+  checkpoints:
+    directory: ""
+debug:
+  directory: ./notarius-debug
 pipelines:
   dnd-session:
     input: seriatim
-    chunk:
-      module: generic
-      options:
-        max_units: 50
     artifacts:
       spells:
         extract: dnd/spells

examples/seriatim-minimal-transcript.json

@@ -5,14 +5,14 @@
   },
   "segments": [
     {
-      "id": "seg-001",
+      "id": 1,
       "start": 0,
       "end": 4,
       "speaker": "Aria",
       "text": "Aria raises her holy symbol and casts Cure Wounds."
     },
     {
-      "id": "seg-002",
+      "id": 2,
       "start": 4,
       "end": 8,
       "speaker": "DM",

go.mod

@@ -1,5 +1,11 @@
 module gitea.maximumdirect.net/eric/notarius
 
-go 1.24.0
+go 1.25.5
 
-require gopkg.in/yaml.v3 v3.0.1
+require (
+	gitea.maximumdirect.net/eric/scriptorium v0.11.0
+	github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
+	gopkg.in/yaml.v3 v3.0.1
+)
+
+require golang.org/x/text v0.14.0 // indirect

go.sum

@@ -1,3 +1,11 @@
+gitea.maximumdirect.net/eric/scriptorium v0.11.0 h1:rjvbt9FTaWHxYlHq7QlUzmMVUt3QdbTmeCkmH81N//o=
+gitea.maximumdirect.net/eric/scriptorium v0.11.0/go.mod h1:FQ5lEuNxmrQyNgIomkpZdxvfTC0jWjbXYuq3tbJWF64=
+github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
+github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
+github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
+github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
+golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
+golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

internal/cli/catalog.go

@@ -3,50 +3,54 @@ package cli
 import (
 	"context"
 	"fmt"
-	"strings"
 
 	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
 	"gitea.maximumdirect.net/eric/notarius/internal/core/config"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
-	"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
+	dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
-	"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
+	genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
-	"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
+	seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
-	"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
-	"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
-	jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
 )
 
-func productionRegistries() (pipeline.Registries, error) {
+type productionComponents struct {
+	registries pipeline.Registries
+	assets     *llm.AssetRegistry
+}
+
+func newProductionComponents() (productionComponents, error) {
 	registries := pipeline.Registries{
-		Inputs:      pipeline.NewInputAdapterRegistry(),
+		Inputs:          pipeline.NewInputAdapterRegistry(),
-		Chunkers:    pipeline.NewChunkerRegistry(),
+		Chunkers:        pipeline.NewChunkerRegistry(),
-		Extractors:  pipeline.NewExtractorRegistry(),
+		ArtifactCodecs:  pipeline.NewArtifactCodecRegistry(),
-		Mergers:     pipeline.NewMergerRegistry(),
+		Extractors:      pipeline.NewExtractorRegistry(),
-		Normalizers: pipeline.NewNormalizerRegistry(),
+		Mergers:         pipeline.NewMergerRegistry(),
-		Validators:  pipeline.NewValidatorRegistry(),
+		Normalizers:     pipeline.NewNormalizerRegistry(),
-		Outputs:     pipeline.NewOutputEncoderRegistry(),
+		Validators:      pipeline.NewValidatorRegistry(),
+		ValidatorChains: pipeline.NewValidatorChainRegistry(),
+		Outputs:         pipeline.NewOutputEncoderRegistry(),
 	}
-	if err := seriatim.Register(registries.Inputs); err != nil {
+	assets := llm.NewAssetRegistry()
-		return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
+	registrars := []struct {
+		name     string
+		register func(pipeline.Registries, *llm.AssetRegistry) error
+	}{
+		{name: "generic", register: genericregister.Register},
+		{name: "seriatim", register: seriatimregister.Register},
+		{name: "dnd", register: dndregister.Register},
 	}
-	if err := generic.Register(registries.Chunkers); err != nil {
+	for _, registrar := range registrars {
-		return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
+		if err := registrar.register(registries, assets); err != nil {
+			return productionComponents{}, fmt.Errorf("register %s module family: %w", registrar.name, err)
+		}
 	}
-	if err := spells.Register(registries.Extractors); err != nil {
+	return productionComponents{registries: registries, assets: assets}, nil
-		return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
+}
-	}
+
-	if err := appendorder.Register(registries.Mergers); err != nil {
+func productionRegistries() (pipeline.Registries, error) {
-		return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
+	components, err := newProductionComponents()
-	}
+	return components.registries, err
-	if err := noop.Register(registries.Normalizers); err != nil {
-		return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
-	}
-	if err := jsonoutput.Register(registries.Outputs); err != nil {
-		return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
-	}
-	return registries, nil
 }
 
 func productionCatalog() (pipeline.ModuleCatalog, error) {
@@ -57,6 +61,11 @@ func productionCatalog() (pipeline.ModuleCatalog, error) {
 	return catalogFromRegistries(registries), nil
 }
 
+func productionPromptAssets() (*llm.AssetRegistry, error) {
+	components, err := newProductionComponents()
+	return components.assets, err
+}
+
 func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
 	if !isEmptyCatalog(opts.Catalog) {
 		return opts.Catalog, nil
@@ -79,45 +88,53 @@ func effectiveRegistries(opts Options) (pipeline.Registries, error) {
 
 func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalog {
 	return pipeline.ModuleCatalog{
-		Inputs:      registries.Inputs,
+		Inputs:          registries.Inputs,
-		Chunkers:    registries.Chunkers,
+		Chunkers:        registries.Chunkers,
-		Extractors:  registries.Extractors,
+		ArtifactCodecs:  registries.ArtifactCodecs,
-		Mergers:     registries.Mergers,
+		Extractors:      registries.Extractors,
-		Normalizers: registries.Normalizers,
+		Mergers:         registries.Mergers,
-		Validators:  registries.Validators,
+		Normalizers:     registries.Normalizers,
-		Outputs:     registries.Outputs,
+		Validators:      registries.Validators,
+		ValidatorChains: registries.ValidatorChains,
+		Outputs:         registries.Outputs,
 	}
 }
 
 func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
 	return pipeline.Registries{
-		Inputs:      catalog.Inputs,
+		Inputs:          catalog.Inputs,
-		Chunkers:    catalog.Chunkers,
+		Chunkers:        catalog.Chunkers,
-		Extractors:  catalog.Extractors,
+		ArtifactCodecs:  catalog.ArtifactCodecs,
-		Mergers:     catalog.Mergers,
+		Extractors:      catalog.Extractors,
-		Normalizers: catalog.Normalizers,
+		Mergers:         catalog.Mergers,
-		Validators:  catalog.Validators,
+		Normalizers:     catalog.Normalizers,
-		Outputs:     catalog.Outputs,
+		Validators:      catalog.Validators,
+		ValidatorChains: catalog.ValidatorChains,
+		Outputs:         catalog.Outputs,
 	}
 }
 
 func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
 	return catalog.Inputs == nil &&
 		catalog.Chunkers == nil &&
+		catalog.ArtifactCodecs == nil &&
 		catalog.Extractors == nil &&
 		catalog.Mergers == nil &&
 		catalog.Normalizers == nil &&
 		catalog.Validators == nil &&
+		catalog.ValidatorChains == nil &&
 		catalog.Outputs == nil
 }
 
 func isEmptyRegistries(registries pipeline.Registries) bool {
 	return registries.Inputs == nil &&
 		registries.Chunkers == nil &&
+		registries.ArtifactCodecs == nil &&
 		registries.Extractors == nil &&
 		registries.Mergers == nil &&
 		registries.Normalizers == nil &&
 		registries.Validators == nil &&
+		registries.ValidatorChains == nil &&
 		registries.Outputs == nil
 }
 
@@ -125,48 +142,39 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
 	if err := ctx.Err(); err != nil {
 		return nil, nil, err
 	}
-	trimmedID := strings.TrimSpace(profileID)
+	assets, err := productionPromptAssets()
-	if trimmedID == "" {
-		trimmedID = pipeline.DefaultLLMProfile
-	}
-
-	profile, ok := cfg.LLMProfile(trimmedID)
-	if !ok {
-		return nil, nil, fmt.Errorf("LLM profile %q is not configured", trimmedID)
-	}
-	clientCfg, err := cfg.OpenAICompatibleClientConfig(trimmedID)
 	if err != nil {
 		return nil, nil, err
 	}
-	client, err := llm.NewOpenAICompatibleClient(clientCfg)
+	return buildProductionLLMClient(ctx, cfg, profileID, assets)
-	if err != nil {
-		return nil, nil, fmt.Errorf("create LLM client for profile %q: %w", trimmedID, err)
-	}
-
-	scheduler, err := llm.NewScheduler(effectiveLLMConcurrency(cfg, profile))
-	if err != nil {
-		return nil, nil, fmt.Errorf("create LLM scheduler for profile %q: %w", trimmedID, err)
-	}
-	provider := strings.TrimSpace(profile.Provider)
-	if provider == "" {
-		provider = "openai-compatible"
-	}
-	metadata := []artifacts.LLMProfileManifest{
-		{
-			ID:       trimmedID,
-			Provider: provider,
-			Model:    strings.TrimSpace(profile.Model),
-		},
-	}
-	return llm.NewScheduledClient(client, scheduler), metadata, nil
 }
 
-func effectiveLLMConcurrency(cfg config.Config, profile config.LLMProfile) int {
+func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
-	if profile.MaxConcurrency > 0 {
+	return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
-		return profile.MaxConcurrency
+		return buildProductionLLMClient(ctx, cfg, profileID, assets)
 	}
-	if cfg.Concurrency.TotalLLM > 0 {
+}
-		return cfg.Concurrency.TotalLLM
+
-	}
+func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
-	return 1
+	if err := ctx.Err(); err != nil {
+		return nil, nil, err
+	}
+	if assets == nil {
+		return nil, nil, fmt.Errorf("production asset registry must not be nil")
+	}
+	recorder := llm.NewLLMProfileRecorder()
+	client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
+		ProfileDir:  cfg.Scriptorium.ProfileDir,
+		ProfileFile: cfg.Scriptorium.ProfileFile,
+		Assets:      assets,
+		Recorder:    recorder,
+	})
+	if err != nil {
+		return nil, nil, fmt.Errorf("create Scriptorium-backed LLM client: %w", err)
+	}
+	scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
+	if err != nil {
+		return nil, nil, fmt.Errorf("create LLM scheduler: %w", err)
+	}
+	return llm.NewScheduledClient(client, scheduler), nil, nil
 }

internal/cli/scriptorium_profiles.go

@@ -0,0 +1,68 @@
+package cli
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"testing/fstest"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/config"
+	"gitea.maximumdirect.net/eric/scriptorium"
+)
+
+const profileCheckPromptID = "notarius.profile.check"
+
+var profileCheckPromptFS = fstest.MapFS{
+	"prompts/profile-check.yaml": &fstest.MapFile{Data: []byte(`id: notarius.profile.check
+version: "1.0.0"
+default_profile: mistral-small-3
+inputs:
+  - name: transcript
+    required: true
+messages:
+  - role: user
+    content: "{{input \"transcript\"}}"
+output:
+  format: text
+  validation_mode: none
+  repair_attempts: 0
+`)},
+}
+
+func validateExplicitScriptoriumProfiles(ctx context.Context, cfg config.Config, profileIDs []string) error {
+	if len(profileIDs) == 0 {
+		return nil
+	}
+	engine, err := newProfileValidationEngine(cfg)
+	if err != nil {
+		return fmt.Errorf("load Scriptorium profiles: %w", err)
+	}
+	for _, profileID := range profileIDs {
+		if _, err := engine.Prepare(ctx, scriptorium.RunRequest{
+			PromptID:  profileCheckPromptID,
+			ProfileID: profileID,
+			Inputs: map[string]scriptorium.ArtifactRef{
+				"transcript": scriptorium.Inline("profile check"),
+			},
+		}); err != nil {
+			if errors.Is(err, scriptorium.ErrProfileNotFound) {
+				return fmt.Errorf("Scriptorium profile %q is not configured", profileID)
+			}
+			return fmt.Errorf("validate Scriptorium profile %q: %w", profileID, err)
+		}
+	}
+	return nil
+}
+
+func newProfileValidationEngine(cfg config.Config) (*scriptorium.Engine, error) {
+	opts := []scriptorium.Option{
+		scriptorium.WithPromptFS(profileCheckPromptFS, "prompts"),
+	}
+	if cfg.Scriptorium.ProfileFile != "" {
+		opts = append(opts, scriptorium.WithProfileFile(cfg.Scriptorium.ProfileFile))
+	}
+	return scriptorium.NewEngine(scriptorium.Config{
+		PromptDir:  "unused",
+		ProfileDir: cfg.Scriptorium.ProfileDir,
+	}, opts...)
+}

internal/cli/state_hardening_test.go

@@ -0,0 +1,528 @@
+package cli
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
+	"gitea.maximumdirect.net/eric/notarius/internal/core/config"
+	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+	frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+const stateTestDigest = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+
+func TestRunStateSurfaceMatrix(t *testing.T) {
+	for _, debug := range []bool{false, true} {
+		for _, resume := range []bool{false, true} {
+			for _, mode := range []string{"auto", "bypass", "refresh"} {
+				name := fmt.Sprintf("debug=%t/resume=%t/cache=%s", debug, resume, mode)
+				t.Run(name, func(t *testing.T) {
+					roots := newStateTestRoots(t)
+					harness := newStateTestHarness()
+					opts := harness.options()
+					var storeRoots []string
+					opts.ChunkPlanStoreFactory = func(root string) (pipeline.ChunkPlanStore, error) {
+						storeRoots = append(storeRoots, root)
+						return chunkplan.NewFilesystemStore(root)
+					}
+					result := runStateTest(t, roots, opts, debug, resume, mode)
+					if result.code != 0 {
+						t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
+					}
+					assertStateTestOutput(t, roots.output)
+					if mode == "bypass" {
+						assertAbsent(t, roots.plans)
+						if len(storeRoots) != 0 {
+							t.Fatalf("chunk plan store roots = %v, want none", storeRoots)
+						}
+					} else {
+						assertFile(t, filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json"))
+						if len(storeRoots) != 1 || storeRoots[0] != roots.plans {
+							t.Fatalf("chunk plan store roots = %v, want [%q]", storeRoots, roots.plans)
+						}
+					}
+					if resume {
+						assertAnyFile(t, roots.checkpoints)
+						assertRestrictedTree(t, roots.checkpoints)
+					} else {
+						assertAbsent(t, roots.checkpoints)
+					}
+					if debug {
+						bundle := onlyChildDir(t, roots.debug)
+						assertFile(t, filepath.Join(bundle, "summary", "invocation.json"))
+						assertAnyFile(t, filepath.Join(bundle, "trace"))
+						assertRestrictedTree(t, roots.debug)
+					} else {
+						assertAbsent(t, roots.debug)
+					}
+				})
+			}
+		}
+	}
+}
+
+func TestRunKeepsStateRootsIndependentAndReusesSelectedCheckpointRoot(t *testing.T) {
+	roots := newStateTestRoots(t)
+	harness := newStateTestHarness()
+	first := runStateTest(t, roots, harness.options(), true, false, "auto")
+	if first.code != 0 {
+		t.Fatalf("first run code=%d stderr=%q", first.code, first.stderr)
+	}
+	planPath := filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json")
+	initialPlan, err := os.ReadFile(planPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	firstBundle := onlyChildDir(t, roots.debug)
+
+	second := runStateTest(t, roots, harness.options(), false, false, "auto")
+	if second.code != 0 {
+		t.Fatalf("second run code=%d stderr=%q", second.code, second.stderr)
+	}
+	if harness.chunkCalls != 1 {
+		t.Fatalf("chunk calls after debug toggle = %d, want 1", harness.chunkCalls)
+	}
+	if got, err := os.ReadFile(planPath); err != nil || !bytes.Equal(got, initialPlan) {
+		t.Fatalf("chunk plan changed after debug toggle: %v", err)
+	}
+	if _, err := os.Stat(firstBundle); err != nil {
+		t.Fatalf("initial debug bundle was removed: %v", err)
+	}
+
+	checkpointRoot := roots.checkpoints
+	seed := runStateTest(t, roots, harness.options(), false, true, "auto")
+	if seed.code != 0 {
+		t.Fatalf("checkpoint seed code=%d stderr=%q", seed.code, seed.stderr)
+	}
+	extractCalls := harness.extractCalls
+	checkpointFiles := readTree(t, checkpointRoot)
+	reused := runStateTest(t, roots, harness.options(), false, true, "auto")
+	if reused.code != 0 {
+		t.Fatalf("checkpoint reuse code=%d stderr=%q", reused.code, reused.stderr)
+	}
+	if harness.extractCalls != extractCalls {
+		t.Fatalf("extract calls after checkpoint reuse = %d, want %d", harness.extractCalls, extractCalls)
+	}
+	if got := readTree(t, checkpointRoot); !sameFiles(got, checkpointFiles) {
+		t.Fatal("reused checkpoint was rewritten")
+	}
+}
+
+func TestRunRecomputesOnlyAfterExplicitChunkPlanRemoval(t *testing.T) {
+	roots := newStateTestRoots(t)
+	harness := newStateTestHarness()
+	first := runStateTest(t, roots, harness.options(), true, false, "auto")
+	if first.code != 0 {
+		t.Fatalf("first run code=%d stderr=%q", first.code, first.stderr)
+	}
+	firstOutput := onlyChildDir(t, roots.output)
+	firstBundle := onlyChildDir(t, roots.debug)
+	entry := filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"))
+	if err := os.RemoveAll(entry); err != nil {
+		t.Fatal(err)
+	}
+	second := runStateTest(t, roots, harness.options(), false, false, "auto")
+	if second.code != 0 {
+		t.Fatalf("second run code=%d stderr=%q", second.code, second.stderr)
+	}
+	if harness.chunkCalls != 2 {
+		t.Fatalf("chunk calls = %d, want 2 after removing exact cache entry", harness.chunkCalls)
+	}
+	assertFile(t, filepath.Join(firstOutput, "result.json"))
+	assertFile(t, filepath.Join(firstBundle, "summary", "run-report.json"))
+}
+
+func TestRunRetainsDebugBundlesAcrossFailures(t *testing.T) {
+	t.Run("configuration failure precedes allocation", func(t *testing.T) {
+		root := filepath.Join(t.TempDir(), "debug")
+		var stdout, stderr bytes.Buffer
+		code := RunWithOptions([]string{"run", "sample", "--config", filepath.Join(t.TempDir(), "missing.yml"), "--input", "missing", "--debug", "--debug-dir", root}, &stdout, &stderr, newStateTestHarness().options())
+		if code != 1 || !strings.Contains(stderr.String(), "config file") {
+			t.Fatalf("code=%d stderr=%q", code, stderr.String())
+		}
+		assertAbsent(t, root)
+	})
+
+	for _, failure := range []struct {
+		name     string
+		expected string
+		setup    func(*testing.T, stateTestRoots, *stateTestHarness) Options
+	}{
+		{"resolution", "pipeline \"missing\"", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options { return h.options() }},
+		{"pipeline", "synthetic extraction failure", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
+			h.extractErr = errors.New("synthetic extraction failure")
+			return h.options()
+		}},
+		{"output", "create output directory", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
+			if err := os.WriteFile(roots.output, []byte("not a directory"), 0o600); err != nil {
+				t.Fatal(err)
+			}
+			return h.options()
+		}},
+		{"summary", "write debug invocation metadata", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
+			opts := h.options()
+			opts.DebugRecorderFactory = func(traceRoot string) (pipeline.DebugRecorder, error) {
+				if err := os.RemoveAll(filepath.Join(filepath.Dir(traceRoot), "summary")); err != nil {
+					return nil, err
+				}
+				if err := os.WriteFile(filepath.Join(filepath.Dir(traceRoot), "summary"), []byte("blocked"), 0o600); err != nil {
+					return nil, err
+				}
+				return frameworkdebug.NewFilesystemRecorder(traceRoot)
+			}
+			return opts
+		}},
+		{"trace", "trace unavailable", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
+			opts := h.options()
+			opts.DebugRecorderFactory = func(string) (pipeline.DebugRecorder, error) { return failingDebugRecorder{}, nil }
+			return opts
+		}},
+	} {
+		t.Run(failure.name, func(t *testing.T) {
+			roots := newStateTestRoots(t)
+			harness := newStateTestHarness()
+			opts := failure.setup(t, roots, harness)
+			failureStderr := ""
+			if failure.name == "resolution" {
+				var stdout, stderr bytes.Buffer
+				code := RunWithOptions([]string{"run", "missing", "--config", roots.config, "--input", roots.input, "--debug"}, &stdout, &stderr, opts)
+				if code != 1 {
+					t.Fatalf("code=%d stderr=%q", code, stderr.String())
+				}
+				failureStderr = stderr.String()
+			} else {
+				result := runStateTest(t, roots, opts, true, false, "bypass")
+				if result.code != 1 {
+					t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
+				}
+				failureStderr = result.stderr
+			}
+			if !strings.Contains(failureStderr, failure.expected) || !strings.Contains(failureStderr, "debug=") {
+				t.Fatalf("stderr=%q, want %q and debug path", failureStderr, failure.expected)
+			}
+			bundle := onlyChildDir(t, roots.debug)
+			if !strings.Contains(readAllFiles(t, bundle), "synthetic") && failure.name == "pipeline" {
+				t.Fatal("pipeline failure was not retained in debug bundle")
+			}
+		})
+	}
+}
+
+func TestRunDebugArtifactsRedactSecretsButRetainApplicationData(t *testing.T) {
+	roots := newStateTestRoots(t)
+	t.Setenv("STATE_TEST_UNRELATED_ENV", "HOST_ONLY_SENTINEL")
+	if err := os.WriteFile(filepath.Join(filepath.Dir(roots.input), "unrelated.txt"), []byte("HOST_ONLY_FILE_SENTINEL"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	harness := newStateTestHarness()
+	result := runStateTest(t, roots, harness.options(), true, false, "bypass")
+	if result.code != 0 {
+		t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
+	}
+	bundle := onlyChildDir(t, roots.debug)
+	summary := readAllFiles(t, filepath.Join(bundle, "summary"))
+	trace := readAllFiles(t, filepath.Join(bundle, "trace"))
+	for _, forbidden := range []string{"sk-secretvalue", "Bearer secretvalue", "HOST_ONLY_SENTINEL", "HOST_ONLY_FILE_SENTINEL"} {
+		if strings.Contains(summary, forbidden) || strings.Contains(trace, forbidden) {
+			t.Fatalf("debug bundle contains %q", forbidden)
+		}
+	}
+	if strings.Contains(summary, "application content") {
+		t.Fatal("summary contains raw application input")
+	}
+	if !strings.Contains(trace, "application content") {
+		t.Fatal("trace does not retain expected application input")
+	}
+}
+
+type stateTestRoots struct{ config, input, output, plans, checkpoints, debug string }
+
+func newStateTestRoots(t *testing.T) stateTestRoots {
+	t.Helper()
+	base := t.TempDir()
+	roots := stateTestRoots{input: filepath.Join(base, "input.txt"), output: filepath.Join(base, "output"), plans: filepath.Join(base, "plans"), checkpoints: filepath.Join(base, "checkpoints"), debug: filepath.Join(base, "debug")}
+	if err := os.WriteFile(roots.input, []byte("application content Bearer secretvalue sk-secretvalue"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	roots.config = filepath.Join(base, "config.yml")
+	config := fmt.Sprintf("version: 3\noutput:\n  directory: %q\ncache:\n  chunk_plans:\n    directory: %q\n    mode: auto\n  checkpoints:\n    directory: %q\ndebug:\n  directory: %q\npipelines:\n  sample:\n    input: test/input\n    chunk: test/chunk\n    artifacts:\n      items:\n        extract: test/extract\n        merge: test/merge\n        normalize: test/normalize\n    output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
+	if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	return roots
+}
+
+type stateTestResult struct {
+	code           int
+	stdout, stderr string
+}
+
+func runStateTest(t *testing.T, roots stateTestRoots, opts Options, debug, resume bool, mode string) stateTestResult {
+	t.Helper()
+	args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", mode}
+	if debug {
+		args = append(args, "--debug")
+	}
+	if resume {
+		args = append(args, "--resume")
+	}
+	var stdout, stderr bytes.Buffer
+	return stateTestResult{RunWithOptions(args, &stdout, &stderr, opts), stdout.String(), stderr.String()}
+}
+
+func assertStateTestOutput(t *testing.T, root string) {
+	t.Helper()
+	output := onlyChildDir(t, root)
+	data, err := os.ReadFile(filepath.Join(output, "result.json"))
+	if err != nil || string(data) != "{\"ok\":true}\n" {
+		t.Fatalf("output = %q, %v", data, err)
+	}
+}
+
+func onlyChildDir(t *testing.T, root string) string {
+	t.Helper()
+	entries, err := os.ReadDir(root)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var dirs []string
+	for _, entry := range entries {
+		if entry.IsDir() {
+			dirs = append(dirs, filepath.Join(root, entry.Name()))
+		}
+	}
+	if len(dirs) != 1 {
+		t.Fatalf("directories in %q = %v, want one", root, dirs)
+	}
+	return dirs[0]
+}
+
+func assertFile(t *testing.T, path string) {
+	t.Helper()
+	if info, err := os.Stat(path); err != nil || info.IsDir() {
+		t.Fatalf("file %q: %v", path, err)
+	}
+}
+func assertAbsent(t *testing.T, path string) {
+	t.Helper()
+	if _, err := os.Stat(path); !os.IsNotExist(err) {
+		t.Fatalf("%q exists or stat failed: %v", path, err)
+	}
+}
+func assertAnyFile(t *testing.T, root string) {
+	t.Helper()
+	if text := readAllFiles(t, root); text == "" {
+		t.Fatalf("no files under %q", root)
+	}
+}
+
+func readAllFiles(t *testing.T, root string) string {
+	t.Helper()
+	var content strings.Builder
+	if err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		if entry.IsDir() {
+			return nil
+		}
+		data, err := os.ReadFile(path)
+		if err != nil {
+			return err
+		}
+		content.Write(data)
+		return nil
+	}); err != nil {
+		t.Fatal(err)
+	}
+	return content.String()
+}
+
+func assertRestrictedTree(t *testing.T, root string) {
+	t.Helper()
+	if runtime.GOOS == "windows" {
+		return
+	}
+	if err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		want := os.FileMode(0o600)
+		if info.IsDir() {
+			want = 0o700
+		}
+		if info.Mode().Perm() != want {
+			return fmt.Errorf("%s has mode %o, want %o", path, info.Mode().Perm(), want)
+		}
+		return nil
+	}); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func readTree(t *testing.T, root string) map[string][]byte {
+	t.Helper()
+	files := map[string][]byte{}
+	if err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		if entry.IsDir() {
+			return nil
+		}
+		data, err := os.ReadFile(path)
+		if err != nil {
+			return err
+		}
+		relative, err := filepath.Rel(root, path)
+		if err != nil {
+			return err
+		}
+		files[relative] = data
+		return nil
+	}); err != nil {
+		t.Fatal(err)
+	}
+	return files
+}
+func sameFiles(left, right map[string][]byte) bool {
+	if len(left) != len(right) {
+		return false
+	}
+	for path, data := range left {
+		if !bytes.Equal(data, right[path]) {
+			return false
+		}
+	}
+	return true
+}
+
+type stateTestHarness struct {
+	mu                       sync.Mutex
+	chunkCalls, extractCalls int
+	extractErr               error
+}
+
+func newStateTestHarness() *stateTestHarness { return &stateTestHarness{} }
+func (h *stateTestHarness) options() Options {
+	registries := pipeline.Registries{Inputs: pipeline.NewInputAdapterRegistry(), Chunkers: pipeline.NewChunkerRegistry(), ArtifactCodecs: pipeline.NewArtifactCodecRegistry(), Extractors: pipeline.NewExtractorRegistry(), Mergers: pipeline.NewMergerRegistry(), Normalizers: pipeline.NewNormalizerRegistry(), Outputs: pipeline.NewOutputEncoderRegistry()}
+	if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
+		panic(err)
+	}
+	if err := registries.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
+		panic(err)
+	}
+	if err := registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, Requires: []string{"source"}, Provides: []string{"chunks"}}, func() (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
+		panic(err)
+	}
+	if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
+		panic(err)
+	}
+	if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }); err != nil {
+		panic(err)
+	}
+	if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }); err != nil {
+		panic(err)
+	}
+	if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }); err != nil {
+		panic(err)
+	}
+	return Options{Catalog: catalogFromRegistries(registries), Registries: registries, LookupEnv: emptyLookup, Now: func() time.Time { return time.Unix(1, 0) }, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
+		return nil, nil, nil
+	}}
+}
+
+type stateTestInput struct{}
+
+func (stateTestInput) Key() string { return "test/input" }
+func (stateTestInput) Parse(_ context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
+	return &source.SourceDocument{ID: "source", Kind: "text", Format: "text/plain", Digest: stateTestDigest, Units: []source.SourceUnit{{ID: 1, Kind: "text", Text: string(req.Raw), Ref: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}}}}, nil
+}
+
+type stateTestChunker struct{ harness *stateTestHarness }
+
+func (stateTestChunker) Key() string                               { return "test/chunk" }
+func (stateTestChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
+func (c stateTestChunker) Plan(_ context.Context, req contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
+	c.harness.mu.Lock()
+	c.harness.chunkCalls++
+	c.harness.mu.Unlock()
+	return contracts.ChunkPlanResult{Plan: source.ChunkPlan{SourceDigest: req.Source.Digest, Ranges: []source.ChunkRange{{StartUnitID: 1, EndUnitID: 1}}}}, nil
+}
+
+const stateTestArtifactKind contracts.ArtifactKind = "test/artifact"
+
+type stateTestArtifact struct {
+	Value string `json:"value"`
+}
+type stateTestCodec struct{}
+
+func (stateTestCodec) Kind() contracts.ArtifactKind { return stateTestArtifactKind }
+func (stateTestCodec) Schema() contracts.ArtifactSchema {
+	return contracts.ArtifactSchema{ID: "test.artifact", Name: "test_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
+}
+func (stateTestCodec) MediaType() string { return "application/json" }
+func (stateTestCodec) EncodeCandidate(v stateTestArtifact) ([]byte, error) {
+	return []byte(`{"value":"ok"}`), nil
+}
+func (stateTestCodec) Encode(v stateTestArtifact) ([]byte, error) {
+	return []byte(`{"value":"ok"}`), nil
+}
+func (stateTestCodec) Decode([]byte) (stateTestArtifact, error) {
+	return stateTestArtifact{Value: "ok"}, nil
+}
+
+type stateTestExtractor struct{ harness *stateTestHarness }
+
+func (stateTestExtractor) Key() string                               { return "test/extract" }
+func (stateTestExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
+func (e stateTestExtractor) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[stateTestArtifact], error) {
+	e.harness.mu.Lock()
+	defer e.harness.mu.Unlock()
+	e.harness.extractCalls++
+	if e.harness.extractErr != nil {
+		return contracts.TypedExtractionResult[stateTestArtifact]{}, e.harness.extractErr
+	}
+	return contracts.TypedExtractionResult[stateTestArtifact]{Value: stateTestArtifact{Value: "ok"}}, nil
+}
+
+type stateTestMerger struct{}
+
+func (stateTestMerger) Key() string { return "test/merge" }
+func (stateTestMerger) Merge(_ context.Context, req contracts.TypedMergeRequest[stateTestArtifact]) (contracts.TypedMergeResult[stateTestArtifact], error) {
+	return contracts.TypedMergeResult[stateTestArtifact]{Value: req.ExtractOutputs[0].Value}, nil
+}
+
+type stateTestNormalizer struct{}
+
+func (stateTestNormalizer) Key() string                               { return "test/normalize" }
+func (stateTestNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
+func (stateTestNormalizer) Normalize(_ context.Context, req contracts.TypedNormalizeRequest[stateTestArtifact]) (contracts.TypedNormalizeResult[stateTestArtifact], error) {
+	return contracts.TypedNormalizeResult[stateTestArtifact]{Value: req.MergeOutput.Value}, nil
+}
+
+type stateTestOutput struct{}
+
+func (stateTestOutput) Key() string { return "test/output" }
+func (stateTestOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
+	return contracts.OutputResult{Files: []contracts.OutputFile{{Name: "result.json", Bytes: []byte("{\"ok\":true}\n")}}}, nil
+}
+
+type failingDebugRecorder struct{}
+
+func (failingDebugRecorder) Enabled() bool                   { return true }
+func (failingDebugRecorder) WriteJSON(string, any) error     { return errors.New("trace unavailable") }
+func (failingDebugRecorder) WriteBytes(string, []byte) error { return errors.New("trace unavailable") }

internal/cli/state_surfaces_test.go

@@ -0,0 +1,88 @@
+package cli
+
+import (
+	"bytes"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestRunRejectsDebugDirectoryWithoutDebug(t *testing.T) {
+	var stdout, stderr bytes.Buffer
+	code := RunWithOptions([]string{"run", "example", "--input", "source.json", "--debug-dir", t.TempDir()}, &stdout, &stderr, Options{})
+	if code != 2 || !strings.Contains(stderr.String(), "--debug-dir requires --debug") {
+		t.Fatalf("code=%d stderr=%q", code, stderr.String())
+	}
+}
+
+func TestRunDebugAllocatesBeforePipelineResolution(t *testing.T) {
+	root := t.TempDir()
+	configPath := writeV3Config(t, "")
+	var stdout, stderr bytes.Buffer
+	code := RunWithOptions([]string{"run", "missing", "--config", configPath, "--input", "source.json", "--debug", "--debug-dir", root, "--chunk_cache", "bypass"}, &stdout, &stderr, Options{LookupEnv: emptyLookup})
+	if code != 1 {
+		t.Fatalf("code=%d stderr=%q", code, stderr.String())
+	}
+	entries, err := os.ReadDir(root)
+	if err != nil || len(entries) != 1 {
+		t.Fatalf("debug bundles: %v, %v", entries, err)
+	}
+	bundle := filepath.Join(root, entries[0].Name())
+	for _, name := range []string{"summary", "trace"} {
+		if info, err := os.Stat(filepath.Join(bundle, name)); err != nil || !info.IsDir() {
+			t.Fatalf("%s: %v", name, err)
+		}
+	}
+	if !strings.Contains(stderr.String(), "debug=") {
+		t.Fatalf("stderr does not include bundle path: %q", stderr.String())
+	}
+}
+
+func TestRunWithoutDebugDoesNotAllocateDebugRoot(t *testing.T) {
+	root := filepath.Join(t.TempDir(), "not-created")
+	configPath := writeV3Config(t, "")
+	var stdout, stderr bytes.Buffer
+	lookup := func(name string) (string, bool) {
+		if name == "NOTARIUS_DEBUG_DIR" {
+			return root, true
+		}
+		return "", false
+	}
+	code := RunWithOptions([]string{"run", "missing", "--config", configPath, "--input", "source.json", "--chunk_cache", "bypass"}, &stdout, &stderr, Options{LookupEnv: lookup})
+	if code != 1 {
+		t.Fatalf("code=%d stderr=%q", code, stderr.String())
+	}
+	if _, err := os.Stat(root); !os.IsNotExist(err) {
+		t.Fatalf("debug root exists or unexpected error: %v", err)
+	}
+}
+
+func TestConfigValidateUsesVersion3AndRemovedFieldsFail(t *testing.T) {
+	configPath := writeV3Config(t, "")
+	var stdout, stderr bytes.Buffer
+	if code := RunWithOptions([]string{"config", "validate", "--config", configPath}, &stdout, &stderr, Options{LookupEnv: emptyLookup}); code != 0 {
+		t.Fatalf("code=%d stderr=%q", code, stderr.String())
+	}
+	legacy := filepath.Join(t.TempDir(), "legacy.yml")
+	if err := os.WriteFile(legacy, []byte("version: 3\nworkspace:\n  directory: /tmp/old\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	stdout.Reset()
+	stderr.Reset()
+	if code := RunWithOptions([]string{"config", "validate", "--config", legacy}, &stdout, &stderr, Options{LookupEnv: emptyLookup}); code != 1 || !strings.Contains(stderr.String(), "field workspace not found") {
+		t.Fatalf("code=%d stderr=%q", code, stderr.String())
+	}
+}
+
+func writeV3Config(t *testing.T, extra string) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "config.yml")
+	data := "version: 3\noutput:\n  directory: ./out\ncache:\n  chunk_plans:\n    mode: bypass\n  checkpoints: {}\ndebug:\n  directory: ./debug\n" + extra + "pipelines: {}\n"
+	if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	return path
+}
+
+func emptyLookup(string) (string, bool) { return "", false }

internal/core/artifacts/artifacts.go

@@ -1,91 +1,125 @@
 package artifacts
 
 import (
-	"encoding/json"
 	"time"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
 )
 
-type ArtifactCandidate struct {
-	Index         int                `json:"index"`
-	ExtractorKey  string             `json:"extractor_key"`
-	ArtifactType  string             `json:"artifact_type"`
-	SchemaVersion string             `json:"schema_version"`
-	Payload       json.RawMessage    `json:"payload"`
-	SourceRefs    []source.SourceRef `json:"source_refs,omitempty"`
-	Metadata      map[string]any     `json:"metadata,omitempty"`
-}
-
-type Artifact struct {
-	ExtractorKey  string             `json:"extractor_key"`
-	ArtifactType  string             `json:"artifact_type"`
-	SchemaVersion string             `json:"schema_version"`
-	Payload       json.RawMessage    `json:"payload"`
-	SourceRefs    []source.SourceRef `json:"source_refs,omitempty"`
-	Metadata      map[string]any     `json:"metadata,omitempty"`
-}
-
-type RejectedArtifact struct {
-	Candidate     ArtifactCandidate `json:"candidate"`
-	ValidatorName string            `json:"validator_name"`
-	ReasonCode    string            `json:"reason_code"`
-	Message       string            `json:"message"`
-}
-
 type ArtifactLaneManifest struct {
 	ID         string         `json:"id"`
 	Extractor  string         `json:"extractor"`
 	Merger     string         `json:"merger"`
 	Normalizer string         `json:"normalizer"`
-	Validators []string       `json:"validators,omitempty"`
 	Metadata   map[string]any `json:"metadata,omitempty"`
 }
 
+type ValidatorChainManifest struct {
+	Stage      string              `json:"stage"`
+	LaneID     string              `json:"lane_id,omitempty"`
+	ModuleKey  string              `json:"module_key"`
+	Validators []ValidatorManifest `json:"validators"`
+}
+
+type ValidatorManifest struct {
+	Key            string `json:"key"`
+	ExecutionClass string `json:"execution_class"`
+}
+
 type LLMProfileManifest struct {
 	ID       string `json:"id"`
 	Provider string `json:"provider,omitempty"`
 	Model    string `json:"model,omitempty"`
 }
 
+type ReferenceProvenance struct {
+	Stage         string `json:"stage,omitempty"`
+	LaneID        string `json:"lane_id,omitempty"`
+	SlotName      string `json:"slot_name"`
+	OriginType    string `json:"origin_type"`
+	OriginURI     string `json:"origin_uri,omitempty"`
+	Digest        string `json:"digest,omitempty"`
+	MediaType     string `json:"media_type,omitempty"`
+	SizeBytes     int64  `json:"size_bytes,omitempty"`
+	BindingSource string `json:"binding_source,omitempty"`
+}
+
+type OutputSchemaProvenance struct {
+	ID      string `json:"id,omitempty"`
+	Name    string `json:"name,omitempty"`
+	Version string `json:"version,omitempty"`
+}
+
+type NormalizedOutputManifest struct {
+	LaneID    string                 `json:"lane_id"`
+	ModuleKey string                 `json:"module_key,omitempty"`
+	SourceID  string                 `json:"source_id,omitempty"`
+	MediaType string                 `json:"media_type,omitempty"`
+	Schema    OutputSchemaProvenance `json:"schema,omitempty"`
+}
+
+type RejectedOutputManifest struct {
+	Stage                  string `json:"stage"`
+	LaneID                 string `json:"lane_id,omitempty"`
+	ModuleKey              string `json:"module_key,omitempty"`
+	ChunkID                string `json:"chunk_id,omitempty"`
+	ChunkIndex             int    `json:"chunk_index,omitempty"`
+	ValidatorName          string `json:"validator_name,omitempty"`
+	ReasonCode             string `json:"reason_code,omitempty"`
+	Message                string `json:"message,omitempty"`
+	AttemptCount           int    `json:"attempt_count,omitempty"`
+	DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
+}
+
+type ChunkPlanManifest struct {
+	Mode                string                `json:"mode"`
+	Action              string                `json:"action,omitempty"`
+	SourceDigest        string                `json:"source_digest,omitempty"`
+	PlanDigest          string                `json:"plan_digest,omitempty"`
+	PlanSchemaVersion   string                `json:"plan_schema_version,omitempty"`
+	RequestedModule     string                `json:"requested_module"`
+	ProducerInputModule string                `json:"producer_input_module,omitempty"`
+	ProducerModule      string                `json:"producer_module,omitempty"`
+	ProducerLLMProfile  string                `json:"producer_llm_profile,omitempty"`
+	ProducerReferences  []ReferenceProvenance `json:"producer_references,omitempty"`
+	ProducerMetadata    map[string]any        `json:"producer_metadata,omitempty"`
+	CreatedAt           *time.Time            `json:"created_at,omitempty"`
+}
+
+// ChunkPlanSummary is deliberately limited to cache and validation decisions.
+// It must never contain plan units, source content, annotations, or model I/O.
+type ChunkPlanSummary struct {
+	Mode              string `json:"mode"`
+	SourceDigest      string `json:"source_digest,omitempty"`
+	CandidateDigest   string `json:"candidate_digest,omitempty"`
+	RequestedModule   string `json:"requested_module"`
+	LookupStatus      string `json:"lookup_status"`
+	LookupReason      string `json:"lookup_reason,omitempty"`
+	Action            string `json:"action,omitempty"`
+	ValidationStatus  string `json:"validation_status"`
+	PublicationStatus string `json:"publication_status"`
+}
+
 type RunManifest struct {
-	RunID            string                 `json:"run_id,omitempty"`
+	RunID             string                     `json:"run_id,omitempty"`
-	PipelineID       string                 `json:"pipeline_id,omitempty"`
+	PipelineID        string                     `json:"pipeline_id,omitempty"`
-	PipelineDigest   string                 `json:"pipeline_digest,omitempty"`
+	PipelineDigest    string                     `json:"pipeline_digest,omitempty"`
-	InputModule      string                 `json:"input_module,omitempty"`
+	InputModule       string                     `json:"input_module,omitempty"`
-	Chunker          string                 `json:"chunker,omitempty"`
+	Chunker           string                     `json:"chunker,omitempty"`
-	SourceDigests    []string               `json:"source_digests,omitempty"`
+	ChunkPlan         *ChunkPlanManifest         `json:"chunk_plan,omitempty"`
-	Extractors       []string               `json:"extractors,omitempty"`
+	SourceDigests     []string                   `json:"source_digests,omitempty"`
-	Merger           string                 `json:"merger,omitempty"`
+	Extractors        []string                   `json:"extractors,omitempty"`
-	Normalizer       string                 `json:"normalizer,omitempty"`
+	Merger            string                     `json:"merger,omitempty"`
-	OutputEncoder    string                 `json:"output_encoder,omitempty"`
+	Normalizer        string                     `json:"normalizer,omitempty"`
-	ArtifactLanes    []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
+	OutputEncoder     string                     `json:"output_encoder,omitempty"`
-	LLMProfiles      []LLMProfileManifest   `json:"llm_profiles,omitempty"`
+	ModuleMetadata    map[string]map[string]any  `json:"module_metadata,omitempty"`
-	SchemaVersion    string                 `json:"schema_version,omitempty"`
+	ArtifactLanes     []ArtifactLaneManifest     `json:"artifact_lanes,omitempty"`
-	ValidationStatus string                 `json:"validation_status,omitempty"`
+	ValidatorChains   []ValidatorChainManifest   `json:"validator_chains,omitempty"`
-	StartedAt        *time.Time             `json:"started_at,omitempty"`
+	References        []ReferenceProvenance      `json:"references,omitempty"`
-	CompletedAt      *time.Time             `json:"completed_at,omitempty"`
+	NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
-}
+	RejectedOutputs   []RejectedOutputManifest   `json:"rejected_outputs,omitempty"`
-
+	LLMProfiles       []LLMProfileManifest       `json:"llm_profiles,omitempty"`
-func ArtifactFromCandidate(candidate ArtifactCandidate) Artifact {
+	Metadata          map[string]any             `json:"metadata,omitempty"`
-	return Artifact{
+	SchemaVersion     string                     `json:"schema_version,omitempty"`
-		ExtractorKey:  candidate.ExtractorKey,
+	ValidationStatus  string                     `json:"validation_status,omitempty"`
-		ArtifactType:  candidate.ArtifactType,
+	StartedAt         *time.Time                 `json:"started_at,omitempty"`
-		SchemaVersion: candidate.SchemaVersion,
+	CompletedAt       *time.Time                 `json:"completed_at,omitempty"`
-		Payload:       append(json.RawMessage(nil), candidate.Payload...),
-		SourceRefs:    append([]source.SourceRef(nil), candidate.SourceRefs...),
-		Metadata:      copyMetadata(candidate.Metadata),
-	}
-}
-
-func copyMetadata(metadata map[string]any) map[string]any {
-	if len(metadata) == 0 {
-		return nil
-	}
-
-	copied := make(map[string]any, len(metadata))
-	for key, value := range metadata {
-		copied[key] = value
-	}
-	return copied
 }

internal/core/artifacts/artifacts_test.go

@@ -2,116 +2,10 @@ package artifacts
 
 import (
 	"encoding/json"
-	"reflect"
+	"strings"
 	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
 )
 
-func TestArtifactFromCandidatePreservesCandidateFields(t *testing.T) {
-	candidate := ArtifactCandidate{
-		Index:         7,
-		ExtractorKey:  "generic-extractor",
-		ArtifactType:  "generic-artifact",
-		SchemaVersion: "v1",
-		Payload:       json.RawMessage(`{"name":"example"}`),
-		SourceRefs: []source.SourceRef{
-			{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u2"},
-		},
-		Metadata: map[string]any{
-			"confidence": 0.75,
-		},
-	}
-
-	artifact := ArtifactFromCandidate(candidate)
-
-	if artifact.ExtractorKey != candidate.ExtractorKey {
-		t.Fatalf("ExtractorKey = %q, want %q", artifact.ExtractorKey, candidate.ExtractorKey)
-	}
-	if artifact.ArtifactType != candidate.ArtifactType {
-		t.Fatalf("ArtifactType = %q, want %q", artifact.ArtifactType, candidate.ArtifactType)
-	}
-	if artifact.SchemaVersion != candidate.SchemaVersion {
-		t.Fatalf("SchemaVersion = %q, want %q", artifact.SchemaVersion, candidate.SchemaVersion)
-	}
-	if string(artifact.Payload) != string(candidate.Payload) {
-		t.Fatalf("Payload = %s, want %s", artifact.Payload, candidate.Payload)
-	}
-	if !reflect.DeepEqual(artifact.SourceRefs, candidate.SourceRefs) {
-		t.Fatalf("SourceRefs = %#v, want %#v", artifact.SourceRefs, candidate.SourceRefs)
-	}
-	if !reflect.DeepEqual(artifact.Metadata, candidate.Metadata) {
-		t.Fatalf("Metadata = %#v, want %#v", artifact.Metadata, candidate.Metadata)
-	}
-
-	candidate.Payload[0] = '['
-	candidate.SourceRefs[0].StartUnitID = "changed"
-	candidate.Metadata["confidence"] = 0.5
-
-	if string(artifact.Payload) != `{"name":"example"}` {
-		t.Fatalf("Payload changed after candidate mutation: %s", artifact.Payload)
-	}
-	if artifact.SourceRefs[0].StartUnitID != "u1" {
-		t.Fatalf("SourceRefs changed after candidate mutation: %#v", artifact.SourceRefs)
-	}
-	if artifact.Metadata["confidence"] != 0.75 {
-		t.Fatalf("Metadata changed after candidate mutation: %#v", artifact.Metadata)
-	}
-}
-
-func TestJSONMarshalUsesExpectedFieldNames(t *testing.T) {
-	candidate := ArtifactCandidate{
-		Index:         1,
-		ExtractorKey:  "generic-extractor",
-		ArtifactType:  "generic-artifact",
-		SchemaVersion: "v1",
-		Payload:       json.RawMessage(`{"value":true}`),
-		SourceRefs: []source.SourceRef{
-			{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
-		},
-		Metadata: map[string]any{
-			"reviewed": true,
-		},
-	}
-	rejected := RejectedArtifact{
-		Candidate:     candidate,
-		ValidatorName: "generic-validator",
-		ReasonCode:    "invalid",
-		Message:       "candidate was not accepted",
-	}
-
-	gotJSON, err := json.Marshal(rejected)
-	if err != nil {
-		t.Fatalf("json.Marshal() error = %v", err)
-	}
-
-	var got map[string]any
-	if err := json.Unmarshal(gotJSON, &got); err != nil {
-		t.Fatalf("json.Unmarshal() error = %v", err)
-	}
-
-	assertHasKeys(t, got, "candidate", "validator_name", "reason_code", "message")
-
-	gotCandidate, ok := got["candidate"].(map[string]any)
-	if !ok {
-		t.Fatalf("candidate = %#v, want object", got["candidate"])
-	}
-	assertHasKeys(t, gotCandidate, "index", "extractor_key", "artifact_type", "schema_version", "payload", "source_refs", "metadata")
-
-	gotRefs, ok := gotCandidate["source_refs"].([]any)
-	if !ok {
-		t.Fatalf("source_refs = %#v, want array", gotCandidate["source_refs"])
-	}
-	if len(gotRefs) != 1 {
-		t.Fatalf("len(source_refs) = %d, want 1", len(gotRefs))
-	}
-	gotRef, ok := gotRefs[0].(map[string]any)
-	if !ok {
-		t.Fatalf("source_refs[0] = %#v, want object", gotRefs[0])
-	}
-	assertHasKeys(t, gotRef, "source_id", "start_unit_id", "end_unit_id")
-}
-
 func TestRunManifestOmitsEmptyOptionalFields(t *testing.T) {
 	gotJSON, err := json.Marshal(RunManifest{})
 	if err != nil {
@@ -123,12 +17,43 @@ func TestRunManifestOmitsEmptyOptionalFields(t *testing.T) {
 	}
 }
 
+func TestRunManifestChunkPlanIsAdditiveAndOmitsPlanContent(t *testing.T) {
+	manifest := RunManifest{ChunkPlan: &ChunkPlanManifest{
+		Mode: "auto", Action: "reused", SourceDigest: "sha256:source", PlanDigest: "sha256:plan",
+		PlanSchemaVersion: "notarius.chunk-plan.v1", RequestedModule: "chunk/current",
+		ProducerInputModule: "input/original", ProducerModule: "chunk/original",
+	}}
+	encoded, err := json.Marshal(manifest)
+	if err != nil {
+		t.Fatal(err)
+	}
+	text := string(encoded)
+	for _, want := range []string{`"chunk_plan"`, `"action":"reused"`, `"requested_module":"chunk/current"`, `"producer_module":"chunk/original"`} {
+		if !strings.Contains(text, want) {
+			t.Fatalf("manifest JSON %s does not contain %s", text, want)
+		}
+	}
+	for _, forbidden := range []string{`"plan"`, `"units"`, `"annotations"`} {
+		if strings.Contains(text, forbidden) {
+			t.Fatalf("manifest JSON contains forbidden field %s: %s", forbidden, text)
+		}
+	}
+
+	var legacy RunManifest
+	if err := json.Unmarshal([]byte(`{"pipeline_id":"legacy"}`), &legacy); err != nil {
+		t.Fatal(err)
+	}
+	if legacy.PipelineID != "legacy" || legacy.ChunkPlan != nil {
+		t.Fatalf("legacy manifest = %#v", legacy)
+	}
+}
+
 func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
 	manifest := RunManifest{
 		PipelineID:     "pipeline-1",
 		PipelineDigest: "sha256:abc123",
 		LLMProfiles: []LLMProfileManifest{
-			{ID: "default", Provider: "openai-compatible", Model: "model-a"},
+			{ID: "default", Provider: "scriptorium", Model: "model-a"},
 		},
 		ArtifactLanes: []ArtifactLaneManifest{
 			{
@@ -136,12 +61,21 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
 				Extractor:  "event-extractor",
 				Merger:     "appendorder",
 				Normalizer: "noop",
-				Validators: []string{"grounded"},
 				Metadata: map[string]any{
 					"extractor": map[string]any{"prompt_id": "test.prompt"},
 				},
 			},
 		},
+		ValidatorChains: []ValidatorChainManifest{
+			{
+				Stage:     "extract",
+				LaneID:    "events",
+				ModuleKey: "event-extractor",
+				Validators: []ValidatorManifest{
+					{Key: "grounded", ExecutionClass: "deterministic"},
+				},
+			},
+		},
 	}
 
 	gotJSON, err := json.Marshal(manifest)
@@ -154,7 +88,7 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
 		t.Fatalf("json.Unmarshal() error = %v", err)
 	}
 
-	assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes", "llm_profiles")
+	assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes", "validator_chains", "llm_profiles")
 
 	profiles, ok := got["llm_profiles"].([]any)
 	if !ok {
@@ -180,7 +114,94 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
 	if !ok {
 		t.Fatalf("artifact_lanes[0] = %#v, want object", lanes[0])
 	}
-	assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "validators", "metadata")
+	assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "metadata")
+
+	chains, ok := got["validator_chains"].([]any)
+	if !ok {
+		t.Fatalf("validator_chains = %#v, want array", got["validator_chains"])
+	}
+	if len(chains) != 1 {
+		t.Fatalf("len(validator_chains) = %d, want 1", len(chains))
+	}
+	chain, ok := chains[0].(map[string]any)
+	if !ok {
+		t.Fatalf("validator_chains[0] = %#v, want object", chains[0])
+	}
+	assertHasKeys(t, chain, "stage", "lane_id", "module_key", "validators")
+}
+
+func TestRunManifestIncludesReferenceProvenance(t *testing.T) {
+	manifest := RunManifest{
+		References: []ReferenceProvenance{
+			{
+				Stage:         "extract",
+				LaneID:        "events",
+				SlotName:      "roster",
+				OriginType:    "file",
+				OriginURI:     "file:///tmp/roster.txt",
+				Digest:        "sha256:reference",
+				MediaType:     "text/plain; charset=utf-8",
+				SizeBytes:     12,
+				BindingSource: "config",
+			},
+		},
+	}
+
+	gotJSON, err := json.Marshal(manifest)
+	if err != nil {
+		t.Fatalf("json.Marshal() error = %v", err)
+	}
+
+	var got RunManifest
+	if err := json.Unmarshal(gotJSON, &got); err != nil {
+		t.Fatalf("json.Unmarshal() error = %v", err)
+	}
+	if len(got.References) != 1 {
+		t.Fatalf("len(References) = %d, want 1", len(got.References))
+	}
+	reference := got.References[0]
+	if reference.Stage != "extract" || reference.LaneID != "events" || reference.SlotName != "roster" || reference.OriginType != "file" || reference.OriginURI != "file:///tmp/roster.txt" {
+		t.Fatalf("reference provenance = %#v, want lane-scoped origin details", reference)
+	}
+	if reference.Digest != "sha256:reference" || reference.MediaType != "text/plain; charset=utf-8" || reference.SizeBytes != 12 || reference.BindingSource != "config" {
+		t.Fatalf("reference provenance = %#v, want digest/media/size/source details", reference)
+	}
+}
+
+func TestRunManifestIncludesTopLevelModuleMetadata(t *testing.T) {
+	manifest := RunManifest{
+		ModuleMetadata: map[string]map[string]any{
+			"chunker": {
+				"prompt_id":            "dnd.scenes",
+				"prompt_version":       "v1",
+				"prompt_sha256":        "sha256:abc123",
+				"response_schema_key":  "dnd_scenes",
+				"response_schema_name": "dnd_scenes",
+			},
+		},
+	}
+
+	gotJSON, err := json.Marshal(manifest)
+	if err != nil {
+		t.Fatalf("json.Marshal() error = %v", err)
+	}
+
+	var got map[string]any
+	if err := json.Unmarshal(gotJSON, &got); err != nil {
+		t.Fatalf("json.Unmarshal() error = %v", err)
+	}
+
+	moduleMetadata, ok := got["module_metadata"].(map[string]any)
+	if !ok {
+		t.Fatalf("module_metadata = %#v, want object", got["module_metadata"])
+	}
+	assertHasKeys(t, moduleMetadata, "chunker")
+
+	chunkerMetadata, ok := moduleMetadata["chunker"].(map[string]any)
+	if !ok {
+		t.Fatalf("module_metadata.chunker = %#v, want object", moduleMetadata["chunker"])
+	}
+	assertHasKeys(t, chunkerMetadata, "prompt_id", "prompt_version", "prompt_sha256", "response_schema_key", "response_schema_name")
 }
 
 func assertHasKeys(t *testing.T, values map[string]any, keys ...string) {

internal/core/config/cache.go

@@ -0,0 +1,31 @@
+package config
+
+import (
+	"fmt"
+	"path/filepath"
+	"strings"
+)
+
+// DefaultChunkPlanRoot resolves the existing per-user chunk-plan cache root.
+func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error) {
+	return defaultCacheFamilyRoot(userCacheDir, "chunk-plans")
+}
+
+func DefaultCheckpointRoot(userCacheDir func() (string, error)) (string, error) {
+	return defaultCacheFamilyRoot(userCacheDir, "checkpoints")
+}
+
+func defaultCacheFamilyRoot(userCacheDir func() (string, error), family string) (string, error) {
+	if userCacheDir == nil {
+		return "", fmt.Errorf("user cache directory resolver must not be nil")
+	}
+	root, err := userCacheDir()
+	if err != nil {
+		return "", fmt.Errorf("resolve user cache directory: %w", err)
+	}
+	root = strings.TrimSpace(root)
+	if root == "" {
+		return "", fmt.Errorf("user cache directory must not be empty")
+	}
+	return filepath.Join(filepath.Clean(root), "notarius", family), nil
+}

internal/core/config/config.go

@@ -1,66 +1,71 @@
 package config
 
 import (
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
 )
 
-const SupportedFileConfigVersion = 1
+const SupportedFileConfigVersion = 3
 
 type Config struct {
-	LLMProfiles map[string]LLMProfile               `json:"llm_profiles"`
+	Scriptorium ScriptoriumConfig                   `json:"scriptorium,omitempty"`
 	Pipelines   map[string]pipeline.PipelineProfile `json:"pipelines"`
 	Concurrency ConcurrencyConfig                   `json:"concurrency"`
-	Diagnostics DiagnosticsConfig                   `json:"diagnostics"`
+	Output      OutputConfig                        `json:"output"`
+	Cache       CacheConfig                         `json:"cache"`
+	Debug       DebugConfig                         `json:"debug"`
 }
 
-type LLMProfile struct {
+type ScriptoriumConfig struct {
-	Provider       string `json:"provider,omitempty"`
+	ProfileDir  string `json:"profile_dir,omitempty"`
-	BaseURL        string `json:"base_url,omitempty"`
+	ProfileFile string `json:"profile_file,omitempty"`
-	Model          string `json:"model,omitempty"`
-	APIKey         string `json:"api_key,omitempty"`
-	APIKeyEnv      string `json:"api_key_env,omitempty"`
-	TimeoutSeconds int    `json:"timeout_seconds,omitempty"`
-	MaxRetries     int    `json:"max_retries,omitempty"`
-	MaxConcurrency int    `json:"max_concurrency,omitempty"`
 }
 
 type ConcurrencyConfig struct {
-	TotalLLM int `json:"total_llm"`
+	TotalLLM     int            `json:"total_llm"`
+	StageWorkers map[string]int `json:"stage_workers"`
+
+	extractWorkersConfigured bool
+	defaultedExtractWorkers  int
 }
 
-type DiagnosticsConfig struct {
+type OutputConfig struct {
-	WorkDir   string                    `json:"work_dir"`
+	Directory string `json:"directory"`
-	Retention diagnostics.RetentionMode `json:"retention"`
+}
+
+type CacheConfig struct {
+	ChunkPlans  ChunkPlanCacheConfig  `json:"chunk_plans"`
+	Checkpoints CheckpointCacheConfig `json:"checkpoints"`
+}
+
+type ChunkPlanCacheConfig struct {
+	Directory string                  `json:"directory,omitempty"`
+	Mode      pipeline.ChunkCacheMode `json:"mode"`
+}
+
+type CheckpointCacheConfig struct {
+	Directory string `json:"directory,omitempty"`
+}
+type DebugConfig struct {
+	Directory string `json:"directory"`
 }
 
 func Default() Config {
 	return Config{
-		LLMProfiles: map[string]LLMProfile{
-			pipeline.DefaultLLMProfile: {
-				Provider:       "openai-compatible",
-				TimeoutSeconds: 600,
-				MaxRetries:     3,
-				MaxConcurrency: 1,
-			},
-		},
 		Pipelines: map[string]pipeline.PipelineProfile{},
 		Concurrency: ConcurrencyConfig{
-			TotalLLM: 1,
+			TotalLLM:                1,
-		},
+			StageWorkers:            map[string]int{"extract": 1},
-		Diagnostics: DiagnosticsConfig{
+			defaultedExtractWorkers: 1,
-			WorkDir:   "/tmp/notarius",
-			Retention: diagnostics.RetentionAuto,
 		},
+		Output: OutputConfig{Directory: "./notarius-output"},
+		Cache:  CacheConfig{ChunkPlans: ChunkPlanCacheConfig{Mode: pipeline.ChunkCacheAuto}},
+		Debug:  DebugConfig{Directory: "./notarius-debug"},
 	}
 }
 
 func cloneConfig(in Config) Config {
 	out := in
-	out.LLMProfiles = make(map[string]LLMProfile, len(in.LLMProfiles))
+	out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
-	for key, profile := range in.LLMProfiles {
-		out.LLMProfiles[key] = profile
-	}
 	out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
 	for key, profile := range in.Pipelines {
 		out.Pipelines[key] = clonePipelineProfile(profile)
@@ -68,11 +73,40 @@ func cloneConfig(in Config) Config {
 	return out
 }
 
+func cloneIntMap(in map[string]int) map[string]int {
+	if len(in) == 0 {
+		return nil
+	}
+	out := make(map[string]int, len(in))
+	for key, value := range in {
+		out[key] = value
+	}
+	return out
+}
+
+func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
+	if c == nil {
+		return
+	}
+	if c.StageWorkers == nil {
+		c.StageWorkers = make(map[string]int)
+	}
+	if !c.extractWorkersConfigured {
+		if value, ok := c.StageWorkers["extract"]; ok && (c.defaultedExtractWorkers == 0 || value != c.defaultedExtractWorkers) {
+			c.extractWorkersConfigured = true
+			return
+		}
+		c.StageWorkers["extract"] = c.TotalLLM
+		c.defaultedExtractWorkers = c.TotalLLM
+	}
+}
+
 func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
 	out := in
 	out.Input = cloneModuleBinding(in.Input)
 	out.Chunk = cloneModuleBinding(in.Chunk)
 	out.Output = cloneModuleBinding(in.Output)
+	out.References = cloneStringMap(in.References)
 	if len(in.Artifacts) > 0 {
 		out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
 		for key, lane := range in.Artifacts {
@@ -87,6 +121,7 @@ func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.Artifact
 	out.Extract = cloneModuleBinding(in.Extract)
 	out.Merge = cloneModuleBinding(in.Merge)
 	out.Normalize = cloneModuleBinding(in.Normalize)
+	out.References = cloneStringMap(in.References)
 	if len(in.Validators) > 0 {
 		out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
 		for i, binding := range in.Validators {
@@ -96,11 +131,35 @@ func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.Artifact
 	return out
 }
 
+func cloneStringMap(in map[string]string) map[string]string {
+	if len(in) == 0 {
+		return nil
+	}
+	out := make(map[string]string, len(in))
+	for key, value := range in {
+		out[key] = value
+	}
+	return out
+}
+
 func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
 	out := in
 	if len(in.Options) > 0 {
 		out.Options = cloneOptions(in.Options)
 	}
+	out.References = cloneStringMap(in.References)
+	out.Validators = cloneValidatorOverride(in.Validators)
+	return out
+}
+
+func cloneValidatorOverride(in pipeline.ValidatorOverride) pipeline.ValidatorOverride {
+	out := pipeline.ValidatorOverride{Set: in.Set}
+	if len(in.Validators) > 0 {
+		out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
+		for i, binding := range in.Validators {
+			out.Validators[i] = cloneModuleBinding(binding)
+		}
+	}
 	return out
 }
 

internal/core/config/config_test.go

@@ -1,78 +0,0 @@
-package config
-
-import (
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
-)
-
-func TestDefaultValues(t *testing.T) {
-	cfg := Default()
-
-	defaultProfile, ok := cfg.LLMProfiles[pipeline.DefaultLLMProfile]
-	if !ok {
-		t.Fatalf("expected default LLM profile")
-	}
-	if defaultProfile.Provider != "openai-compatible" {
-		t.Fatalf("unexpected provider: %q", defaultProfile.Provider)
-	}
-	if defaultProfile.BaseURL != "" || defaultProfile.Model != "" {
-		t.Fatalf("default profile should not require base URL/model yet: %+v", defaultProfile)
-	}
-	if defaultProfile.TimeoutSeconds != 600 || defaultProfile.MaxRetries != 3 || defaultProfile.MaxConcurrency != 1 {
-		t.Fatalf("unexpected default LLM operational values: %+v", defaultProfile)
-	}
-	if len(cfg.Pipelines) != 0 {
-		t.Fatalf("expected no built-in pipeline profiles, got %v", cfg.Pipelines)
-	}
-	if cfg.Concurrency.TotalLLM != 1 {
-		t.Fatalf("unexpected total LLM concurrency: %d", cfg.Concurrency.TotalLLM)
-	}
-	if cfg.Diagnostics.WorkDir != "/tmp/notarius" {
-		t.Fatalf("unexpected diagnostics work dir: %q", cfg.Diagnostics.WorkDir)
-	}
-	if cfg.Diagnostics.Retention != diagnostics.RetentionAuto {
-		t.Fatalf("unexpected diagnostics retention: %q", cfg.Diagnostics.Retention)
-	}
-}
-
-func TestApplyFileConfigMergesWithDefaults(t *testing.T) {
-	fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-llm_profiles:
-  default:
-    model: test-model
-pipelines:
-  example:
-    input: fake/input
-    artifacts:
-      events:
-        extract: fake/extract
-`))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-
-	cfg := Default()
-	if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
-		t.Fatalf("ApplyFileConfig: %v", err)
-	}
-
-	profile := cfg.LLMProfiles[pipeline.DefaultLLMProfile]
-	if profile.Model != "test-model" {
-		t.Fatalf("expected file model, got %+v", profile)
-	}
-	if profile.Provider != "openai-compatible" || profile.TimeoutSeconds != 600 || profile.MaxRetries != 3 {
-		t.Fatalf("expected default LLM fields to be preserved, got %+v", profile)
-	}
-	if cfg.Concurrency.TotalLLM != 1 {
-		t.Fatalf("expected default concurrency preserved, got %d", cfg.Concurrency.TotalLLM)
-	}
-	if cfg.Diagnostics.Retention != diagnostics.RetentionAuto {
-		t.Fatalf("expected default diagnostics retention preserved, got %q", cfg.Diagnostics.Retention)
-	}
-	if _, ok := cfg.Pipelines["example"]; !ok {
-		t.Fatalf("expected file pipeline to be applied")
-	}
-}

internal/core/config/effective_config.go

@@ -3,9 +3,7 @@ package config
 import (
 	"fmt"
 	"strings"
-	"time"
 
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
 )
 
@@ -14,16 +12,21 @@ type ResolveInput struct {
 	Only               []string
 	Catalog            pipeline.ModuleCatalog
 	LLMProfileOverride string
+	ReferenceOverrides []pipeline.ReferenceBinding
+	ReferenceUnbinds   []pipeline.ReferenceUnbind
 }
 
 type EffectiveConfig struct {
-	Config           Config
+	Config             Config
-	PipelineID       string
+	PipelineID         string
-	Only             []string
+	Only               []string
-	ResolvedPipeline pipeline.ResolvedPipeline
+	ReferenceOverrides []pipeline.ReferenceBinding
+	ReferenceUnbinds   []pipeline.ReferenceUnbind
+	ResolvedPipeline   pipeline.ResolvedPipeline
 }
 
 func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
+	c.Concurrency.recomputeStageWorkerDefaults()
 	if err := c.Validate(); err != nil {
 		return EffectiveConfig{}, err
 	}
@@ -40,36 +43,34 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
 	profile = clonePipelineProfile(profile)
 	profile.ID = pipelineID
 	if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
-		if !hasLLMProfile(c.LLMProfiles, override) {
-			return EffectiveConfig{}, fmt.Errorf("LLM profile override %q is not configured", override)
-		}
 		applyLLMProfileOverride(&profile, override)
 	}
 
-	resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{Only: input.Only}, input.Catalog)
+	resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
+		Only:               input.Only,
+		ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
+		ReferenceUnbinds:   append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
+	}, input.Catalog)
 	if err != nil {
 		return EffectiveConfig{}, fmt.Errorf("resolve pipeline %q: %w", pipelineID, err)
 	}
 
 	return EffectiveConfig{
-		Config:           cloneConfig(c),
+		Config:             cloneConfig(c),
-		PipelineID:       pipelineID,
+		PipelineID:         pipelineID,
-		Only:             append([]string(nil), input.Only...),
+		Only:               append([]string(nil), input.Only...),
-		ResolvedPipeline: resolved,
+		ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
+		ReferenceUnbinds:   append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
+		ResolvedPipeline:   resolved,
 	}, nil
 }
 
 func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
-	profile.Input.LLMProfile = profileID
 	profile.Chunk.LLMProfile = profileID
-	profile.Output.LLMProfile = profileID
 	for laneID, lane := range profile.Artifacts {
 		lane.Extract.LLMProfile = profileID
 		lane.Merge.LLMProfile = profileID
 		lane.Normalize.LLMProfile = profileID
-		for i := range lane.Validators {
-			lane.Validators[i].LLMProfile = profileID
-		}
 		profile.Artifacts[laneID] = lane
 	}
 }
@@ -83,36 +84,3 @@ func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelin
 	}
 	return pipeline.PipelineProfile{}, false
 }
-
-func (c Config) OpenAICompatibleClientConfig(profileID string) (llm.OpenAICompatibleClientConfig, error) {
-	trimmedID := strings.TrimSpace(profileID)
-	profile, ok := c.LLMProfile(trimmedID)
-	if !ok {
-		return llm.OpenAICompatibleClientConfig{}, fmt.Errorf("LLM profile %q is not configured", trimmedID)
-	}
-
-	provider := strings.TrimSpace(profile.Provider)
-	if provider == "" {
-		provider = providerOpenAICompatible
-	}
-	if provider != providerOpenAICompatible {
-		return llm.OpenAICompatibleClientConfig{}, fmt.Errorf("LLM profile %q provider %q is not supported", trimmedID, provider)
-	}
-
-	baseURL := strings.TrimSpace(profile.BaseURL)
-	if baseURL == "" {
-		return llm.OpenAICompatibleClientConfig{}, fmt.Errorf("LLM profile %q base URL must not be empty", trimmedID)
-	}
-	model := strings.TrimSpace(profile.Model)
-	if model == "" {
-		return llm.OpenAICompatibleClientConfig{}, fmt.Errorf("LLM profile %q model must not be empty", trimmedID)
-	}
-
-	return llm.OpenAICompatibleClientConfig{
-		BaseURL:        baseURL,
-		Model:          model,
-		APIKey:         profile.APIKey,
-		MaxRetries:     profile.MaxRetries,
-		RequestTimeout: time.Duration(profile.TimeoutSeconds) * time.Second,
-	}, nil
-}

internal/core/config/effective_config_test.go

@@ -1,214 +0,0 @@
-package config
-
-import (
-	"strings"
-	"testing"
-	"time"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
-)
-
-func TestResolveRejectsEmptyAndUnknownPipelineID(t *testing.T) {
-	tests := []struct {
-		name       string
-		pipelineID string
-		want       string
-	}{
-		{name: "empty", pipelineID: " ", want: "pipeline id"},
-		{name: "unknown", pipelineID: "missing", want: "not configured"},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			_, err := validConfig().Resolve(ResolveInput{PipelineID: tc.pipelineID, Catalog: fakeCatalog(t)})
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestResolveLaneFilteringSuccessAndFailure(t *testing.T) {
-	effective, err := validConfig().Resolve(ResolveInput{
-		PipelineID: " example ",
-		Only:       []string{" notes "},
-		Catalog:    fakeCatalog(t),
-	})
-	if err != nil {
-		t.Fatalf("Resolve: %v", err)
-	}
-
-	if effective.PipelineID != "example" {
-		t.Fatalf("unexpected pipeline ID: %q", effective.PipelineID)
-	}
-	if len(effective.ResolvedPipeline.ArtifactLanes) != 1 || effective.ResolvedPipeline.ArtifactLanes[0].ID != "notes" {
-		t.Fatalf("unexpected resolved lanes: %+v", effective.ResolvedPipeline.ArtifactLanes)
-	}
-	if effective.ResolvedPipeline.Digest == "" {
-		t.Fatalf("expected digest")
-	}
-
-	_, err = validConfig().Resolve(ResolveInput{
-		PipelineID: "example",
-		Only:       []string{"missing"},
-		Catalog:    fakeCatalog(t),
-	})
-	if err == nil || !strings.Contains(err.Error(), "selected artifact lane") {
-		t.Fatalf("expected invalid lane error, got %v", err)
-	}
-}
-
-func TestResolveUsesTrimmedPipelineMapKeys(t *testing.T) {
-	cfg := validConfig()
-	cfg.Pipelines[" example "] = cfg.Pipelines["example"]
-	delete(cfg.Pipelines, "example")
-
-	effective, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
-	if err != nil {
-		t.Fatalf("Resolve: %v", err)
-	}
-	if effective.PipelineID != "example" {
-		t.Fatalf("unexpected pipeline ID: %q", effective.PipelineID)
-	}
-}
-
-func TestResolveSurfacesUnknownModuleKeyThroughCatalog(t *testing.T) {
-	cfg := validConfig()
-	lane := cfg.Pipelines["example"].Artifacts["events"]
-	lane.Extract = pipeline.Binding("missing/extract")
-	cfg.Pipelines["example"].Artifacts["events"] = lane
-
-	_, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
-	if err == nil || !strings.Contains(err.Error(), "missing/extract") || !strings.Contains(err.Error(), "events") {
-		t.Fatalf("expected unknown module error with lane context, got %v", err)
-	}
-}
-
-func TestResolveSurfacesMissingCapabilityThroughCatalog(t *testing.T) {
-	_, err := validConfig().Resolve(ResolveInput{
-		PipelineID: "example",
-		Catalog: fakeCatalog(t, pipeline.ModuleSpec{
-			Key:      "json",
-			Stage:    pipeline.StageOutput,
-			Requires: []string{"missing-capability"},
-		}),
-	})
-	if err == nil || !strings.Contains(err.Error(), "missing capability") || !strings.Contains(err.Error(), "json") {
-		t.Fatalf("expected missing capability error, got %v", err)
-	}
-}
-
-func TestResolveDigestChangesWhenEffectiveConfigChanges(t *testing.T) {
-	cfg := validConfig()
-	first, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
-	if err != nil {
-		t.Fatalf("Resolve first: %v", err)
-	}
-
-	lane := cfg.Pipelines["example"].Artifacts["events"]
-	lane.Extract.Options = map[string]any{"temperature": 0.2}
-	cfg.Pipelines["example"].Artifacts["events"] = lane
-	second, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
-	if err != nil {
-		t.Fatalf("Resolve second: %v", err)
-	}
-
-	if first.ResolvedPipeline.Digest == second.ResolvedPipeline.Digest {
-		t.Fatalf("expected digest to change, got %q", first.ResolvedPipeline.Digest)
-	}
-}
-
-func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
-	cfg := validConfig()
-	cfg.LLMProfiles["runtime"] = LLMProfile{Provider: "openai-compatible"}
-
-	base, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
-	if err != nil {
-		t.Fatalf("Resolve base: %v", err)
-	}
-	effective, err := cfg.Resolve(ResolveInput{
-		PipelineID:         "example",
-		Catalog:            fakeCatalog(t),
-		LLMProfileOverride: "runtime",
-	})
-	if err != nil {
-		t.Fatalf("Resolve override: %v", err)
-	}
-
-	if base.ResolvedPipeline.Digest == effective.ResolvedPipeline.Digest {
-		t.Fatalf("expected digest to change after LLM profile override")
-	}
-	for _, binding := range resolvedBindings(effective.ResolvedPipeline) {
-		if binding.LLMProfile != "runtime" {
-			t.Fatalf("binding profile = %q, want runtime", binding.LLMProfile)
-		}
-	}
-
-	_, err = cfg.Resolve(ResolveInput{
-		PipelineID:         "example",
-		Catalog:            fakeCatalog(t),
-		LLMProfileOverride: "missing",
-	})
-	if err == nil || !strings.Contains(err.Error(), "LLM profile override") {
-		t.Fatalf("expected override profile error, got %v", err)
-	}
-}
-
-func resolvedBindings(resolved pipeline.ResolvedPipeline) []pipeline.ModuleBinding {
-	bindings := []pipeline.ModuleBinding{resolved.Input, resolved.Chunk, resolved.Output}
-	for _, lane := range resolved.ArtifactLanes {
-		bindings = append(bindings, lane.Extract, lane.Merge, lane.Normalize)
-		bindings = append(bindings, lane.Validators...)
-	}
-	return bindings
-}
-
-func TestOpenAICompatibleClientConfigRejectsIncompleteDefaultProfile(t *testing.T) {
-	cfg := Default()
-
-	_, err := cfg.OpenAICompatibleClientConfig("default")
-	if err == nil || !strings.Contains(err.Error(), "base URL") {
-		t.Fatalf("expected incomplete profile error, got %v", err)
-	}
-}
-
-func TestOpenAICompatibleClientConfigSuccess(t *testing.T) {
-	cfg := validConfig()
-	profile := cfg.LLMProfiles["default"]
-	profile.APIKey = "secret"
-	profile.TimeoutSeconds = 45
-	profile.MaxRetries = 4
-	cfg.LLMProfiles["default"] = profile
-
-	llmCfg, err := cfg.OpenAICompatibleClientConfig(" default ")
-	if err != nil {
-		t.Fatalf("OpenAICompatibleClientConfig: %v", err)
-	}
-
-	if llmCfg.BaseURL != "https://example.invalid/v1" || llmCfg.Model != "test-model" || llmCfg.APIKey != "secret" {
-		t.Fatalf("unexpected client config strings: %+v", llmCfg)
-	}
-	if llmCfg.MaxRetries != 4 {
-		t.Fatalf("unexpected max retries: %d", llmCfg.MaxRetries)
-	}
-	if llmCfg.RequestTimeout != 45*time.Second {
-		t.Fatalf("unexpected timeout: %s", llmCfg.RequestTimeout)
-	}
-}
-
-func TestOpenAICompatibleClientConfigRejectsUnknownAndUnsupportedProfiles(t *testing.T) {
-	_, err := validConfig().OpenAICompatibleClientConfig("missing")
-	if err == nil || !strings.Contains(err.Error(), "not configured") {
-		t.Fatalf("expected unknown profile error, got %v", err)
-	}
-
-	cfg := validConfig()
-	profile := cfg.LLMProfiles["default"]
-	profile.Provider = "unsupported"
-	cfg.LLMProfiles["default"] = profile
-
-	_, err = cfg.OpenAICompatibleClientConfig("default")
-	if err == nil || !strings.Contains(err.Error(), "provider") {
-		t.Fatalf("expected unsupported provider error, got %v", err)
-	}
-}

internal/core/config/env.go

@@ -6,7 +6,6 @@ import (
 	"strconv"
 	"strings"
 
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
 )
 
@@ -30,43 +29,6 @@ func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool))
 	if c == nil {
 		return fmt.Errorf("config must not be nil")
 	}
-	if c.LLMProfiles == nil {
-		c.LLMProfiles = map[string]LLMProfile{}
-	}
-
-	defaultProfile := c.LLMProfiles[pipeline.DefaultLLMProfile]
-	if raw, ok := lookup("NOTARIUS_LLM_DEFAULT_API_KEY"); ok {
-		defaultProfile.APIKey = raw
-	}
-	if raw, ok := lookup("NOTARIUS_LLM_DEFAULT_BASE_URL"); ok {
-		defaultProfile.BaseURL = strings.TrimSpace(raw)
-	}
-	if raw, ok := lookup("NOTARIUS_LLM_DEFAULT_MODEL"); ok {
-		defaultProfile.Model = strings.TrimSpace(raw)
-	}
-	if raw, ok := lookup("NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS"); ok {
-		value, err := parseIntEnv("NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS", raw)
-		if err != nil {
-			return err
-		}
-		defaultProfile.TimeoutSeconds = value
-	}
-	if raw, ok := lookup("NOTARIUS_LLM_DEFAULT_MAX_RETRIES"); ok {
-		value, err := parseIntEnv("NOTARIUS_LLM_DEFAULT_MAX_RETRIES", raw)
-		if err != nil {
-			return err
-		}
-		defaultProfile.MaxRetries = value
-	}
-	if raw, ok := lookup("NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY"); ok {
-		value, err := parseIntEnv("NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY", raw)
-		if err != nil {
-			return err
-		}
-		defaultProfile.MaxConcurrency = value
-	}
-	c.LLMProfiles[pipeline.DefaultLLMProfile] = defaultProfile
-
 	if raw, ok := lookup("NOTARIUS_TOTAL_LLM_CONCURRENCY"); ok {
 		value, err := parseIntEnv("NOTARIUS_TOTAL_LLM_CONCURRENCY", raw)
 		if err != nil {
@@ -74,11 +36,60 @@ func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool))
 		}
 		c.Concurrency.TotalLLM = value
 	}
-	if raw, ok := lookup("NOTARIUS_WORK_DIR"); ok {
+	if raw, ok := lookup("NOTARIUS_STAGE_WORKERS_EXTRACT"); ok {
-		c.Diagnostics.WorkDir = strings.TrimSpace(raw)
+		value, err := parseIntEnv("NOTARIUS_STAGE_WORKERS_EXTRACT", raw)
+		if err != nil {
+			return err
+		}
+		if c.Concurrency.StageWorkers == nil {
+			c.Concurrency.StageWorkers = make(map[string]int)
+		}
+		c.Concurrency.StageWorkers["extract"] = value
+		c.Concurrency.extractWorkersConfigured = true
 	}
-	if raw, ok := lookup("NOTARIUS_DIAGNOSTICS_RETENTION"); ok {
+	c.Concurrency.recomputeStageWorkerDefaults()
-		c.Diagnostics.Retention = diagnostics.RetentionMode(strings.TrimSpace(raw))
+	if raw, ok := lookup("NOTARIUS_OUTPUT_DIR"); ok {
+		c.Output.Directory = strings.TrimSpace(raw)
+		if c.Output.Directory == "" {
+			return fmt.Errorf("NOTARIUS_OUTPUT_DIR: must not be empty")
+		}
+		if strings.ContainsRune(c.Output.Directory, '\x00') {
+			return fmt.Errorf("NOTARIUS_OUTPUT_DIR: must not contain NUL")
+		}
+	}
+	if raw, ok := lookup("NOTARIUS_CACHE_CHUNK_PLANS_MODE"); ok {
+		mode, err := pipeline.ParseChunkCacheMode(raw)
+		if err != nil {
+			return fmt.Errorf("NOTARIUS_CACHE_CHUNK_PLANS_MODE: %w", err)
+		}
+		c.Cache.ChunkPlans.Mode = mode
+	}
+	if raw, ok := lookup("NOTARIUS_CACHE_CHUNK_PLANS_DIR"); ok {
+		c.Cache.ChunkPlans.Directory = cleanOptionalPath(raw)
+		if c.Cache.ChunkPlans.Directory == "" {
+			return fmt.Errorf("NOTARIUS_CACHE_CHUNK_PLANS_DIR: must not be empty")
+		}
+		if strings.ContainsRune(c.Cache.ChunkPlans.Directory, '\x00') {
+			return fmt.Errorf("NOTARIUS_CACHE_CHUNK_PLANS_DIR: must not contain NUL")
+		}
+	}
+	if raw, ok := lookup("NOTARIUS_CACHE_CHECKPOINTS_DIR"); ok {
+		c.Cache.Checkpoints.Directory = cleanOptionalPath(raw)
+		if c.Cache.Checkpoints.Directory == "" {
+			return fmt.Errorf("NOTARIUS_CACHE_CHECKPOINTS_DIR: must not be empty")
+		}
+		if strings.ContainsRune(c.Cache.Checkpoints.Directory, '\x00') {
+			return fmt.Errorf("NOTARIUS_CACHE_CHECKPOINTS_DIR: must not contain NUL")
+		}
+	}
+	if raw, ok := lookup("NOTARIUS_DEBUG_DIR"); ok {
+		c.Debug.Directory = strings.TrimSpace(raw)
+		if c.Debug.Directory == "" {
+			return fmt.Errorf("NOTARIUS_DEBUG_DIR: must not be empty")
+		}
+		if strings.ContainsRune(c.Debug.Directory, '\x00') {
+			return fmt.Errorf("NOTARIUS_DEBUG_DIR: must not contain NUL")
+		}
 	}
 	return nil
 }

internal/core/config/env_test.go

@@ -1,69 +0,0 @@
-package config
-
-import (
-	"strings"
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
-)
-
-func TestApplyEnvOverridesOperationalAndLLMValues(t *testing.T) {
-	cfg := Default()
-	cfg.Pipelines["example"] = pipeline.PipelineProfile{ID: "example", Input: pipeline.Binding("before")}
-
-	err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
-		"NOTARIUS_LLM_DEFAULT_API_KEY":         "secret",
-		"NOTARIUS_LLM_DEFAULT_BASE_URL":        "https://example.invalid/v1",
-		"NOTARIUS_LLM_DEFAULT_MODEL":           "test-model",
-		"NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS": "120",
-		"NOTARIUS_LLM_DEFAULT_MAX_RETRIES":     "5",
-		"NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY": "2",
-		"NOTARIUS_TOTAL_LLM_CONCURRENCY":       "3",
-		"NOTARIUS_WORK_DIR":                    "/tmp/notarius-env",
-		"NOTARIUS_DIAGNOSTICS_RETENTION":       "never",
-		"NOTARIUS_PIPELINE_INPUT":              "after",
-	}))
-	if err != nil {
-		t.Fatalf("ApplyEnvOverrides: %v", err)
-	}
-
-	profile := cfg.LLMProfiles[pipeline.DefaultLLMProfile]
-	if profile.APIKey != "secret" || profile.BaseURL != "https://example.invalid/v1" || profile.Model != "test-model" {
-		t.Fatalf("unexpected LLM profile strings: %+v", profile)
-	}
-	if profile.TimeoutSeconds != 120 || profile.MaxRetries != 5 || profile.MaxConcurrency != 2 {
-		t.Fatalf("unexpected LLM profile numeric values: %+v", profile)
-	}
-	if cfg.Concurrency.TotalLLM != 3 {
-		t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
-	}
-	if cfg.Diagnostics.WorkDir != "/tmp/notarius-env" || cfg.Diagnostics.Retention != diagnostics.RetentionNever {
-		t.Fatalf("unexpected diagnostics config: %+v", cfg.Diagnostics)
-	}
-	if cfg.Pipelines["example"].Input.Module != "before" {
-		t.Fatalf("environment overrides must not change pipeline wiring: %+v", cfg.Pipelines["example"])
-	}
-}
-
-func TestApplyEnvOverridesRejectsInvalidIntegers(t *testing.T) {
-	cfg := Default()
-	err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
-		"NOTARIUS_TOTAL_LLM_CONCURRENCY": "many",
-	}))
-	if err == nil || !strings.Contains(err.Error(), "NOTARIUS_TOTAL_LLM_CONCURRENCY") {
-		t.Fatalf("expected named integer error, got %v", err)
-	}
-}
-
-func TestLoadFromEnvUsesDefaultConfig(t *testing.T) {
-	t.Setenv("NOTARIUS_LLM_DEFAULT_MODEL", "env-model")
-
-	cfg, err := LoadFromEnv()
-	if err != nil {
-		t.Fatalf("LoadFromEnv: %v", err)
-	}
-	if cfg.LLMProfiles[pipeline.DefaultLLMProfile].Model != "env-model" {
-		t.Fatalf("expected env model, got %+v", cfg.LLMProfiles[pipeline.DefaultLLMProfile])
-	}
-}

internal/core/config/file_config.go

@@ -4,41 +4,35 @@ import (
 	"bytes"
 	"fmt"
 	"os"
-	"regexp"
+	"path/filepath"
 	"sort"
 	"strings"
-	"time"
 
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
 	"gopkg.in/yaml.v3"
 )
 
-var envVarNamePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
-
 type FileConfig struct {
 	Version     int                            `yaml:"version"`
-	LLMProfiles map[string]FileLLMProfile      `yaml:"llm_profiles,omitempty"`
+	Scriptorium *FileScriptoriumConfig         `yaml:"scriptorium,omitempty"`
 	Pipelines   map[string]FilePipelineProfile `yaml:"pipelines,omitempty"`
 	Concurrency *FileConcurrencyConfig         `yaml:"concurrency,omitempty"`
-	Diagnostics *FileDiagnosticsConfig         `yaml:"diagnostics,omitempty"`
+	Output      *FileOutputConfig              `yaml:"output,omitempty"`
+	Cache       *FileCacheConfig               `yaml:"cache,omitempty"`
+	Debug       *FileDebugConfig               `yaml:"debug,omitempty"`
 }
 
-type FileLLMProfile struct {
+type FileScriptoriumConfig struct {
-	Provider       *string              `yaml:"provider,omitempty"`
+	ProfileDir  *string `yaml:"profile_dir,omitempty"`
-	BaseURL        *string              `yaml:"base_url,omitempty"`
+	ProfileFile *string `yaml:"profile_file,omitempty"`
-	Model          *string              `yaml:"model,omitempty"`
-	APIKeyEnv      *string              `yaml:"api_key_env,omitempty"`
-	Timeout        *fileDurationSeconds `yaml:"timeout,omitempty"`
-	MaxRetries     *int                 `yaml:"max_retries,omitempty"`
-	MaxConcurrency *int                 `yaml:"max_concurrency,omitempty"`
 }
 
 type FilePipelineProfile struct {
-	Input     fileModuleBinding                  `yaml:"input"`
+	Input      fileModuleBinding                  `yaml:"input"`
-	Chunk     *fileModuleBinding                 `yaml:"chunk,omitempty"`
+	Chunk      *fileModuleBinding                 `yaml:"chunk,omitempty"`
-	Artifacts map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
+	Artifacts  map[string]FileArtifactLaneProfile `yaml:"artifacts,omitempty"`
-	Output    *fileModuleBinding                 `yaml:"output,omitempty"`
+	Output     *fileModuleBinding                 `yaml:"output,omitempty"`
+	References map[string]string                  `yaml:"references,omitempty"`
 }
 
 type FileArtifactLaneProfile struct {
@@ -46,57 +40,39 @@ type FileArtifactLaneProfile struct {
 	Merge      *fileModuleBinding  `yaml:"merge,omitempty"`
 	Normalize  *fileModuleBinding  `yaml:"normalize,omitempty"`
 	Validators []fileModuleBinding `yaml:"validators,omitempty"`
+	References map[string]string   `yaml:"references,omitempty"`
 }
 
 type FileConcurrencyConfig struct {
-	TotalLLM *int `yaml:"total_llm,omitempty"`
+	TotalLLM     *int           `yaml:"total_llm,omitempty"`
+	StageWorkers map[string]int `yaml:"stage_workers,omitempty"`
 }
 
-type FileDiagnosticsConfig struct {
+type FileOutputConfig struct {
-	WorkDir   *string `yaml:"work_dir,omitempty"`
+	Directory *string `yaml:"directory,omitempty"`
-	Retention *string `yaml:"retention,omitempty"`
 }
-
+type FileCacheConfig struct {
-type fileDurationSeconds struct {
+	ChunkPlans  *FileChunkPlanCacheConfig  `yaml:"chunk_plans,omitempty"`
-	seconds int
+	Checkpoints *FileCheckpointCacheConfig `yaml:"checkpoints,omitempty"`
 }
-
+type FileChunkPlanCacheConfig struct {
-func (d *fileDurationSeconds) UnmarshalYAML(node *yaml.Node) error {
+	Directory *string `yaml:"directory,omitempty"`
-	if node.Kind != yaml.ScalarNode {
+	Mode      *string `yaml:"mode,omitempty"`
-		return fmt.Errorf("must be an integer seconds value or duration string")
-	}
-	if node.Tag == "!!int" {
-		var seconds int
-		if err := node.Decode(&seconds); err != nil {
-			return fmt.Errorf("must be an integer seconds value or duration string")
-		}
-		d.seconds = seconds
-		return nil
-	}
-
-	var raw string
-	if err := node.Decode(&raw); err != nil {
-		return fmt.Errorf("must be an integer seconds value or duration string")
-	}
-	duration, err := time.ParseDuration(strings.TrimSpace(raw))
-	if err != nil {
-		return fmt.Errorf("invalid duration %q", raw)
-	}
-	if duration%time.Second != 0 {
-		return fmt.Errorf("duration %q must resolve to whole seconds", raw)
-	}
-	d.seconds = int(duration / time.Second)
-	return nil
 }
-
+type FileCheckpointCacheConfig struct {
-func (d fileDurationSeconds) Seconds() int {
+	Directory *string `yaml:"directory,omitempty"`
-	return d.seconds
+}
+type FileDebugConfig struct {
+	Directory *string `yaml:"directory,omitempty"`
 }
 
 type fileModuleBinding struct {
 	Module     string
 	LLMProfile string
+	Retries    int
 	Options    map[string]any
+	References map[string]string
+	Validators pipeline.ValidatorOverride
 }
 
 func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
@@ -125,12 +101,34 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
 					return err
 				}
 				b.LLMProfile = strings.TrimSpace(llmProfile)
+			case "retries":
+				var retries int
+				if err := valueNode.Decode(&retries); err != nil {
+					return err
+				}
+				b.Retries = retries
 			case "options":
 				var options map[string]any
 				if err := valueNode.Decode(&options); err != nil {
 					return err
 				}
 				b.Options = normalizeOptions(options)
+			case "references":
+				var references map[string]string
+				if err := valueNode.Decode(&references); err != nil {
+					return err
+				}
+				b.References = references
+			case "validators":
+				b.Validators.Set = true
+				var validators []fileModuleBinding
+				if err := valueNode.Decode(&validators); err != nil {
+					return err
+				}
+				b.Validators.Validators = make([]pipeline.ModuleBinding, len(validators))
+				for i, validator := range validators {
+					b.Validators.Validators[i] = validator.toPipelineBinding()
+				}
 			default:
 				return fmt.Errorf("field %s not found in module binding", keyNode.Value)
 			}
@@ -145,7 +143,10 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
 	return pipeline.ModuleBinding{
 		Module:     strings.TrimSpace(b.Module),
 		LLMProfile: strings.TrimSpace(b.LLMProfile),
+		Retries:    b.Retries,
 		Options:    cloneOptions(b.Options),
+		References: normalizedStringMap(b.References),
+		Validators: b.Validators,
 	}
 }
 
@@ -162,18 +163,27 @@ func LoadFileConfig(path string) (FileConfig, error) {
 }
 
 func ParseFileConfigYAML(data []byte) (FileConfig, error) {
+	var header struct {
+		Version int `yaml:"version"`
+	}
+	if err := yaml.Unmarshal(data, &header); err != nil {
+		return FileConfig{}, fmt.Errorf("decode yaml version header: %w", err)
+	}
+	if header.Version == 0 {
+		return FileConfig{}, fmt.Errorf("config version is required")
+	}
+	if header.Version == 2 {
+		return FileConfig{}, fmt.Errorf("config version 2 is no longer supported; migrate the file using the version 2-to-3 migration in docs/config.md")
+	}
+	if header.Version != SupportedFileConfigVersion {
+		return FileConfig{}, fmt.Errorf("unsupported config version %d (supported version is %d)", header.Version, SupportedFileConfigVersion)
+	}
 	var fileCfg FileConfig
 	decoder := yaml.NewDecoder(bytes.NewReader(data))
 	decoder.KnownFields(true)
 	if err := decoder.Decode(&fileCfg); err != nil {
 		return FileConfig{}, fmt.Errorf("decode yaml: %w", err)
 	}
-	if fileCfg.Version == 0 {
-		return FileConfig{}, fmt.Errorf("config version is required")
-	}
-	if fileCfg.Version != SupportedFileConfigVersion {
-		return FileConfig{}, fmt.Errorf("unsupported config version %d", fileCfg.Version)
-	}
 	return fileCfg, nil
 }
 
@@ -186,23 +196,17 @@ func (c *Config) ApplyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
 }
 
 func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(string) (string, bool)) error {
+	_ = lookup
 	if c == nil {
 		return fmt.Errorf("config must not be nil")
 	}
 	if fileCfg.Version != SupportedFileConfigVersion {
 		return fmt.Errorf("unsupported config version %d", fileCfg.Version)
 	}
-	if c.LLMProfiles == nil {
-		c.LLMProfiles = map[string]LLMProfile{}
-	}
 	if c.Pipelines == nil {
 		c.Pipelines = map[string]pipeline.PipelineProfile{}
 	}
 
-	profileIDs, rawLLMProfileIDs, err := normalizedMapKeys(fileCfg.LLMProfiles, "llm profile id")
-	if err != nil {
-		return err
-	}
 	pipelineIDs, rawPipelineIDs, err := normalizedMapKeys(fileCfg.Pipelines, "pipeline id")
 	if err != nil {
 		return err
@@ -212,38 +216,66 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
 		if _, _, err := normalizedMapKeys(filePipeline.Artifacts, fmt.Sprintf("pipeline %q artifact lane id", pipelineID)); err != nil {
 			return err
 		}
+		if _, _, err := normalizedMapKeys(filePipeline.References, fmt.Sprintf("pipeline %q reference slot", pipelineID)); err != nil {
+			return err
+		}
+		if filePipeline.Chunk != nil {
+			if _, _, err := normalizedMapKeys(filePipeline.Chunk.References, fmt.Sprintf("pipeline %q chunk reference slot", pipelineID)); err != nil {
+				return err
+			}
+		}
+		if _, _, err := normalizedMapKeys(filePipeline.Input.References, fmt.Sprintf("pipeline %q input reference slot", pipelineID)); err != nil {
+			return err
+		}
+		if filePipeline.Output != nil {
+			if _, _, err := normalizedMapKeys(filePipeline.Output.References, fmt.Sprintf("pipeline %q output reference slot", pipelineID)); err != nil {
+				return err
+			}
+		}
+		for rawLaneID, fileLane := range filePipeline.Artifacts {
+			laneID := strings.TrimSpace(rawLaneID)
+			if laneID == "" {
+				continue
+			}
+			if _, _, err := normalizedMapKeys(fileLane.References, fmt.Sprintf("pipeline %q lane %q reference slot", pipelineID, laneID)); err != nil {
+				return err
+			}
+			if _, _, err := normalizedMapKeys(fileLane.Extract.References, fmt.Sprintf("pipeline %q lane %q extract reference slot", pipelineID, laneID)); err != nil {
+				return err
+			}
+			if fileLane.Merge != nil {
+				if _, _, err := normalizedMapKeys(fileLane.Merge.References, fmt.Sprintf("pipeline %q lane %q merge reference slot", pipelineID, laneID)); err != nil {
+					return err
+				}
+			}
+			if fileLane.Normalize != nil {
+				if _, _, err := normalizedMapKeys(fileLane.Normalize.References, fmt.Sprintf("pipeline %q lane %q normalize reference slot", pipelineID, laneID)); err != nil {
+					return err
+				}
+			}
+			for i, validator := range fileLane.Validators {
+				if _, _, err := normalizedMapKeys(validator.References, fmt.Sprintf("pipeline %q lane %q validator[%d] reference slot", pipelineID, laneID, i)); err != nil {
+					return err
+				}
+			}
+		}
 	}
 
-	for _, profileID := range profileIDs {
+	if fileCfg.Scriptorium != nil {
-		fileProfile := fileCfg.LLMProfiles[rawLLMProfileIDs[profileID]]
+		if fileCfg.Scriptorium.ProfileDir != nil {
-		profile := c.LLMProfiles[profileID]
+			value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileDir)
-		if fileProfile.Provider != nil {
+			if value == "" {
-			profile.Provider = strings.TrimSpace(*fileProfile.Provider)
+				return fmt.Errorf("scriptorium.profile_dir must not be empty when set")
-		}
-		if fileProfile.BaseURL != nil {
-			profile.BaseURL = strings.TrimSpace(*fileProfile.BaseURL)
-		}
-		if fileProfile.Model != nil {
-			profile.Model = strings.TrimSpace(*fileProfile.Model)
-		}
-		if fileProfile.APIKeyEnv != nil {
-			apiKey, err := resolveAPIKeyEnv(*fileProfile.APIKeyEnv, lookup)
-			if err != nil {
-				return fmt.Errorf("llm_profiles.%s.api_key_env: %w", profileID, err)
 			}
-			profile.APIKeyEnv = strings.TrimSpace(*fileProfile.APIKeyEnv)
+			c.Scriptorium.ProfileDir = value
-			profile.APIKey = apiKey
 		}
-		if fileProfile.Timeout != nil {
+		if fileCfg.Scriptorium.ProfileFile != nil {
-			profile.TimeoutSeconds = fileProfile.Timeout.Seconds()
+			value := strings.TrimSpace(*fileCfg.Scriptorium.ProfileFile)
+			if value == "" {
+				return fmt.Errorf("scriptorium.profile_file must not be empty when set")
+			}
+			c.Scriptorium.ProfileFile = value
 		}
-		if fileProfile.MaxRetries != nil {
-			profile.MaxRetries = *fileProfile.MaxRetries
-		}
-		if fileProfile.MaxConcurrency != nil {
-			profile.MaxConcurrency = *fileProfile.MaxConcurrency
-		}
-		c.LLMProfiles[profileID] = profile
 	}
 
 	for _, pipelineID := range pipelineIDs {
@@ -253,9 +285,10 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
 			return err
 		}
 		profile := pipeline.PipelineProfile{
-			ID:        pipelineID,
+			ID:         pipelineID,
-			Input:     filePipeline.Input.toPipelineBinding(),
+			Input:      filePipeline.Input.toPipelineBinding(),
-			Artifacts: make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
+			Artifacts:  make(map[string]pipeline.ArtifactLaneProfile, len(filePipeline.Artifacts)),
+			References: normalizedStringMap(filePipeline.References),
 		}
 		if filePipeline.Chunk != nil {
 			profile.Chunk = filePipeline.Chunk.toPipelineBinding()
@@ -265,8 +298,11 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
 		}
 		for _, laneID := range laneIDs {
 			fileLane := filePipeline.Artifacts[rawLaneIDs[laneID]]
+			extract := fileLane.Extract.toPipelineBinding()
+			extract.References = mergeStringMaps(normalizedStringMap(fileLane.References), extract.References)
 			lane := pipeline.ArtifactLaneProfile{
-				Extract: fileLane.Extract.toPipelineBinding(),
+				Extract:    extract,
+				References: normalizedStringMap(fileLane.References),
 			}
 			if fileLane.Merge != nil {
 				lane.Merge = fileLane.Merge.toPipelineBinding()
@@ -288,18 +324,88 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
 	if fileCfg.Concurrency != nil && fileCfg.Concurrency.TotalLLM != nil {
 		c.Concurrency.TotalLLM = *fileCfg.Concurrency.TotalLLM
 	}
-	if fileCfg.Diagnostics != nil {
+	if fileCfg.Concurrency != nil && fileCfg.Concurrency.StageWorkers != nil {
-		if fileCfg.Diagnostics.WorkDir != nil {
+		workers, configured, err := normalizeStageWorkers(fileCfg.Concurrency.StageWorkers)
-			c.Diagnostics.WorkDir = strings.TrimSpace(*fileCfg.Diagnostics.WorkDir)
+		if err != nil {
+			return err
 		}
-		if fileCfg.Diagnostics.Retention != nil {
+		c.Concurrency.StageWorkers = workers
-			c.Diagnostics.Retention = diagnostics.RetentionMode(strings.TrimSpace(*fileCfg.Diagnostics.Retention))
+		c.Concurrency.extractWorkersConfigured = configured
+	}
+	c.Concurrency.recomputeStageWorkerDefaults()
+	if fileCfg.Output != nil && fileCfg.Output.Directory != nil {
+		c.Output.Directory = strings.TrimSpace(*fileCfg.Output.Directory)
+		if c.Output.Directory == "" {
+			return fmt.Errorf("output.directory must not be empty")
+		}
+		if strings.ContainsRune(c.Output.Directory, '\x00') {
+			return fmt.Errorf("output.directory must not contain NUL")
+		}
+	}
+	if fileCfg.Cache != nil {
+		if fileCfg.Cache.ChunkPlans != nil {
+			if fileCfg.Cache.ChunkPlans.Mode != nil {
+				mode, err := pipeline.ParseChunkCacheMode(*fileCfg.Cache.ChunkPlans.Mode)
+				if err != nil {
+					return fmt.Errorf("cache.chunk_plans.mode: %w", err)
+				}
+				c.Cache.ChunkPlans.Mode = mode
+			}
+			if fileCfg.Cache.ChunkPlans.Directory != nil {
+				c.Cache.ChunkPlans.Directory = cleanOptionalPath(*fileCfg.Cache.ChunkPlans.Directory)
+				if strings.ContainsRune(c.Cache.ChunkPlans.Directory, '\x00') {
+					return fmt.Errorf("cache.chunk_plans.directory must not contain NUL")
+				}
+			}
+		}
+		if fileCfg.Cache.Checkpoints != nil && fileCfg.Cache.Checkpoints.Directory != nil {
+			c.Cache.Checkpoints.Directory = cleanOptionalPath(*fileCfg.Cache.Checkpoints.Directory)
+			if strings.ContainsRune(c.Cache.Checkpoints.Directory, '\x00') {
+				return fmt.Errorf("cache.checkpoints.directory must not contain NUL")
+			}
+		}
+	}
+	if fileCfg.Debug != nil && fileCfg.Debug.Directory != nil {
+		c.Debug.Directory = strings.TrimSpace(*fileCfg.Debug.Directory)
+		if c.Debug.Directory == "" {
+			return fmt.Errorf("debug.directory must not be empty")
+		}
+		if strings.ContainsRune(c.Debug.Directory, '\x00') {
+			return fmt.Errorf("debug.directory must not contain NUL")
 		}
 	}
 
 	return nil
 }
 
+func cleanOptionalPath(value string) string {
+	value = strings.TrimSpace(value)
+	if value == "" {
+		return ""
+	}
+	return filepath.Clean(value)
+}
+
+func normalizeStageWorkers(values map[string]int) (map[string]int, bool, error) {
+	workers := make(map[string]int, len(values))
+	configured := false
+	for rawKey, value := range values {
+		key := strings.TrimSpace(rawKey)
+		if key == "" {
+			return nil, false, fmt.Errorf("concurrency.stage_workers key must not be empty")
+		}
+		if key != "extract" {
+			return nil, false, fmt.Errorf("concurrency.stage_workers key %q is not supported", rawKey)
+		}
+		if _, exists := workers[key]; exists {
+			return nil, false, fmt.Errorf("concurrency.stage_workers key %q is duplicated after trimming", key)
+		}
+		workers[key] = value
+		configured = true
+	}
+	return workers, configured, nil
+}
+
 func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, map[string]string, error) {
 	keys := make([]string, 0, len(values))
 	rawByNormalized := make(map[string]string, len(values))
@@ -318,19 +424,37 @@ func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, ma
 	return keys, rawByNormalized, nil
 }
 
-func resolveAPIKeyEnv(envName string, lookup func(string) (string, bool)) (string, error) {
+func normalizedStringMap(values map[string]string) map[string]string {
-	name := strings.TrimSpace(envName)
+	if len(values) == 0 {
-	if name == "" {
+		return nil
-		return "", fmt.Errorf("must not be empty")
 	}
-	if !envVarNamePattern.MatchString(name) {
+	out := make(map[string]string, len(values))
-		return "", fmt.Errorf("must be an environment variable name")
+	keys := make([]string, 0, len(values))
+	rawByNormalized := make(map[string]string, len(values))
+	for rawKey := range values {
+		key := strings.TrimSpace(rawKey)
+		rawByNormalized[key] = rawKey
+		keys = append(keys, key)
 	}
-	value, ok := lookup(name)
+	sort.Strings(keys)
-	if !ok {
+	for _, key := range keys {
-		return "", fmt.Errorf("%s is not set", name)
+		out[key] = strings.TrimSpace(values[rawByNormalized[key]])
 	}
-	return value, nil
+	return out
+}
+
+func mergeStringMaps(base map[string]string, override map[string]string) map[string]string {
+	if len(base) == 0 && len(override) == 0 {
+		return nil
+	}
+	out := make(map[string]string, len(base)+len(override))
+	for key, value := range base {
+		out[key] = value
+	}
+	for key, value := range override {
+		out[key] = value
+	}
+	return out
 }
 
 func normalizeOptions(options map[string]any) map[string]any {

internal/core/config/file_config_test.go

@@ -1,393 +0,0 @@
-package config
-
-import (
-	"os"
-	"path/filepath"
-	"reflect"
-	"strings"
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
-)
-
-func TestParseMinimalValidConfig(t *testing.T) {
-	fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-`))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-	if fileCfg.Version != SupportedFileConfigVersion {
-		t.Fatalf("unexpected version: %d", fileCfg.Version)
-	}
-}
-
-func TestLoadFileConfig(t *testing.T) {
-	path := filepath.Join(t.TempDir(), "config.yml")
-	if err := os.WriteFile(path, []byte("version: 1\n"), 0o644); err != nil {
-		t.Fatalf("write config: %v", err)
-	}
-
-	fileCfg, err := LoadFileConfig(path)
-	if err != nil {
-		t.Fatalf("LoadFileConfig: %v", err)
-	}
-	if fileCfg.Version != SupportedFileConfigVersion {
-		t.Fatalf("unexpected version: %d", fileCfg.Version)
-	}
-}
-
-func TestParseFileConfigRejectsUnknownYAMLFields(t *testing.T) {
-	_, err := ParseFileConfigYAML([]byte(`
-version: 1
-unexpected: true
-`))
-	if err == nil || !strings.Contains(err.Error(), "field unexpected not found") {
-		t.Fatalf("expected unknown field error, got %v", err)
-	}
-}
-
-func TestParseFileConfigRejectsUnknownModuleBindingFields(t *testing.T) {
-	_, err := ParseFileConfigYAML([]byte(`
-version: 1
-pipelines:
-  example:
-    input:
-      module: fake/input
-      unexpected: true
-    artifacts:
-      events:
-        extract: fake/extract
-`))
-	if err == nil || !strings.Contains(err.Error(), "field unexpected not found") {
-		t.Fatalf("expected unknown binding field error, got %v", err)
-	}
-}
-
-func TestParseFileConfigRejectsMissingAndUnsupportedVersion(t *testing.T) {
-	tests := []struct {
-		name string
-		data string
-		want string
-	}{
-		{name: "missing", data: `llm_profiles: {}`, want: "version is required"},
-		{name: "unsupported", data: `version: 2`, want: "unsupported config version"},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			_, err := ParseFileConfigYAML([]byte(tc.data))
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestParseFileConfigModuleBindingForms(t *testing.T) {
-	cfg := parseAndApplyConfig(t, `
-version: 1
-pipelines:
-  example:
-    input: fake/input
-    chunk:
-      module: generic
-      options:
-        size: 10
-        flags:
-          - alpha
-        nested:
-          enabled: true
-    artifacts:
-      events:
-        extract:
-          module: fake/extract
-          llm_profile: fast
-          options:
-            temperature: 0
-        merge: appendorder
-        normalize:
-          module: noop
-    output: json
-`)
-
-	profile := cfg.Pipelines["example"]
-	if profile.Input.Module != "fake/input" {
-		t.Fatalf("unexpected input binding: %+v", profile.Input)
-	}
-	if profile.Chunk.Module != "generic" {
-		t.Fatalf("unexpected chunk binding: %+v", profile.Chunk)
-	}
-	if profile.Chunk.Options["size"] != 10 {
-		t.Fatalf("expected chunk options to preserve scalar, got %#v", profile.Chunk.Options)
-	}
-	if !reflect.DeepEqual(profile.Chunk.Options["flags"], []any{"alpha"}) {
-		t.Fatalf("expected list option, got %#v", profile.Chunk.Options["flags"])
-	}
-	nested, ok := profile.Chunk.Options["nested"].(map[string]any)
-	if !ok || nested["enabled"] != true {
-		t.Fatalf("expected nested map option, got %#v", profile.Chunk.Options["nested"])
-	}
-
-	lane := profile.Artifacts["events"]
-	if lane.Extract.Module != "fake/extract" || lane.Extract.LLMProfile != "fast" {
-		t.Fatalf("unexpected extract binding: %+v", lane.Extract)
-	}
-	if lane.Extract.Options["temperature"] != 0 {
-		t.Fatalf("expected object options, got %#v", lane.Extract.Options)
-	}
-	if lane.Merge.Module != "appendorder" || lane.Normalize.Module != "noop" {
-		t.Fatalf("unexpected lane defaults: %+v", lane)
-	}
-	if profile.Output.Module != "json" {
-		t.Fatalf("unexpected output binding: %+v", profile.Output)
-	}
-}
-
-func TestParseFileConfigValidatorMixedBindingForms(t *testing.T) {
-	cfg := parseAndApplyConfig(t, `
-version: 1
-pipelines:
-  example:
-    input: fake/input
-    artifacts:
-      events:
-        extract: fake/extract
-        validators:
-          - fake/validator
-          - module: fake/llm-validator
-            llm_profile: careful
-            options:
-              threshold: 0.7
-`)
-
-	validators := cfg.Pipelines["example"].Artifacts["events"].Validators
-	if len(validators) != 2 {
-		t.Fatalf("expected two validators, got %d", len(validators))
-	}
-	if validators[0].Module != "fake/validator" {
-		t.Fatalf("unexpected shorthand validator: %+v", validators[0])
-	}
-	if validators[1].Module != "fake/llm-validator" || validators[1].LLMProfile != "careful" {
-		t.Fatalf("unexpected object validator: %+v", validators[1])
-	}
-	if validators[1].Options["threshold"] != 0.7 {
-		t.Fatalf("unexpected validator options: %#v", validators[1].Options)
-	}
-}
-
-func TestParseFileConfigDurationParsing(t *testing.T) {
-	tests := []struct {
-		name string
-		raw  string
-		want int
-	}{
-		{name: "integer seconds", raw: "600", want: 600},
-		{name: "duration string", raw: "10m", want: 600},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			cfg := parseAndApplyConfig(t, `
-version: 1
-llm_profiles:
-  default:
-    timeout: `+tc.raw+`
-`)
-			if got := cfg.LLMProfiles["default"].TimeoutSeconds; got != tc.want {
-				t.Fatalf("TimeoutSeconds = %d, want %d", got, tc.want)
-			}
-		})
-	}
-}
-
-func TestParseFileConfigRejectsSubsecondDuration(t *testing.T) {
-	_, err := ParseFileConfigYAML([]byte(`
-version: 1
-llm_profiles:
-  default:
-    timeout: 1500ms
-`))
-	if err == nil || !strings.Contains(err.Error(), "whole seconds") {
-		t.Fatalf("expected whole-seconds duration error, got %v", err)
-	}
-}
-
-func TestApplyFileConfigResolvesAPIKeyEnv(t *testing.T) {
-	fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-llm_profiles:
-  default:
-    api_key_env: NOTARIUS_TEST_API_KEY
-`))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-
-	cfg := Default()
-	if err := cfg.applyFileConfigWithLookup(fileCfg, mapLookup(map[string]string{"NOTARIUS_TEST_API_KEY": "secret"})); err != nil {
-		t.Fatalf("ApplyFileConfig: %v", err)
-	}
-	profile := cfg.LLMProfiles["default"]
-	if profile.APIKeyEnv != "NOTARIUS_TEST_API_KEY" || profile.APIKey != "secret" {
-		t.Fatalf("unexpected resolved API key: %+v", profile)
-	}
-}
-
-func TestApplyFileConfigRejectsDuplicateTrimmedLLMProfileIDs(t *testing.T) {
-	fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-llm_profiles:
-  default:
-    model: first
-  " default ":
-    model: second
-`))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-
-	cfg := Default()
-	err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
-	if err == nil || !strings.Contains(err.Error(), "llm profile id") || !strings.Contains(err.Error(), "duplicated") {
-		t.Fatalf("expected duplicate LLM profile ID error, got %v", err)
-	}
-}
-
-func TestApplyFileConfigRejectsDuplicateTrimmedPipelineIDs(t *testing.T) {
-	fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-pipelines:
-  example:
-    input: fake/input
-  " example ":
-    input: fake/other-input
-`))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-
-	cfg := Default()
-	err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
-	if err == nil || !strings.Contains(err.Error(), "pipeline id") || !strings.Contains(err.Error(), "duplicated") {
-		t.Fatalf("expected duplicate pipeline ID error, got %v", err)
-	}
-}
-
-func TestApplyFileConfigRejectsDuplicateTrimmedArtifactLaneIDs(t *testing.T) {
-	fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-pipelines:
-  example:
-    input: fake/input
-    artifacts:
-      events:
-        extract: fake/extract
-      " events ":
-        extract: fake/other-extract
-`))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-
-	cfg := Default()
-	err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
-	if err == nil || !strings.Contains(err.Error(), `pipeline "example" artifact lane id`) || !strings.Contains(err.Error(), "duplicated") {
-		t.Fatalf("expected duplicate artifact lane ID error, got %v", err)
-	}
-}
-
-func TestApplyFileConfigAllowsRetryOnlyLLMProfile(t *testing.T) {
-	cfg := parseAndApplyConfig(t, `
-version: 1
-llm_profiles:
-  retry-only:
-    max_retries: 3
-`)
-
-	profile := cfg.LLMProfiles["retry-only"]
-	if profile.MaxRetries != 3 {
-		t.Fatalf("unexpected max retries: %d", profile.MaxRetries)
-	}
-	if profile.TimeoutSeconds != 0 {
-		t.Fatalf("expected unset timeout, got %d", profile.TimeoutSeconds)
-	}
-	if profile.MaxConcurrency != 0 {
-		t.Fatalf("expected unset max concurrency, got %d", profile.MaxConcurrency)
-	}
-}
-
-func TestApplyFileConfigRejectsInvalidAPIKeyEnv(t *testing.T) {
-	tests := []struct {
-		name string
-		env  string
-		want string
-	}{
-		{name: "invalid name", env: "NOTARIUS-KEY", want: "environment variable name"},
-		{name: "not set", env: "NOTARIUS_TEST_API_KEY", want: "is not set"},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			fileCfg, err := ParseFileConfigYAML([]byte(`
-version: 1
-llm_profiles:
-  default:
-    api_key_env: ` + tc.env + `
-`))
-			if err != nil {
-				t.Fatalf("ParseFileConfigYAML: %v", err)
-			}
-			cfg := Default()
-			err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestApplyFileConfigOperationalSections(t *testing.T) {
-	cfg := parseAndApplyConfig(t, `
-version: 1
-concurrency:
-  total_llm: 4
-diagnostics:
-  work_dir: /tmp/notarius-test
-  retention: always
-`)
-
-	if cfg.Concurrency.TotalLLM != 4 {
-		t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
-	}
-	if cfg.Diagnostics.WorkDir != "/tmp/notarius-test" {
-		t.Fatalf("unexpected work dir: %q", cfg.Diagnostics.WorkDir)
-	}
-	if cfg.Diagnostics.Retention != diagnostics.RetentionAlways {
-		t.Fatalf("unexpected retention: %q", cfg.Diagnostics.Retention)
-	}
-}
-
-func parseAndApplyConfig(t *testing.T, raw string) Config {
-	t.Helper()
-	fileCfg, err := ParseFileConfigYAML([]byte(raw))
-	if err != nil {
-		t.Fatalf("ParseFileConfigYAML: %v", err)
-	}
-	cfg := Default()
-	if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
-		t.Fatalf("ApplyFileConfig: %v", err)
-	}
-	return cfg
-}
-
-func emptyLookup(string) (string, bool) {
-	return "", false
-}
-
-func mapLookup(values map[string]string) func(string) (string, bool) {
-	return func(key string) (string, bool) {
-		value, ok := values[key]
-		return value, ok
-	}
-}

internal/core/config/redaction.go

@@ -1,38 +1,42 @@
 package config
 
-import "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+import (
+	"strings"
 
-const redactedSecret = "[REDACTED]"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
 
 func (c Config) Redacted() Config {
-	redacted := cloneConfig(c)
+	return redactConfig(cloneConfig(c))
-	for id, profile := range redacted.LLMProfiles {
-		if profile.APIKey != "" {
-			profile.APIKey = redactedSecret
-		}
-		redacted.LLMProfiles[id] = profile
-	}
-	return redacted
 }
 
-func (c Config) RedactedDiagnosticsPayload() any {
+func (c Config) RedactedSummaryPayload() any {
 	return c.Redacted()
 }
 
-func (e EffectiveConfig) RedactedDiagnosticsPayload() any {
+func (e EffectiveConfig) RedactedSummaryPayload() any {
 	return EffectiveConfig{
-		Config:           e.Config.Redacted(),
+		Config:             e.Config.Redacted(),
-		PipelineID:       e.PipelineID,
+		PipelineID:         e.PipelineID,
-		Only:             append([]string(nil), e.Only...),
+		Only:               append([]string(nil), e.Only...),
-		ResolvedPipeline: cloneResolvedPipeline(e.ResolvedPipeline),
+		ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), e.ReferenceOverrides...),
+		ReferenceUnbinds:   append([]pipeline.ReferenceUnbind(nil), e.ReferenceUnbinds...),
+		ResolvedPipeline:   cloneResolvedPipeline(e.ResolvedPipeline),
 	}
 }
 
 func cloneResolvedPipeline(in pipeline.ResolvedPipeline) pipeline.ResolvedPipeline {
 	out := in
-	out.Input = cloneModuleBinding(in.Input)
+	out.Input = redactBinding(cloneModuleBinding(in.Input))
-	out.Chunk = cloneModuleBinding(in.Chunk)
+	out.Chunk = redactBinding(cloneModuleBinding(in.Chunk))
-	out.Output = cloneModuleBinding(in.Output)
+	out.ChunkReferences = pipeline.CloneReferenceTarget(in.ChunkReferences)
+	out.Output = redactBinding(cloneModuleBinding(in.Output))
+	if len(in.ValidatorChains) > 0 {
+		out.ValidatorChains = make([]pipeline.ResolvedValidatorChain, len(in.ValidatorChains))
+		for i, chain := range in.ValidatorChains {
+			out.ValidatorChains[i] = cloneResolvedValidatorChain(chain)
+		}
+	}
 	if len(in.ArtifactLanes) > 0 {
 		out.ArtifactLanes = make([]pipeline.ResolvedArtifactLane, len(in.ArtifactLanes))
 		for i, lane := range in.ArtifactLanes {
@@ -42,16 +46,97 @@ func cloneResolvedPipeline(in pipeline.ResolvedPipeline) pipeline.ResolvedPipeli
 	return out
 }
 
-func cloneResolvedArtifactLane(in pipeline.ResolvedArtifactLane) pipeline.ResolvedArtifactLane {
+func cloneResolvedValidatorChain(in pipeline.ResolvedValidatorChain) pipeline.ResolvedValidatorChain {
 	out := in
-	out.Extract = cloneModuleBinding(in.Extract)
-	out.Merge = cloneModuleBinding(in.Merge)
-	out.Normalize = cloneModuleBinding(in.Normalize)
 	if len(in.Validators) > 0 {
-		out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
+		out.Validators = make([]pipeline.ResolvedValidator, len(in.Validators))
-		for i, binding := range in.Validators {
+		for i, validator := range in.Validators {
-			out.Validators[i] = cloneModuleBinding(binding)
+			out.Validators[i] = pipeline.ResolvedValidator{
+				Binding:        redactBinding(cloneModuleBinding(validator.Binding)),
+				ExecutionClass: validator.ExecutionClass,
+				Target:         validator.Target,
+				ArtifactKind:   validator.ArtifactKind,
+			}
 		}
 	}
 	return out
 }
+
+func cloneResolvedArtifactLane(in pipeline.ResolvedArtifactLane) pipeline.ResolvedArtifactLane {
+	out := in
+	out.Extract = redactBinding(cloneModuleBinding(in.Extract))
+	out.Merge = redactBinding(cloneModuleBinding(in.Merge))
+	out.Normalize = redactBinding(cloneModuleBinding(in.Normalize))
+	out.ExtractReferences = pipeline.CloneReferenceTarget(in.ExtractReferences)
+	out.MergeReferences = pipeline.CloneReferenceTarget(in.MergeReferences)
+	out.NormalizeReferences = pipeline.CloneReferenceTarget(in.NormalizeReferences)
+	if len(in.Validators) > 0 {
+		out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
+		for i, binding := range in.Validators {
+			out.Validators[i] = redactBinding(cloneModuleBinding(binding))
+		}
+	}
+	return out
+}
+
+func redactConfig(cfg Config) Config {
+	for id, profile := range cfg.Pipelines {
+		profile.Input = redactBinding(profile.Input)
+		profile.Chunk = redactBinding(profile.Chunk)
+		profile.Output = redactBinding(profile.Output)
+		for laneID, lane := range profile.Artifacts {
+			lane.Extract = redactBinding(lane.Extract)
+			lane.Merge = redactBinding(lane.Merge)
+			lane.Normalize = redactBinding(lane.Normalize)
+			for i := range lane.Validators {
+				lane.Validators[i] = redactBinding(lane.Validators[i])
+			}
+			profile.Artifacts[laneID] = lane
+		}
+		cfg.Pipelines[id] = profile
+	}
+	return cfg
+}
+
+func redactBinding(binding pipeline.ModuleBinding) pipeline.ModuleBinding {
+	binding.Options = redactOptions(binding.Options)
+	for i := range binding.Validators.Validators {
+		binding.Validators.Validators[i] = redactBinding(binding.Validators.Validators[i])
+	}
+	return binding
+}
+
+func redactOptions(values map[string]any) map[string]any {
+	if len(values) == 0 {
+		return nil
+	}
+	out := make(map[string]any, len(values))
+	for key, value := range values {
+		if sensitiveConfigKey(key) {
+			out[key] = "[REDACTED]"
+			continue
+		}
+		switch typed := value.(type) {
+		case map[string]any:
+			out[key] = redactOptions(typed)
+		case []any:
+			items := make([]any, len(typed))
+			for i, item := range typed {
+				if nested, ok := item.(map[string]any); ok {
+					items[i] = redactOptions(nested)
+				} else {
+					items[i] = item
+				}
+			}
+			out[key] = items
+		default:
+			out[key] = value
+		}
+	}
+	return out
+}
+
+func sensitiveConfigKey(key string) bool {
+	key = strings.ToLower(key)
+	return strings.Contains(key, "api_key") || strings.Contains(key, "apikey") || strings.Contains(key, "authorization") || strings.Contains(key, "bearer") || strings.Contains(key, "password") || strings.Contains(key, "secret") || strings.Contains(key, "token")
+}

internal/core/config/redaction_test.go

@@ -1,101 +0,0 @@
-package config
-
-import (
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
-)
-
-func TestRedactedConfigRemovesAPIKeyValues(t *testing.T) {
-	cfg := Default()
-	cfg.LLMProfiles[pipeline.DefaultLLMProfile] = LLMProfile{
-		Provider:       "openai-compatible",
-		BaseURL:        "https://example.invalid/v1",
-		Model:          "test-model",
-		APIKey:         "secret",
-		APIKeyEnv:      "NOTARIUS_TEST_API_KEY",
-		TimeoutSeconds: 600,
-		MaxRetries:     3,
-		MaxConcurrency: 1,
-	}
-	cfg.LLMProfiles["other"] = LLMProfile{APIKey: "other-secret", Model: "other-model"}
-
-	redacted := cfg.Redacted()
-
-	if redacted.LLMProfiles[pipeline.DefaultLLMProfile].APIKey != redactedSecret {
-		t.Fatalf("expected default API key redacted, got %+v", redacted.LLMProfiles[pipeline.DefaultLLMProfile])
-	}
-	if redacted.LLMProfiles["other"].APIKey != redactedSecret {
-		t.Fatalf("expected other API key redacted, got %+v", redacted.LLMProfiles["other"])
-	}
-	if redacted.LLMProfiles[pipeline.DefaultLLMProfile].Model != "test-model" {
-		t.Fatalf("expected non-secret fields preserved, got %+v", redacted.LLMProfiles[pipeline.DefaultLLMProfile])
-	}
-	if cfg.LLMProfiles[pipeline.DefaultLLMProfile].APIKey != "secret" {
-		t.Fatalf("redaction mutated original config")
-	}
-}
-
-func TestConfigRedactedDiagnosticsPayloadRedactsAPIKeys(t *testing.T) {
-	cfg := Default()
-	profile := cfg.LLMProfiles[pipeline.DefaultLLMProfile]
-	profile.APIKey = "secret"
-	profile.Model = "test-model"
-	cfg.LLMProfiles[pipeline.DefaultLLMProfile] = profile
-
-	payload, ok := cfg.RedactedDiagnosticsPayload().(Config)
-	if !ok {
-		t.Fatalf("expected Config payload, got %T", cfg.RedactedDiagnosticsPayload())
-	}
-	if payload.LLMProfiles[pipeline.DefaultLLMProfile].APIKey != redactedSecret {
-		t.Fatalf("expected API key redacted, got %+v", payload.LLMProfiles[pipeline.DefaultLLMProfile])
-	}
-	if payload.LLMProfiles[pipeline.DefaultLLMProfile].Model != "test-model" {
-		t.Fatalf("expected non-secret fields preserved, got %+v", payload.LLMProfiles[pipeline.DefaultLLMProfile])
-	}
-	if cfg.LLMProfiles[pipeline.DefaultLLMProfile].APIKey != "secret" {
-		t.Fatalf("redacted diagnostics payload mutated original config")
-	}
-}
-
-func TestEffectiveConfigRedactedDiagnosticsPayloadRedactsAndCopies(t *testing.T) {
-	cfg := validConfig()
-	profile := cfg.LLMProfiles[pipeline.DefaultLLMProfile]
-	profile.APIKey = "secret"
-	cfg.LLMProfiles[pipeline.DefaultLLMProfile] = profile
-	lane := cfg.Pipelines["example"].Artifacts["events"]
-	lane.Extract.Options = map[string]any{"temperature": 0.2}
-	cfg.Pipelines["example"].Artifacts["events"] = lane
-
-	effective, err := cfg.Resolve(ResolveInput{
-		PipelineID: "example",
-		Only:       []string{"events"},
-		Catalog:    fakeCatalog(t),
-	})
-	if err != nil {
-		t.Fatalf("Resolve: %v", err)
-	}
-
-	payload, ok := effective.RedactedDiagnosticsPayload().(EffectiveConfig)
-	if !ok {
-		t.Fatalf("expected EffectiveConfig payload, got %T", effective.RedactedDiagnosticsPayload())
-	}
-	if payload.Config.LLMProfiles[pipeline.DefaultLLMProfile].APIKey != redactedSecret {
-		t.Fatalf("expected nested API key redacted, got %+v", payload.Config.LLMProfiles[pipeline.DefaultLLMProfile])
-	}
-	if cfg.LLMProfiles[pipeline.DefaultLLMProfile].APIKey != "secret" {
-		t.Fatalf("redacted diagnostics payload mutated source config")
-	}
-	if payload.PipelineID != effective.PipelineID || payload.ResolvedPipeline.Digest != effective.ResolvedPipeline.Digest {
-		t.Fatalf("expected pipeline metadata preserved, got %+v", payload)
-	}
-
-	payload.Only[0] = "changed"
-	if effective.Only[0] != "events" {
-		t.Fatalf("expected only lanes to be copied")
-	}
-	payload.ResolvedPipeline.ArtifactLanes[0].Extract.Options["temperature"] = 1.0
-	if effective.ResolvedPipeline.ArtifactLanes[0].Extract.Options["temperature"] != 0.2 {
-		t.Fatalf("expected resolved pipeline options to be copied")
-	}
-}

internal/core/config/v3_test.go

@@ -0,0 +1,124 @@
+package config
+
+import (
+	"encoding/json"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+func TestVersion3DefaultsAndValidation(t *testing.T) {
+	cfg := Default()
+	if cfg.Output.Directory != "./notarius-output" || cfg.Debug.Directory != "./notarius-debug" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheAuto {
+		t.Fatalf("unexpected defaults: %#v", cfg)
+	}
+	if err := cfg.Validate(); err != nil {
+		t.Fatalf("Validate() error = %v", err)
+	}
+}
+
+func TestVersion3FileSchemaIsStrictAndRejectsVersion2BeforeDecode(t *testing.T) {
+	_, err := ParseFileConfigYAML([]byte("version: 2\nworkspace:\n  directory: /tmp/old\n"))
+	if err == nil || !strings.Contains(err.Error(), "version 2-to-3 migration") {
+		t.Fatalf("version 2 error = %v", err)
+	}
+	_, err = ParseFileConfigYAML([]byte("version: 3\nworkspace:\n  directory: /tmp/old\n"))
+	if err == nil || !strings.Contains(err.Error(), "field workspace not found") {
+		t.Fatalf("unknown field error = %v", err)
+	}
+	_, err = ParseFileConfigYAML([]byte("version: 4\n"))
+	if err == nil || !strings.Contains(err.Error(), "unsupported config version 4") {
+		t.Fatalf("version 4 error = %v", err)
+	}
+}
+
+func TestStatePrecedenceAndInvalidSources(t *testing.T) {
+	file, err := ParseFileConfigYAML([]byte(`version: 3
+output:
+  directory: ./file-output
+cache:
+  chunk_plans:
+    directory: ./plans
+    mode: refresh
+  checkpoints:
+    directory: ./checkpoints
+debug:
+  directory: ./debug
+`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	cfg := Default()
+	if err := cfg.ApplyFileConfig(file); err != nil {
+		t.Fatal(err)
+	}
+	lookup := func(name string) (string, bool) {
+		values := map[string]string{
+			"NOTARIUS_OUTPUT_DIR": "/env/output", "NOTARIUS_CACHE_CHUNK_PLANS_MODE": "auto",
+			"NOTARIUS_CACHE_CHUNK_PLANS_DIR": "/env/plans", "NOTARIUS_CACHE_CHECKPOINTS_DIR": "/env/checkpoints", "NOTARIUS_DEBUG_DIR": "/env/debug",
+		}
+		v, ok := values[name]
+		return v, ok
+	}
+	if err := cfg.ApplyEnvOverridesWithLookup(lookup); err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Output.Directory != "/env/output" || cfg.Cache.ChunkPlans.Directory != "/env/plans" || cfg.Cache.Checkpoints.Directory != "/env/checkpoints" || cfg.Debug.Directory != "/env/debug" || cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheAuto {
+		t.Fatalf("unexpected environment precedence: %#v", cfg)
+	}
+
+	bad := Default()
+	err = bad.ApplyEnvOverridesWithLookup(func(name string) (string, bool) {
+		if name == "NOTARIUS_DEBUG_DIR" {
+			return " ", true
+		}
+		return "", false
+	})
+	if err == nil || !strings.Contains(err.Error(), "NOTARIUS_DEBUG_DIR") {
+		t.Fatalf("empty debug environment error = %v", err)
+	}
+	invalidFile, err := ParseFileConfigYAML([]byte("version: 3\noutput:\n  directory: '   '\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	bad = Default()
+	if err := bad.ApplyFileConfig(invalidFile); err == nil || !strings.Contains(err.Error(), "output.directory") {
+		t.Fatalf("invalid file error = %v", err)
+	}
+}
+
+func TestRedactedSummaryContainsOnlyVersion3StateFields(t *testing.T) {
+	cfg := Default()
+	cfg.Pipelines["example"] = pipeline.PipelineProfile{Input: pipeline.ModuleBinding{Module: "input", Options: map[string]any{"api_key": "secret-value", "safe": "value"}}}
+	payload, err := json.Marshal(cfg.RedactedSummaryPayload())
+	if err != nil {
+		t.Fatal(err)
+	}
+	text := string(payload)
+	for _, forbidden := range []string{"workspace", "diagnostics"} {
+		if strings.Contains(text, forbidden) {
+			t.Fatalf("payload contains %q: %s", forbidden, text)
+		}
+	}
+	if strings.Contains(text, "secret-value") || !strings.Contains(text, "[REDACTED]") {
+		t.Fatalf("payload did not redact sensitive option: %s", text)
+	}
+}
+
+func TestCacheFamilyDefaultsAreIndependent(t *testing.T) {
+	base := filepath.Join(t.TempDir(), "cache")
+	resolver := func() (string, error) { return base, nil }
+	plans, err := DefaultChunkPlanRoot(resolver)
+	if err != nil {
+		t.Fatal(err)
+	}
+	checkpoints, err := DefaultCheckpointRoot(resolver)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if plans == checkpoints || plans != filepath.Join(base, "notarius", "chunk-plans") || checkpoints != filepath.Join(base, "notarius", "checkpoints") {
+		t.Fatalf("roots = %q, %q", plans, checkpoints)
+	}
+}

internal/core/config/validation.go

@@ -2,79 +2,84 @@ package config
 
 import (
 	"fmt"
+	"sort"
 	"strings"
 
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
 )
 
-const providerOpenAICompatible = "openai-compatible"
-
 func (c Config) Validate() error {
-	if err := validateLLMProfiles(c.LLMProfiles); err != nil {
+	c.Concurrency.recomputeStageWorkerDefaults()
+	if err := validateScriptorium(c.Scriptorium); err != nil {
 		return err
 	}
-	if err := validateDiagnostics(c.Diagnostics); err != nil {
+	if err := validateStateSurfaces(c); err != nil {
 		return err
 	}
 	if c.Concurrency.TotalLLM <= 0 {
 		return fmt.Errorf("total LLM concurrency must be greater than zero")
 	}
-	return validatePipelineProfiles(c.Pipelines, c.LLMProfiles)
+	if err := validateStageWorkers(c.Concurrency); err != nil {
+		return err
+	}
+	return validatePipelineProfiles(c.Pipelines)
 }
 
-func (c Config) LLMProfile(id string) (LLMProfile, bool) {
+func validateStageWorkers(cfg ConcurrencyConfig) error {
-	trimmedID := strings.TrimSpace(id)
+	keys := make([]string, 0, len(cfg.StageWorkers))
-	for rawID, profile := range c.LLMProfiles {
+	for key := range cfg.StageWorkers {
-		if strings.TrimSpace(rawID) == trimmedID {
+		keys = append(keys, key)
-			return profile, true
+	}
+	sort.Strings(keys)
+	for _, key := range keys {
+		if strings.TrimSpace(key) == "" {
+			return fmt.Errorf("concurrency.stage_workers key must not be empty")
+		}
+		if key != "extract" {
+			return fmt.Errorf("concurrency.stage_workers key %q is not supported", key)
 		}
 	}
-	return LLMProfile{}, false
+	extractWorkers, ok := cfg.StageWorkers["extract"]
+	if !ok {
+		extractWorkers = cfg.TotalLLM
+	}
+	if extractWorkers < 1 || extractWorkers > cfg.TotalLLM {
+		return fmt.Errorf("concurrency.stage_workers.extract must be between 1 and concurrency.total_llm (%d)", cfg.TotalLLM)
+	}
+	return nil
 }
 
-func validateLLMProfiles(profiles map[string]LLMProfile) error {
+func validateScriptorium(cfg ScriptoriumConfig) error {
-	seen := make(map[string]struct{}, len(profiles))
+	if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
-	for rawID, profile := range profiles {
+		return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
-		id := strings.TrimSpace(rawID)
+	}
-		if id == "" {
+	return nil
-			return fmt.Errorf("LLM profile id must not be empty")
+}
-		}
-		if _, ok := seen[id]; ok {
-			return fmt.Errorf("LLM profile id %q is duplicated after trimming", id)
-		}
-		seen[id] = struct{}{}
 
-		provider := strings.TrimSpace(profile.Provider)
+func validateStateSurfaces(cfg Config) error {
-		if provider != "" && provider != providerOpenAICompatible {
+	if strings.TrimSpace(cfg.Output.Directory) == "" {
-			return fmt.Errorf("LLM profile %q provider %q is not supported", id, provider)
+		return fmt.Errorf("output.directory must not be empty")
-		}
+	}
-		if profile.TimeoutSeconds < 0 {
+	if strings.TrimSpace(cfg.Debug.Directory) == "" {
-			return fmt.Errorf("LLM profile %q timeout seconds must not be negative", id)
+		return fmt.Errorf("debug.directory must not be empty")
-		}
+	}
-		if profile.MaxRetries < 0 {
+	if err := cfg.Cache.ChunkPlans.Mode.Validate(); err != nil {
-			return fmt.Errorf("LLM profile %q max retries must not be negative", id)
+		return fmt.Errorf("cache.chunk_plans.mode: %w", err)
-		}
+	}
-		if profile.MaxConcurrency < 0 {
+	for name, value := range map[string]string{
-			return fmt.Errorf("LLM profile %q max concurrency must not be negative", id)
+		"output.directory":            cfg.Output.Directory,
+		"cache.chunk_plans.directory": cfg.Cache.ChunkPlans.Directory,
+		"cache.checkpoints.directory": cfg.Cache.Checkpoints.Directory,
+		"debug.directory":             cfg.Debug.Directory,
+	} {
+		if strings.ContainsRune(value, '\x00') {
+			return fmt.Errorf("%s must not contain NUL", name)
 		}
 	}
 	return nil
 }
 
-func validateDiagnostics(cfg DiagnosticsConfig) error {
+func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile) error {
-	if strings.TrimSpace(cfg.WorkDir) == "" {
-		return fmt.Errorf("diagnostics work dir must not be empty")
-	}
-	switch cfg.Retention {
-	case "", diagnostics.RetentionAuto, diagnostics.RetentionAlways, diagnostics.RetentionNever:
-		return nil
-	default:
-		return fmt.Errorf("diagnostics retention %q is not supported", cfg.Retention)
-	}
-}
-
-func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile, llmProfiles map[string]LLMProfile) error {
 	seen := make(map[string]struct{}, len(profiles))
 	for rawID, profile := range profiles {
 		id := strings.TrimSpace(rawID)
@@ -89,13 +94,16 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile, llmP
 		if profile.ID != "" && strings.TrimSpace(profile.ID) != id {
 			return fmt.Errorf("pipeline %q profile id %q does not match map key", id, profile.ID)
 		}
-		if err := validateBindingLLMProfile(id, "", "input", profile.Input, llmProfiles); err != nil {
+		if err := validateBinding(id, "", "input", profile.Input, false); err != nil {
 			return err
 		}
-		if err := validateBindingLLMProfile(id, "", "chunk", profile.Chunk, llmProfiles); err != nil {
+		if err := validateBinding(id, "", "chunk", profile.Chunk, true); err != nil {
 			return err
 		}
-		if err := validateBindingLLMProfile(id, "", "output", profile.Output, llmProfiles); err != nil {
+		if err := validateBinding(id, "", "output", profile.Output, false); err != nil {
+			return err
+		}
+		if err := validateReferenceMap(id, "", profile.References); err != nil {
 			return err
 		}
 		for rawLaneID, lane := range profile.Artifacts {
@@ -103,51 +111,133 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile, llmP
 			if laneID == "" {
 				return fmt.Errorf("pipeline %q artifact lane id must not be empty", id)
 			}
-			if err := validateBindingLLMProfile(id, laneID, "extract", lane.Extract, llmProfiles); err != nil {
+			if err := validateReferenceMap(id, laneID, lane.References); err != nil {
 				return err
 			}
-			if err := validateBindingLLMProfile(id, laneID, "merge", lane.Merge, llmProfiles); err != nil {
+			if err := validateBinding(id, laneID, "extract", lane.Extract, true); err != nil {
 				return err
 			}
-			if err := validateBindingLLMProfile(id, laneID, "normalize", lane.Normalize, llmProfiles); err != nil {
+			if err := validateBinding(id, laneID, "merge", lane.Merge, true); err != nil {
 				return err
 			}
-			for i, validator := range lane.Validators {
+			if err := validateBinding(id, laneID, "normalize", lane.Normalize, true); err != nil {
-				if err := validateBindingLLMProfile(id, laneID, fmt.Sprintf("validator[%d]", i), validator, llmProfiles); err != nil {
+				return err
-					return err
+			}
-				}
+			if len(lane.Validators) > 0 {
+				return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", id, laneID)
 			}
 		}
 	}
 	return nil
 }
 
+func validateBinding(
+	pipelineID string,
+	laneID string,
+	slot string,
+	binding pipeline.ModuleBinding,
+	referencesAllowed bool,
+) error {
+	if err := validateBindingLLMProfile(pipelineID, laneID, slot, binding); err != nil {
+		return err
+	}
+	if binding.Retries < 0 {
+		if laneID != "" {
+			return fmt.Errorf("pipeline %q lane %q %s retries must be greater than or equal to zero", pipelineID, laneID, slot)
+		}
+		return fmt.Errorf("pipeline %q %s retries must be greater than or equal to zero", pipelineID, slot)
+	}
+	if err := validateValidatorOverride(pipelineID, laneID, slot, binding.Validators); err != nil {
+		return err
+	}
+	if len(binding.References) == 0 {
+		return nil
+	}
+	if !referencesAllowed {
+		if laneID != "" {
+			return fmt.Errorf("pipeline %q lane %q %s references are not supported", pipelineID, laneID, slot)
+		}
+		return fmt.Errorf("pipeline %q %s references are not supported", pipelineID, slot)
+	}
+	return validateReferenceMapForContext(pipelineID, laneID, slot, binding.References)
+}
+
+func validateValidatorOverride(pipelineID string, laneID string, slot string, override pipeline.ValidatorOverride) error {
+	if !override.Set {
+		return nil
+	}
+	switch slot {
+	case "chunk", "extract", "merge", "normalize":
+	default:
+		return fmt.Errorf("%s validators are not supported", referenceContext(pipelineID, laneID, slot))
+	}
+	for i, validator := range override.Validators {
+		context := fmt.Sprintf("%s validators[%d]", referenceContext(pipelineID, laneID, slot), i)
+		if strings.TrimSpace(validator.Module) == "" {
+			return fmt.Errorf("%s module must not be empty", context)
+		}
+		if len(validator.References) > 0 {
+			return fmt.Errorf("%s references are not supported", context)
+		}
+		if validator.Validators.Set {
+			return fmt.Errorf("%s nested validators are not supported", context)
+		}
+		if validator.Retries != 0 {
+			return fmt.Errorf("%s retries are not supported", context)
+		}
+		if validator.LLMProfile != "" && strings.TrimSpace(validator.LLMProfile) == "" {
+			return fmt.Errorf("%s llm_profile must not be empty when set", context)
+		}
+	}
+	return nil
+}
+
+func validateReferenceMap(pipelineID string, laneID string, references map[string]string) error {
+	return validateReferenceMapForContext(pipelineID, laneID, "", references)
+}
+
+func validateReferenceMapForContext(pipelineID string, laneID string, slot string, references map[string]string) error {
+	seen := make(map[string]struct{}, len(references))
+	for rawSlotName, rawSource := range references {
+		slotName := strings.TrimSpace(rawSlotName)
+		if slotName == "" {
+			return fmt.Errorf("%s reference slot name must not be empty", referenceContext(pipelineID, laneID, slot))
+		}
+		if _, ok := seen[slotName]; ok {
+			return fmt.Errorf("%s reference slot %q is duplicated after trimming", referenceContext(pipelineID, laneID, slot), slotName)
+		}
+		seen[slotName] = struct{}{}
+		if strings.TrimSpace(rawSource) == "" {
+			return fmt.Errorf("%s reference slot %q source must not be empty", referenceContext(pipelineID, laneID, slot), slotName)
+		}
+	}
+	return nil
+}
+
+func referenceContext(pipelineID string, laneID string, slot string) string {
+	if laneID != "" && slot != "" {
+		return fmt.Sprintf("pipeline %q lane %q %s", pipelineID, laneID, slot)
+	}
+	if laneID != "" {
+		return fmt.Sprintf("pipeline %q lane %q", pipelineID, laneID)
+	}
+	if slot != "" {
+		return fmt.Sprintf("pipeline %q %s", pipelineID, slot)
+	}
+	return fmt.Sprintf("pipeline %q", pipelineID)
+}
+
 func validateBindingLLMProfile(
 	pipelineID string,
 	laneID string,
 	slot string,
 	binding pipeline.ModuleBinding,
-	profiles map[string]LLMProfile,
 ) error {
-	profileID := strings.TrimSpace(binding.LLMProfile)
+	if binding.LLMProfile != "" && strings.TrimSpace(binding.LLMProfile) == "" {
-	if profileID == "" {
+		if laneID != "" {
-		profileID = pipeline.DefaultLLMProfile
+			return fmt.Errorf("pipeline %q lane %q %s llm_profile must not be empty when set", pipelineID, laneID, slot)
-	}
-	if hasLLMProfile(profiles, profileID) {
-		return nil
-	}
-	if laneID != "" {
-		return fmt.Errorf("pipeline %q lane %q %s references unknown LLM profile %q", pipelineID, laneID, slot, profileID)
-	}
-	return fmt.Errorf("pipeline %q %s references unknown LLM profile %q", pipelineID, slot, profileID)
-}
-
-func hasLLMProfile(profiles map[string]LLMProfile, profileID string) bool {
-	profileID = strings.TrimSpace(profileID)
-	for rawID := range profiles {
-		if strings.TrimSpace(rawID) == profileID {
-			return true
 		}
+		return fmt.Errorf("pipeline %q %s llm_profile must not be empty when set", pipelineID, slot)
 	}
-	return false
+	return nil
 }

internal/core/config/validation_test.go

@@ -1,343 +0,0 @@
-package config
-
-import (
-	"strings"
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
-)
-
-func TestValidateSuccessForValidConfig(t *testing.T) {
-	cfg := validConfig()
-
-	if err := cfg.Validate(); err != nil {
-		t.Fatalf("Validate: %v", err)
-	}
-}
-
-func TestValidateRejectsUnknownLLMProfileReferencedByBinding(t *testing.T) {
-	cfg := validConfig()
-	lane := cfg.Pipelines["example"].Artifacts["events"]
-	lane.Extract.LLMProfile = "missing"
-	cfg.Pipelines["example"].Artifacts["events"] = lane
-
-	err := cfg.Validate()
-	if err == nil || !strings.Contains(err.Error(), "unknown LLM profile") || !strings.Contains(err.Error(), "events") {
-		t.Fatalf("expected unknown LLM profile error with lane context, got %v", err)
-	}
-}
-
-func TestValidateRejectsInvalidProvider(t *testing.T) {
-	cfg := validConfig()
-	profile := cfg.LLMProfiles["default"]
-	profile.Provider = "unsupported"
-	cfg.LLMProfiles["default"] = profile
-
-	err := cfg.Validate()
-	if err == nil || !strings.Contains(err.Error(), "provider") {
-		t.Fatalf("expected provider error, got %v", err)
-	}
-}
-
-func TestValidateRejectsInvalidNumericFields(t *testing.T) {
-	tests := []struct {
-		name   string
-		mutate func(Config) Config
-		want   string
-	}{
-		{
-			name: "total concurrency",
-			mutate: func(cfg Config) Config {
-				cfg.Concurrency.TotalLLM = 0
-				return cfg
-			},
-			want: "total LLM concurrency",
-		},
-		{
-			name: "timeout",
-			mutate: func(cfg Config) Config {
-				profile := cfg.LLMProfiles["default"]
-				profile.TimeoutSeconds = -1
-				cfg.LLMProfiles["default"] = profile
-				return cfg
-			},
-			want: "timeout",
-		},
-		{
-			name: "max retries",
-			mutate: func(cfg Config) Config {
-				profile := cfg.LLMProfiles["default"]
-				profile.MaxRetries = -1
-				cfg.LLMProfiles["default"] = profile
-				return cfg
-			},
-			want: "max retries",
-		},
-		{
-			name: "max concurrency",
-			mutate: func(cfg Config) Config {
-				profile := cfg.LLMProfiles["default"]
-				profile.MaxConcurrency = -1
-				cfg.LLMProfiles["default"] = profile
-				return cfg
-			},
-			want: "max concurrency",
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			err := tc.mutate(validConfig()).Validate()
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestValidateAllowsPartialLLMProfileNumericConfig(t *testing.T) {
-	cfg := validConfig()
-	cfg.LLMProfiles["retry-only"] = LLMProfile{MaxRetries: 3}
-
-	if err := cfg.Validate(); err != nil {
-		t.Fatalf("Validate: %v", err)
-	}
-}
-
-func TestValidateRejectsInvalidDiagnosticsRetention(t *testing.T) {
-	cfg := validConfig()
-	cfg.Diagnostics.Retention = diagnostics.RetentionMode("sometimes")
-
-	err := cfg.Validate()
-	if err == nil || !strings.Contains(err.Error(), "retention") {
-		t.Fatalf("expected retention error, got %v", err)
-	}
-}
-
-func TestValidateRejectsEmptyIDs(t *testing.T) {
-	tests := []struct {
-		name   string
-		mutate func(Config) Config
-		want   string
-	}{
-		{
-			name: "LLM profile",
-			mutate: func(cfg Config) Config {
-				cfg.LLMProfiles[" "] = LLMProfile{}
-				return cfg
-			},
-			want: "LLM profile id",
-		},
-		{
-			name: "pipeline",
-			mutate: func(cfg Config) Config {
-				cfg.Pipelines[" "] = pipeline.PipelineProfile{}
-				return cfg
-			},
-			want: "pipeline id",
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			err := tc.mutate(validConfig()).Validate()
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestValidateRejectsIDsDuplicatedAfterTrimming(t *testing.T) {
-	tests := []struct {
-		name   string
-		mutate func(Config) Config
-		want   string
-	}{
-		{
-			name: "LLM profile",
-			mutate: func(cfg Config) Config {
-				cfg.LLMProfiles[" default "] = cfg.LLMProfiles["default"]
-				return cfg
-			},
-			want: "duplicated",
-		},
-		{
-			name: "pipeline",
-			mutate: func(cfg Config) Config {
-				cfg.Pipelines[" example "] = cfg.Pipelines["example"]
-				return cfg
-			},
-			want: "duplicated",
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			err := tc.mutate(validConfig()).Validate()
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestValidateUsesTrimmedLLMProfileIDs(t *testing.T) {
-	cfg := validConfig()
-	cfg.LLMProfiles[" default "] = cfg.LLMProfiles["default"]
-	delete(cfg.LLMProfiles, "default")
-
-	if err := cfg.Validate(); err != nil {
-		t.Fatalf("Validate: %v", err)
-	}
-	if _, ok := cfg.LLMProfile("default"); !ok {
-		t.Fatalf("expected trimmed LLM profile lookup to succeed")
-	}
-}
-
-func validConfig() Config {
-	cfg := Default()
-	profile := cfg.LLMProfiles["default"]
-	profile.BaseURL = "https://example.invalid/v1"
-	profile.Model = "test-model"
-	cfg.LLMProfiles["default"] = profile
-	cfg.Pipelines["example"] = pipeline.PipelineProfile{
-		Input: pipeline.Binding("fake/input"),
-		Artifacts: map[string]pipeline.ArtifactLaneProfile{
-			"events": {
-				Extract:    pipeline.Binding("fake/extract"),
-				Validators: []pipeline.ModuleBinding{pipeline.Binding("fake/validator")},
-			},
-			"notes": {
-				Extract: pipeline.Binding("fake/extract"),
-			},
-		},
-	}
-	return cfg
-}
-
-func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.ModuleCatalog {
-	t.Helper()
-	specs := map[string]pipeline.ModuleSpec{
-		"fake/input": {
-			Key:      "fake/input",
-			Stage:    pipeline.StageInput,
-			Provides: []string{"source"},
-		},
-		"generic": {
-			Key:      "generic",
-			Stage:    pipeline.StageChunk,
-			Requires: []string{"source"},
-			Provides: []string{"chunks"},
-		},
-		"fake/extract": {
-			Key:      "fake/extract",
-			Stage:    pipeline.StageExtract,
-			Requires: []string{"chunks"},
-			Provides: []string{"artifact"},
-		},
-		"appendorder": {
-			Key:      "appendorder",
-			Stage:    pipeline.StageMerge,
-			Requires: []string{"artifact"},
-			Provides: []string{"merged"},
-		},
-		"noop": {
-			Key:      "noop",
-			Stage:    pipeline.StageNormalize,
-			Requires: []string{"merged"},
-			Provides: []string{"normalized"},
-		},
-		"fake/validator": {
-			Key:      "fake/validator",
-			Stage:    pipeline.StageValidate,
-			Requires: []string{"normalized"},
-			Provides: []string{"validated"},
-		},
-		"json": {
-			Key:      "json",
-			Stage:    pipeline.StageOutput,
-			Requires: []string{"normalized"},
-		},
-	}
-	for _, override := range overrides {
-		specs[override.Key] = override
-	}
-
-	inputs := pipeline.NewInputAdapterRegistry()
-	chunkers := pipeline.NewChunkerRegistry()
-	extractors := pipeline.NewExtractorRegistry()
-	mergers := pipeline.NewMergerRegistry()
-	normalizers := pipeline.NewNormalizerRegistry()
-	validators := pipeline.NewValidatorRegistry()
-	outputs := pipeline.NewOutputEncoderRegistry()
-
-	mustRegisterInput(t, inputs, specs["fake/input"])
-	mustRegisterChunker(t, chunkers, specs["generic"])
-	mustRegisterExtractor(t, extractors, specs["fake/extract"])
-	mustRegisterMerger(t, mergers, specs["appendorder"])
-	mustRegisterNormalizer(t, normalizers, specs["noop"])
-	mustRegisterValidator(t, validators, specs["fake/validator"])
-	mustRegisterOutput(t, outputs, specs["json"])
-
-	return pipeline.ModuleCatalog{
-		Inputs:      inputs,
-		Chunkers:    chunkers,
-		Extractors:  extractors,
-		Mergers:     mergers,
-		Normalizers: normalizers,
-		Validators:  validators,
-		Outputs:     outputs,
-	}
-}
-
-func mustRegisterInput(t *testing.T, registry *pipeline.InputAdapterRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return nil, nil }); err != nil {
-		t.Fatalf("register input: %v", err)
-	}
-}
-
-func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return nil, nil }); err != nil {
-		t.Fatalf("register chunker: %v", err)
-	}
-}
-
-func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
-		t.Fatalf("register extractor: %v", err)
-	}
-}
-
-func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
-		t.Fatalf("register merger: %v", err)
-	}
-}
-
-func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
-		t.Fatalf("register normalizer: %v", err)
-	}
-}
-
-func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.Validator, error) { return nil, nil }); err != nil {
-		t.Fatalf("register validator: %v", err)
-	}
-}
-
-func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
-	t.Helper()
-	if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
-		t.Fatalf("register output: %v", err)
-	}
-}

internal/core/debugbundle/bundle.go

@@ -0,0 +1,89 @@
+// Package debugbundle owns explicitly requested per-run debug bundles.
+package debugbundle
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+const maxCreateAttempts = 16
+
+var utcNow = func() time.Time { return time.Now().UTC() }
+
+type Bundle struct {
+	path, summaryRoot, traceRoot string
+	createdAt                    time.Time
+}
+
+func Allocate(parent string) (*Bundle, error) {
+	parent = strings.TrimSpace(parent)
+	if parent == "" {
+		return nil, fmt.Errorf("debug parent must not be empty")
+	}
+	if err := os.MkdirAll(parent, 0o700); err != nil {
+		return nil, fmt.Errorf("create debug parent %q: %w", parent, err)
+	}
+	var last string
+	for attempt := 0; attempt < maxCreateAttempts; attempt++ {
+		createdAt := utcNow()
+		runID := fmt.Sprintf("run-%d", createdAt.UnixNano())
+		path := filepath.Join(parent, runID)
+		last = path
+		if err := os.Mkdir(path, 0o700); err != nil {
+			if os.IsExist(err) {
+				continue
+			}
+			return nil, fmt.Errorf("create debug bundle %q: %w", path, err)
+		}
+		summary, trace := filepath.Join(path, "summary"), filepath.Join(path, "trace")
+		if err := os.Mkdir(summary, 0o700); err != nil {
+			_ = os.Remove(path)
+			return nil, fmt.Errorf("create debug summary %q: %w", summary, err)
+		}
+		if err := os.Mkdir(trace, 0o700); err != nil {
+			_ = os.RemoveAll(path)
+			return nil, fmt.Errorf("create debug trace %q: %w", trace, err)
+		}
+		return &Bundle{path: path, summaryRoot: summary, traceRoot: trace, createdAt: createdAt}, nil
+	}
+	return nil, fmt.Errorf("create debug bundle %q: exhausted unique run ID attempts", last)
+}
+func (b *Bundle) Path() string {
+	if b == nil {
+		return ""
+	}
+	return b.path
+}
+func (b *Bundle) SummaryRoot() string {
+	if b == nil {
+		return ""
+	}
+	return b.summaryRoot
+}
+func (b *Bundle) TraceRoot() string {
+	if b == nil {
+		return ""
+	}
+	return b.traceRoot
+}
+func (b *Bundle) RunID() string {
+	if b == nil {
+		return ""
+	}
+	return filepath.Base(b.path)
+}
+func (b *Bundle) CreatedAt() time.Time {
+	if b == nil {
+		return time.Time{}
+	}
+	return b.createdAt
+}
+func (b *Bundle) Summary() *SummaryWriter {
+	if b == nil {
+		return nil
+	}
+	return &SummaryWriter{root: b.summaryRoot, runID: b.RunID(), createdAt: b.createdAt}
+}

internal/core/debugbundle/bundle_test.go

@@ -0,0 +1,75 @@
+package debugbundle
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+func TestAllocateCreatesRestrictiveSummaryAndTrace(t *testing.T) {
+	parent := t.TempDir()
+	fixed := time.Unix(0, 42).UTC()
+	previous := utcNow
+	utcNow = func() time.Time { return fixed }
+	defer func() { utcNow = previous }()
+	bundle, err := Allocate(parent)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if bundle.RunID() != "run-42" || bundle.SummaryRoot() != filepath.Join(bundle.Path(), "summary") || bundle.TraceRoot() != filepath.Join(bundle.Path(), "trace") {
+		t.Fatalf("bundle=%#v", bundle)
+	}
+	for _, path := range []string{bundle.Path(), bundle.SummaryRoot(), bundle.TraceRoot()} {
+		info, err := os.Stat(path)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if info.Mode().Perm() != 0o700 {
+			t.Fatalf("%s mode=%#o", path, info.Mode().Perm())
+		}
+	}
+	if err := bundle.Summary().WriteError("failed"); err != nil {
+		t.Fatal(err)
+	}
+	info, err := os.Stat(filepath.Join(bundle.SummaryRoot(), ArtifactErrorLog))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if info.Mode().Perm() != 0o600 {
+		t.Fatalf("file mode=%#o", info.Mode().Perm())
+	}
+}
+func TestAllocateRetriesAndDoesNotDeleteBundle(t *testing.T) {
+	parent := t.TempDir()
+	fixed := time.Unix(0, 9).UTC()
+	previous := utcNow
+	defer func() { utcNow = previous }()
+	calls := 0
+	utcNow = func() time.Time { calls++; return fixed.Add(time.Duration(calls-1) * time.Nanosecond) }
+	if err := os.Mkdir(filepath.Join(parent, "run-9"), 0o700); err != nil {
+		t.Fatal(err)
+	}
+	bundle, err := Allocate(parent)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if bundle.RunID() != "run-10" {
+		t.Fatalf("run id=%q", bundle.RunID())
+	}
+	if _, err := os.Stat(bundle.Path()); err != nil {
+		t.Fatal(err)
+	}
+}
+func TestSummaryWriterConfinesArtifacts(t *testing.T) {
+	bundle, err := Allocate(t.TempDir())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := bundle.Summary().WriteJSON("../outside.json", map[string]any{}); err == nil {
+		t.Fatal("accepted traversal")
+	}
+	if err := bundle.Summary().WriteBytes(`trace\\x`, []byte("x")); err == nil {
+		t.Fatal("accepted backslash")
+	}
+}

internal/core/debugbundle/summary.go

@@ -0,0 +1,120 @@
+package debugbundle
+
+import (
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
+	"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+)
+
+const (
+	ArtifactInvocationMetadata = "invocation.json"
+	ArtifactEffectiveConfig    = "effective-config.json"
+	ArtifactResolvedPipeline   = "resolved-pipeline.json"
+	ArtifactResolvedReferences = "resolved-references.json"
+	ArtifactCheckpointEvents   = "checkpoint-events.json"
+	ArtifactRunManifest        = "run-manifest.json"
+	ArtifactChunkPlan          = "chunk-plan.json"
+	ArtifactRunReport          = "run-report.json"
+	ArtifactWarnings           = "warnings.json"
+	ArtifactErrorLog           = "error.log"
+)
+
+type RedactedSummaryPayload interface{ RedactedSummaryPayload() any }
+type Invocation struct {
+	Operation          string    `json:"operation"`
+	PipelineID         string    `json:"pipeline_id,omitempty"`
+	PipelineDigest     string    `json:"pipeline_digest,omitempty"`
+	Resume             bool      `json:"resume,omitempty"`
+	InputPath          string    `json:"input_path,omitempty"`
+	ConfigPath         string    `json:"config_path,omitempty"`
+	ConfigSource       string    `json:"config_source,omitempty"`
+	OnlyLanes          []string  `json:"only_lanes,omitempty"`
+	ChunkCacheOverride string    `json:"chunk_cache_override,omitempty"`
+	RunID              string    `json:"run_id"`
+	StartedAt          time.Time `json:"started_at"`
+}
+type RunReport struct {
+	RunID            string `json:"run_id"`
+	PipelineID       string `json:"pipeline_id"`
+	OutputPath       string `json:"output_path,omitempty"`
+	DebugPath        string `json:"debug_path,omitempty"`
+	Succeeded        bool   `json:"succeeded"`
+	OutputCount      int    `json:"output_count"`
+	RejectedCount    int    `json:"rejected_count"`
+	WarningCount     int    `json:"warning_count"`
+	ValidationStatus string `json:"validation_status,omitempty"`
+}
+type SummaryWriter struct {
+	root, runID string
+	createdAt   time.Time
+}
+
+func (w *SummaryWriter) WriteInvocation(payload Invocation) error {
+	if w == nil {
+		return fmt.Errorf("debug summary writer must not be nil")
+	}
+	if payload.RunID == "" {
+		payload.RunID = w.runID
+	}
+	if payload.StartedAt.IsZero() {
+		payload.StartedAt = w.createdAt
+	}
+	return w.WriteJSON(ArtifactInvocationMetadata, payload)
+}
+func (w *SummaryWriter) WriteRedactedEffectiveConfig(payload RedactedSummaryPayload) error {
+	if payload == nil {
+		return fmt.Errorf("redacted summary payload must not be nil")
+	}
+	return w.WriteJSON(ArtifactEffectiveConfig, payload.RedactedSummaryPayload())
+}
+func (w *SummaryWriter) WriteResolvedPipeline(v any) error {
+	return w.WriteJSON(ArtifactResolvedPipeline, v)
+}
+func (w *SummaryWriter) WriteResolvedReferences(v any) error {
+	return w.WriteJSON(ArtifactResolvedReferences, v)
+}
+func (w *SummaryWriter) WriteCheckpointEvents(v any) error {
+	return w.WriteJSON(ArtifactCheckpointEvents, v)
+}
+func (w *SummaryWriter) WriteRunManifest(v artifacts.RunManifest) error {
+	return w.WriteJSON(ArtifactRunManifest, v)
+}
+func (w *SummaryWriter) WriteChunkPlan(v artifacts.ChunkPlanSummary) error {
+	return w.WriteJSON(ArtifactChunkPlan, v)
+}
+func (w *SummaryWriter) WriteRunReport(v RunReport) error { return w.WriteJSON(ArtifactRunReport, v) }
+func (w *SummaryWriter) WriteWarnings(v []contracts.Warning) error {
+	return w.WriteJSON(ArtifactWarnings, v)
+}
+func (w *SummaryWriter) WriteError(message string) error {
+	return w.WriteBytes(ArtifactErrorLog, []byte(message+"\n"))
+}
+func (w *SummaryWriter) WriteJSON(name string, v any) error {
+	if w == nil {
+		return fmt.Errorf("debug summary writer must not be nil")
+	}
+	if err := fileio.WriteJSON(w.root, summaryName(name), v, 0o700, 0o600); err != nil {
+		return fmt.Errorf("write debug summary artifact %q: %w", name, err)
+	}
+	return nil
+}
+func (w *SummaryWriter) WriteBytes(name string, v []byte) error {
+	if w == nil {
+		return fmt.Errorf("debug summary writer must not be nil")
+	}
+	if err := fileio.WriteBytes(w.root, summaryName(name), v, 0o700, 0o600); err != nil {
+		return fmt.Errorf("write debug summary artifact %q: %w", name, err)
+	}
+	return nil
+}
+func summaryName(name string) string {
+	name = strings.TrimSpace(name)
+	if name == "" || strings.ContainsAny(name, "/\\") {
+		return "../invalid"
+	}
+	return name
+}

internal/core/diagnostics/artifacts.go

@@ -1,12 +0,0 @@
-package diagnostics
-
-const (
-	ArtifactInvocationMetadata = "invocation.json"
-	ArtifactEffectiveConfig    = "effective-config.json"
-	ArtifactResolvedPipeline   = "resolved-pipeline.json"
-	ArtifactSourceDocument     = "source-document.json"
-	ArtifactRunManifest        = "run-manifest.json"
-	ArtifactRunReport          = "run-report.json"
-	ArtifactWarnings           = "warnings.json"
-	ArtifactErrorLog           = "error.log"
-)

internal/core/diagnostics/artifacts_test.go

@@ -1,22 +0,0 @@
-package diagnostics
-
-import "testing"
-
-func TestArtifactNamesUseExtractionOrientedNames(t *testing.T) {
-	names := []string{
-		ArtifactInvocationMetadata,
-		ArtifactEffectiveConfig,
-		ArtifactResolvedPipeline,
-		ArtifactSourceDocument,
-		ArtifactRunManifest,
-		ArtifactRunReport,
-		ArtifactWarnings,
-		ArtifactErrorLog,
-	}
-
-	for _, name := range names {
-		if name == "" {
-			t.Fatalf("artifact name must not be empty")
-		}
-	}
-}

internal/core/diagnostics/run_dir.go

@@ -1,279 +0,0 @@
-package diagnostics
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-	"time"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
-)
-
-const (
-	defaultWorkDir                = "/tmp/notarius"
-	maxRunDirectoryCreateAttempts = 16
-)
-
-var utcNow = func() time.Time {
-	return time.Now().UTC()
-}
-
-// RunDirectory represents a per-run diagnostics directory.
-type RunDirectory struct {
-	path      string
-	retention RetentionMode
-	createdAt time.Time
-}
-
-type RetentionMode string
-
-const (
-	RetentionAuto   RetentionMode = "auto"
-	RetentionAlways RetentionMode = "always"
-	RetentionNever  RetentionMode = "never"
-)
-
-type RetentionDecisionInput struct {
-	RetentionMode RetentionMode
-	RunSucceeded  bool
-	HasWarnings   bool
-}
-
-type RedactedEffectiveConfigPayload interface {
-	RedactedDiagnosticsPayload() any
-}
-
-// InvocationMetadata captures non-secret invocation details for diagnostics.
-type InvocationMetadata struct {
-	Operation      string    `json:"operation"`
-	PipelineID     string    `json:"pipeline_id,omitempty"`
-	PipelineDigest string    `json:"pipeline_digest,omitempty"`
-	InputPath      string    `json:"input_path,omitempty"`
-	ConfigPath     string    `json:"config_path,omitempty"`
-	ConfigSource   string    `json:"config_source,omitempty"`
-	OnlyLanes      []string  `json:"only_lanes,omitempty"`
-	RunID          string    `json:"run_id"`
-	StartedAt      time.Time `json:"started_at"`
-}
-
-func ShouldRetainRunDirectory(input RetentionDecisionInput) bool {
-	if !input.RunSucceeded {
-		return true
-	}
-
-	switch input.RetentionMode {
-	case RetentionAlways:
-		return true
-	case RetentionNever:
-		return false
-	case RetentionAuto, "":
-		return input.HasWarnings
-	default:
-		return true
-	}
-}
-
-func NewRunDirectory(workDir string, retention RetentionMode) (*RunDirectory, error) {
-	if strings.TrimSpace(workDir) == "" {
-		workDir = defaultWorkDir
-	}
-	if retention == "" {
-		retention = RetentionAuto
-	}
-
-	if err := os.MkdirAll(workDir, 0o755); err != nil {
-		return nil, fmt.Errorf("create diagnostics work directory %q: %w", workDir, err)
-	}
-
-	var lastRunPath string
-	for attempt := 0; attempt < maxRunDirectoryCreateAttempts; attempt++ {
-		createdAt := utcNow()
-		runID := fmt.Sprintf("run-%d", createdAt.UnixNano())
-		runPath := filepath.Join(workDir, runID)
-		lastRunPath = runPath
-		if err := os.Mkdir(runPath, 0o755); err != nil {
-			if os.IsExist(err) {
-				continue
-			}
-			return nil, fmt.Errorf("create diagnostics run directory %q: %w", runPath, err)
-		}
-
-		return &RunDirectory{
-			path:      runPath,
-			retention: retention,
-			createdAt: createdAt,
-		}, nil
-	}
-
-	return nil, fmt.Errorf("create diagnostics run directory %q: exhausted unique run ID attempts", lastRunPath)
-}
-
-func (r *RunDirectory) Path() string {
-	if r == nil {
-		return ""
-	}
-	return r.path
-}
-
-func (r *RunDirectory) RunID() string {
-	if r == nil {
-		return ""
-	}
-	return filepath.Base(r.path)
-}
-
-func (r *RunDirectory) WriteInvocationMetadata(metadata InvocationMetadata) error {
-	if r == nil {
-		return fmt.Errorf("run directory must not be nil")
-	}
-	if metadata.RunID == "" {
-		metadata.RunID = r.RunID()
-	}
-	if metadata.StartedAt.IsZero() {
-		metadata.StartedAt = r.createdAt
-	}
-	return r.WriteJSONArtifact(ArtifactInvocationMetadata, metadata)
-}
-
-func (r *RunDirectory) WriteRedactedEffectiveConfig(payload RedactedEffectiveConfigPayload) error {
-	if payload == nil {
-		return fmt.Errorf("redacted effective config payload must not be nil")
-	}
-	return r.WriteJSONArtifact(ArtifactEffectiveConfig, payload.RedactedDiagnosticsPayload())
-}
-
-func (r *RunDirectory) WriteResolvedPipeline(payload any) error {
-	return r.WriteJSONArtifact(ArtifactResolvedPipeline, payload)
-}
-
-func (r *RunDirectory) WriteSourceDocument(payload any) error {
-	return r.WriteJSONArtifact(ArtifactSourceDocument, payload)
-}
-
-func (r *RunDirectory) WriteRunManifest(manifest artifacts.RunManifest) error {
-	return r.WriteJSONArtifact(ArtifactRunManifest, manifest)
-}
-
-func (r *RunDirectory) WriteRunReport(payload any) error {
-	return r.WriteJSONArtifact(ArtifactRunReport, payload)
-}
-
-func (r *RunDirectory) WriteWarnings(warnings []contracts.Warning) error {
-	return r.WriteJSONArtifact(ArtifactWarnings, warnings)
-}
-
-func (r *RunDirectory) WriteErrorLog(errorMessage string) error {
-	if r == nil {
-		return fmt.Errorf("run directory must not be nil")
-	}
-	path, err := r.artifactPath(ArtifactErrorLog)
-	if err != nil {
-		return err
-	}
-	if err := writeFileAtomic(path, []byte(errorMessage+"\n"), 0o644); err != nil {
-		return fmt.Errorf("write diagnostics artifact %q: %w", ArtifactErrorLog, err)
-	}
-	return nil
-}
-
-func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error {
-	if r == nil {
-		return fmt.Errorf("run directory must not be nil")
-	}
-	path, err := r.artifactPath(name)
-	if err != nil {
-		return err
-	}
-
-	data, err := json.MarshalIndent(payload, "", "  ")
-	if err != nil {
-		return fmt.Errorf("marshal diagnostics artifact %q: %w", name, err)
-	}
-	data = append(data, '\n')
-	if err := writeFileAtomic(path, data, 0o644); err != nil {
-		return fmt.Errorf("write diagnostics artifact %q: %w", name, err)
-	}
-	return nil
-}
-
-func (r *RunDirectory) ApplyRetention(input RetentionDecisionInput) error {
-	if r == nil {
-		return fmt.Errorf("run directory must not be nil")
-	}
-	decision := input
-	if decision.RetentionMode == "" {
-		decision.RetentionMode = r.retention
-	}
-	if ShouldRetainRunDirectory(decision) {
-		return nil
-	}
-	if err := os.RemoveAll(r.path); err != nil {
-		return fmt.Errorf("remove diagnostics run directory %q: %w", r.path, err)
-	}
-	return nil
-}
-
-func (r *RunDirectory) artifactPath(name string) (string, error) {
-	name = strings.TrimSpace(name)
-	if name == "" {
-		return "", fmt.Errorf("diagnostics artifact name must not be empty")
-	}
-	if filepath.IsAbs(name) {
-		return "", fmt.Errorf("diagnostics artifact name %q must not be absolute", name)
-	}
-	if name != filepath.Base(name) || strings.Contains(name, "/") || strings.Contains(name, `\`) {
-		return "", fmt.Errorf("diagnostics artifact name %q must not contain path separators", name)
-	}
-
-	runPath, err := filepath.Abs(r.path)
-	if err != nil {
-		return "", fmt.Errorf("resolve diagnostics run directory %q: %w", r.path, err)
-	}
-	artifactPath, err := filepath.Abs(filepath.Join(runPath, name))
-	if err != nil {
-		return "", fmt.Errorf("resolve diagnostics artifact %q: %w", name, err)
-	}
-	if filepath.Dir(artifactPath) != runPath {
-		return "", fmt.Errorf("diagnostics artifact name %q resolves outside run directory", name)
-	}
-	return artifactPath, nil
-}
-
-func writeFileAtomic(path string, data []byte, perm os.FileMode) error {
-	dir := filepath.Dir(path)
-	if err := os.MkdirAll(dir, 0o755); err != nil {
-		return err
-	}
-
-	temp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".tmp-*")
-	if err != nil {
-		return err
-	}
-	tempPath := temp.Name()
-	removeTemp := true
-	defer func() {
-		if removeTemp {
-			_ = os.Remove(tempPath)
-		}
-	}()
-
-	if _, err := temp.Write(data); err != nil {
-		_ = temp.Close()
-		return err
-	}
-	if err := temp.Chmod(perm); err != nil {
-		_ = temp.Close()
-		return err
-	}
-	if err := temp.Close(); err != nil {
-		return err
-	}
-	if err := os.Rename(tempPath, path); err != nil {
-		return err
-	}
-	removeTemp = false
-	return nil
-}

internal/core/diagnostics/run_dir_test.go

@@ -1,366 +0,0 @@
-package diagnostics
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"path/filepath"
-	"regexp"
-	"strings"
-	"testing"
-	"time"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
-)
-
-func TestNewRunDirectoryCreatesRunDirectoryAndRunID(t *testing.T) {
-	workDir := t.TempDir()
-	runDir, err := NewRunDirectory(workDir, RetentionAuto)
-	if err != nil {
-		t.Fatalf("NewRunDirectory: %v", err)
-	}
-
-	if filepath.Dir(runDir.Path()) != workDir {
-		t.Fatalf("unexpected run directory parent: %q", runDir.Path())
-	}
-	if ok := regexp.MustCompile(`^run-\d+$`).MatchString(runDir.RunID()); !ok {
-		t.Fatalf("unexpected run ID: %q", runDir.RunID())
-	}
-	info, err := os.Stat(runDir.Path())
-	if err != nil {
-		t.Fatalf("stat run directory: %v", err)
-	}
-	if !info.IsDir() {
-		t.Fatalf("expected run path to be a directory")
-	}
-}
-
-func TestNewRunDirectoryRetriesOnRunIDCollision(t *testing.T) {
-	workDir := t.TempDir()
-	first := time.Unix(0, 100).UTC()
-	second := first.Add(time.Nanosecond)
-	if err := os.Mkdir(filepath.Join(workDir, fmt.Sprintf("run-%d", first.UnixNano())), 0o755); err != nil {
-		t.Fatalf("create existing run directory: %v", err)
-	}
-	restoreUTCNow := replaceUTCNow(func() func() time.Time {
-		calls := 0
-		return func() time.Time {
-			calls++
-			if calls == 1 {
-				return first
-			}
-			return second
-		}
-	}())
-	t.Cleanup(restoreUTCNow)
-
-	runDir, err := NewRunDirectory(workDir, RetentionAuto)
-	if err != nil {
-		t.Fatalf("NewRunDirectory: %v", err)
-	}
-
-	wantRunID := fmt.Sprintf("run-%d", second.UnixNano())
-	if runDir.RunID() != wantRunID {
-		t.Fatalf("RunID = %q, want %q", runDir.RunID(), wantRunID)
-	}
-	if _, err := os.Stat(runDir.Path()); err != nil {
-		t.Fatalf("stat run directory: %v", err)
-	}
-}
-
-func TestNewRunDirectoryReturnsErrorAfterRunIDCollisionsExhausted(t *testing.T) {
-	workDir := t.TempDir()
-	collisionTime := time.Unix(0, 200).UTC()
-	collisionPath := filepath.Join(workDir, fmt.Sprintf("run-%d", collisionTime.UnixNano()))
-	if err := os.Mkdir(collisionPath, 0o755); err != nil {
-		t.Fatalf("create existing run directory: %v", err)
-	}
-	restoreUTCNow := replaceUTCNow(func() time.Time {
-		return collisionTime
-	})
-	t.Cleanup(restoreUTCNow)
-
-	_, err := NewRunDirectory(workDir, RetentionAuto)
-	if err == nil || !strings.Contains(err.Error(), "exhausted unique run ID attempts") {
-		t.Fatalf("expected exhausted collision error, got %v", err)
-	}
-}
-
-func TestNewRunDirectoryUsesDefaultWorkDirectory(t *testing.T) {
-	runDir, err := NewRunDirectory("", RetentionAuto)
-	if err != nil {
-		t.Fatalf("NewRunDirectory: %v", err)
-	}
-	t.Cleanup(func() {
-		_ = os.RemoveAll(runDir.Path())
-		_ = os.Remove(defaultWorkDir)
-	})
-
-	if filepath.Dir(runDir.Path()) != defaultWorkDir {
-		t.Fatalf("expected default work directory %q, got %q", defaultWorkDir, filepath.Dir(runDir.Path()))
-	}
-}
-
-func TestWriteJSONArtifactWritesIndentedNewlineTerminatedJSON(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.WriteJSONArtifact("artifact.json", map[string]any{"value": "ok"}); err != nil {
-		t.Fatalf("WriteJSONArtifact: %v", err)
-	}
-
-	data := readArtifact(t, runDir, "artifact.json")
-	if !strings.HasSuffix(string(data), "\n") {
-		t.Fatalf("expected trailing newline, got %q", data)
-	}
-	if !strings.Contains(string(data), "\n  \"value\": \"ok\"\n") {
-		t.Fatalf("expected indented JSON, got %s", data)
-	}
-}
-
-func TestWriteJSONArtifactLeavesNoTemporaryFiles(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.WriteJSONArtifact("artifact.json", map[string]any{"value": "ok"}); err != nil {
-		t.Fatalf("WriteJSONArtifact: %v", err)
-	}
-
-	entries, err := os.ReadDir(runDir.Path())
-	if err != nil {
-		t.Fatalf("read run directory: %v", err)
-	}
-	for _, entry := range entries {
-		if strings.Contains(entry.Name(), ".tmp-") {
-			t.Fatalf("temporary diagnostics file remains after success: %s", entry.Name())
-		}
-	}
-}
-
-func TestWriteInvocationMetadataFillsMissingRunIDAndStartTime(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.WriteInvocationMetadata(InvocationMetadata{Operation: "validate"}); err != nil {
-		t.Fatalf("WriteInvocationMetadata: %v", err)
-	}
-
-	var got InvocationMetadata
-	if err := json.Unmarshal(readArtifact(t, runDir, ArtifactInvocationMetadata), &got); err != nil {
-		t.Fatalf("unmarshal invocation metadata: %v", err)
-	}
-	if got.RunID != runDir.RunID() {
-		t.Fatalf("unexpected run ID: got %q want %q", got.RunID, runDir.RunID())
-	}
-	if got.StartedAt.IsZero() {
-		t.Fatalf("expected started_at to be filled")
-	}
-	if got.Operation != "validate" {
-		t.Fatalf("unexpected operation: %q", got.Operation)
-	}
-}
-
-func TestWriteInvocationMetadataPreservesProvidedRunIDAndStartTime(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-	startedAt := time.Date(2026, 7, 3, 12, 0, 0, 0, time.UTC)
-
-	if err := runDir.WriteInvocationMetadata(InvocationMetadata{
-		Operation: "validate",
-		RunID:     "provided",
-		StartedAt: startedAt,
-	}); err != nil {
-		t.Fatalf("WriteInvocationMetadata: %v", err)
-	}
-
-	var got InvocationMetadata
-	if err := json.Unmarshal(readArtifact(t, runDir, ArtifactInvocationMetadata), &got); err != nil {
-		t.Fatalf("unmarshal invocation metadata: %v", err)
-	}
-	if got.RunID != "provided" {
-		t.Fatalf("unexpected run ID: %q", got.RunID)
-	}
-	if !got.StartedAt.Equal(startedAt) {
-		t.Fatalf("unexpected started_at: %s", got.StartedAt)
-	}
-}
-
-func TestWriteTypedArtifacts(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.WriteRedactedEffectiveConfig(fakeRedactedEffectiveConfig{payload: map[string]any{"redacted": true}}); err != nil {
-		t.Fatalf("WriteRedactedEffectiveConfig: %v", err)
-	}
-	if err := runDir.WriteResolvedPipeline(map[string]any{"pipeline": "test"}); err != nil {
-		t.Fatalf("WriteResolvedPipeline: %v", err)
-	}
-	if err := runDir.WriteSourceDocument(map[string]any{"source_id": "source-1"}); err != nil {
-		t.Fatalf("WriteSourceDocument: %v", err)
-	}
-	if err := runDir.WriteRunManifest(artifacts.RunManifest{RunID: "run-1"}); err != nil {
-		t.Fatalf("WriteRunManifest: %v", err)
-	}
-	if err := runDir.WriteRunReport(map[string]any{"ok": true}); err != nil {
-		t.Fatalf("WriteRunReport: %v", err)
-	}
-	if err := runDir.WriteWarnings([]contracts.Warning{{ReasonCode: "test", Message: "warning"}}); err != nil {
-		t.Fatalf("WriteWarnings: %v", err)
-	}
-
-	for _, name := range []string{
-		ArtifactEffectiveConfig,
-		ArtifactResolvedPipeline,
-		ArtifactSourceDocument,
-		ArtifactRunManifest,
-		ArtifactRunReport,
-		ArtifactWarnings,
-	} {
-		if _, err := os.Stat(filepath.Join(runDir.Path(), name)); err != nil {
-			t.Fatalf("expected artifact %q: %v", name, err)
-		}
-	}
-}
-
-func TestWriteRedactedEffectiveConfigWritesPayloadReturnedByProvider(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.WriteRedactedEffectiveConfig(fakeRedactedEffectiveConfig{
-		payload: map[string]any{
-			"api_key": "[REDACTED]",
-			"model":   "test-model",
-		},
-	}); err != nil {
-		t.Fatalf("WriteRedactedEffectiveConfig: %v", err)
-	}
-
-	data := string(readArtifact(t, runDir, ArtifactEffectiveConfig))
-	if !strings.Contains(data, `"api_key": "[REDACTED]"`) || !strings.Contains(data, `"model": "test-model"`) {
-		t.Fatalf("unexpected effective config artifact: %s", data)
-	}
-}
-
-func TestWriteErrorLogWritesPlainTextWithTrailingNewline(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.WriteErrorLog("something failed"); err != nil {
-		t.Fatalf("WriteErrorLog: %v", err)
-	}
-
-	if got := string(readArtifact(t, runDir, ArtifactErrorLog)); got != "something failed\n" {
-		t.Fatalf("unexpected error log: %q", got)
-	}
-}
-
-func TestArtifactPathRejectsUnsafeNames(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	tests := []string{
-		"",
-		" ",
-		"/absolute.json",
-		"nested/artifact.json",
-		`nested\artifact.json`,
-		"../escape.json",
-	}
-
-	for _, name := range tests {
-		t.Run(name, func(t *testing.T) {
-			if err := runDir.WriteJSONArtifact(name, map[string]any{}); err == nil {
-				t.Fatalf("expected unsafe artifact name %q to be rejected", name)
-			}
-		})
-	}
-}
-
-func TestShouldRetainRunDirectoryDecisions(t *testing.T) {
-	tests := []struct {
-		name  string
-		input RetentionDecisionInput
-		want  bool
-	}{
-		{name: "failed auto retained", input: RetentionDecisionInput{RetentionMode: RetentionAuto, RunSucceeded: false}, want: true},
-		{name: "failed always retained", input: RetentionDecisionInput{RetentionMode: RetentionAlways, RunSucceeded: false}, want: true},
-		{name: "failed never retained", input: RetentionDecisionInput{RetentionMode: RetentionNever, RunSucceeded: false}, want: true},
-		{name: "successful always retained", input: RetentionDecisionInput{RetentionMode: RetentionAlways, RunSucceeded: true}, want: true},
-		{name: "successful never removed", input: RetentionDecisionInput{RetentionMode: RetentionNever, RunSucceeded: true}, want: false},
-		{name: "successful auto without warnings removed", input: RetentionDecisionInput{RetentionMode: RetentionAuto, RunSucceeded: true}, want: false},
-		{name: "successful auto with warnings retained", input: RetentionDecisionInput{RetentionMode: RetentionAuto, RunSucceeded: true, HasWarnings: true}, want: true},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			if got := ShouldRetainRunDirectory(tc.input); got != tc.want {
-				t.Fatalf("ShouldRetainRunDirectory() = %v, want %v", got, tc.want)
-			}
-		})
-	}
-}
-
-func TestApplyRetentionRemovesOnlyRunDirectory(t *testing.T) {
-	workDir := t.TempDir()
-	runDir, err := NewRunDirectory(workDir, RetentionNever)
-	if err != nil {
-		t.Fatalf("NewRunDirectory: %v", err)
-	}
-	siblingPath := filepath.Join(workDir, "sibling")
-	if err := os.WriteFile(siblingPath, []byte("keep"), 0o644); err != nil {
-		t.Fatalf("write sibling: %v", err)
-	}
-
-	if err := runDir.ApplyRetention(RetentionDecisionInput{RunSucceeded: true}); err != nil {
-		t.Fatalf("ApplyRetention: %v", err)
-	}
-
-	if _, err := os.Stat(runDir.Path()); !os.IsNotExist(err) {
-		t.Fatalf("expected run directory removed, stat err=%v", err)
-	}
-	if _, err := os.Stat(workDir); err != nil {
-		t.Fatalf("expected work directory retained: %v", err)
-	}
-	if _, err := os.Stat(siblingPath); err != nil {
-		t.Fatalf("expected sibling retained: %v", err)
-	}
-}
-
-func TestApplyRetentionKeepsRetainedRunDirectory(t *testing.T) {
-	runDir := newTestRunDirectory(t)
-
-	if err := runDir.ApplyRetention(RetentionDecisionInput{RunSucceeded: true, HasWarnings: true}); err != nil {
-		t.Fatalf("ApplyRetention: %v", err)
-	}
-
-	if _, err := os.Stat(runDir.Path()); err != nil {
-		t.Fatalf("expected run directory retained: %v", err)
-	}
-}
-
-func newTestRunDirectory(t *testing.T) *RunDirectory {
-	t.Helper()
-	runDir, err := NewRunDirectory(t.TempDir(), RetentionAuto)
-	if err != nil {
-		t.Fatalf("NewRunDirectory: %v", err)
-	}
-	return runDir
-}
-
-func replaceUTCNow(replacement func() time.Time) func() {
-	original := utcNow
-	utcNow = replacement
-	return func() {
-		utcNow = original
-	}
-}
-
-func readArtifact(t *testing.T, runDir *RunDirectory, name string) []byte {
-	t.Helper()
-	data, err := os.ReadFile(filepath.Join(runDir.Path(), name))
-	if err != nil {
-		t.Fatalf("read artifact %q: %v", name, err)
-	}
-	return data
-}
-
-type fakeRedactedEffectiveConfig struct {
-	payload any
-}
-
-func (f fakeRedactedEffectiveConfig) RedactedDiagnosticsPayload() any {
-	return f.payload
-}

internal/core/fileio/fileio.go

@@ -0,0 +1,102 @@
+// Package fileio provides confined, atomic artifact writes.
+package fileio
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+)
+
+func SafePath(root, name string) (string, error) {
+	root = strings.TrimSpace(root)
+	if root == "" {
+		return "", fmt.Errorf("file root must not be empty")
+	}
+	name = strings.TrimSpace(name)
+	if name == "" {
+		return "", fmt.Errorf("artifact name must not be empty")
+	}
+	if strings.Contains(name, `\\`) {
+		return "", fmt.Errorf("artifact name %q must use slash-separated relative paths", name)
+	}
+	if path.IsAbs(name) || filepath.IsAbs(name) {
+		return "", fmt.Errorf("artifact name %q must be relative", name)
+	}
+	if name == "." || strings.Contains(name, "..") {
+		return "", fmt.Errorf("artifact name %q must not contain ..", name)
+	}
+	if path.Clean(name) != name {
+		return "", fmt.Errorf("artifact name %q must be clean", name)
+	}
+	absRoot, err := filepath.Abs(root)
+	if err != nil {
+		return "", fmt.Errorf("resolve file root %q: %w", root, err)
+	}
+	target, err := filepath.Abs(filepath.Join(absRoot, filepath.FromSlash(name)))
+	if err != nil {
+		return "", fmt.Errorf("resolve artifact %q: %w", name, err)
+	}
+	rel, err := filepath.Rel(absRoot, target)
+	if err != nil {
+		return "", fmt.Errorf("resolve artifact %q: %w", name, err)
+	}
+	if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
+		return "", fmt.Errorf("artifact name %q resolves outside file root", name)
+	}
+	return target, nil
+}
+
+func WriteJSON(root, name string, payload any, dirMode, fileMode os.FileMode) error {
+	data, err := json.MarshalIndent(payload, "", "  ")
+	if err != nil {
+		return fmt.Errorf("marshal artifact %q: %w", name, err)
+	}
+	return WriteBytes(root, name, append(data, '\n'), dirMode, fileMode)
+}
+
+func WriteBytes(root, name string, data []byte, dirMode, fileMode os.FileMode) error {
+	target, err := SafePath(root, name)
+	if err != nil {
+		return err
+	}
+	if err := writeAtomic(target, data, dirMode, fileMode); err != nil {
+		return fmt.Errorf("write artifact %q: %w", name, err)
+	}
+	return nil
+}
+
+func writeAtomic(target string, data []byte, dirMode, fileMode os.FileMode) error {
+	if err := os.MkdirAll(filepath.Dir(target), dirMode); err != nil {
+		return err
+	}
+	temp, err := os.CreateTemp(filepath.Dir(target), "."+filepath.Base(target)+".tmp-*")
+	if err != nil {
+		return err
+	}
+	tempPath := temp.Name()
+	keep := true
+	defer func() {
+		if keep {
+			_ = os.Remove(tempPath)
+		}
+	}()
+	if _, err := temp.Write(data); err != nil {
+		_ = temp.Close()
+		return err
+	}
+	if err := temp.Chmod(fileMode); err != nil {
+		_ = temp.Close()
+		return err
+	}
+	if err := temp.Close(); err != nil {
+		return err
+	}
+	if err := os.Rename(tempPath, target); err != nil {
+		return err
+	}
+	keep = false
+	return nil
+}

internal/core/fileio/fileio_test.go

@@ -0,0 +1,41 @@
+package fileio
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestSafePathRejectsUnsafeNames(t *testing.T) {
+	for _, name := range []string{"/tmp/x", "a/../x", "a//x", `a\\x`} {
+		if _, err := SafePath(t.TempDir(), name); err == nil {
+			t.Fatalf("SafePath(%q) accepted unsafe path", name)
+		}
+	}
+}
+
+func TestWriteBytesIsAtomicAndUsesRequestedModes(t *testing.T) {
+	root := t.TempDir()
+	if err := WriteBytes(root, "nested/value", []byte("value"), 0o700, 0o600); err != nil {
+		t.Fatal(err)
+	}
+	for path, want := range map[string]os.FileMode{filepath.Join(root, "nested"): 0o700, filepath.Join(root, "nested", "value"): 0o600} {
+		info, err := os.Stat(path)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if info.Mode().Perm() != want {
+			t.Fatalf("%s mode=%#o want %#o", path, info.Mode().Perm(), want)
+		}
+	}
+	entries, err := os.ReadDir(filepath.Join(root, "nested"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, entry := range entries {
+		if strings.Contains(entry.Name(), ".tmp-") {
+			t.Fatalf("temporary file remains: %s", entry.Name())
+		}
+	}
+}

internal/core/source/chunk_plan.go

@@ -0,0 +1,209 @@
+package source
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"strings"
+)
+
+// CanonicalizeChunkAnnotations validates annotation namespaces and JSON values
+// and returns an independently owned map whose values use canonical JSON bytes.
+func CanonicalizeChunkAnnotations(annotations ChunkAnnotations) (ChunkAnnotations, error) {
+	if len(annotations) == 0 {
+		return nil, nil
+	}
+	canonical := make(ChunkAnnotations, len(annotations))
+	for namespace, raw := range annotations {
+		if strings.TrimSpace(namespace) == "" {
+			return nil, fmt.Errorf("chunk annotation namespace must not be empty")
+		}
+		if strings.TrimSpace(namespace) != namespace {
+			return nil, fmt.Errorf("chunk annotation namespace %q must not contain leading or trailing whitespace", namespace)
+		}
+		value, err := decodeAnnotation(raw)
+		if err != nil {
+			return nil, fmt.Errorf("chunk annotation %q: %w", namespace, err)
+		}
+		encoded, err := json.Marshal(value)
+		if err != nil {
+			return nil, fmt.Errorf("chunk annotation %q contains an unsupported value: %w", namespace, err)
+		}
+		canonical[namespace] = encoded
+	}
+	return canonical, nil
+}
+
+func decodeAnnotation(raw json.RawMessage) (any, error) {
+	decoder := json.NewDecoder(bytes.NewReader(raw))
+	decoder.UseNumber()
+	var value any
+	if err := decoder.Decode(&value); err != nil {
+		return nil, fmt.Errorf("must contain valid JSON: %w", err)
+	}
+	var trailing any
+	if err := decoder.Decode(&trailing); err != io.EOF {
+		if err == nil {
+			return nil, fmt.Errorf("must contain exactly one JSON value")
+		}
+		return nil, fmt.Errorf("must contain exactly one JSON value: %w", err)
+	}
+	return value, nil
+}
+
+// ValidateChunkAnnotations requires annotations to already contain canonical
+// JSON. CanonicalizeChunkAnnotations can be used at producer boundaries.
+func ValidateChunkAnnotations(annotations ChunkAnnotations) error {
+	canonical, err := CanonicalizeChunkAnnotations(annotations)
+	if err != nil {
+		return err
+	}
+	for namespace, raw := range annotations {
+		if !bytes.Equal(raw, canonical[namespace]) {
+			return fmt.Errorf("chunk annotation %q must use canonical JSON", namespace)
+		}
+	}
+	return nil
+}
+
+// CloneChunkAnnotations returns a deep clone, including every raw JSON value.
+func CloneChunkAnnotations(annotations ChunkAnnotations) ChunkAnnotations {
+	if len(annotations) == 0 {
+		return nil
+	}
+	cloned := make(ChunkAnnotations, len(annotations))
+	for namespace, raw := range annotations {
+		cloned[namespace] = append(json.RawMessage(nil), raw...)
+	}
+	return cloned
+}
+
+// CloneChunkPlan returns a deep clone of a chunk plan.
+func CloneChunkPlan(plan ChunkPlan) ChunkPlan {
+	cloned := ChunkPlan{
+		SourceDigest: plan.SourceDigest,
+		Ranges:       make([]ChunkRange, len(plan.Ranges)),
+		Annotations:  CloneChunkAnnotations(plan.Annotations),
+	}
+	for i, chunkRange := range plan.Ranges {
+		cloned.Ranges[i] = ChunkRange{
+			StartUnitID: chunkRange.StartUnitID,
+			EndUnitID:   chunkRange.EndUnitID,
+			Annotations: CloneChunkAnnotations(chunkRange.Annotations),
+		}
+	}
+	return cloned
+}
+
+// CanonicalizeChunkPlan returns a deep clone with canonical annotation bytes.
+func CanonicalizeChunkPlan(plan ChunkPlan) (ChunkPlan, error) {
+	canonical := CloneChunkPlan(plan)
+	annotations, err := CanonicalizeChunkAnnotations(plan.Annotations)
+	if err != nil {
+		return ChunkPlan{}, fmt.Errorf("chunk plan annotations: %w", err)
+	}
+	canonical.Annotations = annotations
+	for i := range plan.Ranges {
+		annotations, err := CanonicalizeChunkAnnotations(plan.Ranges[i].Annotations)
+		if err != nil {
+			return ChunkPlan{}, fmt.Errorf("chunk plan range[%d] annotations: %w", i, err)
+		}
+		canonical.Ranges[i].Annotations = annotations
+	}
+	return canonical, nil
+}
+
+// ValidateChunkPlan validates a canonical plan against the current source.
+// Ranges may contain gaps or overlap, but their start positions must increase.
+func ValidateChunkPlan(doc *SourceDocument, plan ChunkPlan) error {
+	if err := ValidateDocument(doc); err != nil {
+		return fmt.Errorf("source document: %w", err)
+	}
+	if plan.SourceDigest != doc.Digest {
+		return fmt.Errorf("chunk plan source_digest %q does not match source document digest %q", plan.SourceDigest, doc.Digest)
+	}
+	if len(plan.Ranges) == 0 {
+		return fmt.Errorf("chunk plan ranges must not be empty")
+	}
+	if err := ValidateChunkAnnotations(plan.Annotations); err != nil {
+		return fmt.Errorf("chunk plan annotations: %w", err)
+	}
+	previousStart := -1
+	for i, chunkRange := range plan.Ranges {
+		start, ok := UnitIndex(doc, chunkRange.StartUnitID)
+		if !ok {
+			return fmt.Errorf("chunk plan range[%d] start_unit_id %d was not found", i, chunkRange.StartUnitID)
+		}
+		end, ok := UnitIndex(doc, chunkRange.EndUnitID)
+		if !ok {
+			return fmt.Errorf("chunk plan range[%d] end_unit_id %d was not found", i, chunkRange.EndUnitID)
+		}
+		if start > end {
+			return fmt.Errorf("chunk plan range[%d] start_unit_id %d appears after end_unit_id %d", i, chunkRange.StartUnitID, chunkRange.EndUnitID)
+		}
+		if start <= previousStart {
+			return fmt.Errorf("chunk plan range[%d] start_unit_id %d does not appear after the previous range start", i, chunkRange.StartUnitID)
+		}
+		if err := ValidateChunkAnnotations(chunkRange.Annotations); err != nil {
+			return fmt.Errorf("chunk plan range[%d] annotations: %w", i, err)
+		}
+		previousStart = start
+	}
+	return nil
+}
+
+// MaterializeChunkPlan deterministically expands a validated plan into chunks.
+func MaterializeChunkPlan(doc *SourceDocument, plan ChunkPlan) ([]Chunk, error) {
+	if err := ValidateChunkPlan(doc, plan); err != nil {
+		return nil, err
+	}
+	chunks := make([]Chunk, 0, len(plan.Ranges))
+	for index, chunkRange := range plan.Ranges {
+		start, _ := UnitIndex(doc, chunkRange.StartUnitID)
+		end, _ := UnitIndex(doc, chunkRange.EndUnitID)
+		units, err := cloneSourceUnits(doc.Units[start : end+1])
+		if err != nil {
+			return nil, fmt.Errorf("clone chunk plan range[%d] units: %w", index, err)
+		}
+		content, err := json.Marshal(struct {
+			Units []SourceUnit `json:"units"`
+		}{Units: units})
+		if err != nil {
+			return nil, fmt.Errorf("encode chunk plan range[%d]: %w", index, err)
+		}
+		chunks = append(chunks, Chunk{
+			ID:        fmt.Sprintf("chunk-%06d", index+1),
+			SourceID:  doc.ID,
+			Index:     index,
+			Ref:       SourceRef{SourceID: doc.ID, StartUnitID: chunkRange.StartUnitID, EndUnitID: chunkRange.EndUnitID},
+			Content:   content,
+			MediaType: "application/json",
+			Units:     units,
+			Metadata: map[string]any{
+				"start_unit_id": chunkRange.StartUnitID,
+				"end_unit_id":   chunkRange.EndUnitID,
+				"unit_count":    len(units),
+			},
+			Annotations:     CloneChunkAnnotations(chunkRange.Annotations),
+			PlanAnnotations: CloneChunkAnnotations(plan.Annotations),
+		})
+	}
+	return chunks, nil
+}
+
+func cloneSourceUnits(units []SourceUnit) ([]SourceUnit, error) {
+	if len(units) == 0 {
+		return nil, nil
+	}
+	cloned := make([]SourceUnit, len(units))
+	for i, unit := range units {
+		cloned[i] = unit
+		metadata, err := CloneMetadata(unit.Metadata)
+		if err != nil {
+			return nil, fmt.Errorf("source unit[%d] metadata: %w", i, err)
+		}
+		cloned[i].Metadata = metadata
+	}
+	return cloned, nil
+}

internal/core/source/chunk_plan_test.go

@@ -0,0 +1,332 @@
+package source
+
+import (
+	"bytes"
+	"encoding/json"
+	"math"
+	"reflect"
+	"strings"
+	"testing"
+)
+
+type typedMetadataMap map[string]any
+type typedMetadataSlice []typedMetadataMap
+type typedMetadataArray [2]any
+
+func TestCanonicalizeChunkAnnotations(t *testing.T) {
+	original := ChunkAnnotations{
+		"domain/items": json.RawMessage(` { "z": [3, 2, 1], "a": 1.0 } `),
+	}
+	canonical, err := CanonicalizeChunkAnnotations(original)
+	if err != nil {
+		t.Fatalf("CanonicalizeChunkAnnotations() error = %v, want nil", err)
+	}
+	if got, want := string(canonical["domain/items"]), `{"a":1.0,"z":[3,2,1]}`; got != want {
+		t.Fatalf("canonical annotation = %q, want %q", got, want)
+	}
+	original["domain/items"][0] = '['
+	if got := string(canonical["domain/items"]); got != `{"a":1.0,"z":[3,2,1]}` {
+		t.Fatalf("canonical annotation changed after input mutation: %q", got)
+	}
+	canonical["domain/items"][0] = '['
+	if original["domain/items"][0] == '[' && bytes.Equal(original["domain/items"], canonical["domain/items"]) {
+		t.Fatal("input and canonical annotation share value storage")
+	}
+}
+
+func TestCanonicalizeChunkAnnotationsRejectsInvalidValues(t *testing.T) {
+	tests := []struct {
+		name        string
+		annotations ChunkAnnotations
+		want        string
+	}{
+		{name: "blank namespace", annotations: ChunkAnnotations{" \t": json.RawMessage(`true`)}, want: "namespace must not be empty"},
+		{name: "untrimmed namespace", annotations: ChunkAnnotations{" items ": json.RawMessage(`true`)}, want: "leading or trailing whitespace"},
+		{name: "invalid JSON", annotations: ChunkAnnotations{"items": json.RawMessage(`{"x":`)}, want: "valid JSON"},
+		{name: "trailing JSON", annotations: ChunkAnnotations{"items": json.RawMessage(`true false`)}, want: "exactly one JSON value"},
+		{name: "non-finite number", annotations: ChunkAnnotations{"items": json.RawMessage(`NaN`)}, want: "valid JSON"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			_, err := CanonicalizeChunkAnnotations(tt.annotations)
+			if err == nil || !strings.Contains(err.Error(), tt.want) {
+				t.Fatalf("CanonicalizeChunkAnnotations() error = %v, want containing %q", err, tt.want)
+			}
+		})
+	}
+}
+
+func TestValidateChunkAnnotationsRequiresCanonicalJSON(t *testing.T) {
+	if err := ValidateChunkAnnotations(ChunkAnnotations{"items": json.RawMessage(` {"b":2,"a":1}`)}); err == nil || !strings.Contains(err.Error(), "canonical JSON") {
+		t.Fatalf("ValidateChunkAnnotations() error = %v, want canonical JSON error", err)
+	}
+	if err := ValidateChunkAnnotations(ChunkAnnotations{"items": json.RawMessage(`{"a":1,"b":2}`)}); err != nil {
+		t.Fatalf("ValidateChunkAnnotations(canonical) error = %v, want nil", err)
+	}
+}
+
+func TestCloneChunkPlanDoesNotShareAnnotationBytes(t *testing.T) {
+	plan := validChunkPlan(planDocument())
+	cloned := CloneChunkPlan(plan)
+	cloned.Annotations["plan"][0] = '['
+	cloned.Ranges[0].Annotations["range"][0] = '['
+	if string(plan.Annotations["plan"]) != `{"value":1}` || string(plan.Ranges[0].Annotations["range"]) != `{"value":2}` {
+		t.Fatal("CloneChunkPlan() shares annotation value storage")
+	}
+}
+
+func TestValidateChunkPlanRanges(t *testing.T) {
+	doc := planDocument()
+	tests := []struct {
+		name   string
+		mutate func(*ChunkPlan)
+		want   string
+	}{
+		{name: "source mismatch", mutate: func(plan *ChunkPlan) { plan.SourceDigest = "sha256:other" }, want: "does not match"},
+		{name: "missing ranges", mutate: func(plan *ChunkPlan) { plan.Ranges = nil }, want: "ranges must not be empty"},
+		{name: "missing start", mutate: func(plan *ChunkPlan) { plan.Ranges[0].StartUnitID = 99 }, want: "start_unit_id 99 was not found"},
+		{name: "missing end", mutate: func(plan *ChunkPlan) { plan.Ranges[0].EndUnitID = 99 }, want: "end_unit_id 99 was not found"},
+		{name: "backward range", mutate: func(plan *ChunkPlan) { plan.Ranges[0] = ChunkRange{StartUnitID: 30, EndUnitID: 10} }, want: "appears after end_unit_id"},
+		{name: "duplicate start", mutate: func(plan *ChunkPlan) { plan.Ranges[1].StartUnitID = plan.Ranges[0].StartUnitID }, want: "does not appear after"},
+		{name: "backward starts", mutate: func(plan *ChunkPlan) {
+			plan.Ranges = []ChunkRange{{StartUnitID: 30, EndUnitID: 50}, {StartUnitID: 20, EndUnitID: 40}}
+		}, want: "does not appear after"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			plan := validChunkPlan(doc)
+			tt.mutate(&plan)
+			err := ValidateChunkPlan(doc, plan)
+			if err == nil || !strings.Contains(err.Error(), tt.want) {
+				t.Fatalf("ValidateChunkPlan() error = %v, want containing %q", err, tt.want)
+			}
+		})
+	}
+
+	for name, ranges := range map[string][]ChunkRange{
+		"gap":     {{StartUnitID: 10, EndUnitID: 20}, {StartUnitID: 40, EndUnitID: 50}},
+		"overlap": {{StartUnitID: 10, EndUnitID: 30}, {StartUnitID: 20, EndUnitID: 50}},
+	} {
+		t.Run(name, func(t *testing.T) {
+			plan := validChunkPlan(doc)
+			plan.Ranges = ranges
+			if err := ValidateChunkPlan(doc, plan); err != nil {
+				t.Fatalf("ValidateChunkPlan() error = %v, want nil", err)
+			}
+		})
+	}
+}
+
+func TestDigestChunkPlanIsStableAndCoversLogicalPlan(t *testing.T) {
+	doc := planDocument()
+	plan := validChunkPlan(doc)
+	first, err := DigestChunkPlan(plan)
+	if err != nil {
+		t.Fatalf("DigestChunkPlan() error = %v, want nil", err)
+	}
+	reformatted := CloneChunkPlan(plan)
+	reformatted.Annotations["plan"] = json.RawMessage(` { "value" : 1 } `)
+	second, err := DigestChunkPlan(reformatted)
+	if err != nil {
+		t.Fatalf("DigestChunkPlan(reformatted) error = %v, want nil", err)
+	}
+	if first != second {
+		t.Fatalf("digests = %q and %q, want stable canonical annotation digest", first, second)
+	}
+
+	changes := []func(*ChunkPlan){
+		func(value *ChunkPlan) { value.Ranges[0].EndUnitID = 30 },
+		func(value *ChunkPlan) { value.Annotations["plan"] = json.RawMessage(`{"value":2}`) },
+		func(value *ChunkPlan) { value.Ranges[0].Annotations["range"] = json.RawMessage(`{"value":3}`) },
+	}
+	for i, change := range changes {
+		changed := CloneChunkPlan(plan)
+		change(&changed)
+		digest, err := DigestChunkPlan(changed)
+		if err != nil {
+			t.Fatalf("DigestChunkPlan(change %d) error = %v", i, err)
+		}
+		if digest == first {
+			t.Fatalf("DigestChunkPlan(change %d) = %q, want changed digest", i, digest)
+		}
+	}
+}
+
+func TestMaterializeChunkPlanExactOutputAndMutationSafety(t *testing.T) {
+	doc := planDocument()
+	plan := validChunkPlan(doc)
+	plan.Ranges = []ChunkRange{
+		{StartUnitID: 10, EndUnitID: 30, Annotations: ChunkAnnotations{"range": json.RawMessage(`{"value":2}`)}},
+		{StartUnitID: 20, EndUnitID: 50, Annotations: ChunkAnnotations{"range": json.RawMessage(`{"value":3}`)}},
+	}
+	chunks, err := MaterializeChunkPlan(doc, plan)
+	if err != nil {
+		t.Fatalf("MaterializeChunkPlan() error = %v, want nil", err)
+	}
+	if len(chunks) != 2 {
+		t.Fatalf("chunks = %d, want 2", len(chunks))
+	}
+	first := chunks[0]
+	if first.ID != "chunk-000001" || first.SourceID != doc.ID || first.Index != 0 || first.MediaType != "application/json" {
+		t.Fatalf("first chunk identity = %#v", first)
+	}
+	if want := (SourceRef{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 30}); first.Ref != want {
+		t.Fatalf("first ref = %#v, want %#v", first.Ref, want)
+	}
+	if got, want := unitIDs(first.Units), []int{10, 20, 30}; !reflect.DeepEqual(got, want) {
+		t.Fatalf("first unit ids = %#v, want %#v", got, want)
+	}
+	wantContent, _ := json.Marshal(struct {
+		Units []SourceUnit `json:"units"`
+	}{Units: doc.Units[:3]})
+	if !bytes.Equal(first.Content, wantContent) {
+		t.Fatalf("first content = %s, want %s", first.Content, wantContent)
+	}
+	if !reflect.DeepEqual(first.Metadata, map[string]any{"start_unit_id": 10, "end_unit_id": 30, "unit_count": 3}) {
+		t.Fatalf("first metadata = %#v", first.Metadata)
+	}
+	if string(first.Annotations["range"]) != `{"value":2}` || string(first.PlanAnnotations["plan"]) != `{"value":1}` {
+		t.Fatalf("first annotations = %#v / %#v", first.Annotations, first.PlanAnnotations)
+	}
+	if got, want := unitIDs(chunks[1].Units), []int{20, 30, 40, 50}; !reflect.DeepEqual(got, want) {
+		t.Fatalf("overlapping unit ids = %#v, want %#v", got, want)
+	}
+
+	again, err := MaterializeChunkPlan(doc, plan)
+	if err != nil {
+		t.Fatalf("MaterializeChunkPlan(repeated) error = %v", err)
+	}
+	if !reflect.DeepEqual(chunks, again) {
+		t.Fatalf("repeated materialization differs:\nfirst: %#v\nagain: %#v", chunks, again)
+	}
+	chunks[0].Units[0].Text = "mutated"
+	chunks[0].Annotations["range"][0] = '['
+	chunks[0].PlanAnnotations["plan"][0] = '['
+	if doc.Units[0].Text == "mutated" || string(plan.Ranges[0].Annotations["range"]) != `{"value":2}` || string(plan.Annotations["plan"]) != `{"value":1}` {
+		t.Fatal("materialized chunk shares owned plan or source storage")
+	}
+}
+
+func TestMaterializeChunkPlanDeepClonesUnitMetadata(t *testing.T) {
+	doc := planDocument()
+	doc.Units[0].Metadata = map[string]any{"nested": map[string]any{"values": []any{json.RawMessage(`{"ok":true}`)}}}
+	chunks, err := MaterializeChunkPlan(doc, validChunkPlan(doc))
+	if err != nil {
+		t.Fatal(err)
+	}
+	nested := chunks[0].Units[0].Metadata["nested"].(map[string]any)
+	nested["values"].([]any)[0].(json.RawMessage)[0] = '['
+	nested["changed"] = true
+	original := doc.Units[0].Metadata["nested"].(map[string]any)
+	if _, exists := original["changed"]; exists || string(original["values"].([]any)[0].(json.RawMessage)) != `{"ok":true}` {
+		t.Fatalf("source metadata changed through materialized chunk: %#v", doc.Units[0].Metadata)
+	}
+}
+
+func TestMaterializeChunkPlanClonesConcreteJSONMetadata(t *testing.T) {
+	doc := planDocument()
+	doc.Units[0].Metadata = map[string]any{
+		"typed_map":   typedMetadataMap{"bytes": []byte("map")},
+		"typed_slice": typedMetadataSlice{{"raw": json.RawMessage(`{"slice":true}`)}},
+		"typed_array": typedMetadataArray{map[string]any{"bytes": []byte("array")}, []any{json.RawMessage(`{"array":true}`)}},
+		"interface":   any(typedMetadataMap{"bytes": []byte("interface")}),
+		"raw":         json.RawMessage(`{"raw":true}`),
+		"bytes":       []byte("bytes"),
+	}
+
+	chunks, err := MaterializeChunkPlan(doc, validChunkPlan(doc))
+	if err != nil {
+		t.Fatal(err)
+	}
+	again, err := MaterializeChunkPlan(doc, validChunkPlan(doc))
+	if err != nil {
+		t.Fatal(err)
+	}
+	metadata := chunks[0].Units[0].Metadata
+	metadata["typed_map"].(typedMetadataMap)["bytes"].([]byte)[0] = 'M'
+	metadata["typed_slice"].(typedMetadataSlice)[0]["raw"].(json.RawMessage)[0] = '['
+	metadata["typed_array"].(typedMetadataArray)[0].(map[string]any)["bytes"].([]byte)[0] = 'A'
+	metadata["typed_array"].(typedMetadataArray)[1].([]any)[0].(json.RawMessage)[0] = '['
+	metadata["interface"].(typedMetadataMap)["bytes"].([]byte)[0] = 'I'
+	metadata["raw"].(json.RawMessage)[0] = '['
+	metadata["bytes"].([]byte)[0] = 'B'
+
+	for name, candidate := range map[string]map[string]any{
+		"source": doc.Units[0].Metadata,
+		"again":  again[0].Units[0].Metadata,
+	} {
+		if got := string(candidate["typed_map"].(typedMetadataMap)["bytes"].([]byte)); got != "map" {
+			t.Fatalf("%s typed map bytes = %q, want map", name, got)
+		}
+		if got := string(candidate["typed_slice"].(typedMetadataSlice)[0]["raw"].(json.RawMessage)); got != `{"slice":true}` {
+			t.Fatalf("%s typed slice raw = %q", name, got)
+		}
+		array := candidate["typed_array"].(typedMetadataArray)
+		if got := string(array[0].(map[string]any)["bytes"].([]byte)); got != "array" || string(array[1].([]any)[0].(json.RawMessage)) != `{"array":true}` {
+			t.Fatalf("%s typed array = %#v", name, array)
+		}
+		if got := string(candidate["interface"].(typedMetadataMap)["bytes"].([]byte)); got != "interface" {
+			t.Fatalf("%s interface bytes = %q, want interface", name, got)
+		}
+		if got := string(candidate["raw"].(json.RawMessage)); got != `{"raw":true}` {
+			t.Fatalf("%s raw = %q", name, got)
+		}
+		if got := string(candidate["bytes"].([]byte)); got != "bytes" {
+			t.Fatalf("%s bytes = %q, want bytes", name, got)
+		}
+	}
+}
+
+func TestMaterializeChunkPlanRejectsInvalidMetadata(t *testing.T) {
+	cyclic := make(map[string]any)
+	cyclic["self"] = cyclic
+	for _, tc := range []struct {
+		name  string
+		value any
+		want  string
+	}{
+		{name: "cycle", value: cyclic, want: "metadata.cycle.self contains a cycle"},
+		{name: "unsupported", value: func() {}, want: "metadata.unsupported has unsupported type func()"},
+		{name: "nonfinite", value: math.NaN(), want: "metadata.nonfinite has a non-finite number"},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			doc := planDocument()
+			doc.Units[0].Metadata = map[string]any{tc.name: tc.value}
+			_, first := MaterializeChunkPlan(doc, validChunkPlan(doc))
+			_, second := MaterializeChunkPlan(doc, validChunkPlan(doc))
+			if first == nil || !strings.Contains(first.Error(), "clone chunk plan range[0] units: source unit[0] metadata: "+tc.want) {
+				t.Fatalf("first MaterializeChunkPlan() error = %v, want %q", first, tc.want)
+			}
+			if second == nil || second.Error() != first.Error() {
+				t.Fatalf("MaterializeChunkPlan() errors = %v and %v, want deterministic error", first, second)
+			}
+		})
+	}
+}
+
+func planDocument() *SourceDocument {
+	doc := &SourceDocument{ID: "source-plan", Kind: "test", Format: "application/test", Digest: "sha256:source-plan"}
+	for _, id := range []int{10, 20, 30, 40, 50} {
+		doc.Units = append(doc.Units, SourceUnit{ID: id, Kind: "line", Text: "unit", Ref: SourceRef{SourceID: doc.ID, StartUnitID: id, EndUnitID: id}})
+	}
+	return doc
+}
+
+func validChunkPlan(doc *SourceDocument) ChunkPlan {
+	return ChunkPlan{
+		SourceDigest: doc.Digest,
+		Ranges: []ChunkRange{
+			{StartUnitID: 10, EndUnitID: 20, Annotations: ChunkAnnotations{"range": json.RawMessage(`{"value":2}`)}},
+			{StartUnitID: 30, EndUnitID: 50},
+		},
+		Annotations: ChunkAnnotations{"plan": json.RawMessage(`{"value":1}`)},
+	}
+}
+
+func unitIDs(units []SourceUnit) []int {
+	ids := make([]int, len(units))
+	for i, unit := range units {
+		ids[i] = unit.ID
+	}
+	return ids
+}

internal/core/source/digest.go

@@ -0,0 +1,102 @@
+package source
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+)
+
+// DigestDocument returns a deterministic digest of the canonical source
+// document content. The existing Digest field is excluded from its own digest.
+func DigestDocument(doc *SourceDocument) (string, error) {
+	if doc == nil {
+		return "", fmt.Errorf("source document must not be nil")
+	}
+	payload := struct {
+		ID       string         `json:"id"`
+		Kind     string         `json:"kind"`
+		Format   string         `json:"format"`
+		Units    []SourceUnit   `json:"units"`
+		Metadata map[string]any `json:"metadata,omitempty"`
+	}{
+		ID:       doc.ID,
+		Kind:     doc.Kind,
+		Format:   doc.Format,
+		Units:    doc.Units,
+		Metadata: doc.Metadata,
+	}
+	encoded, err := json.Marshal(payload)
+	if err != nil {
+		return "", fmt.Errorf("encode source document for digest: %w", err)
+	}
+	sum := sha256.Sum256(encoded)
+	return "sha256:" + hex.EncodeToString(sum[:]), nil
+}
+
+// DigestChunk returns a deterministic digest of a chunk, including its source
+// provenance, content, units, and metadata.
+func DigestChunk(chunk Chunk) (string, error) {
+	annotations, err := CanonicalizeChunkAnnotations(chunk.Annotations)
+	if err != nil {
+		return "", fmt.Errorf("canonicalize source chunk annotations: %w", err)
+	}
+	planAnnotations, err := CanonicalizeChunkAnnotations(chunk.PlanAnnotations)
+	if err != nil {
+		return "", fmt.Errorf("canonicalize source chunk plan annotations: %w", err)
+	}
+	payload := struct {
+		ID              string           `json:"id"`
+		SourceID        string           `json:"source_id"`
+		Index           int              `json:"index"`
+		Ref             SourceRef        `json:"ref"`
+		Content         []byte           `json:"content"`
+		MediaType       string           `json:"media_type"`
+		Units           []SourceUnit     `json:"units"`
+		Metadata        map[string]any   `json:"metadata,omitempty"`
+		Annotations     ChunkAnnotations `json:"annotations,omitempty"`
+		PlanAnnotations ChunkAnnotations `json:"plan_annotations,omitempty"`
+	}{
+		ID:              chunk.ID,
+		SourceID:        chunk.SourceID,
+		Index:           chunk.Index,
+		Ref:             chunk.Ref,
+		Content:         chunk.Content,
+		MediaType:       chunk.MediaType,
+		Units:           chunk.Units,
+		Metadata:        chunk.Metadata,
+		Annotations:     annotations,
+		PlanAnnotations: planAnnotations,
+	}
+	encoded, err := json.Marshal(payload)
+	if err != nil {
+		return "", fmt.Errorf("encode source chunk for digest: %w", err)
+	}
+	sum := sha256.Sum256(encoded)
+	return "sha256:" + hex.EncodeToString(sum[:]), nil
+}
+
+// DigestChunkPlan returns a deterministic digest of the logical plan. Storage
+// schema, producer provenance, warnings, and timestamps are intentionally not
+// part of the digest.
+func DigestChunkPlan(plan ChunkPlan) (string, error) {
+	canonical, err := CanonicalizeChunkPlan(plan)
+	if err != nil {
+		return "", err
+	}
+	if isBlank(canonical.SourceDigest) {
+		return "", fmt.Errorf("chunk plan source_digest must not be empty")
+	}
+	if hasSurroundingWhitespace(canonical.SourceDigest) {
+		return "", fmt.Errorf("chunk plan source_digest must not contain leading or trailing whitespace")
+	}
+	if len(canonical.Ranges) == 0 {
+		return "", fmt.Errorf("chunk plan ranges must not be empty")
+	}
+	encoded, err := json.Marshal(canonical)
+	if err != nil {
+		return "", fmt.Errorf("encode chunk plan for digest: %w", err)
+	}
+	sum := sha256.Sum256(encoded)
+	return "sha256:" + hex.EncodeToString(sum[:]), nil
+}

internal/core/source/metadata.go

@@ -0,0 +1,119 @@
+package source
+
+import (
+	"fmt"
+	"math"
+	"reflect"
+	"sort"
+)
+
+// CloneMetadata returns an independently owned copy of JSON-shaped metadata.
+// It preserves concrete map, slice, and array types while rejecting values that
+// cannot be safely represented as JSON-shaped metadata.
+func CloneMetadata(metadata map[string]any) (map[string]any, error) {
+	if len(metadata) == 0 {
+		return nil, nil
+	}
+	cloned, err := cloneMetadataValue(reflect.ValueOf(metadata), "metadata", make(map[metadataVisit]struct{}))
+	if err != nil {
+		return nil, err
+	}
+	return cloned.Interface().(map[string]any), nil
+}
+
+type metadataVisit struct {
+	typ reflect.Type
+	ptr uintptr
+}
+
+func cloneMetadataValue(value reflect.Value, location string, active map[metadataVisit]struct{}) (reflect.Value, error) {
+	if !value.IsValid() {
+		return value, nil
+	}
+
+	switch value.Kind() {
+	case reflect.Interface:
+		if value.IsNil() {
+			return reflect.Zero(value.Type()), nil
+		}
+		cloned, err := cloneMetadataValue(value.Elem(), location, active)
+		if err != nil {
+			return reflect.Value{}, err
+		}
+		result := reflect.New(value.Type()).Elem()
+		result.Set(cloned)
+		return result, nil
+	case reflect.Map:
+		if value.IsNil() {
+			return reflect.Zero(value.Type()), nil
+		}
+		if value.Type().Key().Kind() != reflect.String {
+			return reflect.Value{}, fmt.Errorf("%s has unsupported map key type %s", location, value.Type().Key())
+		}
+		leave, err := enterMetadataValue(value, active, location)
+		if err != nil {
+			return reflect.Value{}, err
+		}
+		defer leave()
+
+		keys := value.MapKeys()
+		sort.Slice(keys, func(i, j int) bool { return keys[i].String() < keys[j].String() })
+		result := reflect.MakeMapWithSize(value.Type(), value.Len())
+		for _, key := range keys {
+			cloned, err := cloneMetadataValue(value.MapIndex(key), location+"."+key.String(), active)
+			if err != nil {
+				return reflect.Value{}, err
+			}
+			result.SetMapIndex(key, cloned)
+		}
+		return result, nil
+	case reflect.Slice:
+		if value.IsNil() {
+			return reflect.Zero(value.Type()), nil
+		}
+		leave, err := enterMetadataValue(value, active, location)
+		if err != nil {
+			return reflect.Value{}, err
+		}
+		defer leave()
+
+		result := reflect.MakeSlice(value.Type(), value.Len(), value.Len())
+		for i := 0; i < value.Len(); i++ {
+			cloned, err := cloneMetadataValue(value.Index(i), fmt.Sprintf("%s[%d]", location, i), active)
+			if err != nil {
+				return reflect.Value{}, err
+			}
+			result.Index(i).Set(cloned)
+		}
+		return result, nil
+	case reflect.Array:
+		result := reflect.New(value.Type()).Elem()
+		for i := 0; i < value.Len(); i++ {
+			cloned, err := cloneMetadataValue(value.Index(i), fmt.Sprintf("%s[%d]", location, i), active)
+			if err != nil {
+				return reflect.Value{}, err
+			}
+			result.Index(i).Set(cloned)
+		}
+		return result, nil
+	case reflect.Float32, reflect.Float64:
+		if math.IsNaN(value.Float()) || math.IsInf(value.Float(), 0) {
+			return reflect.Value{}, fmt.Errorf("%s has a non-finite number", location)
+		}
+		return value, nil
+	case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
+		reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.String:
+		return value, nil
+	default:
+		return reflect.Value{}, fmt.Errorf("%s has unsupported type %s", location, value.Type())
+	}
+}
+
+func enterMetadataValue(value reflect.Value, active map[metadataVisit]struct{}, location string) (func(), error) {
+	visit := metadataVisit{typ: value.Type(), ptr: value.Pointer()}
+	if _, exists := active[visit]; exists {
+		return nil, fmt.Errorf("%s contains a cycle", location)
+	}
+	active[visit] = struct{}{}
+	return func() { delete(active, visit) }, nil
+}

internal/core/source/source.go

@@ -1,5 +1,7 @@
 package source
 
+import "encoding/json"
+
 type SourceDocument struct {
 	ID       string         `json:"id"`
 	Kind     string         `json:"kind"`
@@ -10,14 +12,42 @@ type SourceDocument struct {
 }
 
 type SourceUnit struct {
-	ID       string         `json:"id"`
+	ID       int            `json:"id"`
 	Kind     string         `json:"kind"`
 	Text     string         `json:"text"`
+	Ref      SourceRef      `json:"ref"`
 	Metadata map[string]any `json:"metadata,omitempty"`
 }
 
 type SourceRef struct {
 	SourceID    string `json:"source_id"`
-	StartUnitID string `json:"start_unit_id"`
+	StartUnitID int    `json:"start_unit_id"`
-	EndUnitID   string `json:"end_unit_id"`
+	EndUnitID   int    `json:"end_unit_id"`
+}
+
+type ChunkAnnotations map[string]json.RawMessage
+
+type ChunkPlan struct {
+	SourceDigest string           `json:"source_digest"`
+	Ranges       []ChunkRange     `json:"ranges"`
+	Annotations  ChunkAnnotations `json:"annotations,omitempty"`
+}
+
+type ChunkRange struct {
+	StartUnitID int              `json:"start_unit_id"`
+	EndUnitID   int              `json:"end_unit_id"`
+	Annotations ChunkAnnotations `json:"annotations,omitempty"`
+}
+
+type Chunk struct {
+	ID              string           `json:"id"`
+	SourceID        string           `json:"source_id"`
+	Index           int              `json:"index"`
+	Ref             SourceRef        `json:"ref"`
+	Content         []byte           `json:"-"`
+	MediaType       string           `json:"media_type"`
+	Units           []SourceUnit     `json:"units"`
+	Metadata        map[string]any   `json:"metadata,omitempty"`
+	Annotations     ChunkAnnotations `json:"annotations,omitempty"`
+	PlanAnnotations ChunkAnnotations `json:"plan_annotations,omitempty"`
 }

internal/core/source/source_test.go

@@ -1,6 +1,7 @@
 package source
 
 import (
+	"encoding/json"
 	"strings"
 	"testing"
 )
@@ -96,13 +97,8 @@ func TestValidateDocumentMissingUnitFields(t *testing.T) {
 	}{
 		{
 			name:    "id",
-			mutate:  func(doc *SourceDocument) { doc.Units[1].ID = "" },
+			mutate:  func(doc *SourceDocument) { doc.Units[1].ID = 0 },
-			wantErr: "source unit[1].id must not be empty",
+			wantErr: "source unit[1].id must be positive",
-		},
-		{
-			name:    "id surrounding whitespace",
-			mutate:  func(doc *SourceDocument) { doc.Units[1].ID = " u2 " },
-			wantErr: "source unit[1].id \" u2 \" must not contain leading or trailing whitespace",
 		},
 		{
 			name:    "kind",
@@ -135,24 +131,162 @@ func TestValidateDocumentMissingUnitFields(t *testing.T) {
 
 func TestValidateDocumentDuplicateUnitIDs(t *testing.T) {
 	doc := validDocument()
-	doc.Units[1].ID = "u1"
+	doc.Units[1].ID = 1
 
 	err := ValidateDocument(doc)
 
 	if err == nil {
 		t.Fatal("ValidateDocument() error = nil, want error")
 	}
-	if err.Error() != "source unit id \"u1\" is duplicated" {
+	if err.Error() != "source unit id 1 is duplicated" {
 		t.Fatalf("ValidateDocument() error = %q", err.Error())
 	}
 }
 
+func TestValidateDocumentUnitReferences(t *testing.T) {
+	tests := []struct {
+		name    string
+		mutate  func(*SourceDocument)
+		wantErr string
+	}{
+		{
+			name:    "missing",
+			mutate:  func(doc *SourceDocument) { doc.Units[0].Ref = SourceRef{} },
+			wantErr: "source unit[0].ref: source ref source_id must not be empty",
+		},
+		{
+			name:    "foreign source",
+			mutate:  func(doc *SourceDocument) { doc.Units[0].Ref.SourceID = "source-2" },
+			wantErr: "source unit[0].ref: source ref source_id \"source-2\" does not match document id \"source-1\"",
+		},
+		{
+			name: "non-self range",
+			mutate: func(doc *SourceDocument) {
+				doc.Units[0].Ref.StartUnitID = 2
+				doc.Units[0].Ref.EndUnitID = 2
+			},
+			wantErr: "source unit[0].ref must identify source unit id 1",
+		},
+		{
+			name: "reversed range",
+			mutate: func(doc *SourceDocument) {
+				doc.Units[0].Ref.StartUnitID = 2
+				doc.Units[0].Ref.EndUnitID = 1
+			},
+			wantErr: "source unit[0].ref: source ref start_unit_id 2 appears after end_unit_id 1",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			doc := validDocument()
+			tt.mutate(doc)
+
+			err := ValidateDocument(doc)
+			if err == nil {
+				t.Fatal("ValidateDocument() error = nil, want unit reference error")
+			}
+			if err.Error() != tt.wantErr {
+				t.Fatalf("ValidateDocument() error = %q, want %q", err.Error(), tt.wantErr)
+			}
+		})
+	}
+}
+
+func TestDigestDocumentIsDeterministicAndIncludesUnitReference(t *testing.T) {
+	doc := validDocument()
+	doc.Metadata = map[string]any{"second": "value", "first": true}
+	first, err := DigestDocument(doc)
+	if err != nil {
+		t.Fatalf("DigestDocument() error = %v, want nil", err)
+	}
+
+	reordered := validDocument()
+	reordered.Metadata = map[string]any{"first": true, "second": "value"}
+	second, err := DigestDocument(reordered)
+	if err != nil {
+		t.Fatalf("DigestDocument(reordered) error = %v, want nil", err)
+	}
+	if first != second {
+		t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
+	}
+
+	changed := validDocument()
+	changed.Metadata = map[string]any{"first": true, "second": "value"}
+	changed.Units[0].Ref.SourceID = "different-source"
+	changedDigest, err := DigestDocument(changed)
+	if err != nil {
+		t.Fatalf("DigestDocument(changed) error = %v, want nil", err)
+	}
+	if first == changedDigest {
+		t.Fatalf("digest = %q after reference change, want different digest", changedDigest)
+	}
+}
+
+func TestDigestChunkIsDeterministicAndIncludesReference(t *testing.T) {
+	doc := validDocument()
+	chunk := Chunk{
+		ID:        "chunk-1",
+		SourceID:  doc.ID,
+		Index:     0,
+		Ref:       SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
+		Content:   []byte("chunk content"),
+		MediaType: "text/plain",
+		Units:     doc.Units,
+		Metadata:  map[string]any{"second": "value", "first": true},
+	}
+	first, err := DigestChunk(chunk)
+	if err != nil {
+		t.Fatalf("DigestChunk() error = %v, want nil", err)
+	}
+
+	chunk.Metadata = map[string]any{"first": true, "second": "value"}
+	second, err := DigestChunk(chunk)
+	if err != nil {
+		t.Fatalf("DigestChunk(reordered metadata) error = %v, want nil", err)
+	}
+	if first != second {
+		t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
+	}
+
+	chunk.Ref.EndUnitID = 1
+	changed, err := DigestChunk(chunk)
+	if err != nil {
+		t.Fatalf("DigestChunk(changed ref) error = %v, want nil", err)
+	}
+	if first == changed {
+		t.Fatalf("digest = %q after reference change, want different digest", changed)
+	}
+}
+
+func TestDigestChunkIncludesAnnotationScopes(t *testing.T) {
+	doc := validDocument()
+	chunk := Chunk{
+		ID: "chunk-1", SourceID: doc.ID, Ref: SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
+		Content: []byte("content"), MediaType: "text/plain", Units: doc.Units,
+		Annotations:     ChunkAnnotations{"scope": json.RawMessage(`{"value":1}`)},
+		PlanAnnotations: ChunkAnnotations{"scope": json.RawMessage(`{"value":2}`)},
+	}
+	base, err := DigestChunk(chunk)
+	if err != nil {
+		t.Fatalf("DigestChunk() error = %v", err)
+	}
+	chunk.Annotations["scope"] = json.RawMessage(`{"value":3}`)
+	rangeChanged, _ := DigestChunk(chunk)
+	chunk.Annotations["scope"] = json.RawMessage(`{"value":1}`)
+	chunk.PlanAnnotations["scope"] = json.RawMessage(`{"value":3}`)
+	planChanged, _ := DigestChunk(chunk)
+	if base == rangeChanged || base == planChanged || rangeChanged == planChanged {
+		t.Fatalf("annotation scope digests did not change distinctly: %q %q %q", base, rangeChanged, planChanged)
+	}
+}
+
 func TestValidateRefValid(t *testing.T) {
 	doc := validDocument()
 	ref := SourceRef{
 		SourceID:    "source-1",
-		StartUnitID: "u1",
+		StartUnitID: 1,
-		EndUnitID:   "u2",
+		EndUnitID:   2,
 	}
 
 	if err := ValidateRef(doc, ref); err != nil {
@@ -164,8 +298,8 @@ func TestValidateRefSourceIDMismatch(t *testing.T) {
 	doc := validDocument()
 	ref := SourceRef{
 		SourceID:    "source-2",
-		StartUnitID: "u1",
+		StartUnitID: 1,
-		EndUnitID:   "u2",
+		EndUnitID:   2,
 	}
 
 	err := ValidateRef(doc, ref)
@@ -186,43 +320,33 @@ func TestValidateRefMissingUnitIDs(t *testing.T) {
 	}{
 		{
 			name:    "missing source id",
-			ref:     SourceRef{StartUnitID: "u1", EndUnitID: "u2"},
+			ref:     SourceRef{StartUnitID: 1, EndUnitID: 2},
 			wantErr: "source ref source_id must not be empty",
 		},
 		{
 			name:    "source id surrounding whitespace",
-			ref:     SourceRef{SourceID: " source-1 ", StartUnitID: "u1", EndUnitID: "u2"},
+			ref:     SourceRef{SourceID: " source-1 ", StartUnitID: 1, EndUnitID: 2},
 			wantErr: "source ref source_id \" source-1 \" must not contain leading or trailing whitespace",
 		},
 		{
 			name:    "missing start id",
-			ref:     SourceRef{SourceID: "source-1", EndUnitID: "u2"},
+			ref:     SourceRef{SourceID: "source-1", EndUnitID: 2},
-			wantErr: "source ref start_unit_id must not be empty",
+			wantErr: "source ref start_unit_id must be positive",
-		},
-		{
-			name:    "start id surrounding whitespace",
-			ref:     SourceRef{SourceID: "source-1", StartUnitID: " u1 ", EndUnitID: "u2"},
-			wantErr: "source ref start_unit_id \" u1 \" must not contain leading or trailing whitespace",
 		},
 		{
 			name:    "missing end id",
-			ref:     SourceRef{SourceID: "source-1", StartUnitID: "u1"},
+			ref:     SourceRef{SourceID: "source-1", StartUnitID: 1},
-			wantErr: "source ref end_unit_id must not be empty",
+			wantErr: "source ref end_unit_id must be positive",
-		},
-		{
-			name:    "end id surrounding whitespace",
-			ref:     SourceRef{SourceID: "source-1", StartUnitID: "u1", EndUnitID: " u2 "},
-			wantErr: "source ref end_unit_id \" u2 \" must not contain leading or trailing whitespace",
 		},
 		{
 			name:    "unknown start id",
-			ref:     SourceRef{SourceID: "source-1", StartUnitID: "u9", EndUnitID: "u2"},
+			ref:     SourceRef{SourceID: "source-1", StartUnitID: 9, EndUnitID: 2},
-			wantErr: "source ref start_unit_id \"u9\" was not found",
+			wantErr: "source ref start_unit_id 9 was not found",
 		},
 		{
 			name:    "unknown end id",
-			ref:     SourceRef{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u9"},
+			ref:     SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 9},
-			wantErr: "source ref end_unit_id \"u9\" was not found",
+			wantErr: "source ref end_unit_id 9 was not found",
 		},
 	}
 
@@ -244,8 +368,8 @@ func TestValidateRefReversedUnitOrder(t *testing.T) {
 	doc := validDocument()
 	ref := SourceRef{
 		SourceID:    "source-1",
-		StartUnitID: "u2",
+		StartUnitID: 2,
-		EndUnitID:   "u1",
+		EndUnitID:   1,
 	}
 
 	err := ValidateRef(doc, ref)
@@ -261,7 +385,7 @@ func TestValidateRefReversedUnitOrder(t *testing.T) {
 func TestUnitIndex(t *testing.T) {
 	doc := validDocument()
 
-	index, ok := UnitIndex(doc, "u2")
+	index, ok := UnitIndex(doc, 2)
 	if !ok {
 		t.Fatal("UnitIndex() ok = false, want true")
 	}
@@ -269,7 +393,7 @@ func TestUnitIndex(t *testing.T) {
 		t.Fatalf("UnitIndex() index = %d, want 1", index)
 	}
 
-	index, ok = UnitIndex(doc, "u9")
+	index, ok = UnitIndex(doc, 9)
 	if ok {
 		t.Fatal("UnitIndex() ok = true, want false")
 	}
@@ -286,14 +410,16 @@ func validDocument() *SourceDocument {
 		Digest: "sha256:abc123",
 		Units: []SourceUnit{
 			{
-				ID:   "u1",
+				ID:   1,
 				Kind: "paragraph",
 				Text: "First unit.",
+				Ref:  SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
 			},
 			{
-				ID:   "u2",
+				ID:   2,
 				Kind: "paragraph",
 				Text: "Second unit.",
+				Ref:  SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2},
 			},
 		},
 	}

internal/core/source/validation.go

@@ -28,13 +28,10 @@ func ValidateDocument(doc *SourceDocument) error {
 		return fmt.Errorf("source document units must not be empty")
 	}
 
-	seenUnitIDs := make(map[string]struct{}, len(doc.Units))
+	seenUnitIDs := make(map[int]struct{}, len(doc.Units))
 	for i, unit := range doc.Units {
-		if isBlank(unit.ID) {
+		if unit.ID <= 0 {
-			return fmt.Errorf("source unit[%d].id must not be empty", i)
+			return fmt.Errorf("source unit[%d].id must be positive", i)
-		}
-		if hasSurroundingWhitespace(unit.ID) {
-			return fmt.Errorf("source unit[%d].id %q must not contain leading or trailing whitespace", i, unit.ID)
 		}
 		if isBlank(unit.Kind) {
 			return fmt.Errorf("source unit[%d].kind must not be empty", i)
@@ -43,10 +40,18 @@ func ValidateDocument(doc *SourceDocument) error {
 			return fmt.Errorf("source unit[%d].text must not be empty", i)
 		}
 		if _, ok := seenUnitIDs[unit.ID]; ok {
-			return fmt.Errorf("source unit id %q is duplicated", unit.ID)
+			return fmt.Errorf("source unit id %d is duplicated", unit.ID)
 		}
 		seenUnitIDs[unit.ID] = struct{}{}
 	}
+	for i, unit := range doc.Units {
+		if err := ValidateRef(doc, unit.Ref); err != nil {
+			return fmt.Errorf("source unit[%d].ref: %w", i, err)
+		}
+		if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
+			return fmt.Errorf("source unit[%d].ref must identify source unit id %d", i, unit.ID)
+		}
+	}
 
 	return nil
 }
@@ -61,17 +66,11 @@ func ValidateRef(doc *SourceDocument, ref SourceRef) error {
 	if hasSurroundingWhitespace(ref.SourceID) {
 		return fmt.Errorf("source ref source_id %q must not contain leading or trailing whitespace", ref.SourceID)
 	}
-	if isBlank(ref.StartUnitID) {
+	if ref.StartUnitID <= 0 {
-		return fmt.Errorf("source ref start_unit_id must not be empty")
+		return fmt.Errorf("source ref start_unit_id must be positive")
 	}
-	if hasSurroundingWhitespace(ref.StartUnitID) {
+	if ref.EndUnitID <= 0 {
-		return fmt.Errorf("source ref start_unit_id %q must not contain leading or trailing whitespace", ref.StartUnitID)
+		return fmt.Errorf("source ref end_unit_id must be positive")
-	}
-	if isBlank(ref.EndUnitID) {
-		return fmt.Errorf("source ref end_unit_id must not be empty")
-	}
-	if hasSurroundingWhitespace(ref.EndUnitID) {
-		return fmt.Errorf("source ref end_unit_id %q must not contain leading or trailing whitespace", ref.EndUnitID)
 	}
 	if ref.SourceID != doc.ID {
 		return fmt.Errorf("source ref source_id %q does not match document id %q", ref.SourceID, doc.ID)
@@ -79,20 +78,20 @@ func ValidateRef(doc *SourceDocument, ref SourceRef) error {
 
 	startIndex, ok := UnitIndex(doc, ref.StartUnitID)
 	if !ok {
-		return fmt.Errorf("source ref start_unit_id %q was not found", ref.StartUnitID)
+		return fmt.Errorf("source ref start_unit_id %d was not found", ref.StartUnitID)
 	}
 	endIndex, ok := UnitIndex(doc, ref.EndUnitID)
 	if !ok {
-		return fmt.Errorf("source ref end_unit_id %q was not found", ref.EndUnitID)
+		return fmt.Errorf("source ref end_unit_id %d was not found", ref.EndUnitID)
 	}
 	if startIndex > endIndex {
-		return fmt.Errorf("source ref start_unit_id %q appears after end_unit_id %q", ref.StartUnitID, ref.EndUnitID)
+		return fmt.Errorf("source ref start_unit_id %d appears after end_unit_id %d", ref.StartUnitID, ref.EndUnitID)
 	}
 
 	return nil
 }
 
-func UnitIndex(doc *SourceDocument, unitID string) (int, bool) {
+func UnitIndex(doc *SourceDocument, unitID int) (int, bool) {
 	if doc == nil {
 		return 0, false
 	}

internal/framework/checkpoint/identity.go

@@ -0,0 +1,184 @@
+package checkpoint
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"path/filepath"
+	"sort"
+	"strings"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+const digestPrefixLength = 16
+
+type Fingerprint struct {
+	Name  string `json:"name"`
+	Value string `json:"value"`
+}
+type IdentityInput struct {
+	Pipeline                               pipeline.ResolvedPipeline
+	InputKey, RawInputDigest, SourceDigest string
+	SelectedLanes                          []string
+	RuntimeOverrides                       []Fingerprint
+	References                             []artifacts.ReferenceProvenance
+	ProvenanceFingerprints                 []Fingerprint
+}
+type Identity struct {
+	Digest                 string        `json:"digest"`
+	PipelineID             string        `json:"pipeline_id"`
+	PipelineDigest         string        `json:"pipeline_digest"`
+	InputKey               string        `json:"input_key"`
+	RawInputDigest         string        `json:"raw_input_digest,omitempty"`
+	SourceDigest           string        `json:"source_digest,omitempty"`
+	SelectedLanes          []string      `json:"selected_lanes,omitempty"`
+	RuntimeOverrides       []Fingerprint `json:"runtime_overrides,omitempty"`
+	ReferenceDigests       []Fingerprint `json:"reference_digests,omitempty"`
+	ProvenanceFingerprints []Fingerprint `json:"provenance_fingerprints,omitempty"`
+}
+
+func NewIdentity(input IdentityInput) (Identity, error) {
+	pipelineID, pipelineDigest, inputKey := strings.TrimSpace(input.Pipeline.ID), strings.TrimSpace(input.Pipeline.Digest), strings.TrimSpace(input.InputKey)
+	if pipelineID == "" {
+		return Identity{}, fmt.Errorf("checkpoint identity pipeline id must not be empty")
+	}
+	if pipelineDigest == "" {
+		return Identity{}, fmt.Errorf("checkpoint identity pipeline digest must not be empty")
+	}
+	if inputKey == "" {
+		inputKey = strings.TrimSpace(input.Pipeline.Input.Module)
+	}
+	if inputKey == "" {
+		return Identity{}, fmt.Errorf("checkpoint identity input key must not be empty")
+	}
+	if strings.TrimSpace(input.RawInputDigest) == "" && strings.TrimSpace(input.SourceDigest) == "" {
+		return Identity{}, fmt.Errorf("checkpoint identity raw input digest or source digest must be set")
+	}
+	v := Identity{PipelineID: pipelineID, PipelineDigest: pipelineDigest, InputKey: inputKey, RawInputDigest: strings.TrimSpace(input.RawInputDigest), SourceDigest: strings.TrimSpace(input.SourceDigest), SelectedLanes: normalizedLanes(input.SelectedLanes, input.Pipeline.ArtifactLanes), RuntimeOverrides: normalizeIdentityFingerprints(input.RuntimeOverrides), ReferenceDigests: referenceFingerprints(input.References), ProvenanceFingerprints: normalizeIdentityFingerprints(input.ProvenanceFingerprints)}
+	data, err := json.Marshal(Identity{PipelineID: v.PipelineID, PipelineDigest: v.PipelineDigest, InputKey: v.InputKey, RawInputDigest: v.RawInputDigest, SourceDigest: v.SourceDigest, SelectedLanes: v.SelectedLanes, RuntimeOverrides: v.RuntimeOverrides, ReferenceDigests: v.ReferenceDigests, ProvenanceFingerprints: v.ProvenanceFingerprints})
+	if err != nil {
+		return Identity{}, fmt.Errorf("marshal checkpoint identity: %w", err)
+	}
+	sum := sha256.Sum256(data)
+	v.Digest = "sha256:" + hex.EncodeToString(sum[:])
+	return v, nil
+}
+func (i Identity) RelativePath() (string, error) {
+	p, err := safeComponent(i.PipelineID)
+	if err != nil {
+		return "", fmt.Errorf("checkpoint identity pipeline id: %w", err)
+	}
+	k, err := safeComponent(i.InputKey)
+	if err != nil {
+		return "", fmt.Errorf("checkpoint identity input key: %w", err)
+	}
+	s := digestPrefix(i.SourceDigest)
+	if s == "" {
+		s = digestPrefix(i.RawInputDigest)
+	}
+	d := digestPrefix(i.PipelineDigest)
+	x := digestPrefix(i.Digest)
+	if s == "" || d == "" || x == "" {
+		return "", fmt.Errorf("checkpoint identity digest prefix must not be empty")
+	}
+	s, err = safeComponent(s)
+	if err != nil {
+		return "", fmt.Errorf("checkpoint identity source digest: %w", err)
+	}
+	d, err = safeComponent(d)
+	if err != nil {
+		return "", fmt.Errorf("checkpoint identity pipeline digest: %w", err)
+	}
+	x, err = safeComponent(x)
+	if err != nil {
+		return "", fmt.Errorf("checkpoint identity digest: %w", err)
+	}
+	return filepath.ToSlash(filepath.Join(p, k+"-"+s, d, x)), nil
+}
+func normalizedLanes(selected []string, resolved []pipeline.ResolvedArtifactLane) []string {
+	if len(selected) == 0 {
+		for _, lane := range resolved {
+			selected = append(selected, lane.ID)
+		}
+	}
+	return normalizeStrings(selected)
+}
+func normalizeIdentityFingerprints(values []Fingerprint) []Fingerprint {
+	by := map[string]string{}
+	for _, v := range values {
+		if n, x := strings.TrimSpace(v.Name), strings.TrimSpace(v.Value); n != "" && x != "" {
+			by[n] = x
+		}
+	}
+	names := make([]string, 0, len(by))
+	for n := range by {
+		names = append(names, n)
+	}
+	sort.Strings(names)
+	out := make([]Fingerprint, 0, len(names))
+	for _, n := range names {
+		out = append(out, Fingerprint{Name: n, Value: by[n]})
+	}
+	if len(out) == 0 {
+		return nil
+	}
+	return out
+}
+func referenceFingerprints(refs []artifacts.ReferenceProvenance) []Fingerprint {
+	var values []Fingerprint
+	for _, r := range refs {
+		if d := strings.TrimSpace(r.Digest); d != "" {
+			values = append(values, Fingerprint{Name: strings.Join([]string{strings.TrimSpace(r.Stage), strings.TrimSpace(r.LaneID), strings.TrimSpace(r.SlotName), strings.TrimSpace(r.OriginURI)}, ":"), Value: d})
+		}
+	}
+	return normalizeIdentityFingerprints(values)
+}
+func normalizeStrings(values []string) []string {
+	seen := map[string]struct{}{}
+	for _, v := range values {
+		if v = strings.TrimSpace(v); v != "" {
+			seen[v] = struct{}{}
+		}
+	}
+	out := make([]string, 0, len(seen))
+	for v := range seen {
+		out = append(out, v)
+	}
+	sort.Strings(out)
+	if len(out) == 0 {
+		return nil
+	}
+	return out
+}
+func digestPrefix(v string) string {
+	v = strings.TrimSpace(v)
+	if n := strings.Index(v, ":"); n >= 0 {
+		v = v[n+1:]
+	}
+	if len(v) > digestPrefixLength {
+		return v[:digestPrefixLength]
+	}
+	return v
+}
+func safeComponent(v string) (string, error) {
+	v = strings.TrimSpace(v)
+	if v == "" {
+		return "", fmt.Errorf("must not be empty")
+	}
+	var b strings.Builder
+	for _, r := range v {
+		if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_' || r == '.' {
+			b.WriteRune(r)
+		} else {
+			b.WriteString(fmt.Sprintf("~%x", r))
+		}
+	}
+	out := b.String()
+	if out == "." || out == ".." || strings.Contains(out, "..") || strings.ContainsAny(out, `/\\`) {
+		return "", fmt.Errorf("%q is not filesystem safe", v)
+	}
+	return out, nil
+}

internal/framework/checkpoint/loader.go

@@ -0,0 +1,252 @@
+package checkpoint
+
+import (
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"os"
+	"strings"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
+	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+type FilesystemLoader struct {
+	root           string
+	identityDigest string
+}
+
+func NewFilesystemLoader(root string, identity Identity) (pipeline.CheckpointLoader, error) {
+	root = strings.TrimSpace(root)
+	if root == "" {
+		return pipeline.NoopCheckpointLoader(), nil
+	}
+	relative, err := identity.RelativePath()
+	if err != nil {
+		return nil, err
+	}
+	target, err := fileio.SafePath(root, relative)
+	if err != nil {
+		return nil, err
+	}
+	return &FilesystemLoader{root: target, identityDigest: identity.Digest}, nil
+}
+
+func (l *FilesystemLoader) Enabled() bool {
+	return l != nil && strings.TrimSpace(l.root) != ""
+}
+
+func (l *FilesystemLoader) Source(moduleKey string) (pipeline.SourceCheckpoint, pipeline.CheckpointDecision) {
+	var manifest SourceManifest
+	if decision := l.readJSON("source/manifest.json", &manifest); !decision.Reused {
+		return pipeline.SourceCheckpoint{}, decision
+	}
+	if decision := l.validateManifest(manifest.StageManifest, StageSource, "", moduleKey, StatusSucceeded, nil); !decision.Reused {
+		return pipeline.SourceCheckpoint{}, decision
+	}
+	var payload sourceDocumentEnvelope
+	if decision := l.readJSON("source/source-document.json", &payload); !decision.Reused {
+		return pipeline.SourceCheckpoint{}, decision
+	}
+	doc := cloneSourceDocument(payload.Document)
+	if err := source.ValidateDocument(&doc); err != nil {
+		return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint document is invalid: %v", err)
+	}
+	if strings.TrimSpace(manifest.SourceID) != "" && manifest.SourceID != doc.ID {
+		return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint source id does not match payload")
+	}
+	if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), digestFingerprints("source_document", doc.Digest)) {
+		return pipeline.SourceCheckpoint{}, invalidDecision("source checkpoint output digest does not match payload")
+	}
+	return pipeline.SourceCheckpoint{Document: &doc}, reusedDecision()
+}
+
+func (l *FilesystemLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
+	var manifest ExtractLaneManifest
+	if d := l.readJSON(laneManifestPath("extract", laneID), &manifest); !d.Reused {
+		return pipeline.ExtractCheckpoint{}, d
+	}
+	if d := l.validateLaneManifest(manifest.StageManifest, StageExtract, laneID, moduleKey, dependencies, StatusSucceeded, StatusSucceededWithRejections); !d.Reused {
+		return pipeline.ExtractCheckpoint{}, d
+	}
+	var payload artifactExtractEnvelope
+	if d := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !d.Reused {
+		return pipeline.ExtractCheckpoint{}, d
+	}
+	outputs, err := artifactCheckpointOutputs(payload.Outputs)
+	if err != nil {
+		return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint payload is invalid: %v", err)
+	}
+	if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
+		return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint output digests do not match payload")
+	}
+	return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
+}
+
+func (l *FilesystemLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
+	var manifest MergeLaneManifest
+	if d := l.readJSON(laneManifestPath("merge", laneID), &manifest); !d.Reused {
+		return pipeline.MergeCheckpoint{}, d
+	}
+	if d := l.validateLaneManifest(manifest.StageManifest, StageMerge, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
+		return pipeline.MergeCheckpoint{}, d
+	}
+	var payload artifactSingleEnvelope
+	if d := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !d.Reused {
+		return pipeline.MergeCheckpoint{}, d
+	}
+	values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
+	if err != nil {
+		return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint payload is invalid: %v", err)
+	}
+	if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
+		return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint output digest does not match payload")
+	}
+	return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
+}
+
+func (l *FilesystemLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
+	var manifest NormalizeLaneManifest
+	if d := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !d.Reused {
+		return pipeline.NormalizeCheckpoint{}, d
+	}
+	if d := l.validateLaneManifest(manifest.StageManifest, StageNormalize, laneID, moduleKey, dependencies, StatusSucceeded); !d.Reused {
+		return pipeline.NormalizeCheckpoint{}, d
+	}
+	var payload artifactSingleEnvelope
+	if d := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !d.Reused {
+		return pipeline.NormalizeCheckpoint{}, d
+	}
+	values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
+	if err != nil {
+		return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint payload is invalid: %v", err)
+	}
+	if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
+		return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint output digest does not match payload")
+	}
+	return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
+}
+
+func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) {
+	if len(values) == 0 {
+		return nil, nil
+	}
+	out := make([]pipeline.CheckpointArtifact, 0, len(values))
+	for _, v := range values {
+		content, err := contentFromEnvelope(v.Content)
+		if err != nil {
+			return nil, err
+		}
+		if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
+			return nil, fmt.Errorf("artifact codec identity is incomplete")
+		}
+		out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
+	}
+	return out, nil
+}
+
+func (l *FilesystemLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
+	if !l.Enabled() {
+		return pipeline.CheckpointDecision{Reason: "checkpoint loading disabled"}
+	}
+	target, err := fileio.SafePath(l.root, name)
+	if err != nil {
+		return invalidDecision("checkpoint path is invalid: %v", err)
+	}
+	data, err := os.ReadFile(target)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return pipeline.CheckpointDecision{Reason: "checkpoint artifact is missing"}
+		}
+		return invalidDecision("read checkpoint artifact: %v", err)
+	}
+	if err := json.Unmarshal(data, out); err != nil {
+		return invalidDecision("decode checkpoint artifact: %v", err)
+	}
+	return reusedDecision()
+}
+
+func (l *FilesystemLoader) validateManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, status StageStatus, dependencies []pipeline.CheckpointFingerprint) pipeline.CheckpointDecision {
+	return l.validateLaneManifest(manifest, stage, laneID, moduleKey, dependencies, status)
+}
+
+func (l *FilesystemLoader) validateLaneManifest(manifest StageManifest, stage StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...StageStatus) pipeline.CheckpointDecision {
+	if manifest.WorkspaceSchemaVersion == WorkspaceSchemaVersionV1 {
+		return invalidDecision("checkpoint workspace schema version %q is incompatible with %q and must be recomputed", manifest.WorkspaceSchemaVersion, WorkspaceSchemaVersion)
+	}
+	if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
+		return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion)
+	}
+	if strings.TrimSpace(l.identityDigest) != "" && manifest.Metadata["checkpoint_identity_digest"] != l.identityDigest {
+		return invalidDecision("checkpoint identity digest does not match current invocation")
+	}
+	if manifest.Stage != stage {
+		return invalidDecision("checkpoint stage %q does not match %q", manifest.Stage, stage)
+	}
+	if strings.TrimSpace(laneID) != "" && manifest.LaneID != laneID {
+		return invalidDecision("checkpoint lane %q does not match %q", manifest.LaneID, laneID)
+	}
+	if strings.TrimSpace(moduleKey) != "" && manifest.ModuleKey != moduleKey {
+		return invalidDecision("checkpoint module %q does not match %q", manifest.ModuleKey, moduleKey)
+	}
+	statusOK := false
+	for _, status := range statuses {
+		if manifest.Status == status {
+			statusOK = true
+			break
+		}
+	}
+	if !statusOK {
+		return invalidDecision("checkpoint status %q cannot be reused", manifest.Status)
+	}
+	if !fingerprintsEqual(checkpointToPipelineFingerprints(manifest.DependencyFingerprints), dependencies) {
+		return invalidDecision("checkpoint dependency fingerprints do not match")
+	}
+	return reusedDecision()
+}
+
+func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
+	content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
+	if err != nil {
+		return nil, fmt.Errorf("decode content_base64: %w", err)
+	}
+	if digest := strings.TrimSpace(value.ContentDigest); digest != "" && digest != contentDigest(content) {
+		return nil, fmt.Errorf("content digest mismatch")
+	}
+	return content, nil
+}
+
+func checkpointToPipelineFingerprints(values []Fingerprint) []pipeline.CheckpointFingerprint {
+	if len(values) == 0 {
+		return nil
+	}
+	out := make([]pipeline.CheckpointFingerprint, 0, len(values))
+	for _, value := range values {
+		out = append(out, pipeline.CheckpointFingerprint{Name: value.Name, Value: value.Value})
+	}
+	return normalizeFingerprints(out)
+}
+
+func fingerprintsEqual(a []pipeline.CheckpointFingerprint, b []pipeline.CheckpointFingerprint) bool {
+	a = normalizeFingerprints(a)
+	b = normalizeFingerprints(b)
+	if len(a) != len(b) {
+		return false
+	}
+	for i := range a {
+		if a[i] != b[i] {
+			return false
+		}
+	}
+	return true
+}
+
+func reusedDecision() pipeline.CheckpointDecision {
+	return pipeline.CheckpointDecision{Reused: true, Reason: "checkpoint is valid"}
+}
+
+func invalidDecision(format string, args ...any) pipeline.CheckpointDecision {
+	return pipeline.CheckpointDecision{Reason: fmt.Sprintf(format, args...)}
+}

internal/framework/checkpoint/manifest.go

@@ -0,0 +1,72 @@
+package checkpoint
+
+import "time"
+
+const (
+	// These names and values are frozen checkpoint wire-compatibility
+	// identifiers. They intentionally retain the former terminology.
+	WorkspaceSchemaVersion   = "notarius.workspace.v2"
+	WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
+)
+
+type StageName string
+
+const (
+	StageSource    StageName = "source"
+	StageExtract   StageName = "extract"
+	StageMerge     StageName = "merge"
+	StageNormalize StageName = "normalize"
+)
+
+type StageStatus string
+
+const (
+	StatusPending                 StageStatus = "pending"
+	StatusRunning                 StageStatus = "running"
+	StatusSucceeded               StageStatus = "succeeded"
+	StatusSucceededWithRejections StageStatus = "succeeded_with_rejections"
+	StatusFailed                  StageStatus = "failed"
+	StatusInvalidated             StageStatus = "invalidated"
+)
+
+type StageManifest struct {
+	WorkspaceSchemaVersion string             `json:"workspace_schema_version"`
+	Stage                  StageName          `json:"stage"`
+	LaneID                 string             `json:"lane_id,omitempty"`
+	ModuleKey              string             `json:"module_key,omitempty"`
+	DependencyFingerprints []Fingerprint      `json:"dependency_fingerprints,omitempty"`
+	Status                 StageStatus        `json:"status"`
+	OutputDigests          []Fingerprint      `json:"output_digests,omitempty"`
+	ValidationStatus       string             `json:"validation_status,omitempty"`
+	Rejections             []RejectionSummary `json:"rejections,omitempty"`
+	StartedAt              *time.Time         `json:"started_at,omitempty"`
+	CompletedAt            *time.Time         `json:"completed_at,omitempty"`
+	Metadata               map[string]string  `json:"metadata,omitempty"`
+}
+type RejectionSummary struct {
+	ValidatorName string `json:"validator_name,omitempty"`
+	ReasonCode    string `json:"reason_code,omitempty"`
+	Message       string `json:"message,omitempty"`
+	Count         int    `json:"count,omitempty"`
+}
+type SourceManifest struct {
+	StageManifest
+	SourceID string `json:"source_id,omitempty"`
+}
+type ExtractLaneManifest struct {
+	StageManifest
+	ChunkCount  int `json:"chunk_count,omitempty"`
+	OutputCount int `json:"output_count,omitempty"`
+}
+type MergeLaneManifest struct {
+	StageManifest
+	InputCount int `json:"input_count,omitempty"`
+}
+type NormalizeLaneManifest struct {
+	StageManifest
+	InputCount int `json:"input_count,omitempty"`
+}
+
+func NewStageManifest(stage StageName, status StageStatus) StageManifest {
+	return StageManifest{WorkspaceSchemaVersion: WorkspaceSchemaVersion, Stage: stage, Status: status}
+}

internal/framework/checkpoint/recorder.go

@@ -0,0 +1,469 @@
+package checkpoint
+
+import (
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/hex"
+	"fmt"
+	"path"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
+	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+type FilesystemRecorder struct {
+	root           string
+	identityDigest string
+	now            func() time.Time
+}
+
+func NewFilesystemRecorder(root string, identity Identity) (pipeline.CheckpointRecorder, error) {
+	root = strings.TrimSpace(root)
+	if root == "" {
+		return pipeline.NoopCheckpointRecorder(), nil
+	}
+	relative, err := identity.RelativePath()
+	if err != nil {
+		return nil, err
+	}
+	target, err := fileio.SafePath(root, relative)
+	if err != nil {
+		return nil, err
+	}
+	return &FilesystemRecorder{root: target, identityDigest: identity.Digest, now: time.Now}, nil
+}
+
+func (r *FilesystemRecorder) SourceRunning(moduleKey string) error {
+	manifest := r.newStageManifest(StageSource, StatusRunning)
+	manifest.ModuleKey = moduleKey
+	manifest.StartedAt = timePtr(r.timestamp())
+	return r.writeManifest("source/manifest.json", SourceManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) SourceSucceeded(moduleKey string, doc *source.SourceDocument) error {
+	if doc == nil {
+		return fmt.Errorf("checkpoint source document must not be nil")
+	}
+	if err := r.writePayload("source/source-document.json", sourceDocumentEnvelope{Document: cloneSourceDocument(*doc)}); err != nil {
+		return err
+	}
+	manifest := r.newStageManifest(StageSource, StatusSucceeded)
+	manifest.ModuleKey = moduleKey
+	manifest.OutputDigests = checkpointFingerprints(digestFingerprints("source_document", doc.Digest))
+	manifest.CompletedAt = timePtr(r.timestamp())
+	return r.writeManifest("source/manifest.json", SourceManifest{
+		StageManifest: manifest,
+		SourceID:      doc.ID,
+	})
+}
+
+func (r *FilesystemRecorder) SourceFailed(moduleKey string, err error) error {
+	manifest := r.newStageManifest(StageSource, StatusFailed)
+	manifest.ModuleKey = moduleKey
+	manifest.CompletedAt = timePtr(r.timestamp())
+	manifest.Metadata = errorMetadata(err)
+	return r.writeManifest("source/manifest.json", SourceManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) ExtractRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
+	manifest := r.laneManifest(StageExtract, StatusRunning, laneID, moduleKey, dependencies)
+	manifest.StartedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
+	payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
+	if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil {
+		return err
+	}
+	manifest := r.laneManifest(StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies)
+	manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests(outputs))
+	manifest.ValidationStatus = validationStatusString(warnings, rejected)
+	manifest.Rejections = rejectionSummaries(rejected)
+	manifest.CompletedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
+}
+
+func (r *FilesystemRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
+	manifest := r.laneManifest(StageExtract, StatusFailed, laneID, moduleKey, dependencies)
+	manifest.CompletedAt = timePtr(r.timestamp())
+	manifest.Metadata = errorMetadata(err)
+	return r.writeManifest(laneManifestPath("extract", laneID), ExtractLaneManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) MergeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
+	manifest := r.laneManifest(StageMerge, StatusRunning, laneID, moduleKey, dependencies)
+	manifest.StartedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
+	if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
+		return err
+	}
+	manifest := r.laneManifest(StageMerge, StatusSucceeded, laneID, moduleKey, dependencies)
+	manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
+	manifest.ValidationStatus = validationStatusString(warnings, nil)
+	manifest.CompletedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
+}
+
+func (r *FilesystemRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
+	manifest := r.laneManifest(StageMerge, StatusSucceededWithRejections, laneID, moduleKey, dependencies)
+	manifest.ValidationStatus = "rejected"
+	manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected})
+	manifest.CompletedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
+}
+
+func (r *FilesystemRecorder) MergeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
+	manifest := r.laneManifest(StageMerge, StatusFailed, laneID, moduleKey, dependencies)
+	manifest.CompletedAt = timePtr(r.timestamp())
+	manifest.Metadata = errorMetadata(err)
+	return r.writeManifest(laneManifestPath("merge", laneID), MergeLaneManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) NormalizeRunning(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) error {
+	manifest := r.laneManifest(StageNormalize, StatusRunning, laneID, moduleKey, dependencies)
+	manifest.StartedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
+	if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
+		return err
+	}
+	manifest := r.laneManifest(StageNormalize, StatusSucceeded, laneID, moduleKey, dependencies)
+	manifest.OutputDigests = checkpointFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
+	manifest.ValidationStatus = validationStatusString(warnings, nil)
+	manifest.CompletedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
+}
+
+func (r *FilesystemRecorder) NormalizeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
+	manifest := r.laneManifest(StageNormalize, StatusSucceededWithRejections, laneID, moduleKey, dependencies)
+	manifest.ValidationStatus = "rejected"
+	manifest.Rejections = rejectionSummaries([]contracts.RejectedOutput{rejected})
+	manifest.CompletedAt = timePtr(r.timestamp())
+	return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
+}
+
+func (r *FilesystemRecorder) NormalizeFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
+	manifest := r.laneManifest(StageNormalize, StatusFailed, laneID, moduleKey, dependencies)
+	manifest.CompletedAt = timePtr(r.timestamp())
+	manifest.Metadata = errorMetadata(err)
+	return r.writeManifest(laneManifestPath("normalize", laneID), NormalizeLaneManifest{StageManifest: manifest})
+}
+
+func (r *FilesystemRecorder) writeManifest(name string, payload any) error {
+	return r.writeJSON(name, payload)
+}
+
+func (r *FilesystemRecorder) writePayload(name string, payload any) error {
+	return r.writeJSON(name, payload)
+}
+
+func (r *FilesystemRecorder) writeJSON(name string, payload any) error {
+	if r == nil || strings.TrimSpace(r.root) == "" {
+		return nil
+	}
+	return fileio.WriteJSON(r.root, name, payload, 0o700, 0o600)
+}
+
+func (r *FilesystemRecorder) timestamp() time.Time {
+	if r == nil || r.now == nil {
+		return time.Now().UTC()
+	}
+	return r.now().UTC()
+}
+
+func (r *FilesystemRecorder) newStageManifest(stage StageName, status StageStatus) StageManifest {
+	manifest := NewStageManifest(stage, status)
+	if strings.TrimSpace(r.identityDigest) != "" {
+		manifest.Metadata = map[string]string{"checkpoint_identity_digest": r.identityDigest}
+	}
+	return manifest
+}
+
+func (r *FilesystemRecorder) laneManifest(stage StageName, status StageStatus, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) StageManifest {
+	manifest := r.newStageManifest(stage, status)
+	manifest.LaneID = laneID
+	manifest.ModuleKey = moduleKey
+	manifest.DependencyFingerprints = checkpointFingerprints(dependencies)
+	return manifest
+}
+
+type sourceDocumentEnvelope struct {
+	Document source.SourceDocument `json:"document"`
+}
+
+type binaryEnvelope struct {
+	ContentBase64 string              `json:"content_base64,omitempty"`
+	ContentDigest string              `json:"content_digest,omitempty"`
+	MediaType     string              `json:"media_type,omitempty"`
+	Metadata      map[string]any      `json:"metadata,omitempty"`
+	Warnings      []contracts.Warning `json:"warnings,omitempty"`
+}
+
+type artifactCheckpointEnvelope struct {
+	LaneID       string                   `json:"lane_id"`
+	ModuleKey    string                   `json:"module_key"`
+	SourceID     string                   `json:"source_id,omitempty"`
+	ChunkID      string                   `json:"chunk_id,omitempty"`
+	ChunkIndex   int                      `json:"chunk_index,omitempty"`
+	ChunkRef     source.SourceRef         `json:"chunk_ref,omitempty"`
+	Kind         contracts.ArtifactKind   `json:"artifact_kind"`
+	Schema       contracts.ArtifactSchema `json:"schema"`
+	SchemaDigest string                   `json:"schema_digest"`
+	Content      binaryEnvelope           `json:"content"`
+}
+type artifactExtractEnvelope struct {
+	Outputs  []artifactCheckpointEnvelope `json:"outputs"`
+	Rejected []contracts.RejectedOutput   `json:"rejected,omitempty"`
+	Warnings []contracts.Warning          `json:"warnings,omitempty"`
+}
+type artifactSingleEnvelope struct {
+	Output   artifactCheckpointEnvelope `json:"output"`
+	Warnings []contracts.Warning        `json:"warnings,omitempty"`
+}
+
+func artifactCheckpointEnvelopeFromOutput(output pipeline.CheckpointArtifact) artifactCheckpointEnvelope {
+	schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
+	schema.JSONSchema = nil
+	return artifactCheckpointEnvelope{LaneID: output.LaneID, ModuleKey: output.ModuleKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: output.SchemaDigest, Content: binaryEnvelopeFromContent(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)}
+}
+func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifactCheckpointEnvelope {
+	if len(outputs) == 0 {
+		return nil
+	}
+	out := make([]artifactCheckpointEnvelope, 0, len(outputs))
+	for _, v := range outputs {
+		out = append(out, artifactCheckpointEnvelopeFromOutput(v))
+	}
+	return out
+}
+func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.CheckpointFingerprint {
+	values := make([]pipeline.CheckpointFingerprint, 0, len(outputs))
+	for i, v := range outputs {
+		values = append(values, pipeline.CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: contentDigest(v.Artifact.Content)})
+	}
+	return normalizeFingerprints(values)
+}
+
+func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) binaryEnvelope {
+	return binaryEnvelope{
+		ContentBase64: base64.StdEncoding.EncodeToString(content),
+		ContentDigest: contentDigest(content),
+		MediaType:     mediaType,
+		Metadata:      cloneMetadata(metadata),
+		Warnings:      cloneWarnings(warnings),
+	}
+}
+
+func cloneSourceDocument(doc source.SourceDocument) source.SourceDocument {
+	doc.Units = cloneSourceUnits(doc.Units)
+	doc.Metadata = cloneMetadata(doc.Metadata)
+	return doc
+}
+
+func cloneSourceUnits(units []source.SourceUnit) []source.SourceUnit {
+	if len(units) == 0 {
+		return nil
+	}
+	out := make([]source.SourceUnit, 0, len(units))
+	for _, unit := range units {
+		out = append(out, source.SourceUnit{
+			ID:       unit.ID,
+			Kind:     unit.Kind,
+			Text:     unit.Text,
+			Ref:      unit.Ref,
+			Metadata: cloneMetadata(unit.Metadata),
+		})
+	}
+	return out
+}
+
+func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
+	if len(warnings) == 0 {
+		return nil
+	}
+	return append([]contracts.Warning(nil), warnings...)
+}
+
+func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.RejectedOutput {
+	if len(rejected) == 0 {
+		return nil
+	}
+	return append([]contracts.RejectedOutput(nil), rejected...)
+}
+
+func cloneMetadata(metadata map[string]any) map[string]any {
+	if len(metadata) == 0 {
+		return nil
+	}
+	out := make(map[string]any, len(metadata))
+	for key, value := range metadata {
+		out[key] = value
+	}
+	return out
+}
+
+func digestFingerprints(name string, digest string) []pipeline.CheckpointFingerprint {
+	digest = strings.TrimSpace(digest)
+	if digest == "" {
+		return nil
+	}
+	return []pipeline.CheckpointFingerprint{{Name: name, Value: digest}}
+}
+
+func checkpointFingerprints(values []pipeline.CheckpointFingerprint) []Fingerprint {
+	normalized := normalizeFingerprints(values)
+	if len(normalized) == 0 {
+		return nil
+	}
+	out := make([]Fingerprint, 0, len(normalized))
+	for _, value := range normalized {
+		out = append(out, Fingerprint{Name: value.Name, Value: value.Value})
+	}
+	return out
+}
+
+func normalizeFingerprints(values []pipeline.CheckpointFingerprint) []pipeline.CheckpointFingerprint {
+	if len(values) == 0 {
+		return nil
+	}
+	byName := make(map[string]string, len(values))
+	for _, value := range values {
+		name := strings.TrimSpace(value.Name)
+		fingerprint := strings.TrimSpace(value.Value)
+		if name == "" || fingerprint == "" {
+			continue
+		}
+		byName[name] = fingerprint
+	}
+	if len(byName) == 0 {
+		return nil
+	}
+	names := make([]string, 0, len(byName))
+	for name := range byName {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	out := make([]pipeline.CheckpointFingerprint, 0, len(names))
+	for _, name := range names {
+		out = append(out, pipeline.CheckpointFingerprint{Name: name, Value: byName[name]})
+	}
+	return out
+}
+
+func rejectionSummaries(rejected []contracts.RejectedOutput) []RejectionSummary {
+	if len(rejected) == 0 {
+		return nil
+	}
+	type key struct {
+		validatorName string
+		reasonCode    string
+		message       string
+	}
+	counts := make(map[key]int, len(rejected))
+	for _, item := range rejected {
+		k := key{validatorName: item.ValidatorName, reasonCode: item.ReasonCode, message: item.Message}
+		counts[k]++
+	}
+	keys := make([]key, 0, len(counts))
+	for k := range counts {
+		keys = append(keys, k)
+	}
+	sort.Slice(keys, func(i, j int) bool {
+		if keys[i].validatorName != keys[j].validatorName {
+			return keys[i].validatorName < keys[j].validatorName
+		}
+		if keys[i].reasonCode != keys[j].reasonCode {
+			return keys[i].reasonCode < keys[j].reasonCode
+		}
+		return keys[i].message < keys[j].message
+	})
+	out := make([]RejectionSummary, 0, len(keys))
+	for _, k := range keys {
+		out = append(out, RejectionSummary{
+			ValidatorName: k.validatorName,
+			ReasonCode:    k.reasonCode,
+			Message:       k.message,
+			Count:         counts[k],
+		})
+	}
+	return out
+}
+
+func statusForRejected(rejected []contracts.RejectedOutput) StageStatus {
+	if len(rejected) > 0 {
+		return StatusSucceededWithRejections
+	}
+	return StatusSucceeded
+}
+
+func validationStatusString(warnings []contracts.Warning, rejected []contracts.RejectedOutput) string {
+	if len(rejected) > 0 {
+		return "rejected"
+	}
+	if len(warnings) > 0 {
+		return "approved_with_warnings"
+	}
+	return "approved"
+}
+
+func errorMetadata(err error) map[string]string {
+	if err == nil {
+		return nil
+	}
+	return map[string]string{"error": err.Error()}
+}
+
+func laneManifestPath(stage string, laneID string) string {
+	return lanePayloadPath(stage, laneID, "manifest.json")
+}
+
+func lanePayloadPath(stage string, laneID string, file string) string {
+	return path.Join(stage, checkpointPathComponent(laneID), file)
+}
+
+func checkpointPathComponent(value string) string {
+	value = strings.TrimSpace(value)
+	if value == "" {
+		return "_"
+	}
+	var b strings.Builder
+	for _, r := range value {
+		switch {
+		case r >= 'a' && r <= 'z':
+			b.WriteRune(r)
+		case r >= 'A' && r <= 'Z':
+			b.WriteRune(r)
+		case r >= '0' && r <= '9':
+			b.WriteRune(r)
+		case r == '-' || r == '_' || r == '.':
+			b.WriteRune(r)
+		default:
+			b.WriteString(fmt.Sprintf("~%x", r))
+		}
+	}
+	out := b.String()
+	if out == "." || out == ".." || strings.Contains(out, "..") {
+		return "_"
+	}
+	return out
+}
+
+func contentDigest(content []byte) string {
+	sum := sha256.Sum256(content)
+	return "sha256:" + hex.EncodeToString(sum[:])
+}
+
+func timePtr(t time.Time) *time.Time {
+	return &t
+}

internal/framework/checkpoint/recorder_test.go

@@ -0,0 +1,51 @@
+package checkpoint
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+func TestRootBasedRecorderOutputIsReusable(t *testing.T) {
+	root := t.TempDir()
+	identity := testIdentity(t)
+	recorder, err := NewFilesystemRecorder(root, identity)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := recorder.ExtractSucceeded("lane", "module", nil, nil, nil, nil); err != nil {
+		t.Fatal(err)
+	}
+	loader, err := NewFilesystemLoader(root, identity)
+	if err != nil {
+		t.Fatal(err)
+	}
+	result, decision := loader.Extract("lane", "module", nil)
+	if !decision.Reused || len(result.Outputs) != 0 {
+		t.Fatalf("load result=%#v decision=%#v", result, decision)
+	}
+	relative, err := identity.RelativePath()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := os.Stat(filepath.Join(root, relative, "extract", "lane", "manifest.json")); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestCheckpointSchemaCompatibilityIsUnchanged(t *testing.T) {
+	if WorkspaceSchemaVersion != "notarius.workspace.v2" || WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
+		t.Fatal("checkpoint schema identifiers changed")
+	}
+}
+
+func testIdentity(t *testing.T) Identity {
+	t.Helper()
+	identity, err := NewIdentity(IdentityInput{Pipeline: pipeline.ResolvedPipeline{ID: "pipeline", Digest: "sha256:aaaaaaaaaaaaaaaa", Input: pipeline.Binding("input")}, RawInputDigest: "sha256:bbbbbbbbbbbbbbbb"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	return identity
+}

internal/framework/chunkplan/import_boundaries_test.go

@@ -0,0 +1,62 @@
+package chunkplan
+
+import (
+	"go/parser"
+	"go/token"
+	"path/filepath"
+	"runtime"
+	"strconv"
+	"strings"
+	"testing"
+)
+
+const repositoryImportPrefix = "gitea.maximumdirect.net/eric/notarius/internal/"
+
+func TestPlanStoreAndSourceImportBoundaries(t *testing.T) {
+	repositoryRoot := repositoryRoot(t)
+	for _, tc := range []struct {
+		name      string
+		directory string
+		forbidden []string
+	}{
+		{name: "source is framework and module independent", directory: "internal/core/source", forbidden: []string{"framework/", "modules/"}},
+		{name: "plan store is module independent", directory: "internal/framework/chunkplan", forbidden: []string{"modules/"}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			files, err := filepath.Glob(filepath.Join(repositoryRoot, tc.directory, "*.go"))
+			if err != nil {
+				t.Fatal(err)
+			}
+			for _, filename := range files {
+				if strings.HasSuffix(filename, "_test.go") {
+					continue
+				}
+				parsed, err := parser.ParseFile(token.NewFileSet(), filename, nil, parser.ImportsOnly)
+				if err != nil {
+					t.Fatal(err)
+				}
+				for _, item := range parsed.Imports {
+					path, err := strconv.Unquote(item.Path.Value)
+					if err != nil {
+						t.Fatal(err)
+					}
+					path = strings.TrimPrefix(path, repositoryImportPrefix)
+					for _, prefix := range tc.forbidden {
+						if strings.HasPrefix(path, prefix) {
+							t.Fatalf("%s imports %q, forbidden by %s boundary", filepath.Base(filename), path, tc.name)
+						}
+					}
+				}
+			}
+		})
+	}
+}
+
+func repositoryRoot(t *testing.T) string {
+	t.Helper()
+	_, filename, _, ok := runtime.Caller(0)
+	if !ok {
+		t.Fatal("resolve test location")
+	}
+	return filepath.Clean(filepath.Join(filepath.Dir(filename), "..", "..", ".."))
+}

internal/framework/chunkplan/store.go

@@ -0,0 +1,476 @@
+package chunkplan
+
+import (
+	"bytes"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+const SchemaVersion = pipeline.ChunkPlanSchemaVersion
+
+const planFileName = "plan.json"
+
+type filesystemStore struct {
+	root  string
+	write func(*os.Root, string, []byte) error
+}
+
+func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error) {
+	root = strings.TrimSpace(root)
+	if root == "" {
+		return nil, fmt.Errorf("chunk plan root must not be empty")
+	}
+	if strings.ContainsRune(root, '\x00') {
+		return nil, fmt.Errorf("chunk plan root must not contain NUL")
+	}
+	return &filesystemStore{root: filepath.Clean(root), write: writeAtomic}, nil
+}
+
+func (s *filesystemStore) Load(sourceDigest string) (pipeline.ChunkPlanRecord, pipeline.ChunkPlanDecision, error) {
+	digestDir, err := digestPathSegment(sourceDigest)
+	if err != nil {
+		return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{}, err
+	}
+	root, err := s.openRoot(false)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{Status: pipeline.ChunkPlanMissing, Reason: lookupReason(pipeline.ChunkPlanMissing)}, nil
+		}
+		return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{}, fmt.Errorf("open chunk plan root: %w", err)
+	}
+	defer root.Close()
+
+	digestRoot, state, err := openDigestRoot(root, digestDir, false)
+	if err != nil {
+		return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{}, fmt.Errorf("inspect chunk plan directory: %w", err)
+	}
+	if state == entryMissing {
+		return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{Status: pipeline.ChunkPlanMissing, Reason: lookupReason(pipeline.ChunkPlanMissing)}, nil
+	}
+	if state == entryRejected {
+		return invalidDecision()
+	}
+	defer digestRoot.Close()
+
+	data, state, err := readPlan(digestRoot)
+	if err != nil {
+		return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{}, fmt.Errorf("read chunk plan: %w", err)
+	}
+	if state == entryMissing {
+		return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{Status: pipeline.ChunkPlanMissing, Reason: lookupReason(pipeline.ChunkPlanMissing)}, nil
+	}
+	if state == entryRejected {
+		return invalidDecision()
+	}
+
+	var record pipeline.ChunkPlanRecord
+	decoder := json.NewDecoder(bytes.NewReader(data))
+	decoder.DisallowUnknownFields()
+	decoder.UseNumber()
+	if err := decoder.Decode(&record); err != nil {
+		return invalidDecision()
+	}
+	var trailing any
+	if err := decoder.Decode(&trailing); err != io.EOF {
+		return invalidDecision()
+	}
+	if err := validateRecord(record, sourceDigest); err != nil {
+		return invalidDecision()
+	}
+	return record, pipeline.ChunkPlanDecision{Status: pipeline.ChunkPlanHit, Reason: lookupReason(pipeline.ChunkPlanHit)}, nil
+}
+
+func (s *filesystemStore) Save(record pipeline.ChunkPlanRecord) error {
+	if err := validateRecord(record, record.SourceDigest); err != nil {
+		return fmt.Errorf("validate chunk plan record: %w", err)
+	}
+	digestDir, err := digestPathSegment(record.SourceDigest)
+	if err != nil {
+		return err
+	}
+	data, err := json.Marshal(record)
+	if err != nil {
+		return fmt.Errorf("encode chunk plan record: %w", err)
+	}
+	data = append(data, '\n')
+	root, err := s.openRoot(true)
+	if err != nil {
+		return fmt.Errorf("open chunk plan root: %w", err)
+	}
+	defer root.Close()
+	digestRoot, state, err := openDigestRoot(root, digestDir, true)
+	if err != nil {
+		return fmt.Errorf("prepare chunk plan directory: %w", err)
+	}
+	if state == entryRejected {
+		return fmt.Errorf("chunk plan directory has an unsupported type")
+	}
+	defer digestRoot.Close()
+	state, err = inspectPlan(digestRoot, planFileName)
+	if err != nil {
+		return fmt.Errorf("inspect chunk plan file: %w", err)
+	}
+	if state == entryRejected {
+		return fmt.Errorf("chunk plan file has an unsupported type")
+	}
+	writer := s.write
+	if writer == nil {
+		writer = writeAtomic
+	}
+	if err := writer(digestRoot, planFileName, data); err != nil {
+		return fmt.Errorf("write chunk plan: %w", err)
+	}
+	return nil
+}
+
+func (s *filesystemStore) openRoot(create bool) (*os.Root, error) {
+	if s == nil || strings.TrimSpace(s.root) == "" {
+		return nil, fmt.Errorf("chunk plan store must not be nil")
+	}
+	if create {
+		if err := os.MkdirAll(s.root, 0o700); err != nil {
+			return nil, err
+		}
+	}
+	root, err := os.OpenRoot(s.root)
+	if err != nil {
+		return nil, err
+	}
+	if !create {
+		return root, nil
+	}
+	rootDirectory, err := root.Open(".")
+	if err != nil {
+		_ = root.Close()
+		return nil, err
+	}
+	defer rootDirectory.Close()
+	if err := rootDirectory.Chmod(0o700); err != nil {
+		_ = root.Close()
+		return nil, err
+	}
+	return root, nil
+}
+
+type entryState uint8
+
+const (
+	entryPresent entryState = iota
+	entryMissing
+	entryRejected
+)
+
+func inspectDirectory(root *os.Root, digestDir string) (entryState, os.FileInfo, error) {
+	info, err := root.Lstat(digestDir)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return entryMissing, nil, nil
+		}
+		return entryPresent, nil, err
+	}
+	if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
+		return entryRejected, info, nil
+	}
+	return entryPresent, info, nil
+}
+
+type digestOpenHooks struct {
+	BeforeOpen func() error
+}
+
+func openDigestRoot(root *os.Root, digestDir string, create bool) (*os.Root, entryState, error) {
+	return openDigestRootWithHooks(root, digestDir, create, digestOpenHooks{})
+}
+
+func openDigestRootWithHooks(root *os.Root, digestDir string, create bool, hooks digestOpenHooks) (*os.Root, entryState, error) {
+	for {
+		state, before, err := inspectDirectory(root, digestDir)
+		if err != nil {
+			return nil, entryPresent, err
+		}
+		switch state {
+		case entryRejected:
+			return nil, entryRejected, nil
+		case entryMissing:
+			if !create {
+				return nil, entryMissing, nil
+			}
+			if err := root.Mkdir(digestDir, 0o700); err != nil && !errors.Is(err, os.ErrExist) {
+				return nil, entryPresent, err
+			}
+			continue
+		}
+
+		if hooks.BeforeOpen != nil {
+			if err := hooks.BeforeOpen(); err != nil {
+				return nil, entryPresent, err
+			}
+		}
+		digestRoot, err := root.OpenRoot(digestDir)
+		if err != nil {
+			return nil, entryPresent, err
+		}
+		opened, statErr := digestRoot.Stat(".")
+		afterState, after, afterErr := inspectDirectory(root, digestDir)
+		if statErr != nil || afterErr != nil {
+			_ = digestRoot.Close()
+			if statErr != nil {
+				return nil, entryPresent, statErr
+			}
+			return nil, entryPresent, afterErr
+		}
+		if afterState != entryPresent || !os.SameFile(before, after) || !os.SameFile(opened, after) {
+			_ = digestRoot.Close()
+			if afterState == entryRejected {
+				return nil, entryRejected, nil
+			}
+			continue
+		}
+		if create {
+			directory, openErr := digestRoot.Open(".")
+			if openErr != nil {
+				_ = digestRoot.Close()
+				return nil, entryPresent, openErr
+			}
+			chmodErr := directory.Chmod(0o700)
+			closeErr := directory.Close()
+			if chmodErr != nil || closeErr != nil {
+				_ = digestRoot.Close()
+				if chmodErr != nil {
+					return nil, entryPresent, chmodErr
+				}
+				return nil, entryPresent, closeErr
+			}
+		}
+		return digestRoot, entryPresent, nil
+	}
+}
+
+func inspectPlan(root *os.Root, target string) (entryState, error) {
+	info, err := root.Lstat(target)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return entryMissing, nil
+		}
+		return entryPresent, err
+	}
+	if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
+		return entryRejected, nil
+	}
+	return entryPresent, nil
+}
+
+type planReadHooks struct {
+	BeforeOpen func() error
+}
+
+func readPlan(root *os.Root) ([]byte, entryState, error) {
+	return readPlanWithHooks(root, planReadHooks{})
+}
+
+func readPlanWithHooks(root *os.Root, hooks planReadHooks) ([]byte, entryState, error) {
+	state, err := inspectPlan(root, planFileName)
+	if err != nil || state != entryPresent {
+		return nil, state, err
+	}
+	if hooks.BeforeOpen != nil {
+		if err := hooks.BeforeOpen(); err != nil {
+			return nil, entryPresent, err
+		}
+	}
+	file, err := root.OpenFile(planFileName, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, entryMissing, nil
+		}
+		if errors.Is(err, syscall.ELOOP) {
+			return nil, entryRejected, nil
+		}
+		return nil, entryPresent, err
+	}
+	defer file.Close()
+	opened, err := file.Stat()
+	if err != nil {
+		return nil, entryPresent, err
+	}
+	if !opened.Mode().IsRegular() {
+		return nil, entryRejected, nil
+	}
+	currentState, currentErr := inspectPlan(root, planFileName)
+	if currentErr != nil {
+		return nil, entryPresent, currentErr
+	}
+	if currentState == entryRejected {
+		return nil, entryRejected, nil
+	}
+	data, err := io.ReadAll(file)
+	if err != nil {
+		return nil, entryPresent, err
+	}
+	return data, entryPresent, nil
+}
+
+func digestPathSegment(digest string) (string, error) {
+	if !strings.HasPrefix(digest, "sha256:") {
+		return "", fmt.Errorf("source digest must use sha256:<64 lowercase hex> format")
+	}
+	hexDigest := strings.TrimPrefix(digest, "sha256:")
+	if len(hexDigest) != 64 || strings.ToLower(hexDigest) != hexDigest {
+		return "", fmt.Errorf("source digest must use sha256:<64 lowercase hex> format")
+	}
+	decoded, err := hex.DecodeString(hexDigest)
+	if err != nil || len(decoded) != 32 {
+		return "", fmt.Errorf("source digest must use sha256:<64 lowercase hex> format")
+	}
+	return hexDigest, nil
+}
+
+func validateRecord(record pipeline.ChunkPlanRecord, requestedDigest string) error {
+	if record.SchemaVersion != SchemaVersion {
+		return fmt.Errorf("schema_version %q is not supported", record.SchemaVersion)
+	}
+	if _, err := digestPathSegment(requestedDigest); err != nil {
+		return err
+	}
+	if record.SourceDigest != requestedDigest {
+		return fmt.Errorf("source_digest does not match requested source")
+	}
+	if record.Plan.SourceDigest != record.SourceDigest {
+		return fmt.Errorf("plan source_digest does not match record source_digest")
+	}
+	if len(record.Plan.Ranges) == 0 {
+		return fmt.Errorf("plan ranges must not be empty")
+	}
+	if err := source.ValidateChunkAnnotations(record.Plan.Annotations); err != nil {
+		return fmt.Errorf("plan annotations: %w", err)
+	}
+	seenStarts := make(map[int]struct{}, len(record.Plan.Ranges))
+	for i, chunkRange := range record.Plan.Ranges {
+		if chunkRange.StartUnitID <= 0 || chunkRange.EndUnitID <= 0 {
+			return fmt.Errorf("plan range[%d] boundaries must be positive", i)
+		}
+		if _, exists := seenStarts[chunkRange.StartUnitID]; exists {
+			return fmt.Errorf("plan range[%d] duplicates start_unit_id %d", i, chunkRange.StartUnitID)
+		}
+		seenStarts[chunkRange.StartUnitID] = struct{}{}
+		if err := source.ValidateChunkAnnotations(chunkRange.Annotations); err != nil {
+			return fmt.Errorf("plan range[%d] annotations: %w", i, err)
+		}
+	}
+	wantDigest, err := source.DigestChunkPlan(record.Plan)
+	if err != nil {
+		return fmt.Errorf("digest plan: %w", err)
+	}
+	if record.PlanDigest != wantDigest {
+		return fmt.Errorf("plan_digest does not match plan")
+	}
+	if strings.TrimSpace(record.Producer.InputModule) == "" {
+		return fmt.Errorf("producer input_module must not be empty")
+	}
+	if strings.TrimSpace(record.Producer.ChunkModule) == "" {
+		return fmt.Errorf("producer chunk_module must not be empty")
+	}
+	if record.CreatedAt.IsZero() {
+		return fmt.Errorf("created_at must not be zero")
+	}
+	return nil
+}
+
+func invalidDecision() (pipeline.ChunkPlanRecord, pipeline.ChunkPlanDecision, error) {
+	return pipeline.ChunkPlanRecord{}, pipeline.ChunkPlanDecision{Status: pipeline.ChunkPlanInvalid, Reason: lookupReason(pipeline.ChunkPlanInvalid)}, nil
+}
+
+func lookupReason(status pipeline.ChunkPlanStatus) string {
+	switch status {
+	case pipeline.ChunkPlanHit:
+		return "stored chunk plan is valid"
+	case pipeline.ChunkPlanMissing:
+		return "chunk plan not found"
+	case pipeline.ChunkPlanInvalid:
+		return "stored chunk plan is invalid"
+	default:
+		return "chunk plan lookup skipped"
+	}
+}
+
+type atomicWriteHooks struct {
+	BeforeCreateTemp func() error
+	BeforeRename     func() error
+}
+
+func writeAtomic(root *os.Root, target string, data []byte) error {
+	return writeAtomicWithHooks(root, target, data, atomicWriteHooks{})
+}
+
+func writeAtomicWithHooks(root *os.Root, target string, data []byte, hooks atomicWriteHooks) error {
+	if hooks.BeforeCreateTemp != nil {
+		if err := hooks.BeforeCreateTemp(); err != nil {
+			return err
+		}
+	}
+	temp, tempPath, err := createTemporaryFile(root, target)
+	if err != nil {
+		return err
+	}
+	removeTemp := true
+	defer func() {
+		if removeTemp {
+			_ = root.Remove(tempPath)
+		}
+	}()
+	if err := temp.Chmod(0o600); err != nil {
+		_ = temp.Close()
+		return err
+	}
+	if _, err := temp.Write(data); err != nil {
+		_ = temp.Close()
+		return err
+	}
+	if err := temp.Sync(); err != nil {
+		_ = temp.Close()
+		return err
+	}
+	if err := temp.Close(); err != nil {
+		return err
+	}
+	if hooks.BeforeRename != nil {
+		if err := hooks.BeforeRename(); err != nil {
+			return err
+		}
+	}
+	if err := root.Rename(tempPath, target); err != nil {
+		return err
+	}
+	removeTemp = false
+	return nil
+}
+
+func createTemporaryFile(root *os.Root, target string) (*os.File, string, error) {
+	for attempt := 0; attempt < 32; attempt++ {
+		var suffix [16]byte
+		if _, err := rand.Read(suffix[:]); err != nil {
+			return nil, "", err
+		}
+		path := strings.TrimSuffix(target, planFileName) + ".plan.json.tmp-" + hex.EncodeToString(suffix[:])
+		file, err := root.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
+		if errors.Is(err, os.ErrExist) {
+			continue
+		}
+		if err != nil {
+			return nil, "", err
+		}
+		return file, path, nil
+	}
+	return nil, "", fmt.Errorf("create unique temporary chunk plan file")
+}

internal/framework/chunkplan/store_test.go

@@ -0,0 +1,563 @@
+package chunkplan
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
+	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+const testSourceDigest = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+
+func TestFilesystemStoreRoundTripAndExactPath(t *testing.T) {
+	root := filepath.Join(t.TempDir(), "plans")
+	store := newStore(t, root)
+	record := testRecord(t, 1)
+	if err := store.Save(record); err != nil {
+		t.Fatalf("Save() error = %v", err)
+	}
+
+	target := filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:"), "plan.json")
+	data, err := os.ReadFile(target)
+	if err != nil {
+		t.Fatalf("read exact plan path: %v", err)
+	}
+	if bytes.Contains(data, []byte("RAW REFERENCE CONTENT")) {
+		t.Fatal("stored record contains raw reference content")
+	}
+	var envelope struct {
+		Producer struct {
+			References []map[string]any `json:"references"`
+		} `json:"producer"`
+	}
+	if err := json.Unmarshal(data, &envelope); err != nil {
+		t.Fatal(err)
+	}
+	if len(envelope.Producer.References) != 1 {
+		t.Fatalf("stored references = %#v", envelope.Producer.References)
+	}
+	if _, exists := envelope.Producer.References[0]["content"]; exists {
+		t.Fatalf("stored reference contains content field: %#v", envelope.Producer.References[0])
+	}
+
+	got, decision, err := store.Load(testSourceDigest)
+	if err != nil || decision.Status != pipeline.ChunkPlanHit {
+		t.Fatalf("Load() decision=%#v error=%v", decision, err)
+	}
+	if !reflect.DeepEqual(got, record) {
+		t.Fatalf("round trip record = %#v, want %#v", got, record)
+	}
+}
+
+func TestFilesystemStorePermissions(t *testing.T) {
+	root := filepath.Join(t.TempDir(), "plans")
+	store := newStore(t, root)
+	if err := store.Save(testRecord(t, 1)); err != nil {
+		t.Fatal(err)
+	}
+	digestDir := filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:"))
+	for path, want := range map[string]os.FileMode{
+		root:                                  0o700,
+		digestDir:                             0o700,
+		filepath.Join(digestDir, "plan.json"): 0o600,
+	} {
+		info, err := os.Stat(path)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if got := info.Mode().Perm(); got != want {
+			t.Fatalf("%s mode = %04o, want %04o", path, got, want)
+		}
+	}
+}
+
+func TestFilesystemStoreMissingAndOperationalErrors(t *testing.T) {
+	root := filepath.Join(t.TempDir(), "plans")
+	store := newStore(t, root)
+	_, decision, err := store.Load(testSourceDigest)
+	if err != nil || decision.Status != pipeline.ChunkPlanMissing {
+		t.Fatalf("missing decision=%#v error=%v", decision, err)
+	}
+	if _, err := os.Stat(root); !os.IsNotExist(err) {
+		t.Fatalf("Load() created missing root: %v", err)
+	}
+
+	target := filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:"), "plan.json")
+	if err := os.MkdirAll(target, 0o700); err != nil {
+		t.Fatal(err)
+	}
+	_, decision, err = store.Load(testSourceDigest)
+	if err != nil || decision.Status != pipeline.ChunkPlanInvalid {
+		t.Fatalf("Load(plan.json directory) decision=%#v error=%v", decision, err)
+	}
+}
+
+func TestFilesystemStoreRejectsSymlinkedEntries(t *testing.T) {
+	tests := []struct {
+		name  string
+		setup func(t *testing.T, root, outside string)
+	}{
+		{
+			name: "digest directory",
+			setup: func(t *testing.T, root, outside string) {
+				t.Helper()
+				if err := os.MkdirAll(outside, 0o700); err != nil {
+					t.Fatal(err)
+				}
+				writeFile(t, filepath.Join(outside, "plan.json"), []byte("outside plan"), 0o640)
+				createSymlink(t, outside, filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:")))
+			},
+		},
+		{
+			name: "plan file",
+			setup: func(t *testing.T, root, outside string) {
+				t.Helper()
+				digestDir := filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:"))
+				if err := os.MkdirAll(digestDir, 0o700); err != nil {
+					t.Fatal(err)
+				}
+				writeFile(t, outside, []byte("outside plan"), 0o640)
+				createSymlink(t, outside, filepath.Join(digestDir, "plan.json"))
+			},
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			root := filepath.Join(t.TempDir(), "plans")
+			if err := os.MkdirAll(root, 0o700); err != nil {
+				t.Fatal(err)
+			}
+			outside := filepath.Join(t.TempDir(), "outside")
+			tc.setup(t, root, outside)
+			before := readFile(t, outsidePlanPath(tc.name, outside))
+			beforeMode := fileMode(t, outsidePlanPath(tc.name, outside))
+
+			store := newStore(t, root)
+			_, decision, err := store.Load(testSourceDigest)
+			if err != nil || decision.Status != pipeline.ChunkPlanInvalid {
+				t.Fatalf("Load() decision=%#v error=%v", decision, err)
+			}
+			if err := store.Save(testRecord(t, 1)); err == nil {
+				t.Fatal("Save() error = nil")
+			}
+			outsidePlan := outsidePlanPath(tc.name, outside)
+			if got := readFile(t, outsidePlan); !bytes.Equal(got, before) {
+				t.Fatalf("outside content = %q, want %q", got, before)
+			}
+			if got := fileMode(t, outsidePlan); got != beforeMode {
+				t.Fatalf("outside mode = %04o, want %04o", got, beforeMode)
+			}
+		})
+	}
+}
+
+func TestOpenDigestRootRejectsEntryReplacedDuringOpen(t *testing.T) {
+	rootPath := t.TempDir()
+	digestDir := strings.TrimPrefix(testSourceDigest, "sha256:")
+	if err := os.Mkdir(filepath.Join(rootPath, digestDir), 0o700); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Mkdir(filepath.Join(rootPath, "redirect"), 0o700); err != nil {
+		t.Fatal(err)
+	}
+	root, err := os.OpenRoot(rootPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer root.Close()
+
+	traced := false
+	opened, state, err := openDigestRootWithHooks(root, digestDir, false, digestOpenHooks{BeforeOpen: func() error {
+		if traced {
+			return nil
+		}
+		traced = true
+		if err := os.Rename(filepath.Join(rootPath, digestDir), filepath.Join(rootPath, "original")); err != nil {
+			return err
+		}
+		return os.Symlink("redirect", filepath.Join(rootPath, digestDir))
+	}})
+	if opened != nil {
+		_ = opened.Close()
+	}
+	if err != nil || state != entryRejected {
+		t.Fatalf("openDigestRootWithHooks() root=%v state=%v error=%v, want nil/rejected/nil", opened, state, err)
+	}
+}
+
+func TestReadPlanRejectsEntryReplacedDuringOpen(t *testing.T) {
+	rootPath := t.TempDir()
+	writeFile(t, filepath.Join(rootPath, planFileName), []byte("original"), 0o600)
+	writeFile(t, filepath.Join(rootPath, "redirect.json"), []byte("redirect"), 0o600)
+	root, err := os.OpenRoot(rootPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer root.Close()
+
+	data, state, err := readPlanWithHooks(root, planReadHooks{BeforeOpen: func() error {
+		if err := os.Remove(filepath.Join(rootPath, planFileName)); err != nil {
+			return err
+		}
+		return os.Symlink("redirect.json", filepath.Join(rootPath, planFileName))
+	}})
+	if err != nil || state != entryRejected || data != nil {
+		t.Fatalf("readPlanWithHooks() data=%q state=%v error=%v, want nil/rejected/nil", data, state, err)
+	}
+}
+
+func TestFilesystemStoreRejectsUnexpectedEntryTypes(t *testing.T) {
+	tests := []struct {
+		name  string
+		setup func(t *testing.T, root string)
+	}{
+		{
+			name: "digest file",
+			setup: func(t *testing.T, root string) {
+				writeFile(t, filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:")), []byte("not a directory"), 0o600)
+			},
+		},
+		{
+			name: "plan directory",
+			setup: func(t *testing.T, root string) {
+				if err := os.MkdirAll(filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:"), "plan.json"), 0o700); err != nil {
+					t.Fatal(err)
+				}
+			},
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			root := t.TempDir()
+			tc.setup(t, root)
+			store := newStore(t, root)
+			_, decision, err := store.Load(testSourceDigest)
+			if err != nil || decision.Status != pipeline.ChunkPlanInvalid {
+				t.Fatalf("Load() decision=%#v error=%v", decision, err)
+			}
+			if err := store.Save(testRecord(t, 1)); err == nil {
+				t.Fatal("Save() error = nil")
+			}
+		})
+	}
+}
+
+func TestFilesystemStoreRejectsMalformedSourceDigests(t *testing.T) {
+	store := newStore(t, t.TempDir())
+	for _, digest := range []string{"", "sha1:" + strings.Repeat("a", 64), "sha256:../escape", "sha256:" + strings.Repeat("A", 64), "sha256:" + strings.Repeat("a", 63)} {
+		t.Run(digest, func(t *testing.T) {
+			if _, _, err := store.Load(digest); err == nil {
+				t.Fatal("Load() error = nil")
+			}
+			record := testRecord(t, 1)
+			record.SourceDigest = digest
+			record.Plan.SourceDigest = digest
+			record.PlanDigest, _ = source.DigestChunkPlan(record.Plan)
+			if err := store.Save(record); err == nil {
+				t.Fatal("Save() error = nil")
+			}
+		})
+	}
+}
+
+func TestFilesystemStoreReportsInvalidRecordsAsRecoverable(t *testing.T) {
+	tests := []struct {
+		name   string
+		mutate func([]byte) []byte
+	}{
+		{name: "unknown field", mutate: func(data []byte) []byte {
+			return bytes.Replace(data, []byte(`{"schema_version"`), []byte(`{"SENTINEL_UNKNOWN_FIELD":true,"schema_version"`), 1)
+		}},
+		{name: "truncated JSON", mutate: func(data []byte) []byte { return data[:len(data)/2] }},
+		{name: "schema mismatch", mutate: replaceJSON(`notarius.chunk-plan.v1`, `SENTINEL_SCHEMA_VALUE`)},
+		{name: "source mismatch", mutate: replaceJSON(testSourceDigest, "sha256:"+strings.Repeat("b", 64))},
+		{name: "plan digest mismatch", mutate: func(data []byte) []byte {
+			prefix := []byte(`"plan_digest":"sha256:`)
+			index := bytes.Index(data, prefix)
+			if index >= 0 {
+				data[index+len(prefix)] = '0'
+			}
+			return data
+		}},
+		{name: "noncanonical annotation", mutate: func(data []byte) []byte {
+			return bytes.Replace(data, []byte(`"test/value"`), []byte(`"SENTINEL_ANNOTATION_NAMESPACE"`), 1)
+		}},
+		{name: "bad boundary", mutate: func(data []byte) []byte {
+			return bytes.Replace(data, []byte(`"start_unit_id":1`), []byte(`"start_unit_id":0`), 1)
+		}},
+		{name: "timestamp", mutate: replaceJSON(`2026-07-18T12:00:00Z`, `SENTINEL_TIMESTAMP`)},
+		{name: "trailing JSON", mutate: func(data []byte) []byte { return append(data, []byte(` {"SENTINEL_TRAILING":true}`)...) }},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			root := t.TempDir()
+			store := newStore(t, root)
+			record := testRecord(t, 1)
+			if err := store.Save(record); err != nil {
+				t.Fatal(err)
+			}
+			path := filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:"), "plan.json")
+			data, err := os.ReadFile(path)
+			if err != nil {
+				t.Fatal(err)
+			}
+			if err := os.WriteFile(path, tc.mutate(data), 0o600); err != nil {
+				t.Fatal(err)
+			}
+			got, decision, err := store.Load(testSourceDigest)
+			if err != nil || decision.Status != pipeline.ChunkPlanInvalid || !reflect.DeepEqual(got, pipeline.ChunkPlanRecord{}) || decision.Reason != "stored chunk plan is invalid" {
+				t.Fatalf("record=%#v decision=%#v error=%v", got, decision, err)
+			}
+			for _, sentinel := range []string{"SENTINEL_UNKNOWN_FIELD", "SENTINEL_SCHEMA_VALUE", "SENTINEL_ANNOTATION_NAMESPACE", "SENTINEL_TIMESTAMP", "SENTINEL_TRAILING"} {
+				if strings.Contains(decision.Reason, sentinel) {
+					t.Fatalf("decision leaked %q: %#v", sentinel, decision)
+				}
+			}
+		})
+	}
+}
+
+func TestFilesystemStoreAtomicallyReplacesAndPreservesValidRecordOnFailure(t *testing.T) {
+	root := t.TempDir()
+	store := newStore(t, root)
+	first := testRecord(t, 1)
+	second := testRecord(t, 2)
+	if err := store.Save(first); err != nil {
+		t.Fatal(err)
+	}
+	invalid := second
+	invalid.PlanDigest = "sha256:" + strings.Repeat("0", 64)
+	if err := store.Save(invalid); err == nil {
+		t.Fatal("Save(invalid) error = nil")
+	}
+	got, _, _ := store.Load(testSourceDigest)
+	if !reflect.DeepEqual(got, first) {
+		t.Fatalf("record after failed replacement = %#v", got)
+	}
+	if err := store.Save(second); err != nil {
+		t.Fatal(err)
+	}
+	got, _, _ = store.Load(testSourceDigest)
+	if !reflect.DeepEqual(got, second) {
+		t.Fatalf("record after replacement = %#v", got)
+	}
+	assertNoTemps(t, filepath.Join(root, strings.TrimPrefix(testSourceDigest, "sha256:")))
+}
+
+func TestFilesystemStoreConcurrentWritersExposeCompleteRecord(t *testing.T) {
+	store := newStore(t, t.TempDir())
+	const writers = 24
+	records := make([]pipeline.ChunkPlanRecord, writers)
+	for i := range records {
+		records[i] = testRecord(t, i+1)
+	}
+	var wg sync.WaitGroup
+	errs := make(chan error, writers)
+	for i := range records {
+		wg.Add(1)
+		go func(record pipeline.ChunkPlanRecord) {
+			defer wg.Done()
+			errs <- store.Save(record)
+		}(records[i])
+	}
+	wg.Wait()
+	close(errs)
+	for err := range errs {
+		if err != nil {
+			t.Fatalf("concurrent Save() error = %v", err)
+		}
+	}
+	got, decision, err := store.Load(testSourceDigest)
+	if err != nil || decision.Status != pipeline.ChunkPlanHit {
+		t.Fatalf("Load() decision=%#v error=%v", decision, err)
+	}
+	var annotation struct {
+		Value int `json:"value"`
+	}
+	if err := json.Unmarshal(got.Plan.Annotations["test/value"], &annotation); err != nil || annotation.Value < 1 || annotation.Value > writers {
+		t.Fatalf("final annotation=%#v error=%v", annotation, err)
+	}
+}
+
+func TestFilesystemStoreReadersObserveOnlyCompleteRecordsDuringWrites(t *testing.T) {
+	store := newStore(t, t.TempDir())
+	if err := store.Save(testRecord(t, 1)); err != nil {
+		t.Fatal(err)
+	}
+	const writers = 12
+	const readers = 12
+	errs := make(chan error, writers+readers)
+	start := make(chan struct{})
+	var writersDone sync.WaitGroup
+	for i := 0; i < writers; i++ {
+		writersDone.Add(1)
+		go func(value int) {
+			defer writersDone.Done()
+			<-start
+			errs <- store.Save(testRecord(t, value+2))
+		}(i)
+	}
+	for i := 0; i < readers; i++ {
+		go func() {
+			<-start
+			for attempt := 0; attempt < 50; attempt++ {
+				record, decision, err := store.Load(testSourceDigest)
+				if err != nil || decision.Status != pipeline.ChunkPlanHit {
+					errs <- fmt.Errorf("Load() decision=%#v error=%v", decision, err)
+					return
+				}
+				if err := validateRecord(record, testSourceDigest); err != nil {
+					errs <- fmt.Errorf("reader observed invalid record: %w", err)
+					return
+				}
+			}
+			errs <- nil
+		}()
+	}
+	close(start)
+	writersDone.Wait()
+	for i := 0; i < readers+writers; i++ {
+		if err := <-errs; err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+func TestFilesystemStoreInterruptedWritesPreservePreviousRecord(t *testing.T) {
+	store := newStore(t, t.TempDir()).(*filesystemStore)
+	first := testRecord(t, 1)
+	if err := store.Save(first); err != nil {
+		t.Fatal(err)
+	}
+	for _, tc := range []struct {
+		name  string
+		hooks atomicWriteHooks
+	}{
+		{name: "before temporary file", hooks: atomicWriteHooks{BeforeCreateTemp: func() error { return errors.New("interrupted before temporary file") }}},
+		{name: "before rename", hooks: atomicWriteHooks{BeforeRename: func() error { return errors.New("interrupted before rename") }}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			store.write = func(root *os.Root, target string, data []byte) error {
+				return writeAtomicWithHooks(root, target, data, tc.hooks)
+			}
+			if err := store.Save(testRecord(t, 2)); err == nil {
+				t.Fatal("Save() error = nil")
+			}
+			store.write = writeAtomic
+			got, decision, err := store.Load(testSourceDigest)
+			if err != nil || decision.Status != pipeline.ChunkPlanHit || !reflect.DeepEqual(got, first) {
+				t.Fatalf("record after interruption=%#v decision=%#v error=%v", got, decision, err)
+			}
+		})
+	}
+}
+
+func newStore(t *testing.T, root string) pipeline.ChunkPlanStore {
+	t.Helper()
+	store, err := NewFilesystemStore(root)
+	if err != nil {
+		t.Fatalf("NewFilesystemStore() error = %v", err)
+	}
+	return store
+}
+
+func testRecord(t *testing.T, value int) pipeline.ChunkPlanRecord {
+	t.Helper()
+	annotation, err := json.Marshal(map[string]int{"value": value})
+	if err != nil {
+		t.Fatal(err)
+	}
+	plan := source.ChunkPlan{
+		SourceDigest: testSourceDigest,
+		Ranges:       []source.ChunkRange{{StartUnitID: 1, EndUnitID: 2, Annotations: source.ChunkAnnotations{"test/range": json.RawMessage(`{"range":true}`)}}},
+		Annotations:  source.ChunkAnnotations{"test/value": annotation},
+	}
+	planDigest, err := source.DigestChunkPlan(plan)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return pipeline.ChunkPlanRecord{
+		SchemaVersion: SchemaVersion,
+		SourceDigest:  testSourceDigest,
+		PlanDigest:    planDigest,
+		Plan:          plan,
+		Producer: pipeline.ChunkPlanProducer{
+			InputModule: "input/test", ChunkModule: "chunk/test", LLMProfile: "profile/test",
+			References: []artifacts.ReferenceProvenance{{Stage: "chunk", SlotName: "guide", OriginType: "file", OriginURI: "file:///guide.txt", Digest: "sha256:reference"}},
+			Metadata:   map[string]any{"prompt_id": "test/prompt", "enabled": true},
+		},
+		Warnings:  []contracts.Warning{{Scope: "chunk/test", ReasonCode: "observed", Message: "warning"}},
+		CreatedAt: time.Date(2026, 7, 18, 12, 0, 0, 0, time.UTC),
+	}
+}
+
+func replaceJSON(old, replacement string) func([]byte) []byte {
+	return func(data []byte) []byte { return bytes.Replace(data, []byte(old), []byte(replacement), 1) }
+}
+
+func assertNoTemps(t *testing.T, dir string) {
+	t.Helper()
+	entries, err := os.ReadDir(dir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, entry := range entries {
+		if strings.Contains(entry.Name(), ".tmp-") {
+			t.Fatalf("temporary file remains: %s", entry.Name())
+		}
+	}
+}
+
+func createSymlink(t *testing.T, target, link string) {
+	t.Helper()
+	if err := os.Symlink(target, link); err != nil {
+		t.Skipf("create symlink: %v", err)
+	}
+}
+
+func writeFile(t *testing.T, path string, data []byte, mode os.FileMode) {
+	t.Helper()
+	if err := os.WriteFile(path, data, mode); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Chmod(path, mode); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func readFile(t *testing.T, path string) []byte {
+	t.Helper()
+	data, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return data
+}
+
+func fileMode(t *testing.T, path string) os.FileMode {
+	t.Helper()
+	info, err := os.Stat(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return info.Mode().Perm()
+}
+
+func outsidePlanPath(name, outside string) string {
+	if name == "digest directory" {
+		return filepath.Join(outside, "plan.json")
+	}
+	return outside
+}

internal/framework/contracts/artifact.go

@@ -0,0 +1,85 @@
+package contracts
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+)
+
+// ArtifactKind is the stable logical identity of a domain artifact.
+type ArtifactKind string
+
+// ArtifactSchema describes the durable representation owned by an artifact
+// codec. JSONSchema is cloned whenever framework ownership changes.
+type ArtifactSchema struct {
+	ID         string `json:"id"`
+	Name       string `json:"name"`
+	Version    string `json:"version"`
+	JSONSchema []byte `json:"-"`
+}
+
+// SerializedArtifact is the domain-neutral representation of a typed
+// artifact at an explicit serialization boundary.
+type SerializedArtifact struct {
+	Kind      ArtifactKind   `json:"kind"`
+	Schema    ArtifactSchema `json:"schema"`
+	MediaType string         `json:"media_type"`
+	Content   []byte         `json:"-"`
+	Metadata  map[string]any `json:"metadata,omitempty"`
+}
+
+// SerializedOutput associates a domain-neutral artifact with the pipeline
+// operation that produced it. Provenance remains outside codec-owned bytes.
+type SerializedOutput struct {
+	LaneID        string             `json:"lane_id"`
+	NormalizerKey string             `json:"normalizer_key"`
+	SourceID      string             `json:"source_id,omitempty"`
+	Artifact      SerializedArtifact `json:"artifact"`
+}
+
+// ArtifactCodec owns the stable encoding for one concrete artifact type.
+type ArtifactCodec[T any] interface {
+	Kind() ArtifactKind
+	Schema() ArtifactSchema
+	MediaType() string
+	// EncodeCandidate serializes a stage result before semantic validation. It
+	// must not apply validity checks owned by typed validators; Encode remains
+	// the strict final-artifact boundary used after validation succeeds.
+	EncodeCandidate(T) ([]byte, error)
+	Encode(T) ([]byte, error)
+	Decode([]byte) (T, error)
+}
+
+// DigestArtifactSchema returns the SHA-256 digest of the exact JSON Schema
+// bytes. Schema formatting is therefore part of the registered identity.
+func DigestArtifactSchema(schema ArtifactSchema) string {
+	sum := sha256.Sum256(schema.JSONSchema)
+	return "sha256:" + hex.EncodeToString(sum[:])
+}
+
+func CloneArtifactSchema(schema ArtifactSchema) ArtifactSchema {
+	schema.JSONSchema = append([]byte(nil), schema.JSONSchema...)
+	return schema
+}
+
+func CloneSerializedArtifact(artifact SerializedArtifact) SerializedArtifact {
+	artifact.Schema = CloneArtifactSchema(artifact.Schema)
+	artifact.Content = append([]byte(nil), artifact.Content...)
+	artifact.Metadata = cloneArtifactMetadata(artifact.Metadata)
+	return artifact
+}
+
+func CloneSerializedOutput(output SerializedOutput) SerializedOutput {
+	output.Artifact = CloneSerializedArtifact(output.Artifact)
+	return output
+}
+
+func cloneArtifactMetadata(metadata map[string]any) map[string]any {
+	if len(metadata) == 0 {
+		return nil
+	}
+	out := make(map[string]any, len(metadata))
+	for key, value := range metadata {
+		out[key] = value
+	}
+	return out
+}

internal/framework/contracts/artifact_test.go

@@ -0,0 +1,45 @@
+package contracts
+
+import "testing"
+
+func TestDigestArtifactSchemaUsesExactBytes(t *testing.T) {
+	first := ArtifactSchema{JSONSchema: []byte(`{"type":"object"}`)}
+	second := ArtifactSchema{JSONSchema: []byte("{\n  \"type\": \"object\"\n}")}
+
+	if got := DigestArtifactSchema(first); got != "sha256:a2c799262a3ce3c19ef5cdd983bf3d12b43ab3c426227091b909dcb7054738c0" {
+		t.Fatalf("DigestArtifactSchema() = %q, want stable SHA-256", got)
+	}
+	if DigestArtifactSchema(first) == DigestArtifactSchema(second) {
+		t.Fatal("schema digests match for different exact bytes")
+	}
+}
+
+func TestArtifactCloneHelpersOwnSlicesAndMaps(t *testing.T) {
+	schema := ArtifactSchema{ID: "notes.v1", Name: "notes", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
+	artifact := SerializedArtifact{
+		Kind:      "test/notes",
+		Schema:    schema,
+		MediaType: "application/json",
+		Content:   []byte(`{"items":["one"]}`),
+		Metadata:  map[string]any{"origin": "test"},
+	}
+
+	clonedSchema := CloneArtifactSchema(schema)
+	cloned := CloneSerializedArtifact(artifact)
+	schema.JSONSchema[0] = '['
+	artifact.Content[0] = '['
+	artifact.Metadata["origin"] = "changed"
+
+	if string(clonedSchema.JSONSchema) != `{"type":"object"}` {
+		t.Fatalf("cloned schema = %q, want original bytes", clonedSchema.JSONSchema)
+	}
+	if string(cloned.Schema.JSONSchema) != `{"type":"object"}` {
+		t.Fatalf("serialized artifact schema = %q, want original bytes", cloned.Schema.JSONSchema)
+	}
+	if string(cloned.Content) != `{"items":["one"]}` {
+		t.Fatalf("cloned content = %q, want original bytes", cloned.Content)
+	}
+	if cloned.Metadata["origin"] != "test" {
+		t.Fatalf("cloned metadata = %#v, want independent map", cloned.Metadata)
+	}
+}

internal/framework/contracts/composition_test.go

@@ -1,307 +0,0 @@
-package contracts_test
-
-import (
-	"context"
-	"encoding/json"
-	"errors"
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
-	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
-)
-
-var _ contracts.InputAdapter = compositionAdapter{}
-var _ contracts.Chunker = compositionChunker{}
-var _ contracts.Extractor = compositionExtractor{}
-var _ contracts.Merger = compositionMerger{}
-var _ contracts.Normalizer = compositionNormalizer{}
-var _ contracts.Validator = compositionValidator{}
-var _ contracts.OutputEncoder = compositionOutputEncoder{}
-
-func TestContractsComposeAcrossPackages(t *testing.T) {
-	ctx := context.Background()
-	adapter := compositionAdapter{}
-	chunker := compositionChunker{}
-	extractor := compositionExtractor{}
-	merger := compositionMerger{}
-	normalizer := compositionNormalizer{}
-	validator := compositionValidator{}
-	encoder := compositionOutputEncoder{}
-
-	doc, err := adapter.Parse(ctx, contracts.ParseRequest{SourceID: "source-1"})
-	if err != nil {
-		t.Fatalf("Parse() error = %v, want nil", err)
-	}
-	if err := source.ValidateDocument(doc); err != nil {
-		t.Fatalf("ValidateDocument() error = %v, want nil", err)
-	}
-
-	chunking, err := chunker.Chunk(ctx, contracts.ChunkRequest{
-		Source:   doc,
-		Metadata: map[string]any{"max_units": 2},
-	})
-	if err != nil {
-		t.Fatalf("Chunk() error = %v, want nil", err)
-	}
-	if len(chunking.Chunks) != 1 {
-		t.Fatalf("len(Chunks) = %d, want 1", len(chunking.Chunks))
-	}
-
-	extraction, err := extractor.Extract(ctx, contracts.ExtractionRequest{
-		Source:         doc,
-		Chunk:          &chunking.Chunks[0],
-		AmbientContext: map[string]any{"synopsis": "example synopsis"},
-	})
-	if err != nil {
-		t.Fatalf("Extract() error = %v, want nil", err)
-	}
-	if len(extraction.Candidates) != 1 {
-		t.Fatalf("len(Candidates) = %d, want 1", len(extraction.Candidates))
-	}
-
-	candidate := extraction.Candidates[0]
-	for _, ref := range candidate.SourceRefs {
-		if err := source.ValidateRef(doc, ref); err != nil {
-			t.Fatalf("ValidateRef() error = %v, want nil", err)
-		}
-	}
-
-	merge, err := merger.Merge(ctx, contracts.MergeRequest{
-		Source: doc,
-		LaneID: candidate.ArtifactType,
-		ChunkArtifacts: []contracts.ChunkArtifacts{
-			{
-				Chunk:      chunking.Chunks[0],
-				Candidates: extraction.Candidates,
-			},
-		},
-	})
-	if err != nil {
-		t.Fatalf("Merge() error = %v, want nil", err)
-	}
-	if len(merge.Candidates) != 1 {
-		t.Fatalf("len(merge.Candidates) = %d, want 1", len(merge.Candidates))
-	}
-
-	normalize, err := normalizer.Normalize(ctx, contracts.NormalizeRequest{
-		Source:     doc,
-		LaneID:     candidate.ArtifactType,
-		Candidates: merge.Candidates,
-	})
-	if err != nil {
-		t.Fatalf("Normalize() error = %v, want nil", err)
-	}
-	if len(normalize.Candidates) != 1 {
-		t.Fatalf("len(normalize.Candidates) = %d, want 1", len(normalize.Candidates))
-	}
-
-	validation, err := validator.Validate(ctx, contracts.ValidationRequest{
-		Source:     doc,
-		Candidates: normalize.Candidates,
-	})
-	if err != nil {
-		t.Fatalf("Validate() error = %v, want nil", err)
-	}
-	if len(validation.Decisions) != 1 {
-		t.Fatalf("len(Decisions) = %d, want 1", len(validation.Decisions))
-	}
-
-	decision := validation.Decisions[0]
-	if !decision.Approved {
-		t.Fatal("Approved = false, want true")
-	}
-	if decision.CandidateIndex != candidate.Index {
-		t.Fatalf("CandidateIndex = %d, want %d", decision.CandidateIndex, candidate.Index)
-	}
-
-	output, err := encoder.Encode(ctx, contracts.OutputRequest{
-		Manifest: artifacts.RunManifest{RunID: "run-1"},
-		Approved: []artifacts.Artifact{
-			artifacts.ArtifactFromCandidate(normalize.Candidates[0]),
-		},
-	})
-	if err != nil {
-		t.Fatalf("Encode() error = %v, want nil", err)
-	}
-	if len(output.Files) != 1 {
-		t.Fatalf("len(Files) = %d, want 1", len(output.Files))
-	}
-	if output.Files[0].ContentType != "application/json" {
-		t.Fatalf("ContentType = %q, want application/json", output.Files[0].ContentType)
-	}
-	if len(output.Files[0].Bytes) == 0 {
-		t.Fatal("len(Bytes) = 0, want encoded bytes")
-	}
-}
-
-type compositionAdapter struct{}
-
-func (adapter compositionAdapter) Key() string {
-	return "generic-input"
-}
-
-func (adapter compositionAdapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
-	return &source.SourceDocument{
-		ID:     req.SourceID,
-		Kind:   "document",
-		Format: "text/plain",
-		Digest: "sha256:abc123",
-		Units: []source.SourceUnit{
-			{ID: "u1", Kind: "unit", Text: "First source unit."},
-			{ID: "u2", Kind: "unit", Text: "Second source unit."},
-		},
-	}, nil
-}
-
-type compositionChunker struct{}
-
-func (chunker compositionChunker) Key() string {
-	return "generic-chunker"
-}
-
-func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
-	if req.Source == nil {
-		return contracts.ChunkResult{}, errors.New("source document is required")
-	}
-
-	return contracts.ChunkResult{
-		Chunks: []contracts.SourceChunk{
-			{
-				ID:       req.Source.ID + ":chunk:0",
-				SourceID: req.Source.ID,
-				Index:    0,
-				Units:    append([]source.SourceUnit(nil), req.Source.Units...),
-				Metadata: map[string]any{"strategy": "whole-document"},
-			},
-		},
-	}, nil
-}
-
-type compositionExtractor struct{}
-
-func (extractor compositionExtractor) Key() string {
-	return "generic-extractor"
-}
-
-func (extractor compositionExtractor) ArtifactType() string {
-	return "generic-artifact"
-}
-
-func (extractor compositionExtractor) SchemaVersion() string {
-	return "v1"
-}
-
-func (extractor compositionExtractor) Validators() []contracts.Validator {
-	return []contracts.Validator{compositionValidator{}}
-}
-
-func (extractor compositionExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
-	if req.Source == nil {
-		return contracts.ExtractionResult{}, errors.New("source document is required")
-	}
-	units := req.Source.Units
-	if req.Chunk != nil {
-		units = req.Chunk.Units
-	}
-	if req.AmbientContext["synopsis"] == "" {
-		return contracts.ExtractionResult{}, errors.New("ambient synopsis is required")
-	}
-
-	return contracts.ExtractionResult{
-		Candidates: []artifacts.ArtifactCandidate{
-			{
-				Index:         0,
-				ExtractorKey:  extractor.Key(),
-				ArtifactType:  extractor.ArtifactType(),
-				SchemaVersion: extractor.SchemaVersion(),
-				Payload:       json.RawMessage(`{"value":"example"}`),
-				SourceRefs: []source.SourceRef{
-					{
-						SourceID:    req.Source.ID,
-						StartUnitID: units[0].ID,
-						EndUnitID:   units[len(units)-1].ID,
-					},
-				},
-			},
-		},
-	}, nil
-}
-
-type compositionMerger struct{}
-
-func (merger compositionMerger) Key() string {
-	return "generic-merger"
-}
-
-func (merger compositionMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
-	var candidates []artifacts.ArtifactCandidate
-	for _, chunkArtifacts := range req.ChunkArtifacts {
-		candidates = append(candidates, chunkArtifacts.Candidates...)
-	}
-
-	return contracts.MergeResult{Candidates: candidates}, nil
-}
-
-type compositionNormalizer struct{}
-
-func (normalizer compositionNormalizer) Key() string {
-	return "generic-normalizer"
-}
-
-func (normalizer compositionNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
-	return contracts.NormalizeResult{Candidates: req.Candidates}, nil
-}
-
-type compositionValidator struct{}
-
-func (validator compositionValidator) Name() string {
-	return "generic-validator"
-}
-
-func (validator compositionValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
-	decisions := make([]contracts.ValidationDecision, 0, len(req.Candidates))
-	for _, candidate := range req.Candidates {
-		decisions = append(decisions, contracts.ValidationDecision{
-			CandidateIndex: candidate.Index,
-			Approved:       true,
-			ReasonCode:     "accepted",
-			Message:        "candidate accepted",
-		})
-	}
-
-	return contracts.ValidationResult{
-		ValidatorName: validator.Name(),
-		Decisions:     decisions,
-	}, nil
-}
-
-type compositionOutputEncoder struct{}
-
-func (encoder compositionOutputEncoder) Key() string {
-	return "generic-output"
-}
-
-func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
-	payload := struct {
-		RunID         string `json:"run_id"`
-		ApprovedCount int    `json:"approved_count"`
-	}{
-		RunID:         req.Manifest.RunID,
-		ApprovedCount: len(req.Approved),
-	}
-	encoded, err := json.Marshal(payload)
-	if err != nil {
-		return contracts.OutputResult{}, err
-	}
-
-	return contracts.OutputResult{
-		Files: []contracts.OutputFile{
-			{
-				Name:        "artifacts/generic.json",
-				ContentType: "application/json",
-				Bytes:       encoded,
-			},
-		},
-	}, nil
-}

internal/framework/contracts/contracts.go

@@ -8,38 +8,127 @@ import (
 	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
 )
 
-type LLMMessage struct {
-	Role    string `json:"role"`
-	Content string `json:"content"`
-}
-
 type StructuredCompletionRequest struct {
-	StageName          string          `json:"stage_name"`
+	StageName     string         `json:"stage_name"`
-	Messages           []LLMMessage    `json:"messages"`
+	PromptID      string         `json:"prompt_id,omitempty"`
-	Model              string          `json:"model,omitempty"`
+	PromptVersion string         `json:"prompt_version,omitempty"`
-	ResponseSchemaName string          `json:"response_schema_name,omitempty"`
+	ProfileID     string         `json:"profile_id,omitempty"`
-	ResponseSchema     json.RawMessage `json:"response_schema,omitempty"`
+	SessionID     string         `json:"session_id,omitempty"`
+	Inputs        LLMInputSet    `json:"inputs,omitempty"`
+	Vars          map[string]any `json:"vars,omitempty"`
 }
 
 type StructuredCompletionResponse struct {
-	Content          json.RawMessage `json:"content"`
+	Content          json.RawMessage   `json:"content"`
-	Provider         string          `json:"provider,omitempty"`
+	Provider         string            `json:"provider,omitempty"`
-	Model            string          `json:"model,omitempty"`
+	Model            string            `json:"model,omitempty"`
-	PromptTokens     int             `json:"prompt_tokens,omitempty"`
+	ProfileID        string            `json:"profile_id,omitempty"`
-	CompletionTokens int             `json:"completion_tokens,omitempty"`
+	PromptTokens     int               `json:"prompt_tokens,omitempty"`
-	TotalTokens      int             `json:"total_tokens,omitempty"`
+	CompletionTokens int               `json:"completion_tokens,omitempty"`
+	TotalTokens      int               `json:"total_tokens,omitempty"`
+	Debug            *LLMDebugMaterial `json:"debug,omitempty"`
 }
 
 type StructuredLLMClient interface {
 	CompleteStructured(ctx context.Context, req StructuredCompletionRequest, out any) (StructuredCompletionResponse, error)
 }
 
+type LLMDebugMaterial struct {
+	Prompt   *LLMDebugPrompt   `json:"prompt,omitempty"`
+	Response *LLMDebugResponse `json:"response,omitempty"`
+}
+
+type LLMDebugPrompt struct {
+	PromptID             string            `json:"prompt_id,omitempty"`
+	PromptVersion        string            `json:"prompt_version,omitempty"`
+	PromptHash           string            `json:"prompt_hash,omitempty"`
+	SelectedProfileID    string            `json:"selected_profile_id,omitempty"`
+	SessionID            string            `json:"session_id,omitempty"`
+	RenderedPromptHash   string            `json:"rendered_prompt_hash,omitempty"`
+	Messages             []LLMDebugMessage `json:"messages,omitempty"`
+	EffectiveModelParams map[string]any    `json:"effective_model_params,omitempty"`
+	OutputContract       map[string]any    `json:"output_contract,omitempty"`
+	StructuredOutput     map[string]any    `json:"structured_output,omitempty"`
+	InputHashes          map[string]string `json:"input_hashes,omitempty"`
+}
+
+type LLMDebugMessage struct {
+	Role         string         `json:"role"`
+	Content      string         `json:"content"`
+	CacheControl map[string]any `json:"cache_control,omitempty"`
+}
+
+type LLMDebugResponse struct {
+	Content              string            `json:"content,omitempty"`
+	RunID                string            `json:"run_id,omitempty"`
+	PromptID             string            `json:"prompt_id,omitempty"`
+	PromptVersion        string            `json:"prompt_version,omitempty"`
+	PromptHash           string            `json:"prompt_hash,omitempty"`
+	RenderedPromptHash   string            `json:"rendered_prompt_hash,omitempty"`
+	SelectedProfileID    string            `json:"selected_profile_id,omitempty"`
+	ModelName            string            `json:"model_name,omitempty"`
+	Endpoint             string            `json:"endpoint,omitempty"`
+	EffectiveModelParams map[string]any    `json:"effective_model_params,omitempty"`
+	InputHashes          map[string]string `json:"input_hashes,omitempty"`
+	Validation           map[string]any    `json:"validation,omitempty"`
+	Usage                LLMDebugUsage     `json:"usage,omitempty"`
+}
+
+type LLMDebugUsage struct {
+	PromptTokens     int `json:"prompt_tokens,omitempty"`
+	CompletionTokens int `json:"completion_tokens,omitempty"`
+	TotalTokens      int `json:"total_tokens,omitempty"`
+	CachedTokens     int `json:"cached_tokens,omitempty"`
+	CacheWriteTokens int `json:"cache_write_tokens,omitempty"`
+}
+
+type LLMProfileManifestProvider interface {
+	LLMProfileManifests() []artifacts.LLMProfileManifest
+}
+
+type LLMInputMaterial struct {
+	Name      string `json:"name"`
+	MediaType string `json:"media_type,omitempty"`
+	Content   []byte `json:"-"`
+	Digest    string `json:"digest,omitempty"`
+	OriginURI string `json:"origin_uri,omitempty"`
+	SizeBytes int64  `json:"size_bytes,omitempty"`
+}
+
+func NewLLMInputMaterial(name string, mediaType string, content []byte, digest string, originURI string) LLMInputMaterial {
+	return LLMInputMaterial{
+		Name:      name,
+		MediaType: mediaType,
+		Content:   append([]byte(nil), content...),
+		Digest:    digest,
+		OriginURI: originURI,
+		SizeBytes: int64(len(content)),
+	}
+}
+
+func (material LLMInputMaterial) Clone() LLMInputMaterial {
+	material.Content = append([]byte(nil), material.Content...)
+	return material
+}
+
+type LLMInputSet map[string]LLMInputMaterial
+
+func (set LLMInputSet) Clone() LLMInputSet {
+	if len(set) == 0 {
+		return nil
+	}
+	out := make(LLMInputSet, len(set))
+	for key, material := range set {
+		out[key] = material.Clone()
+	}
+	return out
+}
+
 type ParseRequest struct {
 	SourceID   string         `json:"source_id,omitempty"`
 	Path       string         `json:"path,omitempty"`
 	Raw        []byte         `json:"-"`
 	LLMProfile string         `json:"llm_profile,omitempty"`
-	Options    map[string]any `json:"options,omitempty"`
 	Metadata   map[string]any `json:"metadata,omitempty"`
 }
 
@@ -48,122 +137,93 @@ type InputAdapter interface {
 	Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
 }
 
-type SourceChunk struct {
-	ID       string              `json:"id"`
-	SourceID string              `json:"source_id"`
-	Index    int                 `json:"index"`
-	Units    []source.SourceUnit `json:"units"`
-	Metadata map[string]any      `json:"metadata,omitempty"`
-}
-
 type ChunkRequest struct {
-	Source     *source.SourceDocument `json:"-"`
+	Source      *source.SourceDocument `json:"-"`
-	LLMProfile string                 `json:"llm_profile,omitempty"`
+	SourceInput LLMInputMaterial       `json:"source_input,omitempty"`
-	Options    map[string]any         `json:"options,omitempty"`
+	SessionID   string                 `json:"session_id,omitempty"`
-	Metadata   map[string]any         `json:"metadata,omitempty"`
+	References  ReferenceSet           `json:"references,omitempty"`
+	LLMProfile  string                 `json:"llm_profile,omitempty"`
+	Metadata    map[string]any         `json:"metadata,omitempty"`
 }
 
-type ChunkResult struct {
+type ChunkPlanResult struct {
-	Chunks   []SourceChunk `json:"chunks"`
+	Plan     source.ChunkPlan `json:"plan"`
-	Warnings []Warning     `json:"warnings,omitempty"`
+	Warnings []Warning        `json:"warnings,omitempty"`
 }
 
 type Chunker interface {
 	Key() string
-	Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error)
+	ReferenceSlots() []ReferenceSlot
+	Plan(ctx context.Context, req ChunkRequest) (ChunkPlanResult, error)
 }
 
-type ExtractionRequest struct {
+const (
-	Source         *source.SourceDocument `json:"-"`
+	ReferenceBindingSourceConfig = "config"
-	Chunk          *SourceChunk           `json:"chunk,omitempty"`
+	ReferenceBindingSourceCLI    = "cli"
-	AmbientContext map[string]any         `json:"ambient_context,omitempty"`
+)
-	LLMClient      StructuredLLMClient    `json:"-"`
+
-	LLMProfile     string                 `json:"llm_profile,omitempty"`
+type ReferenceSlot struct {
-	Options        map[string]any         `json:"options,omitempty"`
+	Name               string   `json:"name"`
-	Metadata       map[string]any         `json:"metadata,omitempty"`
+	Description        string   `json:"description,omitempty"`
+	Required           bool     `json:"required,omitempty"`
+	AcceptedMediaTypes []string `json:"accepted_media_types,omitempty"`
+	Multiple           bool     `json:"multiple,omitempty"`
+	MaxBytes           int64    `json:"max_bytes,omitempty"`
 }
 
-type ExtractionResult struct {
+func CloneReferenceSlots(slots []ReferenceSlot) []ReferenceSlot {
-	Candidates []artifacts.ArtifactCandidate `json:"candidates,omitempty"`
+	if len(slots) == 0 {
-	Warnings   []Warning                     `json:"warnings,omitempty"`
+		return nil
+	}
+	out := make([]ReferenceSlot, len(slots))
+	for i, slot := range slots {
+		slot.AcceptedMediaTypes = append([]string(nil), slot.AcceptedMediaTypes...)
+		out[i] = slot
+	}
+	return out
 }
 
-type Extractor interface {
+type ReferenceOrigin struct {
-	Key() string
+	Type string `json:"type"`
-	ArtifactType() string
+	URI  string `json:"uri,omitempty"`
-	SchemaVersion() string
-	Validators() []Validator
-	Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
 }
 
-type ChunkArtifacts struct {
+type ReferenceItem struct {
-	Chunk      SourceChunk                   `json:"chunk"`
+	SlotName      string          `json:"slot_name"`
-	Candidates []artifacts.ArtifactCandidate `json:"candidates"`
+	MediaType     string          `json:"media_type,omitempty"`
+	Content       []byte          `json:"-"`
+	Digest        string          `json:"digest,omitempty"`
+	Origin        ReferenceOrigin `json:"origin"`
+	SizeBytes     int64           `json:"size_bytes,omitempty"`
+	BindingSource string          `json:"binding_source,omitempty"`
 }
 
-type MergeRequest struct {
+type ResolvedReferenceSlot struct {
-	Source         *source.SourceDocument `json:"-"`
+	Slot  ReferenceSlot   `json:"slot"`
-	LaneID         string                 `json:"lane_id"`
+	Items []ReferenceItem `json:"items,omitempty"`
-	ChunkArtifacts []ChunkArtifacts       `json:"chunk_artifacts"`
-	LLMProfile     string                 `json:"llm_profile,omitempty"`
-	Options        map[string]any         `json:"options,omitempty"`
-	Metadata       map[string]any         `json:"metadata,omitempty"`
 }
 
-type MergeResult struct {
+type ReferenceSet struct {
-	Candidates []artifacts.ArtifactCandidate `json:"candidates"`
+	Slots map[string]ResolvedReferenceSlot `json:"slots,omitempty"`
-	Warnings   []Warning                     `json:"warnings,omitempty"`
 }
 
-type Merger interface {
+type ExecutionClass string
-	Key() string
-	Merge(ctx context.Context, req MergeRequest) (MergeResult, error)
-}
 
-type NormalizeRequest struct {
+const (
-	Source     *source.SourceDocument        `json:"-"`
+	ExecutionClassDeterministic ExecutionClass = "deterministic"
-	LaneID     string                        `json:"lane_id"`
+	ExecutionClassLLMBacked     ExecutionClass = "llm_backed"
-	Candidates []artifacts.ArtifactCandidate `json:"candidates"`
+)
-	LLMProfile string                        `json:"llm_profile,omitempty"`
-	Options    map[string]any                `json:"options,omitempty"`
-	Metadata   map[string]any                `json:"metadata,omitempty"`
-}
 
-type NormalizeResult struct {
+type ChunkExecutionClassProvider interface {
-	Candidates []artifacts.ArtifactCandidate `json:"candidates"`
+	ExecutionClass() ExecutionClass
-	Warnings   []Warning                     `json:"warnings,omitempty"`
-}
-
-type Normalizer interface {
-	Key() string
-	Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
-}
-
-type ValidationRequest struct {
-	Source     *source.SourceDocument        `json:"-"`
-	Candidates []artifacts.ArtifactCandidate `json:"candidates"`
-	LLMProfile string                        `json:"llm_profile,omitempty"`
-	Options    map[string]any                `json:"options,omitempty"`
-	Metadata   map[string]any                `json:"metadata,omitempty"`
-}
-
-type ValidationDecision struct {
-	CandidateIndex         int    `json:"candidate_index"`
-	Approved               bool   `json:"approved"`
-	ReasonCode             string `json:"reason_code"`
-	Message                string `json:"message"`
-	DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
 }
 
 type ValidationResult struct {
-	ValidatorName string               `json:"validator_name"`
+	Approved               bool      `json:"approved"`
-	Decisions     []ValidationDecision `json:"decisions"`
+	ReasonCode             string    `json:"reason_code,omitempty"`
-	Warnings      []Warning            `json:"warnings,omitempty"`
+	Message                string    `json:"message,omitempty"`
-}
+	DiagnosticArtifactPath string    `json:"diagnostic_artifact_path,omitempty"`
-
+	Warnings               []Warning `json:"warnings,omitempty"`
-type Validator interface {
-	Name() string
-	Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error)
 }
 
 type Warning struct {
@@ -173,13 +233,12 @@ type Warning struct {
 }
 
 type OutputRequest struct {
-	Manifest   artifacts.RunManifest        `json:"manifest"`
+	Manifest         artifacts.RunManifest `json:"manifest"`
-	Approved   []artifacts.Artifact         `json:"approved,omitempty"`
+	NormalizeOutputs []SerializedOutput    `json:"normalize_outputs,omitempty"`
-	Rejected   []artifacts.RejectedArtifact `json:"rejected,omitempty"`
+	Rejected         []RejectedOutput      `json:"rejected,omitempty"`
-	Warnings   []Warning                    `json:"warnings,omitempty"`
+	Warnings         []Warning             `json:"warnings,omitempty"`
-	LLMProfile string                       `json:"llm_profile,omitempty"`
+	LLMProfile       string                `json:"llm_profile,omitempty"`
-	Options    map[string]any               `json:"options,omitempty"`
+	Metadata         map[string]any        `json:"metadata,omitempty"`
-	Metadata   map[string]any               `json:"metadata,omitempty"`
 }
 
 type OutputFile struct {
@@ -198,6 +257,19 @@ type OutputEncoder interface {
 	Encode(ctx context.Context, req OutputRequest) (OutputResult, error)
 }
 
+type RejectedOutput struct {
+	Stage                  string `json:"stage"`
+	LaneID                 string `json:"lane_id,omitempty"`
+	ModuleKey              string `json:"module_key,omitempty"`
+	ChunkID                string `json:"chunk_id,omitempty"`
+	ChunkIndex             int    `json:"chunk_index,omitempty"`
+	ValidatorName          string `json:"validator_name,omitempty"`
+	ReasonCode             string `json:"reason_code,omitempty"`
+	Message                string `json:"message"`
+	AttemptCount           int    `json:"attempt_count,omitempty"`
+	DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
+}
+
 type ManifestMetadataProvider interface {
 	ManifestMetadata() map[string]any
 }

internal/framework/contracts/contracts_test.go

@@ -3,6 +3,7 @@ package contracts
 import (
 	"context"
 	"encoding/json"
+	"reflect"
 	"testing"
 
 	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
@@ -11,20 +12,15 @@ import (
 
 var _ InputAdapter = fakeAdapter{}
 var _ Chunker = fakeChunker{}
-var _ Extractor = fakeExtractor{}
+var _ Extractor[fakeArtifact] = fakeExtractor{}
-var _ Merger = fakeMerger{}
+var _ Merger[fakeArtifact] = fakeMerger{}
-var _ Normalizer = fakeNormalizer{}
+var _ Normalizer[fakeArtifact] = fakeNormalizer{}
-var _ Validator = fakeValidator{}
 var _ StructuredLLMClient = fakeLLMClient{}
 var _ OutputEncoder = fakeOutputEncoder{}
 
-func TestFakeExtractorReturnsCandidateAndValidator(t *testing.T) {
+func TestFakeExtractorReturnsTypedOutput(t *testing.T) {
-	validator := fakeValidator{name: "generic-validator"}
 	extractor := fakeExtractor{
-		key:           "generic-extractor",
+		key: "generic-extractor",
-		artifactType:  "generic-artifact",
-		schemaVersion: "v1",
-		validators:    []Validator{validator},
 	}
 	doc := &source.SourceDocument{
 		ID:     "source-1",
@@ -32,11 +28,11 @@ func TestFakeExtractorReturnsCandidateAndValidator(t *testing.T) {
 		Format: "text/plain",
 		Digest: "sha256:abc123",
 		Units: []source.SourceUnit{
-			{ID: "u1", Kind: "section", Text: "Source text."},
+			{ID: 1, Kind: "section", Text: "Source text."},
 		},
 	}
 
-	result, err := extractor.Extract(context.Background(), ExtractionRequest{Source: doc})
+	result, err := extractor.Extract(context.Background(), TypedExtractionRequest{Source: doc})
 	if err != nil {
 		t.Fatalf("Extract() error = %v, want nil", err)
 	}
@@ -44,103 +40,82 @@ func TestFakeExtractorReturnsCandidateAndValidator(t *testing.T) {
 	if extractor.Key() != "generic-extractor" {
 		t.Fatalf("Key() = %q, want generic-extractor", extractor.Key())
 	}
-	if extractor.ArtifactType() != "generic-artifact" {
+	if result.Value.Value != "example" {
-		t.Fatalf("ArtifactType() = %q, want generic-artifact", extractor.ArtifactType())
+		t.Fatalf("Value = %q, want example", result.Value.Value)
-	}
-	if extractor.SchemaVersion() != "v1" {
-		t.Fatalf("SchemaVersion() = %q, want v1", extractor.SchemaVersion())
-	}
-	if len(extractor.Validators()) != 1 {
-		t.Fatalf("len(Validators()) = %d, want 1", len(extractor.Validators()))
-	}
-	if extractor.Validators()[0].Name() != "generic-validator" {
-		t.Fatalf("Validators()[0].Name() = %q, want generic-validator", extractor.Validators()[0].Name())
-	}
-	if len(result.Candidates) != 1 {
-		t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
-	}
-
-	candidate := result.Candidates[0]
-	if candidate.Index != 0 {
-		t.Fatalf("ArtifactCandidate.Index = %d, want 0", candidate.Index)
-	}
-	if candidate.ExtractorKey != extractor.Key() {
-		t.Fatalf("ArtifactCandidate.ExtractorKey = %q, want %q", candidate.ExtractorKey, extractor.Key())
-	}
-	if candidate.ArtifactType != extractor.ArtifactType() {
-		t.Fatalf("ArtifactCandidate.ArtifactType = %q, want %q", candidate.ArtifactType, extractor.ArtifactType())
-	}
-	if candidate.SchemaVersion != extractor.SchemaVersion() {
-		t.Fatalf("ArtifactCandidate.SchemaVersion = %q, want %q", candidate.SchemaVersion, extractor.SchemaVersion())
-	}
-	if string(candidate.Payload) != `{"value":"example"}` {
-		t.Fatalf("ArtifactCandidate.Payload = %s, want example payload", candidate.Payload)
 	}
 }
 
-func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
+func TestFakeChunkerReturnsSourcePlan(t *testing.T) {
 	doc := &source.SourceDocument{
 		ID:     "source-1",
 		Kind:   "document",
 		Format: "text/plain",
 		Digest: "sha256:abc123",
 		Units: []source.SourceUnit{
-			{ID: "u1", Kind: "section", Text: "Source text."},
+			{ID: 1, Kind: "section", Text: "Source text."},
 		},
 	}
 	chunker := fakeChunker{key: "generic-chunker"}
 
-	result, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc})
+	result, err := chunker.Plan(context.Background(), ChunkRequest{Source: doc})
 	if err != nil {
-		t.Fatalf("Chunk() error = %v, want nil", err)
+		t.Fatalf("Plan() error = %v, want nil", err)
 	}
 
 	if chunker.Key() != "generic-chunker" {
 		t.Fatalf("Key() = %q, want generic-chunker", chunker.Key())
 	}
-	if len(result.Chunks) != 1 {
+	if len(result.Plan.Ranges) != 1 {
-		t.Fatalf("len(Chunks) = %d, want 1", len(result.Chunks))
+		t.Fatalf("len(Ranges) = %d, want 1", len(result.Plan.Ranges))
 	}
-
+	if result.Plan.SourceDigest != doc.Digest || result.Plan.Ranges[0].StartUnitID != 1 || result.Plan.Ranges[0].EndUnitID != 1 {
-	chunk := result.Chunks[0]
+		t.Fatalf("Plan = %#v, want source digest and unit range", result.Plan)
-	if chunk.ID != "source-1:chunk:0" {
-		t.Fatalf("SourceChunk.ID = %q, want source-1:chunk:0", chunk.ID)
-	}
-	if chunk.SourceID != doc.ID {
-		t.Fatalf("SourceChunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
-	}
-	if chunk.Index != 0 {
-		t.Fatalf("SourceChunk.Index = %d, want 0", chunk.Index)
-	}
-	if len(chunk.Units) != 1 {
-		t.Fatalf("len(SourceChunk.Units) = %d, want 1", len(chunk.Units))
 	}
 }
 
-func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
+func TestFakeChunkerReceivesPerRunContext(t *testing.T) {
-	extractor := fakeExtractor{
-		key:           "generic-extractor",
-		artifactType:  "generic-artifact",
-		schemaVersion: "v1",
-	}
 	doc := &source.SourceDocument{
 		ID:     "source-1",
 		Kind:   "document",
 		Format: "text/plain",
 		Digest: "sha256:abc123",
 		Units: []source.SourceUnit{
-			{ID: "u1", Kind: "section", Text: "First source text."},
+			{ID: 1, Kind: "section", Text: "Source text."},
-			{ID: "u2", Kind: "section", Text: "Second source text."},
 		},
 	}
-	chunk := SourceChunk{
+	chunker := &recordingChunker{key: "llm-chunker"}
-		ID:       "source-1:chunk:1",
+
-		SourceID: doc.ID,
+	if _, err := chunker.Plan(context.Background(), ChunkRequest{Source: doc, SessionID: "session", LLMProfile: "profile"}); err != nil {
-		Index:    1,
+		t.Fatalf("Plan() error = %v, want nil", err)
-		Units:    []source.SourceUnit{doc.Units[1]},
+	}
+	if chunker.request.SessionID != "session" || chunker.request.LLMProfile != "profile" {
+		t.Fatalf("ChunkRequest = %#v, want per-run session and profile", chunker.request)
+	}
+}
+
+func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
+	extractor := fakeExtractor{key: "generic-extractor"}
+	doc := &source.SourceDocument{
+		ID:     "source-1",
+		Kind:   "document",
+		Format: "text/plain",
+		Digest: "sha256:abc123",
+		Units: []source.SourceUnit{
+			{ID: 1, Kind: "section", Text: "First source text."},
+			{ID: 2, Kind: "section", Text: "Second source text."},
+		},
+	}
+	chunk := source.Chunk{
+		ID:        "source-1:chunk:1",
+		SourceID:  doc.ID,
+		Index:     1,
+		Ref:       source.SourceRef{SourceID: doc.ID, StartUnitID: 2, EndUnitID: 2},
+		Content:   []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
+		MediaType: "application/json",
+		Units:     []source.SourceUnit{doc.Units[1]},
 	}
 
-	result, err := extractor.Extract(context.Background(), ExtractionRequest{
+	result, err := extractor.Extract(context.Background(), TypedExtractionRequest{
 		Source:         doc,
 		Chunk:          &chunk,
 		AmbientContext: map[string]any{"mode": "chunked"},
@@ -148,51 +123,211 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
 	if err != nil {
 		t.Fatalf("Extract() error = %v, want nil", err)
 	}
-	if len(result.Candidates) != 1 {
+	if result.Value.Value != "chunked" {
-		t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
+		t.Fatalf("Value = %q, want chunked", result.Value.Value)
+	}
+}
+
+func TestReferenceSetDataTypes(t *testing.T) {
+	references := ReferenceSet{
+		Slots: map[string]ResolvedReferenceSlot{
+			"roster": {
+				Slot: ReferenceSlot{
+					Name:               "roster",
+					Description:        "Known characters",
+					Required:           true,
+					AcceptedMediaTypes: []string{"text/plain"},
+					Multiple:           true,
+					MaxBytes:           4096,
+				},
+				Items: []ReferenceItem{
+					{
+						SlotName:  "roster",
+						MediaType: "text/plain",
+						Content:   []byte("Aria\nBryn\n"),
+						Digest:    "sha256:reference",
+						Origin: ReferenceOrigin{
+							Type: "file",
+							URI:  "file:///tmp/roster.txt",
+						},
+						SizeBytes:     10,
+						BindingSource: ReferenceBindingSourceConfig,
+					},
+				},
+			},
+		},
 	}
 
-	candidate := result.Candidates[0]
+	item := references.Slots["roster"].Items[0]
-	if string(candidate.Payload) != `{"value":"chunked"}` {
+	if item.SlotName != "roster" || item.MediaType != "text/plain" || string(item.Content) != "Aria\nBryn\n" {
-		t.Fatalf("ArtifactCandidate.Payload = %s, want chunked payload", candidate.Payload)
+		t.Fatalf("reference item = %#v, want constructed item fields", item)
 	}
-	if len(candidate.SourceRefs) != 1 {
+	if item.BindingSource != ReferenceBindingSourceConfig {
-		t.Fatalf("len(SourceRefs) = %d, want 1", len(candidate.SourceRefs))
+		t.Fatalf("BindingSource = %q, want %q", item.BindingSource, ReferenceBindingSourceConfig)
 	}
-	ref := candidate.SourceRefs[0]
+}
-	if ref.StartUnitID != "u2" || ref.EndUnitID != "u2" {
+
-		t.Fatalf("SourceRef = %+v, want u2 range", ref)
+func TestCloneReferenceSlotsEmptyInputReturnsNil(t *testing.T) {
+	if got := CloneReferenceSlots(nil); got != nil {
+		t.Fatalf("CloneReferenceSlots(nil) = %#v, want nil", got)
+	}
+	if got := CloneReferenceSlots([]ReferenceSlot{}); got != nil {
+		t.Fatalf("CloneReferenceSlots(empty) = %#v, want nil", got)
+	}
+}
+
+func TestCloneReferenceSlotsPreservesFields(t *testing.T) {
+	slots := []ReferenceSlot{
+		{
+			Name:               "roster",
+			Description:        "Known characters",
+			Required:           true,
+			AcceptedMediaTypes: []string{"text/plain", "text/markdown"},
+			Multiple:           true,
+			MaxBytes:           4096,
+		},
+		{
+			Name:        "glossary",
+			Description: "Campaign terms",
+			MaxBytes:    2048,
+		},
+	}
+
+	got := CloneReferenceSlots(slots)
+
+	if !reflect.DeepEqual(got, slots) {
+		t.Fatalf("CloneReferenceSlots() = %#v, want %#v", got, slots)
+	}
+}
+
+func TestCloneReferenceSlotsCopiesAcceptedMediaTypes(t *testing.T) {
+	slots := []ReferenceSlot{
+		{
+			Name:               "party",
+			AcceptedMediaTypes: []string{"application/json", "text/plain"},
+		},
+	}
+
+	got := CloneReferenceSlots(slots)
+	got[0].Name = "changed"
+	got[0].AcceptedMediaTypes[0] = "text/markdown"
+
+	if slots[0].Name != "party" {
+		t.Fatalf("source slot name = %q, want unchanged", slots[0].Name)
+	}
+	if slots[0].AcceptedMediaTypes[0] != "application/json" {
+		t.Fatalf("source AcceptedMediaTypes aliased clone: %#v", slots[0].AcceptedMediaTypes)
+	}
+}
+
+func TestReferenceItemJSONOmitsContent(t *testing.T) {
+	item := ReferenceItem{
+		SlotName:  "roster",
+		MediaType: "text/plain",
+		Content:   []byte("reference content"),
+		Digest:    "sha256:reference",
+		Origin:    ReferenceOrigin{Type: "file", URI: "file:///tmp/roster.txt"},
+	}
+
+	encoded, err := json.Marshal(item)
+	if err != nil {
+		t.Fatalf("json.Marshal() error = %v, want nil", err)
+	}
+
+	var got map[string]any
+	if err := json.Unmarshal(encoded, &got); err != nil {
+		t.Fatalf("json.Unmarshal() error = %v, want nil", err)
+	}
+	if _, ok := got["content"]; ok {
+		t.Fatalf("encoded reference item leaked content: %s", encoded)
+	}
+	if _, ok := got["Content"]; ok {
+		t.Fatalf("encoded reference item leaked Content: %s", encoded)
+	}
+}
+
+func TestLLMInputMaterialCopiesContentAndOmitsContentFromJSON(t *testing.T) {
+	content := []byte("raw source bytes")
+	material := NewLLMInputMaterial("transcript", "application/json", content, "sha256:source", "file:///tmp/source.json")
+	content[0] = 'R'
+	if got := string(material.Content); got != "raw source bytes" {
+		t.Fatalf("material content = %q, want defensive copy", got)
+	}
+	if material.SizeBytes != int64(len("raw source bytes")) {
+		t.Fatalf("SizeBytes = %d, want content length", material.SizeBytes)
+	}
+
+	clone := material.Clone()
+	clone.Content[0] = 'X'
+	if got := string(material.Content); got != "raw source bytes" {
+		t.Fatalf("cloned material content aliased original: %q", got)
+	}
+
+	encoded, err := json.Marshal(material)
+	if err != nil {
+		t.Fatalf("json.Marshal() error = %v, want nil", err)
+	}
+	var got map[string]any
+	if err := json.Unmarshal(encoded, &got); err != nil {
+		t.Fatalf("json.Unmarshal() error = %v, want nil", err)
+	}
+	if _, ok := got["content"]; ok {
+		t.Fatalf("encoded material leaked content: %s", encoded)
+	}
+	if _, ok := got["Content"]; ok {
+		t.Fatalf("encoded material leaked Content: %s", encoded)
+	}
+	if got["digest"] != "sha256:source" || got["origin_uri"] != "file:///tmp/source.json" {
+		t.Fatalf("encoded material = %#v, want non-secret provenance", got)
+	}
+}
+
+func TestLLMInputSetCloneCopiesContent(t *testing.T) {
+	set := LLMInputSet{
+		"transcript": NewLLMInputMaterial("transcript", "application/json", []byte("source"), "sha256:source", "file:///tmp/source.json"),
+	}
+	clone := set.Clone()
+	clone["transcript"].Content[0] = 'S'
+	if got := string(set["transcript"].Content); got != "source" {
+		t.Fatalf("input set clone aliased content: %q", got)
+	}
+}
+
+func TestArtifactSchemaJSONOmitsSchemaContent(t *testing.T) {
+	schema := ArtifactSchema{
+		ID:         "schema-id",
+		Name:       "schema-name",
+		Version:    "v1",
+		JSONSchema: []byte(`{"type":"object"}`),
+	}
+	encoded, err := json.Marshal(schema)
+	if err != nil {
+		t.Fatalf("json.Marshal() error = %v, want nil", err)
+	}
+	var got map[string]any
+	if err := json.Unmarshal(encoded, &got); err != nil {
+		t.Fatalf("json.Unmarshal() error = %v, want nil", err)
+	}
+	if got["id"] != "schema-id" || got["name"] != "schema-name" || got["version"] != "v1" {
+		t.Fatalf("encoded schema = %#v, want schema provenance", got)
+	}
+	if _, ok := got["json_schema"]; ok {
+		t.Fatalf("encoded schema leaked raw schema content: %s", encoded)
+	}
+	if _, ok := got["JSONSchema"]; ok {
+		t.Fatalf("encoded schema leaked raw schema content: %s", encoded)
 	}
 }
 
 func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
-	candidate := artifacts.ArtifactCandidate{
+	extractOutput := ExtractArtifact[fakeArtifact]{LaneID: "generic-lane", ExtractorKey: "generic-extractor", SourceID: "source-1", ChunkID: "source-1:chunk:0", ChunkIndex: 0, Value: fakeArtifact{Value: "example"}}
-		Index:         0,
-		ExtractorKey:  "generic-extractor",
-		ArtifactType:  "generic-artifact",
-		SchemaVersion: "v1",
-		Payload:       json.RawMessage(`{"value":"example"}`),
-	}
-	chunk := SourceChunk{
-		ID:       "source-1:chunk:0",
-		SourceID: "source-1",
-		Index:    0,
-		Units: []source.SourceUnit{
-			{ID: "u1", Kind: "section", Text: "Source text."},
-		},
-	}
 	merger := fakeMerger{key: "generic-merger"}
 	normalizer := fakeNormalizer{key: "generic-normalizer"}
 	encoder := fakeOutputEncoder{key: "generic-output"}
 
-	merged, err := merger.Merge(context.Background(), MergeRequest{
+	merged, err := merger.Merge(context.Background(), TypedMergeRequest[fakeArtifact]{
-		LaneID: "generic-artifact",
+		LaneID:         "generic-lane",
-		ChunkArtifacts: []ChunkArtifacts{
+		ExtractOutputs: []ExtractArtifact[fakeArtifact]{extractOutput},
-			{
-				Chunk:      chunk,
-				Candidates: []artifacts.ArtifactCandidate{candidate},
-			},
-		},
 	})
 	if err != nil {
 		t.Fatalf("Merge() error = %v, want nil", err)
@@ -200,13 +335,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
 	if merger.Key() != "generic-merger" {
 		t.Fatalf("Merger.Key() = %q, want generic-merger", merger.Key())
 	}
-	if len(merged.Candidates) != 1 {
+	if merged.Value.Value != "example" {
-		t.Fatalf("len(merged.Candidates) = %d, want 1", len(merged.Candidates))
+		t.Fatalf("merged value = %q, want example", merged.Value.Value)
 	}
 
-	normalized, err := normalizer.Normalize(context.Background(), NormalizeRequest{
+	normalized, err := normalizer.Normalize(context.Background(), TypedNormalizeRequest[fakeArtifact]{
-		LaneID:     "generic-artifact",
+		LaneID:      "generic-lane",
-		Candidates: merged.Candidates,
+		MergeOutput: MergeArtifact[fakeArtifact]{LaneID: "generic-lane", MergerKey: merger.Key(), SourceID: "source-1", Value: merged.Value},
 	})
 	if err != nil {
 		t.Fatalf("Normalize() error = %v, want nil", err)
@@ -214,15 +349,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
 	if normalizer.Key() != "generic-normalizer" {
 		t.Fatalf("Normalizer.Key() = %q, want generic-normalizer", normalizer.Key())
 	}
-	if len(normalized.Candidates) != 1 {
+	if normalized.Value.Value != "example" {
-		t.Fatalf("len(normalized.Candidates) = %d, want 1", len(normalized.Candidates))
+		t.Fatalf("normalized value = %q, want example", normalized.Value.Value)
 	}
 
 	encoded, err := encoder.Encode(context.Background(), OutputRequest{
-		Manifest: artifacts.RunManifest{RunID: "run-1"},
+		Manifest:         artifacts.RunManifest{RunID: "run-1"},
-		Approved: []artifacts.Artifact{
+		NormalizeOutputs: []SerializedOutput{{LaneID: "generic-lane", NormalizerKey: normalizer.Key(), SourceID: "source-1", Artifact: SerializedArtifact{Kind: "test/artifact", Schema: ArtifactSchema{ID: "schema-id", Name: "schema-name", Version: "v1"}, MediaType: "application/json", Content: []byte(`{"value":"example"}`)}}},
-			artifacts.ArtifactFromCandidate(normalized.Candidates[0]),
-		},
 	})
 	if err != nil {
 		t.Fatalf("Encode() error = %v, want nil", err)
@@ -236,7 +369,7 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
 	if encoded.Files[0].ContentType != "application/json" {
 		t.Fatalf("ContentType = %q, want application/json", encoded.Files[0].ContentType)
 	}
-	if string(encoded.Files[0].Bytes) != `{"run_id":"run-1","approved_count":1}` {
+	if string(encoded.Files[0].Bytes) != `{"run_id":"run-1","output_count":1}` {
 		t.Fatalf("Bytes = %s, want encoded output", encoded.Files[0].Bytes)
 	}
 }
@@ -292,70 +425,60 @@ func (chunker fakeChunker) Key() string {
 	return chunker.key
 }
 
-func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error) {
+func (chunker fakeChunker) ReferenceSlots() []ReferenceSlot {
-	return ChunkResult{
+	return nil
-		Chunks: []SourceChunk{
+}
-			{
+
-				ID:       req.Source.ID + ":chunk:0",
+func (chunker fakeChunker) Plan(ctx context.Context, req ChunkRequest) (ChunkPlanResult, error) {
-				SourceID: req.Source.ID,
+	return ChunkPlanResult{
-				Index:    0,
+		Plan: source.ChunkPlan{
-				Units:    append([]source.SourceUnit(nil), req.Source.Units...),
+			SourceDigest: req.Source.Digest,
-			},
+			Ranges: []source.ChunkRange{{
+				StartUnitID: req.Source.Units[0].ID,
+				EndUnitID:   req.Source.Units[len(req.Source.Units)-1].ID,
+			}},
 		},
 	}, nil
 }
 
-type fakeExtractor struct {
+type recordingChunker struct {
-	key           string
+	key     string
-	artifactType  string
+	request ChunkRequest
-	schemaVersion string
-	validators    []Validator
 }
 
+func (chunker *recordingChunker) Key() string {
+	return chunker.key
+}
+
+func (chunker *recordingChunker) ReferenceSlots() []ReferenceSlot {
+	return nil
+}
+
+func (chunker *recordingChunker) Plan(ctx context.Context, req ChunkRequest) (ChunkPlanResult, error) {
+	chunker.request = req
+	return fakeChunker{key: chunker.key}.Plan(ctx, req)
+}
+
+type fakeExtractor struct {
+	key string
+}
+
+type fakeArtifact struct{ Value string }
+
 func (extractor fakeExtractor) Key() string {
 	return extractor.key
 }
 
-func (extractor fakeExtractor) ArtifactType() string {
+func (extractor fakeExtractor) ReferenceSlots() []ReferenceSlot {
-	return extractor.artifactType
+	return nil
 }
 
-func (extractor fakeExtractor) SchemaVersion() string {
+func (extractor fakeExtractor) Extract(ctx context.Context, req TypedExtractionRequest) (TypedExtractionResult[fakeArtifact], error) {
-	return extractor.schemaVersion
+	value := "example"
-}
-
-func (extractor fakeExtractor) Validators() []Validator {
-	return extractor.validators
-}
-
-func (extractor fakeExtractor) Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error) {
-	units := req.Source.Units
-	if req.Chunk != nil {
-		units = req.Chunk.Units
-	}
-	payload := json.RawMessage(`{"value":"example"}`)
 	if req.AmbientContext["mode"] == "chunked" {
-		payload = json.RawMessage(`{"value":"chunked"}`)
+		value = "chunked"
 	}
-
+	return TypedExtractionResult[fakeArtifact]{Value: fakeArtifact{Value: value}}, nil
-	return ExtractionResult{
-		Candidates: []artifacts.ArtifactCandidate{
-			{
-				Index:         0,
-				ExtractorKey:  extractor.key,
-				ArtifactType:  extractor.artifactType,
-				SchemaVersion: extractor.schemaVersion,
-				Payload:       payload,
-				SourceRefs: []source.SourceRef{
-					{
-						SourceID:    req.Source.ID,
-						StartUnitID: units[0].ID,
-						EndUnitID:   units[len(units)-1].ID,
-					},
-				},
-			},
-		},
-	}, nil
 }
 
 type fakeMerger struct {
@@ -366,13 +489,8 @@ func (merger fakeMerger) Key() string {
 	return merger.key
 }
 
-func (merger fakeMerger) Merge(ctx context.Context, req MergeRequest) (MergeResult, error) {
+func (merger fakeMerger) Merge(ctx context.Context, req TypedMergeRequest[fakeArtifact]) (TypedMergeResult[fakeArtifact], error) {
-	var candidates []artifacts.ArtifactCandidate
+	return TypedMergeResult[fakeArtifact]{Value: req.ExtractOutputs[0].Value}, nil
-	for _, chunkArtifacts := range req.ChunkArtifacts {
-		candidates = append(candidates, chunkArtifacts.Candidates...)
-	}
-
-	return MergeResult{Candidates: candidates}, nil
 }
 
 type fakeNormalizer struct {
@@ -383,33 +501,12 @@ func (normalizer fakeNormalizer) Key() string {
 	return normalizer.key
 }
 
-func (normalizer fakeNormalizer) Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error) {
+func (normalizer fakeNormalizer) ReferenceSlots() []ReferenceSlot {
-	return NormalizeResult{Candidates: req.Candidates}, nil
+	return nil
 }
 
-type fakeValidator struct {
+func (normalizer fakeNormalizer) Normalize(ctx context.Context, req TypedNormalizeRequest[fakeArtifact]) (TypedNormalizeResult[fakeArtifact], error) {
-	name string
+	return TypedNormalizeResult[fakeArtifact]{Value: req.MergeOutput.Value}, nil
-}
-
-func (validator fakeValidator) Name() string {
-	return validator.name
-}
-
-func (validator fakeValidator) Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error) {
-	decisions := make([]ValidationDecision, 0, len(req.Candidates))
-	for _, candidate := range req.Candidates {
-		decisions = append(decisions, ValidationDecision{
-			CandidateIndex: candidate.Index,
-			Approved:       true,
-			ReasonCode:     "accepted",
-			Message:        "candidate accepted",
-		})
-	}
-
-	return ValidationResult{
-		ValidatorName: validator.name,
-		Decisions:     decisions,
-	}, nil
 }
 
 type fakeLLMClient struct{}
@@ -434,7 +531,7 @@ func (encoder fakeOutputEncoder) Encode(ctx context.Context, req OutputRequest)
 			{
 				Name:        "artifacts/generic.json",
 				ContentType: "application/json",
-				Bytes:       []byte(`{"run_id":"` + req.Manifest.RunID + `","approved_count":1}`),
+				Bytes:       []byte(`{"run_id":"` + req.Manifest.RunID + `","output_count":1}`),
 			},
 		},
 	}, nil

internal/framework/contracts/typed_pipeline.go

@@ -0,0 +1,164 @@
+package contracts
+
+import (
+	"context"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/source"
+)
+
+// ExtractArtifact carries a typed per-chunk value with framework provenance.
+type ExtractArtifact[T any] struct {
+	LaneID       string
+	ExtractorKey string
+	SourceID     string
+	ChunkID      string
+	ChunkIndex   int
+	ChunkRef     source.SourceRef
+	Value        T
+}
+
+// MergeArtifact carries a typed merged value with framework provenance.
+type MergeArtifact[T any] struct {
+	LaneID    string
+	MergerKey string
+	SourceID  string
+	Value     T
+}
+
+// NormalizeArtifact carries a typed normalized value with framework provenance.
+type NormalizeArtifact[T any] struct {
+	LaneID        string
+	NormalizerKey string
+	SourceID      string
+	Value         T
+}
+
+type TypedExtractionRequest struct {
+	Source         *source.SourceDocument
+	Chunk          *source.Chunk
+	AmbientContext map[string]any
+	SourceInput    LLMInputMaterial
+	SessionID      string
+	References     ReferenceSet
+	LLMProfile     string
+	Metadata       map[string]any
+}
+
+type TypedExtractionResult[T any] struct {
+	Value    T
+	Warnings []Warning
+}
+
+type Extractor[T any] interface {
+	Key() string
+	ReferenceSlots() []ReferenceSlot
+	Extract(context.Context, TypedExtractionRequest) (TypedExtractionResult[T], error)
+}
+
+type TypedMergeRequest[T any] struct {
+	Source         *source.SourceDocument
+	LaneID         string
+	ExtractOutputs []ExtractArtifact[T]
+	SourceInput    LLMInputMaterial
+	SessionID      string
+	References     ReferenceSet
+	LLMProfile     string
+	Metadata       map[string]any
+}
+
+type TypedMergeResult[T any] struct {
+	Value    T
+	Warnings []Warning
+}
+
+type Merger[T any] interface {
+	Key() string
+	Merge(context.Context, TypedMergeRequest[T]) (TypedMergeResult[T], error)
+}
+
+type TypedNormalizeRequest[T any] struct {
+	Source      *source.SourceDocument
+	LaneID      string
+	MergeOutput MergeArtifact[T]
+	SourceInput LLMInputMaterial
+	SessionID   string
+	References  ReferenceSet
+	LLMProfile  string
+	Metadata    map[string]any
+}
+
+type TypedNormalizeResult[T any] struct {
+	Value    T
+	Warnings []Warning
+}
+
+type Normalizer[T any] interface {
+	Key() string
+	ReferenceSlots() []ReferenceSlot
+	Normalize(context.Context, TypedNormalizeRequest[T]) (TypedNormalizeResult[T], error)
+}
+
+type TypedValidationRequest[T any] struct {
+	Stage       string
+	LaneID      string
+	ModuleKey   string
+	Source      *source.SourceDocument
+	SourceID    string
+	SourceInput LLMInputMaterial
+	SessionID   string
+	References  ReferenceSet
+	LLMProfile  string
+	Metadata    map[string]any
+	Chunk       *source.Chunk
+	Chunks      []source.Chunk
+	Ref         source.SourceRef
+	Value       T
+}
+
+type TypedValidator[T any] interface {
+	Name() string
+	ExecutionClass() ExecutionClass
+	Validate(context.Context, TypedValidationRequest[T]) (ValidationResult, error)
+}
+
+type ChunkValidationRequest struct {
+	ModuleKey   string
+	Source      *source.SourceDocument
+	SourceID    string
+	SourceInput LLMInputMaterial
+	SessionID   string
+	References  ReferenceSet
+	LLMProfile  string
+	Metadata    map[string]any
+	Chunks      []source.Chunk
+}
+
+type ChunkValidator interface {
+	Name() string
+	ExecutionClass() ExecutionClass
+	Validate(context.Context, ChunkValidationRequest) (ValidationResult, error)
+}
+
+type SerializedValidationRequest struct {
+	Stage       string
+	LaneID      string
+	ModuleKey   string
+	Source      *source.SourceDocument
+	SourceID    string
+	SourceInput LLMInputMaterial
+	SessionID   string
+	References  ReferenceSet
+	LLMProfile  string
+	Metadata    map[string]any
+	Chunk       *source.Chunk
+	Chunks      []source.Chunk
+	Schema      ArtifactSchema
+	MediaType   string
+	Content     []byte
+}
+
+type SerializedValidator interface {
+	Name() string
+	ExecutionClass() ExecutionClass
+	Validate(context.Context, SerializedValidationRequest) (ValidationResult, error)
+}

internal/framework/debug/recorder.go

@@ -0,0 +1,38 @@
+package debug
+
+import (
+	"strings"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/fileio"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
+)
+
+type FilesystemRecorder struct {
+	root string
+}
+
+func NewFilesystemRecorder(root string) (pipeline.DebugRecorder, error) {
+	root = strings.TrimSpace(root)
+	if strings.TrimSpace(root) == "" {
+		return pipeline.NoopDebugRecorder(), nil
+	}
+	return &FilesystemRecorder{root: root}, nil
+}
+
+func (r *FilesystemRecorder) Enabled() bool {
+	return r != nil && strings.TrimSpace(r.root) != ""
+}
+
+func (r *FilesystemRecorder) WriteJSON(name string, payload any) error {
+	if !r.Enabled() {
+		return nil
+	}
+	return fileio.WriteJSON(r.root, name, payload, 0o700, 0o600)
+}
+
+func (r *FilesystemRecorder) WriteBytes(name string, data []byte) error {
+	if !r.Enabled() {
+		return nil
+	}
+	return fileio.WriteBytes(r.root, name, data, 0o700, 0o600)
+}

internal/framework/debug/recorder_test.go

@@ -0,0 +1,28 @@
+package debug
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func TestFilesystemRecorderWritesWithinTraceRoot(t *testing.T) {
+	root := t.TempDir()
+	recorder, err := NewFilesystemRecorder(root)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := recorder.WriteBytes("attempt/data", []byte("payload")); err != nil {
+		t.Fatal(err)
+	}
+	info, err := os.Stat(filepath.Join(root, "attempt", "data"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if info.Mode().Perm() != 0o600 {
+		t.Fatalf("mode=%#o", info.Mode().Perm())
+	}
+	if err := recorder.WriteBytes("../outside", nil); err == nil {
+		t.Fatal("accepted traversal")
+	}
+}

internal/framework/llm/asset_registry.go

@@ -0,0 +1,333 @@
+package llm
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"io/fs"
+	"path"
+	"sort"
+	"strings"
+	"time"
+
+	"gitea.maximumdirect.net/eric/scriptorium"
+)
+
+type AssetSource struct {
+	FS   fs.FS
+	Root string
+}
+
+type AssetRegistry struct {
+	prompts []AssetSource
+	schemas []AssetSource
+}
+
+type AssetHashPart struct {
+	FS   fs.FS
+	Path string
+}
+
+func NewAssetRegistry() *AssetRegistry {
+	return &AssetRegistry{}
+}
+
+func (r *AssetRegistry) RegisterPromptFS(fsys fs.FS, root string) error {
+	if r == nil {
+		return fmt.Errorf("asset registry must not be nil")
+	}
+	source, err := newAssetSource(fsys, root)
+	if err != nil {
+		return fmt.Errorf("register prompt assets: %w", err)
+	}
+	r.prompts = append(r.prompts, source)
+	return nil
+}
+
+func (r *AssetRegistry) RegisterSchemaFS(fsys fs.FS, root string) error {
+	if r == nil {
+		return fmt.Errorf("asset registry must not be nil")
+	}
+	source, err := newAssetSource(fsys, root)
+	if err != nil {
+		return fmt.Errorf("register schema assets: %w", err)
+	}
+	r.schemas = append(r.schemas, source)
+	return nil
+}
+
+func (r *AssetRegistry) PromptFS() (fs.FS, error) {
+	if r == nil {
+		return nil, fmt.Errorf("asset registry must not be nil")
+	}
+	return flattenAssetSources(r.prompts)
+}
+
+func (r *AssetRegistry) SchemaFS() (fs.FS, error) {
+	if r == nil {
+		return nil, fmt.Errorf("asset registry must not be nil")
+	}
+	return flattenAssetSources(r.schemas)
+}
+
+func (r *AssetRegistry) ScriptoriumOptions() ([]scriptorium.Option, error) {
+	promptFS, err := r.PromptFS()
+	if err != nil {
+		return nil, fmt.Errorf("prepare prompt assets: %w", err)
+	}
+	schemaFS, err := r.SchemaFS()
+	if err != nil {
+		return nil, fmt.Errorf("prepare schema assets: %w", err)
+	}
+	return []scriptorium.Option{
+		scriptorium.WithPromptFS(promptFS, "."),
+		scriptorium.WithSchemaFS(schemaFS, "."),
+	}, nil
+}
+
+func HashAssets(parts []AssetHashPart) (string, error) {
+	if len(parts) == 0 {
+		return "", fmt.Errorf("asset hash requires at least one part")
+	}
+	hash := sha256.New()
+	for _, part := range parts {
+		cleanPath, err := cleanAssetPath(part.Path)
+		if err != nil {
+			return "", fmt.Errorf("hash asset %q: %w", part.Path, err)
+		}
+		data, err := fs.ReadFile(part.FS, cleanPath)
+		if err != nil {
+			return "", fmt.Errorf("read hash asset %s: %w", cleanPath, err)
+		}
+		if _, err := io.WriteString(hash, cleanPath); err != nil {
+			return "", err
+		}
+		if _, err := hash.Write([]byte{0}); err != nil {
+			return "", err
+		}
+		if _, err := hash.Write(data); err != nil {
+			return "", err
+		}
+		if _, err := hash.Write([]byte{0}); err != nil {
+			return "", err
+		}
+	}
+	return "sha256:" + hex.EncodeToString(hash.Sum(nil)), nil
+}
+
+func newAssetSource(fsys fs.FS, root string) (AssetSource, error) {
+	if fsys == nil {
+		return AssetSource{}, fmt.Errorf("filesystem must not be nil")
+	}
+	cleanRoot, err := cleanAssetRoot(root)
+	if err != nil {
+		return AssetSource{}, err
+	}
+	return AssetSource{FS: fsys, Root: cleanRoot}, nil
+}
+
+func flattenAssetSources(sources []AssetSource) (fs.FS, error) {
+	out := assetMapFS{}
+	for _, source := range sources {
+		if err := fs.WalkDir(source.FS, source.Root, func(name string, entry fs.DirEntry, walkErr error) error {
+			if walkErr != nil {
+				return walkErr
+			}
+			if entry.IsDir() {
+				return nil
+			}
+			rel := name
+			if source.Root != "." {
+				rel = strings.TrimPrefix(name, source.Root+"/")
+			}
+			rel, err := cleanAssetPath(rel)
+			if err != nil {
+				return err
+			}
+			if _, exists := out[rel]; exists {
+				return fmt.Errorf("duplicate asset path %q", rel)
+			}
+			data, err := fs.ReadFile(source.FS, name)
+			if err != nil {
+				return err
+			}
+			out[rel] = append([]byte(nil), data...)
+			return nil
+		}); err != nil {
+			return nil, fmt.Errorf("walk asset root %s: %w", source.Root, err)
+		}
+	}
+	return out, nil
+}
+
+func cleanAssetRoot(root string) (string, error) {
+	trimmed := strings.TrimSpace(root)
+	if trimmed == "" || trimmed == "." {
+		return ".", nil
+	}
+	return cleanAssetPath(trimmed)
+}
+
+func cleanAssetPath(name string) (string, error) {
+	trimmed := strings.TrimSpace(name)
+	if trimmed == "" {
+		return "", fmt.Errorf("path must not be empty")
+	}
+	cleaned := path.Clean(strings.TrimPrefix(trimmed, "/"))
+	if cleaned == "." || !fs.ValidPath(cleaned) {
+		return "", fmt.Errorf("invalid path %q", name)
+	}
+	return cleaned, nil
+}
+
+type assetMapFS map[string][]byte
+
+func (m assetMapFS) Open(name string) (fs.File, error) {
+	cleaned, err := cleanOpenPath(name)
+	if err != nil {
+		return nil, &fs.PathError{Op: "open", Path: name, Err: err}
+	}
+	if data, ok := m[cleaned]; ok {
+		return &assetFile{
+			reader: bytes.NewReader(data),
+			info:   assetFileInfo{name: path.Base(cleaned), size: int64(len(data))},
+		}, nil
+	}
+	entries := m.dirEntries(cleaned)
+	if entries != nil {
+		return &assetDir{name: path.Base(cleaned), entries: entries}, nil
+	}
+	return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
+}
+
+func (m assetMapFS) ReadFile(name string) ([]byte, error) {
+	cleaned, err := cleanOpenPath(name)
+	if err != nil {
+		return nil, &fs.PathError{Op: "readfile", Path: name, Err: err}
+	}
+	data, ok := m[cleaned]
+	if !ok {
+		return nil, &fs.PathError{Op: "readfile", Path: name, Err: fs.ErrNotExist}
+	}
+	return append([]byte(nil), data...), nil
+}
+
+func (m assetMapFS) ReadDir(name string) ([]fs.DirEntry, error) {
+	cleaned, err := cleanOpenPath(name)
+	if err != nil {
+		return nil, &fs.PathError{Op: "readdir", Path: name, Err: err}
+	}
+	entries := m.dirEntries(cleaned)
+	if entries == nil {
+		return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrNotExist}
+	}
+	return entries, nil
+}
+
+func (m assetMapFS) dirEntries(dir string) []fs.DirEntry {
+	children := map[string]assetDirEntry{}
+	prefix := ""
+	if dir != "." {
+		prefix = dir + "/"
+	}
+	for name, data := range m {
+		if !strings.HasPrefix(name, prefix) {
+			continue
+		}
+		rest := strings.TrimPrefix(name, prefix)
+		if rest == "" {
+			continue
+		}
+		childName, _, hasSlash := strings.Cut(rest, "/")
+		entry := assetDirEntry{name: childName, dir: hasSlash}
+		if !hasSlash {
+			entry.size = int64(len(data))
+		}
+		children[childName] = entry
+	}
+	if len(children) == 0 {
+		return nil
+	}
+	names := make([]string, 0, len(children))
+	for name := range children {
+		names = append(names, name)
+	}
+	sort.Strings(names)
+	entries := make([]fs.DirEntry, 0, len(names))
+	for _, name := range names {
+		entries = append(entries, children[name])
+	}
+	return entries
+}
+
+func cleanOpenPath(name string) (string, error) {
+	if name == "." {
+		return ".", nil
+	}
+	return cleanAssetPath(name)
+}
+
+type assetFile struct {
+	reader *bytes.Reader
+	info   assetFileInfo
+}
+
+func (f *assetFile) Stat() (fs.FileInfo, error) { return f.info, nil }
+func (f *assetFile) Read(p []byte) (int, error) { return f.reader.Read(p) }
+func (f *assetFile) Close() error               { return nil }
+
+type assetDir struct {
+	name    string
+	offset  int
+	entries []fs.DirEntry
+}
+
+func (d *assetDir) Stat() (fs.FileInfo, error) { return assetFileInfo{name: d.name, dir: true}, nil }
+func (d *assetDir) Read([]byte) (int, error)   { return 0, fmt.Errorf("cannot read directory") }
+func (d *assetDir) Close() error               { return nil }
+func (d *assetDir) ReadDir(n int) ([]fs.DirEntry, error) {
+	if d.offset >= len(d.entries) {
+		return nil, io.EOF
+	}
+	end := len(d.entries)
+	if n > 0 && d.offset+n < end {
+		end = d.offset + n
+	}
+	out := append([]fs.DirEntry(nil), d.entries[d.offset:end]...)
+	d.offset = end
+	return out, nil
+}
+
+type assetDirEntry struct {
+	name string
+	dir  bool
+	size int64
+}
+
+func (e assetDirEntry) Name() string      { return e.name }
+func (e assetDirEntry) IsDir() bool       { return e.dir }
+func (e assetDirEntry) Type() fs.FileMode { return e.InfoMode().Type() }
+func (e assetDirEntry) Info() (fs.FileInfo, error) {
+	return assetFileInfo{name: e.name, dir: e.dir, size: e.size}, nil
+}
+func (e assetDirEntry) InfoMode() fs.FileMode {
+	if e.dir {
+		return fs.ModeDir | 0o555
+	}
+	return 0o444
+}
+
+type assetFileInfo struct {
+	name string
+	dir  bool
+	size int64
+}
+
+func (i assetFileInfo) Name() string       { return i.name }
+func (i assetFileInfo) Size() int64        { return i.size }
+func (i assetFileInfo) Mode() fs.FileMode  { return assetDirEntry{dir: i.dir}.InfoMode() }
+func (i assetFileInfo) ModTime() time.Time { return time.Time{} }
+func (i assetFileInfo) IsDir() bool        { return i.dir }
+func (i assetFileInfo) Sys() any           { return nil }

internal/framework/llm/asset_registry_test.go

@@ -0,0 +1,198 @@
+package llm
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"testing/fstest"
+	"time"
+
+	"gitea.maximumdirect.net/eric/scriptorium"
+)
+
+func TestAssetRegistryCombinesPromptAndSchemaSources(t *testing.T) {
+	registry := NewAssetRegistry()
+	mustRegisterPromptFS(t, registry, fstest.MapFS{
+		"prompts/test.yaml":           {Data: []byte(validPromptYAML("schemas/out.json"))},
+		"prompts/messages/user.tmpl":  {Data: []byte(`Input: {{ input "transcript" }}`)},
+		"prompts/messages/task.tmpl":  {Data: []byte("Return JSON.")},
+		"schemas/ignored/schema.json": {Data: []byte(`{"type":"object"}`)},
+	}, "prompts")
+	mustRegisterSchemaFS(t, registry, fstest.MapFS{
+		"root/schemas/out.json": {Data: []byte(`{"type":"object"}`)},
+	}, "root")
+
+	engine := newAssetTestEngine(t, registry)
+	prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
+		PromptID:  "asset.test",
+		ProfileID: "asset-test-profile",
+		Inputs: map[string]scriptorium.ArtifactRef{
+			"transcript": scriptorium.Inline(`{"ok":true}`),
+		},
+	})
+	if err != nil {
+		t.Fatalf("Prepare() error = %v, want nil", err)
+	}
+	if got := len(prepared.Messages); got != 2 {
+		t.Fatalf("message count = %d, want 2", got)
+	}
+	if prepared.OutputContract.SchemaPath != "schemas/out.json" {
+		t.Fatalf("schema path = %q, want schemas/out.json", prepared.OutputContract.SchemaPath)
+	}
+}
+
+func TestAssetRegistryPrepareFailsForMissingPromptAsset(t *testing.T) {
+	registry := NewAssetRegistry()
+	mustRegisterPromptFS(t, registry, fstest.MapFS{
+		"test.yaml": {Data: []byte(validPromptYAML("out.json"))},
+	}, ".")
+	mustRegisterSchemaFS(t, registry, fstest.MapFS{
+		"out.json": {Data: []byte(`{"type":"object"}`)},
+	}, ".")
+
+	engine := newAssetTestEngine(t, registry)
+	_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
+		PromptID:  "asset.test",
+		ProfileID: "asset-test-profile",
+		Inputs: map[string]scriptorium.ArtifactRef{
+			"transcript": scriptorium.Inline(`{"ok":true}`),
+		},
+	})
+	if err == nil || !strings.Contains(err.Error(), "content_file") {
+		t.Fatalf("Prepare() error = %v, want missing content_file error", err)
+	}
+}
+
+func TestAssetRegistryPrepareFailsForMissingSchemaAsset(t *testing.T) {
+	registry := NewAssetRegistry()
+	mustRegisterPromptFS(t, registry, fstest.MapFS{
+		"test.yaml":          {Data: []byte(validPromptYAML("missing.json"))},
+		"messages/user.tmpl": {Data: []byte(`Input: {{ input "transcript" }}`)},
+		"messages/task.tmpl": {Data: []byte("Return JSON.")},
+	}, ".")
+	mustRegisterSchemaFS(t, registry, fstest.MapFS{
+		"present.json": {Data: []byte(`{"type":"object"}`)},
+	}, ".")
+
+	engine := newAssetTestEngine(t, registry)
+	_, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
+		PromptID:  "asset.test",
+		ProfileID: "asset-test-profile",
+		Inputs: map[string]scriptorium.ArtifactRef{
+			"transcript": scriptorium.Inline(`{"ok":true}`),
+		},
+	})
+	if err == nil || !strings.Contains(err.Error(), "missing.json") {
+		t.Fatalf("Prepare() error = %v, want missing schema error", err)
+	}
+}
+
+func TestAssetRegistryRejectsDuplicateAssetPaths(t *testing.T) {
+	registry := NewAssetRegistry()
+	mustRegisterPromptFS(t, registry, fstest.MapFS{"one/prompt.yaml": {Data: []byte("id: one")}}, "one")
+	mustRegisterPromptFS(t, registry, fstest.MapFS{"two/prompt.yaml": {Data: []byte("id: two")}}, "two")
+
+	_, err := registry.PromptFS()
+	if err == nil || !strings.Contains(err.Error(), "duplicate asset path") {
+		t.Fatalf("PromptFS() error = %v, want duplicate path error", err)
+	}
+}
+
+func TestAssetRegistryCombinesNamespacedPromptSources(t *testing.T) {
+	registry := NewAssetRegistry()
+	mustRegisterPromptFS(t, registry, fstest.MapFS{
+		"dnd.spells/dnd.spells.yaml": {Data: []byte(validPromptYAML("schema.json"))},
+		"dnd.spells/task.md":         {Data: []byte("spell task")},
+		"dnd.spells/instructions.md": {Data: []byte("spell instructions")},
+	}, ".")
+	mustRegisterPromptFS(t, registry, fstest.MapFS{
+		"dnd.scenes/dnd.scenes.yaml": {Data: []byte(validPromptYAML("schema.json"))},
+		"dnd.scenes/task.md":         {Data: []byte("scene task")},
+		"dnd.scenes/instructions.md": {Data: []byte("scene instructions")},
+	}, ".")
+
+	fsys, err := registry.PromptFS()
+	if err != nil {
+		t.Fatalf("PromptFS() error = %v, want nil", err)
+	}
+	for _, name := range []string{
+		"dnd.spells/dnd.spells.yaml",
+		"dnd.spells/task.md",
+		"dnd.spells/instructions.md",
+		"dnd.scenes/dnd.scenes.yaml",
+		"dnd.scenes/task.md",
+		"dnd.scenes/instructions.md",
+	} {
+		if _, err := fsys.Open(name); err != nil {
+			t.Fatalf("PromptFS().Open(%q) error = %v, want nil", name, err)
+		}
+	}
+}
+
+func TestHashAssetsOmitsRawAssetContent(t *testing.T) {
+	hash, err := HashAssets([]AssetHashPart{{
+		FS:   fstest.MapFS{"prompt.md": {Data: []byte("secret prompt text")}},
+		Path: "prompt.md",
+	}})
+	if err != nil {
+		t.Fatalf("HashAssets() error = %v, want nil", err)
+	}
+	if !strings.HasPrefix(hash, "sha256:") {
+		t.Fatalf("hash = %q, want sha256-prefixed value", hash)
+	}
+	if strings.Contains(hash, "secret prompt text") {
+		t.Fatalf("hash leaked asset content")
+	}
+}
+
+func newAssetTestEngine(t *testing.T, registry *AssetRegistry) *scriptorium.Engine {
+	t.Helper()
+	options, err := registry.ScriptoriumOptions()
+	if err != nil {
+		t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
+	}
+	options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
+		ID:       "asset-test-profile",
+		Endpoint: "http://127.0.0.1:1/v1",
+		Model:    "asset-test-model",
+	})))
+	engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
+	if err != nil {
+		t.Fatalf("NewEngine() error = %v, want nil", err)
+	}
+	return engine
+}
+
+func mustRegisterPromptFS(t *testing.T, registry *AssetRegistry, fsys fstest.MapFS, root string) {
+	t.Helper()
+	if err := registry.RegisterPromptFS(fsys, root); err != nil {
+		t.Fatalf("RegisterPromptFS() error = %v, want nil", err)
+	}
+}
+
+func mustRegisterSchemaFS(t *testing.T, registry *AssetRegistry, fsys fstest.MapFS, root string) {
+	t.Helper()
+	if err := registry.RegisterSchemaFS(fsys, root); err != nil {
+		t.Fatalf("RegisterSchemaFS() error = %v, want nil", err)
+	}
+}
+
+func validPromptYAML(schemaPath string) string {
+	return `id: asset.test
+version: "v1"
+inputs:
+  - name: transcript
+    required: true
+    content_type: application/json
+messages:
+  - role: user
+    content_file: ./messages/user.tmpl
+  - role: user
+    content_file: ./messages/task.tmpl
+output:
+  format: json
+  validation_mode: json_schema
+  schema_path: ` + schemaPath + `
+  repair_attempts: 0
+`
+}

internal/framework/llm/openai_compatible_client.go

@@ -1,364 +0,0 @@
-package llm
-
-import (
-	"bytes"
-	"context"
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"net/url"
-	"strings"
-	"time"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
-)
-
-const openAICompatibleProviderName = "openai-compatible"
-
-// OpenAICompatibleClientConfig configures the direct HTTP structured-output adapter.
-type OpenAICompatibleClientConfig struct {
-	BaseURL        string
-	Model          string
-	APIKey         string
-	MaxRetries     int
-	HTTPClient     *http.Client
-	RequestTimeout time.Duration
-}
-
-// OpenAICompatibleClient sends OpenAI-compatible chat-completion requests with
-// response_format.type=json_schema.
-type OpenAICompatibleClient struct {
-	baseURL        string
-	model          string
-	apiKey         string
-	maxRetries     int
-	httpClient     *http.Client
-	requestTimeout time.Duration
-}
-
-var _ contracts.StructuredLLMClient = (*OpenAICompatibleClient)(nil)
-
-func NewOpenAICompatibleClient(cfg OpenAICompatibleClientConfig) (*OpenAICompatibleClient, error) {
-	normalized, err := normalizeOpenAICompatibleConfig(cfg)
-	if err != nil {
-		return nil, err
-	}
-
-	client := normalized.HTTPClient
-	if client == nil {
-		client = http.DefaultClient
-	}
-
-	return &OpenAICompatibleClient{
-		baseURL:        normalized.BaseURL,
-		model:          normalized.Model,
-		apiKey:         normalized.APIKey,
-		maxRetries:     normalized.MaxRetries,
-		httpClient:     client,
-		requestTimeout: normalized.RequestTimeout,
-	}, nil
-}
-
-func (c *OpenAICompatibleClient) CompleteStructured(
-	ctx context.Context,
-	req contracts.StructuredCompletionRequest,
-	out any,
-) (contracts.StructuredCompletionResponse, error) {
-	if c == nil {
-		return contracts.StructuredCompletionResponse{}, fmt.Errorf("openai-compatible client must not be nil")
-	}
-	if err := validateOutputTarget(out); err != nil {
-		return contracts.StructuredCompletionResponse{}, err
-	}
-
-	model := strings.TrimSpace(req.Model)
-	if model == "" {
-		model = c.model
-	}
-	if model == "" {
-		return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion model must not be empty")
-	}
-
-	schemaName := strings.TrimSpace(req.ResponseSchemaName)
-	if schemaName == "" {
-		return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion response schema name must not be empty")
-	}
-	if len(bytes.TrimSpace(req.ResponseSchema)) == 0 || !json.Valid(req.ResponseSchema) {
-		return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion response schema JSON must be valid")
-	}
-
-	messages, err := toOpenAICompatibleMessages(req.Messages)
-	if err != nil {
-		return contracts.StructuredCompletionResponse{}, err
-	}
-
-	endpoint := buildChatCompletionsURL(c.baseURL)
-	var lastErr error
-	for attempt := 0; attempt <= c.maxRetries; attempt++ {
-		content, metadata, callErr := c.completeStructuredOnce(ctx, endpoint, model, messages, schemaName, req.ResponseSchema)
-		if callErr == nil {
-			if decodeErr := json.Unmarshal(content, out); decodeErr != nil {
-				callErr = retryableError{err: fmt.Errorf("decode structured output: %w", decodeErr)}
-			} else {
-				return contracts.StructuredCompletionResponse{
-					Content:          content,
-					Provider:         openAICompatibleProviderName,
-					Model:            firstNonEmpty(metadata.Model, model),
-					PromptTokens:     metadata.PromptTokens,
-					CompletionTokens: metadata.CompletionTokens,
-					TotalTokens:      metadata.TotalTokens,
-				}, nil
-			}
-		}
-
-		if ctx.Err() != nil {
-			return contracts.StructuredCompletionResponse{}, ctx.Err()
-		}
-		lastErr = c.redactError(callErr)
-		if !canRetry(ctx, attempt, c.maxRetries, callErr) {
-			return contracts.StructuredCompletionResponse{}, lastErr
-		}
-	}
-
-	if lastErr == nil {
-		lastErr = fmt.Errorf("structured completion failed")
-	}
-	return contracts.StructuredCompletionResponse{}, lastErr
-}
-
-type openAICompatibleMessage struct {
-	Role    string `json:"role"`
-	Content string `json:"content"`
-}
-
-type openAICompatibleRequest struct {
-	Model          string                                `json:"model"`
-	Messages       []openAICompatibleMessage             `json:"messages"`
-	ResponseFormat openAICompatibleStructuredOutputShape `json:"response_format"`
-}
-
-type openAICompatibleStructuredOutputShape struct {
-	Type       string                         `json:"type"`
-	JSONSchema openAICompatibleSchemaEnvelope `json:"json_schema"`
-}
-
-type openAICompatibleSchemaEnvelope struct {
-	Name   string          `json:"name"`
-	Strict bool            `json:"strict"`
-	Schema json.RawMessage `json:"schema"`
-}
-
-type openAICompatibleChatCompletionsResponse struct {
-	Model   string `json:"model"`
-	Choices []struct {
-		Message struct {
-			Content json.RawMessage `json:"content"`
-		} `json:"message"`
-	} `json:"choices"`
-	Usage *openAICompatibleUsage `json:"usage,omitempty"`
-}
-
-type openAICompatibleUsage struct {
-	PromptTokens     int `json:"prompt_tokens"`
-	CompletionTokens int `json:"completion_tokens"`
-	TotalTokens      int `json:"total_tokens"`
-}
-
-type openAICompatibleResponseMetadata struct {
-	Model            string
-	PromptTokens     int
-	CompletionTokens int
-	TotalTokens      int
-}
-
-func normalizeOpenAICompatibleConfig(cfg OpenAICompatibleClientConfig) (OpenAICompatibleClientConfig, error) {
-	cfg.BaseURL = strings.TrimSpace(cfg.BaseURL)
-	cfg.Model = strings.TrimSpace(cfg.Model)
-	cfg.APIKey = strings.TrimSpace(cfg.APIKey)
-	if cfg.MaxRetries < 0 {
-		return OpenAICompatibleClientConfig{}, fmt.Errorf("max retries must be zero or greater")
-	}
-	if cfg.BaseURL == "" {
-		return OpenAICompatibleClientConfig{}, fmt.Errorf("base URL must not be empty")
-	}
-	if _, err := url.ParseRequestURI(cfg.BaseURL); err != nil {
-		return OpenAICompatibleClientConfig{}, fmt.Errorf("base URL must be valid: %w", err)
-	}
-	if cfg.Model == "" {
-		return OpenAICompatibleClientConfig{}, fmt.Errorf("model must not be empty")
-	}
-	cfg.BaseURL = strings.TrimRight(cfg.BaseURL, "/")
-	return cfg, nil
-}
-
-func (c *OpenAICompatibleClient) completeStructuredOnce(
-	ctx context.Context,
-	endpoint string,
-	model string,
-	messages []openAICompatibleMessage,
-	responseSchemaName string,
-	responseSchemaJSON json.RawMessage,
-) (json.RawMessage, openAICompatibleResponseMetadata, error) {
-	requestCtx := ctx
-	var cancel context.CancelFunc
-	if c.requestTimeout > 0 {
-		requestCtx, cancel = context.WithTimeout(ctx, c.requestTimeout)
-		defer cancel()
-	}
-
-	requestBody := openAICompatibleRequest{
-		Model:    model,
-		Messages: messages,
-		ResponseFormat: openAICompatibleStructuredOutputShape{
-			Type: "json_schema",
-			JSONSchema: openAICompatibleSchemaEnvelope{
-				Name:   responseSchemaName,
-				Strict: true,
-				Schema: responseSchemaJSON,
-			},
-		},
-	}
-
-	payload, err := json.Marshal(requestBody)
-	if err != nil {
-		return nil, openAICompatibleResponseMetadata{}, fmt.Errorf("marshal provider request: %w", err)
-	}
-
-	httpReq, err := http.NewRequestWithContext(requestCtx, http.MethodPost, endpoint, bytes.NewReader(payload))
-	if err != nil {
-		return nil, openAICompatibleResponseMetadata{}, fmt.Errorf("build provider request: %w", err)
-	}
-	httpReq.Header.Set("Content-Type", "application/json")
-	if c.apiKey != "" {
-		httpReq.Header.Set("Authorization", "Bearer "+c.apiKey)
-	}
-
-	httpResp, err := c.httpClient.Do(httpReq)
-	if err != nil {
-		return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("provider request failed: %w", err)}
-	}
-	defer func() {
-		_ = httpResp.Body.Close()
-	}()
-
-	rawResp, err := io.ReadAll(httpResp.Body)
-	if err != nil {
-		return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("read provider response: %w", err)}
-	}
-
-	if httpResp.StatusCode < 200 || httpResp.StatusCode >= 300 {
-		statusErr := parseProviderErrorBody(httpResp.StatusCode, rawResp)
-		if httpResp.StatusCode == http.StatusTooManyRequests || httpResp.StatusCode >= 500 {
-			return nil, openAICompatibleResponseMetadata{}, retryableError{err: statusErr}
-		}
-		return nil, openAICompatibleResponseMetadata{}, statusErr
-	}
-
-	return decodeChatCompletionsResponse(rawResp)
-}
-
-func toOpenAICompatibleMessages(messages []contracts.LLMMessage) ([]openAICompatibleMessage, error) {
-	if len(messages) == 0 {
-		return nil, fmt.Errorf("structured completion messages must not be empty")
-	}
-
-	result := make([]openAICompatibleMessage, len(messages))
-	for i, message := range messages {
-		role := strings.TrimSpace(message.Role)
-		content := strings.TrimSpace(message.Content)
-		if role == "" {
-			return nil, fmt.Errorf("message[%d] role must not be empty", i)
-		}
-		if content == "" {
-			return nil, fmt.Errorf("message[%d] content must not be empty", i)
-		}
-		result[i] = openAICompatibleMessage{
-			Role:    role,
-			Content: content,
-		}
-	}
-	return result, nil
-}
-
-func buildChatCompletionsURL(baseURL string) string {
-	return strings.TrimRight(baseURL, "/") + "/chat/completions"
-}
-
-func decodeChatCompletionsResponse(raw []byte) (json.RawMessage, openAICompatibleResponseMetadata, error) {
-	var parsed openAICompatibleChatCompletionsResponse
-	if err := json.Unmarshal(raw, &parsed); err != nil {
-		return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("decode provider response envelope: %w", err)}
-	}
-	if len(parsed.Choices) == 0 {
-		return nil, openAICompatibleResponseMetadata{}, retryableError{err: fmt.Errorf("provider response missing choices")}
-	}
-
-	content, err := extractAssistantContentJSON(parsed.Choices[0].Message.Content)
-	if err != nil {
-		return nil, openAICompatibleResponseMetadata{}, retryableError{err: err}
-	}
-
-	metadata := openAICompatibleResponseMetadata{
-		Model: parsed.Model,
-	}
-	if parsed.Usage != nil {
-		metadata.PromptTokens = parsed.Usage.PromptTokens
-		metadata.CompletionTokens = parsed.Usage.CompletionTokens
-		metadata.TotalTokens = parsed.Usage.TotalTokens
-	}
-	return content, metadata, nil
-}
-
-func extractAssistantContentJSON(raw json.RawMessage) (json.RawMessage, error) {
-	trimmedRaw := bytes.TrimSpace(raw)
-	if len(trimmedRaw) == 0 || bytes.Equal(trimmedRaw, []byte("null")) {
-		return nil, fmt.Errorf("provider response missing assistant message content")
-	}
-
-	var textContent string
-	if err := json.Unmarshal(trimmedRaw, &textContent); err == nil {
-		textContent = strings.TrimSpace(textContent)
-		if textContent == "" {
-			return nil, fmt.Errorf("provider response assistant message content is empty")
-		}
-		if !json.Valid([]byte(textContent)) {
-			return nil, fmt.Errorf("provider response assistant message content is not valid JSON")
-		}
-		return json.RawMessage(textContent), nil
-	}
-
-	if json.Valid(trimmedRaw) {
-		return append(json.RawMessage(nil), trimmedRaw...), nil
-	}
-	return nil, fmt.Errorf("provider response assistant message content is not valid JSON")
-}
-
-func parseProviderErrorBody(status int, body []byte) error {
-	trimmed := strings.TrimSpace(string(body))
-	if trimmed == "" {
-		return fmt.Errorf("provider returned status %d", status)
-	}
-
-	var payload map[string]any
-	if err := json.Unmarshal(body, &payload); err == nil {
-		if nested, ok := payload["error"].(map[string]any); ok {
-			if msg, ok := nested["message"].(string); ok && strings.TrimSpace(msg) != "" {
-				return fmt.Errorf("provider returned status %d: %s", status, strings.TrimSpace(msg))
-			}
-		}
-		if msg, ok := payload["message"].(string); ok && strings.TrimSpace(msg) != "" {
-			return fmt.Errorf("provider returned status %d: %s", status, strings.TrimSpace(msg))
-		}
-	}
-
-	return fmt.Errorf("provider returned status %d: %s", status, trimmed)
-}
-
-func (c *OpenAICompatibleClient) redactError(err error) error {
-	secrets := []string{c.apiKey}
-	if c.apiKey != "" {
-		secrets = append(secrets, "Bearer "+c.apiKey)
-	}
-	return ErrorWithSecretsRedacted(err, secrets)
-}

internal/framework/llm/openai_compatible_client_test.go

@@ -1,494 +0,0 @@
-package llm
-
-import (
-	"context"
-	"encoding/json"
-	"errors"
-	"fmt"
-	"io"
-	"net/http"
-	"net/http/httptest"
-	"strings"
-	"sync/atomic"
-	"testing"
-
-	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
-)
-
-type testArtifact struct {
-	Value string `json:"value"`
-}
-
-func TestNewOpenAICompatibleClientValidation(t *testing.T) {
-	tests := []struct {
-		name string
-		cfg  OpenAICompatibleClientConfig
-		want string
-	}{
-		{
-			name: "empty base URL",
-			cfg: OpenAICompatibleClientConfig{
-				BaseURL: " ",
-				Model:   "model",
-			},
-			want: "base URL",
-		},
-		{
-			name: "invalid base URL",
-			cfg: OpenAICompatibleClientConfig{
-				BaseURL: "://bad",
-				Model:   "model",
-			},
-			want: "base URL",
-		},
-		{
-			name: "empty model",
-			cfg: OpenAICompatibleClientConfig{
-				BaseURL: "https://example.test/v1",
-				Model:   " ",
-			},
-			want: "model",
-		},
-		{
-			name: "negative retries",
-			cfg: OpenAICompatibleClientConfig{
-				BaseURL:    "https://example.test/v1",
-				Model:      "model",
-				MaxRetries: -1,
-			},
-			want: "max retries",
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			_, err := NewOpenAICompatibleClient(tc.cfg)
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestOpenAICompatibleClientSuccessfulStructuredCompletion(t *testing.T) {
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		w.Header().Set("Content-Type", "application/json")
-		_, _ = io.WriteString(w, `{
-  "model":"provider-model",
-  "choices":[{"message":{"content":"{\"value\":\"ok\"}"}}],
-  "usage":{"prompt_tokens":11,"completion_tokens":7,"total_tokens":18}
-}`)
-	}))
-	defer server.Close()
-
-	client := newTestClient(t, server.URL, "default-model", 0)
-	var out testArtifact
-	resp, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), &out)
-	if err != nil {
-		t.Fatalf("CompleteStructured: %v", err)
-	}
-
-	if out.Value != "ok" {
-		t.Fatalf("unexpected decoded output: %+v", out)
-	}
-	if string(resp.Content) != `{"value":"ok"}` {
-		t.Fatalf("unexpected raw content: %s", resp.Content)
-	}
-	if resp.Provider != openAICompatibleProviderName {
-		t.Fatalf("unexpected provider: %q", resp.Provider)
-	}
-	if resp.Model != "provider-model" {
-		t.Fatalf("unexpected model: %q", resp.Model)
-	}
-	if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
-		t.Fatalf("unexpected token metadata: %+v", resp)
-	}
-}
-
-func TestOpenAICompatibleClientRequestBodyIncludesStructuredOutputShape(t *testing.T) {
-	var seenPath string
-	var seenAuthorization string
-	var seenReq map[string]any
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		seenPath = r.URL.Path
-		seenAuthorization = r.Header.Get("Authorization")
-		if err := json.NewDecoder(r.Body).Decode(&seenReq); err != nil {
-			t.Fatalf("decode request: %v", err)
-		}
-		_, _ = io.WriteString(w, `{"choices":[{"message":{"content":"{\"value\":\"ok\"}"}}]}`)
-	}))
-	defer server.Close()
-
-	client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
-		BaseURL:    server.URL + "/v1",
-		Model:      "default-model",
-		APIKey:     "secret-key",
-		MaxRetries: 0,
-	})
-	if err != nil {
-		t.Fatalf("NewOpenAICompatibleClient: %v", err)
-	}
-
-	var out testArtifact
-	_, err = client.CompleteStructured(context.Background(), validStructuredRequest("request-model"), &out)
-	if err != nil {
-		t.Fatalf("CompleteStructured: %v", err)
-	}
-
-	if seenPath != "/v1/chat/completions" {
-		t.Fatalf("unexpected request path: %q", seenPath)
-	}
-	if seenAuthorization != "Bearer secret-key" {
-		t.Fatalf("unexpected authorization header: %q", seenAuthorization)
-	}
-	if seenReq["model"] != "request-model" {
-		t.Fatalf("unexpected model: %v", seenReq["model"])
-	}
-
-	messages, ok := seenReq["messages"].([]any)
-	if !ok || len(messages) != 1 {
-		t.Fatalf("unexpected messages: %#v", seenReq["messages"])
-	}
-	message, ok := messages[0].(map[string]any)
-	if !ok {
-		t.Fatalf("unexpected message shape: %#v", messages[0])
-	}
-	if message["role"] != "user" || message["content"] != "extract this" {
-		t.Fatalf("unexpected message: %#v", message)
-	}
-
-	responseFormat, ok := seenReq["response_format"].(map[string]any)
-	if !ok {
-		t.Fatalf("expected response_format object, got %T", seenReq["response_format"])
-	}
-	if responseFormat["type"] != "json_schema" {
-		t.Fatalf("unexpected response_format.type: %v", responseFormat["type"])
-	}
-	jsonSchema, ok := responseFormat["json_schema"].(map[string]any)
-	if !ok {
-		t.Fatalf("expected response_format.json_schema object, got %T", responseFormat["json_schema"])
-	}
-	if jsonSchema["name"] != "test_artifact" {
-		t.Fatalf("unexpected schema name: %v", jsonSchema["name"])
-	}
-	if jsonSchema["strict"] != true {
-		t.Fatalf("expected strict=true, got %v", jsonSchema["strict"])
-	}
-	schema, ok := jsonSchema["schema"].(map[string]any)
-	if !ok {
-		t.Fatalf("expected schema object, got %T", jsonSchema["schema"])
-	}
-	if schema["type"] != "object" {
-		t.Fatalf("unexpected schema: %#v", schema)
-	}
-}
-
-func TestOpenAICompatibleClientDefaultModelFallbackAndOverride(t *testing.T) {
-	var seenModels []string
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		var req map[string]any
-		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
-			t.Fatalf("decode request: %v", err)
-		}
-		seenModels = append(seenModels, fmt.Sprint(req["model"]))
-		_, _ = io.WriteString(w, `{"choices":[{"message":{"content":"{\"value\":\"ok\"}"}}]}`)
-	}))
-	defer server.Close()
-
-	client := newTestClient(t, server.URL, "default-model", 0)
-	var first testArtifact
-	if _, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), &first); err != nil {
-		t.Fatalf("first CompleteStructured: %v", err)
-	}
-	var second testArtifact
-	if _, err := client.CompleteStructured(context.Background(), validStructuredRequest("override-model"), &second); err != nil {
-		t.Fatalf("second CompleteStructured: %v", err)
-	}
-
-	if len(seenModels) != 2 || seenModels[0] != "default-model" || seenModels[1] != "override-model" {
-		t.Fatalf("unexpected models: %v", seenModels)
-	}
-}
-
-func TestOpenAICompatibleClientInvalidOutputTarget(t *testing.T) {
-	client := newTestClient(t, "https://example.test/v1", "default-model", 0)
-
-	tests := []struct {
-		name string
-		out  any
-	}{
-		{name: "nil", out: nil},
-		{name: "non-pointer", out: testArtifact{}},
-		{name: "nil pointer", out: (*testArtifact)(nil)},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			_, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), tc.out)
-			if err == nil || !strings.Contains(err.Error(), "output target") {
-				t.Fatalf("expected output target error, got %v", err)
-			}
-		})
-	}
-}
-
-func TestOpenAICompatibleClientMissingAndInvalidSchema(t *testing.T) {
-	client := newTestClient(t, "https://example.test/v1", "default-model", 0)
-
-	tests := []struct {
-		name   string
-		mutate func(*contracts.StructuredCompletionRequest)
-		want   string
-	}{
-		{
-			name: "missing schema name",
-			mutate: func(req *contracts.StructuredCompletionRequest) {
-				req.ResponseSchemaName = " "
-			},
-			want: "schema name",
-		},
-		{
-			name: "missing schema JSON",
-			mutate: func(req *contracts.StructuredCompletionRequest) {
-				req.ResponseSchema = nil
-			},
-			want: "schema JSON",
-		},
-		{
-			name: "invalid schema JSON",
-			mutate: func(req *contracts.StructuredCompletionRequest) {
-				req.ResponseSchema = json.RawMessage(`{"type":`)
-			},
-			want: "schema JSON",
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			req := validStructuredRequest("")
-			tc.mutate(&req)
-			var out testArtifact
-			_, err := client.CompleteStructured(context.Background(), req, &out)
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestOpenAICompatibleClientRejectsEmptyMessages(t *testing.T) {
-	client := newTestClient(t, "https://example.test/v1", "default-model", 0)
-
-	tests := []struct {
-		name   string
-		mutate func(*contracts.StructuredCompletionRequest)
-		want   string
-	}{
-		{
-			name: "no messages",
-			mutate: func(req *contracts.StructuredCompletionRequest) {
-				req.Messages = nil
-			},
-			want: "messages",
-		},
-		{
-			name: "empty role",
-			mutate: func(req *contracts.StructuredCompletionRequest) {
-				req.Messages[0].Role = " "
-			},
-			want: "role",
-		},
-		{
-			name: "empty content",
-			mutate: func(req *contracts.StructuredCompletionRequest) {
-				req.Messages[0].Content = " "
-			},
-			want: "content",
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			req := validStructuredRequest("")
-			tc.mutate(&req)
-			var out testArtifact
-			_, err := client.CompleteStructured(context.Background(), req, &out)
-			if err == nil || !strings.Contains(err.Error(), tc.want) {
-				t.Fatalf("expected error containing %q, got %v", tc.want, err)
-			}
-		})
-	}
-}
-
-func TestOpenAICompatibleClientProviderNon2xxBehavior(t *testing.T) {
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		w.WriteHeader(http.StatusBadRequest)
-		_, _ = io.WriteString(w, `{"error":{"message":"bad request"}}`)
-	}))
-	defer server.Close()
-
-	client := newTestClient(t, server.URL, "default-model", 0)
-	var out testArtifact
-	_, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), &out)
-	if err == nil || !strings.Contains(err.Error(), "status 400: bad request") {
-		t.Fatalf("expected provider status error, got %v", err)
-	}
-}
-
-func TestOpenAICompatibleClientRetries429And5xx(t *testing.T) {
-	var attempts atomic.Int32
-	statuses := []int{http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusOK}
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		attempt := int(attempts.Add(1)) - 1
-		if statuses[attempt] != http.StatusOK {
-			w.WriteHeader(statuses[attempt])
-			_, _ = io.WriteString(w, `{"error":{"message":"try again"}}`)
-			return
-		}
-		_, _ = io.WriteString(w, `{"choices":[{"message":{"content":"{\"value\":\"ok\"}"}}]}`)
-	}))
-	defer server.Close()
-
-	client := newTestClient(t, server.URL, "default-model", 2)
-	var out testArtifact
-	if _, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), &out); err != nil {
-		t.Fatalf("CompleteStructured: %v", err)
-	}
-	if attempts.Load() != 3 {
-		t.Fatalf("expected 3 attempts, got %d", attempts.Load())
-	}
-}
-
-func TestOpenAICompatibleClientRetriesMalformedResponses(t *testing.T) {
-	tests := []struct {
-		name      string
-		firstBody string
-	}{
-		{
-			name:      "malformed provider envelope",
-			firstBody: `{"choices":[]}`,
-		},
-		{
-			name:      "malformed assistant JSON",
-			firstBody: `{"choices":[{"message":{"content":"{"}}]}`,
-		},
-	}
-
-	for _, tc := range tests {
-		t.Run(tc.name, func(t *testing.T) {
-			var attempts atomic.Int32
-			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				if attempts.Add(1) == 1 {
-					_, _ = io.WriteString(w, tc.firstBody)
-					return
-				}
-				_, _ = io.WriteString(w, `{"choices":[{"message":{"content":"{\"value\":\"ok\"}"}}]}`)
-			}))
-			defer server.Close()
-
-			client := newTestClient(t, server.URL, "default-model", 1)
-			var out testArtifact
-			if _, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), &out); err != nil {
-				t.Fatalf("CompleteStructured: %v", err)
-			}
-			if attempts.Load() != 2 {
-				t.Fatalf("expected 2 attempts, got %d", attempts.Load())
-			}
-		})
-	}
-}
-
-func TestOpenAICompatibleClientNoRetryForNonRetryable4xx(t *testing.T) {
-	var attempts atomic.Int32
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		attempts.Add(1)
-		w.WriteHeader(http.StatusForbidden)
-		_, _ = io.WriteString(w, `{"error":{"message":"forbidden"}}`)
-	}))
-	defer server.Close()
-
-	client := newTestClient(t, server.URL, "default-model", 3)
-	var out testArtifact
-	_, err := client.CompleteStructured(context.Background(), validStructuredRequest(""), &out)
-	if err == nil || !strings.Contains(err.Error(), "status 403") {
-		t.Fatalf("expected forbidden error, got %v", err)
-	}
-	if attempts.Load() != 1 {
-		t.Fatalf("expected 1 attempt, got %d", attempts.Load())
-	}
-}
-
-func TestOpenAICompatibleClientProviderErrorRedactsAPIKey(t *testing.T) {
-	const apiKey = "secret-api-key"
-	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		w.WriteHeader(http.StatusInternalServerError)
-		_, _ = io.WriteString(w, `{"error":{"message":"Bearer secret-api-key failed for secret-api-key"}}`)
-	}))
-	defer server.Close()
-
-	client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
-		BaseURL:    server.URL,
-		Model:      "default-model",
-		APIKey:     apiKey,
-		MaxRetries: 0,
-	})
-	if err != nil {
-		t.Fatalf("NewOpenAICompatibleClient: %v", err)
-	}
-
-	var out testArtifact
-	_, err = client.CompleteStructured(context.Background(), validStructuredRequest(""), &out)
-	if err == nil {
-		t.Fatalf("expected provider error")
-	}
-	if strings.Contains(err.Error(), apiKey) || strings.Contains(err.Error(), "Bearer "+apiKey) {
-		t.Fatalf("expected API key to be redacted, got %q", err.Error())
-	}
-}
-
-func TestOpenAICompatibleClientRespectsContextCancellation(t *testing.T) {
-	ctx, cancel := context.WithCancel(context.Background())
-	cancel()
-
-	client := newTestClient(t, "https://example.test/v1", "default-model", 1)
-	var out testArtifact
-	_, err := client.CompleteStructured(ctx, validStructuredRequest(""), &out)
-	if !errors.Is(err, context.Canceled) {
-		t.Fatalf("expected context canceled, got %v", err)
-	}
-}
-
-func newTestClient(t *testing.T, baseURL string, model string, maxRetries int) *OpenAICompatibleClient {
-	t.Helper()
-	client, err := NewOpenAICompatibleClient(OpenAICompatibleClientConfig{
-		BaseURL:    baseURL,
-		Model:      model,
-		MaxRetries: maxRetries,
-	})
-	if err != nil {
-		t.Fatalf("NewOpenAICompatibleClient: %v", err)
-	}
-	return client
-}
-
-func validStructuredRequest(model string) contracts.StructuredCompletionRequest {
-	return contracts.StructuredCompletionRequest{
-		Messages: []contracts.LLMMessage{
-			{Role: " user ", Content: " extract this "},
-		},
-		Model:              model,
-		ResponseSchemaName: " test_artifact ",
-		ResponseSchema:     testResponseSchema(),
-	}
-}
-
-func testResponseSchema() json.RawMessage {
-	return json.RawMessage(`{
-  "type": "object",
-  "properties": {
-    "value": {"type": "string"}
-  },
-  "required": ["value"],
-  "additionalProperties": false
-}`)
-}

internal/framework/llm/scheduled_client.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 
+	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
 	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
 )
 
@@ -41,3 +42,14 @@ func (c *scheduledClient) CompleteStructured(ctx context.Context, req contracts.
 	}
 	return response, nil
 }
+
+func (c *scheduledClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
+	if c == nil || c.client == nil {
+		return nil
+	}
+	provider, ok := c.client.(contracts.LLMProfileManifestProvider)
+	if !ok {
+		return nil
+	}
+	return provider.LLMProfileManifests()
+}

internal/framework/llm/scriptorium_api_test.go

@@ -0,0 +1,175 @@
+package llm
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"testing/fstest"
+	"time"
+
+	"gitea.maximumdirect.net/eric/scriptorium"
+)
+
+func TestScriptoriumPublicAPIGrounding(t *testing.T) {
+	// Keep this compile-time grounding close to the future Notarius adapter so
+	// dependency upgrades reveal API drift before the runtime cutover.
+	engine, err := scriptorium.NewEngine(
+		scriptorium.Config{
+			PromptDir:  "unused-when-prompt-option-is-set",
+			ProfileDir: "",
+			SchemaDir:  "",
+			Timeout:    time.Second,
+		},
+		scriptorium.WithPromptFS(fstest.MapFS{}, "."),
+		scriptorium.WithProfileFS(fstest.MapFS{}, "."),
+		scriptorium.WithSchemaFS(fstest.MapFS{}, "."),
+		scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
+			ID:             "test-profile",
+			Endpoint:       "http://127.0.0.1:1/v1",
+			Model:          "test-model",
+			APIKeyRequired: true,
+			ExtraParams:    map[string]any{"mode": "test"},
+		})),
+		scriptorium.WithLLMClient(scriptoriumGroundingLLMClient{}),
+	)
+	if err != nil {
+		t.Fatalf("NewEngine() error = %v, want nil", err)
+	}
+	if engine == nil {
+		t.Fatalf("NewEngine() = nil, want engine")
+	}
+
+	var (
+		_ func(string) scriptorium.Option = scriptorium.WithPromptFile
+		_ func(string) scriptorium.Option = scriptorium.WithProfileFile
+		_ func(string) scriptorium.Option = scriptorium.WithSchemaFile
+	)
+
+	req := scriptorium.RunRequest{
+		PromptID:      "dnd.spells",
+		PromptVersion: "v1",
+		ProfileID:     "test-profile",
+		APIKey:        "request-scoped-secret",
+		Inputs: map[string]scriptorium.ArtifactRef{
+			"transcript": scriptorium.InlineWithURI("file:///tmp/transcript.json", `{"segments":[]}`),
+			"glossary":   scriptorium.Inline(""),
+			"roster":     scriptorium.File("/tmp/roster.txt"),
+		},
+		Vars: map[string]string{
+			"session_id": "session-1",
+		},
+		Execution: &scriptorium.ExecutionTargetOverride{
+			Model:           "override-model",
+			Temperature:     ptr(0.2),
+			MaxTokens:       ptr(100),
+			TopP:            ptr(0.9),
+			TimeoutSeconds:  ptr(30),
+			ServiceTier:     "standard",
+			ReasoningEffort: "low",
+			APIKeyEnv:       "SCRIPTORIUM_API_KEY",
+			ExtraParams:     map[string]any{"provider_option": "value"},
+		},
+		Validation: &scriptorium.OutputContract{
+			Format:         scriptorium.FormatJSON,
+			ValidationMode: scriptorium.ValidationJSONSchema,
+			SchemaPath:     "schemas/dnd_spells.v1.json",
+			RepairAttempts: 1,
+		},
+		Metadata: map[string]string{
+			"artifact_kind": "dnd_spell",
+		},
+	}
+	if req.Inputs["transcript"].Type != scriptorium.ArtifactRefInline {
+		t.Fatalf("inline input type = %q, want %q", req.Inputs["transcript"].Type, scriptorium.ArtifactRefInline)
+	}
+	if req.Inputs["roster"].Type != scriptorium.ArtifactRefFile {
+		t.Fatalf("file input type = %q, want %q", req.Inputs["roster"].Type, scriptorium.ArtifactRefFile)
+	}
+
+	result := scriptorium.RunResult{
+		RunID: "run-1",
+		Artifact: scriptorium.Artifact{
+			Name:        "output",
+			ContentType: "application/json",
+			Body:        []byte(`{"ok":true}`),
+			URI:         "inline://output",
+			Size:        int64(len(`{"ok":true}`)),
+			Hash:        "sha256:abc",
+		},
+		RawOutput:          `{"ok":true}`,
+		PromptID:           req.PromptID,
+		PromptVersion:      req.PromptVersion,
+		PromptHash:         "prompt-hash",
+		RenderedPromptHash: "rendered-prompt-hash",
+		SelectedProfileID:  req.ProfileID,
+		ModelName:          "test-model",
+		Endpoint:           "http://127.0.0.1:1/v1",
+		EffectiveModelParams: scriptorium.ExecutionTarget{
+			Model:           "test-model",
+			APIKeyEnv:       "SCRIPTORIUM_API_KEY",
+			ExtraParams:     map[string]any{"provider_option": "value"},
+			ReasoningEffort: "low",
+		},
+		InputHashes: map[string]string{
+			"transcript": "sha256:def",
+		},
+		Validation: scriptorium.ValidationResult{
+			Status:         scriptorium.ValidationPassed,
+			Mode:           scriptorium.ValidationJSONSchema,
+			SchemaPath:     req.Validation.SchemaPath,
+			RepairAttempts: 1,
+			IsValid:        true,
+		},
+		Usage: scriptorium.TokenUsage{
+			PromptTokens:     10,
+			CompletionTokens: 5,
+			TotalTokens:      15,
+			CachedTokens:     3,
+			CacheWriteTokens: 2,
+		},
+		StartTime: time.Unix(1, 0),
+		EndTime:   time.Unix(2, 0),
+		Duration:  time.Second,
+	}
+	if result.Validation.Status != scriptorium.ValidationPassed {
+		t.Fatalf("validation status = %q, want %q", result.Validation.Status, scriptorium.ValidationPassed)
+	}
+	if result.Usage.TotalTokens != 15 {
+		t.Fatalf("total tokens = %d, want 15", result.Usage.TotalTokens)
+	}
+
+	publicErrors := []error{
+		scriptorium.ErrInvalidConfig,
+		scriptorium.ErrInvalidRequest,
+		scriptorium.ErrPromptNotFound,
+		scriptorium.ErrProfileNotFound,
+		scriptorium.ErrPromptLoad,
+		scriptorium.ErrProfileLoad,
+		scriptorium.ErrArtifactLoad,
+		scriptorium.ErrPromptRender,
+		scriptorium.ErrLLMGenerate,
+		scriptorium.ErrValidation,
+	}
+	for _, publicErr := range publicErrors {
+		if !errors.Is(publicErr, publicErr) {
+			t.Fatalf("sentinel error does not match itself: %v", publicErr)
+		}
+	}
+}
+
+type scriptoriumGroundingLLMClient struct{}
+
+func (scriptoriumGroundingLLMClient) Generate(context.Context, scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
+	return &scriptorium.GenerateResponse{
+		Content: `{"ok":true}`,
+		Usage: scriptorium.TokenUsage{
+			PromptTokens:     1,
+			CompletionTokens: 1,
+			TotalTokens:      2,
+		},
+	}, nil
+}
+
+func ptr[T any](v T) *T {
+	return &v
+}

internal/framework/llm/scriptorium_client.go

@@ -0,0 +1,379 @@
+package llm
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"regexp"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+	"gitea.maximumdirect.net/eric/scriptorium"
+)
+
+const scriptoriumProviderName = "scriptorium"
+
+type ScriptoriumClientConfig struct {
+	ProfileDir    string
+	ProfileFile   string
+	Assets        *AssetRegistry
+	Timeout       time.Duration
+	HTTPClient    *http.Client
+	EngineOptions []scriptorium.Option
+	Recorder      *LLMProfileRecorder
+}
+
+type ScriptoriumClient struct {
+	engine   *scriptorium.Engine
+	recorder *LLMProfileRecorder
+}
+
+type LLMProfileRecorder struct {
+	mu       sync.Mutex
+	profiles map[string]artifacts.LLMProfileManifest
+}
+
+var _ contracts.StructuredLLMClient = (*ScriptoriumClient)(nil)
+var _ contracts.LLMProfileManifestProvider = (*ScriptoriumClient)(nil)
+
+func NewScriptoriumClient(cfg ScriptoriumClientConfig) (*ScriptoriumClient, error) {
+	if cfg.Assets == nil {
+		return nil, fmt.Errorf("scriptorium client assets must not be nil")
+	}
+	if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
+		return nil, fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
+	}
+	options, err := cfg.Assets.ScriptoriumOptions()
+	if err != nil {
+		return nil, err
+	}
+	if profileFile := strings.TrimSpace(cfg.ProfileFile); profileFile != "" {
+		options = append(options, scriptorium.WithProfileFile(profileFile))
+	}
+	options = append(options, cfg.EngineOptions...)
+
+	engine, err := scriptorium.NewEngine(scriptorium.Config{
+		ProfileDir: strings.TrimSpace(cfg.ProfileDir),
+		Timeout:    cfg.Timeout,
+		HTTPClient: cfg.HTTPClient,
+	}, options...)
+	if err != nil {
+		return nil, fmt.Errorf("create Scriptorium engine: %w", err)
+	}
+	recorder := cfg.Recorder
+	if recorder == nil {
+		recorder = NewLLMProfileRecorder()
+	}
+	return &ScriptoriumClient{
+		engine:   engine,
+		recorder: recorder,
+	}, nil
+}
+
+func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
+	if c == nil {
+		return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client must not be nil")
+	}
+	if c.engine == nil {
+		return contracts.StructuredCompletionResponse{}, fmt.Errorf("scriptorium client engine must not be nil")
+	}
+	if err := validateOutputTarget(out); err != nil {
+		return contracts.StructuredCompletionResponse{}, err
+	}
+	promptID := strings.TrimSpace(req.PromptID)
+	if promptID == "" {
+		return contracts.StructuredCompletionResponse{}, fmt.Errorf("structured completion prompt_id must not be empty")
+	}
+
+	runReq := scriptorium.RunRequest{
+		PromptID:      promptID,
+		PromptVersion: strings.TrimSpace(req.PromptVersion),
+		ProfileID:     strings.TrimSpace(req.ProfileID),
+		Inputs:        scriptoriumInputs(req.Inputs),
+		Vars:          scriptoriumVars(req),
+		Metadata:      scriptoriumMetadata(req),
+	}
+	prepared, err := c.engine.Prepare(ctx, runReq)
+	if err != nil {
+		if ctxErr := ctx.Err(); ctxErr != nil {
+			return contracts.StructuredCompletionResponse{}, ctxErr
+		}
+		return contracts.StructuredCompletionResponse{}, fmt.Errorf("prepare Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
+	}
+	result, err := c.engine.Run(ctx, runReq)
+	if err != nil {
+		if ctxErr := ctx.Err(); ctxErr != nil {
+			return contracts.StructuredCompletionResponse{}, ctxErr
+		}
+		return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
+	}
+	if result == nil {
+		return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: empty result", promptID)
+	}
+	response := c.responseFromResult(result, prepared)
+	if result.Validation.Status == scriptorium.ValidationFailed || !result.Validation.IsValid {
+		return response, fmt.Errorf("run Scriptorium prompt %q: validation failed: %s", promptID, strings.Join(result.Validation.Errors, "; "))
+	}
+	if len(strings.TrimSpace(string(response.Content))) == 0 {
+		return response, fmt.Errorf("run Scriptorium prompt %q: empty structured output", promptID)
+	}
+	if err := json.Unmarshal(response.Content, out); err != nil {
+		return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w", promptID, err)
+	}
+	return response, nil
+}
+
+func (c *ScriptoriumClient) responseFromResult(result *scriptorium.RunResult, prepared *scriptorium.PreparedRun) contracts.StructuredCompletionResponse {
+	content := result.Artifact.Body
+	if len(content) == 0 {
+		content = []byte(result.RawOutput)
+	}
+	profile := artifacts.LLMProfileManifest{
+		ID:       strings.TrimSpace(result.SelectedProfileID),
+		Provider: scriptoriumProviderName,
+		Model:    firstNonEmpty(result.ModelName, result.EffectiveModelParams.Model),
+	}
+	if c.recorder != nil {
+		c.recorder.Record(profile)
+	}
+	return contracts.StructuredCompletionResponse{
+		Content:          append(json.RawMessage(nil), content...),
+		Provider:         profile.Provider,
+		Model:            profile.Model,
+		ProfileID:        profile.ID,
+		PromptTokens:     result.Usage.PromptTokens,
+		CompletionTokens: result.Usage.CompletionTokens,
+		TotalTokens:      result.Usage.TotalTokens,
+		Debug:            scriptoriumDebugMaterial(prepared, result),
+	}
+}
+
+func scriptoriumDebugMaterial(prepared *scriptorium.PreparedRun, result *scriptorium.RunResult) *contracts.LLMDebugMaterial {
+	material := &contracts.LLMDebugMaterial{}
+	if prepared != nil {
+		material.Prompt = scriptoriumDebugPrompt(prepared)
+	}
+	if result != nil {
+		material.Response = scriptoriumDebugResponse(result)
+	}
+	if material.Prompt == nil && material.Response == nil {
+		return nil
+	}
+	return material
+}
+
+func scriptoriumDebugPrompt(prepared *scriptorium.PreparedRun) *contracts.LLMDebugPrompt {
+	if prepared == nil {
+		return nil
+	}
+	messages := make([]contracts.LLMDebugMessage, 0, len(prepared.Messages))
+	for _, message := range prepared.Messages {
+		messages = append(messages, contracts.LLMDebugMessage{
+			Role:         message.Role,
+			Content:      message.Content,
+			CacheControl: jsonObject(message.CacheControl),
+		})
+	}
+	return &contracts.LLMDebugPrompt{
+		PromptID:             prepared.PromptID,
+		PromptVersion:        prepared.PromptVersion,
+		PromptHash:           prepared.PromptHash,
+		SelectedProfileID:    prepared.SelectedProfileID,
+		SessionID:            prepared.SessionID,
+		RenderedPromptHash:   prepared.RenderedPromptHash,
+		Messages:             messages,
+		EffectiveModelParams: jsonObject(prepared.EffectiveModelParams),
+		OutputContract:       jsonObject(prepared.OutputContract),
+		StructuredOutput:     jsonObject(prepared.StructuredOutput),
+		InputHashes:          cloneStringMap(prepared.InputHashes),
+	}
+}
+
+func scriptoriumDebugResponse(result *scriptorium.RunResult) *contracts.LLMDebugResponse {
+	if result == nil {
+		return nil
+	}
+	content := result.RawOutput
+	if content == "" {
+		content = string(result.Artifact.Body)
+	}
+	return &contracts.LLMDebugResponse{
+		Content:              content,
+		RunID:                result.RunID,
+		PromptID:             result.PromptID,
+		PromptVersion:        result.PromptVersion,
+		PromptHash:           result.PromptHash,
+		RenderedPromptHash:   result.RenderedPromptHash,
+		SelectedProfileID:    result.SelectedProfileID,
+		ModelName:            result.ModelName,
+		Endpoint:             result.Endpoint,
+		EffectiveModelParams: jsonObject(result.EffectiveModelParams),
+		InputHashes:          cloneStringMap(result.InputHashes),
+		Validation:           jsonObject(result.Validation),
+		Usage: contracts.LLMDebugUsage{
+			PromptTokens:     result.Usage.PromptTokens,
+			CompletionTokens: result.Usage.CompletionTokens,
+			TotalTokens:      result.Usage.TotalTokens,
+			CachedTokens:     result.Usage.CachedTokens,
+			CacheWriteTokens: result.Usage.CacheWriteTokens,
+		},
+	}
+}
+
+func jsonObject(value any) map[string]any {
+	if value == nil {
+		return nil
+	}
+	data, err := json.Marshal(value)
+	if err != nil || string(data) == "null" {
+		return nil
+	}
+	var out map[string]any
+	if err := json.Unmarshal(data, &out); err != nil {
+		return nil
+	}
+	if len(out) == 0 {
+		return nil
+	}
+	return out
+}
+
+func cloneStringMap(values map[string]string) map[string]string {
+	if len(values) == 0 {
+		return nil
+	}
+	out := make(map[string]string, len(values))
+	for key, value := range values {
+		out[key] = value
+	}
+	return out
+}
+
+func (c *ScriptoriumClient) LLMProfileManifests() []artifacts.LLMProfileManifest {
+	if c == nil || c.recorder == nil {
+		return nil
+	}
+	return c.recorder.Manifests()
+}
+
+func NewLLMProfileRecorder() *LLMProfileRecorder {
+	return &LLMProfileRecorder{profiles: map[string]artifacts.LLMProfileManifest{}}
+}
+
+func (r *LLMProfileRecorder) Record(profile artifacts.LLMProfileManifest) {
+	if r == nil {
+		return
+	}
+	profile.ID = strings.TrimSpace(profile.ID)
+	profile.Provider = strings.TrimSpace(profile.Provider)
+	profile.Model = strings.TrimSpace(profile.Model)
+	key := profile.ID + "\x00" + profile.Provider + "\x00" + profile.Model
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if r.profiles == nil {
+		r.profiles = map[string]artifacts.LLMProfileManifest{}
+	}
+	r.profiles[key] = profile
+}
+
+func (r *LLMProfileRecorder) Manifests() []artifacts.LLMProfileManifest {
+	if r == nil {
+		return nil
+	}
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	if len(r.profiles) == 0 {
+		return nil
+	}
+	keys := make([]string, 0, len(r.profiles))
+	for key := range r.profiles {
+		keys = append(keys, key)
+	}
+	sort.Strings(keys)
+	out := make([]artifacts.LLMProfileManifest, 0, len(keys))
+	for _, key := range keys {
+		out = append(out, r.profiles[key])
+	}
+	return out
+}
+
+func scriptoriumInputs(inputs contracts.LLMInputSet) map[string]scriptorium.ArtifactRef {
+	if len(inputs) == 0 {
+		return nil
+	}
+	out := make(map[string]scriptorium.ArtifactRef, len(inputs))
+	for key, material := range inputs {
+		name := strings.TrimSpace(key)
+		if name == "" {
+			name = strings.TrimSpace(material.Name)
+		}
+		if name == "" {
+			continue
+		}
+		body := string(material.Content)
+		if body == "" {
+			body = " "
+		}
+		if origin := strings.TrimSpace(material.OriginURI); origin != "" {
+			out[name] = scriptorium.InlineWithURI(origin, body)
+		} else {
+			out[name] = scriptorium.Inline(body)
+		}
+	}
+	return out
+}
+
+func scriptoriumVars(req contracts.StructuredCompletionRequest) map[string]string {
+	vars := make(map[string]string, len(req.Vars)+1)
+	for key, value := range req.Vars {
+		name := strings.TrimSpace(key)
+		if name == "" || value == nil {
+			continue
+		}
+		vars[name] = fmt.Sprint(value)
+	}
+	if sessionID := strings.TrimSpace(req.SessionID); sessionID != "" {
+		vars["session_id"] = sessionID
+	}
+	if len(vars) == 0 {
+		return nil
+	}
+	return vars
+}
+
+func scriptoriumMetadata(req contracts.StructuredCompletionRequest) map[string]string {
+	metadata := map[string]string{}
+	if stageName := strings.TrimSpace(req.StageName); stageName != "" {
+		metadata["stage_name"] = stageName
+	}
+	if len(metadata) == 0 {
+		return nil
+	}
+	return metadata
+}
+
+var bearerTokenPattern = regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._~+/=-]+`)
+
+func redactScriptoriumError(err error) error {
+	if err == nil {
+		return nil
+	}
+	return redactedProviderError{err: err}
+}
+
+type redactedProviderError struct {
+	err error
+}
+
+func (e redactedProviderError) Error() string {
+	return bearerTokenPattern.ReplaceAllString(e.err.Error(), "Bearer "+secretReplacement)
+}
+
+func (e redactedProviderError) Unwrap() error {
+	return e.err
+}

internal/framework/llm/scriptorium_client_test.go

@@ -0,0 +1,352 @@
+package llm
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"testing/fstest"
+	"time"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+	"gitea.maximumdirect.net/eric/scriptorium"
+)
+
+func TestScriptoriumClientMapsPromptRequestAndUnmarshalsOutput(t *testing.T) {
+	fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
+	client := newTestScriptoriumClient(t, fake)
+
+	var out struct {
+		OK bool `json:"ok"`
+	}
+	resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
+		StageName:     "test-stage",
+		PromptID:      "adapter.test",
+		PromptVersion: "v1",
+		ProfileID:     "explicit-profile",
+		SessionID:     "session-123",
+		Inputs: contracts.LLMInputSet{
+			"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "sha256:source", "file:///source.json"),
+		},
+		Vars: map[string]any{"custom": "value"},
+	}, &out)
+	if err != nil {
+		t.Fatalf("CompleteStructured() error = %v, want nil", err)
+	}
+	if !out.OK {
+		t.Fatalf("decoded output OK = false, want true")
+	}
+	if resp.Provider != scriptoriumProviderName || resp.Model != "explicit-model" || resp.ProfileID != "explicit-profile" {
+		t.Fatalf("response metadata = %#v", resp)
+	}
+	if resp.PromptTokens != 11 || resp.CompletionTokens != 7 || resp.TotalTokens != 18 {
+		t.Fatalf("usage = %#v, want mapped token counts", resp)
+	}
+	if resp.Debug == nil || resp.Debug.Prompt == nil {
+		t.Fatalf("debug prompt = nil, want prepared prompt material")
+	}
+	if resp.Debug.Prompt.PromptID != "adapter.test" || resp.Debug.Prompt.SelectedProfileID != "explicit-profile" {
+		t.Fatalf("debug prompt metadata = %#v, want prompt/profile", resp.Debug.Prompt)
+	}
+	if len(resp.Debug.Prompt.Messages) != 1 || !strings.Contains(resp.Debug.Prompt.Messages[0].Content, `{"source":true}`) {
+		t.Fatalf("debug prompt messages = %#v, want rendered input content", resp.Debug.Prompt.Messages)
+	}
+	if resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
+		t.Fatalf("debug response = %#v, want raw response content", resp.Debug.Response)
+	}
+	debugJSON, err := json.Marshal(resp.Debug)
+	if err != nil {
+		t.Fatalf("marshal debug material: %v", err)
+	}
+	if strings.Contains(string(debugJSON), "secret-token") || strings.Contains(string(debugJSON), "sk-") {
+		t.Fatalf("debug material contains secret material: %s", debugJSON)
+	}
+	gotReq := fake.lastRequest()
+	if gotReq.Prompt.SessionID != "session-123" {
+		t.Fatalf("session id = %q, want session-123", gotReq.Prompt.SessionID)
+	}
+	if gotReq.Target.Model != "explicit-model" {
+		t.Fatalf("model = %q, want explicit-model", gotReq.Target.Model)
+	}
+	if len(gotReq.Prompt.Messages) != 1 || !strings.Contains(gotReq.Prompt.Messages[0].Content, `{"source":true}`) {
+		t.Fatalf("rendered messages = %#v, want transcript input content", gotReq.Prompt.Messages)
+	}
+	if gotReq.StructuredOutput == nil {
+		t.Fatalf("structured output = nil, want JSON schema")
+	}
+	manifests := client.LLMProfileManifests()
+	if len(manifests) != 1 || manifests[0].ID != "explicit-profile" || manifests[0].Model != "explicit-model" {
+		t.Fatalf("profile manifests = %#v", manifests)
+	}
+}
+
+func TestScriptoriumClientUsesPromptDefaultProfileWhenRequestProfileEmpty(t *testing.T) {
+	fake := &fakeScriptoriumLLM{content: `{"ok":true}`}
+	client := newTestScriptoriumClient(t, fake)
+
+	var out map[string]any
+	if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
+		PromptID:  "adapter.test",
+		SessionID: "session-123",
+		Inputs: contracts.LLMInputSet{
+			"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
+		},
+	}, &out); err != nil {
+		t.Fatalf("CompleteStructured() error = %v, want nil", err)
+	}
+	if got := fake.lastRequest().Target.Model; got != "default-model" {
+		t.Fatalf("model = %q, want prompt default profile model", got)
+	}
+}
+
+func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
+	client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"bad":true}`})
+
+	var out map[string]any
+	resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
+		PromptID:  "adapter.test",
+		SessionID: "session-123",
+		Inputs: contracts.LLMInputSet{
+			"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
+		},
+	}, &out)
+	if err == nil || !strings.Contains(err.Error(), "validation failed") {
+		t.Fatalf("CompleteStructured() error = %v, want validation failure", err)
+	}
+	if got := string(resp.Content); got != `{"bad":true}` {
+		t.Fatalf("response content = %q, want raw failed output", got)
+	}
+	if resp.Debug == nil || resp.Debug.Response == nil || resp.Debug.Response.Content != `{"bad":true}` {
+		t.Fatalf("debug response = %#v, want raw failed output", resp.Debug)
+	}
+	if resp.Debug.Prompt == nil || len(resp.Debug.Prompt.Messages) == 0 {
+		t.Fatalf("debug prompt = %#v, want prepared prompt material", resp.Debug.Prompt)
+	}
+}
+
+func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
+	client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
+
+	var out []any
+	resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
+		PromptID:  "adapter.test",
+		SessionID: "session-123",
+		Inputs: contracts.LLMInputSet{
+			"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
+		},
+	}, &out)
+	if err == nil || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
+		t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
+	}
+	if got := string(resp.Content); got != `{"ok":true}` {
+		t.Fatalf("response content = %q, want raw decode-failed output", got)
+	}
+	if resp.Debug == nil || resp.Debug.Response == nil || resp.Debug.Response.Content != `{"ok":true}` {
+		t.Fatalf("debug response = %#v, want raw decode-failed output", resp.Debug)
+	}
+}
+
+func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t *testing.T) {
+	client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{err: errors.New("provider failed with Bearer secret-token")})
+
+	var out map[string]any
+	resp, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
+		PromptID:  "adapter.test",
+		SessionID: "session-123",
+		Inputs: contracts.LLMInputSet{
+			"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
+		},
+	}, &out)
+	if err == nil {
+		t.Fatalf("CompleteStructured() error = nil, want provider error")
+	}
+	if !strings.Contains(err.Error(), `run Scriptorium prompt "adapter.test"`) {
+		t.Fatalf("error = %q, want operation context", err.Error())
+	}
+	if strings.Contains(err.Error(), "secret-token") || !strings.Contains(err.Error(), "Bearer [REDACTED]") {
+		t.Fatalf("error = %q, want redacted bearer token", err.Error())
+	}
+	if resp.Debug != nil {
+		t.Fatalf("debug material = %#v, want none for provider failure without result", resp.Debug)
+	}
+}
+
+func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
+
+	var out map[string]any
+	_, err := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
+		PromptID: "adapter.test",
+		Inputs: contracts.LLMInputSet{
+			"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
+		},
+	}, &out)
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
+	}
+}
+
+func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
+	fake := &fakeScriptoriumLLM{
+		content: `{"ok":true}`,
+		block:   make(chan struct{}),
+	}
+	client := newTestScriptoriumClient(t, fake)
+	scheduler, err := NewScheduler(1)
+	if err != nil {
+		t.Fatalf("NewScheduler() error = %v, want nil", err)
+	}
+	scheduled := NewScheduledClient(client, scheduler)
+
+	var wg sync.WaitGroup
+	for i := 0; i < 3; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			var out map[string]any
+			_, callErr := scheduled.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
+				PromptID:  "adapter.test",
+				SessionID: "session-123",
+				Inputs: contracts.LLMInputSet{
+					"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
+				},
+			}, &out)
+			if callErr != nil {
+				t.Errorf("CompleteStructured() error = %v, want nil", callErr)
+			}
+		}()
+	}
+	waitForAtomicAtLeast(t, &fake.calls, 1)
+	time.Sleep(20 * time.Millisecond)
+	if got := atomic.LoadInt32(&fake.maxInFlight); got > 1 {
+		t.Fatalf("max in-flight calls = %d, want <= 1", got)
+	}
+	close(fake.block)
+	wg.Wait()
+}
+
+func TestScriptoriumClientValidatesRequest(t *testing.T) {
+	client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{content: `{"ok":true}`})
+	var out map[string]any
+	if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err == nil || !strings.Contains(err.Error(), "prompt_id") {
+		t.Fatalf("missing prompt id error = %v, want prompt_id validation", err)
+	}
+	if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{PromptID: "adapter.test"}, nil); err == nil || !strings.Contains(err.Error(), "non-nil pointer") {
+		t.Fatalf("nil output error = %v, want output validation", err)
+	}
+}
+
+func newTestScriptoriumClient(t *testing.T, fake *fakeScriptoriumLLM) *ScriptoriumClient {
+	t.Helper()
+	registry := NewAssetRegistry()
+	if err := registry.RegisterPromptFS(fstest.MapFS{
+		"adapter.test.yaml": {Data: []byte(`id: adapter.test
+version: "v1"
+default_profile: default-profile
+session_id: "{{ .session_id }}"
+inputs:
+  - name: transcript
+    required: true
+    content_type: application/json
+messages:
+  - role: user
+    content: "Transcript: {{ input \"transcript\" }}"
+output:
+  format: json
+  validation_mode: json_schema
+  schema_path: adapter.schema.json
+  repair_attempts: 0
+`)},
+	}, "."); err != nil {
+		t.Fatalf("RegisterPromptFS() error = %v", err)
+	}
+	if err := registry.RegisterSchemaFS(fstest.MapFS{
+		"adapter.schema.json": {Data: []byte(`{"type":"object","required":["ok"],"properties":{"ok":{"type":"boolean"}}}`)},
+	}, "."); err != nil {
+		t.Fatalf("RegisterSchemaFS() error = %v", err)
+	}
+	client, err := NewScriptoriumClient(ScriptoriumClientConfig{
+		Assets: registry,
+		EngineOptions: []scriptorium.Option{
+			scriptorium.WithProfiles(
+				scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
+					ID:       "default-profile",
+					Endpoint: "http://127.0.0.1:1/v1",
+					Model:    "default-model",
+				}),
+				scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
+					ID:       "explicit-profile",
+					Endpoint: "http://127.0.0.1:1/v1",
+					Model:    "explicit-model",
+				}),
+			),
+			scriptorium.WithLLMClient(fake),
+		},
+	})
+	if err != nil {
+		t.Fatalf("NewScriptoriumClient() error = %v, want nil", err)
+	}
+	return client
+}
+
+type fakeScriptoriumLLM struct {
+	content     string
+	err         error
+	block       chan struct{}
+	mu          sync.Mutex
+	last        scriptorium.GenerateRequest
+	calls       int32
+	inFlight    int32
+	maxInFlight int32
+}
+
+func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.GenerateRequest) (*scriptorium.GenerateResponse, error) {
+	f.mu.Lock()
+	f.last = req
+	f.mu.Unlock()
+	atomic.AddInt32(&f.calls, 1)
+	current := atomic.AddInt32(&f.inFlight, 1)
+	for {
+		seen := atomic.LoadInt32(&f.maxInFlight)
+		if current <= seen || atomic.CompareAndSwapInt32(&f.maxInFlight, seen, current) {
+			break
+		}
+	}
+	defer atomic.AddInt32(&f.inFlight, -1)
+	if f.block != nil {
+		select {
+		case <-f.block:
+		case <-ctx.Done():
+			return nil, ctx.Err()
+		}
+	}
+	if f.err != nil {
+		return nil, f.err
+	}
+	content := f.content
+	if content == "" {
+		content = `{"ok":true}`
+	}
+	if !json.Valid([]byte(content)) {
+		return nil, errors.New("test fake must return JSON content")
+	}
+	return &scriptorium.GenerateResponse{
+		Content: content,
+		Usage: scriptorium.TokenUsage{
+			PromptTokens:     11,
+			CompletionTokens: 7,
+			TotalTokens:      18,
+		},
+	}, nil
+}
+
+func (f *fakeScriptoriumLLM) lastRequest() scriptorium.GenerateRequest {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return f.last
+}

internal/framework/pipeline/artifact_codec_registry.go

@@ -0,0 +1,315 @@
+package pipeline
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"reflect"
+	"sort"
+	"strings"
+
+	"github.com/santhosh-tekuri/jsonschema/v6"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
+)
+
+type ArtifactCodecSpec struct {
+	Kind         contracts.ArtifactKind
+	Schema       contracts.ArtifactSchema
+	SchemaDigest string
+	MediaType    string
+}
+
+// ArtifactCodecTypeError reports a value that does not have the exact Go type
+// registered for an artifact kind.
+type ArtifactCodecTypeError struct {
+	Operation    string
+	Kind         contracts.ArtifactKind
+	ExpectedType string
+	ActualType   string
+}
+
+func (e *ArtifactCodecTypeError) Error() string {
+	return fmt.Sprintf("%s artifact %q: expected exact Go type %s, got %s", e.Operation, e.Kind, e.ExpectedType, e.ActualType)
+}
+
+// ArtifactCodecCompatibilityError reports serialized metadata that does not
+// identify the registered representation for an artifact kind.
+type ArtifactCodecCompatibilityError struct {
+	Kind   contracts.ArtifactKind
+	Reason string
+}
+
+func (e *ArtifactCodecCompatibilityError) Error() string {
+	return fmt.Sprintf("decode artifact %q: %s", e.Kind, e.Reason)
+}
+
+// ArtifactCodecOperationError preserves an encode or decode failure from the
+// registered domain codec.
+type ArtifactCodecOperationError struct {
+	Operation string
+	Kind      contracts.ArtifactKind
+	Err       error
+}
+
+func (e *ArtifactCodecOperationError) Error() string {
+	return fmt.Sprintf("%s artifact %q: %v", e.Operation, e.Kind, e.Err)
+}
+
+func (e *ArtifactCodecOperationError) Unwrap() error { return e.Err }
+
+type ArtifactCodecRegistry struct {
+	entries map[contracts.ArtifactKind]artifactCodecEntry
+}
+
+type artifactCodecEntry struct {
+	spec            ArtifactCodecSpec
+	valueType       reflect.Type
+	encode          func(any) ([]byte, error)
+	encodeCandidate func(any) ([]byte, error)
+	metadata        func(any) (map[string]any, error)
+	decode          func([]byte) (any, error)
+}
+
+func NewArtifactCodecRegistry() *ArtifactCodecRegistry {
+	return &ArtifactCodecRegistry{entries: make(map[contracts.ArtifactKind]artifactCodecEntry)}
+}
+
+// RegisterArtifactCodec registers one codec for T. The concrete type is kept
+// private and checked at every erased encode boundary.
+func RegisterArtifactCodec[T any](registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) error {
+	if registry == nil {
+		return fmt.Errorf("artifact codec registry must not be nil")
+	}
+	if nilInterface(codec) {
+		return fmt.Errorf("artifact codec must not be nil")
+	}
+
+	spec, err := artifactCodecSpec(codec.Kind(), codec.Schema(), codec.MediaType())
+	if err != nil {
+		return err
+	}
+	if _, ok := registry.entries[spec.Kind]; ok {
+		return fmt.Errorf("artifact codec %q is already registered", spec.Kind)
+	}
+
+	valueType := reflect.TypeFor[T]()
+	entry := artifactCodecEntry{
+		spec:      cloneArtifactCodecSpec(spec),
+		valueType: valueType,
+		encode: func(value any) ([]byte, error) {
+			actualType := reflect.TypeOf(value)
+			if actualType != valueType {
+				return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
+			}
+			typed, ok := value.(T)
+			if !ok {
+				return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
+			}
+			encoded, err := codec.Encode(typed)
+			if err != nil {
+				return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
+			}
+			return append([]byte(nil), encoded...), nil
+		},
+		decode: func(content []byte) (any, error) {
+			decoded, err := codec.Decode(append([]byte(nil), content...))
+			if err != nil {
+				return nil, &ArtifactCodecOperationError{Operation: "decode", Kind: spec.Kind, Err: err}
+			}
+			return decoded, nil
+		},
+	}
+	entry.encodeCandidate = func(value any) ([]byte, error) {
+		typed, err := exactTypedValue[T]("encode candidate artifact", value)
+		if err != nil {
+			return nil, err
+		}
+		content, err := codec.EncodeCandidate(typed)
+		if err != nil {
+			return nil, &ArtifactCodecOperationError{Operation: "encode candidate", Kind: spec.Kind, Err: err}
+		}
+		return append([]byte(nil), content...), nil
+	}
+	if provider, ok := any(codec).(interface{ Metadata(T) map[string]any }); ok {
+		entry.metadata = func(value any) (map[string]any, error) {
+			typed, err := exactTypedValue[T]("artifact metadata", value)
+			if err != nil {
+				return nil, err
+			}
+			return cloneMetadata(provider.Metadata(typed))
+		}
+	}
+	if registry.entries == nil {
+		registry.entries = make(map[contracts.ArtifactKind]artifactCodecEntry)
+	}
+	registry.entries[spec.Kind] = entry
+	return nil
+}
+
+func (r *ArtifactCodecRegistry) Spec(kind contracts.ArtifactKind) (ArtifactCodecSpec, bool) {
+	if r == nil {
+		return ArtifactCodecSpec{}, false
+	}
+	entry, ok := r.entries[normalizeArtifactKind(kind)]
+	if !ok {
+		return ArtifactCodecSpec{}, false
+	}
+	return cloneArtifactCodecSpec(entry.spec), true
+}
+
+func (r *ArtifactCodecRegistry) valueType(kind contracts.ArtifactKind) (reflect.Type, bool) {
+	if r == nil {
+		return nil, false
+	}
+	entry, ok := r.entries[normalizeArtifactKind(kind)]
+	if !ok {
+		return nil, false
+	}
+	return entry.valueType, true
+}
+
+func (r *ArtifactCodecRegistry) RegisteredKinds() []contracts.ArtifactKind {
+	if r == nil || len(r.entries) == 0 {
+		return nil
+	}
+	kinds := make([]contracts.ArtifactKind, 0, len(r.entries))
+	for kind := range r.entries {
+		kinds = append(kinds, kind)
+	}
+	sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
+	return kinds
+}
+
+// Encode serializes an erased framework value after proving its exact
+// registered Go type.
+func (r *ArtifactCodecRegistry) Encode(kind contracts.ArtifactKind, value any) (contracts.SerializedArtifact, error) {
+	entry, normalizedKind, err := r.entry(kind)
+	if err != nil {
+		return contracts.SerializedArtifact{}, err
+	}
+	content, err := entry.encode(value)
+	if err != nil {
+		return contracts.SerializedArtifact{}, err
+	}
+	return contracts.SerializedArtifact{
+		Kind:      normalizedKind,
+		Schema:    contracts.CloneArtifactSchema(entry.spec.Schema),
+		MediaType: entry.spec.MediaType,
+		Content:   content,
+	}, nil
+}
+
+// Decode verifies serialized identity before invoking the registered codec.
+func (r *ArtifactCodecRegistry) Decode(artifact contracts.SerializedArtifact) (any, error) {
+	entry, normalizedKind, err := r.entry(artifact.Kind)
+	if err != nil {
+		return nil, err
+	}
+	if artifact.Schema.ID != entry.spec.Schema.ID || artifact.Schema.Name != entry.spec.Schema.Name || artifact.Schema.Version != entry.spec.Schema.Version {
+		return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema identity does not match registered codec"}
+	}
+	if contracts.DigestArtifactSchema(artifact.Schema) != entry.spec.SchemaDigest {
+		return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema digest does not match registered codec"}
+	}
+	if strings.TrimSpace(artifact.MediaType) != entry.spec.MediaType {
+		return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "media type does not match registered codec"}
+	}
+	return entry.decode(artifact.Content)
+}
+
+func (r *ArtifactCodecRegistry) entry(kind contracts.ArtifactKind) (artifactCodecEntry, contracts.ArtifactKind, error) {
+	if r == nil {
+		return artifactCodecEntry{}, "", fmt.Errorf("artifact codec registry must not be nil")
+	}
+	normalizedKind := normalizeArtifactKind(kind)
+	if normalizedKind == "" {
+		return artifactCodecEntry{}, "", fmt.Errorf("artifact kind must not be empty")
+	}
+	entry, ok := r.entries[normalizedKind]
+	if !ok {
+		return artifactCodecEntry{}, normalizedKind, fmt.Errorf("artifact codec %q is not registered", normalizedKind)
+	}
+	return entry, normalizedKind, nil
+}
+
+func artifactCodecSpec(kind contracts.ArtifactKind, schema contracts.ArtifactSchema, mediaType string) (ArtifactCodecSpec, error) {
+	kind = normalizeArtifactKind(kind)
+	schema.ID = strings.TrimSpace(schema.ID)
+	schema.Name = strings.TrimSpace(schema.Name)
+	schema.Version = strings.TrimSpace(schema.Version)
+	mediaType = strings.TrimSpace(mediaType)
+	switch {
+	case kind == "":
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec kind must not be empty")
+	case schema.ID == "":
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema id must not be empty", kind)
+	case schema.Name == "":
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema name must not be empty", kind)
+	case schema.Version == "":
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema version must not be empty", kind)
+	case mediaType == "":
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q media type must not be empty", kind)
+	case len(bytes.TrimSpace(schema.JSONSchema)) == 0:
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must not be empty", kind)
+	case !json.Valid(schema.JSONSchema):
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must be valid JSON", kind)
+	}
+	schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(schema.JSONSchema))
+	if err != nil {
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
+	}
+	compiler := jsonschema.NewCompiler()
+	if err := compiler.AddResource("artifact-schema.json", schemaDocument); err != nil {
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
+	}
+	if _, err := compiler.Compile("artifact-schema.json"); err != nil {
+		return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
+	}
+	schema = contracts.CloneArtifactSchema(schema)
+	return ArtifactCodecSpec{
+		Kind:         kind,
+		Schema:       schema,
+		SchemaDigest: contracts.DigestArtifactSchema(schema),
+		MediaType:    mediaType,
+	}, nil
+}
+
+func normalizeArtifactKind(kind contracts.ArtifactKind) contracts.ArtifactKind {
+	return contracts.ArtifactKind(strings.TrimSpace(string(kind)))
+}
+
+func cloneArtifactCodecSpec(spec ArtifactCodecSpec) ArtifactCodecSpec {
+	spec.Schema = contracts.CloneArtifactSchema(spec.Schema)
+	return spec
+}
+
+func nilInterface(value any) bool {
+	if value == nil {
+		return true
+	}
+	reflected := reflect.ValueOf(value)
+	switch reflected.Kind() {
+	case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
+		return reflected.IsNil()
+	default:
+		return false
+	}
+}
+
+func newArtifactCodecTypeError(operation string, kind contracts.ArtifactKind, expected reflect.Type, actual reflect.Type) error {
+	expectedName := "<nil>"
+	if expected != nil {
+		expectedName = expected.String()
+	}
+	actualName := "<nil>"
+	if actual != nil {
+		actualName = actual.String()
+	}
+	return &ArtifactCodecTypeError{
+		Operation:    operation,
+		Kind:         kind,
+		ExpectedType: expectedName,
+		ActualType:   actualName,
+	}
+}