diff --git a/docs/internal/overview.md b/docs/internal/overview.md index bbe34d8..dcae244 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -1,195 +1,56 @@ # Internal Overview -This document inventories the implemented Notarius components. Normative +This document is the implemented component map for Notarius. Normative boundaries and dependency direction belong in -[Architecture](../policy/architecture.md); external behavior belongs in the -[CLI](../cli.md), [Configuration](../config.md), +[Architecture](../policy/architecture.md). User and operator contracts belong +in the [CLI](../cli.md), [Configuration](../config.md), [Operations](../operations.md), and [integration contracts](../integrations/). ## Execution Path -`cmd/notarius` delegates to `internal/cli`, the production composition root. -The CLI loads configuration, builds the production catalogs and runtime -collaborators, invokes `internal/framework/pipeline`, and places the logical -output files returned by the runner. Cache and debug collaborators are supplied -at this boundary. +~~~ +cmd/notarius -> internal/cli -> configuration and production composition + -> internal/framework/pipeline -> logical output files + -> internal/cli -> durable output and optional state/debug data +~~~ -Resolution produces a fixed ordered workflow of steps and globally unique, -sorted artifact lanes. Preparation constructs the complete module and validator -set before the runner receives source bytes. Source parsing and chunking are -serial. Each step then uses a bounded run-wide extraction pool followed by -serial per-lane merge and normalize continuations. A step barrier prevents -later consumers from starting until all earlier lanes are terminal and their -required normalized artifacts have crossed the typed handoff. +The CLI is the application boundary: it discovers configuration, composes +production registries and runtime collaborators, invokes the framework, and +places returned files. The framework resolves and prepares a fixed extraction +pipeline, then returns logical results without owning process behavior or +physical state roots. -## Application Boundary +## Components -| Package | Implemented responsibility | -| --- | --- | -| `cmd/notarius` | Executable entry point and process exit delegation. | -| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. | - -## Core Packages - -| Package | Implemented responsibility | -| --- | --- | -| `internal/core/artifacts` | Run-manifest and provenance models. | -| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. | -| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. | -| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. | -| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. | - -## Framework Packages - -| Package | Implemented responsibility | -| --- | --- | -| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. | -| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. | -| `internal/framework/validate` | Shared validator decision and cardinality helpers. | -| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. | -| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. | -| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. | -| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. | -| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. | -| `internal/framework/debug` | Root-based framework and LLM debug recording. | - -Framework contracts provide typed artifact, provenance-wrapper, chunk-validator, -serialized-validator, and -typed-validator interfaces. The runner owns handoff provenance, validation -sequencing, rejection handling, checkpoint and debug boundaries, and final -manifest assembly. - -Artifact registries support heterogeneous typed extraction entries and -kind-specific merger, normalizer, and validator variants. Resolution derives a -lane's kind from its extractor, requires the matching codec, verifies exact Go -type equality across the lane, and records schema identity in the resolved lane -and pipeline digest. Registry entries carry separate option-validation and -run-local construction closures. Preparation injects shared dependencies and -constructs input, chunk, validators, ordered lanes, and output before source -parsing. Production modules use strict construction-time option decoding, and -LLM-backed modules retain the injected shared client. The D&D family registers -the canonical `dnd/spell-list`, `dnd/npc-list`, `dnd/combat-turn-list`, -`dnd/item-event-list`, `dnd/npc-interaction-list`, and -`dnd/scene-description-list` codecs, typed spell, NPC, combat, item-event, -interaction, and scene-description extractors and normalizers, validators, -plus kind-specific generic merge strategies; generic JSON validators use the -serialized-validation contract. The runner executes lanes through -private exact-type-checked closures, coordinates extract results independently -of completion timing, and serializes artifacts only through their codec at -checkpoint, debug, and output boundaries. - -## Production Extensions - -The canonical catalogs of user-selectable -[module](../config.md#implemented-production-modules) and -[validator](../config.md#implemented-production-validators) keys are in -Configuration. The implemented module packages are: - -| Package | Implemented responsibility | -| --- | --- | -| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. | -| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. | -| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. | -| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, combat-turn-list, combat-turn, item-event-list, item-event, NPC-interaction-list, and scene-description-list artifact types. | -| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. | -| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. | -| `internal/modules/dnd/codec/combatturns` | Strictly decodes and stably encodes the durable D&D combat-turn-list representation. | -| `internal/modules/dnd/codec/itemevents` | Strictly decodes and stably encodes the durable D&D item-event-list representation. | -| `internal/modules/dnd/codec/npcinteractions` | Strictly decodes and stably encodes the durable D&D NPC-interaction-list representation. | -| `internal/modules/dnd/codec/scenedescriptions` | Strictly decodes and stably encodes the durable D&D scene-description-list representation. | -| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. | -| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. | -| `internal/modules/dnd/extract/combatturns` | Uses exact scene eligibility to select combat chunks, then maps private structured model output to source-grounded D&D combat-turn candidates. | -| `internal/modules/dnd/extract/itemevents` | Maps private structured model output to source-grounded D&D item-event candidates. | -| `internal/modules/dnd/extract/npcinteractions` | Maps private structured model output to current-source NPC interaction candidates grounded by a required registry. | -| `internal/modules/dnd/extract/scenedescriptions` | Maps one private scene description to the current accepted chunk's ID and exact range. | -| `internal/modules/dnd/npcinteractions` | Owns interaction occurrence ordering, valid-evidence checks, and exact interaction identity shared by normalization and invariant validation. | -| `internal/modules/dnd/normalize/combatturns` | Canonicalizes and orders merged combat turns, applies exact NPC identity matches, and collapses only exact valid-evidence duplicates. | -| `internal/modules/dnd/normalize/itemevents` | Trims, source-orders, and removes only exact valid-evidence item-event duplicates. | -| `internal/modules/dnd/normalize/npcinteractions` | Canonicalizes required-registry names, orders interaction occurrences, and collapses only exact valid-evidence duplicates. | -| `internal/modules/dnd/normalize/scenedescriptions` | Trims, source-orders, and removes only exactly identical scene descriptions while rejecting ID and range conflicts. | -| `internal/modules/dnd/validate/combatturns` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for the production combat chains. | -| `internal/modules/dnd/validate/itemevents` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for item-event chains. | -| `internal/modules/dnd/validate/npcinteractions` | Provides deterministic shape, registry, source-reference, source-relatedness, and normalized-invariant validation for interaction chains. | -| `internal/modules/dnd/validate/scenedescriptions` | Provides deterministic shape, exact extraction attachment, source-relatedness, and normalized-invariant validation for scene-description chains. | -| `internal/modules/dnd/npcs/registry` | Resolves validated normalized NPC references into immutable grounding data and exact identity lookup. | -| `internal/modules/dnd/scenedescriptions/registry` | Resolves approved scene descriptions into immutable exact-match combat eligibility data without retaining scene prose. | -| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. | -| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. | -| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. | -| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. | -| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. | -| `internal/modules/dnd/normalize/npcs` | Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics. | -| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. | - -`internal/modules/dnd/shared` owns reusable D&D prompt fragments, -reference declarations, prompt input assembly, document-aware source-reference -ordering and canonicalization, and bounded diagnostics under -`internal/modules/dnd/shared/diagnostics`. -The shared NPC grounding fragment is mounted for D&D prompts and is owned by -this package. Domain-neutral prompt filesystem composition lives in -`internal/framework/promptfs`. - -The `dnd/npcs/registry` package owns the optional `npcs` registry boundary. -External references are strictly decoded and identity-validated during -preparation; generated references are decoded and identity-validated at the -ordered step handoff. Both paths retain canonical registry JSON for provenance -and emit a names-only projection to operation-time spell, combat, and -interaction prompts. Combat and interaction normalization use the canonical -registry for exact name lookup. The -framework records generated identity and bounded producer provenance, while -the raw external reference remains independently tracked by pipeline -provenance. An absent registry is represented only by the empty prompt value -`{"npcs":[]}`. Spell -and combat consumers use this shared boundary without changing their public -module contracts. Interaction consumers require it and retain only current -transcript references as durable evidence. - -Generic validators under `internal/modules/generic/validate` provide -unconditional test decisions, JSON syntax validation, and JSON Schema -validation. D&D spell validators under `internal/modules/dnd/validate/spells` -consume the canonical spell-list type directly to provide shape, -effective-catalog, source-reference, and source-relatedness decisions. - -Production composition is grouped behind package-family registrars, and every -implemented production extension uses its domain-first tree: - -| Package | Implemented responsibility | -| --- | --- | -| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. | -| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. | -| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. | - -The CLI allocates the framework registries and asset registry, then invokes -these registrars in generic, Seriatim, and D&D order. - -Implementation details for all production extensions are in -[Module Internals](modules.md). - -## Run-State Components - -| Surface | Implemented owners | Internal purpose | +| Area | Implemented owners | Responsibility | | --- | --- | --- | -| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. | -| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. | -| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. | -| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. | +| Executable and command boundary | **cmd/notarius**, **internal/cli** | Process entry, command dispatch, configuration discovery, production composition, runtime collaborator setup, durable file placement, and user-facing reporting. | +| Configuration | **internal/core/config** | Defaults, strict YAML parsing, environment overrides, structural validation, effective resolution, redaction, and resolved-composition summaries. | +| Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. | +| Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. | +| LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. | +| Runtime state | **internal/core/fileio**, **internal/core/debugbundle**, **internal/framework/checkpoint**, **internal/framework/chunkplan**, **internal/framework/chunkmap**, **internal/framework/debug** | Confined atomic files, debug bundles, checkpoint and chunk-plan state, accepted chunk maps, and pipeline-facing debug recording. | +| Production extensions | **internal/modules/generic**, **internal/modules/seriatim**, **internal/modules/dnd** | Domain-neutral extensions, Seriatim input support, and D&D extraction families registered into the production catalog. | -Physical layout, cleanup, recovery, and sensitive-data handling are defined -in [Operations](../operations.md). Concrete modules receive recorder -interfaces and request data, not physical state roots. +Generic core and framework packages do not depend on production extensions. +Concrete extensions depend inward on their contracts and are registered only at +the CLI composition boundary. ## Focused Documentation - [Configuration Internals](configuration.md): loading, validation, effective resolution, redaction, and resolved-composition identity. - [CLI Internals](cli.md): command dispatch, production composition, run - orchestration, terminal reporting, and focused CLI tests. -- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries, - checkpoint/debug hooks, and result assembly. -- [Module Internals](modules.md): production modules, validators, assets, - registration, and the contributor recipe for adding an extension. -- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter, - assets, scheduling, profile recording, and redaction. + orchestration, and terminal reporting. +- [Pipeline Internals](pipeline.md): resolution, preparation, execution, + validation, typed handoff, and framework state hooks. - [Run State Internals](state.md): output, cache, debug collaborator composition, and path safety. +- [LLM Runtime](llm.md): structured completion, scheduling, prompt assets, + profiles, and secret handling. +- [Module Internals](modules.md): extension registration and production module + mechanics. Durable D&D and Seriatim data shapes remain in the + [integration contracts](../integrations/). + +Use this map to find an owner, then read the focused document and its tests +before changing behavior. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 26ac052..1cf487c 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -1,491 +1,168 @@ # Pipeline Internals -The implemented resolver and runner live in `internal/framework/pipeline`. -Their fixed workflow and ownership boundaries are defined by -[Architecture](../policy/architecture.md#system-shape). Configuration fields, -defaults, and selectable keys are defined in -[Configuration](../config.md#pipelines). +This document describes the framework-owned pipeline mechanics in +**internal/framework/pipeline**. [Configuration](../config.md) owns selectable +profiles, bindings, and retry settings; [Operations](../operations.md) owns +state lifecycle and recovery; and the [integration contracts](../integrations/) +own durable output shapes. Concrete production extensions are covered by +[Module Internals](modules.md). -Resolution fixes the ordered steps, selected lanes, and all stage bindings; -preparation constructs every selected implementation before the runner begins -source work. After serial input parsing and plan selection or generation, the -runner materializes chunks and executes one step at a time. Within a step, -extract work uses one bounded run-wide worker pool in chunk-first, lane-second -order. Each lane's merge and normalize operations remain serial, and lanes in -the same step may overlap once their extracts are terminal. A later step cannot -start across its barrier until every earlier lane is terminal and each required -generated artifact has been accepted and handed off. +## Boundary -## Resolution +The pipeline framework accepts a resolved composition, registries, shared +dependencies, input bytes, and state/debug collaborators. It returns logical +output files, normalized artifacts, recorded rejections and warnings, manifest +provenance, and checkpoint decisions. The CLI owns process arguments, +configuration discovery, physical roots, and placement of returned output +files. -`internal/core/config.Config.Resolve` validates the loaded configuration, -selects the named profile, applies the runtime inputs supplied by the CLI, and -calls `pipeline.ResolvePipeline`. +The framework has one fixed shape: -`ResolvePipeline`: +~~~ +input -> chunk -> extract -> merge -> normalize -> output +~~~ -1. selects the explicit ordered steps, or creates the implicit `default` step - from the legacy top-level `artifacts` map; -2. selects and sorts artifact lanes within each step while enforcing global lane - identity; -3. completes omitted bindings using the documented configuration defaults; -4. looks up each module and validator spec without constructing it; -5. for a typed extractor, derives its artifact kind, requires the codec, and - selects exact-type merger, normalizer, and validator variants; -6. checks required and provided capabilities in workflow order; -7. resolves external and generated target-aware reference bindings and - validates producer order, consumer slot declarations, and artifact-kind - compatibility; -8. validates each selected module and validator option set through its registry - entry; and -9. calculates a digest over the resolved structure, including step order, step - IDs, lane membership, generated topology, producer and consumer identities, - typed artifact kind and schema identity, and the effective validator policy - in its resolved execution order. +Input and chunking are pipeline-wide. A selected artifact lane owns extract, +merge, and normalize; output aggregates the terminal lane outcomes. A pipeline +is an ordered list of steps, not an arbitrary workflow graph. -Resolution returns a `ResolvedPipeline` containing ordered steps, lanes, -concrete bindings, validator chains, reference targets, and the digest. It does -not read external reference bytes or construct runtime modules. CLI lane and -reference selector syntax is defined in the [CLI reference](../cli.md#run). +## Resolve, Materialize, Prepare -The digest includes each resolved step's ID and lane membership, generated -producer/consumer topology, and each validator chain's stage, lane, owning -module, ordered validator bindings, execution classes, targets, and artifact -kinds. Changing step order, a dependency, a default chain, or an explicit -override therefore changes pipeline identity whenever it changes effective -execution policy. +Resolution turns a configured pipeline profile into a **ResolvedPipeline**. +It normalizes the pipeline and lane identities, applies stage defaults, selects +requested lanes where that is supported, resolves validator chains, checks +module capabilities and typed artifact compatibility, validates options, and +assigns a deterministic resolved-composition digest. The resolved pipeline +contains bindings and declared reference targets, not external reference bytes. +Configuration resolution supplies the selected profile and catalog; see +[Configuration Internals](configuration.md). -## Reference Materialization +External reference materialization happens before preparation. The materializer +checks that each slot is declared by the selected module, resolves a file path +relative to the correct configuration or working-directory origin, reads +UTF-8 text, verifies media type and size limits, and retains bounded +provenance. A generated-artifact selector remains declared but has no bytes +until its producing step completes. -The CLI calls `MaterializeReferences` after resolution and before constructing -the LLM client or running the pipeline. For external bindings, the materializer -checks each binding against its resolved target declaration, reads and validates -the file, and builds both a `contracts.ReferenceSet` and provenance-only -metadata on the corresponding `ResolvedReferenceTarget`. A structured -generated binding is declaration-only at this point: its producer bytes do not -exist until the producer lane reaches an accepted normalized result. +Preparation is the construction boundary. It validates the resolved shape and +registry set, clones the resolved data, then constructs the input adapter, +chunker, stage-local validators, every typed lane, and output encoder with +cloned options, references, and shared dependencies. It also collects stable +checkpoint fingerprints. Missing registrations, incompatible typed entries, +nil implementations, and constructor failures are reported before source +parsing or any stage operation begins. -Preparation delivers the materialized external set for each target through -`pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target; -extractors and extract validators receive the lane extract target; mergers and -merge validators receive the lane merge target; and normalizers and normalize -validators receive the lane normalize target. Input and output builders receive -an empty set because those stages cannot declare references. Every builder gets -an isolated deep clone of its target set, so construction-time mutation cannot -change another builder, the resolved pipeline, or later runtime requests. +## Typed Lanes And References -Prepared consumers do not need to be reconstructed when generated content is -available. At the step boundary, the runner encodes the accepted producer value -through its registered canonical codec, validates the generated bytes against -each target slot's kind, schema, media type, and size, and clones one immutable -reference item into the operation request. The item includes canonical digest, -size, and bounded producer provenance but no filesystem URI. A handoff failure -is a framework dependency error and prevents every consumer in that step from -starting. +Each resolved lane has one artifact kind, codec, and exact Go type. The +framework uses private type erasure only around those typed operations; every +handoff checks exact type and codec identity and reports incompatibility as an +error rather than panicking. Encoding through the registered codec is the +boundary for output, checkpoints, debug records, and generated references. -The runner continues to clone the resulting set into the chunk, extract, merge, -or normalize request that owns the target. LLM-backed extensions may convert -those items into named prompt inputs. Reference content remains separate from -source evidence and source digests, whether the item came from a file or a -generated handoff. +Reference targets are stage- and lane-specific. External reference bytes are +cloned into the operation request. Generated references are built at the next +step boundary from exactly one accepted normalized producer output. The +framework decodes and re-encodes that output with the registered producer +codec, checks its complete schema and media identity, and records a content +digest plus bounded producer provenance. A missing, ambiguous, invalid, or +incompatible producer prevents the consumer step from starting. -Binding precedence, path resolution, accepted content, and media-type behavior -are configuration contracts; see [Configuration](../config.md#pipelines). -Durable provenance is defined in the -[JSON output contract](../integrations/json-output.md#manifestjson), while -runtime sensitive-data handling belongs in [Operations](../operations.md). +## Execution And Ordering -## Registries And Specs +The runner validates its input, installs no-op state collaborators when none +were supplied, and serially performs source parsing and chunk-plan selection. +An accepted plan is materialized into source-addressed chunks and passes the +configured chunk validators before any lane runs. A chunk rejection is a +recorded pipeline outcome: lanes do not start, but the output stage can encode +the terminal result. -`pipeline.Registries` holds option validators and run-local builders used during -resolution and preparation. -`pipeline.ModuleCatalog` exposes their specs during configuration validation and -resolution. Separate registries exist for every stage and for validators; -`ValidatorChainRegistry` stores production default-chain mappings. Both -containers also carry an `ArtifactCodecRegistry`. Generic registration records -one codec per stable artifact kind, validates its schema metadata and JSON -Schema, retains the exact schema digest and Go type, and safely encodes or -decodes framework-erased values with typed errors on incompatibility. +For each ordered step, the runner first builds generated reference sets from +the accepted normalized outputs of earlier steps. It then executes the step's +lanes. Later steps do not begin until the current step is terminal and its +generated handoffs have succeeded. -Typed extractor entries are keyed by module key and declare one artifact kind. -Merger, normalizer, and typed-validator variants are keyed by module or -validator key plus artifact kind. Chunk and serialized validators occupy -separate target namespaces; serialized registrations declare whether they -support chunks, artifacts, or both. Duplicate variants and exact Go-type -mismatches are rejected deterministically. +Within a step, the lane engine dispatches extraction jobs in deterministic +chunk-first, lane-second order to a bounded worker group. When all extraction +jobs for one lane are terminal, a bounded continuation group can run that +lane's merge and normalize work while extraction for other lanes continues. +The framework does not create an unbounded goroutine per chunk or lane. -Lane-sensitive merger and normalizer spec discovery always supplies the -extractor's artifact kind, so variants under one reusable key may declare -different capabilities and reference slots. Kind-neutral registry inspection -selects the first registered artifact kind in sorted order. +Completion timing does not determine public results. The coordinator restores +lane and chunk order before merging results, and selects a framework error by +stable stage, lane, and chunk position. A validator rejection records a lane +outcome without cancelling unrelated work. A framework error or parent +cancellation cancels derived work, prevents queued work from starting, waits +for started workers, and prevents output encoding. -Production composition registers the D&D spell-list, NPC-list, combat-turn-list, -NPC-interaction-list, and scene-description-list codecs and typed lane -variants, plus serialized JSON validators. Every artifact lane resolves through -the typed registries and a matching codec. +## Validation, Retries, And Output -A `ModuleSpec` declares its stage plus required and provided capabilities. -Chunk, extract, merge, and normalize specs may also declare reference slots. -Registry implementations defensively copy spec metadata, reject duplicate keys, -and verify that a constructed implementation reports the registered key. -Builder registrations accept `ModuleDependencies` and cloned configuration -options through one `BuildRequest`. Builders decode those options and retain -typed values or injected dependencies in the constructed implementation. -Extractors declare their artifact kind, and merger, normalizer, and validator -resolution selects the matching typed variant. +Every chunk, extract, merge, and normalize candidate passes its resolved +validator chain. Validators receive immutable canonical input appropriate to +their target: chunks, typed values, or serialized codec bytes. They may +approve, approve with warnings, reject, or fail. A rejection is an ordinary +pipeline result; a validator error is a framework error. -A `ValidatorSpec` declares a validator key and execution class. Resolution uses -the execution class to reject incompatible profile bindings before execution. -The current production catalog and default chain are listed only in -[Configuration](../config.md#implemented-production-validators). +The runner applies the binding's retry policy around a stage operation and its +complete validation chain. It preserves warnings only from the final accepted +or rejected attempt. Cancellation stops retries. Normalizer-specific retry +directives consume this same budget and validate any final safe fallback through +the normalizer chain. -## Preparation And Runner Boundary - -`pipeline.Prepare` receives a resolved pipeline, the registries, and shared -module dependencies. It constructs input; chunk and its validators; every -step's lane extract, merge, and normalize modules and validator chains in -resolved order; then output. It stops at the first error with pipeline, step, -stage, lane, module, and validator context as applicable. It never invokes an -operation method. Generated references are not available during preparation; -the operation request is the handoff boundary. - -`PreparedPipeline` keeps private constructed executors and exposes cloned -resolved input, chunk, lane, and output identities. Prepared components may -implement `pipeline.CheckpointFingerprintProvider` to contribute explicit -semantic identities to checkpoint reuse. Preparation trims and validates each -non-secret name and value, prefixes it with the component's stage, lane, -module, and validator scope, rejects duplicates, and retains the resulting -sorted collection behind a defensive-copy accessor. Fingerprints must be -stable and must not contain source content, credentials, local paths, -timestamps, or other invocation-specific values. - -`pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional -session and profile metadata, a chunk-plan store and mode, a checkpoint -execution policy, and checkpoint/debug collaborators. The runner -parses source bytes through the already constructed input adapter. Later stage -requests receive the generic source model; extract requests receive -chunk-scoped input material, while chunk, merge, and normalize requests retain -access to the original source material. Input, chunk, and output operation -requests do not carry raw module options. The chunk request also does not carry -an LLM client; an LLM-backed chunker receives the shared client during -preparation. Their operation requests retain run-specific source, reference, -profile, session, metadata, and step-handoff context as applicable. A generated -reference is cloned into each compatible consumer request and is never exposed -as a path. - -Prepared lanes retain exact-type-checked erased operation closures. The runner -uses those closures to keep each value typed through extraction, validation, -merge, and normalization. - -Source validation requires every unit to carry a canonical self-reference to -its containing document and its own unit ID. Explicit clone, checkpoint, and -debug boundaries retain that reference, and the canonical source digest covers -it deterministically. Chunks use the same source model and carry one canonical -reference spanning the first selected unit through the last. - -`pipeline.RunOutput` carries the run manifest, accepted normalized serialized -artifacts with lane and normalizer provenance, -rejected results, warnings, checkpoint events, and logical files returned by the -output encoder. The CLI owns debug-summary and durable filesystem writes after -the runner returns. - -## Execution Flow - -The pipeline-wide coordinator owns the ordered step loop, generated-reference -sets at each barrier, and deterministic merging of step outcomes. For one step, -one run-local lane engine owns worker lifecycle, cancellation, dispatch, -continuation queues, and result collection. It initializes checkpoint state in -lane order, dispatches bounded extract work, advances terminal lanes through -serial merge and normalize work, selects failures by stable pipeline scope, and -merges lane-local outcomes back in resolved order. Completion timing never -becomes public ordering. - -The runner: - -1. validates its prepared input; -2. parses the raw input with the prepared adapter and validates the generic - source document; -3. selects a stored plan or executes the configured chunker's `Plan` operation; -4. canonicalizes and materializes the plan, then validates the resulting - chunks; -5. builds the framework-owned accepted chunk map from the accepted source, - logical plan, and exact materialized chunks, then supplies it to the output - request independently of output-module options; -6. executes each resolved step in configuration order. For one step, it - dispatches extract jobs in source-chunk then resolved-lane order, starts a - bounded lane continuation when all extracts for that lane are terminal, and - waits for every lane to become terminal; -7. encodes and validates each accepted normalized producer artifact, then - builds the immutable generated reference sets for the next step; -8. invokes the prepared output encoder only after every step succeeds and - validates its logical file results; -9. returns the assembled manifest, outcomes, warnings, and files. - -Within each artifact lane, it reuses the prepared extractor, merger, normalizer, -and validators while performing these transitions: - -1. extract once per accepted chunk and add runner-owned lane, source, and chunk - provenance; -2. validate each extract result and omit rejected results from merge input; -3. skip the rest of the lane when no extract result is accepted; -4. merge accepted extract results in their existing order; -5. validate the merge result and skip normalization on rejection; -6. normalize the accepted merge result; -7. validate and append the accepted normalized result. - -At a step barrier, a lane with no accepted normalized output is still a regular -rejection unless a later generated binding names that lane as a required -producer. In that case the runner raises a deterministic dependency error and -does not start the consumer step. One accepted typed artifact may fan out to -multiple compatible target slots. Consumers in the same step may run -concurrently after the handoff; no work crosses the barrier early. - -Module-provided warnings and payload warnings are promoted only from attempts -whose results are accepted and used. - -## Chunk Plans And Reuse - -`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest, -ordered unit-ID ranges, and optional plan or range annotations. The framework -owns plan canonicalization and materialization. It creates the generic chunks -and therefore owns their IDs, indexes, source references, JSON content, units, -media type, and generic metadata. Plan and range annotations are independently -owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`. - -In `auto`, the runner looks up the source digest before invoking the chunker. A -valid hit is materialized and sent through the current run's configured chunk -validators; it does not invoke the chunk module, consume its retry budget, or -make a chunk-stage LLM call. A missing, invalid, or unmaterializable record -generates a candidate. `refresh` generates without lookup; `bypass` generates -without cache access. Generated plans are published only after the full chunk -validator chain approves them. A validator rejection is a regular rejected -pipeline outcome and never replaces a cached plan. - -The store is source-addressed, not pipeline-addressed. Changes to pipeline -configuration, requested chunker, options, references, lanes, validators, or -LLM profile do not prevent a source-digest hit. The manifest records both the -currently requested chunker and the effective plan producer. Cache state and -paths are configured and operated outside the runner; see -[Configuration](../config.md#state-surfaces) and [Operations](../operations.md). - -For an accepted plan, the runner also constructs the strict framework-owned -[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It -uses the current resolved chunker as `requested_chunker` and the stored or -generated record as `producer`, preserving that distinction on reuse. Chunk -rejection supplies no map; later lane rejection does not discard it. Output -encoders receive a defensively owned serialized value and may explicitly -ignore it. - -The extract job channel has the same capacity as the effective extract worker -count, so dispatch applies backpressure. A fixed continuation executor prevents -ready or checkpoint-reused lanes from creating one goroutine each. Workers and -continuations publish lane-local results; the coordinator is the only writer of -aggregate output and merges those results in resolved lane and source-chunk -order. - -## Plan Canonicalization And Chunk Materialization - -Plan canonicalization requires canonical JSON annotations, a matching source -digest, at least one range, existing ordered boundaries, and increasing range -starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy. -Materialization deterministically reconstructs each range from the current -source document, deep-clones JSON-shaped source-unit metadata, and copies -annotations without interpreting their namespaces. Materialized chunks and -separate materializations do not share mutable unit metadata; unsupported or -cyclic metadata fails materialization with context. - -Before lane execution, generic chunk validation checks the materialized chunks' -identities, order, source references, content, media type, units, and metadata. -No chunk checkpoint participates in plan selection: plan storage is the only -chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to -use materialized chunk digests as their dependencies. - -## Validation And Retries - -Chunk, extract, merge, and normalize results pass through the resolved validator -chain for their stage and module. Chunk validators receive canonical chunks; -typed validators receive the domain value; and serialized validators receive -canonical chunk JSON or artifact codec bytes. Validators execute in resolved -order and stop at the first error or rejection. An empty chain approves the -result. - -Production D&D artifact chains keep generic JSON syntax validation first, then -run every rejecting domain validator before generic JSON Schema validation. The -domain validator therefore owns expected semantic diagnostics; the generic -schema validator remains the final rejecting representation backstop, before -warning-only relatedness validation. Explicitly configured validator chains -retain their configured order. - -`runWithRetry` applies the effective retry policy around module execution and -its complete validation chain. A module or validator error becomes a framework -error when attempts are exhausted. A rejection becomes a recorded -`RejectedOutput` when attempts are exhausted. Cancellation stops retry -processing immediately. - -Structured-completion adapters classify malformed or undecodable provider -output with the provider-neutral `contracts.ErrInvalidStructuredOutput` error. -A typed normalizer may turn that condition, or another unsafe proposal, into a -normalize retry directive with a module-supplied safe candidate, stable -diagnostic, and fallback warnings. The directive consumes the same configured -normalize retry budget: `retries` permits that many additional attempts after -the initial attempt. It neither creates a normalizer-local retry loop nor -records an accepted checkpoint for the discarded attempt. - -Before adding a normalize retry directive to attempt debug data, the runner -requires a nonblank, valid UTF-8 reason code of at most 128 bytes and a -nonblank, valid UTF-8 message of at most 4,096 bytes. These are encoded-byte -limits. The framework rejects an invalid directive without truncating or -rewriting either field. It validates only this mechanical contract; normalizers -remain responsible for ensuring their otherwise valid diagnostics do not expose -source material, credentials, paths, names, or other sensitive content. - -The framework treats a module-supplied candidate as opaque. The normalizer owns -its safety determination, and the configured normalizer validator chain remains -the acceptance boundary for the final fallback. - -When a later normalize attempt succeeds, its candidate alone proceeds through -the usual validation and checkpoint path. When the final attempt still returns -a directive, the runner validates its supplied safe fallback through that same -normalizer validator chain before accepting or rejecting it. Ordinary -attempt-local warnings and fallback warnings remain unpromoted while another -attempt is available; only final exhaustion promotes the supplied fallback -warnings. Rejected output is a non-fatal pipeline outcome and does not advance. -Configuration owns retry counts and validator overrides; see -[Module Bindings](../config.md#module-bindings). +After terminal lane work, the runner assembles manifest provenance, normalized +artifacts, rejections, warnings, and an optional accepted chunk map. The output +encoder returns logical files; it does not choose a physical directory. The CLI +publishes those files only after the runner returns without a framework error. +Logical file names and schemas are defined by the +[output integration contracts](../integrations/). ## Checkpoint And Debug Hooks -The runner depends on recorder and loader interfaces, using no-op -implementations when collaborators are absent. Each checkpointed workflow -boundary records a running, succeeded, or failed transition. Reuse decisions -are consulted in workflow order and accepted payloads are cloned before -entering the normal handoff path. Typed extract, merge, and normalize -checkpoints store codec bytes with artifact kind, schema ID, name, version and -exact digest, and media type. Reuse compares that identity with the prepared -codec and decodes through the codec; missing identity, mismatches, corrupt -bytes, and decode failures become explicit reuse misses and execute the lane -normally. Dependency fingerprints and debug content digests use the same stable -codec bytes that cross those boundaries. +The runner receives checkpoint and debug interfaces rather than roots. It +records workflow transitions and reuse decisions through the supplied +collaborators, and clones reusable artifacts before they re-enter normal typed +handoff. Generated-reference dependencies participate in checkpoint decisions. +Selective recomputation can require a canonical accepted normalized predecessor +before a dependent lane starts. -That progressive extract, merge, and normalize reuse is the ordinary resume -path. A lane marked as a required predecessor for selective recomputation takes -a separate accepted-output path before extract scheduling. The loader reads the -existing successful normalize manifest and payload by step, lane, and -normalizer, without consulting extract or merge dependencies. It requires the -current non-empty checkpoint identity to match, so the invocation identity -still binds the input, resolved topology and configuration, references, runtime -overrides, profiles, and component fingerprints. +Debug recording is attempt-scoped and application-owned. A failure to persist +required debug data is a framework error. State roots, persistence, reason-code +meanings, resume, and cleanup are intentionally owned by +[Run State Internals](state.md) and [Operations](../operations.md). -The runner decodes and canonically re-encodes each reusable artifact once with -the prepared codec, requiring exact kind, schema identity and digest, media -type, canonical bytes, content digest, and producer provenance. A valid accepted -producer becomes a runner-owned cloned normalized output, restores only -normalize-checkpoint warnings, and records one `accepted_artifact_reused` -normalize decision. It does not invoke or record extract, merge, normalize, or -their validators. Invalid or unavailable accepted state records its decision -and fails the producer step; the dependent step never starts and the producer -is not implicitly rerun. If a later required lane fails during initialization, -already hydrated terminal lanes remain in the failed output in resolved order. +## Invariants To Preserve -Generated references add downstream dependencies containing the producer's -artifact kind, complete schema identity, media type, canonical content digest, -and size. Compatible accepted producer outputs may therefore feed a later step -without re-executing the producer. Forced lanes bypass accepted-output -hydration and execute normally. A missing, rejected, corrupt, incompatible, or -changed producer blocks its dependent while leaving independent work eligible -for reuse. The runner records bounded decision -categories: `reused`, `executed`, `forced_recompute`, and -`dependency_invalidated`. Operator meanings for the stable reason codes belong -to [Operations](../operations.md#resume-and-selective-recompute). +- The six fixed stages remain explicit; a pipeline is not a general DAG. +- Resolution and preparation reject statically discoverable incompatibility + before parsing or execution. +- Every typed lane uses one compatible artifact kind, codec, and exact Go type. +- Generated references come only from one earlier accepted normalized producer + and carry canonical identity rather than an unverified value. +- Rejections are recorded outcomes; framework errors cancel derived work and + prevent output encoding. +- Public ordering and selected errors are independent of goroutine completion + order. +- Pipeline modules receive collaborators and data, never CLI streams or + physical output, cache, or debug roots. -The CLI includes prepared-component fingerprints in the run-wide checkpoint -identity alongside resolved configuration, raw input, reference provenance, -runtime overrides, and LLM-profile fingerprints. Module metadata is not used -implicitly for cache identity: components opt in only with stable semantic -values that can change accepted output. Adding or changing a component -fingerprint intentionally produces a cold cache miss. Existing checkpoint -schemas and paths remain unchanged. +## Focused Tests -The CLI's `--recompute-step` policy forces the selected step and all transitive -dependents, but requires accepted normalized artifacts for every unselected -producer on which that closure depends. It changes execution policy only; it -does not alter persistent checkpoint identity. +- **internal/framework/pipeline/profile_test.go** and + **typed_resolution_test.go** cover resolution, defaults, ordered steps, + compatibility, validators, references, and resolved identity. +- **internal/framework/pipeline/preparation_test.go** covers complete + construction before execution and contextual construction failures. +- **internal/framework/pipeline/references_test.go** and **handoff_test.go** + cover external materialization, generated references, provenance, and typed + producer checks. +- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded + execution, ordered steps, stable error selection, rejections, and + cancellation. +- **internal/framework/pipeline/runner_chunk_plan_test.go**, + **runner_typed_checkpoint_test.go**, and + **runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior. +- **internal/framework/pipeline/runner_attempt_debug_test.go** and + **runner_terminal_debug_test.go** cover attempt and terminal debug behavior. -Debug instrumentation wraps run, stage, attempt, validator, and structured LLM -boundaries. Every executed chunk, extract, merge, and normalize attempt writes -one terminal envelope for acceptance, validator rejection, module or validator -error, or applicable candidate or final serialization error. The envelope -contains its attempt-local warnings, any available candidate and rejection, -and terminal error text; normalize retry directives retain their attempt-local -candidate and diagnostic, while only the final safe fallback reaches validation. -Failures before a candidate exists omit that payload. -Only LLM calls made by the module operation belong to the module attempt. -Validator calls retain independent scopes under `validate/` and are not -duplicated into the module envelope. A failed terminal-envelope write is a -non-retryable framework error and is joined with any primary attempt error. -Debug data is never used as a checkpoint source. Typed artifact debug envelopes -are domain-neutral, redact sensitive metadata and bytes through the common -debug policy, and record codec identity plus schema and content digests. - -Merge and normalize attempts serialize their in-memory candidate with the -codec's required candidate encoder before typed validation. Serialized -validators and attempt debug use that candidate representation, which carries -the codec media type and schema identity but is never checkpointed or passed -downstream. Only a validator-approved value is encoded through the strict final -codec and made eligible for a checkpoint or stage output. - -Checkpoint identity, physical layout, reuse behavior, and debug artifact -handling are operator contracts in [Operations](../operations.md). Serialization -and recorder implementation are inventoried in -[Internal Overview](overview.md#run-state-components). - -## Results And Failures - -The runner owns manifest assembly and handoff summaries but not the durable JSON -schema. It records resolved module and lane provenance, validator chains, -source/reference identities, selected LLM profiles, normalized and rejected -summaries, status, and timing. Serialized artifact content remains outside the manifest. -Module metadata providers may add non-secret singleton or lane-scoped metadata. - -Execution errors include stage, module, lane, or validator context. Once a -manifest exists, a failing run returns it with failed status and completion -time. Successful status reflects whether any result was rejected. The -durable manifest and logical file schemas are defined in the -[JSON output contract](../integrations/json-output.md). - -On a framework failure, the runner cancels its derived context, stops submitting -new extract work, drains started tasks, and skips the output encoder. Parent -cancellation takes precedence. Otherwise context-cancellation fallout is -discarded when a substantive error exists, and the primary error is selected by -stage, resolved lane, and source chunk rather than completion time. - -## Tests To Inspect - -- `internal/core/config/effective_config_test.go`: config-to-resolution boundary. -- `internal/framework/pipeline/profile_test.go`: selection, defaults, - capabilities, validator chains, and digest behavior. -- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec - metadata, registration, erasure safety, strict decoding, and cloning. -- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed - lane resolution and preparation, target-specific validators, - incompatibilities, ordering, and schema-sensitive pipeline identity. -- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and - continuations, reverse completion, stable errors, rejection, cancellation, - retries, and independent provider-call limits. -- `internal/framework/pipeline/preparation_test.go`: option validation, - construction order, dependency failures, and the before-source-work boundary. -- `internal/framework/pipeline/references_test.go`: target resolution and - materialization. -- `internal/cli/run_contract_test.go`: production run transitions, retries, - rejections, warnings, CLI recomputation controls, debug hooks, and manifests. -- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed - selective recomputation and accepted-producer recovery. -- `internal/cli/production_contract_test.go`: production composition and - configuration-resolution smoke coverage. -- `internal/cli/example_contract_test.go`: maintained example resolution and - execution ownership. -- `internal/modules/integration/*_test.go` and - `internal/modules/seriatim/input/transcript/runner_test.go`: typed runner - composition across concrete module families. -- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse - collaborators. +Run **go test ./internal/framework/pipeline ./internal/cli** after changing a +pipeline boundary. Use the more focused tests above while iterating.