eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite pipeline and component overview documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-26 13:26:20 +00:00
parent fd3f7b85cc
commit b5835fbc37
2 changed files with 172 additions and 634 deletions
Download Patch File Download Diff File Expand all files Collapse all files

211
docs/internal/overview.md
View File

@@ -1,195 +1,56 @@
# Internal Overview # Internal Overview
This document inventories the implemented Notarius components. Normative This document is the implemented component map for Notarius. Normative
boundaries and dependency direction belong in boundaries and dependency direction belong in
[Architecture](../policy/architecture.md); external behavior belongs in the [Architecture](../policy/architecture.md). User and operator contracts belong
[CLI](../cli.md), [Configuration](../config.md), in the [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/). [Operations](../operations.md), and [integration contracts](../integrations/).
## Execution Path ## Execution Path
`cmd/notarius` delegates to `internal/cli`, the production composition root. ~~~
The CLI loads configuration, builds the production catalogs and runtime cmd/notarius -> internal/cli -> configuration and production composition
collaborators, invokes `internal/framework/pipeline`, and places the logical -> internal/framework/pipeline -> logical output files
output files returned by the runner. Cache and debug collaborators are supplied -> internal/cli -> durable output and optional state/debug data
at this boundary. ~~~
Resolution produces a fixed ordered workflow of steps and globally unique, The CLI is the application boundary: it discovers configuration, composes
sorted artifact lanes. Preparation constructs the complete module and validator production registries and runtime collaborators, invokes the framework, and
set before the runner receives source bytes. Source parsing and chunking are places returned files. The framework resolves and prepares a fixed extraction
serial. Each step then uses a bounded run-wide extraction pool followed by pipeline, then returns logical results without owning process behavior or
serial per-lane merge and normalize continuations. A step barrier prevents physical state roots.
later consumers from starting until all earlier lanes are terminal and their
required normalized artifacts have crossed the typed handoff.
## Application Boundary ## Components
| Package | Implemented responsibility | | Area | Implemented owners | Responsibility |
| --- | --- |
| `cmd/notarius` | Executable entry point and process exit delegation. |
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. |
## Core Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/core/artifacts` | Run-manifest and provenance models. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. |
| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. |
| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |
## Framework Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
serialized-validator, and
typed-validator interfaces. The runner owns handoff provenance, validation
sequencing, rejection handling, checkpoint and debug boundaries, and final
manifest assembly.
Artifact registries support heterogeneous typed extraction entries and
kind-specific merger, normalizer, and validator variants. Resolution derives a
lane's kind from its extractor, requires the matching codec, verifies exact Go
type equality across the lane, and records schema identity in the resolved lane
and pipeline digest. Registry entries carry separate option-validation and
run-local construction closures. Preparation injects shared dependencies and
constructs input, chunk, validators, ordered lanes, and output before source
parsing. Production modules use strict construction-time option decoding, and
LLM-backed modules retain the injected shared client. The D&D family registers
the canonical `dnd/spell-list`, `dnd/npc-list`, `dnd/combat-turn-list`,
`dnd/item-event-list`, `dnd/npc-interaction-list`, and
`dnd/scene-description-list` codecs, typed spell, NPC, combat, item-event,
interaction, and scene-description extractors and normalizers, validators,
plus kind-specific generic merge strategies; generic JSON validators use the
serialized-validation contract. The runner executes lanes through
private exact-type-checked closures, coordinates extract results independently
of completion timing, and serializes artifacts only through their codec at
checkpoint, debug, and output boundaries.
## Production Extensions
The canonical catalogs of user-selectable
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) keys are in
Configuration. The implemented module packages are:
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, combat-turn-list, combat-turn, item-event-list, item-event, NPC-interaction-list, and scene-description-list artifact types. |
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. |
| `internal/modules/dnd/codec/combatturns` | Strictly decodes and stably encodes the durable D&D combat-turn-list representation. |
| `internal/modules/dnd/codec/itemevents` | Strictly decodes and stably encodes the durable D&D item-event-list representation. |
| `internal/modules/dnd/codec/npcinteractions` | Strictly decodes and stably encodes the durable D&D NPC-interaction-list representation. |
| `internal/modules/dnd/codec/scenedescriptions` | Strictly decodes and stably encodes the durable D&D scene-description-list representation. |
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. |
| `internal/modules/dnd/extract/combatturns` | Uses exact scene eligibility to select combat chunks, then maps private structured model output to source-grounded D&D combat-turn candidates. |
| `internal/modules/dnd/extract/itemevents` | Maps private structured model output to source-grounded D&D item-event candidates. |
| `internal/modules/dnd/extract/npcinteractions` | Maps private structured model output to current-source NPC interaction candidates grounded by a required registry. |
| `internal/modules/dnd/extract/scenedescriptions` | Maps one private scene description to the current accepted chunk's ID and exact range. |
| `internal/modules/dnd/npcinteractions` | Owns interaction occurrence ordering, valid-evidence checks, and exact interaction identity shared by normalization and invariant validation. |
| `internal/modules/dnd/normalize/combatturns` | Canonicalizes and orders merged combat turns, applies exact NPC identity matches, and collapses only exact valid-evidence duplicates. |
| `internal/modules/dnd/normalize/itemevents` | Trims, source-orders, and removes only exact valid-evidence item-event duplicates. |
| `internal/modules/dnd/normalize/npcinteractions` | Canonicalizes required-registry names, orders interaction occurrences, and collapses only exact valid-evidence duplicates. |
| `internal/modules/dnd/normalize/scenedescriptions` | Trims, source-orders, and removes only exactly identical scene descriptions while rejecting ID and range conflicts. |
| `internal/modules/dnd/validate/combatturns` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for the production combat chains. |
| `internal/modules/dnd/validate/itemevents` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for item-event chains. |
| `internal/modules/dnd/validate/npcinteractions` | Provides deterministic shape, registry, source-reference, source-relatedness, and normalized-invariant validation for interaction chains. |
| `internal/modules/dnd/validate/scenedescriptions` | Provides deterministic shape, exact extraction attachment, source-relatedness, and normalized-invariant validation for scene-description chains. |
| `internal/modules/dnd/npcs/registry` | Resolves validated normalized NPC references into immutable grounding data and exact identity lookup. |
| `internal/modules/dnd/scenedescriptions/registry` | Resolves approved scene descriptions into immutable exact-match combat eligibility data without retaining scene prose. |
| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. |
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
| `internal/modules/dnd/normalize/npcs` | Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
reference declarations, prompt input assembly, document-aware source-reference
ordering and canonicalization, and bounded diagnostics under
`internal/modules/dnd/shared/diagnostics`.
The shared NPC grounding fragment is mounted for D&D prompts and is owned by
this package. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`.
The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
External references are strictly decoded and identity-validated during
preparation; generated references are decoded and identity-validated at the
ordered step handoff. Both paths retain canonical registry JSON for provenance
and emit a names-only projection to operation-time spell, combat, and
interaction prompts. Combat and interaction normalization use the canonical
registry for exact name lookup. The
framework records generated identity and bounded producer provenance, while
the raw external reference remains independently tracked by pipeline
provenance. An absent registry is represented only by the empty prompt value
`{"npcs":[]}`. Spell
and combat consumers use this shared boundary without changing their public
module contracts. Interaction consumers require it and retain only current
transcript references as durable evidence.
Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
consume the canonical spell-list type directly to provide shape,
effective-catalog, source-reference, and source-relatedness decisions.
Production composition is grouped behind package-family registrars, and every
implemented production extension uses its domain-first tree:
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
The CLI allocates the framework registries and asset registry, then invokes
these registrars in generic, Seriatim, and D&D order.
Implementation details for all production extensions are in
[Module Internals](modules.md).
## Run-State Components
| Surface | Implemented owners | Internal purpose |
| --- | --- | --- | | --- | --- | --- |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. | | Executable and command boundary | **cmd/notarius**, **internal/cli** | Process entry, command dispatch, configuration discovery, production composition, runtime collaborator setup, durable file placement, and user-facing reporting. |
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. | | Configuration | **internal/core/config** | Defaults, strict YAML parsing, environment overrides, structural validation, effective resolution, redaction, and resolved-composition summaries. |
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. | | Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. |
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. | | Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. |
| LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. |
| Runtime state | **internal/core/fileio**, **internal/core/debugbundle**, **internal/framework/checkpoint**, **internal/framework/chunkplan**, **internal/framework/chunkmap**, **internal/framework/debug** | Confined atomic files, debug bundles, checkpoint and chunk-plan state, accepted chunk maps, and pipeline-facing debug recording. |
| Production extensions | **internal/modules/generic**, **internal/modules/seriatim**, **internal/modules/dnd** | Domain-neutral extensions, Seriatim input support, and D&D extraction families registered into the production catalog. |
Physical layout, cleanup, recovery, and sensitive-data handling are defined Generic core and framework packages do not depend on production extensions.
in [Operations](../operations.md). Concrete modules receive recorder Concrete extensions depend inward on their contracts and are registered only at
interfaces and request data, not physical state roots. the CLI composition boundary.
## Focused Documentation ## Focused Documentation
- [Configuration Internals](configuration.md): loading, validation, effective - [Configuration Internals](configuration.md): loading, validation, effective
resolution, redaction, and resolved-composition identity. resolution, redaction, and resolved-composition identity.
- [CLI Internals](cli.md): command dispatch, production composition, run - [CLI Internals](cli.md): command dispatch, production composition, run
orchestration, terminal reporting, and focused CLI tests. orchestration, and terminal reporting.
- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries, - [Pipeline Internals](pipeline.md): resolution, preparation, execution,
checkpoint/debug hooks, and result assembly. validation, typed handoff, and framework state hooks.
- [Module Internals](modules.md): production modules, validators, assets,
registration, and the contributor recipe for adding an extension.
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
assets, scheduling, profile recording, and redaction.
- [Run State Internals](state.md): output, cache, debug collaborator - [Run State Internals](state.md): output, cache, debug collaborator
composition, and path safety. composition, and path safety.
- [LLM Runtime](llm.md): structured completion, scheduling, prompt assets,
profiles, and secret handling.
- [Module Internals](modules.md): extension registration and production module
mechanics. Durable D&D and Seriatim data shapes remain in the
[integration contracts](../integrations/).
Use this map to find an owner, then read the focused document and its tests
before changing behavior.

595
docs/internal/pipeline.md
View File

@@ -1,491 +1,168 @@
# Pipeline Internals # Pipeline Internals
The implemented resolver and runner live in `internal/framework/pipeline`. This document describes the framework-owned pipeline mechanics in
Their fixed workflow and ownership boundaries are defined by **internal/framework/pipeline**. [Configuration](../config.md) owns selectable
[Architecture](../policy/architecture.md#system-shape). Configuration fields, profiles, bindings, and retry settings; [Operations](../operations.md) owns
defaults, and selectable keys are defined in state lifecycle and recovery; and the [integration contracts](../integrations/)
[Configuration](../config.md#pipelines). own durable output shapes. Concrete production extensions are covered by
[Module Internals](modules.md).
Resolution fixes the ordered steps, selected lanes, and all stage bindings; ## Boundary
preparation constructs every selected implementation before the runner begins
source work. After serial input parsing and plan selection or generation, the
runner materializes chunks and executes one step at a time. Within a step,
extract work uses one bounded run-wide worker pool in chunk-first, lane-second
order. Each lane's merge and normalize operations remain serial, and lanes in
the same step may overlap once their extracts are terminal. A later step cannot
start across its barrier until every earlier lane is terminal and each required
generated artifact has been accepted and handed off.
## Resolution The pipeline framework accepts a resolved composition, registries, shared
dependencies, input bytes, and state/debug collaborators. It returns logical
output files, normalized artifacts, recorded rejections and warnings, manifest
provenance, and checkpoint decisions. The CLI owns process arguments,
configuration discovery, physical roots, and placement of returned output
files.
`internal/core/config.Config.Resolve` validates the loaded configuration, The framework has one fixed shape:
selects the named profile, applies the runtime inputs supplied by the CLI, and
calls `pipeline.ResolvePipeline`.
`ResolvePipeline`: ~~~
input -> chunk -> extract -> merge -> normalize -> output
~~~
1. selects the explicit ordered steps, or creates the implicit `default` step Input and chunking are pipeline-wide. A selected artifact lane owns extract,
from the legacy top-level `artifacts` map; merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
2. selects and sorts artifact lanes within each step while enforcing global lane is an ordered list of steps, not an arbitrary workflow graph.
identity;
3. completes omitted bindings using the documented configuration defaults;
4. looks up each module and validator spec without constructing it;
5. for a typed extractor, derives its artifact kind, requires the codec, and
selects exact-type merger, normalizer, and validator variants;
6. checks required and provided capabilities in workflow order;
7. resolves external and generated target-aware reference bindings and
validates producer order, consumer slot declarations, and artifact-kind
compatibility;
8. validates each selected module and validator option set through its registry
entry; and
9. calculates a digest over the resolved structure, including step order, step
IDs, lane membership, generated topology, producer and consumer identities,
typed artifact kind and schema identity, and the effective validator policy
in its resolved execution order.
Resolution returns a `ResolvedPipeline` containing ordered steps, lanes, ## Resolve, Materialize, Prepare
concrete bindings, validator chains, reference targets, and the digest. It does
not read external reference bytes or construct runtime modules. CLI lane and
reference selector syntax is defined in the [CLI reference](../cli.md#run).
The digest includes each resolved step's ID and lane membership, generated Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
producer/consumer topology, and each validator chain's stage, lane, owning It normalizes the pipeline and lane identities, applies stage defaults, selects
module, ordered validator bindings, execution classes, targets, and artifact requested lanes where that is supported, resolves validator chains, checks
kinds. Changing step order, a dependency, a default chain, or an explicit module capabilities and typed artifact compatibility, validates options, and
override therefore changes pipeline identity whenever it changes effective assigns a deterministic resolved-composition digest. The resolved pipeline
execution policy. contains bindings and declared reference targets, not external reference bytes.
Configuration resolution supplies the selected profile and catalog; see
[Configuration Internals](configuration.md).
## Reference Materialization External reference materialization happens before preparation. The materializer
checks that each slot is declared by the selected module, resolves a file path
relative to the correct configuration or working-directory origin, reads
UTF-8 text, verifies media type and size limits, and retains bounded
provenance. A generated-artifact selector remains declared but has no bytes
until its producing step completes.
The CLI calls `MaterializeReferences` after resolution and before constructing Preparation is the construction boundary. It validates the resolved shape and
the LLM client or running the pipeline. For external bindings, the materializer registry set, clones the resolved data, then constructs the input adapter,
checks each binding against its resolved target declaration, reads and validates chunker, stage-local validators, every typed lane, and output encoder with
the file, and builds both a `contracts.ReferenceSet` and provenance-only cloned options, references, and shared dependencies. It also collects stable
metadata on the corresponding `ResolvedReferenceTarget`. A structured checkpoint fingerprints. Missing registrations, incompatible typed entries,
generated binding is declaration-only at this point: its producer bytes do not nil implementations, and constructor failures are reported before source
exist until the producer lane reaches an accepted normalized result. parsing or any stage operation begins.
Preparation delivers the materialized external set for each target through ## Typed Lanes And References
`pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
extractors and extract validators receive the lane extract target; mergers and
merge validators receive the lane merge target; and normalizers and normalize
validators receive the lane normalize target. Input and output builders receive
an empty set because those stages cannot declare references. Every builder gets
an isolated deep clone of its target set, so construction-time mutation cannot
change another builder, the resolved pipeline, or later runtime requests.
Prepared consumers do not need to be reconstructed when generated content is Each resolved lane has one artifact kind, codec, and exact Go type. The
available. At the step boundary, the runner encodes the accepted producer value framework uses private type erasure only around those typed operations; every
through its registered canonical codec, validates the generated bytes against handoff checks exact type and codec identity and reports incompatibility as an
each target slot's kind, schema, media type, and size, and clones one immutable error rather than panicking. Encoding through the registered codec is the
reference item into the operation request. The item includes canonical digest, boundary for output, checkpoints, debug records, and generated references.
size, and bounded producer provenance but no filesystem URI. A handoff failure
is a framework dependency error and prevents every consumer in that step from
starting.
The runner continues to clone the resulting set into the chunk, extract, merge, Reference targets are stage- and lane-specific. External reference bytes are
or normalize request that owns the target. LLM-backed extensions may convert cloned into the operation request. Generated references are built at the next
those items into named prompt inputs. Reference content remains separate from step boundary from exactly one accepted normalized producer output. The
source evidence and source digests, whether the item came from a file or a framework decodes and re-encodes that output with the registered producer
generated handoff. codec, checks its complete schema and media identity, and records a content
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
incompatible producer prevents the consumer step from starting.
Binding precedence, path resolution, accepted content, and media-type behavior ## Execution And Ordering
are configuration contracts; see [Configuration](../config.md#pipelines).
Durable provenance is defined in the
[JSON output contract](../integrations/json-output.md#manifestjson), while
runtime sensitive-data handling belongs in [Operations](../operations.md).
## Registries And Specs The runner validates its input, installs no-op state collaborators when none
were supplied, and serially performs source parsing and chunk-plan selection.
An accepted plan is materialized into source-addressed chunks and passes the
configured chunk validators before any lane runs. A chunk rejection is a
recorded pipeline outcome: lanes do not start, but the output stage can encode
the terminal result.
`pipeline.Registries` holds option validators and run-local builders used during For each ordered step, the runner first builds generated reference sets from
resolution and preparation. the accepted normalized outputs of earlier steps. It then executes the step's
`pipeline.ModuleCatalog` exposes their specs during configuration validation and lanes. Later steps do not begin until the current step is terminal and its
resolution. Separate registries exist for every stage and for validators; generated handoffs have succeeded.
`ValidatorChainRegistry` stores production default-chain mappings. Both
containers also carry an `ArtifactCodecRegistry`. Generic registration records
one codec per stable artifact kind, validates its schema metadata and JSON
Schema, retains the exact schema digest and Go type, and safely encodes or
decodes framework-erased values with typed errors on incompatibility.
Typed extractor entries are keyed by module key and declare one artifact kind. Within a step, the lane engine dispatches extraction jobs in deterministic
Merger, normalizer, and typed-validator variants are keyed by module or chunk-first, lane-second order to a bounded worker group. When all extraction
validator key plus artifact kind. Chunk and serialized validators occupy jobs for one lane are terminal, a bounded continuation group can run that
separate target namespaces; serialized registrations declare whether they lane's merge and normalize work while extraction for other lanes continues.
support chunks, artifacts, or both. Duplicate variants and exact Go-type The framework does not create an unbounded goroutine per chunk or lane.
mismatches are rejected deterministically.
Lane-sensitive merger and normalizer spec discovery always supplies the Completion timing does not determine public results. The coordinator restores
extractor's artifact kind, so variants under one reusable key may declare lane and chunk order before merging results, and selects a framework error by
different capabilities and reference slots. Kind-neutral registry inspection stable stage, lane, and chunk position. A validator rejection records a lane
selects the first registered artifact kind in sorted order. outcome without cancelling unrelated work. A framework error or parent
cancellation cancels derived work, prevents queued work from starting, waits
for started workers, and prevents output encoding.
Production composition registers the D&D spell-list, NPC-list, combat-turn-list, ## Validation, Retries, And Output
NPC-interaction-list, and scene-description-list codecs and typed lane
variants, plus serialized JSON validators. Every artifact lane resolves through
the typed registries and a matching codec.
A `ModuleSpec` declares its stage plus required and provided capabilities. Every chunk, extract, merge, and normalize candidate passes its resolved
Chunk, extract, merge, and normalize specs may also declare reference slots. validator chain. Validators receive immutable canonical input appropriate to
Registry implementations defensively copy spec metadata, reject duplicate keys, their target: chunks, typed values, or serialized codec bytes. They may
and verify that a constructed implementation reports the registered key. approve, approve with warnings, reject, or fail. A rejection is an ordinary
Builder registrations accept `ModuleDependencies` and cloned configuration pipeline result; a validator error is a framework error.
options through one `BuildRequest`. Builders decode those options and retain
typed values or injected dependencies in the constructed implementation.
Extractors declare their artifact kind, and merger, normalizer, and validator
resolution selects the matching typed variant.
A `ValidatorSpec` declares a validator key and execution class. Resolution uses The runner applies the binding's retry policy around a stage operation and its
the execution class to reject incompatible profile bindings before execution. complete validation chain. It preserves warnings only from the final accepted
The current production catalog and default chain are listed only in or rejected attempt. Cancellation stops retries. Normalizer-specific retry
[Configuration](../config.md#implemented-production-validators). directives consume this same budget and validate any final safe fallback through
the normalizer chain.
## Preparation And Runner Boundary After terminal lane work, the runner assembles manifest provenance, normalized
artifacts, rejections, warnings, and an optional accepted chunk map. The output
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared encoder returns logical files; it does not choose a physical directory. The CLI
module dependencies. It constructs input; chunk and its validators; every publishes those files only after the runner returns without a framework error.
step's lane extract, merge, and normalize modules and validator chains in Logical file names and schemas are defined by the
resolved order; then output. It stops at the first error with pipeline, step, [output integration contracts](../integrations/).
stage, lane, module, and validator context as applicable. It never invokes an
operation method. Generated references are not available during preparation;
the operation request is the handoff boundary.
`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. Prepared components may
implement `pipeline.CheckpointFingerprintProvider` to contribute explicit
semantic identities to checkpoint reuse. Preparation trims and validates each
non-secret name and value, prefixes it with the component's stage, lane,
module, and validator scope, rejects duplicates, and retains the resulting
sorted collection behind a defensive-copy accessor. Fingerprints must be
stable and must not contain source content, credentials, local paths,
timestamps, or other invocation-specific values.
`pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, a chunk-plan store and mode, a checkpoint
execution policy, and checkpoint/debug collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
access to the original source material. Input, chunk, and output operation
requests do not carry raw module options. The chunk request also does not carry
an LLM client; an LLM-backed chunker receives the shared client during
preparation. Their operation requests retain run-specific source, reference,
profile, session, metadata, and step-handoff context as applicable. A generated
reference is cloned into each compatible consumer request and is never exposed
as a path.
Prepared lanes retain exact-type-checked erased operation closures. The runner
uses those closures to keep each value typed through extraction, validation,
merge, and normalization.
Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
debug boundaries retain that reference, and the canonical source digest covers
it deterministically. Chunks use the same source model and carry one canonical
reference spanning the first selected unit through the last.
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
artifacts with lane and normalizer provenance,
rejected results, warnings, checkpoint events, and logical files returned by the
output encoder. The CLI owns debug-summary and durable filesystem writes after
the runner returns.
## Execution Flow
The pipeline-wide coordinator owns the ordered step loop, generated-reference
sets at each barrier, and deterministic merging of step outcomes. For one step,
one run-local lane engine owns worker lifecycle, cancellation, dispatch,
continuation queues, and result collection. It initializes checkpoint state in
lane order, dispatches bounded extract work, advances terminal lanes through
serial merge and normalize work, selects failures by stable pipeline scope, and
merges lane-local outcomes back in resolved order. Completion timing never
becomes public ordering.
The runner:
1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
source document;
3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting
chunks;
5. builds the framework-owned accepted chunk map from the accepted source,
logical plan, and exact materialized chunks, then supplies it to the output
request independently of output-module options;
6. executes each resolved step in configuration order. For one step, it
dispatches extract jobs in source-chunk then resolved-lane order, starts a
bounded lane continuation when all extracts for that lane are terminal, and
waits for every lane to become terminal;
7. encodes and validates each accepted normalized producer artifact, then
builds the immutable generated reference sets for the next step;
8. invokes the prepared output encoder only after every step succeeds and
validates its logical file results;
9. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
provenance;
2. validate each extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.
At a step barrier, a lane with no accepted normalized output is still a regular
rejection unless a later generated binding names that lane as a required
producer. In that case the runner raises a deterministic dependency error and
does not start the consumer step. One accepted typed artifact may fan out to
multiple compatible target slots. Consumers in the same step may run
concurrently after the handoff; no work crosses the barrier early.
Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.
## Chunk Plans And Reuse
`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
ordered unit-ID ranges, and optional plan or range annotations. The framework
owns plan canonicalization and materialization. It creates the generic chunks
and therefore owns their IDs, indexes, source references, JSON content, units,
media type, and generic metadata. Plan and range annotations are independently
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
In `auto`, the runner looks up the source digest before invoking the chunker. A
valid hit is materialized and sent through the current run's configured chunk
validators; it does not invoke the chunk module, consume its retry budget, or
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
generates a candidate. `refresh` generates without lookup; `bypass` generates
without cache access. Generated plans are published only after the full chunk
validator chain approves them. A validator rejection is a regular rejected
pipeline outcome and never replaces a cached plan.
The store is source-addressed, not pipeline-addressed. Changes to pipeline
configuration, requested chunker, options, references, lanes, validators, or
LLM profile do not prevent a source-digest hit. The manifest records both the
currently requested chunker and the effective plan producer. Cache state and
paths are configured and operated outside the runner; see
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
For an accepted plan, the runner also constructs the strict framework-owned
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
uses the current resolved chunker as `requested_chunker` and the stored or
generated record as `producer`, preserving that distinction on reuse. Chunk
rejection supplies no map; later lane rejection does not discard it. Output
encoders receive a defensively owned serialized value and may explicitly
ignore it.
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
continuations publish lane-local results; the coordinator is the only writer of
aggregate output and merges those results in resolved lane and source-chunk
order.
## Plan Canonicalization And Chunk Materialization
Plan canonicalization requires canonical JSON annotations, a matching source
digest, at least one range, existing ordered boundaries, and increasing range
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
Materialization deterministically reconstructs each range from the current
source document, deep-clones JSON-shaped source-unit metadata, and copies
annotations without interpreting their namespaces. Materialized chunks and
separate materializations do not share mutable unit metadata; unsupported or
cyclic metadata fails materialization with context.
Before lane execution, generic chunk validation checks the materialized chunks'
identities, order, source references, content, media type, units, and metadata.
No chunk checkpoint participates in plan selection: plan storage is the only
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
use materialized chunk digests as their dependencies.
## Validation And Retries
Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Chunk validators receive canonical chunks;
typed validators receive the domain value; and serialized validators receive
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
order and stop at the first error or rejection. An empty chain approves the
result.
Production D&D artifact chains keep generic JSON syntax validation first, then
run every rejecting domain validator before generic JSON Schema validation. The
domain validator therefore owns expected semantic diagnostics; the generic
schema validator remains the final rejecting representation backstop, before
warning-only relatedness validation. Explicitly configured validator chains
retain their configured order.
`runWithRetry` applies the effective retry policy around module execution and
its complete validation chain. A module or validator error becomes a framework
error when attempts are exhausted. A rejection becomes a recorded
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
processing immediately.
Structured-completion adapters classify malformed or undecodable provider
output with the provider-neutral `contracts.ErrInvalidStructuredOutput` error.
A typed normalizer may turn that condition, or another unsafe proposal, into a
normalize retry directive with a module-supplied safe candidate, stable
diagnostic, and fallback warnings. The directive consumes the same configured
normalize retry budget: `retries` permits that many additional attempts after
the initial attempt. It neither creates a normalizer-local retry loop nor
records an accepted checkpoint for the discarded attempt.
Before adding a normalize retry directive to attempt debug data, the runner
requires a nonblank, valid UTF-8 reason code of at most 128 bytes and a
nonblank, valid UTF-8 message of at most 4,096 bytes. These are encoded-byte
limits. The framework rejects an invalid directive without truncating or
rewriting either field. It validates only this mechanical contract; normalizers
remain responsible for ensuring their otherwise valid diagnostics do not expose
source material, credentials, paths, names, or other sensitive content.
The framework treats a module-supplied candidate as opaque. The normalizer owns
its safety determination, and the configured normalizer validator chain remains
the acceptance boundary for the final fallback.
When a later normalize attempt succeeds, its candidate alone proceeds through
the usual validation and checkpoint path. When the final attempt still returns
a directive, the runner validates its supplied safe fallback through that same
normalizer validator chain before accepting or rejecting it. Ordinary
attempt-local warnings and fallback warnings remain unpromoted while another
attempt is available; only final exhaustion promotes the supplied fallback
warnings. Rejected output is a non-fatal pipeline outcome and does not advance.
Configuration owns retry counts and validator overrides; see
[Module Bindings](../config.md#module-bindings).
## Checkpoint And Debug Hooks ## Checkpoint And Debug Hooks
The runner depends on recorder and loader interfaces, using no-op The runner receives checkpoint and debug interfaces rather than roots. It
implementations when collaborators are absent. Each checkpointed workflow records workflow transitions and reuse decisions through the supplied
boundary records a running, succeeded, or failed transition. Reuse decisions collaborators, and clones reusable artifacts before they re-enter normal typed
are consulted in workflow order and accepted payloads are cloned before handoff. Generated-reference dependencies participate in checkpoint decisions.
entering the normal handoff path. Typed extract, merge, and normalize Selective recomputation can require a canonical accepted normalized predecessor
checkpoints store codec bytes with artifact kind, schema ID, name, version and before a dependent lane starts.
exact digest, and media type. Reuse compares that identity with the prepared
codec and decodes through the codec; missing identity, mismatches, corrupt
bytes, and decode failures become explicit reuse misses and execute the lane
normally. Dependency fingerprints and debug content digests use the same stable
codec bytes that cross those boundaries.
That progressive extract, merge, and normalize reuse is the ordinary resume Debug recording is attempt-scoped and application-owned. A failure to persist
path. A lane marked as a required predecessor for selective recomputation takes required debug data is a framework error. State roots, persistence, reason-code
a separate accepted-output path before extract scheduling. The loader reads the meanings, resume, and cleanup are intentionally owned by
existing successful normalize manifest and payload by step, lane, and [Run State Internals](state.md) and [Operations](../operations.md).
normalizer, without consulting extract or merge dependencies. It requires the
current non-empty checkpoint identity to match, so the invocation identity
still binds the input, resolved topology and configuration, references, runtime
overrides, profiles, and component fingerprints.
The runner decodes and canonically re-encodes each reusable artifact once with ## Invariants To Preserve
the prepared codec, requiring exact kind, schema identity and digest, media
type, canonical bytes, content digest, and producer provenance. A valid accepted
producer becomes a runner-owned cloned normalized output, restores only
normalize-checkpoint warnings, and records one `accepted_artifact_reused`
normalize decision. It does not invoke or record extract, merge, normalize, or
their validators. Invalid or unavailable accepted state records its decision
and fails the producer step; the dependent step never starts and the producer
is not implicitly rerun. If a later required lane fails during initialization,
already hydrated terminal lanes remain in the failed output in resolved order.
Generated references add downstream dependencies containing the producer's - The six fixed stages remain explicit; a pipeline is not a general DAG.
artifact kind, complete schema identity, media type, canonical content digest, - Resolution and preparation reject statically discoverable incompatibility
and size. Compatible accepted producer outputs may therefore feed a later step before parsing or execution.
without re-executing the producer. Forced lanes bypass accepted-output - Every typed lane uses one compatible artifact kind, codec, and exact Go type.
hydration and execute normally. A missing, rejected, corrupt, incompatible, or - Generated references come only from one earlier accepted normalized producer
changed producer blocks its dependent while leaving independent work eligible and carry canonical identity rather than an unverified value.
for reuse. The runner records bounded decision - Rejections are recorded outcomes; framework errors cancel derived work and
categories: `reused`, `executed`, `forced_recompute`, and prevent output encoding.
`dependency_invalidated`. Operator meanings for the stable reason codes belong - Public ordering and selected errors are independent of goroutine completion
to [Operations](../operations.md#resume-and-selective-recompute). order.
- Pipeline modules receive collaborators and data, never CLI streams or
physical output, cache, or debug roots.
The CLI includes prepared-component fingerprints in the run-wide checkpoint ## Focused Tests
identity alongside resolved configuration, raw input, reference provenance,
runtime overrides, and LLM-profile fingerprints. Module metadata is not used
implicitly for cache identity: components opt in only with stable semantic
values that can change accepted output. Adding or changing a component
fingerprint intentionally produces a cold cache miss. Existing checkpoint
schemas and paths remain unchanged.
The CLI's `--recompute-step` policy forces the selected step and all transitive - **internal/framework/pipeline/profile_test.go** and
dependents, but requires accepted normalized artifacts for every unselected **typed_resolution_test.go** cover resolution, defaults, ordered steps,
producer on which that closure depends. It changes execution policy only; it compatibility, validators, references, and resolved identity.
does not alter persistent checkpoint identity. - **internal/framework/pipeline/preparation_test.go** covers complete
construction before execution and contextual construction failures.
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
cover external materialization, generated references, provenance, and typed
producer checks.
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
execution, ordered steps, stable error selection, rejections, and
cancellation.
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
**runner_typed_checkpoint_test.go**, and
**runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
**runner_terminal_debug_test.go** cover attempt and terminal debug behavior.
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
boundaries. Every executed chunk, extract, merge, and normalize attempt writes pipeline boundary. Use the more focused tests above while iterating.
one terminal envelope for acceptance, validator rejection, module or validator
error, or applicable candidate or final serialization error. The envelope
contains its attempt-local warnings, any available candidate and rejection,
and terminal error text; normalize retry directives retain their attempt-local
candidate and diagnostic, while only the final safe fallback reaches validation.
Failures before a candidate exists omit that payload.
Only LLM calls made by the module operation belong to the module attempt.
Validator calls retain independent scopes under `validate/` and are not
duplicated into the module envelope. A failed terminal-envelope write is a
non-retryable framework error and is joined with any primary attempt error.
Debug data is never used as a checkpoint source. Typed artifact debug envelopes
are domain-neutral, redact sensitive metadata and bytes through the common
debug policy, and record codec identity plus schema and content digests.
Merge and normalize attempts serialize their in-memory candidate with the
codec's required candidate encoder before typed validation. Serialized
validators and attempt debug use that candidate representation, which carries
the codec media type and schema identity but is never checkpointed or passed
downstream. Only a validator-approved value is encoded through the strict final
codec and made eligible for a checkpoint or stage output.
Checkpoint identity, physical layout, reuse behavior, and debug artifact
handling are operator contracts in [Operations](../operations.md). Serialization
and recorder implementation are inventoried in
[Internal Overview](overview.md#run-state-components).
## Results And Failures
The runner owns manifest assembly and handoff summaries but not the durable JSON
schema. It records resolved module and lane provenance, validator chains,
source/reference identities, selected LLM profiles, normalized and rejected
summaries, status, and timing. Serialized artifact content remains outside the manifest.
Module metadata providers may add non-secret singleton or lane-scoped metadata.
Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).
On a framework failure, the runner cancels its derived context, stops submitting
new extract work, drains started tasks, and skips the output encoder. Parent
cancellation takes precedence. Otherwise context-cancellation fallout is
discarded when a substantive error exists, and the primary error is selected by
stage, resolved lane, and source chunk rather than completion time.
## Tests To Inspect
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
metadata, registration, erasure safety, strict decoding, and cloning.
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
lane resolution and preparation, target-specific validators,
incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
continuations, reverse completion, stable errors, rejection, cancellation,
retries, and independent provider-call limits.
- `internal/framework/pipeline/preparation_test.go`: option validation,
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/cli/run_contract_test.go`: production run transitions, retries,
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
selective recomputation and accepted-producer recovery.
- `internal/cli/production_contract_test.go`: production composition and
configuration-resolution smoke coverage.
- `internal/cli/example_contract_test.go`: maintained example resolution and
execution ownership.
- `internal/modules/integration/*_test.go` and
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
composition across concrete module families.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
collaborators.