Refocus developer and internal documentation
This commit is contained in:
@@ -1,295 +1,185 @@
|
||||
# Pipeline Internals
|
||||
|
||||
The implemented pipeline runner lives in `internal/framework/pipeline`. It
|
||||
executes the fixed workflow defined by the architecture policy:
|
||||
The implemented resolver and runner live in `internal/framework/pipeline`.
|
||||
Their fixed workflow and ownership boundaries are defined by
|
||||
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
|
||||
defaults, and selectable keys are defined in
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
```text
|
||||
input -> chunk -> extract -> merge -> normalize -> output
|
||||
```
|
||||
Pipeline execution is serial. Resolution fixes the selected lanes and all
|
||||
stage bindings before the runner constructs stage implementations.
|
||||
|
||||
Pipeline execution is serial. The runner executes the resolved lanes one after
|
||||
another in the fixed workflow order.
|
||||
## Resolution
|
||||
|
||||
## Profile Resolution
|
||||
`internal/core/config.Config.Resolve` validates the loaded configuration,
|
||||
selects the named profile, applies the runtime inputs supplied by the CLI, and
|
||||
calls `pipeline.ResolvePipeline`.
|
||||
|
||||
Config loading produces `pipeline.PipelineProfile` values. Resolution happens
|
||||
before execution:
|
||||
`ResolvePipeline`:
|
||||
|
||||
1. `internal/core/config.Config.Resolve` validates config and finds the named
|
||||
pipeline.
|
||||
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
|
||||
3. Module bindings are defaulted:
|
||||
- chunk: `generic`
|
||||
- merge: `appendorder`
|
||||
- normalize: `noop`
|
||||
- output: `json`
|
||||
- LLM profile: empty, which lets Scriptorium prompt defaults choose a
|
||||
profile.
|
||||
4. The module catalog is checked for each bound module key.
|
||||
5. Module capabilities are checked in workflow order.
|
||||
6. A digest is calculated from the resolved pipeline without the digest field.
|
||||
1. selects and sorts artifact lanes;
|
||||
2. completes omitted bindings using the documented configuration defaults;
|
||||
3. looks up each module and validator spec without constructing it;
|
||||
4. checks required and provided capabilities in workflow order;
|
||||
5. resolves target-aware reference bindings and validator chains;
|
||||
6. calculates a digest over the resolved structure.
|
||||
|
||||
The CLI writes the resolved pipeline and digest to diagnostics.
|
||||
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
||||
bindings, validator chains, reference targets, and the digest. It does not read
|
||||
reference bytes or construct runtime modules. CLI lane and reference selector
|
||||
syntax is defined in the [CLI reference](../cli.md#run).
|
||||
|
||||
Pipeline profiles and artifact lanes may include reference binding maps keyed by
|
||||
reference slot name. During resolution, pipeline-level bindings act as defaults
|
||||
for selected chunk, extractor, merger, and normalizer targets that declare the
|
||||
slot; target-local bindings override or add bindings for that target. Runtime
|
||||
`--reference` requests override target config bindings, and runtime unbinds
|
||||
remove optional target bindings. Flat runtime slot names are resolved only when
|
||||
exactly one selected target declares the slot; otherwise the CLI requires a more
|
||||
specific selector such as `chunk.slot`, `lane.extract.slot`,
|
||||
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
|
||||
against the declaring target specs and stores the bindings in target-aware
|
||||
resolved reference holders. It does not read reference files or include
|
||||
reference bytes in source digests.
|
||||
## Reference Materialization
|
||||
|
||||
During run preparation, resolved file references for chunk, extractor, merger,
|
||||
and normalizer targets are materialized before any LLM-backed pipeline work. Config
|
||||
bindings resolve relative to the config file, and CLI bindings resolve relative
|
||||
to the current working directory. Materialization accepts UTF-8 text files,
|
||||
computes `sha256:` content digests, records file origins, infers canonical base
|
||||
media types from file extensions, enforces declared byte limits, and warns for
|
||||
empty bound files. Media-type acceptance is checked only when a slot declares
|
||||
`AcceptedMediaTypes`; unknown extensions are recorded as
|
||||
`application/octet-stream`. Reference content is omitted from diagnostics and
|
||||
manifests. The CLI writes provenance-only resolved reference diagnostics, and
|
||||
the run manifest records target-stage reference provenance separately from
|
||||
source digests. Runtime reference content is passed to the matching chunker,
|
||||
extractor, merger, or normalizer request. LLM-backed modules pass that material
|
||||
onward as named Scriptorium prompt inputs.
|
||||
The CLI calls `MaterializeReferences` after resolution and before constructing
|
||||
the LLM client or running the pipeline. The materializer checks each binding
|
||||
against its resolved target declaration, reads and validates the file, and
|
||||
builds both a `contracts.ReferenceSet` and provenance-only metadata on the
|
||||
corresponding `ResolvedReferenceTarget`.
|
||||
|
||||
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
|
||||
those bytes into the source document. Chunk, merge, and normalize requests
|
||||
receive the original source material as `SourceInput`; extraction requests
|
||||
receive chunk-scoped source material built from the current `SourceChunk`
|
||||
content, media type, and origin metadata. The raw input payload is not written
|
||||
to manifests or default diagnostics.
|
||||
The runner clones the resulting set into the chunk, extract, merge, or normalize
|
||||
request that owns the target. LLM-backed extensions may convert those items into
|
||||
named prompt inputs. Reference content remains separate from source evidence and
|
||||
source digests.
|
||||
|
||||
The CLI also carries an optional run `session_id`. The runner makes it available
|
||||
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
|
||||
through their structured completion requests so Scriptorium can include it in
|
||||
prompt execution metadata.
|
||||
Binding precedence, path resolution, accepted content, and media-type behavior
|
||||
are configuration contracts; see [Configuration](../config.md#pipelines).
|
||||
Durable provenance is defined in the
|
||||
[JSON output contract](../integrations/json-output.md#manifestjson), while
|
||||
runtime sensitive-data handling belongs in [Operations](../operations.md).
|
||||
|
||||
When workspace resume checkpointing is enabled, the CLI constructs a checkpoint
|
||||
recorder after pipeline resolution and reference materialization and passes it
|
||||
through `pipeline.RunInput`. The runner records source, chunk, extract, merge,
|
||||
and normalize outcomes through that interface. Concrete modules do not receive
|
||||
workspace paths and do not write checkpoint files directly.
|
||||
## Registries And Specs
|
||||
|
||||
For `run --resume`, the CLI also passes a checkpoint loader. The runner consults
|
||||
the loader in workflow order and reuses only checkpoints whose manifest schema,
|
||||
status, identity digest, dependency fingerprints, payload files, and payload
|
||||
digests validate for the current invocation. The identity includes the resolved
|
||||
pipeline, selected lanes, source/input digest, runtime overrides that affect
|
||||
execution, and materialized reference digests. Missing or invalid checkpoints
|
||||
fall back to normal execution and are refreshed by the recorder.
|
||||
`pipeline.Registries` holds constructors used during execution.
|
||||
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
||||
resolution. Separate registries exist for every stage and for validators;
|
||||
`ValidatorChainRegistry` stores production default-chain mappings.
|
||||
|
||||
When workspace debug output is enabled, the CLI passes a debug recorder for the
|
||||
current run ID. The runner writes framework-boundary inputs, outputs,
|
||||
structured LLM calls, validator calls, timing, and retry attempt metadata
|
||||
through that interface. Each retry or validator attempt records any LLM calls
|
||||
made within that attempt in an `llm_calls` array and writes paired
|
||||
`prompt-000N.json` and `response-000N.json` metadata files under the attempt
|
||||
directory. LLM response bodies are written as sibling `response-content-000N.*`
|
||||
files, using pretty-printed JSON when the content is valid JSON and raw text
|
||||
otherwise. Debug output is not used for resume and can contain sensitive source,
|
||||
reference, prompt, and model-output material. Concrete modules still do not
|
||||
receive workspace paths.
|
||||
A `ModuleSpec` declares its stage plus required and provided capabilities.
|
||||
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
||||
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
||||
and verify that a constructed implementation reports the registered key.
|
||||
|
||||
## Registries And Module Specs
|
||||
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
||||
the execution class to reject incompatible profile bindings before execution.
|
||||
The current production catalog and default chain are listed only in
|
||||
[Configuration](../config.md#implemented-production-validators).
|
||||
|
||||
`pipeline.Registries` holds concrete constructors for execution. A
|
||||
`pipeline.ModuleCatalog` exposes module specs for config validation and
|
||||
resolution. The catalog also exposes validator specs and central default
|
||||
validator-chain mappings without constructing modules or validators.
|
||||
## Runner Boundary
|
||||
|
||||
Every production module registers a `ModuleSpec` with:
|
||||
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
|
||||
LLM client, run identity and timing, optional session and profile metadata, and
|
||||
checkpoint/debug collaborators. The runner parses source bytes through the
|
||||
selected input adapter. Later stage requests receive the generic source model;
|
||||
extract requests receive chunk-scoped input material, while chunk, merge, and
|
||||
normalize requests retain access to the original source material.
|
||||
|
||||
- `Key`: module key used in config;
|
||||
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
|
||||
validate, or output;
|
||||
- `Provides`: capabilities added after that module runs;
|
||||
- `Requires`: capabilities that must already be available.
|
||||
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
|
||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
||||
output encoder. The CLI owns diagnostics and durable filesystem writes after the
|
||||
runner returns.
|
||||
|
||||
Chunk, extract, merge, and normalize specs may also declare reference slots. Slot
|
||||
declarations are available from registry metadata without constructing module
|
||||
instances. Input, validate, and output specs must not declare reference slots.
|
||||
|
||||
Capability checks prevent incompatible pipeline composition before a run starts.
|
||||
|
||||
Every production validator registers a `ValidatorSpec` with:
|
||||
|
||||
- `Key`: validator key used in config and manifests;
|
||||
- `ExecutionClass`: `deterministic` or `llm_backed`.
|
||||
|
||||
Default validator chains are keyed by workflow stage and module key. Production
|
||||
currently registers a default chain for `extract` module `dnd/spells` only.
|
||||
|
||||
## Runner Input And Output
|
||||
|
||||
`pipeline.RunInput` carries:
|
||||
|
||||
- a `ResolvedPipeline`;
|
||||
- optional source ID, input path, and raw input bytes;
|
||||
- a structured LLM client;
|
||||
- run ID, start time, LLM profile manifest metadata, and CLI metadata.
|
||||
|
||||
`pipeline.RunOutput` carries:
|
||||
|
||||
- run manifest;
|
||||
- normalized raw outputs;
|
||||
- rejected raw outputs;
|
||||
- warnings;
|
||||
- logical output files returned by the output encoder.
|
||||
|
||||
The CLI owns durable file writes and diagnostics writes after the runner returns.
|
||||
|
||||
## Execution
|
||||
## Execution Flow
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates run input and registries;
|
||||
2. builds the input adapter and parses the raw input into a source document;
|
||||
3. validates the source document;
|
||||
4. builds the chunker and produces source chunks, retrying when configured;
|
||||
5. validates source chunks against framework invariants and the resolved chunk
|
||||
validator chain;
|
||||
6. runs each selected artifact lane in sorted resolved order;
|
||||
7. builds the output encoder and validates logical output file names.
|
||||
8. passes accepted normalized raw outputs, rejected output records, warnings,
|
||||
and the manifest to the output encoder.
|
||||
1. validates its input and registries;
|
||||
2. builds the input adapter, parses the raw input, and validates the generic
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
5. executes each resolved artifact lane in order;
|
||||
6. builds the output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
## Chunk Results
|
||||
Within each artifact lane, it builds the extractor, merger, and normalizer,
|
||||
then performs these transitions:
|
||||
|
||||
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
|
||||
with the validated source document, reference set, structured LLM client, the
|
||||
configured LLM profile, module options, and run metadata. Deterministic and
|
||||
LLM-backed chunkers use the same contract; provider construction stays outside
|
||||
chunk modules.
|
||||
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
||||
provenance;
|
||||
2. validate each raw extract result and omit rejected results from merge input;
|
||||
3. skip the rest of the lane when no extract result is accepted;
|
||||
4. merge accepted extract results in their existing order;
|
||||
5. validate the merge result and skip normalization on rejection;
|
||||
6. normalize the accepted merge result;
|
||||
7. validate and append the accepted normalized result.
|
||||
|
||||
When chunking succeeds, the runner validates generic chunk invariants before
|
||||
running extractors:
|
||||
Module-provided warnings and payload warnings are promoted only from attempts
|
||||
whose results are accepted and used.
|
||||
|
||||
- chunk IDs must be non-empty and unique in the chunk result;
|
||||
- each chunk `SourceID` must match the source document ID;
|
||||
- each chunk `Index` must match its zero-based returned order;
|
||||
- each chunk start and end unit ID must exist in the source document, with the
|
||||
start unit at or before the end unit;
|
||||
- each chunk must include non-empty extraction content and media type;
|
||||
- each chunk must contain at least one source unit;
|
||||
- a chunk must not repeat a source unit;
|
||||
- every chunk source unit must exist in the source document;
|
||||
- source units inside each chunk must appear in source-document order.
|
||||
## Chunk Canonicalization
|
||||
|
||||
After validation, the runner rebuilds each chunk from source-document units by
|
||||
integer ID, preserving chunk boundaries, content bytes, media type, and cloned
|
||||
chunk metadata. Extractors and downstream stages therefore see canonical source
|
||||
units, while `SourceChunk.Metadata` remains the supported place for
|
||||
chunker-owned context.
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
source identity, indexes matching returned order, valid ordered boundaries,
|
||||
non-empty content and media type, and at least one valid source unit per chunk.
|
||||
Units may not repeat inside a chunk and must preserve source-document order.
|
||||
|
||||
If chunk validation rejects the chunk result after configured retries, the runner
|
||||
records a rejected raw output and skips downstream lane execution. Framework-level
|
||||
chunking or validation errors that remain after configured retries fail the run.
|
||||
The runner then rebuilds each chunk's unit slice from the source document by
|
||||
unit ID. It preserves the module-owned boundaries, content, media type, and
|
||||
cloned metadata. The framework permits gaps and overlap between separate
|
||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
||||
|
||||
The framework does not require complete source-unit coverage and does not reject
|
||||
overlap between different chunks. Stricter policies, such as full coverage or
|
||||
non-overlap, belong to individual chunk modules when they are part of that
|
||||
module's contract.
|
||||
## Validation And Retries
|
||||
|
||||
Within an artifact lane, the runner:
|
||||
Chunk, extract, merge, and normalize results pass through the resolved validator
|
||||
chain for their stage and module. Each validator receives the raw payload plus
|
||||
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
|
||||
and run context. Validators execute in resolved order and stop at the first
|
||||
error or rejection. An empty chain approves the result.
|
||||
|
||||
1. builds the extractor, merger, and normalizer;
|
||||
2. records module manifest metadata when modules provide it;
|
||||
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
|
||||
configured;
|
||||
4. fills runner-owned provenance on each extract output, including lane ID,
|
||||
extractor key, source ID, chunk ID, and chunk index;
|
||||
5. validates raw extract outputs and omits rejected outputs from merge input;
|
||||
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
|
||||
when configured;
|
||||
7. validates raw merge output and skips normalization for rejected merge output;
|
||||
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
|
||||
retrying when configured;
|
||||
9. validates raw normalize output and appends accepted normalized raw output to
|
||||
`RunOutput.NormalizeOutputs`.
|
||||
`runWithRetry` performs the initial module call plus the configured additional
|
||||
attempts. Each attempt includes module execution and its complete validation
|
||||
chain. A module or validator error retries and becomes a framework error after
|
||||
the final attempt. A rejection retries and becomes a recorded `RejectedOutput`
|
||||
after the final attempt. Cancellation stops retry processing immediately.
|
||||
|
||||
## Validators
|
||||
Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
|
||||
from discarded attempts are not promoted. Configuration owns retry counts and
|
||||
validator overrides; see [Module Bindings](../config.md#module-bindings).
|
||||
|
||||
The runner handoff is raw-output based. Chunkers, extractors, mergers, and
|
||||
normalizers do not advertise validator chains through their module interfaces.
|
||||
Resolved validation chains receive the raw module output plus stage, lane,
|
||||
module, source, chunk, schema, session, reference, LLM client/profile, binding
|
||||
option, and run metadata context. Chunk validators receive the chunk result
|
||||
collection, merge validators receive the ordered extract outputs used by the
|
||||
merge, and normalize validators receive the accepted merge output. Empty chains
|
||||
approve output by default. Response-schema provenance may include in-memory JSON
|
||||
schema bytes for validators. Those bytes are omitted from manifests,
|
||||
diagnostics, and encoded output files.
|
||||
## Checkpoint And Debug Hooks
|
||||
|
||||
Resolved validator chains come from central default mappings unless a
|
||||
stage-local config override is set on `chunk`, lane `extract`, lane `merge`, or
|
||||
lane `normalize`. Explicit empty overrides are valid and are recorded as empty
|
||||
chains in manifests. Explicit non-empty overrides replace the default chain and
|
||||
preserve configured order.
|
||||
The runner depends on recorder and loader interfaces, using no-op
|
||||
implementations when collaborators are absent. Each checkpointed workflow
|
||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||
are consulted in workflow order and accepted payloads are cloned before
|
||||
entering the normal handoff path. Dependency fingerprints connect later
|
||||
checkpoints to the exact accepted results on which they depend.
|
||||
|
||||
The production default chain for `extract` module `dnd/spells` is:
|
||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||
boundaries. Context scopes associate nested LLM calls with the module or
|
||||
validator attempt that made them. Debug-write failures are framework errors;
|
||||
debug data is never used as a checkpoint source.
|
||||
|
||||
1. `generic/valid_json`
|
||||
2. `generic/valid_json_schema`
|
||||
3. `extract/dnd/spells/shape`
|
||||
4. `extract/dnd/spells/source_refs`
|
||||
5. `extract/dnd/spells/source_relatedness`
|
||||
Checkpoint identity, physical layout, reuse behavior, and debug artifact
|
||||
handling are operator contracts in [Operations](../operations.md). Serialization
|
||||
and recorder implementation are inventoried in
|
||||
[Internal Overview](overview.md#run-state-components).
|
||||
|
||||
No other production module currently has a default validator chain.
|
||||
## Results And Failures
|
||||
|
||||
Validator rejection is a non-fatal run outcome: the rejected output is recorded
|
||||
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
|
||||
errors are framework-level errors and retry according to the relevant binding.
|
||||
Warning-only validators return approved results with warnings; those warnings
|
||||
are promoted only from successful attempts whose outputs are used.
|
||||
The runner owns manifest assembly and handoff summaries but not the durable JSON
|
||||
schema. It records resolved module and lane provenance, validator chains,
|
||||
source/reference identities, selected LLM profiles, normalized and rejected
|
||||
summaries, status, and timing. Raw payload bytes remain outside the manifest.
|
||||
Module metadata providers may add non-secret singleton or lane-scoped metadata.
|
||||
|
||||
## Warnings And Failures
|
||||
Execution errors include stage, module, lane, or validator context. Once a
|
||||
manifest exists, a failing run returns it with failed status and completion
|
||||
time. Successful status reflects whether any raw result was rejected. The
|
||||
durable manifest and logical file schemas are defined in the
|
||||
[JSON output contract](../integrations/json-output.md).
|
||||
|
||||
Warnings from the successful chunking, extraction, merging, and normalization
|
||||
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
|
||||
with output encoder warnings. Warnings from discarded retry attempts are not
|
||||
promoted to final warnings.
|
||||
## Tests To Inspect
|
||||
|
||||
Errors wrap the operation and module key or lane context. If execution fails
|
||||
after a manifest exists, the returned manifest is marked `failed` and receives a
|
||||
completion timestamp.
|
||||
|
||||
On successful execution, the manifest validation status is:
|
||||
|
||||
- `approved` when no raw outputs were rejected;
|
||||
- `rejected` when at least one raw output was rejected.
|
||||
|
||||
## Manifest Population
|
||||
|
||||
The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
|
||||
module metadata, artifact lanes, LLM profile metadata, source digest,
|
||||
reference provenance, normalized raw output summaries, rejected output
|
||||
summaries, validation status, and timing. Raw output summaries include lane ID,
|
||||
normalizer module key, media type, source ID, and response-schema provenance
|
||||
when present. Rejected output summaries include stage, lane, module, chunk,
|
||||
validator or reason, message, attempt count, and optional diagnostic artifact
|
||||
path. The manifest does not include raw output payload bytes.
|
||||
|
||||
Singleton pipeline modules may add non-secret metadata by implementing
|
||||
`contracts.ManifestMetadataProvider`. The runner records that metadata under
|
||||
`module_metadata` with stable keys for `input`, `chunker`, and `output`.
|
||||
|
||||
Lane-owned modules may add non-secret metadata through
|
||||
`artifact_lanes[].metadata`. The runner records extractor, merger, and
|
||||
normalizer metadata there. The D&D spell extractor uses lane metadata for
|
||||
prompt and response-schema provenance.
|
||||
|
||||
## JSON Output
|
||||
|
||||
The production JSON output encoder writes `manifest.json`, `index.json`,
|
||||
`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted
|
||||
normalized lane output under `lanes/`. It accepts only normalized outputs with
|
||||
valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe
|
||||
logical paths, and duplicate sanitized lane file names fail the run before
|
||||
durable output files are written.
|
||||
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
||||
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
|
||||
capabilities, validator chains, and digest behavior.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
|
||||
rejections, warnings, checkpoints, debug hooks, and manifests.
|
||||
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
|
||||
workflow composition.
|
||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
|
||||
collaborators.
|
||||
|
||||
Reference in New Issue
Block a user