234 lines
13 KiB
Markdown
234 lines
13 KiB
Markdown
# Pipeline Internals
|
|
|
|
This document describes the framework-owned pipeline mechanics in
|
|
**internal/framework/pipeline**. [Configuration](../config.md) owns selectable
|
|
profiles, bindings, and retry settings; [Operations](../operations.md) owns
|
|
state lifecycle and recovery; and the [integration contracts](../integrations/)
|
|
own durable output shapes. Concrete production extensions are covered by
|
|
[Module Internals](modules.md).
|
|
|
|
## Boundary
|
|
|
|
The pipeline framework accepts a resolved composition, registries, shared
|
|
dependencies, input bytes, a supplied prompt session, and state/debug
|
|
collaborators. It returns logical output files, normalized artifacts, recorded
|
|
rejections and warnings, manifest provenance, and checkpoint decisions. The
|
|
CLI owns process arguments, configuration discovery, session resolution,
|
|
physical roots, and placement of returned output files.
|
|
|
|
The framework has one fixed shape:
|
|
|
|
~~~
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
~~~
|
|
|
|
Input and chunking are pipeline-wide. A selected artifact lane owns extract,
|
|
merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
|
|
is an ordered list of steps, not an arbitrary workflow graph.
|
|
|
|
## Resolve, Materialize, Prepare
|
|
|
|
Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
|
|
It normalizes the pipeline and lane identities, applies stage defaults, selects
|
|
requested lanes where that is supported, resolves validator chains, checks
|
|
module capabilities and typed artifact compatibility, validates options, and
|
|
assigns a deterministic resolved-composition digest. A correction protocol is
|
|
selected from each eligible LLM-backed producer specification and becomes part
|
|
of that resolved identity; only `single_response_v1` is currently supported.
|
|
Preparation rejects an LLM-backed producer that combines a non-empty validator
|
|
chain with positive producer retries unless it declares that protocol. Producers
|
|
without validators or without retries remain valid without correction support.
|
|
The resolved pipeline contains bindings and declared reference targets, not
|
|
external reference bytes.
|
|
After selection, the resolver applies command, binding, and pipeline profile
|
|
precedence to LLM-backed bindings and validators only; prompt defaults remain
|
|
an empty resolved binding profile. It resolves structural output repair
|
|
separately: a binding's `structured_output_repair_attempts` value wins, then a
|
|
pipeline value applies to LLM-backed bindings and validators, and omission
|
|
leaves the prompt-owned policy intact. An explicit repair value on a
|
|
deterministic binding is rejected. Resolved bindings own copied repair values,
|
|
and these effective values are part of the digest, so execution and checkpoint
|
|
consumers do not repeat profile inheritance or configuration resolution.
|
|
Each LLM request receives its own copy of that resolved value. PromptKit spends
|
|
it only for structural correction inside one completion; the runner's binding
|
|
retry policy remains the separate outer budget for complete stage attempts.
|
|
Configuration resolution supplies the selected profile and catalog; see
|
|
[Configuration Internals](configuration.md).
|
|
|
|
External reference materialization happens before preparation. The materializer
|
|
checks that each slot is declared by the selected module, resolves a file path
|
|
relative to the correct configuration or working-directory origin, reads
|
|
UTF-8 text, verifies media type and size limits, and retains bounded
|
|
provenance. For a positive slot limit, it reads at most the limit plus one byte
|
|
and rejects overflow before retaining content. A generated-artifact selector
|
|
remains declared but has no bytes until its producing step completes.
|
|
|
|
Preparation is the construction boundary. It validates the resolved shape and
|
|
registry set, clones the resolved data, then constructs the input adapter,
|
|
chunker, stage-local validators, every typed lane, and output encoder. The
|
|
prepared producer metadata preserves each selected correction protocol, and
|
|
the resolved digest carrying that metadata participates in checkpoint identity.
|
|
Each registered builder receives its own cloned build request immediately
|
|
before its module-owned code runs. Preparation also collects stable checkpoint
|
|
fingerprints. Missing registrations, incompatible typed entries, nil
|
|
implementations, and constructor failures are reported before source parsing
|
|
or any stage operation begins.
|
|
|
|
An output encoder can opt into source-evidence publication through its output
|
|
policy. Preparation keeps the configured lane allowlist and active lanes
|
|
separate, then verifies an exact typed evidence projector and registered codec
|
|
for each active lane. The resulting private plan is immutable; lanes excluded
|
|
by invocation filtering remain configured but do not acquire a projector for
|
|
that run.
|
|
|
|
## Typed Lanes And References
|
|
|
|
Each resolved lane has one artifact kind, codec, and exact Go type. The
|
|
framework uses private type erasure only around those typed operations; every
|
|
handoff checks exact type and codec identity and reports incompatibility as an
|
|
error rather than panicking. Encoding through the registered codec is the
|
|
boundary for output, checkpoints, debug records, and generated references.
|
|
|
|
Reference targets are stage- and lane-specific. External reference bytes are
|
|
cloned into the operation request. Generated references are built at the next
|
|
step boundary from exactly one accepted normalized producer output. The
|
|
framework decodes and re-encodes that output with the registered producer
|
|
codec, checks its complete schema and media identity, and records a content
|
|
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
|
|
incompatible producer prevents the consumer step from starting.
|
|
|
|
## Execution And Ordering
|
|
|
|
The runner validates its input, installs no-op state collaborators when none
|
|
were supplied, and serially performs source parsing and chunk-plan selection.
|
|
It transports the supplied session unchanged to prompt-facing operations and
|
|
run-manifest metadata; it neither derives a session nor substitutes a parsed
|
|
source document identifier. The public session contract is owned by the
|
|
[CLI reference](../cli.md#run).
|
|
An accepted plan is materialized into source-addressed chunks and passes the
|
|
configured chunk validators before any lane runs. A chunk rejection is a
|
|
recorded pipeline outcome: lanes do not start, but the output stage can encode
|
|
the terminal result.
|
|
|
|
For each ordered step, the runner first builds generated reference sets from
|
|
the accepted normalized outputs of earlier steps. It then executes the step's
|
|
lanes. Later steps do not begin until the current step is terminal and its
|
|
generated handoffs have succeeded.
|
|
|
|
Within a step, the lane engine dispatches extraction jobs in deterministic
|
|
chunk-first, lane-second order to a bounded worker group. When all extraction
|
|
jobs for one lane are terminal, a bounded continuation group can run that
|
|
lane's merge and normalize work while extraction for other lanes continues.
|
|
The framework does not create an unbounded goroutine per chunk or lane.
|
|
|
|
Completion timing does not determine public results. The coordinator restores
|
|
lane and chunk order before merging results, and selects a framework error by
|
|
stable stage, lane, and chunk position. A validator rejection records a lane
|
|
outcome without cancelling unrelated work. A framework error or parent
|
|
cancellation cancels derived work, prevents queued work from starting, waits
|
|
for started workers, and prevents output encoding.
|
|
|
|
## Validation, Retries, And Output
|
|
|
|
Every chunk, extract, merge, and normalize candidate passes its resolved
|
|
validator chain. Validators receive immutable canonical input appropriate to
|
|
their target: chunks, codec-decoded typed candidates, or serialized codec
|
|
bytes. Each typed validator receives a newly decoded value from the one
|
|
candidate serialization for that attempt, while serialized validators receive
|
|
separately owned representation bytes and schema metadata. They may approve,
|
|
approve with warnings, reject, fail, or be skipped when a runtime prerequisite
|
|
is unavailable. The shared executor settles every configured validator in
|
|
order. A failed LLM-backed validator retries only itself against the same
|
|
immutable candidate; it does not regenerate the producer or alter the
|
|
validator request. Rejections stop that validator, while other configured
|
|
validators still run. The executor retains ordered results, bounded
|
|
deduplicated correction guidance from rejections, and only the final exhausted
|
|
failure outcome for each validator.
|
|
|
|
The runner applies the binding's retry policy around a stage operation and its
|
|
complete validation chain. It preserves warnings only from the final accepted
|
|
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
|
|
directives consume this same budget and validate any final safe fallback through
|
|
the normalizer chain.
|
|
|
|
The artifact-neutral producer-attempt state machine owns that shared budget,
|
|
attempt provenance, semantic-correction material, and terminal-policy
|
|
selection. It accepts producer and complete-validation closures, so artifact
|
|
materialization, cache handling, checkpoints, and debug output stay at the
|
|
operation boundary. It distinguishes operational, structural, module-requested,
|
|
and semantic retries. A semantic retry is available only for a valid latest
|
|
`single_response_v1` candidate; a deterministic or no-model rejection instead
|
|
settles the semantic policy immediately. Structural-output errors alone use the
|
|
structural policy, and validation failure without rejection settles the
|
|
validator-failure policy without regenerating the producer.
|
|
|
|
Chunk planning uses this state machine for generated plans. A rejected
|
|
automatic cache hit is not model material and therefore falls through to a
|
|
fresh initial generation; it neither receives a correction nor overwrites the
|
|
stored record. Only a newly generated, completely validated plan is published
|
|
to the chunk-plan store. Rejected plans never advance, and validation-incomplete
|
|
plans remain unpublishable.
|
|
|
|
After terminal lane work, the runner assembles manifest provenance, normalized
|
|
artifacts, rejections, warnings, and an optional accepted chunk map. When an
|
|
output policy selected evidence lanes, it decodes accepted serialized normalize
|
|
outputs through their registered codecs and invokes the prepared typed
|
|
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
|
|
also used after normalized-checkpoint reuse, so no second typed output channel
|
|
is retained. The runner passes the resulting owned artifact to the output
|
|
encoder, which returns logical files and does not choose a physical directory.
|
|
The CLI publishes those files only after the runner returns without a framework
|
|
error. Logical file names and schemas are defined by the [output integration
|
|
contracts](../integrations/).
|
|
|
|
## Checkpoint And Debug Hooks
|
|
|
|
The runner receives checkpoint and debug interfaces rather than roots. It
|
|
records workflow transitions and reuse decisions through the supplied
|
|
collaborators, and clones reusable artifacts before they re-enter normal typed
|
|
handoff. Generated-reference dependencies participate in checkpoint decisions.
|
|
Selective recomputation can require a canonical accepted normalized predecessor
|
|
before a dependent lane starts.
|
|
|
|
Debug recording is attempt-scoped and application-owned. A failure to persist
|
|
required debug data is a framework error. State roots, persistence, reason-code
|
|
meanings, resume, and cleanup are intentionally owned by
|
|
[Run State Internals](state.md) and [Operations](../operations.md).
|
|
|
|
## Invariants To Preserve
|
|
|
|
- The six fixed stages remain explicit; a pipeline is not a general DAG.
|
|
- Resolution and preparation reject statically discoverable incompatibility
|
|
before parsing or execution.
|
|
- Every typed lane uses one compatible artifact kind, codec, and exact Go type.
|
|
- Generated references come only from one earlier accepted normalized producer
|
|
and carry canonical identity rather than an unverified value.
|
|
- Rejections are recorded outcomes; framework errors cancel derived work and
|
|
prevent output encoding.
|
|
- Public ordering and selected errors are independent of goroutine completion
|
|
order.
|
|
- Pipeline modules receive collaborators and data, never CLI streams or
|
|
physical output, cache, or debug roots.
|
|
|
|
## Focused Tests
|
|
|
|
- **internal/framework/pipeline/profile_test.go** and
|
|
**typed_resolution_test.go** cover resolution, defaults, ordered steps,
|
|
compatibility, validators, references, and resolved identity.
|
|
- **internal/framework/pipeline/preparation_test.go** covers complete
|
|
construction before execution and contextual construction failures.
|
|
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
|
|
cover external materialization, generated references, provenance, and typed
|
|
producer checks.
|
|
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
|
|
execution, ordered steps, stable error selection, rejections, and
|
|
cancellation.
|
|
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
|
|
**runner_typed_checkpoint_test.go**, and
|
|
**runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
|
|
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
|
|
**runner_terminal_debug_test.go** cover attempt and terminal debug behavior.
|
|
|
|
Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
|
|
pipeline boundary. Use the more focused tests above while iterating.
|