Add merge references and retry config

This commit is contained in:
2026-07-07 19:10:55 +00:00
parent c05ecb58d8
commit bcedf19a08
25 changed files with 466 additions and 95 deletions

View File

@@ -41,6 +41,10 @@ The runtime records the actual selected Scriptorium profile, provider, and model
used during execution. Manifest population does not rely on a precomputed
profile ID before pipeline execution.
Explicit profile validation and `--llm-profile` overrides apply to LLM-capable
pipeline stages: chunk, extract, merge, and normalize. Input, output, and
validator bindings are not part of the current production LLM profile scope.
## Scriptorium Adapter
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting

View File

@@ -20,7 +20,7 @@ A production module package should provide:
Module specs should describe capabilities accurately. Resolution uses specs to
reject incompatible pipelines before execution.
Chunk, extract, and normalize modules that accept auxiliary reference material
Chunk, extract, merge, and normalize modules that accept auxiliary reference material
must declare slots through both `ReferenceSlots()` and
`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata
should match so config validation can inspect slots without constructing module
@@ -31,10 +31,10 @@ must still be UTF-8 text. When a slot declares accepted media types, Notarius
compares the canonical base media type inferred from the file extension,
case-insensitively and without parameters.
The resolver materializes reference content for chunk, extractor, and
The resolver materializes reference content for chunk, extractor, merger, and
normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`,
`contracts.ExtractionRequest.References`, and
`contracts.NormalizeRequest.References`. Reference material is not source
`contracts.ExtractionRequest.References`, `contracts.MergeRequest.References`,
and `contracts.NormalizeRequest.References`. Reference material is not source
evidence and must not be converted into `SourceRef` values. If a module prompt
uses references, pass them as prompt input materials through the structured LLM
request. Prompt metadata hashes remain based on prompt asset source, not
@@ -48,7 +48,7 @@ Common D&D prompt fragments, reference slot helpers, prompt input assembly, and
reference rendering live under `internal/modules/sharedassets/dnd`. Module
contracts should expose prompt IDs, versions, input material names, and
non-secret prompt/schema hashes through manifest metadata; they should not
expose Scriptorium public types through chunk, extract, or normalize contracts.
expose Scriptorium public types through chunk, extract, merge, or normalize contracts.
Chunk modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, and raw source input material through
@@ -60,6 +60,10 @@ Normalize modules receive the structured LLM client, configured Scriptorium
profile ID, prompt session ID, and reference material through
`contracts.NormalizeRequest` when they need model-backed reconciliation.
Merge modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, raw source input material, and reference material through
`contracts.MergeRequest` when they need model-backed merge behavior.
## `seriatim` Input
Package: `internal/modules/input/seriatim`

View File

@@ -33,18 +33,19 @@ The CLI writes the resolved pipeline and digest to diagnostics.
Pipeline profiles and artifact lanes may include reference binding maps keyed by
reference slot name. During resolution, pipeline-level bindings act as defaults
for selected chunk, extractor, and normalizer targets that declare the slot;
target-local bindings override or add bindings for that target. Runtime
for selected chunk, extractor, merger, and normalizer targets that declare the
slot; target-local bindings override or add bindings for that target. Runtime
`--reference` requests override target config bindings, and runtime unbinds
remove optional target bindings. Flat runtime slot names are resolved only when
exactly one selected target declares the slot; otherwise the CLI requires a more
specific selector such as `chunk.slot`, `lane.extract.slot`, or
`lane.normalize.slot`. Resolution validates bindings against the declaring
target specs and stores the bindings in target-aware resolved reference holders.
It does not read reference files or include reference bytes in source digests.
specific selector such as `chunk.slot`, `lane.extract.slot`,
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
against the declaring target specs and stores the bindings in target-aware
resolved reference holders. It does not read reference files or include
reference bytes in source digests.
During run preparation, resolved file references for chunk, extractor, and
normalizer targets are materialized before any LLM-backed pipeline work. Config
During run preparation, resolved file references for chunk, extractor, merger,
and normalizer targets are materialized before any LLM-backed pipeline work. Config
bindings resolve relative to the config file, and CLI bindings resolve relative
to the current working directory. Materialization accepts UTF-8 text files,
computes `sha256:` content digests, records file origins, infers canonical base
@@ -55,8 +56,8 @@ empty bound files. Media-type acceptance is checked only when a slot declares
manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from
source digests. Runtime reference content is passed to the matching chunker,
extractor, or normalizer request. LLM-backed modules pass that material onward
as named Scriptorium prompt inputs.
extractor, merger, or normalizer request. LLM-backed modules pass that material
onward as named Scriptorium prompt inputs.
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
those bytes into the source document, while LLM-backed modules that need the
@@ -65,9 +66,9 @@ origin metadata. The raw input payload is not written to manifests or default
diagnostics.
The CLI also carries an optional run `session_id`. The runner makes it available
to chunk, extract, and normalize requests; LLM-backed modules forward it through
their structured completion requests so Scriptorium can include it in prompt
execution metadata.
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
through their structured completion requests so Scriptorium can include it in
prompt execution metadata.
## Registries And Module Specs
@@ -83,10 +84,9 @@ Every production module registers a `ModuleSpec` with:
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.
Chunk, extract, and normalize specs may also declare reference slots. Slot
Chunk, extract, merge, and normalize specs may also declare reference slots. Slot
declarations are available from registry metadata without constructing module
instances. Input, merge, validate, and output specs must not declare reference
slots.
instances. Input, validate, and output specs must not declare reference slots.
Capability checks prevent incompatible pipeline composition before a run starts.