Move chunks into the canonical source model
This commit is contained in:
@@ -57,7 +57,8 @@ rules are defined in the
|
||||
The generic chunker validates the source document, walks units in configured
|
||||
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
||||
Overlap changes the next window start but never reorders units. It records the
|
||||
first and last unit and unit count in chunk metadata.
|
||||
first and last unit and unit count in chunk metadata, and derives the chunk's
|
||||
canonical source reference from those unit references.
|
||||
|
||||
The accepted options and defaults are defined in
|
||||
[Configuration](../config.md#implemented-production-modules). Generic
|
||||
@@ -68,7 +69,7 @@ framework validation canonicalizes the returned unit slices before extraction.
|
||||
The scene chunker prepares a structured Scriptorium request from the full
|
||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
||||
scene boundaries against source-unit IDs and converts them into deterministic
|
||||
chunks.
|
||||
chunks with canonical source references spanning each scene's units.
|
||||
|
||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
||||
the first source unit through the last. Each chunk contains JSON scene content
|
||||
|
||||
@@ -31,7 +31,7 @@ a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||
| `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
|
||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
||||
|
||||
## Framework Packages
|
||||
|
||||
@@ -77,7 +77,8 @@ normalize requests retain access to the original source material.
|
||||
Source validation requires every unit to carry a canonical self-reference to
|
||||
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
||||
debug boundaries retain that reference, and the canonical source digest covers
|
||||
it deterministically.
|
||||
it deterministically. Chunks use the same source model and carry one canonical
|
||||
reference spanning the first selected unit through the last.
|
||||
|
||||
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
|
||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
||||
@@ -115,13 +116,15 @@ whose results are accepted and used.
|
||||
## Chunk Canonicalization
|
||||
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
source identity, indexes matching returned order, valid ordered boundaries,
|
||||
source identity, indexes matching returned order, a valid canonical reference,
|
||||
non-empty content and media type, and at least one valid source unit per chunk.
|
||||
Units may not repeat inside a chunk and must preserve source-document order.
|
||||
Units may not repeat inside a chunk and must form a contiguous range in
|
||||
source-document order. The chunk reference must exactly match the source and
|
||||
the first and last unit references.
|
||||
|
||||
The runner then rebuilds each chunk's unit slice from the source document by
|
||||
unit ID. It preserves the module-owned boundaries, content, media type, and
|
||||
cloned metadata. The framework permits gaps and overlap between separate
|
||||
unit ID. It preserves the canonical reference, content, media type, and cloned
|
||||
metadata. The framework permits gaps and overlap between separate
|
||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
||||
|
||||
## Validation And Retries
|
||||
|
||||
Reference in New Issue
Block a user