Document chunk plan remediation
This commit is contained in:
@@ -32,7 +32,10 @@ digests, requested module, lookup decision, materialization action, validation
|
|||||||
decision, and publication decision. Its closed decision values make failures
|
decision, and publication decision. Its closed decision values make failures
|
||||||
and recoverable invalid records inspectable without serializing plan ranges,
|
and recoverable invalid records inspectable without serializing plan ranges,
|
||||||
annotations, source content, reference content, prompts, model responses, or
|
annotations, source content, reference content, prompts, model responses, or
|
||||||
raw invalid-file bytes.
|
raw invalid-file bytes. Lookup reasons are derived only from the lookup status:
|
||||||
|
`stored chunk plan is valid`, `chunk plan not found`, `stored chunk plan is
|
||||||
|
invalid`, or `chunk plan lookup skipped`. Store-provided reasons and malformed
|
||||||
|
record details never enter this artifact.
|
||||||
|
|
||||||
JSON methods indent their payload and append a newline. All artifact writes use
|
JSON methods indent their payload and append a newline. All artifact writes use
|
||||||
a temporary file in the target directory, apply the requested permissions, and
|
a temporary file in the target directory, apply the requested permissions, and
|
||||||
|
|||||||
@@ -71,7 +71,8 @@ Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
|
|||||||
unit ranges and may carry optional namespaced JSON annotations; it does not
|
unit ranges and may carry optional namespaced JSON annotations; it does not
|
||||||
contain materialized chunk content. The framework canonicalizes annotations,
|
contain materialized chunk content. The framework canonicalizes annotations,
|
||||||
validates ranges against the current source, and materializes chunk IDs,
|
validates ranges against the current source, and materializes chunk IDs,
|
||||||
indexes, references, content, units, and generic metadata. Annotation
|
indexes, references, content, units, and generic metadata. Materialized source
|
||||||
|
unit metadata is independently owned. Annotation
|
||||||
namespaces remain optional data: generic framework code and downstream modules
|
namespaces remain optional data: generic framework code and downstream modules
|
||||||
must not require D&D scene annotations or import `dnd/scenes`.
|
must not require D&D scene annotations or import `dnd/scenes`.
|
||||||
|
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ normalize continuations that may overlap across lanes.
|
|||||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
||||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |
|
||||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
|
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
|
||||||
|
|
||||||
## Framework Packages
|
## Framework Packages
|
||||||
|
|||||||
@@ -214,7 +214,10 @@ Plan canonicalization requires canonical JSON annotations, a matching source
|
|||||||
digest, at least one range, existing ordered boundaries, and increasing range
|
digest, at least one range, existing ordered boundaries, and increasing range
|
||||||
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
|
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
|
||||||
Materialization deterministically reconstructs each range from the current
|
Materialization deterministically reconstructs each range from the current
|
||||||
source document and copies annotations without interpreting their namespaces.
|
source document, deep-clones JSON-shaped source-unit metadata, and copies
|
||||||
|
annotations without interpreting their namespaces. Materialized chunks and
|
||||||
|
separate materializations do not share mutable unit metadata; unsupported or
|
||||||
|
cyclic metadata fails materialization with context.
|
||||||
|
|
||||||
Before lane execution, generic chunk validation checks the materialized chunks'
|
Before lane execution, generic chunk validation checks the materialized chunks'
|
||||||
identities, order, source references, content, media type, units, and metadata.
|
identities, order, source references, content, media type, units, and metadata.
|
||||||
|
|||||||
@@ -97,6 +97,11 @@ an `auto` run regenerates and atomically replaces it only after validation
|
|||||||
succeeds. Delete an exact cache root or digest directory only when regeneration
|
succeeds. Delete an exact cache root or digest directory only when regeneration
|
||||||
cost is acceptable.
|
cost is acceptable.
|
||||||
|
|
||||||
|
The configured root is the cache trust boundary. An operator-supplied root path
|
||||||
|
may itself resolve through a symlink, but cache-owned digest directories and
|
||||||
|
plan files must be real directory and regular-file entries. Links or other
|
||||||
|
unexpected entry types are rejected rather than followed.
|
||||||
|
|
||||||
Publication uses atomic replacement. Concurrent readers observe a complete old
|
Publication uses atomic replacement. Concurrent readers observe a complete old
|
||||||
or new plan, and concurrent writers leave one complete valid winner; there is
|
or new plan, and concurrent writers leave one complete valid winner; there is
|
||||||
no history, lock protocol, or rollback facility. Do not share a cache root
|
no history, lock protocol, or rollback facility. Do not share a cache root
|
||||||
|
|||||||
@@ -1,9 +1,8 @@
|
|||||||
# ADR-0005 Feature Roadmap
|
# ADR-0005 Feature Roadmap
|
||||||
|
|
||||||
This roadmap defines the intended end state for
|
This roadmap records the implemented target state for
|
||||||
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature
|
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The
|
||||||
is not implemented. The ordered coding work belongs in the
|
[implementation record](implementation.md) preserves the completed work.
|
||||||
[implementation plan](implementation.md).
|
|
||||||
|
|
||||||
## Intent
|
## Intent
|
||||||
|
|
||||||
@@ -156,11 +155,11 @@ This feature does not provide:
|
|||||||
|
|
||||||
## Completion Outcomes
|
## Completion Outcomes
|
||||||
|
|
||||||
The feature is complete when independent runs over the same canonical source
|
**Achieved.** Independent runs over the same canonical source
|
||||||
reuse byte-stable materialized chunks across pipeline, lane, reference,
|
reuse byte-stable materialized chunks across pipeline, lane, reference,
|
||||||
chunk-module, and LLM configuration changes; bypass and refresh obey their
|
chunk-module, and LLM configuration changes; bypass and refresh obey their
|
||||||
documented state semantics; provenance identifies the effective producer;
|
documented state semantics; provenance identifies the effective producer;
|
||||||
legacy chunk checkpoints cannot compete with plan reuse; and all focused,
|
legacy chunk checkpoints cannot compete with plan reuse; and all focused,
|
||||||
integration, compatibility, CLI, and repository-wide validation passes. The
|
integration, compatibility, CLI, and repository-wide validation pass. The
|
||||||
configuration and operations references must also document both the per-user
|
configuration and operations references must also document both the per-user
|
||||||
default and the recommended system-wide Linux setting.
|
default and the recommended system-wide Linux setting.
|
||||||
|
|||||||
@@ -5,10 +5,11 @@ This document records the implementation of the target state in
|
|||||||
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
|
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md).
|
||||||
|
|
||||||
Stages 1 through 8 summarize the completed initial implementation. Stages 9
|
Stages 1 through 8 summarize the completed initial implementation. Stages 9
|
||||||
through 12 track pending remediation identified by the post-implementation
|
through 12 record remediation identified by the post-implementation review and
|
||||||
review. Current behavior is documented in the canonical references linked from
|
retain their individual completion status. Current behavior is documented in the
|
||||||
[Development](../development.md); those references must not describe the Stages
|
canonical references linked from [Development](../development.md); those
|
||||||
9 through 12 target state until the corresponding stage is complete.
|
references must not describe a remediation target state until its corresponding
|
||||||
|
work is complete.
|
||||||
|
|
||||||
## Execution Rules
|
## Execution Rules
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user