notarius/docs/roadmap/implementation.md

ADR-0005 Implementation Record

This document records the implementation of the target state in ADR-0005 Feature Roadmap, governed by ADR-0005.

All stages are complete. This document remains a concise historical implementation record. Current behavior is documented in the canonical references linked from Development.

Execution Rules

Before every stage:

1. read Development, both documents under docs/policy/, ADR-0005, this plan, and every stage-specific document named below;
2. inspect git status and preserve user changes;
3. inspect the focused package contracts and tests before editing; and
4. confirm that all earlier stages are complete.

During every stage:

• implement only that stage and prerequisites discovered to be inseparable;
• use existing package boundaries and helpers before introducing abstractions;
• keep framework and core packages independent of concrete D&D modules;
• preserve cancellation, deterministic ordering, redaction, path confinement, and atomic-write invariants;
• add or update focused tests with every behavior change;
• do not update current-behavior documentation before the final documentation stage, except for ADR acceptance in Stage 1; and
• do not begin the next stage after meeting the current exit criteria.

At the end of every stage:

1. run the focused tests listed for that stage;
2. run go test ./...;
3. run go vet ./...;
4. run go build ./cmd/notarius;
5. run git diff --check; and
6. mark the stage complete in this document only after all checks pass.

If repository reality conflicts with this plan, stop and update the plan in the same change before implementing a materially different design. Do not silently invent a new contract.

Fixed Decisions

These choices are settled for this implementation:

• source.SourceDocument.Digest is the cache lookup identity.
• The chunk-plan root is independent of workspace.directory and every other state root.
• Its default joins the directory returned by os.UserCacheDir with notarius/chunk-plans.
• workspace.chunk_cache.directory overrides that default and names the chunk-plan root itself. The recommended value for a system-wide Linux service is /var/cache/notarius/chunk-plans.
• The plan file is <chunk-plan-root>/<64-character-lowercase-source-sha256-hex>/plan.json. The validated sha256: prefix remains part of the logical digest but is removed from the path segment.
• The plan envelope schema is notarius.chunk-plan.v1.
• There is one mutable plan file per source, atomically replaced. There are no immutable plan objects, pointers, history, locks, or rollback.
• Concurrent writers use last-successful-atomic-write semantics; they must never expose a partial file.
• Chunk modules retain the contracts.Chunker name but replace Chunk with a Plan operation. No legacy interface remains after Stage 2.
• Plans and ranges use map[string]json.RawMessage annotations. Annotation values are canonical valid JSON and namespaces are non-empty trimmed strings.
• Materialized chunks expose range annotations through source.Chunk.Annotations and plan annotations through source.Chunk.PlanAnnotations, never through Metadata.
• Framework materialization owns chunk IDs, indexes, references, content, media type, units, and generic metadata.
• Only the generic chunks capability is hard; chunks.scenes is removed.
• Cached-plan hits still construct the configured chunker during preparation but never call its operation or the chunk-stage LLM.
• The current pipeline's configured chunk validators run against materialized chunks on every path.
• Canonical plan storage is the only chunk-reuse mechanism; chunk checkpoint APIs and payloads are removed.
• The mode values are auto, bypass, and refresh. They are valid in config, environment, and CLI, including persistent refresh configuration.
• The YAML field is workspace.chunk_cache.mode.
• The YAML directory field is workspace.chunk_cache.directory.
• The environment variable is NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE.
• The root environment variable is NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR.
• The CLI flag is --chunk_cache <auto|bypass|refresh>.
• Precedence is CLI, environment, file, then default auto.
• Chunk-plan-root precedence is environment, file, then the per-user default; there is no CLI root override.
• Every supplied file, environment, and CLI value is parsed strictly even when a higher-precedence source would override it; precedence selects among valid values and does not mask malformed configuration.
• Existing chunk checkpoint files are ignored and never migrated.
• A structurally invalid stored plan is an auto miss. A configured-validator rejection of a structurally valid hit does not trigger regeneration.

Planned Contracts

Use these names unless an existing collision requires the smallest obvious adjustment.

Source-zone types

Add to internal/core/source:

type ChunkAnnotations map[string]json.RawMessage

type ChunkPlan struct {
    SourceDigest string           `json:"source_digest"`
    Ranges       []ChunkRange     `json:"ranges"`
    Annotations  ChunkAnnotations `json:"annotations,omitempty"`
}

type ChunkRange struct {
    StartUnitID int              `json:"start_unit_id"`
    EndUnitID   int              `json:"end_unit_id"`
    Annotations ChunkAnnotations `json:"annotations,omitempty"`
}

Add Annotations ChunkAnnotations and PlanAnnotations ChunkAnnotations to source.Chunk, with JSON names annotations and plan_annotations and omitempty on both.

The plan digest covers SourceDigest, ordered ranges, and canonical annotation bytes. It excludes producer provenance, warnings, timestamps, and storage schema. Materialization produces:

• IDs chunk-%06d, numbered from one;
• zero-based indexes in plan order;
• references spanning the first and last units in each range;
• cloned contiguous source units;
• media type application/json;
• canonical JSON content shaped as {"units":[...]};
• generic metadata containing start_unit_id, end_unit_id, and unit_count;
• cloned range annotations in Chunk.Annotations; and
• cloned plan annotations in Chunk.PlanAnnotations.

The plan and range scopes remain distinct even when both use the same namespace. Copying plan annotations into each runtime chunk is intentional: it makes them available to downstream chunk-scoped operations without duplicating source content in durable storage.

Framework validation permits gaps and overlap between ranges, but requires strictly increasing start positions and rejects duplicate or backward ranges. Producing modules may impose stricter policy before returning a plan.

Module contract

Replace the current chunk result and operation with:

type ChunkPlanResult struct {
    Plan     source.ChunkPlan
    Warnings []Warning
}

type Chunker interface {
    Key() string
    ReferenceSlots() []ReferenceSlot
    Plan(context.Context, ChunkRequest) (ChunkPlanResult, error)
}

Keep ChunkRequest, chunk module keys, stage name chunk, configuration bindings, registries, and preparation ownership unchanged.

Durable plan record

Define framework-facing record and store contracts beside other pipeline collaborator contracts:

type ChunkPlanProducer struct {
    InputModule  string
    ChunkModule  string
    LLMProfile   string
    References   []artifacts.ReferenceProvenance
    Metadata     map[string]any
}

type ChunkPlanRecord struct {
    SchemaVersion string
    SourceDigest  string
    PlanDigest    string
    Plan          source.ChunkPlan
    Producer      ChunkPlanProducer
    Warnings      []contracts.Warning
    CreatedAt     time.Time
}

type ChunkPlanStore interface {
    Load(sourceDigest string) (ChunkPlanRecord, ChunkPlanDecision, error)
    Save(ChunkPlanRecord) error
}

type ChunkPlanStoreFactory func(root string) (ChunkPlanStore, error)

type ChunkPlanDecision struct {
    Status ChunkPlanStatus
    Reason string
}

type ChunkPlanStatus string

const (
    ChunkPlanHit     ChunkPlanStatus = "hit"
    ChunkPlanMissing ChunkPlanStatus = "missing"
    ChunkPlanInvalid ChunkPlanStatus = "invalid"
)

ChunkPlanDecision reports hit, missing, or invalid plus a redacted reason. A miss or invalid result returns a zero record and nil error. Missing and structurally invalid records are recoverable in auto. Filesystem access failures other than absence return an error and are framework failures.

The concrete constructor is:

// Package chunkplan
func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error)

It treats root as the exact chunk-plan root. CLI composition injects this constructor through cli.Options.ChunkPlanStoreFactory; tests may inject a recording factory. cli.Options.UserCacheDir has signature func() (string, error) and defaults to os.UserCacheDir.

The JSON envelope uses snake-case equivalents of the record fields. It stores references as provenance only, never reference content. Before returning a hit, the loader validates schema version, source digest, plan digest, plan structure, annotation JSON, and materializability against the current source in the runner.

Cache policy and provenance

Add a ChunkCacheMode enum with auto, bypass, and refresh, plus strict parsing and validation.

Add to RunInput:

ChunkCacheMode ChunkCacheMode
ChunkPlans     ChunkPlanStore

An empty mode used by direct framework callers means bypass; the production CLI always supplies the effective non-empty mode.

Add a run-manifest summary:

type ChunkPlanManifest struct {
    Mode                string                `json:"mode"`
    Action              string                `json:"action,omitempty"`
    SourceDigest        string                `json:"source_digest,omitempty"`
    PlanDigest          string                `json:"plan_digest,omitempty"`
    PlanSchemaVersion   string                `json:"plan_schema_version,omitempty"`
    RequestedModule     string                `json:"requested_module"`
    ProducerInputModule string                `json:"producer_input_module,omitempty"`
    ProducerModule      string                `json:"producer_module,omitempty"`
    ProducerLLMProfile  string                `json:"producer_llm_profile,omitempty"`
    ProducerReferences  []ReferenceProvenance `json:"producer_references,omitempty"`
    ProducerMetadata    map[string]any        `json:"producer_metadata,omitempty"`
    CreatedAt           *time.Time            `json:"created_at,omitempty"`
}

Action is one of reused, generated, refreshed, or bypassed. Retain the existing top-level input_module and chunker fields as the requested pipeline modules. Add this field to artifacts.RunManifest:

ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`

The runner initializes it with the effective mode and requested chunk module, then fills candidate and producer fields when a plan reaches materialization.

Completed Stage Summaries

Stage 1: Finalize ADR and add the source-zone plan model

Status: Complete

Accepted ADR-0005 and added the domain-neutral chunk-plan, range, annotation, validation, digest, cloning, and deterministic materialization primitives in internal/core/source.

Stage 2: Replace the chunk operation with plan generation

Status: Complete

Replaced chunk-returning module operations with Chunker.Plan, converted the generic and D&D scene chunkers to produce ranges and optional annotations, and made the runner materialize and validate generated plans.

Stage 3: Add dedicated cache configuration and the plan store

Status: Complete

Added strict cache modes, independent per-user or explicitly configured chunk-plan roots, the source-addressed filesystem store, restrictive permissions, envelope validation, and atomic replacement.

Stage 4: Integrate cache policy and remove chunk checkpoints

Status: Complete

Implemented runner semantics for auto, bypass, and refresh; retained current chunk-validator behavior; made accepted canonical plans the sole chunk reuse mechanism; and removed chunk checkpoint loading and recording.

Stage 5: Wire persistent policy into the CLI

Status: Complete

Added CLI, environment, and file configuration precedence; per-user root resolution; persistent store construction; bypass isolation; and cross-run reuse through normal CLI execution.

Stage 6: Expose producer provenance and safe diagnostics

Status: Complete

Added requested-versus-producer chunk-plan provenance to manifests and a payload-free chunk-plan decision summary to normal diagnostics, with complete plan material confined to opt-in debug artifacts.

Stage 7: Add cross-run, corruption, and concurrency hardening

Status: Complete

Added integration, corruption, interruption, concurrency, compatibility, cross-domain annotation, import-boundary, and race coverage for the implemented cache design.

Stage 8: Publish current-behavior documentation

Status: Complete

Updated architecture, pipeline, module, configuration, CLI, operations, diagnostics, integration, and overview documentation for the implemented feature and its per-user and system-wide deployment guidance.

Stage 9: Redact invalid cache-record diagnostics

Status: Complete

Objective

Ensure malformed or incompatible cache contents cannot enter normal errors, warnings, manifests, or diagnostics while retaining useful closed lookup decisions.

Read first

• Architecture Policy, especially secret handling
• Diagnostics Internals
• internal/framework/chunkplan/store.go and its focused tests
• internal/framework/pipeline/runner_chunk_plan.go and its focused tests
• internal/core/artifacts/artifacts.go
• the CLI path that writes chunk-plan.json

Implement

1. Stop interpolating decoder errors, unknown field names, schema values, annotation namespaces, timestamps, paths, or any other stored content into recoverable ChunkPlanDecision.Reason values.
2. At the normal-diagnostic boundary, ignore the reason supplied by a ChunkPlanStore and derive ChunkPlanSummary.LookupReason solely from status using exactly:
   • hit: stored chunk plan is valid;
   • missing: chunk plan not found;
   • invalid: stored chunk plan is invalid; and
   • skipped: chunk plan lookup skipped.
3. Keep operational read failures as framework errors rather than recoverable invalid-record decisions. Their normal error text must identify the failed operation without including cache-file content.
4. Do not add raw invalid-record details to another default surface. Direct operator inspection or a future explicitly sensitive debug facility may expose them, but this stage does not need to add such a facility.

Tests

Add focused tests proving that sentinel secrets placed in unknown field names, schema values, annotation namespaces, timestamps, and malformed trailing JSON appear in neither the store decision nor chunk-plan.json. Add a runner test with a custom store whose reason contains a sentinel and prove that the summary uses the fixed status-derived reason. Preserve useful missing, hit, invalid, and operational-error classification.

Run focused tests for internal/framework/chunkplan, internal/framework/pipeline, and internal/cli, then perform the repository-wide validation in the execution rules.

Exit criteria

Every normal lookup reason belongs to the fixed safe vocabulary, hostile invalid-file content cannot reach default diagnostics, operational failures remain correctly classified, and all focused and repository-wide checks pass.

Stage 10: Enforce rooted chunk-plan filesystem access

Status: Complete

Objective

Make the cache's path-confinement guarantee resistant to symlinked cache entries while preserving atomicity, permissions, and concurrent access semantics.

Read first

• Architecture Policy, especially state and path invariants
• Operations
• internal/framework/chunkplan/store.go and all store tests
• Go's os.Root API for the repository's declared Go version

Implement

1. Treat the configured cache root as the filesystem trust boundary. A symlink in the operator-supplied root path itself may be followed when the root is opened, but every store-owned operation after that point must remain beneath the opened root.
2. Perform digest-directory, plan-file, temporary-file, and rename operations through os.Root or an equivalently race-resistant directory-relative mechanism. Do not rely on filepath.Join plus a preflight Lstat as the confinement boundary.
3. Reject symlinks and unexpected file types at the store-owned digest-directory and plan.json positions. A missing cache root on Load remains a normal miss and is not created; Save creates the root and digest directory as needed.
4. Preserve the exact <root>/<full-source-sha256-hex>/plan.json layout, 0700 directories, 0600 plan files, strict digest validation, and no implicit path suffix.
5. Publish using an exclusively created random temporary file inside the digest directory, sync and close it, then atomically rename it through the same rooted handle. Remove temporary files on every pre-rename failure. Preserve last-successful-write and complete-reader semantics.
6. Keep store access serial at the runner boundary and safe for concurrent independent store callers. Do not introduce a lock, history, or rollback protocol.

Tests

Add load and save cases for digest-directory and plan-file symlinks that resolve outside the configured root. Prove that they are rejected and that no outside file is read, created, chmodded, or replaced. Include unexpected file types and failed-publication preservation.

Retain and run tests for absent-root behavior, exact paths, permissions, round trip, malformed digests, atomic replacement, interrupted writes, temporary-file cleanup, concurrent readers, and concurrent writers. Run the race detector for internal/framework/chunkplan, then perform the repository-wide validation in the execution rules.

Exit criteria

No store-owned operation can escape the opened root through a cache entry, normal layout and permissions remain unchanged, publication remains atomic, concurrency tests and the race detector pass, and the repository is green.

Stage 11: Deep-clone source metadata during materialization

Status: Complete

Objective

Ensure source documents, materialized chunks, and separate materializations never share mutable metadata storage, regardless of the concrete JSON-shaped map, slice, array, interface, raw-message, or byte-slice types used by an input adapter.

Read first

• internal/core/source/source.go
• internal/core/source/chunk_plan.go and its focused tests
• internal/core/source/digest.go and validation behavior
• every metadata clone helper and ownership boundary found with rg
• focused runner tests that hand chunks to validators and extractors

Implement

1. Replace the concrete-type allowlist used by plan materialization with one reusable source-owned deep-clone implementation for supported JSON-serializable metadata.
2. Recursively clone string-keyed maps, slices, arrays, interfaces, json.RawMessage, and byte slices without sharing mutable backing storage. Preserve scalar values and the logical JSON representation. Typed composite containers must not fall through as shared values.
3. Detect cyclic or unsupported values and return a contextual error from materialization rather than retaining an alias, panicking, or recursing indefinitely.
4. Use the source-owned helper at pipeline ownership boundaries where doing so removes a duplicate incomplete implementation. Do not introduce a dependency from internal/core/source to a framework package and do not expand this stage into unrelated metadata-schema redesign.
5. Preserve deterministic materialized content, chunk digests, annotations, IDs, references, and metadata fields for all currently supported production inputs.

Tests

Cover typed nested maps, slices of typed maps, arrays, interfaces, json.RawMessage, and byte slices. Mutate each materialized value and prove that neither the source document nor a separately materialized chunk changes. Add cyclic and unsupported-value cases that fail deterministically with contextual errors. Retain the existing exact-materialization and digest tests.

Run focused tests for internal/core/source, internal/framework/pipeline, and affected module packages, then perform the repository-wide validation in the execution rules.

Exit criteria

Every supported mutable metadata value has independent ownership across source, chunk, validation, and extraction boundaries; invalid metadata fails safely; current materialized bytes remain stable; and all checks pass.

Stage 12: Finalize remediation documentation and roadmap status

Status: Complete

Objective

Make the roadmap and canonical documentation accurately describe the completed feature after Stages 9 through 11 land.

Read first

• Documentation Policy
• ADR-0005 Feature Roadmap
• Architecture Policy
• Diagnostics Internals
• Operations
• the final code and tests from Stages 9 through 11

Implement

1. Remove the feature roadmap's statement that ADR-0005 is unimplemented and describe it as the implemented target-state record. Mark its completion outcomes as achieved without adding staged implementation detail.
2. Recheck architecture, diagnostics, operations, pipeline, and module documentation against the remediated behavior. Update a canonical document only where Stages 9 through 11 changed a current operator or developer contract; do not duplicate implementation detail across documents.
3. Mark Stages 9 through 12 complete only after their respective code, documentation, focused tests, and repository-wide validation have passed. Update this document's introduction to state that all stages are complete and that it remains as a concise historical implementation record.
4. Validate every changed link, heading, symbol, diagnostic field, permission, and filesystem claim against the implementation.

Tests and validation

Run:

go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check

Run any repository documentation or link checker if present, and manually inspect the reading map in Development and component links in Internal Overview.

Exit criteria

The feature roadmap and canonical references describe implemented behavior, this plan accurately records completion, documentation ownership remains clear, all links and names are current, and the repository-wide validation is green.