eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6d0a19c94c2bb307823edd7c1079fe23dc678dc8
notarius/docs/roadmap/implementation.md

41 KiB
Raw Blame History

ADR-0005 Staged Implementation Plan

This document is the executable implementation plan for the target state in ADR-0005 Feature Roadmap, governed by ADR-0005. The feature is not implemented.

The audience is an LLM coding agent. Implement the stages in order. Each stage is deliberately scoped to finish with a compiling, tested repository and may be assigned as one implementation prompt.

Execution Rules

Before every stage:

  1. read Development, both documents under docs/policy/, ADR-0005, this plan, and every stage-specific document named below;
  2. inspect git status and preserve user changes;
  3. inspect the focused package contracts and tests before editing; and
  4. confirm that all earlier stages are complete.

During every stage:

  • implement only that stage and prerequisites discovered to be inseparable;
  • use existing package boundaries and helpers before introducing abstractions;
  • keep framework and core packages independent of concrete D&D modules;
  • preserve cancellation, deterministic ordering, redaction, path confinement, and atomic-write invariants;
  • add or update focused tests with every behavior change;
  • do not update current-behavior documentation before the final documentation stage, except for ADR acceptance in Stage 1; and
  • do not begin the next stage after meeting the current exit criteria.

At the end of every stage:

  1. run the focused tests listed for that stage;
  2. run go test ./...;
  3. run go vet ./...;
  4. run go build ./cmd/notarius;
  5. run git diff --check; and
  6. mark the stage complete in this document only after all checks pass.

If repository reality conflicts with this plan, stop and update the plan in the same change before implementing a materially different design. Do not silently invent a new contract.

Fixed Decisions

These choices are settled for this implementation:

  • source.SourceDocument.Digest is the cache lookup identity.
  • The chunk-plan root is independent of workspace.directory and every other state root.
  • Its default joins the directory returned by os.UserCacheDir with notarius/chunk-plans.
  • workspace.chunk_cache.directory overrides that default and names the chunk-plan root itself. The recommended value for a system-wide Linux service is /var/cache/notarius/chunk-plans.
  • The plan file is <chunk-plan-root>/<64-character-lowercase-source-sha256-hex>/plan.json. The validated sha256: prefix remains part of the logical digest but is removed from the path segment.
  • The plan envelope schema is notarius.chunk-plan.v1.
  • There is one mutable plan file per source, atomically replaced. There are no immutable plan objects, pointers, history, locks, or rollback.
  • Concurrent writers use last-successful-atomic-write semantics; they must never expose a partial file.
  • Chunk modules retain the contracts.Chunker name but replace Chunk with a Plan operation. No legacy interface remains after Stage 2.
  • Plans and ranges use map[string]json.RawMessage annotations. Annotation values are canonical valid JSON and namespaces are non-empty trimmed strings.
  • Materialized chunks expose range annotations through source.Chunk.Annotations and plan annotations through source.Chunk.PlanAnnotations, never through Metadata.
  • Framework materialization owns chunk IDs, indexes, references, content, media type, units, and generic metadata.
  • Only the generic chunks capability is hard; chunks.scenes is removed.
  • Cached-plan hits still construct the configured chunker during preparation but never call its operation or the chunk-stage LLM.
  • The current pipeline's configured chunk validators run against materialized chunks on every path.
  • Canonical plan storage is the only chunk-reuse mechanism; chunk checkpoint APIs and payloads are removed.
  • The mode values are auto, bypass, and refresh. They are valid in config, environment, and CLI, including persistent refresh configuration.
  • The YAML field is workspace.chunk_cache.mode.
  • The YAML directory field is workspace.chunk_cache.directory.
  • The environment variable is NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE.
  • The root environment variable is NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR.
  • The CLI flag is --chunk_cache <auto|bypass|refresh>.
  • Precedence is CLI, environment, file, then default auto.
  • Chunk-plan-root precedence is environment, file, then the per-user default; there is no CLI root override.
  • Every supplied file, environment, and CLI value is parsed strictly even when a higher-precedence source would override it; precedence selects among valid values and does not mask malformed configuration.
  • Existing chunk checkpoint files are ignored and never migrated.
  • A structurally invalid stored plan is an auto miss. A configured-validator rejection of a structurally valid hit does not trigger regeneration.

Planned Contracts

Use these names unless an existing collision requires the smallest obvious adjustment.

Source-zone types

Add to internal/core/source:

type ChunkAnnotations map[string]json.RawMessage

type ChunkPlan struct {
    SourceDigest string           `json:"source_digest"`
    Ranges       []ChunkRange     `json:"ranges"`
    Annotations  ChunkAnnotations `json:"annotations,omitempty"`
}

type ChunkRange struct {
    StartUnitID int              `json:"start_unit_id"`
    EndUnitID   int              `json:"end_unit_id"`
    Annotations ChunkAnnotations `json:"annotations,omitempty"`
}

Add Annotations ChunkAnnotations and PlanAnnotations ChunkAnnotations to source.Chunk, with JSON names annotations and plan_annotations and omitempty on both.

The plan digest covers SourceDigest, ordered ranges, and canonical annotation bytes. It excludes producer provenance, warnings, timestamps, and storage schema. Materialization produces:

  • IDs chunk-%06d, numbered from one;
  • zero-based indexes in plan order;
  • references spanning the first and last units in each range;
  • cloned contiguous source units;
  • media type application/json;
  • canonical JSON content shaped as {"units":[...]};
  • generic metadata containing start_unit_id, end_unit_id, and unit_count;
  • cloned range annotations in Chunk.Annotations; and
  • cloned plan annotations in Chunk.PlanAnnotations.

The plan and range scopes remain distinct even when both use the same namespace. Copying plan annotations into each runtime chunk is intentional: it makes them available to downstream chunk-scoped operations without duplicating source content in durable storage.

Framework validation permits gaps and overlap between ranges, but requires strictly increasing start positions and rejects duplicate or backward ranges. Producing modules may impose stricter policy before returning a plan.

Module contract

Replace the current chunk result and operation with:

type ChunkPlanResult struct {
    Plan     source.ChunkPlan
    Warnings []Warning
}

type Chunker interface {
    Key() string
    ReferenceSlots() []ReferenceSlot
    Plan(context.Context, ChunkRequest) (ChunkPlanResult, error)
}

Keep ChunkRequest, chunk module keys, stage name chunk, configuration bindings, registries, and preparation ownership unchanged.

Durable plan record

Define framework-facing record and store contracts beside other pipeline collaborator contracts:

type ChunkPlanProducer struct {
    InputModule  string
    ChunkModule  string
    LLMProfile   string
    References   []artifacts.ReferenceProvenance
    Metadata     map[string]any
}

type ChunkPlanRecord struct {
    SchemaVersion string
    SourceDigest  string
    PlanDigest    string
    Plan          source.ChunkPlan
    Producer      ChunkPlanProducer
    Warnings      []contracts.Warning
    CreatedAt     time.Time
}

type ChunkPlanStore interface {
    Load(sourceDigest string) (ChunkPlanRecord, ChunkPlanDecision, error)
    Save(ChunkPlanRecord) error
}

type ChunkPlanStoreFactory func(root string) (ChunkPlanStore, error)

type ChunkPlanDecision struct {
    Status ChunkPlanStatus
    Reason string
}

type ChunkPlanStatus string

const (
    ChunkPlanHit     ChunkPlanStatus = "hit"
    ChunkPlanMissing ChunkPlanStatus = "missing"
    ChunkPlanInvalid ChunkPlanStatus = "invalid"
)

ChunkPlanDecision reports hit, missing, or invalid plus a redacted reason. A miss or invalid result returns a zero record and nil error. Missing and structurally invalid records are recoverable in auto. Filesystem access failures other than absence return an error and are framework failures.

The concrete constructor is:

// Package chunkplan
func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error)

It treats root as the exact chunk-plan root. CLI composition injects this constructor through cli.Options.ChunkPlanStoreFactory; tests may inject a recording factory. cli.Options.UserCacheDir has signature func() (string, error) and defaults to os.UserCacheDir.

The JSON envelope uses snake-case equivalents of the record fields. It stores references as provenance only, never reference content. Before returning a hit, the loader validates schema version, source digest, plan digest, plan structure, annotation JSON, and materializability against the current source in the runner.

Cache policy and provenance

Add a ChunkCacheMode enum with auto, bypass, and refresh, plus strict parsing and validation.

Add to RunInput:

ChunkCacheMode ChunkCacheMode
ChunkPlans     ChunkPlanStore

An empty mode used by direct framework callers means bypass; the production CLI always supplies the effective non-empty mode.

Add a run-manifest summary:

type ChunkPlanManifest struct {
    Mode                string                `json:"mode"`
    Action              string                `json:"action,omitempty"`
    SourceDigest        string                `json:"source_digest,omitempty"`
    PlanDigest          string                `json:"plan_digest,omitempty"`
    PlanSchemaVersion   string                `json:"plan_schema_version,omitempty"`
    RequestedModule     string                `json:"requested_module"`
    ProducerInputModule string                `json:"producer_input_module,omitempty"`
    ProducerModule      string                `json:"producer_module,omitempty"`
    ProducerLLMProfile  string                `json:"producer_llm_profile,omitempty"`
    ProducerReferences  []ReferenceProvenance `json:"producer_references,omitempty"`
    ProducerMetadata    map[string]any        `json:"producer_metadata,omitempty"`
    CreatedAt           *time.Time            `json:"created_at,omitempty"`
}

Action is one of reused, generated, refreshed, or bypassed. Retain the existing top-level input_module and chunker fields as the requested pipeline modules. Add this field to artifacts.RunManifest:

ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`

The runner initializes it with the effective mode and requested chunk module, then fills candidate and producer fields when a plan reaches materialization.

Stage 1: Finalize ADR and add the source-zone plan model

Status: Complete

Objective

Accept the architectural decision and introduce domain-neutral plan, annotation, validation, digest, clone, and materialization primitives without changing chunk-module execution yet.

Read first

  • internal/core/source/source.go
  • internal/core/source/digest.go
  • internal/core/source/validation.go
  • internal/core/source/source_test.go
  • every current source.Chunk clone or serialization helper found with rg

Implement

  1. Revise ADR-0005 before acceptance:
    • confirm it retains one source-addressed mutable plan and atomic refresh;
    • preserve the independent cache-surface decision;
    • keep filesystem and CLI details out of the ADR; and
    • change status from Proposed to Accepted.
  2. Add ChunkAnnotations, ChunkPlan, and ChunkRange in internal/core/source.
  3. Add canonical annotation validation and cloning. Canonicalize each raw JSON value by decoding with json.Decoder.UseNumber and re-encoding; reject blank namespaces, invalid JSON, trailing JSON values, and non-finite or unsupported values.
  4. Add plan validation against a SourceDocument using the rules in Planned Contracts.
  5. Add DigestChunkPlan and deterministic MaterializeChunkPlan.
  6. Add Annotations and PlanAnnotations to source.Chunk, DigestChunk, every clone helper, debug envelope, and serialized chunk-validation representation. Ensure all ownership boundaries clone raw bytes and retain the two scopes separately.
  7. Do not change the chunker interface or runner flow in this stage.

Tests

Add focused tests for:

  • annotation canonicalization, invalid JSON, blank namespaces, and mutation safety;
  • plan source mismatch, missing ranges, missing units, backward ranges, duplicate starts, gaps, and overlap;
  • stable plan digests and digest changes for boundary or annotation changes;
  • exact materialized IDs, references, content bytes, metadata, annotations, gaps, and overlap;
  • repeated byte-identical materialization; and
  • plan- and range-annotation preservation through existing chunk clones, debug conversion, serialized validation, and downstream extract requests.

Run at minimum:

go test ./internal/core/source
go test ./internal/framework/contracts ./internal/framework/pipeline

Exit criteria

The repository is green; ADR-0005 is accepted; plan primitives are complete and tested; current modules and the runner still behave as before.

Stage 2: Replace the chunk operation with plan generation

Status: Complete

Objective

Make all chunk modules produce plans and make the runner materialize those plans without adding durable reuse yet.

Read first

  • docs/internal/modules.md
  • docs/internal/pipeline.md
  • internal/framework/contracts/contracts.go
  • internal/framework/pipeline/chunker_registry.go
  • internal/framework/pipeline/runner.go
  • internal/framework/pipeline/chunk_validation.go
  • both production chunk-module packages and all chunk fakes in tests

Implement

  1. Replace ChunkResult with ChunkPlanResult and Chunker.Chunk with Chunker.Plan; keep the request, registry, module keys, and stage name.
  2. Update all fakes, registry tests, preparation tests, integration helpers, and compile-time interface assertions in the same stage. Do not retain adapters or a legacy chunk-returning interface.
  3. Convert generic/chunk/units to emit ranges only. Preserve max_units and overlap_units semantics.
  4. Convert dnd/chunk/scenes to emit namespace dnd/scenes at both scopes:
    • each range value is an object with exact keys short_title, primary_mode, main_participants, summary, boundary_note, and boundary_confidence, preserving the current normalized scene values; and
    • the plan value is an object with exact key boundary_caveats, containing the normalized caveat array.
  5. Preserve accepted scene boundary caveats as contracts.Warning values as well as in the plan annotation so warning behavior remains visible.
  6. Remove chunks.scenes from the D&D scene module's provided capabilities. Keep only chunks.
  7. Change the runner’s non-cached chunk path to:
    • call Plan under the existing retry and debug attempt boundary;
    • validate the plan structurally;
    • materialize chunks;
    • run the configured chunk-validator chain on materialized chunks;
    • preserve retry and rejection semantics; and
    • pass materialized chunks to lanes exactly as before.
  8. Update chunk debug payloads to distinguish the generated plan from materialized chunks. Plan annotations may appear only in opt-in debug data.
  9. Leave current chunk checkpoint loading and recording temporarily in place, storing materialized chunks, so existing resume tests remain green until Stage 4 removes that path.

Tests

Update focused module and framework tests to cover:

  • exact generic ranges and overlap;
  • exact D&D scene annotations and warning conversion;
  • annotations absent from generic plans;
  • framework materialization rather than module materialization;
  • chunk validators receiving materialized annotated chunks;
  • retries around generation plus validation;
  • stable chunk IDs independent of producing module; and
  • no chunks.scenes resolution dependency.

Run at minimum:

go test ./internal/modules/generic/chunk/units
go test ./internal/modules/dnd/chunk/scenes
go test ./internal/framework/contracts ./internal/framework/pipeline
go test ./internal/modules/integration ./internal/cli

Exit criteria

Every chunk module uses the single plan contract, the runner always materializes, production behavior remains functionally equivalent apart from framework-owned stable chunk IDs and namespaced annotations, and the repository is green.

Stage 3: Add dedicated cache configuration and the plan store

Status: Complete

Objective

Implement the durable source-addressed store and its independent cache-root configuration without changing the runner to consume the store.

Read first

  • docs/config.md
  • docs/operations.md
  • internal/core/config/*
  • internal/core/workspace/*
  • internal/framework/checkpoint/ as the local pattern for a workspace-backed implementation
  • internal/cli/run.go configuration loading, without wiring the store yet

Implement

  1. Add ChunkCacheMode and strict parsing in the pipeline package.

  2. Add WorkspaceChunkCacheConfig with Mode and Directory to config and file config. The YAML fields are workspace.chunk_cache.mode and workspace.chunk_cache.directory.

  3. Set default mode to auto; support NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE; preserve precedence file then environment. CLI precedence arrives in Stage 5.

  4. Support NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR with environment-over-file precedence. Trim the value; an unset or empty effective value selects the per-user default. Do not add a CLI directory override.

  5. Add workspace.DefaultChunkPlanRoot, accepting an injectable user-cache-directory resolver and returning filepath.Join(userCacheDir, "notarius", "chunk-plans"). Production later supplies os.UserCacheDir; tests supply a stub. Reject resolver errors and empty returned directories. Use this exact signature:

    func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error)

  6. Treat a non-empty workspace.chunk_cache.directory as the plan root itself, cleaned with the same path rules as other workspace roots. It does not gain an implicit chunk-plans suffix. workspace.directory must not affect this root, and the chunk-cache directory must not affect checkpoint, diagnostics, debug, or output roots. Do not change workspace.FromConfig or workspace.Settings for this feature.

  7. Add the plan record, producer, decision, and store interfaces described above in the pipeline package.

  8. Add internal/framework/chunkplan with:

    • a filesystem implementation;
    • a constructor that accepts the already resolved chunk-plan root;
    • path <chunk-plan-root>/<64-character-source-sha256-hex>/plan.json, after strictly validating and stripping the logical sha256: prefix;
    • schema notarius.chunk-plan.v1;
    • directories mode 0700 and file mode 0600;
    • atomic same-directory temp-file rename;
    • strict envelope decoding and unknown-field rejection;
    • complete digest and annotation validation; and
    • no raw reference content.

  9. Load distinguishes missing, invalid, and operational error. Save always atomically replaces and implements last-successful-write semantics.

  10. Keep store fakes local to their tests; do not add a production in-memory or no-op store because bypass and a nil collaborator already express disabled persistence.

  11. Do not wire the store into Runner or CLI execution in this stage.

Tests

Cover:

  • config defaults, YAML decoding, environment override, invalid values, cloning, redaction, and validation;
  • directory environment-over-file precedence, empty-value fallback, cloning, redaction, cleaning, and validation;
  • exact injected per-user default and resolver failure or empty-result errors;
  • independence in both directions between workspace.directory and workspace.chunk_cache.directory;
  • exact acceptance of /var/cache/notarius/chunk-plans as a configured root without writing to that real path;
  • safe full-digest paths and rejection of malformed digests;
  • 0700/0600 permissions where supported;
  • envelope round trip, strict decoding, source and plan digest mismatch, invalid annotations, schema mismatch, missing files, operational read errors, atomic replacement, and concurrent writers; and
  • absence of raw reference content in persisted JSON.

Run at minimum:

go test ./internal/core/config ./internal/core/workspace
go test ./internal/framework/chunkplan ./internal/framework/pipeline
go test ./internal/cli

Exit criteria

Configuration and storage are complete and independently tested, but normal runs still use Stage 2’s generate-and-materialize behavior.

Stage 4: Integrate cache policy and remove chunk checkpoints

Status: Complete

Objective

Make the runner’s chunk path use the canonical plan store under all three modes and establish it as the only chunk-reuse mechanism.

Read first

  • internal/framework/pipeline/runner.go
  • internal/framework/pipeline/checkpoint.go
  • internal/framework/pipeline/runner_concurrent.go
  • internal/framework/checkpoint/loader.go
  • internal/framework/checkpoint/recorder.go
  • focused runner retry, rejection, debug, and checkpoint tests

Implement

  1. Add ChunkCacheMode and ChunkPlans to RunInput. Empty mode means bypass for direct framework callers.
  2. Add synchronized store wrapping only if runner access can be concurrent; otherwise document and test the intentionally serial chunk-plan access.
  3. Implement mode behavior:
    • auto hit: load, structurally validate against the current source, materialize, then run configured chunk validators once;
    • auto miss or invalid: generate through existing retries, validate, materialize, validate chunks, then save;
    • bypass: generate and validate normally without calling Load or Save;
    • refresh: generate and validate normally, then replace through Save.
  4. A valid-hit validator rejection is recorded without retry, store mutation, or implicit regeneration. A store I/O error is a framework error.
  5. A failed or rejected generated plan is never saved. An invalid old file is overwritten only after a replacement passes module execution, structural validation, materialization, and configured validators.
  6. Build producer provenance from the prepared input module key, prepared chunk module key, resolved chunk-stage LLM profile, chunk-target auxiliary reference provenance, and chunk-module manifest metadata. Clone and redact according to existing boundaries.
  7. Replay stored producer warnings on a hit, then append current validation warnings. Do not duplicate warnings from discarded generation attempts.
  8. Remove all chunk methods and types from CheckpointRecorder, CheckpointLoader, synchronized wrappers, workspace recorder/loader, and checkpoint manifests and envelopes.
  9. Remove runner chunk-checkpoint decisions and events. Retain source, extract, merge, and normalize checkpoints. Keep downstream fingerprints based on the digest of materialized chunks.
  10. Ignore existing on-disk chunk/ checkpoint files; do not read, delete, or migrate them.
  11. Update debug behavior:
    • hit records lookup and materialization but no chunk module attempt or LLM call;
    • generated, bypassed, and refreshed paths retain attempt debug; and
    • default debug envelopes remain redacted.

Tests

Add table-driven runner tests for every mode and outcome:

  • hit, missing, invalid, and I/O error;
  • generation success, error, rejection, retry, and cancellation;
  • validator warning, rejection, and error on hits and generated plans;
  • bypass performs zero store calls;
  • refresh performs no load and exactly one successful save;
  • invalid record remains untouched when replacement fails;
  • stored warnings replay once;
  • no module or LLM call on hit;
  • downstream chunk digest changes on refresh;
  • legacy chunk checkpoint files are ignored; and
  • source and lane checkpoint behavior remains intact.

Run at minimum:

go test ./internal/framework/pipeline
go test ./internal/framework/checkpoint
go test ./internal/modules/integration

Exit criteria

The runner implements complete cache semantics, chunk checkpoints no longer exist as a code path, all direct framework tests are green, and production CLI runs are not yet wired to persistent plan storage.

Stage 5: Wire persistent policy into the CLI

Status: Complete

Objective

Make normal CLI runs resolve the effective mode and dedicated chunk-plan root, construct the durable plan store when appropriate, and pass it into the runner.

Read first

  • docs/cli.md
  • docs/config.md
  • cmd/notarius/main.go and internal/cli/run.go
  • CLI flag, environment, invocation-metadata, and exit-code tests
  • the workspace and configuration code changed in Stage 3

Implement

  1. Add the exact flag --chunk_cache <auto|bypass|refresh>. Represent an omitted CLI value separately from an explicit value so precedence is not inferred from the default printed by the flag package.
  2. Resolve the effective mode in this order: explicit CLI flag, environment, configuration file, default auto. An invalid CLI flag is a usage error with exit code 2; invalid file or environment configuration follows the existing configuration-load failure path and exit code 1.
  3. Include an explicitly requested CLI value in invocation metadata and the resolved mode in effective-configuration metadata without adding field-level origin tracking or plan and annotation payloads. Name the optional invocation JSON field chunk_cache_override; the resolved mode already appears at workspace.chunk_cache.mode in the redacted effective config.
  4. Add UserCacheDir func() (string, error) and ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory to cli.Options, defaulting them to os.UserCacheDir and chunkplan.NewFilesystemStore. For auto and refresh, choose the non-empty effective workspace.chunk_cache.directory; otherwise resolve the per-user default with workspace.DefaultChunkPlanRoot. Invoke the store factory with that exact root and pass the result and effective mode through RunInput. Never append chunk-plans to an explicit directory. The injectable factory lets CLI tests assert root selection without writing to production paths.
  5. For bypass, do not resolve, inspect, or create the chunk-plan root and pass no store. Existing general-workspace setup remains unchanged and independent: workspace.directory, resume, diagnostics, and debug continue to use their current paths and semantics.
  6. Keep resume policy independent of plan reuse. Enabling, disabling, or selecting a resume run must not change the chunk-cache mode or path.
  7. Configuration validation and pipeline-listing commands validate the mode but do not create workspace directories or plan files.
  8. Update CLI usage text only as needed to expose the new flag. Defer all narrative documentation to Stage 8.

Tests

Add CLI and integration tests for:

  • a default auto run writing a plan and a second independent invocation reusing it without a chunk operation or chunk LLM call;
  • file, environment, and CLI precedence, including persistent refresh in file and environment configuration;
  • bypass performing no plan-store access and resolving or creating no default chunk-plan root;
  • invalid flag, environment, and file values with their established error classifications;
  • an explicit chunk-plan root from file and environment, including exact use of /var/cache/notarius/chunk-plans without an appended suffix, using a recording store factory and never writing to that real path;
  • an explicit root succeeding without calling a failing per-user cache resolver;
  • an injected per-user default resolving exactly to <user-cache-dir>/notarius/chunk-plans;
  • workspace.directory changes having no effect on chunk-plan placement and a chunk-cache directory having no effect on other workspace state;
  • per-user cache-root discovery failure under auto, refresh, and bypass, including the allowed no-state bypass case, plus unwritable-store failures;
  • resume plus each cache mode, showing independent behavior;
  • validation and listing commands having no filesystem side effects; and
  • reuse across changed pipeline, chunk module/options, references, validators, and LLM settings when the source digest is unchanged.

Run at minimum:

go test ./internal/cli
go test ./internal/framework/pipeline ./internal/framework/chunkplan
go test ./internal/modules/integration

Exit criteria

The production CLI implements the complete mode and precedence contract, independent invocations reuse source-addressed plans by default, bypass and refresh have exact state semantics, and the repository is green.

Stage 6: Expose producer provenance and safe diagnostics

Status: Complete

Objective

Make every output and diagnostic record distinguish the requested chunk module from the producer of the effective plan without exposing sensitive payloads by default.

Read first

  • docs/integrations/json-output.md
  • docs/internal/diagnostics.md
  • manifest assembly in internal/framework/pipeline/runner.go
  • internal/core/artifacts/artifacts.go
  • current diagnostics and debug-envelope implementations and tests

Implement

  1. Add ChunkPlanManifest to the run manifest and all required clone, conversion, and serialization paths.
  2. Populate mode and exactly one action after a candidate plan has passed structural validation and materialization:
    • reused for an auto hit;
    • generated for an auto miss or invalid record;
    • refreshed for generation under refresh; and
    • bypassed for generation under bypass. Retain that action if the current chunk-validator chain rejects the candidate; publication still follows Stage 4's accept-only rule. If no candidate reaches materialization, leave action and candidate plan fields empty in the partial failure manifest and report the failure in diagnostics.
  3. Preserve the existing top-level chunker and its module metadata as the requested module. Never overwrite those fields with cached producer data.
  4. Populate producer module, LLM profile, references, metadata, creation time, source digest, plan digest, and schema from the effective stored or newly generated record. A reused record reports its original producer, even when it differs from the requested module.
  5. Define producer LLM profile as the resolved chunk-stage profile used during generation. Leave it empty for producers that did not use an LLM; do not substitute the current run's requested profile on reuse.
  6. Keep producer references as existing redacted provenance records. Deep-clone metadata, references, warnings, annotations, and raw JSON at every ownership boundary.
  7. Add a redacted chunk-plan.json diagnostic artifact, backed by a summary on RunOutput so the CLI can write it on success, rejection, or framework failure. It contains mode, source and candidate plan digests, requested module, lookup status and redacted reason, action, validation status, and publication status. Use these closed values:
    • lookup: hit, missing, invalid, or skipped;
    • validation: not_run, approved, approved_with_warnings, rejected, or error; and
    • publication: not_requested, published, or not_published. It contains no source units, materialized chunk content, annotation bytes, reference content, prompts, model responses, or raw invalid-file content.
  8. Keep complete plan, annotations, attempts, and materialized chunks confined to existing opt-in debug artifacts and apply their normal restrictive file handling.
  9. Preserve JSON-output compatibility by adding fields rather than renaming unrelated existing fields. Omit producer-only optional values when absent in the same style as the surrounding manifest schema.

Tests

Cover:

  • all four actions and their exact manifest fields;
  • requested and producer modules differing on a hit;
  • producer metadata remaining stable when current configuration changes;
  • LLM-backed and non-LLM producers;
  • deep-clone and mutation safety for raw annotations and metadata;
  • default diagnostic redaction, including invalid stored files;
  • complete data appearing only under explicit debug configuration; and
  • JSON serialization and compatibility fixtures.

Run at minimum:

go test ./internal/core/artifacts
go test ./internal/framework/pipeline
go test ./internal/cli

Exit criteria

Manifests and diagnostics explain which plan was requested, selected, and produced; default output remains redacted; debug behavior is explicit; and all compatibility tests pass.

Stage 7: Add cross-run, corruption, and concurrency hardening

Status: Complete

Objective

Exercise the completed design across package boundaries and adversarial state, then close any correctness gaps without changing the settled contract.

Read first

  • all tests added in Stages 1–6
  • end-to-end CLI and module integration test harnesses
  • atomic-write, cancellation, retry, and race-sensitive framework code
  • repository import-boundary tests

Implement

  1. Add black-box tests using separate CLI application instances and a shared temporary chunk-plan root. Prove reuse across different pipelines, configured chunk modules, options, references, lanes, validators, and LLM profiles.
  2. Prove that a changed canonical source digest selects a different path, while byte-identical canonical sources reuse the same plan.
  3. Exercise generic plans through D&D pipelines and D&D-annotated plans through generic pipelines. Optional annotations must survive, while no downstream component may require dnd/scenes merely because the requested chunker is the D&D scene module.
  4. Add corruption cases for truncated JSON, unknown fields, schema mismatch, source mismatch, plan-digest mismatch, bad boundaries, invalid annotations, and materialization failure. Verify auto regeneration and non-destructive failure behavior exactly.
  5. Add concurrent reader/writer and writer/writer tests. Readers observe either the old complete envelope or the new complete envelope; after successful writers, the final file is one complete valid record. Do not assert which writer wins.
  6. Exercise interruption before save, during temp-file creation, and around rename where the implementation permits deterministic fault injection. Prior valid state must remain readable unless a complete replacement was published.
  7. Verify cancellation, retry limits, warning order, error classification, and debug ordering on misses and hits. No cache hit may consume chunk-stage retry budget or invoke the module.
  8. Verify legacy chunk checkpoint files cannot influence reuse and that lane resume still keys from effective materialized chunk digests after refresh.
  9. Extend import-boundary tests so internal/core/source owns no framework or module dependencies and the generic store owns no concrete module imports.
  10. Ensure examples and all tests isolate workspace state in temporary directories and isolate chunk-plan state with a temporary configured root or explicitly select bypass; tests must never write to the real per-user default or /var/cache/notarius/chunk-plans.
  11. Run the race detector over the packages that access the plan store and runner concurrently, and fix any race within feature scope.

Tests

In addition to the new focused tests, run:

go test ./...
go test -race ./internal/framework/chunkplan ./internal/framework/pipeline
go vet ./...
go build ./cmd/notarius

Exit criteria

The feature is demonstrated across independent runs, corruption and concurrent publication cannot expose partial state, cross-domain annotations remain optional, cancellation and retry behavior is deterministic, and all repository and race checks pass.

Stage 8: Publish current-behavior documentation

Status: Not started

Objective

After implementation is complete, make the supported behavior discoverable without duplicating canonical ownership across documents.

Read first

  • docs/policy/architecture.md
  • docs/policy/documentation.md
  • every current-behavior document linked below
  • the final code, flags, schemas, examples, and tests from Stages 1–7

Implement

  1. Update docs/policy/architecture.md with the plan-generation/materialization boundary, source-addressed reuse, and the independent cache-state surface at the level appropriate for the canonical system overview.

  2. Update docs/internal/pipeline.md with mode behavior, hit/miss control flow, validator placement, retry behavior, and removal of chunk checkpoints.

  3. Update docs/internal/modules.md with the Chunker.Plan contract, generic boundaries, optional annotation namespaces, and framework materialization.

  4. Update docs/internal/overview.md with the new plan-store package and current state ownership, and remove chunk checkpoint ownership from its checkpoint description. Keep detailed checkpoint and plan execution flow in docs/internal/pipeline.md rather than creating a parallel internal state reference.

  5. Update docs/config.md with workspace.chunk_cache.mode, workspace.chunk_cache.directory, both environment variables, strict values, precedence, and the per-user cache-root default. State explicitly that workspace.directory does not affect plan placement and that the configured directory is the root itself. Keep field-level details owned here. Document that Unix os.UserCacheDir rejects a relative XDG_CACHE_HOME and that this is a configuration error rather than a fallback.

  6. Update docs/cli.md with --chunk_cache semantics and examples. Link to configuration rather than restating its entire precedence model.

  7. Update docs/operations.md with plan location, permissions, invalid-state recovery, refresh and bypass operations, concurrency semantics, sensitive debug data, deletion as a recoverable but potentially expensive regeneration event, and the no-history/no-rollback limitation. Document both:

    • the ordinary Linux per-user default, $XDG_CACHE_HOME/notarius/chunk-plans when XDG_CACHE_HOME is a valid absolute path, or $HOME/.cache/notarius/chunk-plans when it is unset; and
    • /var/cache/notarius/chunk-plans as the recommended configured root for a system-wide Linux deployment running under a dedicated service account. Explain that the system path is not the unprivileged default, show the following minimal configuration, require the operator or package installer to provision it with restrictive service-account ownership and permissions, and warn against sharing the cache across mutually untrusted users:
    workspace:
  chunk_cache:
    directory: /var/cache/notarius/chunk-plans

  8. Update docs/integrations/json-output.md and docs/internal/diagnostics.md with requested versus producer provenance, actions, warnings, and redaction boundaries.

  9. Keep package-level plan/range/annotation and materialization invariants in docs/internal/pipeline.md; no separate source document is needed unless the implemented subsystem becomes too large for that canonical owner.

  10. Update examples only when they materially clarify operation; examples must use temporary or explicitly configured workspace and chunk-cache paths.

  11. Remove future-tense disclaimers from the feature roadmap, mark its completion outcomes as achieved, and mark all completed stages in this document. Keep the roadmap as historical target-state context rather than copying implementation details into it.

  12. Validate every changed link, heading, code symbol, flag, environment name, path, and JSON field against the implementation. Search for stale Chunker.Chunk, ChunkResult, chunks.scenes, and chunk-checkpoint claims.

Tests and validation

Run:

go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check

Also run repository link or documentation checks if present, and manually inspect the reading map in docs/development.md and component links in docs/internal/overview.md for canonical ownership and navigation consistency.

Exit criteria

All canonical documents describe implemented behavior, operational guidance is complete and non-duplicative, no stale chunk API or checkpoint documentation remains, roadmap status is accurate, and the repository is green.

Reference in New Issue View Git Blame Copy Permalink