notarius/docs/roadmap/implementation.md

Workspace Implementation Plan

This plan implements the workspace target described in Workspace Roadmap. Follow the stages in order. Keep code changes focused on the current stage, and update tests and canonical docs in the same stage when behavior changes.

The intended end state is:

• one configurable workspace root for Notarius-owned local state;
• diagnostics written under workspace/diagnostics/<run-id>/ when workspace diagnostics are enabled;
• resumability checkpoints written under deterministic workspace/checkpoints/ paths only when resume checkpointing is enabled;
• debug artifacts written under workspace/debug/<run-id>/ only when debug is enabled;
• stage-owned checkpoint manifests only, with no root-level checkpoint summary;
• explicit resume behavior through run --resume, not implicit skipping during ordinary run.

Stage 1: Add Workspace Configuration

Add workspace configuration while preserving legacy diagnostics configuration long enough for backward compatibility.

Implement in internal/core/config:

• Add WorkspaceConfig to Config.

• Add nested structs:

  • WorkspaceDiagnosticsConfig
  • WorkspaceResumeConfig
  • WorkspaceDebugConfig

• Add file config parsing for:

  workspace:
    directory: /var/lib/notarius
    diagnostics:
      enabled: true
      retention: auto
    resume:
      enabled: false
    debug:
      enabled: false
  

• Preserve existing diagnostics.work_dir and diagnostics.retention parsing as deprecated compatibility input.

• Resolve effective diagnostics behavior as follows:

  • if workspace.directory is set, diagnostics root is <workspace.directory>/diagnostics;
  • if workspace.directory is unset and legacy diagnostics.work_dir is set, use legacy diagnostics behavior unchanged;
  • if neither is set, preserve the current default diagnostics behavior for migration compatibility;
  • workspace.diagnostics.retention overrides legacy diagnostics retention when set;
  • legacy diagnostics retention remains accepted when workspace diagnostics retention is unset.

• If workspace.diagnostics.enabled is explicitly false, do not create a diagnostics run directory and do not write diagnostics artifacts. CLI failure handling must still print concise errors to stderr without assuming a diagnostics directory exists.

• Add environment support:

  • NOTARIUS_WORKSPACE_DIR sets workspace.directory;
  • NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED parses a boolean;
  • NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION sets workspace diagnostics retention;
  • NOTARIUS_WORKSPACE_RESUME_ENABLED parses a boolean;
  • NOTARIUS_WORKSPACE_DEBUG_ENABLED parses a boolean;
  • keep NOTARIUS_WORK_DIR and NOTARIUS_DIAGNOSTICS_RETENTION as deprecated compatibility overrides for legacy diagnostics config.

• Keep --diagnostics-dir as a backwards-compatible per-invocation diagnostics root override. It should affect diagnostics only and should not change checkpoint or debug roots.

• Update redaction/effective-config diagnostics so workspace config is included and no secrets are introduced.

Tests:

• Add config default tests for workspace defaults.
• Add file config parsing tests for nested workspace fields.
• Add env override tests for all new env vars.
• Add precedence tests covering workspace config, legacy diagnostics config, env overrides, and --diagnostics-dir.
• Update redacted/effective config tests.

Stage 2: Introduce Workspace Filesystem Helpers

Create reusable filesystem helpers for workspace state. Keep concrete module packages out of this layer.

Implement a new package, recommended path internal/core/workspace, with:

• Config or Settings describing effective workspace roots:
  • root directory;
  • diagnostics root;
  • checkpoints root;
  • debug root;
  • enabled flags.
• path construction helpers for:
  • diagnostics run directories;
  • checkpoint identity directories;
  • debug run directories.
• safe path helpers that reject absolute artifact names, .., backslashes, and paths that escape their intended root.
• atomic JSON and byte-file writes using the existing diagnostics atomic-write behavior as the model.
• optional shared internal helper for atomic file writes so diagnostics and workspace writers do not duplicate low-level write logic.

Do not add resume behavior in this stage.

Tests:

• Unit-test path construction and path-safety failures.
• Unit-test atomic JSON/byte writes.
• Unit-test disabled workspace settings returning no-op or empty roots as appropriate.
• Confirm no helper permits writes outside the configured root.

Stage 3: Move Diagnostics Under Workspace

Update diagnostics creation to use the effective diagnostics root from workspace configuration when workspace is configured.

Implementation requirements:

• Keep diagnostics artifacts and retention behavior unchanged.

• Honor disabled diagnostics by using a no-op diagnostics writer or nil-safe diagnostics path through the CLI failure and success paths.

• Preserve the existing diagnostics.RunDirectory contract unless a narrow constructor addition is cleaner.

• Route normal workspace diagnostics to:

  <workspace.directory>/diagnostics/<run-id>/
  

• Preserve legacy behavior when only legacy diagnostics config is present.

• Preserve --diagnostics-dir behavior as diagnostics-only override.

• Keep diagnostics run IDs run-based and non-deterministic.

• Do not write checkpoint or debug output in this stage.

Docs to update after behavior exists:

• docs/config.md
• docs/operations.md
• docs/internal/diagnostics.md
• docs/cli.md if CLI help text changes.

Tests:

• Update diagnostics run directory tests for workspace diagnostics roots.
• Update CLI tests that inspect diagnostics paths.
• Add migration tests proving legacy diagnostics config still works.
• Run focused checks:
  • go test ./internal/core/config
  • go test ./internal/core/diagnostics
  • go test ./internal/cli

Stage 4: Define Checkpoint Identity And Stage Manifest Types

Add checkpoint identity and manifest data structures before writing stage payloads.

Implement in internal/core/workspace or a closely related core package:

• CheckpointIdentity, derived from:

  • resolved pipeline ID;
  • resolved pipeline digest;
  • input adapter key;
  • raw input digest or source digest;
  • selected lanes;
  • runtime overrides that affect execution;
  • materialized reference digests;
  • prompt/schema/profile provenance not already represented by the pipeline digest.

• A deterministic, filesystem-safe checkpoint path:

  <workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-digest-prefix>/<pipeline-digest-prefix>/
  

• Stage manifest structs for:

  • source;
  • chunk;
  • extract lane;
  • merge lane;
  • normalize lane.

• Shared manifest fields:

  • workspace schema version;
  • stage name;
  • lane ID when applicable;
  • module key;
  • dependency fingerprints;
  • status;
  • output digests;
  • validation status and rejection summaries where applicable;
  • started/completed timestamps where useful.

• Status values:

  • pending;
  • running;
  • succeeded;
  • succeeded_with_rejections;
  • failed;
  • invalidated.

Do not add a root-level checkpoint manifest.

Tests:

• Unit-test deterministic identity generation.
• Unit-test identity changes when pipeline digest, input digest, selected lanes, or reference digests change.
• Unit-test manifest JSON round trips.
• Unit-test filesystem-safe path generation.

Stage 5: Write Checkpoints Without Resuming

Add write-only checkpoint support behind workspace.resume.enabled.

Implementation requirements:

• Add a framework-owned checkpoint recorder to pipeline execution. Recommended shape:

  • CLI constructs the effective workspace/checkpoint recorder after pipeline resolution and reference materialization.
  • pipeline.RunInput receives a recorder interface or no-op recorder.
  • concrete modules do not receive workspace paths and do not write directly to workspace.

• Write checkpoint artifacts only when workspace.resume.enabled is true.

• Use stage-owned manifests and no root summary.

• Use JSON envelope types for checkpointed payloads that preserve byte content, such as content_base64, media type, metadata, warnings, and content digest. Do not rely on existing runtime structs whose byte fields are tagged json:"-".

• Recommended checkpoint files:

  checkpoints/<identity>/source/manifest.json
  checkpoints/<identity>/source/source-document.json
  checkpoints/<identity>/chunk/manifest.json
  checkpoints/<identity>/chunk/chunks.json
  checkpoints/<identity>/extract/<lane-id>/manifest.json
  checkpoints/<identity>/extract/<lane-id>/outputs.json
  checkpoints/<identity>/merge/<lane-id>/manifest.json
  checkpoints/<identity>/merge/<lane-id>/output.json
  checkpoints/<identity>/normalize/<lane-id>/manifest.json
  checkpoints/<identity>/normalize/<lane-id>/output.json
  

• Record rejected extract outputs as checkpointed stage outcomes. Under current pipeline policy they do not pass downstream, but they are not framework errors.

• Mark a stage running before writing its payloads, then atomically replace the manifest with a final success/failure status after payload writes complete.

• Ensure interrupted or partial writes cannot be mistaken for successful checkpoints.

• Do not write raw prompts, raw references, raw LLM request payloads, or debug traces in checkpoint output.

Tests:

• Add runner/CLI tests proving no checkpoint files are written when resume is disabled.
• Add tests proving checkpoint files are written when resume is enabled.
• Add tests for successful runs, rejected extract outputs, failed stages, and warning-only validation.
• Add tests proving checkpoint payload content digests match written content.
• Add tests proving modules do not receive filesystem paths.

Stage 6: Add Explicit Resume Reads

Add explicit resume behavior after write-only checkpoints are stable.

CLI behavior:

• Add run --resume.
• --resume should require workspace.resume.enabled: true; otherwise return a clear configuration error.
• Plain run should continue to execute stages normally and should not silently skip completed stages.

Resume behavior:

• Load and validate checkpoint stage manifests in workflow order.
• Reuse a checkpoint only when:
  • workspace schema version is supported;
  • stage status is successful for the current purpose;
  • dependency fingerprints match the current invocation;
  • referenced checkpoint payload files exist;
  • payload digests match manifest digests;
  • selected lanes and runtime overrides are compatible.
• If a checkpoint is missing or invalid, execute that stage normally and write a fresh checkpoint if resume checkpointing remains enabled.
• If a source or chunk checkpoint is reused, downstream dependency fingerprints must still be validated before downstream reuse.
• If an extract checkpoint includes rejected outputs, preserve those rejection records and continue to omit rejected outputs from merge input.
• If merge is reused, skip merge execution only for the matching lane.
• If normalize is reused, pass the reused normalized output to the output stage.
• Always create fresh diagnostics for the current invocation, including records showing which stages were reused versus executed.

Tests:

• Add CLI tests for --resume without workspace resume enabled.
• Add tests for reusing source, chunk, extract, merge, and normalize checkpoints.
• Add tests for invalidation when input, pipeline digest, references, selected lanes, or runtime LLM profile override changes.
• Add tests for corrupt or missing checkpoint payloads.
• Add tests proving fresh diagnostics are written for resumed invocations.

Stage 7: Add Debug Output

Add debug output behind workspace.debug.enabled.

Implementation requirements:

• Write debug artifacts under:

  <workspace.directory>/debug/<run-id>/
  

• Debug is per-invocation and should not be used for resume.

• Debug and resume are independent:

  • debug enabled does not imply checkpoint writing;
  • resume enabled does not imply debug writing.

• Start with artifacts available from Notarius framework boundaries:

  • source/checkpoint-like stage inputs and outputs;
  • structured LLM request inputs and response content from Notarius contracts;
  • validator requests and results;
  • stage timing and attempt metadata.

• Do not depend on Scriptorium internals for rendered upstream provider requests. If Scriptorium later exposes rendered prompt traces safely, add them in a separate pass.

• Never write API keys, bearer tokens, or raw provider credentials.

• Document clearly that debug output may contain source material, references, prompts, model outputs, and other sensitive content.

Tests:

• Add tests proving no debug files are written when debug is disabled.
• Add tests proving debug files are written under debug/<run-id>/ when enabled.
• Add tests proving debug and resume can be enabled independently.
• Add tests proving obvious secrets are not written.
• Add diagnostics/debug path tests to ensure path roots do not overlap accidentally.

Stage 8: Documentation, Examples, And Cleanup

After behavior is implemented, update canonical current-behavior docs.

Update:

• docs/config.md
• docs/cli.md
• docs/operations.md
• docs/troubleshooting.md
• docs/internal/diagnostics.md
• docs/internal/pipeline.md
• maintained examples under examples/ when practical.

Documentation requirements:

• Document workspace config and defaults.
• Document legacy diagnostics config compatibility and any deprecation language.
• Document /var/lib/notarius as the production recommendation.
• Document local development recommendations.
• Document checkpoint sensitivity and debug sensitivity.
• Document explicit resume behavior and invalidation rules.
• Keep future or deferred behavior only in roadmap files.

Cleanup:

• Update docs/roadmap/workspace.md to a completed-status note after the implementation lands.
• Remove or replace this implementation plan with a completed note after the implementation lands.

Validation:

go test ./internal/core/config
go test ./internal/core/diagnostics
go test ./internal/core/workspace
go test ./internal/framework/pipeline
go test ./internal/cli
go test ./...
go vet ./...
go build ./cmd/notarius