Add a planning roadmap and a staged implementation plan for the workspace configuration

2026-07-07 21:08:52 -05:00
parent c5f2b14ff4
commit 304c68f9fc
2 changed files with 627 additions and 0 deletions
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,394 @@
+# Workspace Implementation Plan
+
+This plan implements the workspace target described in
+[Workspace Roadmap](workspace.md). Follow the stages in order. Keep code changes
+focused on the current stage, and update tests and canonical docs in the same
+stage when behavior changes.
+
+The intended end state is:
+
+- one configurable workspace root for Notarius-owned local state;
+- diagnostics written under `workspace/diagnostics/<run-id>/` when workspace
+  diagnostics are enabled;
+- resumability checkpoints written under deterministic `workspace/checkpoints/`
+  paths only when resume checkpointing is enabled;
+- debug artifacts written under `workspace/debug/<run-id>/` only when debug is
+  enabled;
+- stage-owned checkpoint manifests only, with no root-level checkpoint summary;
+- explicit resume behavior through `run --resume`, not implicit skipping during
+  ordinary `run`.
+
+## Stage 1: Add Workspace Configuration
+
+Add workspace configuration while preserving legacy diagnostics configuration
+long enough for backward compatibility.
+
+Implement in `internal/core/config`:
+
+- Add `WorkspaceConfig` to `Config`.
+- Add nested structs:
+  - `WorkspaceDiagnosticsConfig`
+  - `WorkspaceResumeConfig`
+  - `WorkspaceDebugConfig`
+- Add file config parsing for:
+
+  ```yaml
+  workspace:
+    directory: /var/lib/notarius
+    diagnostics:
+      enabled: true
+      retention: auto
+    resume:
+      enabled: false
+    debug:
+      enabled: false
+  ```
+
+- Preserve existing `diagnostics.work_dir` and `diagnostics.retention` parsing as
+  deprecated compatibility input.
+- Resolve effective diagnostics behavior as follows:
+  - if `workspace.directory` is set, diagnostics root is
+    `<workspace.directory>/diagnostics`;
+  - if `workspace.directory` is unset and legacy `diagnostics.work_dir` is set,
+    use legacy diagnostics behavior unchanged;
+  - if neither is set, preserve the current default diagnostics behavior for
+    migration compatibility;
+  - `workspace.diagnostics.retention` overrides legacy diagnostics retention
+    when set;
+  - legacy diagnostics retention remains accepted when workspace diagnostics
+    retention is unset.
+- If `workspace.diagnostics.enabled` is explicitly false, do not create a
+  diagnostics run directory and do not write diagnostics artifacts. CLI failure
+  handling must still print concise errors to stderr without assuming a
+  diagnostics directory exists.
+- Add environment support:
+  - `NOTARIUS_WORKSPACE_DIR` sets `workspace.directory`;
+  - `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED` parses a boolean;
+  - `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION` sets workspace diagnostics
+    retention;
+  - `NOTARIUS_WORKSPACE_RESUME_ENABLED` parses a boolean;
+  - `NOTARIUS_WORKSPACE_DEBUG_ENABLED` parses a boolean;
+  - keep `NOTARIUS_WORK_DIR` and `NOTARIUS_DIAGNOSTICS_RETENTION` as deprecated
+    compatibility overrides for legacy diagnostics config.
+- Keep `--diagnostics-dir` as a backwards-compatible per-invocation diagnostics
+  root override. It should affect diagnostics only and should not change
+  checkpoint or debug roots.
+- Update redaction/effective-config diagnostics so workspace config is included
+  and no secrets are introduced.
+
+Tests:
+
+- Add config default tests for workspace defaults.
+- Add file config parsing tests for nested workspace fields.
+- Add env override tests for all new env vars.
+- Add precedence tests covering workspace config, legacy diagnostics config,
+  env overrides, and `--diagnostics-dir`.
+- Update redacted/effective config tests.
+
+## Stage 2: Introduce Workspace Filesystem Helpers
+
+Create reusable filesystem helpers for workspace state. Keep concrete module
+packages out of this layer.
+
+Implement a new package, recommended path `internal/core/workspace`, with:
+
+- `Config` or `Settings` describing effective workspace roots:
+  - root directory;
+  - diagnostics root;
+  - checkpoints root;
+  - debug root;
+  - enabled flags.
+- path construction helpers for:
+  - diagnostics run directories;
+  - checkpoint identity directories;
+  - debug run directories.
+- safe path helpers that reject absolute artifact names, `..`, backslashes, and
+  paths that escape their intended root.
+- atomic JSON and byte-file writes using the existing diagnostics atomic-write
+  behavior as the model.
+- optional shared internal helper for atomic file writes so diagnostics and
+  workspace writers do not duplicate low-level write logic.
+
+Do not add resume behavior in this stage.
+
+Tests:
+
+- Unit-test path construction and path-safety failures.
+- Unit-test atomic JSON/byte writes.
+- Unit-test disabled workspace settings returning no-op or empty roots as
+  appropriate.
+- Confirm no helper permits writes outside the configured root.
+
+## Stage 3: Move Diagnostics Under Workspace
+
+Update diagnostics creation to use the effective diagnostics root from workspace
+configuration when workspace is configured.
+
+Implementation requirements:
+
+- Keep diagnostics artifacts and retention behavior unchanged.
+- Honor disabled diagnostics by using a no-op diagnostics writer or nil-safe
+  diagnostics path through the CLI failure and success paths.
+- Preserve the existing `diagnostics.RunDirectory` contract unless a narrow
+  constructor addition is cleaner.
+- Route normal workspace diagnostics to:
+
+  ```text
+  <workspace.directory>/diagnostics/<run-id>/
+  ```
+
+- Preserve legacy behavior when only legacy diagnostics config is present.
+- Preserve `--diagnostics-dir` behavior as diagnostics-only override.
+- Keep diagnostics run IDs run-based and non-deterministic.
+- Do not write checkpoint or debug output in this stage.
+
+Docs to update after behavior exists:
+
+- `docs/config.md`
+- `docs/operations.md`
+- `docs/internal/diagnostics.md`
+- `docs/cli.md` if CLI help text changes.
+
+Tests:
+
+- Update diagnostics run directory tests for workspace diagnostics roots.
+- Update CLI tests that inspect diagnostics paths.
+- Add migration tests proving legacy diagnostics config still works.
+- Run focused checks:
+  - `go test ./internal/core/config`
+  - `go test ./internal/core/diagnostics`
+  - `go test ./internal/cli`
+
+## Stage 4: Define Checkpoint Identity And Stage Manifest Types
+
+Add checkpoint identity and manifest data structures before writing stage
+payloads.
+
+Implement in `internal/core/workspace` or a closely related core package:
+
+- `CheckpointIdentity`, derived from:
+  - resolved pipeline ID;
+  - resolved pipeline digest;
+  - input adapter key;
+  - raw input digest or source digest;
+  - selected lanes;
+  - runtime overrides that affect execution;
+  - materialized reference digests;
+  - prompt/schema/profile provenance not already represented by the pipeline
+    digest.
+- A deterministic, filesystem-safe checkpoint path:
+
+  ```text
+  <workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-digest-prefix>/<pipeline-digest-prefix>/
+  ```
+
+- Stage manifest structs for:
+  - source;
+  - chunk;
+  - extract lane;
+  - merge lane;
+  - normalize lane.
+- Shared manifest fields:
+  - workspace schema version;
+  - stage name;
+  - lane ID when applicable;
+  - module key;
+  - dependency fingerprints;
+  - status;
+  - output digests;
+  - validation status and rejection summaries where applicable;
+  - started/completed timestamps where useful.
+- Status values:
+  - `pending`;
+  - `running`;
+  - `succeeded`;
+  - `succeeded_with_rejections`;
+  - `failed`;
+  - `invalidated`.
+
+Do not add a root-level checkpoint manifest.
+
+Tests:
+
+- Unit-test deterministic identity generation.
+- Unit-test identity changes when pipeline digest, input digest, selected lanes,
+  or reference digests change.
+- Unit-test manifest JSON round trips.
+- Unit-test filesystem-safe path generation.
+
+## Stage 5: Write Checkpoints Without Resuming
+
+Add write-only checkpoint support behind `workspace.resume.enabled`.
+
+Implementation requirements:
+
+- Add a framework-owned checkpoint recorder to pipeline execution. Recommended
+  shape:
+  - CLI constructs the effective workspace/checkpoint recorder after pipeline
+    resolution and reference materialization.
+  - `pipeline.RunInput` receives a recorder interface or no-op recorder.
+  - concrete modules do not receive workspace paths and do not write directly to
+    workspace.
+- Write checkpoint artifacts only when `workspace.resume.enabled` is true.
+- Use stage-owned manifests and no root summary.
+- Use JSON envelope types for checkpointed payloads that preserve byte content,
+  such as `content_base64`, media type, metadata, warnings, and content digest.
+  Do not rely on existing runtime structs whose byte fields are tagged
+  `json:"-"`.
+- Recommended checkpoint files:
+
+  ```text
+  checkpoints/<identity>/source/manifest.json
+  checkpoints/<identity>/source/source-document.json
+  checkpoints/<identity>/chunk/manifest.json
+  checkpoints/<identity>/chunk/chunks.json
+  checkpoints/<identity>/extract/<lane-id>/manifest.json
+  checkpoints/<identity>/extract/<lane-id>/outputs.json
+  checkpoints/<identity>/merge/<lane-id>/manifest.json
+  checkpoints/<identity>/merge/<lane-id>/output.json
+  checkpoints/<identity>/normalize/<lane-id>/manifest.json
+  checkpoints/<identity>/normalize/<lane-id>/output.json
+  ```
+
+- Record rejected extract outputs as checkpointed stage outcomes. Under current
+  pipeline policy they do not pass downstream, but they are not framework errors.
+- Mark a stage `running` before writing its payloads, then atomically replace the
+  manifest with a final success/failure status after payload writes complete.
+- Ensure interrupted or partial writes cannot be mistaken for successful
+  checkpoints.
+- Do not write raw prompts, raw references, raw LLM request payloads, or debug
+  traces in checkpoint output.
+
+Tests:
+
+- Add runner/CLI tests proving no checkpoint files are written when resume is
+  disabled.
+- Add tests proving checkpoint files are written when resume is enabled.
+- Add tests for successful runs, rejected extract outputs, failed stages, and
+  warning-only validation.
+- Add tests proving checkpoint payload content digests match written content.
+- Add tests proving modules do not receive filesystem paths.
+
+## Stage 6: Add Explicit Resume Reads
+
+Add explicit resume behavior after write-only checkpoints are stable.
+
+CLI behavior:
+
+- Add `run --resume`.
+- `--resume` should require `workspace.resume.enabled: true`; otherwise return a
+  clear configuration error.
+- Plain `run` should continue to execute stages normally and should not silently
+  skip completed stages.
+
+Resume behavior:
+
+- Load and validate checkpoint stage manifests in workflow order.
+- Reuse a checkpoint only when:
+  - workspace schema version is supported;
+  - stage status is successful for the current purpose;
+  - dependency fingerprints match the current invocation;
+  - referenced checkpoint payload files exist;
+  - payload digests match manifest digests;
+  - selected lanes and runtime overrides are compatible.
+- If a checkpoint is missing or invalid, execute that stage normally and write a
+  fresh checkpoint if resume checkpointing remains enabled.
+- If a source or chunk checkpoint is reused, downstream dependency fingerprints
+  must still be validated before downstream reuse.
+- If an extract checkpoint includes rejected outputs, preserve those rejection
+  records and continue to omit rejected outputs from merge input.
+- If merge is reused, skip merge execution only for the matching lane.
+- If normalize is reused, pass the reused normalized output to the output stage.
+- Always create fresh diagnostics for the current invocation, including records
+  showing which stages were reused versus executed.
+
+Tests:
+
+- Add CLI tests for `--resume` without workspace resume enabled.
+- Add tests for reusing source, chunk, extract, merge, and normalize
+  checkpoints.
+- Add tests for invalidation when input, pipeline digest, references, selected
+  lanes, or runtime LLM profile override changes.
+- Add tests for corrupt or missing checkpoint payloads.
+- Add tests proving fresh diagnostics are written for resumed invocations.
+
+## Stage 7: Add Debug Output
+
+Add debug output behind `workspace.debug.enabled`.
+
+Implementation requirements:
+
+- Write debug artifacts under:
+
+  ```text
+  <workspace.directory>/debug/<run-id>/
+  ```
+
+- Debug is per-invocation and should not be used for resume.
+- Debug and resume are independent:
+  - debug enabled does not imply checkpoint writing;
+  - resume enabled does not imply debug writing.
+- Start with artifacts available from Notarius framework boundaries:
+  - source/checkpoint-like stage inputs and outputs;
+  - structured LLM request inputs and response content from Notarius contracts;
+  - validator requests and results;
+  - stage timing and attempt metadata.
+- Do not depend on Scriptorium internals for rendered upstream provider
+  requests. If Scriptorium later exposes rendered prompt traces safely, add them
+  in a separate pass.
+- Never write API keys, bearer tokens, or raw provider credentials.
+- Document clearly that debug output may contain source material, references,
+  prompts, model outputs, and other sensitive content.
+
+Tests:
+
+- Add tests proving no debug files are written when debug is disabled.
+- Add tests proving debug files are written under `debug/<run-id>/` when enabled.
+- Add tests proving debug and resume can be enabled independently.
+- Add tests proving obvious secrets are not written.
+- Add diagnostics/debug path tests to ensure path roots do not overlap
+  accidentally.
+
+## Stage 8: Documentation, Examples, And Cleanup
+
+After behavior is implemented, update canonical current-behavior docs.
+
+Update:
+
+- `docs/config.md`
+- `docs/cli.md`
+- `docs/operations.md`
+- `docs/troubleshooting.md`
+- `docs/internal/diagnostics.md`
+- `docs/internal/pipeline.md`
+- maintained examples under `examples/` when practical.
+
+Documentation requirements:
+
+- Document workspace config and defaults.
+- Document legacy diagnostics config compatibility and any deprecation language.
+- Document `/var/lib/notarius` as the production recommendation.
+- Document local development recommendations.
+- Document checkpoint sensitivity and debug sensitivity.
+- Document explicit resume behavior and invalidation rules.
+- Keep future or deferred behavior only in roadmap files.
+
+Cleanup:
+
+- Update `docs/roadmap/workspace.md` to a completed-status note after the
+  implementation lands.
+- Remove or replace this implementation plan with a completed note after the
+  implementation lands.
+
+Validation:
+
+```sh
+go test ./internal/core/config
+go test ./internal/core/diagnostics
+go test ./internal/core/workspace
+go test ./internal/framework/pipeline
+go test ./internal/cli
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
--- a/docs/roadmap/workspace.md
+++ b/docs/roadmap/workspace.md
@@ -0,0 +1,233 @@
+# Workspace Roadmap
+
+Notarius should gain a single configurable workspace root for application-owned
+local state. The workspace exists only to support enabled workspace features;
+ordinary runs should not write workspace files by default.
+
+The workspace is distinct from durable output:
+
+- durable output is the user-requested final product of a run;
+- the workspace is application-owned local state for diagnostics, resumable
+  checkpoints, and development/debug workflows.
+
+Within the workspace, diagnostics, checkpoints, and debug artifacts should use
+separate subdirectories because they have different identity and lifecycle
+models.
+
+## Target Configuration
+
+Workspace behavior should be configured under a dedicated top-level object:
+
+```yaml
+workspace:
+  directory: /var/lib/notarius
+  diagnostics:
+    enabled: true
+    retention: auto
+  resume:
+    enabled: false
+  debug:
+    enabled: false
+```
+
+Policy:
+
+- `workspace.diagnostics.enabled` defaults to the current diagnostics behavior
+  during migration.
+- `workspace.resume.enabled` defaults to `false`.
+- `workspace.debug.enabled` defaults to `false`.
+- no checkpoint or debug files are written when those features are disabled.
+- `/var/lib/notarius` should be the standard production recommendation in docs
+  and examples.
+- local development docs may recommend a project-local path such as
+  `./.notarius/workspace`.
+
+`workspace.directory` should become the single configured root for Notarius local
+state. Existing `diagnostics.work_dir` behavior should be migrated carefully for
+backward compatibility, but the long-term configuration model should avoid two
+separate roots for application-owned local state.
+
+## Workspace Layout
+
+The workspace root should contain separate subtrees:
+
+```text
+<workspace.directory>/
+  diagnostics/
+  checkpoints/
+  debug/
+```
+
+Diagnostics remain run-id based. They are invocation history and should preserve
+the existing diagnostics retention model:
+
+```text
+<workspace.directory>/
+  diagnostics/
+    <run-id>/
+      invocation.json
+      effective-config.json
+      resolved-pipeline.json
+      resolved-references.json
+      run-manifest.json
+      warnings.json
+      run-report.json
+      error.log
+```
+
+Checkpoints should be deterministic and should include enough identity material
+to prevent unsafe reuse across incompatible runs:
+
+```text
+<workspace.directory>/
+  checkpoints/
+    <pipeline-id>/
+      <input-key>-<source-digest-prefix>/
+        <pipeline-digest-prefix>/
+          source/
+          chunk/
+          extract/<lane-id>/
+          merge/<lane-id>/
+          normalize/<lane-id>/
+```
+
+Checkpoint identity should account for:
+
+- resolved pipeline ID;
+- resolved pipeline digest;
+- input adapter key;
+- source/input digest;
+- selected lanes and runtime overrides that affect execution;
+- reference digests;
+- prompt, schema, and profile provenance when not already captured by the
+  resolved pipeline digest.
+
+Debug output should be run-id based like diagnostics, because debug traces are
+development artifacts from a specific invocation rather than resumable state:
+
+```text
+<workspace.directory>/
+  debug/
+    <run-id>/
+```
+
+Human-readable path segments are useful, but content hashes should be
+authoritative for correctness. The exact path shape may change during
+implementation, but checkpoint identity must be stable, deterministic, and safe
+to compare across runs. Diagnostics and debug output do not need deterministic
+resume identity because they are per-invocation artifacts.
+
+## Stage Ownership
+
+Stage manifests should be stage-owned. There should not be a root-level manifest
+that summarizes all stages.
+
+Rationale:
+
+- stage manifests are the authoritative state for the stage that wrote them;
+- avoiding a root summary prevents duplicate state from drifting;
+- stage-local manifests make partial writes, failure recovery, and future stage
+  invalidation easier to reason about.
+
+Each stage manifest should record enough information to determine whether its
+outputs can be trusted for resume:
+
+- workspace schema/version;
+- stage name and lane ID when applicable;
+- module key and relevant module provenance;
+- dependency fingerprints and input digests;
+- status such as pending, running, succeeded, succeeded with rejections, failed,
+  or invalidated;
+- output content digests;
+- validation status and rejection summaries where relevant;
+- timing metadata when useful and non-sensitive.
+
+## Checkpoint Output
+
+When `workspace.resume.enabled` is enabled, Notarius should write under the
+`checkpoints/` subtree. Checkpoints should contain only artifacts required to
+resume safely.
+
+Checkpoint artifacts may include:
+
+- source document checkpoint;
+- chunk collection checkpoint;
+- accepted extract outputs;
+- rejected extract records;
+- merge output;
+- normalize output;
+- stage manifests with dependency fingerprints and output digests.
+
+Checkpoint output should avoid raw prompts, raw references, raw LLM request
+payloads, and other sensitive development artifacts unless they are strictly
+required for safe resume. Prefer digests and provenance over copying sensitive
+inputs.
+
+Checkpoint writes should be atomic at the file level. Partial or interrupted
+writes must not be mistaken for successful stage completion.
+
+## Debug Output
+
+When `workspace.debug.enabled` is enabled, Notarius should write under the
+`debug/<run-id>/` subtree. Debug artifacts are useful for inspection but not
+required for resume.
+
+Debug artifacts may include:
+
+- rendered prompt inputs;
+- structured LLM request and response traces;
+- raw model responses;
+- copied reference content;
+- copied source snippets;
+- intermediate raw payloads;
+- validator request/response details.
+
+Debug output is explicitly sensitive. Documentation should warn that debug
+workspace data may contain source material, references, prompts, and model
+outputs. Debug output should remain disabled by default.
+
+`workspace.debug.enabled` and `workspace.resume.enabled` should be independent.
+Enabling debug should not imply resume, and enabling resume should not imply
+debug.
+
+## Resume Semantics
+
+Resume should be explicit at first, for example through a future `resume` command
+or a `run --resume` flag. Plain `run` should not silently skip completed stages
+in the initial workspace feature.
+
+Before reusing a checkpoint, Notarius should verify that:
+
+- the workspace schema/version is supported;
+- the checkpoint stage manifest is complete and successful;
+- dependency fingerprints match the current invocation;
+- referenced checkpoint files exist and match recorded digests;
+- selected lanes and runtime overrides are compatible with the checkpoint.
+
+Rejected extract outputs should be treated as recorded stage outcomes, not
+framework failures. Under current pipeline policy, rejected outputs do not pass
+to downstream stages.
+
+## Boundaries
+
+Workspace writing should be framework-owned. Concrete modules should not write
+directly to workspace paths.
+
+If modules need to expose debug material later, they should return logical
+artifacts through framework contracts and let framework or CLI code perform path
+validation, redaction policy, and writes.
+
+Workspace state should preserve existing Notarius boundaries:
+
+- provider plumbing remains behind LLM runtime contracts;
+- prompt ownership remains with modules and prompt asset helpers;
+- diagnostics remain per-run invocation artifacts under the workspace root;
+- output modules continue to own final durable output encoding;
+- secrets and sensitive payloads are not written unless an explicit debug policy
+  enables them.
+
+## Deferred Work
+
+Default-idempotent `run` behavior with a `--force` override, remote workspace
+storage, workspace garbage collection, archival policy, and cross-machine resume
+are deferred.