Add a planning roadmap and a staged implementation plan for the workspace configuration

docs/roadmap/implementation.md

@@ -0,0 +1,394 @@
+# Workspace Implementation Plan
+
+This plan implements the workspace target described in
+[Workspace Roadmap](workspace.md). Follow the stages in order. Keep code changes
+focused on the current stage, and update tests and canonical docs in the same
+stage when behavior changes.
+
+The intended end state is:
+
+- one configurable workspace root for Notarius-owned local state;
+- diagnostics written under `workspace/diagnostics/<run-id>/` when workspace
+  diagnostics are enabled;
+- resumability checkpoints written under deterministic `workspace/checkpoints/`
+  paths only when resume checkpointing is enabled;
+- debug artifacts written under `workspace/debug/<run-id>/` only when debug is
+  enabled;
+- stage-owned checkpoint manifests only, with no root-level checkpoint summary;
+- explicit resume behavior through `run --resume`, not implicit skipping during
+  ordinary `run`.
+
+## Stage 1: Add Workspace Configuration
+
+Add workspace configuration while preserving legacy diagnostics configuration
+long enough for backward compatibility.
+
+Implement in `internal/core/config`:
+
+- Add `WorkspaceConfig` to `Config`.
+- Add nested structs:
+  - `WorkspaceDiagnosticsConfig`
+  - `WorkspaceResumeConfig`
+  - `WorkspaceDebugConfig`
+- Add file config parsing for:
+
+  ```yaml
+  workspace:
+    directory: /var/lib/notarius
+    diagnostics:
+      enabled: true
+      retention: auto
+    resume:
+      enabled: false
+    debug:
+      enabled: false
+  ```
+
+- Preserve existing `diagnostics.work_dir` and `diagnostics.retention` parsing as
+  deprecated compatibility input.
+- Resolve effective diagnostics behavior as follows:
+  - if `workspace.directory` is set, diagnostics root is
+    `<workspace.directory>/diagnostics`;
+  - if `workspace.directory` is unset and legacy `diagnostics.work_dir` is set,
+    use legacy diagnostics behavior unchanged;
+  - if neither is set, preserve the current default diagnostics behavior for
+    migration compatibility;
+  - `workspace.diagnostics.retention` overrides legacy diagnostics retention
+    when set;
+  - legacy diagnostics retention remains accepted when workspace diagnostics
+    retention is unset.
+- If `workspace.diagnostics.enabled` is explicitly false, do not create a
+  diagnostics run directory and do not write diagnostics artifacts. CLI failure
+  handling must still print concise errors to stderr without assuming a
+  diagnostics directory exists.
+- Add environment support:
+  - `NOTARIUS_WORKSPACE_DIR` sets `workspace.directory`;
+  - `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED` parses a boolean;
+  - `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION` sets workspace diagnostics
+    retention;
+  - `NOTARIUS_WORKSPACE_RESUME_ENABLED` parses a boolean;
+  - `NOTARIUS_WORKSPACE_DEBUG_ENABLED` parses a boolean;
+  - keep `NOTARIUS_WORK_DIR` and `NOTARIUS_DIAGNOSTICS_RETENTION` as deprecated
+    compatibility overrides for legacy diagnostics config.
+- Keep `--diagnostics-dir` as a backwards-compatible per-invocation diagnostics
+  root override. It should affect diagnostics only and should not change
+  checkpoint or debug roots.
+- Update redaction/effective-config diagnostics so workspace config is included
+  and no secrets are introduced.
+
+Tests:
+
+- Add config default tests for workspace defaults.
+- Add file config parsing tests for nested workspace fields.
+- Add env override tests for all new env vars.
+- Add precedence tests covering workspace config, legacy diagnostics config,
+  env overrides, and `--diagnostics-dir`.
+- Update redacted/effective config tests.
+
+## Stage 2: Introduce Workspace Filesystem Helpers
+
+Create reusable filesystem helpers for workspace state. Keep concrete module
+packages out of this layer.
+
+Implement a new package, recommended path `internal/core/workspace`, with:
+
+- `Config` or `Settings` describing effective workspace roots:
+  - root directory;
+  - diagnostics root;
+  - checkpoints root;
+  - debug root;
+  - enabled flags.
+- path construction helpers for:
+  - diagnostics run directories;
+  - checkpoint identity directories;
+  - debug run directories.
+- safe path helpers that reject absolute artifact names, `..`, backslashes, and
+  paths that escape their intended root.
+- atomic JSON and byte-file writes using the existing diagnostics atomic-write
+  behavior as the model.
+- optional shared internal helper for atomic file writes so diagnostics and
+  workspace writers do not duplicate low-level write logic.
+
+Do not add resume behavior in this stage.
+
+Tests:
+
+- Unit-test path construction and path-safety failures.
+- Unit-test atomic JSON/byte writes.
+- Unit-test disabled workspace settings returning no-op or empty roots as
+  appropriate.
+- Confirm no helper permits writes outside the configured root.
+
+## Stage 3: Move Diagnostics Under Workspace
+
+Update diagnostics creation to use the effective diagnostics root from workspace
+configuration when workspace is configured.
+
+Implementation requirements:
+
+- Keep diagnostics artifacts and retention behavior unchanged.
+- Honor disabled diagnostics by using a no-op diagnostics writer or nil-safe
+  diagnostics path through the CLI failure and success paths.
+- Preserve the existing `diagnostics.RunDirectory` contract unless a narrow
+  constructor addition is cleaner.
+- Route normal workspace diagnostics to:
+
+  ```text
+  <workspace.directory>/diagnostics/<run-id>/
+  ```
+
+- Preserve legacy behavior when only legacy diagnostics config is present.
+- Preserve `--diagnostics-dir` behavior as diagnostics-only override.
+- Keep diagnostics run IDs run-based and non-deterministic.
+- Do not write checkpoint or debug output in this stage.
+
+Docs to update after behavior exists:
+
+- `docs/config.md`
+- `docs/operations.md`
+- `docs/internal/diagnostics.md`
+- `docs/cli.md` if CLI help text changes.
+
+Tests:
+
+- Update diagnostics run directory tests for workspace diagnostics roots.
+- Update CLI tests that inspect diagnostics paths.
+- Add migration tests proving legacy diagnostics config still works.
+- Run focused checks:
+  - `go test ./internal/core/config`
+  - `go test ./internal/core/diagnostics`
+  - `go test ./internal/cli`
+
+## Stage 4: Define Checkpoint Identity And Stage Manifest Types
+
+Add checkpoint identity and manifest data structures before writing stage
+payloads.
+
+Implement in `internal/core/workspace` or a closely related core package:
+
+- `CheckpointIdentity`, derived from:
+  - resolved pipeline ID;
+  - resolved pipeline digest;
+  - input adapter key;
+  - raw input digest or source digest;
+  - selected lanes;
+  - runtime overrides that affect execution;
+  - materialized reference digests;
+  - prompt/schema/profile provenance not already represented by the pipeline
+    digest.
+- A deterministic, filesystem-safe checkpoint path:
+
+  ```text
+  <workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-digest-prefix>/<pipeline-digest-prefix>/
+  ```
+
+- Stage manifest structs for:
+  - source;
+  - chunk;
+  - extract lane;
+  - merge lane;
+  - normalize lane.
+- Shared manifest fields:
+  - workspace schema version;
+  - stage name;
+  - lane ID when applicable;
+  - module key;
+  - dependency fingerprints;
+  - status;
+  - output digests;
+  - validation status and rejection summaries where applicable;
+  - started/completed timestamps where useful.
+- Status values:
+  - `pending`;
+  - `running`;
+  - `succeeded`;
+  - `succeeded_with_rejections`;
+  - `failed`;
+  - `invalidated`.
+
+Do not add a root-level checkpoint manifest.
+
+Tests:
+
+- Unit-test deterministic identity generation.
+- Unit-test identity changes when pipeline digest, input digest, selected lanes,
+  or reference digests change.
+- Unit-test manifest JSON round trips.
+- Unit-test filesystem-safe path generation.
+
+## Stage 5: Write Checkpoints Without Resuming
+
+Add write-only checkpoint support behind `workspace.resume.enabled`.
+
+Implementation requirements:
+
+- Add a framework-owned checkpoint recorder to pipeline execution. Recommended
+  shape:
+  - CLI constructs the effective workspace/checkpoint recorder after pipeline
+    resolution and reference materialization.
+  - `pipeline.RunInput` receives a recorder interface or no-op recorder.
+  - concrete modules do not receive workspace paths and do not write directly to
+    workspace.
+- Write checkpoint artifacts only when `workspace.resume.enabled` is true.
+- Use stage-owned manifests and no root summary.
+- Use JSON envelope types for checkpointed payloads that preserve byte content,
+  such as `content_base64`, media type, metadata, warnings, and content digest.
+  Do not rely on existing runtime structs whose byte fields are tagged
+  `json:"-"`.
+- Recommended checkpoint files:
+
+  ```text
+  checkpoints/<identity>/source/manifest.json
+  checkpoints/<identity>/source/source-document.json
+  checkpoints/<identity>/chunk/manifest.json
+  checkpoints/<identity>/chunk/chunks.json
+  checkpoints/<identity>/extract/<lane-id>/manifest.json
+  checkpoints/<identity>/extract/<lane-id>/outputs.json
+  checkpoints/<identity>/merge/<lane-id>/manifest.json
+  checkpoints/<identity>/merge/<lane-id>/output.json
+  checkpoints/<identity>/normalize/<lane-id>/manifest.json
+  checkpoints/<identity>/normalize/<lane-id>/output.json
+  ```
+
+- Record rejected extract outputs as checkpointed stage outcomes. Under current
+  pipeline policy they do not pass downstream, but they are not framework errors.
+- Mark a stage `running` before writing its payloads, then atomically replace the
+  manifest with a final success/failure status after payload writes complete.
+- Ensure interrupted or partial writes cannot be mistaken for successful
+  checkpoints.
+- Do not write raw prompts, raw references, raw LLM request payloads, or debug
+  traces in checkpoint output.
+
+Tests:
+
+- Add runner/CLI tests proving no checkpoint files are written when resume is
+  disabled.
+- Add tests proving checkpoint files are written when resume is enabled.
+- Add tests for successful runs, rejected extract outputs, failed stages, and
+  warning-only validation.
+- Add tests proving checkpoint payload content digests match written content.
+- Add tests proving modules do not receive filesystem paths.
+
+## Stage 6: Add Explicit Resume Reads
+
+Add explicit resume behavior after write-only checkpoints are stable.
+
+CLI behavior:
+
+- Add `run --resume`.
+- `--resume` should require `workspace.resume.enabled: true`; otherwise return a
+  clear configuration error.
+- Plain `run` should continue to execute stages normally and should not silently
+  skip completed stages.
+
+Resume behavior:
+
+- Load and validate checkpoint stage manifests in workflow order.
+- Reuse a checkpoint only when:
+  - workspace schema version is supported;
+  - stage status is successful for the current purpose;
+  - dependency fingerprints match the current invocation;
+  - referenced checkpoint payload files exist;
+  - payload digests match manifest digests;
+  - selected lanes and runtime overrides are compatible.
+- If a checkpoint is missing or invalid, execute that stage normally and write a
+  fresh checkpoint if resume checkpointing remains enabled.
+- If a source or chunk checkpoint is reused, downstream dependency fingerprints
+  must still be validated before downstream reuse.
+- If an extract checkpoint includes rejected outputs, preserve those rejection
+  records and continue to omit rejected outputs from merge input.
+- If merge is reused, skip merge execution only for the matching lane.
+- If normalize is reused, pass the reused normalized output to the output stage.
+- Always create fresh diagnostics for the current invocation, including records
+  showing which stages were reused versus executed.
+
+Tests:
+
+- Add CLI tests for `--resume` without workspace resume enabled.
+- Add tests for reusing source, chunk, extract, merge, and normalize
+  checkpoints.
+- Add tests for invalidation when input, pipeline digest, references, selected
+  lanes, or runtime LLM profile override changes.
+- Add tests for corrupt or missing checkpoint payloads.
+- Add tests proving fresh diagnostics are written for resumed invocations.
+
+## Stage 7: Add Debug Output
+
+Add debug output behind `workspace.debug.enabled`.
+
+Implementation requirements:
+
+- Write debug artifacts under:
+
+  ```text
+  <workspace.directory>/debug/<run-id>/
+  ```
+
+- Debug is per-invocation and should not be used for resume.
+- Debug and resume are independent:
+  - debug enabled does not imply checkpoint writing;
+  - resume enabled does not imply debug writing.
+- Start with artifacts available from Notarius framework boundaries:
+  - source/checkpoint-like stage inputs and outputs;
+  - structured LLM request inputs and response content from Notarius contracts;
+  - validator requests and results;
+  - stage timing and attempt metadata.
+- Do not depend on Scriptorium internals for rendered upstream provider
+  requests. If Scriptorium later exposes rendered prompt traces safely, add them
+  in a separate pass.
+- Never write API keys, bearer tokens, or raw provider credentials.
+- Document clearly that debug output may contain source material, references,
+  prompts, model outputs, and other sensitive content.
+
+Tests:
+
+- Add tests proving no debug files are written when debug is disabled.
+- Add tests proving debug files are written under `debug/<run-id>/` when enabled.
+- Add tests proving debug and resume can be enabled independently.
+- Add tests proving obvious secrets are not written.
+- Add diagnostics/debug path tests to ensure path roots do not overlap
+  accidentally.
+
+## Stage 8: Documentation, Examples, And Cleanup
+
+After behavior is implemented, update canonical current-behavior docs.
+
+Update:
+
+- `docs/config.md`
+- `docs/cli.md`
+- `docs/operations.md`
+- `docs/troubleshooting.md`
+- `docs/internal/diagnostics.md`
+- `docs/internal/pipeline.md`
+- maintained examples under `examples/` when practical.
+
+Documentation requirements:
+
+- Document workspace config and defaults.
+- Document legacy diagnostics config compatibility and any deprecation language.
+- Document `/var/lib/notarius` as the production recommendation.
+- Document local development recommendations.
+- Document checkpoint sensitivity and debug sensitivity.
+- Document explicit resume behavior and invalidation rules.
+- Keep future or deferred behavior only in roadmap files.
+
+Cleanup:
+
+- Update `docs/roadmap/workspace.md` to a completed-status note after the
+  implementation lands.
+- Remove or replace this implementation plan with a completed note after the
+  implementation lands.
+
+Validation:
+
+```sh
+go test ./internal/core/config
+go test ./internal/core/diagnostics
+go test ./internal/core/workspace
+go test ./internal/framework/pipeline
+go test ./internal/cli
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```