From 304c68f9fcf186eac6d16af1e4ee86197601a82b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 7 Jul 2026 21:08:52 -0500 Subject: [PATCH] Add a planning roadmap and a staged implementation plan for the workspace configuration --- docs/roadmap/implementation.md | 394 +++++++++++++++++++++++++++++++++ docs/roadmap/workspace.md | 233 +++++++++++++++++++ 2 files changed, 627 insertions(+) create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/workspace.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..18c0b17 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,394 @@ +# Workspace Implementation Plan + +This plan implements the workspace target described in +[Workspace Roadmap](workspace.md). Follow the stages in order. Keep code changes +focused on the current stage, and update tests and canonical docs in the same +stage when behavior changes. + +The intended end state is: + +- one configurable workspace root for Notarius-owned local state; +- diagnostics written under `workspace/diagnostics//` when workspace + diagnostics are enabled; +- resumability checkpoints written under deterministic `workspace/checkpoints/` + paths only when resume checkpointing is enabled; +- debug artifacts written under `workspace/debug//` only when debug is + enabled; +- stage-owned checkpoint manifests only, with no root-level checkpoint summary; +- explicit resume behavior through `run --resume`, not implicit skipping during + ordinary `run`. + +## Stage 1: Add Workspace Configuration + +Add workspace configuration while preserving legacy diagnostics configuration +long enough for backward compatibility. + +Implement in `internal/core/config`: + +- Add `WorkspaceConfig` to `Config`. +- Add nested structs: + - `WorkspaceDiagnosticsConfig` + - `WorkspaceResumeConfig` + - `WorkspaceDebugConfig` +- Add file config parsing for: + + ```yaml + workspace: + directory: /var/lib/notarius + diagnostics: + enabled: true + retention: auto + resume: + enabled: false + debug: + enabled: false + ``` + +- Preserve existing `diagnostics.work_dir` and `diagnostics.retention` parsing as + deprecated compatibility input. +- Resolve effective diagnostics behavior as follows: + - if `workspace.directory` is set, diagnostics root is + `/diagnostics`; + - if `workspace.directory` is unset and legacy `diagnostics.work_dir` is set, + use legacy diagnostics behavior unchanged; + - if neither is set, preserve the current default diagnostics behavior for + migration compatibility; + - `workspace.diagnostics.retention` overrides legacy diagnostics retention + when set; + - legacy diagnostics retention remains accepted when workspace diagnostics + retention is unset. +- If `workspace.diagnostics.enabled` is explicitly false, do not create a + diagnostics run directory and do not write diagnostics artifacts. CLI failure + handling must still print concise errors to stderr without assuming a + diagnostics directory exists. +- Add environment support: + - `NOTARIUS_WORKSPACE_DIR` sets `workspace.directory`; + - `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED` parses a boolean; + - `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION` sets workspace diagnostics + retention; + - `NOTARIUS_WORKSPACE_RESUME_ENABLED` parses a boolean; + - `NOTARIUS_WORKSPACE_DEBUG_ENABLED` parses a boolean; + - keep `NOTARIUS_WORK_DIR` and `NOTARIUS_DIAGNOSTICS_RETENTION` as deprecated + compatibility overrides for legacy diagnostics config. +- Keep `--diagnostics-dir` as a backwards-compatible per-invocation diagnostics + root override. It should affect diagnostics only and should not change + checkpoint or debug roots. +- Update redaction/effective-config diagnostics so workspace config is included + and no secrets are introduced. + +Tests: + +- Add config default tests for workspace defaults. +- Add file config parsing tests for nested workspace fields. +- Add env override tests for all new env vars. +- Add precedence tests covering workspace config, legacy diagnostics config, + env overrides, and `--diagnostics-dir`. +- Update redacted/effective config tests. + +## Stage 2: Introduce Workspace Filesystem Helpers + +Create reusable filesystem helpers for workspace state. Keep concrete module +packages out of this layer. + +Implement a new package, recommended path `internal/core/workspace`, with: + +- `Config` or `Settings` describing effective workspace roots: + - root directory; + - diagnostics root; + - checkpoints root; + - debug root; + - enabled flags. +- path construction helpers for: + - diagnostics run directories; + - checkpoint identity directories; + - debug run directories. +- safe path helpers that reject absolute artifact names, `..`, backslashes, and + paths that escape their intended root. +- atomic JSON and byte-file writes using the existing diagnostics atomic-write + behavior as the model. +- optional shared internal helper for atomic file writes so diagnostics and + workspace writers do not duplicate low-level write logic. + +Do not add resume behavior in this stage. + +Tests: + +- Unit-test path construction and path-safety failures. +- Unit-test atomic JSON/byte writes. +- Unit-test disabled workspace settings returning no-op or empty roots as + appropriate. +- Confirm no helper permits writes outside the configured root. + +## Stage 3: Move Diagnostics Under Workspace + +Update diagnostics creation to use the effective diagnostics root from workspace +configuration when workspace is configured. + +Implementation requirements: + +- Keep diagnostics artifacts and retention behavior unchanged. +- Honor disabled diagnostics by using a no-op diagnostics writer or nil-safe + diagnostics path through the CLI failure and success paths. +- Preserve the existing `diagnostics.RunDirectory` contract unless a narrow + constructor addition is cleaner. +- Route normal workspace diagnostics to: + + ```text + /diagnostics// + ``` + +- Preserve legacy behavior when only legacy diagnostics config is present. +- Preserve `--diagnostics-dir` behavior as diagnostics-only override. +- Keep diagnostics run IDs run-based and non-deterministic. +- Do not write checkpoint or debug output in this stage. + +Docs to update after behavior exists: + +- `docs/config.md` +- `docs/operations.md` +- `docs/internal/diagnostics.md` +- `docs/cli.md` if CLI help text changes. + +Tests: + +- Update diagnostics run directory tests for workspace diagnostics roots. +- Update CLI tests that inspect diagnostics paths. +- Add migration tests proving legacy diagnostics config still works. +- Run focused checks: + - `go test ./internal/core/config` + - `go test ./internal/core/diagnostics` + - `go test ./internal/cli` + +## Stage 4: Define Checkpoint Identity And Stage Manifest Types + +Add checkpoint identity and manifest data structures before writing stage +payloads. + +Implement in `internal/core/workspace` or a closely related core package: + +- `CheckpointIdentity`, derived from: + - resolved pipeline ID; + - resolved pipeline digest; + - input adapter key; + - raw input digest or source digest; + - selected lanes; + - runtime overrides that affect execution; + - materialized reference digests; + - prompt/schema/profile provenance not already represented by the pipeline + digest. +- A deterministic, filesystem-safe checkpoint path: + + ```text + /checkpoints//-// + ``` + +- Stage manifest structs for: + - source; + - chunk; + - extract lane; + - merge lane; + - normalize lane. +- Shared manifest fields: + - workspace schema version; + - stage name; + - lane ID when applicable; + - module key; + - dependency fingerprints; + - status; + - output digests; + - validation status and rejection summaries where applicable; + - started/completed timestamps where useful. +- Status values: + - `pending`; + - `running`; + - `succeeded`; + - `succeeded_with_rejections`; + - `failed`; + - `invalidated`. + +Do not add a root-level checkpoint manifest. + +Tests: + +- Unit-test deterministic identity generation. +- Unit-test identity changes when pipeline digest, input digest, selected lanes, + or reference digests change. +- Unit-test manifest JSON round trips. +- Unit-test filesystem-safe path generation. + +## Stage 5: Write Checkpoints Without Resuming + +Add write-only checkpoint support behind `workspace.resume.enabled`. + +Implementation requirements: + +- Add a framework-owned checkpoint recorder to pipeline execution. Recommended + shape: + - CLI constructs the effective workspace/checkpoint recorder after pipeline + resolution and reference materialization. + - `pipeline.RunInput` receives a recorder interface or no-op recorder. + - concrete modules do not receive workspace paths and do not write directly to + workspace. +- Write checkpoint artifacts only when `workspace.resume.enabled` is true. +- Use stage-owned manifests and no root summary. +- Use JSON envelope types for checkpointed payloads that preserve byte content, + such as `content_base64`, media type, metadata, warnings, and content digest. + Do not rely on existing runtime structs whose byte fields are tagged + `json:"-"`. +- Recommended checkpoint files: + + ```text + checkpoints//source/manifest.json + checkpoints//source/source-document.json + checkpoints//chunk/manifest.json + checkpoints//chunk/chunks.json + checkpoints//extract//manifest.json + checkpoints//extract//outputs.json + checkpoints//merge//manifest.json + checkpoints//merge//output.json + checkpoints//normalize//manifest.json + checkpoints//normalize//output.json + ``` + +- Record rejected extract outputs as checkpointed stage outcomes. Under current + pipeline policy they do not pass downstream, but they are not framework errors. +- Mark a stage `running` before writing its payloads, then atomically replace the + manifest with a final success/failure status after payload writes complete. +- Ensure interrupted or partial writes cannot be mistaken for successful + checkpoints. +- Do not write raw prompts, raw references, raw LLM request payloads, or debug + traces in checkpoint output. + +Tests: + +- Add runner/CLI tests proving no checkpoint files are written when resume is + disabled. +- Add tests proving checkpoint files are written when resume is enabled. +- Add tests for successful runs, rejected extract outputs, failed stages, and + warning-only validation. +- Add tests proving checkpoint payload content digests match written content. +- Add tests proving modules do not receive filesystem paths. + +## Stage 6: Add Explicit Resume Reads + +Add explicit resume behavior after write-only checkpoints are stable. + +CLI behavior: + +- Add `run --resume`. +- `--resume` should require `workspace.resume.enabled: true`; otherwise return a + clear configuration error. +- Plain `run` should continue to execute stages normally and should not silently + skip completed stages. + +Resume behavior: + +- Load and validate checkpoint stage manifests in workflow order. +- Reuse a checkpoint only when: + - workspace schema version is supported; + - stage status is successful for the current purpose; + - dependency fingerprints match the current invocation; + - referenced checkpoint payload files exist; + - payload digests match manifest digests; + - selected lanes and runtime overrides are compatible. +- If a checkpoint is missing or invalid, execute that stage normally and write a + fresh checkpoint if resume checkpointing remains enabled. +- If a source or chunk checkpoint is reused, downstream dependency fingerprints + must still be validated before downstream reuse. +- If an extract checkpoint includes rejected outputs, preserve those rejection + records and continue to omit rejected outputs from merge input. +- If merge is reused, skip merge execution only for the matching lane. +- If normalize is reused, pass the reused normalized output to the output stage. +- Always create fresh diagnostics for the current invocation, including records + showing which stages were reused versus executed. + +Tests: + +- Add CLI tests for `--resume` without workspace resume enabled. +- Add tests for reusing source, chunk, extract, merge, and normalize + checkpoints. +- Add tests for invalidation when input, pipeline digest, references, selected + lanes, or runtime LLM profile override changes. +- Add tests for corrupt or missing checkpoint payloads. +- Add tests proving fresh diagnostics are written for resumed invocations. + +## Stage 7: Add Debug Output + +Add debug output behind `workspace.debug.enabled`. + +Implementation requirements: + +- Write debug artifacts under: + + ```text + /debug// + ``` + +- Debug is per-invocation and should not be used for resume. +- Debug and resume are independent: + - debug enabled does not imply checkpoint writing; + - resume enabled does not imply debug writing. +- Start with artifacts available from Notarius framework boundaries: + - source/checkpoint-like stage inputs and outputs; + - structured LLM request inputs and response content from Notarius contracts; + - validator requests and results; + - stage timing and attempt metadata. +- Do not depend on Scriptorium internals for rendered upstream provider + requests. If Scriptorium later exposes rendered prompt traces safely, add them + in a separate pass. +- Never write API keys, bearer tokens, or raw provider credentials. +- Document clearly that debug output may contain source material, references, + prompts, model outputs, and other sensitive content. + +Tests: + +- Add tests proving no debug files are written when debug is disabled. +- Add tests proving debug files are written under `debug//` when enabled. +- Add tests proving debug and resume can be enabled independently. +- Add tests proving obvious secrets are not written. +- Add diagnostics/debug path tests to ensure path roots do not overlap + accidentally. + +## Stage 8: Documentation, Examples, And Cleanup + +After behavior is implemented, update canonical current-behavior docs. + +Update: + +- `docs/config.md` +- `docs/cli.md` +- `docs/operations.md` +- `docs/troubleshooting.md` +- `docs/internal/diagnostics.md` +- `docs/internal/pipeline.md` +- maintained examples under `examples/` when practical. + +Documentation requirements: + +- Document workspace config and defaults. +- Document legacy diagnostics config compatibility and any deprecation language. +- Document `/var/lib/notarius` as the production recommendation. +- Document local development recommendations. +- Document checkpoint sensitivity and debug sensitivity. +- Document explicit resume behavior and invalidation rules. +- Keep future or deferred behavior only in roadmap files. + +Cleanup: + +- Update `docs/roadmap/workspace.md` to a completed-status note after the + implementation lands. +- Remove or replace this implementation plan with a completed note after the + implementation lands. + +Validation: + +```sh +go test ./internal/core/config +go test ./internal/core/diagnostics +go test ./internal/core/workspace +go test ./internal/framework/pipeline +go test ./internal/cli +go test ./... +go vet ./... +go build ./cmd/notarius +``` diff --git a/docs/roadmap/workspace.md b/docs/roadmap/workspace.md new file mode 100644 index 0000000..8c29300 --- /dev/null +++ b/docs/roadmap/workspace.md @@ -0,0 +1,233 @@ +# Workspace Roadmap + +Notarius should gain a single configurable workspace root for application-owned +local state. The workspace exists only to support enabled workspace features; +ordinary runs should not write workspace files by default. + +The workspace is distinct from durable output: + +- durable output is the user-requested final product of a run; +- the workspace is application-owned local state for diagnostics, resumable + checkpoints, and development/debug workflows. + +Within the workspace, diagnostics, checkpoints, and debug artifacts should use +separate subdirectories because they have different identity and lifecycle +models. + +## Target Configuration + +Workspace behavior should be configured under a dedicated top-level object: + +```yaml +workspace: + directory: /var/lib/notarius + diagnostics: + enabled: true + retention: auto + resume: + enabled: false + debug: + enabled: false +``` + +Policy: + +- `workspace.diagnostics.enabled` defaults to the current diagnostics behavior + during migration. +- `workspace.resume.enabled` defaults to `false`. +- `workspace.debug.enabled` defaults to `false`. +- no checkpoint or debug files are written when those features are disabled. +- `/var/lib/notarius` should be the standard production recommendation in docs + and examples. +- local development docs may recommend a project-local path such as + `./.notarius/workspace`. + +`workspace.directory` should become the single configured root for Notarius local +state. Existing `diagnostics.work_dir` behavior should be migrated carefully for +backward compatibility, but the long-term configuration model should avoid two +separate roots for application-owned local state. + +## Workspace Layout + +The workspace root should contain separate subtrees: + +```text +/ + diagnostics/ + checkpoints/ + debug/ +``` + +Diagnostics remain run-id based. They are invocation history and should preserve +the existing diagnostics retention model: + +```text +/ + diagnostics/ + / + invocation.json + effective-config.json + resolved-pipeline.json + resolved-references.json + run-manifest.json + warnings.json + run-report.json + error.log +``` + +Checkpoints should be deterministic and should include enough identity material +to prevent unsafe reuse across incompatible runs: + +```text +/ + checkpoints/ + / + -/ + / + source/ + chunk/ + extract// + merge// + normalize// +``` + +Checkpoint identity should account for: + +- resolved pipeline ID; +- resolved pipeline digest; +- input adapter key; +- source/input digest; +- selected lanes and runtime overrides that affect execution; +- reference digests; +- prompt, schema, and profile provenance when not already captured by the + resolved pipeline digest. + +Debug output should be run-id based like diagnostics, because debug traces are +development artifacts from a specific invocation rather than resumable state: + +```text +/ + debug/ + / +``` + +Human-readable path segments are useful, but content hashes should be +authoritative for correctness. The exact path shape may change during +implementation, but checkpoint identity must be stable, deterministic, and safe +to compare across runs. Diagnostics and debug output do not need deterministic +resume identity because they are per-invocation artifacts. + +## Stage Ownership + +Stage manifests should be stage-owned. There should not be a root-level manifest +that summarizes all stages. + +Rationale: + +- stage manifests are the authoritative state for the stage that wrote them; +- avoiding a root summary prevents duplicate state from drifting; +- stage-local manifests make partial writes, failure recovery, and future stage + invalidation easier to reason about. + +Each stage manifest should record enough information to determine whether its +outputs can be trusted for resume: + +- workspace schema/version; +- stage name and lane ID when applicable; +- module key and relevant module provenance; +- dependency fingerprints and input digests; +- status such as pending, running, succeeded, succeeded with rejections, failed, + or invalidated; +- output content digests; +- validation status and rejection summaries where relevant; +- timing metadata when useful and non-sensitive. + +## Checkpoint Output + +When `workspace.resume.enabled` is enabled, Notarius should write under the +`checkpoints/` subtree. Checkpoints should contain only artifacts required to +resume safely. + +Checkpoint artifacts may include: + +- source document checkpoint; +- chunk collection checkpoint; +- accepted extract outputs; +- rejected extract records; +- merge output; +- normalize output; +- stage manifests with dependency fingerprints and output digests. + +Checkpoint output should avoid raw prompts, raw references, raw LLM request +payloads, and other sensitive development artifacts unless they are strictly +required for safe resume. Prefer digests and provenance over copying sensitive +inputs. + +Checkpoint writes should be atomic at the file level. Partial or interrupted +writes must not be mistaken for successful stage completion. + +## Debug Output + +When `workspace.debug.enabled` is enabled, Notarius should write under the +`debug//` subtree. Debug artifacts are useful for inspection but not +required for resume. + +Debug artifacts may include: + +- rendered prompt inputs; +- structured LLM request and response traces; +- raw model responses; +- copied reference content; +- copied source snippets; +- intermediate raw payloads; +- validator request/response details. + +Debug output is explicitly sensitive. Documentation should warn that debug +workspace data may contain source material, references, prompts, and model +outputs. Debug output should remain disabled by default. + +`workspace.debug.enabled` and `workspace.resume.enabled` should be independent. +Enabling debug should not imply resume, and enabling resume should not imply +debug. + +## Resume Semantics + +Resume should be explicit at first, for example through a future `resume` command +or a `run --resume` flag. Plain `run` should not silently skip completed stages +in the initial workspace feature. + +Before reusing a checkpoint, Notarius should verify that: + +- the workspace schema/version is supported; +- the checkpoint stage manifest is complete and successful; +- dependency fingerprints match the current invocation; +- referenced checkpoint files exist and match recorded digests; +- selected lanes and runtime overrides are compatible with the checkpoint. + +Rejected extract outputs should be treated as recorded stage outcomes, not +framework failures. Under current pipeline policy, rejected outputs do not pass +to downstream stages. + +## Boundaries + +Workspace writing should be framework-owned. Concrete modules should not write +directly to workspace paths. + +If modules need to expose debug material later, they should return logical +artifacts through framework contracts and let framework or CLI code perform path +validation, redaction policy, and writes. + +Workspace state should preserve existing Notarius boundaries: + +- provider plumbing remains behind LLM runtime contracts; +- prompt ownership remains with modules and prompt asset helpers; +- diagnostics remain per-run invocation artifacts under the workspace root; +- output modules continue to own final durable output encoding; +- secrets and sensitive payloads are not written unless an explicit debug policy + enables them. + +## Deferred Work + +Default-idempotent `run` behavior with a `--force` override, remote workspace +storage, workspace garbage collection, archival policy, and cross-machine resume +are deferred.