From ae65b953746a8b12cbe532fc7540aba03b7db5b5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 8 Jul 2026 03:18:50 +0000 Subject: [PATCH] Document completed workspace behavior --- docs/cli.md | 7 +- docs/config.md | 47 +++- docs/internal/diagnostics.md | 7 + docs/internal/pipeline.md | 10 +- docs/operations.md | 20 +- docs/roadmap/implementation.md | 420 ++------------------------------- docs/roadmap/workspace.md | 231 +----------------- docs/troubleshooting.md | 44 ++++ examples/dnd-spells.config.yml | 15 ++ 9 files changed, 178 insertions(+), 623 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 942cbdc..5b22d92 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -42,7 +42,7 @@ Flags: - `--only lane-a,lane-b`: run only the named artifact lanes. Values are comma-separated and must be non-empty. - `--resume`: reuse valid workspace checkpoints for this invocation. Requires - `workspace.resume.enabled: true`. + an effective workspace directory and `workspace.resume.enabled: true`. - `--output-dir path`: output root. The run writes to `//`. Defaults to `./notarius-output`. - `--diagnostics-dir path`: diagnostics work directory override for this @@ -152,6 +152,11 @@ go run ./cmd/notarius run dnd-session \ --resume ``` +Plain `run` does not skip completed work. It executes the pipeline normally and +refreshes checkpoints when checkpointing is enabled. `--resume` verifies each +checkpoint before reuse and executes any missing, corrupt, or incompatible step +normally. + For durable output, diagnostics, retention, and failure inspection, see [Operations](operations.md). diff --git a/docs/config.md b/docs/config.md index aeae3f9..562b46a 100644 --- a/docs/config.md +++ b/docs/config.md @@ -67,6 +67,10 @@ workspace: enabled: false ``` +`workspace.directory` is unset by default. Without a workspace directory, +diagnostics continue to use `/tmp/notarius`, and checkpoint and debug workspace +features have no storage root. + No pipelines are built in. A run requires a configured pipeline. If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile @@ -352,12 +356,40 @@ casts still must be present in the source transcript. - `resume.enabled`: boolean resume checkpointing setting. Default: `false`. - `debug.enabled`: boolean debug artifact setting. Default: `false`. -When `workspace.resume.enabled` is true, runs write stage-owned checkpoint -artifacts under `/checkpoints/`. `notarius run --resume` -can reuse valid checkpoints from a compatible invocation. +Use `/var/lib/notarius` as the standard production workspace directory. For +local development, prefer a project-local ignored path such as +`./.notarius/workspace`. -When `workspace.debug.enabled` is true, runs write per-invocation debug -artifacts under `/debug//`. +```yaml +workspace: + directory: /var/lib/notarius + diagnostics: + enabled: true + retention: auto + resume: + enabled: false + debug: + enabled: false +``` + +When `workspace.directory` is set, diagnostics are written under +`/diagnostics/`. + +When both `workspace.directory` and `workspace.resume.enabled` are set, runs +write stage-owned checkpoint artifacts under +`/checkpoints/`. `notarius run --resume` can reuse valid +checkpoints from a compatible invocation. Checkpoints may contain source text, +intermediate raw outputs, rejected outputs, metadata, and warnings. Protect the +workspace as sensitive local state. + +When both `workspace.directory` and `workspace.debug.enabled` are set, runs +write per-invocation debug artifacts under +`/debug//`. Debug artifacts may contain source +material, reference material, prompt inputs, model outputs, validation payloads, +and other sensitive content. Debug is disabled by default. + +`workspace.resume.enabled` and `workspace.debug.enabled` are independent. +Enabling one does not enable the other. ## Diagnostics @@ -380,6 +412,11 @@ set. - `retention`: deprecated compatibility retention mode. `auto`, `always`, or `never`. Empty uses `auto`. +Existing `diagnostics.work_dir`, `diagnostics.retention`, `NOTARIUS_WORK_DIR`, +and `NOTARIUS_DIAGNOSTICS_RETENTION` inputs remain supported for compatibility. +New configuration should use `workspace.directory` and +`workspace.diagnostics.retention` instead. + `auto` retains diagnostics for failed runs and successful runs with warnings. `always` retains diagnostics for every run. `never` removes diagnostics for successful runs without regard to warnings; failed runs are retained. diff --git a/docs/internal/diagnostics.md b/docs/internal/diagnostics.md index d6b3347..7878dad 100644 --- a/docs/internal/diagnostics.md +++ b/docs/internal/diagnostics.md @@ -22,6 +22,11 @@ Diagnostics must not expose secrets. If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults to `auto`. +The CLI passes the effective diagnostics root from workspace configuration. +When `workspace.directory` is set and diagnostics are enabled, that root is +`/diagnostics`. The legacy diagnostics work directory and +`--diagnostics-dir` still pass a diagnostics-only root to this constructor. + The writer makes the work directory if needed, then attempts to create a unique run directory. It retries run ID creation a bounded number of times if a collision occurs. @@ -93,3 +98,5 @@ manifest before logging the failure. information needed for recovery. - Durable output file contracts belong to output modules and integration docs, not to diagnostics. +- Checkpoint and debug workspace files are separate framework-owned artifacts, + not diagnostics artifacts. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 4616617..959d20f 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -80,13 +80,17 @@ workspace paths and do not write checkpoint files directly. For `run --resume`, the CLI also passes a checkpoint loader. The runner consults the loader in workflow order and reuses only checkpoints whose manifest schema, status, identity digest, dependency fingerprints, payload files, and payload -digests validate for the current invocation. Missing or invalid checkpoints fall -back to normal execution and are refreshed by the recorder. +digests validate for the current invocation. The identity includes the resolved +pipeline, selected lanes, source/input digest, runtime overrides that affect +execution, and materialized reference digests. Missing or invalid checkpoints +fall back to normal execution and are refreshed by the recorder. When workspace debug output is enabled, the CLI passes a debug recorder for the current run ID. The runner writes framework-boundary inputs, outputs, structured LLM calls, validator calls, timing, and retry attempt metadata -through that interface. Concrete modules still do not receive workspace paths. +through that interface. Debug output is not used for resume and can contain +sensitive source, reference, prompt, and model-output material. Concrete modules +still do not receive workspace paths. ## Registries And Module Specs diff --git a/docs/operations.md b/docs/operations.md index 3e5f331..c7b3f93 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -19,6 +19,10 @@ go run ./cmd/notarius run dnd-session \ The command prints a success line with the pipeline ID, normalized output count, rejected output count, and the output path. +For production, configure a workspace such as `/var/lib/notarius` and ensure the +Notarius process can create files below it. For local development, prefer an +ignored project-local workspace such as `./.notarius/workspace`. + ## Output Directory Durable output is written to: @@ -103,7 +107,9 @@ checkpoints and executes any missing, invalid, or incompatible step normally. Checkpoint payloads preserve byte content with base64 envelopes, media type, metadata, warnings, and content digests where applicable. Checkpoints do not include raw prompts, raw reference contents, raw LLM request payloads, or debug -traces. +traces. They can still contain source text, intermediate extracted content, +rejected outputs, metadata, and warnings. Treat checkpoint directories as +sensitive local state. A checkpoint is reused only when its workspace schema version, checkpoint identity digest, step status, dependency fingerprints, payload files, and @@ -111,6 +117,10 @@ payload digests match the current invocation. Changes to input bytes, resolved pipeline digest, selected lanes, runtime LLM profile override, or materialized reference digests invalidate reuse. +Plain `notarius run` does not reuse checkpoints. It executes the workflow and +refreshes checkpoint files when checkpointing is enabled. `notarius run +--resume` is the explicit reuse path. + ## Debug When `workspace.debug.enabled: true` and `workspace.directory` is set, runs @@ -181,6 +191,14 @@ rm -rf /tmp/notarius/run-1234567890 rm -rf ./notarius-output/run-1234567890 ``` +Workspace checkpoint and debug directories can also be removed when no longer +needed. Remove exact identity or run directories, for example: + +```sh +rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234 +rm -rf /var/lib/notarius/debug/run-1234567890 +``` + Use exact run-directory paths. Avoid broad cleanup commands against parent directories unless they are part of your own operational policy. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 18c0b17..0c1a3a1 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,394 +1,26 @@ -# Workspace Implementation Plan - -This plan implements the workspace target described in -[Workspace Roadmap](workspace.md). Follow the stages in order. Keep code changes -focused on the current stage, and update tests and canonical docs in the same -stage when behavior changes. - -The intended end state is: - -- one configurable workspace root for Notarius-owned local state; -- diagnostics written under `workspace/diagnostics//` when workspace - diagnostics are enabled; -- resumability checkpoints written under deterministic `workspace/checkpoints/` - paths only when resume checkpointing is enabled; -- debug artifacts written under `workspace/debug//` only when debug is - enabled; -- stage-owned checkpoint manifests only, with no root-level checkpoint summary; -- explicit resume behavior through `run --resume`, not implicit skipping during - ordinary `run`. - -## Stage 1: Add Workspace Configuration - -Add workspace configuration while preserving legacy diagnostics configuration -long enough for backward compatibility. - -Implement in `internal/core/config`: - -- Add `WorkspaceConfig` to `Config`. -- Add nested structs: - - `WorkspaceDiagnosticsConfig` - - `WorkspaceResumeConfig` - - `WorkspaceDebugConfig` -- Add file config parsing for: - - ```yaml - workspace: - directory: /var/lib/notarius - diagnostics: - enabled: true - retention: auto - resume: - enabled: false - debug: - enabled: false - ``` - -- Preserve existing `diagnostics.work_dir` and `diagnostics.retention` parsing as - deprecated compatibility input. -- Resolve effective diagnostics behavior as follows: - - if `workspace.directory` is set, diagnostics root is - `/diagnostics`; - - if `workspace.directory` is unset and legacy `diagnostics.work_dir` is set, - use legacy diagnostics behavior unchanged; - - if neither is set, preserve the current default diagnostics behavior for - migration compatibility; - - `workspace.diagnostics.retention` overrides legacy diagnostics retention - when set; - - legacy diagnostics retention remains accepted when workspace diagnostics - retention is unset. -- If `workspace.diagnostics.enabled` is explicitly false, do not create a - diagnostics run directory and do not write diagnostics artifacts. CLI failure - handling must still print concise errors to stderr without assuming a - diagnostics directory exists. -- Add environment support: - - `NOTARIUS_WORKSPACE_DIR` sets `workspace.directory`; - - `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED` parses a boolean; - - `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION` sets workspace diagnostics - retention; - - `NOTARIUS_WORKSPACE_RESUME_ENABLED` parses a boolean; - - `NOTARIUS_WORKSPACE_DEBUG_ENABLED` parses a boolean; - - keep `NOTARIUS_WORK_DIR` and `NOTARIUS_DIAGNOSTICS_RETENTION` as deprecated - compatibility overrides for legacy diagnostics config. -- Keep `--diagnostics-dir` as a backwards-compatible per-invocation diagnostics - root override. It should affect diagnostics only and should not change - checkpoint or debug roots. -- Update redaction/effective-config diagnostics so workspace config is included - and no secrets are introduced. - -Tests: - -- Add config default tests for workspace defaults. -- Add file config parsing tests for nested workspace fields. -- Add env override tests for all new env vars. -- Add precedence tests covering workspace config, legacy diagnostics config, - env overrides, and `--diagnostics-dir`. -- Update redacted/effective config tests. - -## Stage 2: Introduce Workspace Filesystem Helpers - -Create reusable filesystem helpers for workspace state. Keep concrete module -packages out of this layer. - -Implement a new package, recommended path `internal/core/workspace`, with: - -- `Config` or `Settings` describing effective workspace roots: - - root directory; - - diagnostics root; - - checkpoints root; - - debug root; - - enabled flags. -- path construction helpers for: - - diagnostics run directories; - - checkpoint identity directories; - - debug run directories. -- safe path helpers that reject absolute artifact names, `..`, backslashes, and - paths that escape their intended root. -- atomic JSON and byte-file writes using the existing diagnostics atomic-write - behavior as the model. -- optional shared internal helper for atomic file writes so diagnostics and - workspace writers do not duplicate low-level write logic. - -Do not add resume behavior in this stage. - -Tests: - -- Unit-test path construction and path-safety failures. -- Unit-test atomic JSON/byte writes. -- Unit-test disabled workspace settings returning no-op or empty roots as - appropriate. -- Confirm no helper permits writes outside the configured root. - -## Stage 3: Move Diagnostics Under Workspace - -Update diagnostics creation to use the effective diagnostics root from workspace -configuration when workspace is configured. - -Implementation requirements: - -- Keep diagnostics artifacts and retention behavior unchanged. -- Honor disabled diagnostics by using a no-op diagnostics writer or nil-safe - diagnostics path through the CLI failure and success paths. -- Preserve the existing `diagnostics.RunDirectory` contract unless a narrow - constructor addition is cleaner. -- Route normal workspace diagnostics to: - - ```text - /diagnostics// - ``` - -- Preserve legacy behavior when only legacy diagnostics config is present. -- Preserve `--diagnostics-dir` behavior as diagnostics-only override. -- Keep diagnostics run IDs run-based and non-deterministic. -- Do not write checkpoint or debug output in this stage. - -Docs to update after behavior exists: - -- `docs/config.md` -- `docs/operations.md` -- `docs/internal/diagnostics.md` -- `docs/cli.md` if CLI help text changes. - -Tests: - -- Update diagnostics run directory tests for workspace diagnostics roots. -- Update CLI tests that inspect diagnostics paths. -- Add migration tests proving legacy diagnostics config still works. -- Run focused checks: - - `go test ./internal/core/config` - - `go test ./internal/core/diagnostics` - - `go test ./internal/cli` - -## Stage 4: Define Checkpoint Identity And Stage Manifest Types - -Add checkpoint identity and manifest data structures before writing stage -payloads. - -Implement in `internal/core/workspace` or a closely related core package: - -- `CheckpointIdentity`, derived from: - - resolved pipeline ID; - - resolved pipeline digest; - - input adapter key; - - raw input digest or source digest; - - selected lanes; - - runtime overrides that affect execution; - - materialized reference digests; - - prompt/schema/profile provenance not already represented by the pipeline - digest. -- A deterministic, filesystem-safe checkpoint path: - - ```text - /checkpoints//-// - ``` - -- Stage manifest structs for: - - source; - - chunk; - - extract lane; - - merge lane; - - normalize lane. -- Shared manifest fields: - - workspace schema version; - - stage name; - - lane ID when applicable; - - module key; - - dependency fingerprints; - - status; - - output digests; - - validation status and rejection summaries where applicable; - - started/completed timestamps where useful. -- Status values: - - `pending`; - - `running`; - - `succeeded`; - - `succeeded_with_rejections`; - - `failed`; - - `invalidated`. - -Do not add a root-level checkpoint manifest. - -Tests: - -- Unit-test deterministic identity generation. -- Unit-test identity changes when pipeline digest, input digest, selected lanes, - or reference digests change. -- Unit-test manifest JSON round trips. -- Unit-test filesystem-safe path generation. - -## Stage 5: Write Checkpoints Without Resuming - -Add write-only checkpoint support behind `workspace.resume.enabled`. - -Implementation requirements: - -- Add a framework-owned checkpoint recorder to pipeline execution. Recommended - shape: - - CLI constructs the effective workspace/checkpoint recorder after pipeline - resolution and reference materialization. - - `pipeline.RunInput` receives a recorder interface or no-op recorder. - - concrete modules do not receive workspace paths and do not write directly to - workspace. -- Write checkpoint artifacts only when `workspace.resume.enabled` is true. -- Use stage-owned manifests and no root summary. -- Use JSON envelope types for checkpointed payloads that preserve byte content, - such as `content_base64`, media type, metadata, warnings, and content digest. - Do not rely on existing runtime structs whose byte fields are tagged - `json:"-"`. -- Recommended checkpoint files: - - ```text - checkpoints//source/manifest.json - checkpoints//source/source-document.json - checkpoints//chunk/manifest.json - checkpoints//chunk/chunks.json - checkpoints//extract//manifest.json - checkpoints//extract//outputs.json - checkpoints//merge//manifest.json - checkpoints//merge//output.json - checkpoints//normalize//manifest.json - checkpoints//normalize//output.json - ``` - -- Record rejected extract outputs as checkpointed stage outcomes. Under current - pipeline policy they do not pass downstream, but they are not framework errors. -- Mark a stage `running` before writing its payloads, then atomically replace the - manifest with a final success/failure status after payload writes complete. -- Ensure interrupted or partial writes cannot be mistaken for successful - checkpoints. -- Do not write raw prompts, raw references, raw LLM request payloads, or debug - traces in checkpoint output. - -Tests: - -- Add runner/CLI tests proving no checkpoint files are written when resume is - disabled. -- Add tests proving checkpoint files are written when resume is enabled. -- Add tests for successful runs, rejected extract outputs, failed stages, and - warning-only validation. -- Add tests proving checkpoint payload content digests match written content. -- Add tests proving modules do not receive filesystem paths. - -## Stage 6: Add Explicit Resume Reads - -Add explicit resume behavior after write-only checkpoints are stable. - -CLI behavior: - -- Add `run --resume`. -- `--resume` should require `workspace.resume.enabled: true`; otherwise return a - clear configuration error. -- Plain `run` should continue to execute stages normally and should not silently - skip completed stages. - -Resume behavior: - -- Load and validate checkpoint stage manifests in workflow order. -- Reuse a checkpoint only when: - - workspace schema version is supported; - - stage status is successful for the current purpose; - - dependency fingerprints match the current invocation; - - referenced checkpoint payload files exist; - - payload digests match manifest digests; - - selected lanes and runtime overrides are compatible. -- If a checkpoint is missing or invalid, execute that stage normally and write a - fresh checkpoint if resume checkpointing remains enabled. -- If a source or chunk checkpoint is reused, downstream dependency fingerprints - must still be validated before downstream reuse. -- If an extract checkpoint includes rejected outputs, preserve those rejection - records and continue to omit rejected outputs from merge input. -- If merge is reused, skip merge execution only for the matching lane. -- If normalize is reused, pass the reused normalized output to the output stage. -- Always create fresh diagnostics for the current invocation, including records - showing which stages were reused versus executed. - -Tests: - -- Add CLI tests for `--resume` without workspace resume enabled. -- Add tests for reusing source, chunk, extract, merge, and normalize - checkpoints. -- Add tests for invalidation when input, pipeline digest, references, selected - lanes, or runtime LLM profile override changes. -- Add tests for corrupt or missing checkpoint payloads. -- Add tests proving fresh diagnostics are written for resumed invocations. - -## Stage 7: Add Debug Output - -Add debug output behind `workspace.debug.enabled`. - -Implementation requirements: - -- Write debug artifacts under: - - ```text - /debug// - ``` - -- Debug is per-invocation and should not be used for resume. -- Debug and resume are independent: - - debug enabled does not imply checkpoint writing; - - resume enabled does not imply debug writing. -- Start with artifacts available from Notarius framework boundaries: - - source/checkpoint-like stage inputs and outputs; - - structured LLM request inputs and response content from Notarius contracts; - - validator requests and results; - - stage timing and attempt metadata. -- Do not depend on Scriptorium internals for rendered upstream provider - requests. If Scriptorium later exposes rendered prompt traces safely, add them - in a separate pass. -- Never write API keys, bearer tokens, or raw provider credentials. -- Document clearly that debug output may contain source material, references, - prompts, model outputs, and other sensitive content. - -Tests: - -- Add tests proving no debug files are written when debug is disabled. -- Add tests proving debug files are written under `debug//` when enabled. -- Add tests proving debug and resume can be enabled independently. -- Add tests proving obvious secrets are not written. -- Add diagnostics/debug path tests to ensure path roots do not overlap - accidentally. - -## Stage 8: Documentation, Examples, And Cleanup - -After behavior is implemented, update canonical current-behavior docs. - -Update: - -- `docs/config.md` -- `docs/cli.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/internal/diagnostics.md` -- `docs/internal/pipeline.md` -- maintained examples under `examples/` when practical. - -Documentation requirements: - -- Document workspace config and defaults. -- Document legacy diagnostics config compatibility and any deprecation language. -- Document `/var/lib/notarius` as the production recommendation. -- Document local development recommendations. -- Document checkpoint sensitivity and debug sensitivity. -- Document explicit resume behavior and invalidation rules. -- Keep future or deferred behavior only in roadmap files. - -Cleanup: - -- Update `docs/roadmap/workspace.md` to a completed-status note after the - implementation lands. -- Remove or replace this implementation plan with a completed note after the - implementation lands. - -Validation: - -```sh -go test ./internal/core/config -go test ./internal/core/diagnostics -go test ./internal/core/workspace -go test ./internal/framework/pipeline -go test ./internal/cli -go test ./... -go vet ./... -go build ./cmd/notarius -``` +# Workspace Implementation Status + +The workspace implementation described by this roadmap has landed. Current +behavior is documented in the canonical current-behavior docs: + +- [Configuration](../config.md) +- [CLI Reference](../cli.md) +- [Operations](../operations.md) +- [Troubleshooting](../troubleshooting.md) +- [Diagnostics Internals](../internal/diagnostics.md) +- [Pipeline Internals](../internal/pipeline.md) + +Implemented behavior includes: + +- `workspace.directory` as the root for Notarius-owned local state; +- workspace diagnostics under `/diagnostics//`; +- compatibility for legacy `diagnostics.work_dir`, `diagnostics.retention`, + `NOTARIUS_WORK_DIR`, and `NOTARIUS_DIAGNOSTICS_RETENTION`; +- checkpoint writes under `/checkpoints/` when resume + checkpointing is enabled; +- explicit checkpoint reuse through `notarius run --resume`; +- debug artifacts under `/debug//` when debug + output is enabled; +- independent resume and debug settings. + +Deferred workspace ideas remain in [Workspace Roadmap](workspace.md). diff --git a/docs/roadmap/workspace.md b/docs/roadmap/workspace.md index 8c29300..934e708 100644 --- a/docs/roadmap/workspace.md +++ b/docs/roadmap/workspace.md @@ -1,55 +1,11 @@ -# Workspace Roadmap +# Workspace Roadmap Status -Notarius should gain a single configurable workspace root for application-owned -local state. The workspace exists only to support enabled workspace features; -ordinary runs should not write workspace files by default. +The local workspace feature has been implemented. Current behavior is documented +in [Configuration](../config.md), [CLI Reference](../cli.md), +[Operations](../operations.md), and the relevant internal docs. -The workspace is distinct from durable output: - -- durable output is the user-requested final product of a run; -- the workspace is application-owned local state for diagnostics, resumable - checkpoints, and development/debug workflows. - -Within the workspace, diagnostics, checkpoints, and debug artifacts should use -separate subdirectories because they have different identity and lifecycle -models. - -## Target Configuration - -Workspace behavior should be configured under a dedicated top-level object: - -```yaml -workspace: - directory: /var/lib/notarius - diagnostics: - enabled: true - retention: auto - resume: - enabled: false - debug: - enabled: false -``` - -Policy: - -- `workspace.diagnostics.enabled` defaults to the current diagnostics behavior - during migration. -- `workspace.resume.enabled` defaults to `false`. -- `workspace.debug.enabled` defaults to `false`. -- no checkpoint or debug files are written when those features are disabled. -- `/var/lib/notarius` should be the standard production recommendation in docs - and examples. -- local development docs may recommend a project-local path such as - `./.notarius/workspace`. - -`workspace.directory` should become the single configured root for Notarius local -state. Existing `diagnostics.work_dir` behavior should be migrated carefully for -backward compatibility, but the long-term configuration model should avoid two -separate roots for application-owned local state. - -## Workspace Layout - -The workspace root should contain separate subtrees: +The implemented workspace provides one configurable root for Notarius-owned +local state: ```text / @@ -58,176 +14,13 @@ The workspace root should contain separate subtrees: debug/ ``` -Diagnostics remain run-id based. They are invocation history and should preserve -the existing diagnostics retention model: - -```text -/ - diagnostics/ - / - invocation.json - effective-config.json - resolved-pipeline.json - resolved-references.json - run-manifest.json - warnings.json - run-report.json - error.log -``` - -Checkpoints should be deterministic and should include enough identity material -to prevent unsafe reuse across incompatible runs: - -```text -/ - checkpoints/ - / - -/ - / - source/ - chunk/ - extract// - merge// - normalize// -``` - -Checkpoint identity should account for: - -- resolved pipeline ID; -- resolved pipeline digest; -- input adapter key; -- source/input digest; -- selected lanes and runtime overrides that affect execution; -- reference digests; -- prompt, schema, and profile provenance when not already captured by the - resolved pipeline digest. - -Debug output should be run-id based like diagnostics, because debug traces are -development artifacts from a specific invocation rather than resumable state: - -```text -/ - debug/ - / -``` - -Human-readable path segments are useful, but content hashes should be -authoritative for correctness. The exact path shape may change during -implementation, but checkpoint identity must be stable, deterministic, and safe -to compare across runs. Diagnostics and debug output do not need deterministic -resume identity because they are per-invocation artifacts. - -## Stage Ownership - -Stage manifests should be stage-owned. There should not be a root-level manifest -that summarizes all stages. - -Rationale: - -- stage manifests are the authoritative state for the stage that wrote them; -- avoiding a root summary prevents duplicate state from drifting; -- stage-local manifests make partial writes, failure recovery, and future stage - invalidation easier to reason about. - -Each stage manifest should record enough information to determine whether its -outputs can be trusted for resume: - -- workspace schema/version; -- stage name and lane ID when applicable; -- module key and relevant module provenance; -- dependency fingerprints and input digests; -- status such as pending, running, succeeded, succeeded with rejections, failed, - or invalidated; -- output content digests; -- validation status and rejection summaries where relevant; -- timing metadata when useful and non-sensitive. - -## Checkpoint Output - -When `workspace.resume.enabled` is enabled, Notarius should write under the -`checkpoints/` subtree. Checkpoints should contain only artifacts required to -resume safely. - -Checkpoint artifacts may include: - -- source document checkpoint; -- chunk collection checkpoint; -- accepted extract outputs; -- rejected extract records; -- merge output; -- normalize output; -- stage manifests with dependency fingerprints and output digests. - -Checkpoint output should avoid raw prompts, raw references, raw LLM request -payloads, and other sensitive development artifacts unless they are strictly -required for safe resume. Prefer digests and provenance over copying sensitive -inputs. - -Checkpoint writes should be atomic at the file level. Partial or interrupted -writes must not be mistaken for successful stage completion. - -## Debug Output - -When `workspace.debug.enabled` is enabled, Notarius should write under the -`debug//` subtree. Debug artifacts are useful for inspection but not -required for resume. - -Debug artifacts may include: - -- rendered prompt inputs; -- structured LLM request and response traces; -- raw model responses; -- copied reference content; -- copied source snippets; -- intermediate raw payloads; -- validator request/response details. - -Debug output is explicitly sensitive. Documentation should warn that debug -workspace data may contain source material, references, prompts, and model -outputs. Debug output should remain disabled by default. - -`workspace.debug.enabled` and `workspace.resume.enabled` should be independent. -Enabling debug should not imply resume, and enabling resume should not imply -debug. - -## Resume Semantics - -Resume should be explicit at first, for example through a future `resume` command -or a `run --resume` flag. Plain `run` should not silently skip completed stages -in the initial workspace feature. - -Before reusing a checkpoint, Notarius should verify that: - -- the workspace schema/version is supported; -- the checkpoint stage manifest is complete and successful; -- dependency fingerprints match the current invocation; -- referenced checkpoint files exist and match recorded digests; -- selected lanes and runtime overrides are compatible with the checkpoint. - -Rejected extract outputs should be treated as recorded stage outcomes, not -framework failures. Under current pipeline policy, rejected outputs do not pass -to downstream stages. - -## Boundaries - -Workspace writing should be framework-owned. Concrete modules should not write -directly to workspace paths. - -If modules need to expose debug material later, they should return logical -artifacts through framework contracts and let framework or CLI code perform path -validation, redaction policy, and writes. - -Workspace state should preserve existing Notarius boundaries: - -- provider plumbing remains behind LLM runtime contracts; -- prompt ownership remains with modules and prompt asset helpers; -- diagnostics remain per-run invocation artifacts under the workspace root; -- output modules continue to own final durable output encoding; -- secrets and sensitive payloads are not written unless an explicit debug policy - enables them. +Implemented behavior includes workspace-backed diagnostics, checkpoint writing, +explicit checkpoint reuse through `notarius run --resume`, workspace debug +artifacts, safe workspace-relative writes, and compatibility for legacy +diagnostics configuration. ## Deferred Work -Default-idempotent `run` behavior with a `--force` override, remote workspace +Default-idempotent `run` behavior with a force override, remote workspace storage, workspace garbage collection, archival policy, and cross-machine resume -are deferred. +remain deferred. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d90f596..cce7326 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -280,6 +280,50 @@ Fix: - Pass `--session-id ` to `notarius run`. - Use a stable, non-secret identifier from the external orchestrator. +## Resume Or Checkpoint Reuse Failure + +Symptoms include: + +- `--resume requires workspace.resume.enabled: true` +- `checkpoint artifact is missing` +- `checkpoint workspace schema version` +- `checkpoint dependency fingerprints do not match` +- a resumed run executes work instead of reusing a checkpoint + +Fix: + +- Set both `workspace.directory` and `workspace.resume.enabled: true`. +- Use `--resume`; plain `notarius run` executes normally and refreshes + checkpoints. +- Confirm the current run uses the same input bytes, resolved pipeline, selected + lanes, runtime LLM profile override, and materialized references as the run + that wrote the checkpoint. +- Inspect retained diagnostics `checkpoint-events.json` to see which workflow + steps were reused or executed. +- If a checkpoint payload is missing or corrupt, rerun without relying on that + checkpoint. Notarius executes invalidated steps normally and writes fresh + checkpoints when checkpointing remains enabled. + +Checkpoint files can contain source text, intermediate outputs, rejected +outputs, metadata, and warnings. Protect the workspace directory accordingly. + +## Debug Output Missing Or Too Verbose + +Symptoms: + +- no files appear under `/debug//`; +- debug files contain more source, reference, prompt, or model-output material + than expected. + +Fix: + +- Set both `workspace.directory` and `workspace.debug.enabled: true`. +- Confirm you are inspecting the current run ID. Debug output is per invocation + and is not used for resume. +- Disable `workspace.debug.enabled` after the inspection run. Debug output may + contain sensitive source material, reference material, prompt inputs, model + outputs, and validation payloads. + ## Output Write Failure Symptoms include: diff --git a/examples/dnd-spells.config.yml b/examples/dnd-spells.config.yml index 3531394..ff1aee9 100644 --- a/examples/dnd-spells.config.yml +++ b/examples/dnd-spells.config.yml @@ -1,4 +1,19 @@ version: 2 +# For production runs, use a writable application-owned workspace such as: +# +# workspace: +# directory: /var/lib/notarius +# diagnostics: +# retention: auto +# resume: +# enabled: false +# debug: +# enabled: false +# +# For local development, use a project-local ignored path such as: +# +# workspace: +# directory: ./.notarius/workspace pipelines: dnd-session: input: seriatim