eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update the debug workflow to provide raw LLM output

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-07 23:08:14 -05:00
parent ae65b95374
commit 3011dd91ca
10 changed files with 476 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/internal/pipeline.md
View File

@@ -88,9 +88,11 @@ fall back to normal execution and are refreshed by the recorder.
When workspace debug output is enabled, the CLI passes a debug recorder for the
current run ID. The runner writes framework-boundary inputs, outputs,
structured LLM calls, validator calls, timing, and retry attempt metadata
through that interface. Debug output is not used for resume and can contain
sensitive source, reference, prompt, and model-output material. Concrete modules
still do not receive workspace paths.
through that interface. Each retry or validator attempt records any LLM calls
made within that attempt in an `llm_calls` array and writes scoped copies of the
canonical LLM call artifacts under the attempt directory. Debug output is not
used for resume and can contain sensitive source, reference, prompt, and
model-output material. Concrete modules still do not receive workspace paths.
## Registries And Module Specs

18
docs/operations.md
View File

@@ -96,7 +96,7 @@ When `workspace.resume.enabled: true` and `workspace.directory` is set, runs
write checkpoints under:
```text
<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-digest>/<pipeline-digest>/
<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/
```
Each workflow step owns its own manifest and payload files. There is no
@@ -114,8 +114,9 @@ sensitive local state.
A checkpoint is reused only when its workspace schema version, checkpoint
identity digest, step status, dependency fingerprints, payload files, and
payload digests match the current invocation. Changes to input bytes, resolved
pipeline digest, selected lanes, runtime LLM profile override, or materialized
reference digests invalidate reuse.
pipeline digest, selected lanes, runtime LLM profile override, materialized
reference digests, or other identity material invalidate reuse and use a
separate checkpoint directory.
Plain `notarius run` does not reuse checkpoints. It executes the workflow and
refreshes checkpoint files when checkpointing is enabled. `notarius run
@@ -137,9 +138,12 @@ checkpointing does not write debug output.
Debug artifacts include framework-boundary inputs and outputs for source,
chunk, extract, merge, normalize, and output work, structured LLM request and
response data from Notarius contracts, validator requests and results, timing,
and retry attempt metadata. Debug artifacts may contain source material,
reference material, prompt inputs, model outputs, and other sensitive data.
Obvious credential-shaped values and sensitive map keys are redacted, but debug
and retry attempt metadata. Canonical LLM call artifacts are written under
`llm/call-000N.json`; LLM calls made inside a retry or validator attempt are
also copied under that attempt directory and linked from the attempt
`llm_calls` array. Debug artifacts may contain source material, reference
material, prompt inputs, model outputs, and other sensitive data. Obvious
credential-shaped values and sensitive map keys are redacted, but debug
directories should still be protected as sensitive local state.
## Retention
@@ -195,7 +199,7 @@ Workspace checkpoint and debug directories can also be removed when no longer
needed. Remove exact identity or run directories, for example:
```sh
rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234
rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234
rm -rf /var/lib/notarius/debug/run-1234567890
```