Document output cache and debug state model
This commit is contained in:
@@ -1,105 +0,0 @@
|
||||
# Diagnostics Internals
|
||||
|
||||
`internal/core/diagnostics` provides the scoped writer and retention decision
|
||||
used by `internal/cli`. The physical layout, artifact inventory, retention
|
||||
semantics, failure inspection, and cleanup procedures are canonical in
|
||||
[Operations](../operations.md#diagnostics-directory). Configuration fields and
|
||||
defaults are canonical in [Configuration](../config.md#diagnostics).
|
||||
|
||||
## Run Directory
|
||||
|
||||
`NewRunDirectory` normalizes empty constructor inputs, creates the effective
|
||||
diagnostics root when needed, and allocates a unique timestamp-based child
|
||||
directory. It retries a bounded number of collisions before failing. The
|
||||
resulting `RunDirectory` retains its creation time and retention mode for later
|
||||
metadata and cleanup decisions.
|
||||
|
||||
The package does not resolve workspace configuration. `internal/cli` derives
|
||||
effective workspace settings first and passes the diagnostics root into the
|
||||
constructor.
|
||||
|
||||
## Scoped Writers
|
||||
|
||||
Typed methods on `RunDirectory` write invocation metadata, redacted effective
|
||||
configuration, resolved pipeline/reference data, checkpoint events, source data
|
||||
when explicitly requested, manifests, reports, warnings, redacted chunk-plan
|
||||
summaries, and error text. The
|
||||
current filenames and their operator-facing contents are listed in
|
||||
[Operations](../operations.md#diagnostics-directory).
|
||||
|
||||
The chunk-plan summary records the effective mode, source and candidate
|
||||
digests, requested module, lookup decision, materialization action, validation
|
||||
decision, and publication decision. Its closed decision values make failures
|
||||
and recoverable invalid records inspectable without serializing plan ranges,
|
||||
annotations, source content, reference content, prompts, model responses, or
|
||||
raw invalid-file bytes. Lookup reasons are derived only from the lookup status:
|
||||
`stored chunk plan is valid`, `chunk plan not found`, `stored chunk plan is
|
||||
invalid`, or `chunk plan lookup skipped`. Store-provided reasons and malformed
|
||||
record details never enter this artifact.
|
||||
|
||||
JSON methods indent their payload and append a newline. All artifact writes use
|
||||
a temporary file in the target directory, apply the requested permissions, and
|
||||
rename it into place. Artifact resolution accepts only a single relative base
|
||||
name; absolute paths, separators, and paths escaping the run directory fail
|
||||
before writing.
|
||||
|
||||
## Redacted Configuration
|
||||
|
||||
`WriteRedactedEffectiveConfig` accepts a `RedactedDiagnosticsPayload` provider
|
||||
rather than a raw config value. `internal/core/config` implements that contract
|
||||
by cloning effective config data and removing secret-shaped values before JSON
|
||||
encoding. The diagnostics package therefore never needs configuration-specific
|
||||
field knowledge.
|
||||
|
||||
## Retention Coordination
|
||||
|
||||
`ShouldRetainRunDirectory` is a pure decision over the effective retention mode,
|
||||
run success, and warning presence. `ApplyRetention` uses that result to remove
|
||||
only its own run directory. Unsupported modes retain data as a fail-safe, though
|
||||
normal CLI execution rejects them during config validation.
|
||||
|
||||
The meaning of each supported mode belongs in
|
||||
[Operations](../operations.md#retention); this package implements that contract
|
||||
without loading config or inspecting run artifacts.
|
||||
|
||||
## CLI State Flow
|
||||
|
||||
When diagnostics are enabled, `internal/cli` creates the run directory after
|
||||
configuration loading and before pipeline resolution. It then writes artifacts
|
||||
as state becomes available: invocation data, effective resolution data,
|
||||
pipeline results, and the final report. This ordering permits later failures to
|
||||
retain the context already established.
|
||||
|
||||
Failures before construction have no `RunDirectory`. Later failures write an
|
||||
error log, preserve any available partial manifest and chunk-plan summary, and
|
||||
apply a failed-run
|
||||
retention decision. A diagnostics write failure is itself a command failure so
|
||||
the CLI does not report success after losing requested inspection data.
|
||||
|
||||
When diagnostics are disabled, the CLI carries a nil run directory and the
|
||||
shared `writeDiagnostics` helper turns writes into no-ops. User-facing errors
|
||||
still go to stderr; that invocation behavior is documented in
|
||||
[Operations](../operations.md#failures).
|
||||
|
||||
## Package Guarantees
|
||||
|
||||
- A `RunDirectory` writes and removes only within its allocated directory.
|
||||
- JSON and error artifacts use atomic replacement.
|
||||
- Nil receivers and invalid typed payloads return errors rather than panicking.
|
||||
- Retention never removes a failed run and never targets the diagnostics root.
|
||||
- Diagnostics models contain inspection metadata, not the durable output
|
||||
contract.
|
||||
- Checkpoint and debug serializers remain separate framework components.
|
||||
- Secret-handling follows the invariant in
|
||||
[Architecture](../policy/architecture.md#state-output-and-safety).
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/core/diagnostics/run_dir_test.go`: allocation, artifact confinement,
|
||||
atomic writes, retention, and failure behavior.
|
||||
- `internal/core/diagnostics/artifacts_test.go`: stable artifact identifiers.
|
||||
- `internal/core/config/redaction_test.go`: clone-and-redact payload behavior.
|
||||
- `internal/core/workspace/settings_test.go`: effective diagnostics-root and
|
||||
enablement handoff.
|
||||
- `internal/cli/run_test.go`: creation timing, artifact sequencing, disabled
|
||||
diagnostics, overrides, failures, and retention integration.
|
||||
@@ -106,14 +106,13 @@ schemas remain package-owned.
|
||||
## Debug And Redaction Boundaries
|
||||
|
||||
The pipeline may wrap the client with a debug recorder that captures prepared
|
||||
prompt/response material for an explicitly enabled debug run. Default
|
||||
diagnostics and manifests receive identities, hashes, usage, and selected
|
||||
profile summaries rather than prompt, source, reference, schema, or response
|
||||
content.
|
||||
prompt/response material for an explicitly requested debug run. Debug summaries
|
||||
and manifests receive identities, hashes, usage, and selected profile summaries
|
||||
rather than prompt, source, reference, schema, or response content.
|
||||
|
||||
The Scriptorium error wrapper removes bearer credential values from surfaced
|
||||
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
|
||||
secret values elsewhere in the runtime. Config diagnostics use a separate
|
||||
secret values elsewhere in the runtime. Config summaries use a separate
|
||||
clone-and-redact path in `internal/core/config`. These mechanisms implement the
|
||||
security invariant in
|
||||
[Architecture](../policy/architecture.md#state-output-and-safety); operator
|
||||
|
||||
@@ -11,8 +11,8 @@ boundaries and dependency direction belong in
|
||||
`cmd/notarius` delegates to `internal/cli`, the production composition root.
|
||||
The CLI loads configuration, builds the production catalogs and runtime
|
||||
collaborators, invokes `internal/framework/pipeline`, and places the logical
|
||||
output files returned by the runner. Diagnostics, checkpoints, and debug
|
||||
recorders are optional side-channel collaborators supplied at this boundary.
|
||||
output files returned by the runner. Cache and debug collaborators are supplied
|
||||
at this boundary.
|
||||
|
||||
Resolution produces a fixed ordered workflow and a sorted set of artifact
|
||||
lanes. Preparation constructs the complete module and validator set before the
|
||||
@@ -25,7 +25,7 @@ normalize continuations that may overlap across lanes.
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
||||
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. |
|
||||
|
||||
## Core Packages
|
||||
|
||||
@@ -33,9 +33,9 @@ normalize continuations that may overlap across lanes.
|
||||
| --- | --- |
|
||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||
| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. |
|
||||
| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. |
|
||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |
|
||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
|
||||
|
||||
## Framework Packages
|
||||
|
||||
@@ -46,9 +46,9 @@ normalize continuations that may overlap across lanes.
|
||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
||||
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
|
||||
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
|
||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
||||
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
|
||||
|
||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||
serialized-validator, and
|
||||
@@ -122,14 +122,13 @@ Implementation details for all production extensions are in
|
||||
| Surface | Implemented owners | Internal purpose |
|
||||
| --- | --- | --- |
|
||||
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
||||
| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
|
||||
| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable extract, merge, and normalize outcomes. |
|
||||
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. |
|
||||
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
|
||||
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
|
||||
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
|
||||
|
||||
Physical layout, retention, recovery, and sensitive-data handling are defined
|
||||
Physical layout, cleanup, recovery, and sensitive-data handling are defined
|
||||
in [Operations](../operations.md). Concrete stage modules receive recorder
|
||||
interfaces and request data, not workspace paths.
|
||||
interfaces and request data, not physical state roots.
|
||||
|
||||
## Focused Documentation
|
||||
|
||||
@@ -139,5 +138,5 @@ interfaces and request data, not workspace paths.
|
||||
registration, and the contributor recipe for adding an extension.
|
||||
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
|
||||
assets, scheduling, profile recording, and redaction.
|
||||
- [Diagnostics Internals](diagnostics.md): scoped writers, retention
|
||||
coordination, CLI failure flow, and path safety.
|
||||
- [Run State Internals](state.md): output, cache, debug collaborator
|
||||
composition, and path safety.
|
||||
|
||||
@@ -143,8 +143,8 @@ reference spanning the first selected unit through the last.
|
||||
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
|
||||
artifacts with lane and normalizer provenance,
|
||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
||||
output encoder. The CLI owns diagnostics and durable filesystem writes after the
|
||||
runner returns.
|
||||
output encoder. The CLI owns debug-summary and durable filesystem writes after
|
||||
the runner returns.
|
||||
|
||||
## Execution Flow
|
||||
|
||||
@@ -199,7 +199,7 @@ configuration, requested chunker, options, references, lanes, validators, or
|
||||
LLM profile do not prevent a source-digest hit. The manifest records both the
|
||||
currently requested chunker and the effective plan producer. Cache state and
|
||||
paths are configured and operated outside the runner; see
|
||||
[Configuration](../config.md#workspace) and [Operations](../operations.md).
|
||||
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
|
||||
56
docs/internal/state.md
Normal file
56
docs/internal/state.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Run State Internals
|
||||
|
||||
This document describes the implementation collaborators behind output, cache,
|
||||
and debug state. User-visible fields belong in [Configuration](../config.md),
|
||||
and layouts and lifecycle belong in [Operations](../operations.md).
|
||||
|
||||
## Composition
|
||||
|
||||
`internal/cli` is the only physical-path composition root. It resolves the
|
||||
effective configuration, selects exact roots, allocates requested debug bundles,
|
||||
constructs cache collaborators, writes logical output files, and reports paths.
|
||||
Pipeline modules receive interfaces and request data, never output, cache, or
|
||||
debug roots.
|
||||
|
||||
## Output And Cache
|
||||
|
||||
The pipeline runner returns logical output files. The CLI places them beneath
|
||||
the selected output root with confined, atomic writes.
|
||||
|
||||
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
|
||||
and atomic publication. Its store is constructed only when the selected mode is
|
||||
not `bypass`.
|
||||
|
||||
`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
|
||||
codecs, loader, and recorder. The CLI constructs both loader and recorder only
|
||||
for a `--resume` invocation. The serialized
|
||||
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
|
||||
they do not describe a current public state surface.
|
||||
|
||||
`internal/core/fileio` provides confined atomic file writes used by state
|
||||
collaborators. The chunk-plan store retains its stronger entry validation.
|
||||
|
||||
## Debug Bundles
|
||||
|
||||
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
||||
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
|
||||
resolution, run, warning, and failure artifacts. `internal/framework/debug`
|
||||
implements the pipeline-facing trace recorder under the trace root.
|
||||
|
||||
The CLI allocates a bundle before pipeline resolution and treats requested
|
||||
summary or trace persistence failures as command failures. The pipeline's debug
|
||||
boundaries redact sensitive metadata and credential-shaped bytes while allowing
|
||||
application-owned trace material. Debug data is never a checkpoint source or
|
||||
cache input.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/cli/state_surfaces_test.go`: debug allocation and configuration
|
||||
boundaries.
|
||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||
permissions, cleanup, and redaction.
|
||||
- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes.
|
||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and
|
||||
reuse.
|
||||
- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement,
|
||||
publication, and permissions.
|
||||
Reference in New Issue
Block a user