eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Strengthen validation retries and checkpoint safety

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-27 11:50:37 +00:00
parent b2e83bd6e7
commit e6b2ae88d2
73 changed files with 934 additions and 1560 deletions
Download Patch File Download Diff File Expand all files Collapse all files

731
docs/roadmap/implementation.md
View File

@@ -1,731 +0,0 @@
# Feedback-Aware Validation Retry Implementation Plan
## Status
Ready for implementation. This plan implements the target state defined by
[Feedback-Aware Stage Validation Retries](validation-retries.md). Its stages
are ordered dependencies, and each stage is intentionally scoped for one
gpt-5.6-terra implementation prompt.
The feature roadmap owns product intent and durable policy. This document owns
implementation order, concrete boundaries, and stage-level verification. If a
conflict is discovered, preserve the feature roadmap and the repository
policies, stop the affected stage, and revise this plan rather than silently
choosing a different architecture.
## Settled Decisions
The implementation agent must treat these decisions as fixed:
- PromptKit v0.9.0 is the minimum and exact supported PromptKit release for
this work. Notarius uses `RunRequest.AppendedMessages`; it does not create
paired correction manifests or bypass PromptKit preparation and execution.
- A correction request reconstructs the ordinary initial request and appends
exactly two messages: the latest exact producer response as `assistant`,
followed by one deterministic aggregate correction request as `user`.
Earlier correction turns never accumulate.
- A correction-capable producer supplies the exact single LLM response that
directly controlled the candidate. The initial protocol does not synthesize
a model-facing projection of a compound artifact.
- The only supported producer correction protocol is
`single_response_v1`. An empty protocol means unsupported. Configuration
rejects semantic-retry workflows for LLM-backed compound producers or any
other producer unable to satisfy `single_response_v1`.
- The existing producer binding `retries` value is the sole outer stage
budget. PromptKit repair attempts and validator execution retries are
independent budgets.
- Terminal policy is configured as pipeline defaults with field-by-field
producer-binding overrides. Validators never own candidate disposition.
- Application defaults are `fail_run` for exhausted producer structural
failure, `fail_run` for exhausted semantic rejection, and `warn_continue`
for exhausted validator execution failure.
- All applicable validators run sequentially in configured order. Rejections
and validator failures are aggregated; failures are not represented to the
producer as candidate defects.
- Deterministic producers do not consume retries after rejection. An
LLM-backed attempt that took a deterministic fast path and produced no model
response is likewise not correctable for that candidate.
- Known rejected or structurally invalid output never advances. A candidate
accepted under `validator_failure: warn_continue` advances with explicit
incomplete-validation provenance but is not checkpointed.
- The D&D combat-scene validator and broader warning-system redesign remain
out of scope.
## Cross-Stage Implementation Rules
Apply these rules in every stage:
1. Read the feature roadmap, `docs/development.md`, all `docs/policy/`
documents, and the focused current documentation named by the stage before
editing.
2. Inspect current code and tests rather than assuming paths or private helper
layouts. Preserve unrelated work and existing public ordering,
cancellation, scheduler, checkpoint, and sensitive-data invariants.
3. Use transport-neutral types outside `internal/framework/llm`. PromptKit
types must not escape the adapter boundary.
4. Keep correction material attempt-local. Do not put raw assistant responses
or correction text in ordinary errors, warnings, manifests, receipts,
checkpoints, caches, or default debug summaries.
5. Add the smallest durable tests that protect the stage's contracts. Do not
duplicate PromptKit's own prepared-execution, copying, hashing, capacity,
role-validation, or structural-repair test matrix.
6. Use `gofmt` on changed Go files. Run the focused checks listed for the
stage and fix failures before stopping. Do not proceed into the next stage
in the same prompt.
7. Do not delete this plan or the feature roadmap during implementation. They
are retired only after a final review confirms the complete target state.
## Contract Names And Bounds
Use one vocabulary consistently across packages. Exact private helper names
may follow local conventions, but the following contracts and values are not
open for redesign:
- `CorrectionProtocol` has only the empty unsupported value and
`single_response_v1`.
- An application-owned `SemanticCorrection` carries owned
`AssistantResponse []byte` and `UserGuidance string` values. It is exposed
on chunk, typed extract, typed merge, typed normalize, and structured
completion requests as an optional pointer.
- An application-owned `ModelCandidate` carries the owned exact response bytes
and `CorrectionProtocol`. It is exposed on the corresponding producer
results as an optional pointer and is never serialized as part of a durable
artifact.
- `ValidationResult` gains optional `CorrectionGuidance`. Operator-facing
`Message` is not copied into this field implicitly.
- Each reason code is valid UTF-8, nonblank after trimming, and at most 128
bytes. Each validator-supplied correction guidance value is valid UTF-8,
nonblank after trimming, and at most 4 KiB.
- The exact defective assistant response may be at most 1 MiB. The aggregate
user correction message may be at most 64 KiB. The combined appended
content may therefore be at most 1,114,112 bytes. Reject invalid UTF-8,
empty/whitespace-only content, or over-limit content; never truncate it.
- Invalid producer-owned correction material is a framework contract error.
It is not a validator failure and cannot be converted to `reject_output` or
`warn_continue`.
- Aggregate correction entries are ordered by validator order, contain the
stable reason code plus guidance, and collapse only byte-identical duplicate
`(reason_code, correction_guidance)` pairs while retaining first occurrence.
A rejection without guidance receives bounded generic guidance based only on
its reason code.
- Terminal actions are closed enums: structural failure and semantic
rejection accept `fail_run` or `reject_output`; validator failure accepts
`warn_continue` or `fail_run`.
## Stage 1 ✅ — Adopt PromptKit v0.9.0
### Goal
Upgrade the dependency without changing Notarius stage behavior, prove the
existing integration remains compatible, and make the new upstream request
primitive available to later stages.
### Work
- Update `go.mod` and `go.sum` to PromptKit v0.9.0 with `go get` and
`go mod tidy`. Accept the catalog modules selected transitively by PromptKit;
do not import or register them directly.
- Review every Notarius `promptkit.RunRequest` literal and retain keyed form.
Verify production prompt roles are limited to `system` and `user`.
- Update the conservative built-in-profile checkpoint marker in
`internal/framework/llm/promptkit_profile_fingerprint.go` from v0.8.0 to
v0.9.0. Do not duplicate the separately versioned catalog module versions.
- Update `docs/integrations/pkg-promptkit.md` only to describe the newly
implemented dependency version and unchanged current integration boundary.
Include the strict role vocabulary and external catalog ownership, but do
not document Notarius correction behavior yet.
### Verification
- Run `go test ./internal/framework/llm ./internal/cli`.
- Run `go test ./...`, `go test -race ./internal/framework/llm`, `go vet ./...`,
and `go build ./cmd/notarius`.
### Exit Criteria
Notarius builds and passes its suite on PromptKit v0.9.0, its compatibility
document and checkpoint marker name v0.9.0, and no Notarius package directly
depends on either upstream catalog module. This stage is one Terra prompt.
## Stage 2 ✅ — Record The Architecture And Add Transport-Neutral Contracts
### Goal
Record the durable decision and introduce owned correction/candidate types
without changing runtime retry behavior.
### Work
- Add ADR-0014 under `docs/adr/` using the repository ADR format. Record the
three distinct retry budgets, complete validator aggregation, the fresh
two-message correction protocol, exact-response producer capability,
non-recursive validator failure handling, terminal-policy ownership and
defaults, checkpoint conservatism, and sensitive-data constraints.
- Add `CorrectionProtocol`, `SemanticCorrection`, and `ModelCandidate` to the
domain-neutral framework contracts, with constructors/clone helpers that
validate the settled bounds and defensively copy bytes.
- Add optional correction input to `ChunkRequest`,
`TypedExtractionRequest`, `TypedMergeRequest`, and
`TypedNormalizeRequest`.
- Add optional model-candidate output to `ChunkPlanResult`,
`TypedExtractionResult`, `TypedMergeResult`, and
`TypedNormalizeResult`.
- Add `CorrectionGuidance` to `ValidationResult` and centralize validation of
reason codes and corrective guidance at the framework boundary.
- Update type-erasure adapters, stored fakes, and clone paths so the new values
retain ownership and cannot alias caller buffers. Do not declare production
module capabilities or use the fields in the runner yet.
### Verification
- Add contract-level tests for accepted values, UTF-8 and size rejection,
defensive copying, nil behavior, and typed-erasure preservation.
- Run `go test ./internal/framework/contracts ./internal/framework/pipeline`.
- Run `go test ./...`.
### Exit Criteria
The generic contracts can safely carry correction input and exact candidate
material across every producer stage, no PromptKit type crosses the boundary,
and existing runtime behavior is unchanged. This stage is one Terra prompt.
## Stage 3 ✅ — Add Validation Policy Configuration And Resolution
### Goal
Implement strict configuration, inheritance, identity, and provenance for
terminal validation policy before the runner consumes it.
### Work
- Add optional `validation_policy` to a pipeline profile and to chunk,
extract, merge, and normalize producer bindings. Use pointer-backed override
fields so omission is distinguishable from an explicit value.
- Reject an explicitly null policy object, null policy fields, duplicate or
unknown fields, and values outside the settled enums. Keep the current file
configuration version.
- Reject `validation_policy` on input, output, and validator bindings. Reject
a binding-level explicit `producer_structural_failure` value on a
deterministic producer; a pipeline-level structural default remains valid
because a pipeline may contain LLM-backed producers.
- Resolve each policy field independently in binding, pipeline, application
default order. Store one detached concrete effective policy for chunk and
for each lane's extract, merge, and normalize stage; do not leave runtime
inheritance to the runner.
- Include both configured overrides and effective policies at their existing
appropriate configuration/provenance boundaries. Include the effective
values in resolved-pipeline digest and checkpoint identity. Preserve
deterministic serialization and redaction.
- Update `examples/dnd-minimal/config.yml` and
`examples/dnd-complete/config.yml` only if their current shape must change to
remain valid; do not add redundant explicit defaults merely to demonstrate
the feature.
### Verification
- Add focused parsing tests for omission versus null, unknown/duplicate
fields, invalid placements, and invalid enums.
- Add resolution tests for field-by-field inheritance and mixed overrides.
- Add digest, clone, JSON/YAML round-trip, and redacted-effective-config tests.
- Run `go test ./internal/core/config ./internal/framework/pipeline ./internal/cli`.
### Exit Criteria
Every producer stage receives one immutable effective policy, invalid
configuration fails before execution, and policy changes invalidate resolved
identity. The runner still follows its old behavior until later stages. This
stage is one Terra prompt.
## Stage 4 ✅ — Declare And Validate Producer Correction Capability
### Goal
Make correction support an explicit module contract and prepare the framework
to reject impossible workflows.
### Work
- Extend `pipeline.ModuleSpec` with `CorrectionProtocol`; normalize, clone,
validate, and include it in relevant registry/spec fingerprints.
- Permit `single_response_v1` only for LLM-backed chunk, extract, merge, or
normalize modules. Reject it for deterministic, input, output, or validator
specs. Empty remains the default unsupported value.
- Carry the selected protocol into resolved and prepared producer metadata and
checkpoint fingerprints.
- Reject positive `retries` on deterministic validator bindings during
resolution. LLM-backed validator retries remain valid and independent of
producer capability.
### Verification
- Add spec normalization/clone tests, invalid stage/class combinations,
validator retry-class tests, and preparation tests for supported and
unsupported producer workflows.
- Run `go test ./internal/framework/pipeline ./internal/core/config`.
### Exit Criteria
Capability is explicit, fingerprinted, and testable, while existing production
pipelines remain executable pending their migrations. No dormant feature gate
or unused preparation check is introduced. This stage is one Terra prompt.
## Stage 5 — Adapt Corrections Through PromptKit ✅
### Goal
Map the transport-neutral correction contract onto PromptKit v0.9.0 without
changing ordinary requests.
### Work
- Add optional `SemanticCorrection` to `StructuredCompletionRequest` and its
debug-safe request representation.
- In `PromptKitClient.CompleteStructured`, validate and defensively copy the
correction, then map it to exactly two `promptkit.RenderedMessage` values in
`RunRequest.AppendedMessages`, using `promptkit.RoleAssistant` followed by
`promptkit.RoleUser`.
- Leave `AppendedMessages` nil for an ordinary request. Do not allow callers to
choose other roles through the Notarius contract.
- Ensure request formatting and default debug summaries expose only safe
counts/digests. The existing explicitly requested detailed PromptKit trace
may contain complete effective messages and must retain its existing
sensitive-data treatment.
- Preserve the same prompt identity, inputs, variables, session, profile,
execution overrides, repair attempts, scheduler, and prepared-execution
path for corrected calls.
### Verification
- Add focused adapter tests proving an ordinary request is unchanged and a
corrected request has the ordinary rendered prefix plus exactly the two
supplied messages in order with exact content.
- Test invalid/oversized correction rejection before preparation and absence
of raw content from ordinary formatting/errors.
- Add one representative test showing PromptKit structural repair remains
available on a request with appended messages; do not reproduce upstream's
complete repair suite.
- Run `go test ./internal/framework/llm ./internal/framework/pipeline` and
`go test -race ./internal/framework/llm`.
### Exit Criteria
Notarius has one safe, tested adapter path for fresh corrected requests, and no
stage invokes it yet. This stage is one Terra prompt.
## Stage 6 — Migrate The Scene Chunker And Foundational Extractors ✅
### Goal
Make the scene chunker and the foundational direct D&D extractors satisfy
`single_response_v1`.
### Work
- Update `dnd/scenes`, `dnd/npc-registry`, `dnd/item-registry`,
`dnd/location-registry`, `dnd/scene-descriptions`, and `dnd/spells` producer
implementations to pass request correction to their structured completion
and return an owned copy of the successful response's exact validated raw
bytes as `ModelCandidate`.
- Declare `single_response_v1` in each corresponding module spec.
- Do not serialize normalized artifacts to fabricate candidate material. Keep
artifact parsing, deterministic identity attachment, evidence validation,
warnings, and durable schemas unchanged.
- Update shared D&D extraction helpers only where the behavior is genuinely
common; keep domain prompt ownership in each module.
### Verification
- Use representative package-level tests to prove exact-response propagation,
correction forwarding, and input/result ownership. Update spec tests for the
declared protocol without multiplying the same behavior test across all six
modules.
- Run the affected module tests and `go test ./internal/modules/dnd/...`.
### Exit Criteria
All named producers truthfully advertise and implement the exact-response
protocol, with unchanged ordinary extraction behavior. This stage is one
Terra prompt.
## Stage 7 — Migrate Downstream D&D Extractors ✅
### Goal
Complete `single_response_v1` support for the remaining direct D&D extraction
producers.
### Work
- Migrate `dnd/npc-occurrences`, `dnd/item-occurrences`,
`dnd/location-occurrences`, `dnd/combat-turns`, and `dnd/enemy-events` using
the same contract as Stage 6.
- Preserve their contextual-name and deterministic-ID rules. Correction
messages must never introduce opaque registry IDs or ask the model to copy
them.
- Preserve combat-scene gating, reference projections, and prompt-cache prefix
ordering. The correction pair is appended after the complete existing
request and never changes stable prompt assets.
### Verification
- Add or adapt focused tests for correction forwarding and exact raw-response
retention at the shared boundary, plus one registry-grounded extractor case
proving no opaque IDs enter correction material.
- Run the affected module tests, `go test ./internal/modules/dnd/...`, and the
assembled D&D CLI contract tests.
### Exit Criteria
Every direct production D&D LLM extractor implements the same correction
protocol without changing artifact semantics. This stage is one Terra prompt.
## Stage 8 — Migrate Semantic Reconciliation Normalizers ✅
### Goal
Support correction for the current single-proposal reconciliation path and
activate capability enforcement for all eligible production producers.
### Work
- Extend `semanticreconcile.Request` to accept `SemanticCorrection` and its
result to expose the exact validated proposal response as owned
`ModelCandidate` when an LLM call actually occurred.
- Forward correction through the shared engine's ordinary structured
completion request. Preserve request-local integer candidate handles,
proposal validation, typed application policies, and fallback behavior.
- Update the NPC-, item-, and location-registry normalizers to carry the exact
proposal material through their typed result and declare
`single_response_v1`.
- Mark deterministic skip/limit/fallback outcomes as having no model candidate.
If such a candidate is later rejected, it is not correctable and must not
consume a retry merely because its module execution class is LLM-backed.
- Audit all production LLM-backed producer specs. Add the preparation rule that
an LLM-backed producer with a non-empty validator chain and `retries > 0`
must declare `single_response_v1`. Any genuine multi-response producer
remains unsupported and a configured validator-backed semantic retry for it
must fail preparation.
### Verification
- Add engine tests for initial and corrected requests, exact proposal bytes,
deterministic no-call outcomes, invalid proposal outcomes, and ownership.
- Add representative normalizer and preparation tests for supported,
unsupported, and no-validator/no-retry configurations.
- Run `go test ./internal/framework/semanticreconcile ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli`.
### Exit Criteria
All eligible production LLM producers implement the declared protocol,
unsupported workflows fail before source parsing, and ordinary operational
retries remain allowed when semantic correction cannot occur. This stage is
one Terra prompt.
## Stage 9 — Build Complete Validator-Chain Execution ✅
### Goal
Replace first-result validation with one reusable, deterministic executor that
aggregates the complete chain and isolates validator retries.
### Work
- Introduce an internal immutable `validationReport` model with one ordered
invocation record per validator: approved, rejected, failed, or skipped;
validator attempt count; safe reason/message metadata; warnings; and bounded
correction guidance.
- Implement one shared sequential executor around stage-specific invocation
closures. It must continue after rejection and isolated execution failure,
retry an LLM-backed failed validator against the same immutable candidate up
to its binding budget, and stop retrying after a contract-valid approval or
rejection.
- Reconstruct the same ordinary validator request for each validator retry.
Do not append feedback to validator prompts and do not regenerate the
producer candidate.
- Preserve only one final warning for an exhausted validator failure; retain
individual attempt details for debug. Keep warnings from completed validators
in configured order.
- Build aggregate correction guidance from all rejections using the settled
ordering, deduplication, generic fallback, and bounds. Validator failures,
skips, warnings, and operator messages must not enter it.
- Retain `skipped` as a framework-owned outcome for an otherwise selected
validator whose runtime prerequisites are unavailable. Module construction,
type incompatibility, candidate cloning failure, cancellation, and debug
persistence failure remain framework errors rather than skips.
### Verification
- Add table-driven behavior tests for all approved, multiple rejections,
rejection plus failure, failure only, skipped, retry success, retry
exhaustion, immutable candidate reuse, deterministic ordering, duplicate
guidance, and bounds.
- Prove deterministic validators cannot receive positive retry budgets and
LLM validator retry counts do not consume producer attempts.
- Run `go test ./internal/framework/pipeline` and
`go test -race ./internal/framework/pipeline`.
### Exit Criteria
The shared executor returns a complete ordered report without deciding whether
the producer candidate advances. Existing stage callers may still adapt the
report through their old disposition path until the next stages. This stage is
one Terra prompt.
## Stage 10 — Implement The Generic Producer Attempt State Machine ✅
### Goal
Replace the boolean retry helper with one explicit domain-neutral state
machine used by later stage integrations.
### Work
- Refactor `runWithRetry` into an attempt engine that accepts producer and
complete-validation closures and returns a terminal result plus ordered
attempt provenance. Keep stage-specific artifact handling outside it.
- Classify attempts as initial, operational-error retry, structural retry,
module-requested retry, or semantic correction. Preserve one total producer
budget of `retries + 1` attempts.
- On semantic rejection, correct only when another attempt exists and the
current candidate contains valid `single_response_v1` material. Construct a
fresh `SemanticCorrection` from that latest response and aggregate feedback.
- Apply outcome precedence and effective terminal policy exactly as the
feature roadmap specifies. Ordinary producer errors remain framework errors
after retries. Only `ErrInvalidStructuredOutput` uses the structural-failure
policy; `reject_output` records no usable artifact.
- A deterministic or no-model candidate applies semantic terminal policy
immediately without spending an ineffective retry.
- With rejection plus validator failure, use rejection guidance for a
correction while recording incomplete validation. With failure only, apply
validator-failure policy without regenerating the producer.
- Keep cancellation and debug-persistence failures immediately terminal.
Abandoned-attempt warnings must not be promoted.
### Verification
- Test the state machine through stable behavior with fake closures: budget
accounting, fresh correction history, latest-response replacement, all
terminal actions, failure precedence, deterministic candidates,
cancellation, and warning promotion.
- Do not assert private helper call choreography or exact correction prose.
- Run `go test ./internal/framework/pipeline` and its race-enabled suite.
### Exit Criteria
One tested state machine owns attempt budgets and disposition, but no stage is
partially migrated. This stage is one Terra prompt.
## Stage 11 — Integrate Chunk Validation And Correction ✅
### Goal
Move chunk generation, validation, cache reuse, and retry disposition onto the
new state machine.
### Work
- Adapt chunk candidate materialization and both chunk/serialized validator
targets to the complete-chain executor.
- Pass correction only to a generated chunker attempt and retain the exact raw
chunker response associated with the materialized plan.
- Preserve chunk-plan cache identity and mode semantics. A rejected automatic
cache hit is unusable for the invocation and falls through to generated
attempt one without pretending the cache hit has model material. Publish a
newly generated plan only after complete accepted validation.
- Apply chunk effective terminal policy. A terminal chunk rejection prevents
lane execution; a framework error still prevents output encoding.
- Preserve annotation materialization, deterministic plan canonicalization,
attempt debug paths, scheduler use, and checkpoint behavior.
### Verification
- Add focused tests for corrected plan acceptance, multi-validator feedback,
rejected cache hit regeneration, cache non-overwrite, incomplete-validation
non-publication, policy outcomes, cancellation, and debug failure.
- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/chunk/... ./internal/cli` and the pipeline race tests.
### Exit Criteria
Chunk is the first complete production stage using feedback-aware retries, with
cache and terminal behavior matching the roadmap. This stage is one Terra
prompt.
## Stage 12 ✅ — Integrate Per-Chunk Extraction
### Goal
Apply the state machine independently to every concurrent extraction job.
### Work
- Adapt typed and serialized extract validation to the complete-chain executor
and use the extract binding's effective policy.
- Keep correction state local to one lane/chunk job. Rebuild the same initial
extraction request with correction attached only on a semantic retry.
- Preserve the run-wide worker pool, scheduled LLM client, chunk-first/lane-
second public ordering, stable error selection, cancellation, and lane
continuation rules.
- Ensure `reject_output` records the terminal chunk-scoped rejection without
advancing it to merge. `warn_continue` is available only for validator
failure with no semantic rejection and marks the accepted extract as
validation-incomplete.
### Verification
- Add representative extraction tests for correction success, exhausted
rejection, validator failure policy, independent concurrent jobs, stable
ordering, cancellation, and warning promotion.
- Add one assembled D&D pipeline test using fakes—not a live provider—to prove
a rejected direct extraction is corrected using the exact prior response.
- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli` and relevant race tests.
### Exit Criteria
Every extraction job has an isolated bounded correction conversation and
continues to obey existing concurrency and ordering contracts. This stage is
one Terra prompt.
## Stage 13 ✅ — Integrate Merge And Normalize
### Goal
Complete stage coverage and reconcile semantic correction with the existing
normalizer fallback retry directive.
### Work
- Adapt typed and serialized merge and normalize validation to the shared
executor and their respective effective policies.
- Keep merge and normalize serial within a lane and reuse the exact same
accepted upstream artifacts, references, source input, profile, and session
on every attempt.
- Fold `NormalizeRetry` into the one producer attempt state machine. It consumes
the same remaining stage budget, retains its validated safe fallback when
exhausted, and never overrides a semantic rejection of that fallback or a
later candidate.
- Do not send correction to deterministic mergers/normalizers. A rejected
deterministic candidate applies terminal policy immediately.
- Preserve typed erasure checks, codec boundaries, generated handoffs, and
lane continuation/cancellation behavior.
### Verification
- Add one representative merge path and normalize paths for correction
success, deterministic rejection, `NormalizeRetry` success/exhaustion,
fallback rejection, validator failure, and terminal policies.
- Run `go test ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/... ./internal/cli` and relevant race tests.
### Exit Criteria
Chunk, extract, merge, and normalize all use the same attempt and validation
semantics, with no nested normalizer retry loop. This stage is one Terra
prompt.
## Stage 14 ✅ — Finalize Provenance, Checkpoints, Debug, And Durable Results
### Goal
Make the new state machine auditable and safe without leaking correction
content or allowing degraded output to become reusable state.
### Work
- Extend internal attempt/debug records with attempt kind, PromptKit repair
count and usage, ordered validator outcomes and attempt counts, aggregate
reason codes, validation completeness, effective policy, and terminal
decision.
- Keep raw assistant responses and complete correction messages only in the
existing explicitly requested detailed trace. Default summaries contain
counts, digests, identities, and bounded safe fields.
- Add one reusable durable validation summary with the JSON fields
`status`, `rejecting_validators`, `reason_codes`,
`incomplete_validators`, `producer_attempt_count`, and `terminal_action`.
`status` is one of `complete`, `rejected`, or `incomplete`; lists preserve
configured order and omit duplicates after their first occurrence. Embed or
project this summary at the manifest, rejection, and run-result boundaries
that already expose the affected stage outcome. Preserve the existing
singular rejection fields as the first configured rejection for downstream
continuity; do not put raw guidance or responses in the summary.
- Emit one genuine, bounded, deterministically ordered warning per validator
whose execution budget is exhausted under `warn_continue`; do not emit a
warning merely because a later correction succeeded.
- Checkpoint only accepted and completely validated results. Never write or
reuse rejected, structurally invalid, or validation-incomplete producer
output. Ensure correction protocol and effective policy participate in
fingerprints.
- Confirm cache, resume, generated-reference handoff, output encoding, and CLI
result publication cannot treat a rejected or incomplete checkpoint as
accepted.
### Verification
- Add focused manifest, receipt, warning, debug, redaction, checkpoint reuse,
generated-handoff, and sensitive-content tests at their canonical owners.
- Test that corrected success is auditable but quiet, warn-continue output is
never checkpointed, and raw content is absent from ordinary durable files.
- Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint ./internal/framework/artifacts ./internal/cli` and relevant race tests.
### Exit Criteria
Every terminal path has accurate bounded provenance, no sensitive correction
content leaks by default, and only completely validated output is reusable.
This stage is one Terra prompt.
## Stage 15 ✅ — Update Canonical Documentation And Perform Final Verification
### Goal
Document the implemented contracts in their canonical homes and prove the
repository is ready for review.
### Work
- Update `docs/policy/architecture.md` with durable validator aggregation,
retry-budget separation, producer correction capability, terminal-policy,
checkpoint, and sensitive-data invariants. Link to ADR-0014 for rationale.
- Update `docs/config.md` with the exact `validation_policy` schema, enum
values, defaults, inheritance, placement restrictions, retry-budget
meanings, and invalid combinations.
- Update `docs/operations.md` with costs, terminal outcomes, warnings, debug
sensitivity, retry exhaustion, resume/cache consequences, and recovery.
- Update `docs/internal/pipeline.md`, `docs/internal/llm.md`, and
`docs/internal/modules.md` with implemented mechanics and focused test
routing. Keep public configuration definitions in `docs/config.md`.
- Complete `docs/integrations/pkg-promptkit.md` for v0.9.0 appended messages,
supported roles, application-owned bounds, and the external catalog
boundary. Update the run-result and affected output/subprocess integration
contracts for any durable fields added in Stage 14.
- Update maintained examples only to demonstrate implemented behavior. Use a
minimal policy override in the complete D&D example if it materially aids
operators; keep the minimal example minimal. Ensure all links point to
canonical owners and remove stale v0.8.0 claims outside historical release
notes.
- Do not create a release note until an actual release is prepared.
### Verification
- Run `gofmt` on all changed Go files.
- Run `go test ./...`, `go test -race ./...`, `go vet ./...`, and
`go build ./cmd/notarius`.
- Run the repository's example/config validation tests and documentation link
checks. If no standalone link checker exists, verify changed relative links
and record that manual check in the implementation report.
- Use targeted searches to confirm no current documentation still claims
PromptKit v0.8.0, no production correction-capable spec is missing its
protocol, no PromptKit type escaped `internal/framework/llm`, and no raw
correction content is serialized outside explicit detailed debug data.
- Review `git diff --check` and `git status --short`; do not include generated
binaries, temporary files, or unrelated work.
### Exit Criteria
All roadmap acceptance criteria are satisfied, canonical documentation matches
the code, maintained examples validate, the complete ordinary and race-enabled
test suites pass, and the worktree contains only intentional implementation
changes. This stage is one Terra prompt.
## Open Questions
None. PromptKit transport, producer representation, terminal-policy ownership,
bounds, retry budgets, outcome precedence, and checkpoint treatment are all
settled by the feature roadmap and this plan.