732 lines
34 KiB
Markdown
732 lines
34 KiB
Markdown
# Feedback-Aware Validation Retry Implementation Plan
|
|
|
|
## Status
|
|
|
|
Ready for implementation. This plan implements the target state defined by
|
|
[Feedback-Aware Stage Validation Retries](validation-retries.md). Its stages
|
|
are ordered dependencies, and each stage is intentionally scoped for one
|
|
gpt-5.6-terra implementation prompt.
|
|
|
|
The feature roadmap owns product intent and durable policy. This document owns
|
|
implementation order, concrete boundaries, and stage-level verification. If a
|
|
conflict is discovered, preserve the feature roadmap and the repository
|
|
policies, stop the affected stage, and revise this plan rather than silently
|
|
choosing a different architecture.
|
|
|
|
## Settled Decisions
|
|
|
|
The implementation agent must treat these decisions as fixed:
|
|
|
|
- PromptKit v0.9.0 is the minimum and exact supported PromptKit release for
|
|
this work. Notarius uses `RunRequest.AppendedMessages`; it does not create
|
|
paired correction manifests or bypass PromptKit preparation and execution.
|
|
- A correction request reconstructs the ordinary initial request and appends
|
|
exactly two messages: the latest exact producer response as `assistant`,
|
|
followed by one deterministic aggregate correction request as `user`.
|
|
Earlier correction turns never accumulate.
|
|
- A correction-capable producer supplies the exact single LLM response that
|
|
directly controlled the candidate. The initial protocol does not synthesize
|
|
a model-facing projection of a compound artifact.
|
|
- The only supported producer correction protocol is
|
|
`single_response_v1`. An empty protocol means unsupported. Configuration
|
|
rejects semantic-retry workflows for LLM-backed compound producers or any
|
|
other producer unable to satisfy `single_response_v1`.
|
|
- The existing producer binding `retries` value is the sole outer stage
|
|
budget. PromptKit repair attempts and validator execution retries are
|
|
independent budgets.
|
|
- Terminal policy is configured as pipeline defaults with field-by-field
|
|
producer-binding overrides. Validators never own candidate disposition.
|
|
- Application defaults are `fail_run` for exhausted producer structural
|
|
failure, `fail_run` for exhausted semantic rejection, and `warn_continue`
|
|
for exhausted validator execution failure.
|
|
- All applicable validators run sequentially in configured order. Rejections
|
|
and validator failures are aggregated; failures are not represented to the
|
|
producer as candidate defects.
|
|
- Deterministic producers do not consume retries after rejection. An
|
|
LLM-backed attempt that took a deterministic fast path and produced no model
|
|
response is likewise not correctable for that candidate.
|
|
- Known rejected or structurally invalid output never advances. A candidate
|
|
accepted under `validator_failure: warn_continue` advances with explicit
|
|
incomplete-validation provenance but is not checkpointed.
|
|
- The D&D combat-scene validator and broader warning-system redesign remain
|
|
out of scope.
|
|
|
|
## Cross-Stage Implementation Rules
|
|
|
|
Apply these rules in every stage:
|
|
|
|
1. Read the feature roadmap, `docs/development.md`, all `docs/policy/`
|
|
documents, and the focused current documentation named by the stage before
|
|
editing.
|
|
2. Inspect current code and tests rather than assuming paths or private helper
|
|
layouts. Preserve unrelated work and existing public ordering,
|
|
cancellation, scheduler, checkpoint, and sensitive-data invariants.
|
|
3. Use transport-neutral types outside `internal/framework/llm`. PromptKit
|
|
types must not escape the adapter boundary.
|
|
4. Keep correction material attempt-local. Do not put raw assistant responses
|
|
or correction text in ordinary errors, warnings, manifests, receipts,
|
|
checkpoints, caches, or default debug summaries.
|
|
5. Add the smallest durable tests that protect the stage's contracts. Do not
|
|
duplicate PromptKit's own prepared-execution, copying, hashing, capacity,
|
|
role-validation, or structural-repair test matrix.
|
|
6. Use `gofmt` on changed Go files. Run the focused checks listed for the
|
|
stage and fix failures before stopping. Do not proceed into the next stage
|
|
in the same prompt.
|
|
7. Do not delete this plan or the feature roadmap during implementation. They
|
|
are retired only after a final review confirms the complete target state.
|
|
|
|
## Contract Names And Bounds
|
|
|
|
Use one vocabulary consistently across packages. Exact private helper names
|
|
may follow local conventions, but the following contracts and values are not
|
|
open for redesign:
|
|
|
|
- `CorrectionProtocol` has only the empty unsupported value and
|
|
`single_response_v1`.
|
|
- An application-owned `SemanticCorrection` carries owned
|
|
`AssistantResponse []byte` and `UserGuidance string` values. It is exposed
|
|
on chunk, typed extract, typed merge, typed normalize, and structured
|
|
completion requests as an optional pointer.
|
|
- An application-owned `ModelCandidate` carries the owned exact response bytes
|
|
and `CorrectionProtocol`. It is exposed on the corresponding producer
|
|
results as an optional pointer and is never serialized as part of a durable
|
|
artifact.
|
|
- `ValidationResult` gains optional `CorrectionGuidance`. Operator-facing
|
|
`Message` is not copied into this field implicitly.
|
|
- Each reason code is valid UTF-8, nonblank after trimming, and at most 128
|
|
bytes. Each validator-supplied correction guidance value is valid UTF-8,
|
|
nonblank after trimming, and at most 4 KiB.
|
|
- The exact defective assistant response may be at most 1 MiB. The aggregate
|
|
user correction message may be at most 64 KiB. The combined appended
|
|
content may therefore be at most 1,114,112 bytes. Reject invalid UTF-8,
|
|
empty/whitespace-only content, or over-limit content; never truncate it.
|
|
- Invalid producer-owned correction material is a framework contract error.
|
|
It is not a validator failure and cannot be converted to `reject_output` or
|
|
`warn_continue`.
|
|
- Aggregate correction entries are ordered by validator order, contain the
|
|
stable reason code plus guidance, and collapse only byte-identical duplicate
|
|
`(reason_code, correction_guidance)` pairs while retaining first occurrence.
|
|
A rejection without guidance receives bounded generic guidance based only on
|
|
its reason code.
|
|
- Terminal actions are closed enums: structural failure and semantic
|
|
rejection accept `fail_run` or `reject_output`; validator failure accepts
|
|
`warn_continue` or `fail_run`.
|
|
|
|
## Stage 1 — Adopt PromptKit v0.9.0
|
|
|
|
### Goal
|
|
|
|
Upgrade the dependency without changing Notarius stage behavior, prove the
|
|
existing integration remains compatible, and make the new upstream request
|
|
primitive available to later stages.
|
|
|
|
### Work
|
|
|
|
- Update `go.mod` and `go.sum` to PromptKit v0.9.0 with `go get` and
|
|
`go mod tidy`. Accept the catalog modules selected transitively by PromptKit;
|
|
do not import or register them directly.
|
|
- Review every Notarius `promptkit.RunRequest` literal and retain keyed form.
|
|
Verify production prompt roles are limited to `system` and `user`.
|
|
- Update the conservative built-in-profile checkpoint marker in
|
|
`internal/framework/llm/promptkit_profile_fingerprint.go` from v0.8.0 to
|
|
v0.9.0. Do not duplicate the separately versioned catalog module versions.
|
|
- Update `docs/integrations/pkg-promptkit.md` only to describe the newly
|
|
implemented dependency version and unchanged current integration boundary.
|
|
Include the strict role vocabulary and external catalog ownership, but do
|
|
not document Notarius correction behavior yet.
|
|
|
|
### Verification
|
|
|
|
- Run `go test ./internal/framework/llm ./internal/cli`.
|
|
- Run `go test ./...`, `go test -race ./internal/framework/llm`, `go vet ./...`,
|
|
and `go build ./cmd/notarius`.
|
|
|
|
### Exit Criteria
|
|
|
|
Notarius builds and passes its suite on PromptKit v0.9.0, its compatibility
|
|
document and checkpoint marker name v0.9.0, and no Notarius package directly
|
|
depends on either upstream catalog module. This stage is one Terra prompt.
|
|
|
|
## Stage 2 — Record The Architecture And Add Transport-Neutral Contracts
|
|
|
|
### Goal
|
|
|
|
Record the durable decision and introduce owned correction/candidate types
|
|
without changing runtime retry behavior.
|
|
|
|
### Work
|
|
|
|
- Add ADR-0014 under `docs/adr/` using the repository ADR format. Record the
|
|
three distinct retry budgets, complete validator aggregation, the fresh
|
|
two-message correction protocol, exact-response producer capability,
|
|
non-recursive validator failure handling, terminal-policy ownership and
|
|
defaults, checkpoint conservatism, and sensitive-data constraints.
|
|
- Add `CorrectionProtocol`, `SemanticCorrection`, and `ModelCandidate` to the
|
|
domain-neutral framework contracts, with constructors/clone helpers that
|
|
validate the settled bounds and defensively copy bytes.
|
|
- Add optional correction input to `ChunkRequest`,
|
|
`TypedExtractionRequest`, `TypedMergeRequest`, and
|
|
`TypedNormalizeRequest`.
|
|
- Add optional model-candidate output to `ChunkPlanResult`,
|
|
`TypedExtractionResult`, `TypedMergeResult`, and
|
|
`TypedNormalizeResult`.
|
|
- Add `CorrectionGuidance` to `ValidationResult` and centralize validation of
|
|
reason codes and corrective guidance at the framework boundary.
|
|
- Update type-erasure adapters, stored fakes, and clone paths so the new values
|
|
retain ownership and cannot alias caller buffers. Do not declare production
|
|
module capabilities or use the fields in the runner yet.
|
|
|
|
### Verification
|
|
|
|
- Add contract-level tests for accepted values, UTF-8 and size rejection,
|
|
defensive copying, nil behavior, and typed-erasure preservation.
|
|
- Run `go test ./internal/framework/contracts ./internal/framework/pipeline`.
|
|
- Run `go test ./...`.
|
|
|
|
### Exit Criteria
|
|
|
|
The generic contracts can safely carry correction input and exact candidate
|
|
material across every producer stage, no PromptKit type crosses the boundary,
|
|
and existing runtime behavior is unchanged. This stage is one Terra prompt.
|
|
|
|
## Stage 3 — Add Validation Policy Configuration And Resolution
|
|
|
|
### Goal
|
|
|
|
Implement strict configuration, inheritance, identity, and provenance for
|
|
terminal validation policy before the runner consumes it.
|
|
|
|
### Work
|
|
|
|
- Add optional `validation_policy` to a pipeline profile and to chunk,
|
|
extract, merge, and normalize producer bindings. Use pointer-backed override
|
|
fields so omission is distinguishable from an explicit value.
|
|
- Reject an explicitly null policy object, null policy fields, duplicate or
|
|
unknown fields, and values outside the settled enums. Keep the current file
|
|
configuration version.
|
|
- Reject `validation_policy` on input, output, and validator bindings. Reject
|
|
a binding-level explicit `producer_structural_failure` value on a
|
|
deterministic producer; a pipeline-level structural default remains valid
|
|
because a pipeline may contain LLM-backed producers.
|
|
- Resolve each policy field independently in binding, pipeline, application
|
|
default order. Store one detached concrete effective policy for chunk and
|
|
for each lane's extract, merge, and normalize stage; do not leave runtime
|
|
inheritance to the runner.
|
|
- Include both configured overrides and effective policies at their existing
|
|
appropriate configuration/provenance boundaries. Include the effective
|
|
values in resolved-pipeline digest and checkpoint identity. Preserve
|
|
deterministic serialization and redaction.
|
|
- Update `examples/dnd-minimal/config.yml` and
|
|
`examples/dnd-complete/config.yml` only if their current shape must change to
|
|
remain valid; do not add redundant explicit defaults merely to demonstrate
|
|
the feature.
|
|
|
|
### Verification
|
|
|
|
- Add focused parsing tests for omission versus null, unknown/duplicate
|
|
fields, invalid placements, and invalid enums.
|
|
- Add resolution tests for field-by-field inheritance and mixed overrides.
|
|
- Add digest, clone, JSON/YAML round-trip, and redacted-effective-config tests.
|
|
- Run `go test ./internal/core/config ./internal/framework/pipeline ./internal/cli`.
|
|
|
|
### Exit Criteria
|
|
|
|
Every producer stage receives one immutable effective policy, invalid
|
|
configuration fails before execution, and policy changes invalidate resolved
|
|
identity. The runner still follows its old behavior until later stages. This
|
|
stage is one Terra prompt.
|
|
|
|
## Stage 4 — Declare And Validate Producer Correction Capability
|
|
|
|
### Goal
|
|
|
|
Make correction support an explicit module contract and prepare the framework
|
|
to reject impossible workflows.
|
|
|
|
### Work
|
|
|
|
- Extend `pipeline.ModuleSpec` with `CorrectionProtocol`; normalize, clone,
|
|
validate, and include it in relevant registry/spec fingerprints.
|
|
- Permit `single_response_v1` only for LLM-backed chunk, extract, merge, or
|
|
normalize modules. Reject it for deterministic, input, output, or validator
|
|
specs. Empty remains the default unsupported value.
|
|
- Carry the selected protocol into resolved and prepared producer metadata and
|
|
checkpoint fingerprints.
|
|
- Reject positive `retries` on deterministic validator bindings during
|
|
resolution. LLM-backed validator retries remain valid and independent of
|
|
producer capability.
|
|
|
|
### Verification
|
|
|
|
- Add spec normalization/clone tests, invalid stage/class combinations,
|
|
validator retry-class tests, and preparation tests for supported and
|
|
unsupported producer workflows.
|
|
- Run `go test ./internal/framework/pipeline ./internal/core/config`.
|
|
|
|
### Exit Criteria
|
|
|
|
Capability is explicit, fingerprinted, and testable, while existing production
|
|
pipelines remain executable pending their migrations. No dormant feature gate
|
|
or unused preparation check is introduced. This stage is one Terra prompt.
|
|
|
|
## Stage 5 — Adapt Corrections Through PromptKit
|
|
|
|
### Goal
|
|
|
|
Map the transport-neutral correction contract onto PromptKit v0.9.0 without
|
|
changing ordinary requests.
|
|
|
|
### Work
|
|
|
|
- Add optional `SemanticCorrection` to `StructuredCompletionRequest` and its
|
|
debug-safe request representation.
|
|
- In `PromptKitClient.CompleteStructured`, validate and defensively copy the
|
|
correction, then map it to exactly two `promptkit.RenderedMessage` values in
|
|
`RunRequest.AppendedMessages`, using `promptkit.RoleAssistant` followed by
|
|
`promptkit.RoleUser`.
|
|
- Leave `AppendedMessages` nil for an ordinary request. Do not allow callers to
|
|
choose other roles through the Notarius contract.
|
|
- Ensure request formatting and default debug summaries expose only safe
|
|
counts/digests. The existing explicitly requested detailed PromptKit trace
|
|
may contain complete effective messages and must retain its existing
|
|
sensitive-data treatment.
|
|
- Preserve the same prompt identity, inputs, variables, session, profile,
|
|
execution overrides, repair attempts, scheduler, and prepared-execution
|
|
path for corrected calls.
|
|
|
|
### Verification
|
|
|
|
- Add focused adapter tests proving an ordinary request is unchanged and a
|
|
corrected request has the ordinary rendered prefix plus exactly the two
|
|
supplied messages in order with exact content.
|
|
- Test invalid/oversized correction rejection before preparation and absence
|
|
of raw content from ordinary formatting/errors.
|
|
- Add one representative test showing PromptKit structural repair remains
|
|
available on a request with appended messages; do not reproduce upstream's
|
|
complete repair suite.
|
|
- Run `go test ./internal/framework/llm ./internal/framework/pipeline` and
|
|
`go test -race ./internal/framework/llm`.
|
|
|
|
### Exit Criteria
|
|
|
|
Notarius has one safe, tested adapter path for fresh corrected requests, and no
|
|
stage invokes it yet. This stage is one Terra prompt.
|
|
|
|
## Stage 6 — Migrate The Scene Chunker And Foundational Extractors
|
|
|
|
### Goal
|
|
|
|
Make the scene chunker and the foundational direct D&D extractors satisfy
|
|
`single_response_v1`.
|
|
|
|
### Work
|
|
|
|
- Update `dnd/scenes`, `dnd/npc-registry`, `dnd/item-registry`,
|
|
`dnd/location-registry`, `dnd/scene-descriptions`, and `dnd/spells` producer
|
|
implementations to pass request correction to their structured completion
|
|
and return an owned copy of the successful response's exact validated raw
|
|
bytes as `ModelCandidate`.
|
|
- Declare `single_response_v1` in each corresponding module spec.
|
|
- Do not serialize normalized artifacts to fabricate candidate material. Keep
|
|
artifact parsing, deterministic identity attachment, evidence validation,
|
|
warnings, and durable schemas unchanged.
|
|
- Update shared D&D extraction helpers only where the behavior is genuinely
|
|
common; keep domain prompt ownership in each module.
|
|
|
|
### Verification
|
|
|
|
- Use representative package-level tests to prove exact-response propagation,
|
|
correction forwarding, and input/result ownership. Update spec tests for the
|
|
declared protocol without multiplying the same behavior test across all six
|
|
modules.
|
|
- Run the affected module tests and `go test ./internal/modules/dnd/...`.
|
|
|
|
### Exit Criteria
|
|
|
|
All named producers truthfully advertise and implement the exact-response
|
|
protocol, with unchanged ordinary extraction behavior. This stage is one
|
|
Terra prompt.
|
|
|
|
## Stage 7 — Migrate Downstream D&D Extractors
|
|
|
|
### Goal
|
|
|
|
Complete `single_response_v1` support for the remaining direct D&D extraction
|
|
producers.
|
|
|
|
### Work
|
|
|
|
- Migrate `dnd/npc-occurrences`, `dnd/item-occurrences`,
|
|
`dnd/location-occurrences`, `dnd/combat-turns`, and `dnd/enemy-events` using
|
|
the same contract as Stage 6.
|
|
- Preserve their contextual-name and deterministic-ID rules. Correction
|
|
messages must never introduce opaque registry IDs or ask the model to copy
|
|
them.
|
|
- Preserve combat-scene gating, reference projections, and prompt-cache prefix
|
|
ordering. The correction pair is appended after the complete existing
|
|
request and never changes stable prompt assets.
|
|
|
|
### Verification
|
|
|
|
- Add or adapt focused tests for correction forwarding and exact raw-response
|
|
retention at the shared boundary, plus one registry-grounded extractor case
|
|
proving no opaque IDs enter correction material.
|
|
- Run the affected module tests, `go test ./internal/modules/dnd/...`, and the
|
|
assembled D&D CLI contract tests.
|
|
|
|
### Exit Criteria
|
|
|
|
Every direct production D&D LLM extractor implements the same correction
|
|
protocol without changing artifact semantics. This stage is one Terra prompt.
|
|
|
|
## Stage 8 — Migrate Semantic Reconciliation Normalizers
|
|
|
|
### Goal
|
|
|
|
Support correction for the current single-proposal reconciliation path and
|
|
activate capability enforcement for all eligible production producers.
|
|
|
|
### Work
|
|
|
|
- Extend `semanticreconcile.Request` to accept `SemanticCorrection` and its
|
|
result to expose the exact validated proposal response as owned
|
|
`ModelCandidate` when an LLM call actually occurred.
|
|
- Forward correction through the shared engine's ordinary structured
|
|
completion request. Preserve request-local integer candidate handles,
|
|
proposal validation, typed application policies, and fallback behavior.
|
|
- Update the NPC-, item-, and location-registry normalizers to carry the exact
|
|
proposal material through their typed result and declare
|
|
`single_response_v1`.
|
|
- Mark deterministic skip/limit/fallback outcomes as having no model candidate.
|
|
If such a candidate is later rejected, it is not correctable and must not
|
|
consume a retry merely because its module execution class is LLM-backed.
|
|
- Audit all production LLM-backed producer specs. Add the preparation rule that
|
|
an LLM-backed producer with a non-empty validator chain and `retries > 0`
|
|
must declare `single_response_v1`. Any genuine multi-response producer
|
|
remains unsupported and a configured validator-backed semantic retry for it
|
|
must fail preparation.
|
|
|
|
### Verification
|
|
|
|
- Add engine tests for initial and corrected requests, exact proposal bytes,
|
|
deterministic no-call outcomes, invalid proposal outcomes, and ownership.
|
|
- Add representative normalizer and preparation tests for supported,
|
|
unsupported, and no-validator/no-retry configurations.
|
|
- Run `go test ./internal/framework/semanticreconcile ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli`.
|
|
|
|
### Exit Criteria
|
|
|
|
All eligible production LLM producers implement the declared protocol,
|
|
unsupported workflows fail before source parsing, and ordinary operational
|
|
retries remain allowed when semantic correction cannot occur. This stage is
|
|
one Terra prompt.
|
|
|
|
## Stage 9 — Build Complete Validator-Chain Execution
|
|
|
|
### Goal
|
|
|
|
Replace first-result validation with one reusable, deterministic executor that
|
|
aggregates the complete chain and isolates validator retries.
|
|
|
|
### Work
|
|
|
|
- Introduce an internal immutable `validationReport` model with one ordered
|
|
invocation record per validator: approved, rejected, failed, or skipped;
|
|
validator attempt count; safe reason/message metadata; warnings; and bounded
|
|
correction guidance.
|
|
- Implement one shared sequential executor around stage-specific invocation
|
|
closures. It must continue after rejection and isolated execution failure,
|
|
retry an LLM-backed failed validator against the same immutable candidate up
|
|
to its binding budget, and stop retrying after a contract-valid approval or
|
|
rejection.
|
|
- Reconstruct the same ordinary validator request for each validator retry.
|
|
Do not append feedback to validator prompts and do not regenerate the
|
|
producer candidate.
|
|
- Preserve only one final warning for an exhausted validator failure; retain
|
|
individual attempt details for debug. Keep warnings from completed validators
|
|
in configured order.
|
|
- Build aggregate correction guidance from all rejections using the settled
|
|
ordering, deduplication, generic fallback, and bounds. Validator failures,
|
|
skips, warnings, and operator messages must not enter it.
|
|
- Retain `skipped` as a framework-owned outcome for an otherwise selected
|
|
validator whose runtime prerequisites are unavailable. Module construction,
|
|
type incompatibility, candidate cloning failure, cancellation, and debug
|
|
persistence failure remain framework errors rather than skips.
|
|
|
|
### Verification
|
|
|
|
- Add table-driven behavior tests for all approved, multiple rejections,
|
|
rejection plus failure, failure only, skipped, retry success, retry
|
|
exhaustion, immutable candidate reuse, deterministic ordering, duplicate
|
|
guidance, and bounds.
|
|
- Prove deterministic validators cannot receive positive retry budgets and
|
|
LLM validator retry counts do not consume producer attempts.
|
|
- Run `go test ./internal/framework/pipeline` and
|
|
`go test -race ./internal/framework/pipeline`.
|
|
|
|
### Exit Criteria
|
|
|
|
The shared executor returns a complete ordered report without deciding whether
|
|
the producer candidate advances. Existing stage callers may still adapt the
|
|
report through their old disposition path until the next stages. This stage is
|
|
one Terra prompt.
|
|
|
|
## Stage 10 — Implement The Generic Producer Attempt State Machine
|
|
|
|
### Goal
|
|
|
|
Replace the boolean retry helper with one explicit domain-neutral state
|
|
machine used by later stage integrations.
|
|
|
|
### Work
|
|
|
|
- Refactor `runWithRetry` into an attempt engine that accepts producer and
|
|
complete-validation closures and returns a terminal result plus ordered
|
|
attempt provenance. Keep stage-specific artifact handling outside it.
|
|
- Classify attempts as initial, operational-error retry, structural retry,
|
|
module-requested retry, or semantic correction. Preserve one total producer
|
|
budget of `retries + 1` attempts.
|
|
- On semantic rejection, correct only when another attempt exists and the
|
|
current candidate contains valid `single_response_v1` material. Construct a
|
|
fresh `SemanticCorrection` from that latest response and aggregate feedback.
|
|
- Apply outcome precedence and effective terminal policy exactly as the
|
|
feature roadmap specifies. Ordinary producer errors remain framework errors
|
|
after retries. Only `ErrInvalidStructuredOutput` uses the structural-failure
|
|
policy; `reject_output` records no usable artifact.
|
|
- A deterministic or no-model candidate applies semantic terminal policy
|
|
immediately without spending an ineffective retry.
|
|
- With rejection plus validator failure, use rejection guidance for a
|
|
correction while recording incomplete validation. With failure only, apply
|
|
validator-failure policy without regenerating the producer.
|
|
- Keep cancellation and debug-persistence failures immediately terminal.
|
|
Abandoned-attempt warnings must not be promoted.
|
|
|
|
### Verification
|
|
|
|
- Test the state machine through stable behavior with fake closures: budget
|
|
accounting, fresh correction history, latest-response replacement, all
|
|
terminal actions, failure precedence, deterministic candidates,
|
|
cancellation, and warning promotion.
|
|
- Do not assert private helper call choreography or exact correction prose.
|
|
- Run `go test ./internal/framework/pipeline` and its race-enabled suite.
|
|
|
|
### Exit Criteria
|
|
|
|
One tested state machine owns attempt budgets and disposition, but no stage is
|
|
partially migrated. This stage is one Terra prompt.
|
|
|
|
## Stage 11 — Integrate Chunk Validation And Correction
|
|
|
|
### Goal
|
|
|
|
Move chunk generation, validation, cache reuse, and retry disposition onto the
|
|
new state machine.
|
|
|
|
### Work
|
|
|
|
- Adapt chunk candidate materialization and both chunk/serialized validator
|
|
targets to the complete-chain executor.
|
|
- Pass correction only to a generated chunker attempt and retain the exact raw
|
|
chunker response associated with the materialized plan.
|
|
- Preserve chunk-plan cache identity and mode semantics. A rejected automatic
|
|
cache hit is unusable for the invocation and falls through to generated
|
|
attempt one without pretending the cache hit has model material. Publish a
|
|
newly generated plan only after complete accepted validation.
|
|
- Apply chunk effective terminal policy. A terminal chunk rejection prevents
|
|
lane execution; a framework error still prevents output encoding.
|
|
- Preserve annotation materialization, deterministic plan canonicalization,
|
|
attempt debug paths, scheduler use, and checkpoint behavior.
|
|
|
|
### Verification
|
|
|
|
- Add focused tests for corrected plan acceptance, multi-validator feedback,
|
|
rejected cache hit regeneration, cache non-overwrite, incomplete-validation
|
|
non-publication, policy outcomes, cancellation, and debug failure.
|
|
- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/chunk/... ./internal/cli` and the pipeline race tests.
|
|
|
|
### Exit Criteria
|
|
|
|
Chunk is the first complete production stage using feedback-aware retries, with
|
|
cache and terminal behavior matching the roadmap. This stage is one Terra
|
|
prompt.
|
|
|
|
## Stage 12 — Integrate Per-Chunk Extraction
|
|
|
|
### Goal
|
|
|
|
Apply the state machine independently to every concurrent extraction job.
|
|
|
|
### Work
|
|
|
|
- Adapt typed and serialized extract validation to the complete-chain executor
|
|
and use the extract binding's effective policy.
|
|
- Keep correction state local to one lane/chunk job. Rebuild the same initial
|
|
extraction request with correction attached only on a semantic retry.
|
|
- Preserve the run-wide worker pool, scheduled LLM client, chunk-first/lane-
|
|
second public ordering, stable error selection, cancellation, and lane
|
|
continuation rules.
|
|
- Ensure `reject_output` records the terminal chunk-scoped rejection without
|
|
advancing it to merge. `warn_continue` is available only for validator
|
|
failure with no semantic rejection and marks the accepted extract as
|
|
validation-incomplete.
|
|
|
|
### Verification
|
|
|
|
- Add representative extraction tests for correction success, exhausted
|
|
rejection, validator failure policy, independent concurrent jobs, stable
|
|
ordering, cancellation, and warning promotion.
|
|
- Add one assembled D&D pipeline test using fakes—not a live provider—to prove
|
|
a rejected direct extraction is corrected using the exact prior response.
|
|
- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli` and relevant race tests.
|
|
|
|
### Exit Criteria
|
|
|
|
Every extraction job has an isolated bounded correction conversation and
|
|
continues to obey existing concurrency and ordering contracts. This stage is
|
|
one Terra prompt.
|
|
|
|
## Stage 13 — Integrate Merge And Normalize
|
|
|
|
### Goal
|
|
|
|
Complete stage coverage and reconcile semantic correction with the existing
|
|
normalizer fallback retry directive.
|
|
|
|
### Work
|
|
|
|
- Adapt typed and serialized merge and normalize validation to the shared
|
|
executor and their respective effective policies.
|
|
- Keep merge and normalize serial within a lane and reuse the exact same
|
|
accepted upstream artifacts, references, source input, profile, and session
|
|
on every attempt.
|
|
- Fold `NormalizeRetry` into the one producer attempt state machine. It consumes
|
|
the same remaining stage budget, retains its validated safe fallback when
|
|
exhausted, and never overrides a semantic rejection of that fallback or a
|
|
later candidate.
|
|
- Do not send correction to deterministic mergers/normalizers. A rejected
|
|
deterministic candidate applies terminal policy immediately.
|
|
- Preserve typed erasure checks, codec boundaries, generated handoffs, and
|
|
lane continuation/cancellation behavior.
|
|
|
|
### Verification
|
|
|
|
- Add one representative merge path and normalize paths for correction
|
|
success, deterministic rejection, `NormalizeRetry` success/exhaustion,
|
|
fallback rejection, validator failure, and terminal policies.
|
|
- Run `go test ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/... ./internal/cli` and relevant race tests.
|
|
|
|
### Exit Criteria
|
|
|
|
Chunk, extract, merge, and normalize all use the same attempt and validation
|
|
semantics, with no nested normalizer retry loop. This stage is one Terra
|
|
prompt.
|
|
|
|
## Stage 14 — Finalize Provenance, Checkpoints, Debug, And Durable Results
|
|
|
|
### Goal
|
|
|
|
Make the new state machine auditable and safe without leaking correction
|
|
content or allowing degraded output to become reusable state.
|
|
|
|
### Work
|
|
|
|
- Extend internal attempt/debug records with attempt kind, PromptKit repair
|
|
count and usage, ordered validator outcomes and attempt counts, aggregate
|
|
reason codes, validation completeness, effective policy, and terminal
|
|
decision.
|
|
- Keep raw assistant responses and complete correction messages only in the
|
|
existing explicitly requested detailed trace. Default summaries contain
|
|
counts, digests, identities, and bounded safe fields.
|
|
- Add one reusable durable validation summary with the JSON fields
|
|
`status`, `rejecting_validators`, `reason_codes`,
|
|
`incomplete_validators`, `producer_attempt_count`, and `terminal_action`.
|
|
`status` is one of `complete`, `rejected`, or `incomplete`; lists preserve
|
|
configured order and omit duplicates after their first occurrence. Embed or
|
|
project this summary at the manifest, rejection, and run-result boundaries
|
|
that already expose the affected stage outcome. Preserve the existing
|
|
singular rejection fields as the first configured rejection for downstream
|
|
continuity; do not put raw guidance or responses in the summary.
|
|
- Emit one genuine, bounded, deterministically ordered warning per validator
|
|
whose execution budget is exhausted under `warn_continue`; do not emit a
|
|
warning merely because a later correction succeeded.
|
|
- Checkpoint only accepted and completely validated results. Never write or
|
|
reuse rejected, structurally invalid, or validation-incomplete producer
|
|
output. Ensure correction protocol and effective policy participate in
|
|
fingerprints.
|
|
- Confirm cache, resume, generated-reference handoff, output encoding, and CLI
|
|
result publication cannot treat a rejected or incomplete checkpoint as
|
|
accepted.
|
|
|
|
### Verification
|
|
|
|
- Add focused manifest, receipt, warning, debug, redaction, checkpoint reuse,
|
|
generated-handoff, and sensitive-content tests at their canonical owners.
|
|
- Test that corrected success is auditable but quiet, warn-continue output is
|
|
never checkpointed, and raw content is absent from ordinary durable files.
|
|
- Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint ./internal/framework/artifacts ./internal/cli` and relevant race tests.
|
|
|
|
### Exit Criteria
|
|
|
|
Every terminal path has accurate bounded provenance, no sensitive correction
|
|
content leaks by default, and only completely validated output is reusable.
|
|
This stage is one Terra prompt.
|
|
|
|
## Stage 15 — Update Canonical Documentation And Perform Final Verification
|
|
|
|
### Goal
|
|
|
|
Document the implemented contracts in their canonical homes and prove the
|
|
repository is ready for review.
|
|
|
|
### Work
|
|
|
|
- Update `docs/policy/architecture.md` with durable validator aggregation,
|
|
retry-budget separation, producer correction capability, terminal-policy,
|
|
checkpoint, and sensitive-data invariants. Link to ADR-0014 for rationale.
|
|
- Update `docs/config.md` with the exact `validation_policy` schema, enum
|
|
values, defaults, inheritance, placement restrictions, retry-budget
|
|
meanings, and invalid combinations.
|
|
- Update `docs/operations.md` with costs, terminal outcomes, warnings, debug
|
|
sensitivity, retry exhaustion, resume/cache consequences, and recovery.
|
|
- Update `docs/internal/pipeline.md`, `docs/internal/llm.md`, and
|
|
`docs/internal/modules.md` with implemented mechanics and focused test
|
|
routing. Keep public configuration definitions in `docs/config.md`.
|
|
- Complete `docs/integrations/pkg-promptkit.md` for v0.9.0 appended messages,
|
|
supported roles, application-owned bounds, and the external catalog
|
|
boundary. Update the run-result and affected output/subprocess integration
|
|
contracts for any durable fields added in Stage 14.
|
|
- Update maintained examples only to demonstrate implemented behavior. Use a
|
|
minimal policy override in the complete D&D example if it materially aids
|
|
operators; keep the minimal example minimal. Ensure all links point to
|
|
canonical owners and remove stale v0.8.0 claims outside historical release
|
|
notes.
|
|
- Do not create a release note until an actual release is prepared.
|
|
|
|
### Verification
|
|
|
|
- Run `gofmt` on all changed Go files.
|
|
- Run `go test ./...`, `go test -race ./...`, `go vet ./...`, and
|
|
`go build ./cmd/notarius`.
|
|
- Run the repository's example/config validation tests and documentation link
|
|
checks. If no standalone link checker exists, verify changed relative links
|
|
and record that manual check in the implementation report.
|
|
- Use targeted searches to confirm no current documentation still claims
|
|
PromptKit v0.8.0, no production correction-capable spec is missing its
|
|
protocol, no PromptKit type escaped `internal/framework/llm`, and no raw
|
|
correction content is serialized outside explicit detailed debug data.
|
|
- Review `git diff --check` and `git status --short`; do not include generated
|
|
binaries, temporary files, or unrelated work.
|
|
|
|
### Exit Criteria
|
|
|
|
All roadmap acceptance criteria are satisfied, canonical documentation matches
|
|
the code, maintained examples validate, the complete ordinary and race-enabled
|
|
test suites pass, and the worktree contains only intentional implementation
|
|
changes. This stage is one Terra prompt.
|
|
|
|
## Open Questions
|
|
|
|
None. PromptKit transport, producer representation, terminal-policy ownership,
|
|
bounds, retry budgets, outcome precedence, and checkpoint treatment are all
|
|
settled by the feature roadmap and this plan.
|