diff --git a/docs/adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md b/docs/adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md index 4d07fdc4..a1568333 100644 --- a/docs/adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md +++ b/docs/adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md @@ -83,9 +83,7 @@ entity identity, checkpoint contracts, or cross-request correlation. Changes to shared protocol and policy assets must participate in the normal prompt, schema, and checkpoint fingerprint mechanisms. -Acceptance of this decision does not imply that the shared mechanism or its -consumer migrations are implemented. The -[feature roadmap](../roadmap/semantic-reconciliation.md) owns target behavior -and status, and the -[implementation plan](../roadmap/implementation.md) owns delivery sequence -until the work is complete. +The shared mechanism and its initial D&D registry consumers are now +implemented. Current behavior is documented in +[Module Internals](../internal/modules.md#semantic-reconciliation) and +[D&D Module Internals](../internal/dnd.md#semantic-registry-reconciliation). diff --git a/docs/roadmap/dnd-subprocess-documentation.md b/docs/roadmap/dnd-subprocess-documentation.md deleted file mode 100644 index 46abf998..00000000 --- a/docs/roadmap/dnd-subprocess-documentation.md +++ /dev/null @@ -1,224 +0,0 @@ -# D&D Subprocess Consumer Documentation - -## Status - -Completed. The target guide is `docs/consumers/dnd-pipeline.md`. - -## Purpose - -Provide one task-oriented guide for applications that run Notarius as a -subprocess to execute the maintained complete D&D pipeline and consume its -published artifacts. The initial concrete consumer is Narratio, but the guide -must describe the public Notarius workflow rather than depend on Narratio -internals. - -The guide should make the safe integration path obvious without duplicating -the CLI, input, receipt, output-bundle, or individual artifact contracts that -already have canonical documentation. - -## Current State - -The public integration surface is documented accurately but is distributed -across several documents: - -- `docs/consumers/subprocess.md` defines the generic subprocess workflow; -- `docs/cli.md` owns commands, flags, stream behavior, and exit statuses; -- `docs/integrations/seriatim.md` owns the accepted transcript input shape; -- `docs/integrations/run-result.md` owns the machine-readable successful-run - receipt; -- `docs/integrations/json-output.md` owns bundle discovery and logical files; -- the D&D integration documents own the individual lane payload contracts; -- `examples/dnd-complete.config.yml` is the maintained complete pipeline. - -A consumer can reconstruct the full workflow from those documents, but there -is no D&D-focused guide that connects the maintained example to its input, -invocation, complete artifact inventory, discovery procedure, and downstream -acceptance decisions. - -## Target Documentation Set - -### Create `docs/consumers/dnd-pipeline.md` - -This document should own the end-to-end consumer workflow for the maintained -complete D&D configuration. It should be useful to Narratio and to another -subprocess orchestrator with the same needs. - -The guide should contain the following sections. - -#### Prerequisites And Deployment Configuration - -- Link to `examples/dnd-complete.config.yml` rather than embedding a second - complete configuration. -- Explain that a deployment must provide the configured PromptKit profile and - campaign reference files. -- Recommend absolute paths for a service or orchestrator deployment. -- Call out the path-resolution distinction explicitly: YAML reference paths - are relative to the Notarius configuration file, while - `promptkit.profile_file` is relative to the Notarius process working - directory. -- Recommend validating the selected configuration and `dnd-session` pipeline - before processing sessions. - -#### Transcript Input - -- State that the complete pipeline consumes a Seriatim JSON document. -- Link to the canonical Seriatim contract for required fields and validation. -- Recommend the caller's final trimmed transcript when the caller maintains - transcript tiers. For Narratio, identify the implemented source as - `narratio.transcript.final_trimmed`, normally stored at - `transcripts/final.trimmed.json`. -- Explain that segment IDs must remain stable because D&D source references - cite those units. -- Explain that Notarius derives its default prompt session from the input - module and exact input bytes and that ordinary callers should not supply - `--session-id`. - -#### Subprocess Invocation - -- Show one concise invocation using `notarius run dnd-session`, explicit - absolute `--config`, `--input`, and `--output-dir` paths, and `--json`. -- Direct callers to capture stdout and stderr separately, propagate - cancellation, impose an operator-appropriate timeout, and wait for process - completion before parsing stdout. -- State that only exit status zero permits receipt decoding and link to the CLI - contract for the complete exit-status definition. -- Recommend retaining stderr and the invocation context for diagnosis without - logging secrets or transcript content. - -#### Receipt And Bundle Discovery - -- Require callers to accept only supported run-result schema versions while - tolerating unknown fields allowed by that version. -- Direct callers to obtain the exact run-specific bundle from the receipt's - absolute `output_directory`; they must not scan for the newest run directory - or construct a run ID. -- Require a confinement check when resolving `index_file` beneath the reported - bundle root. -- Direct callers to discover lane payloads by `lane_id` in `index.json`, then - verify descriptor media type and schema identity before decoding them. -- Explain that descriptor paths are untrusted relative paths and require the - same confinement discipline. - -#### Complete D&D Artifact Inventory - -Include a compact table for the ten lane IDs selected by the maintained -complete configuration: - -- `item-registry`; -- `npc-registry`; -- `location-registry`; -- `scene-descriptions`; -- `item-occurrences`; -- `spells`; -- `combat-turns`; -- `npc-occurrences`; -- `location-occurrences`; -- `enemy-events`. - -For each row, give a one-line purpose and link to the corresponding canonical -D&D artifact contract. Do not copy its fields or schema rules into the -consumer guide. - -Document the four always-published bundle files—`index.json`, `manifest.json`, -`rejected.json`, and `warnings.json`—and the complete example's configured -`chunk-map.json` and `evidence-context.json` pipeline-wide artifacts. Link to -their canonical contracts and distinguish pipeline-wide artifacts from lane -outputs. - -The inventory must say that a file is available only when its corresponding -artifact was accepted and published. It must not imply that process success -guarantees every configured lane. - -#### Downstream Acceptance And Retention - -- Explain that exit status zero can coexist with rejected outputs, warnings, - or absent lane descriptors. -- Require the consumer to define its required lane set explicitly. Recommend - treating all ten lanes as required when the caller claims to consume the - complete D&D workflow, while allowing another consumer to adopt a narrower - documented policy. -- Recommend retaining the receipt, the complete published bundle, and captured - diagnostic streams long enough to support provenance and failure analysis. -- Explain that `evidence-context.json` is a reading excerpt; authoritative - citations remain in lane payloads. -- Treat transcripts, lane artifacts, evidence context, manifests, and logs as - sensitive campaign data. - -#### Compatibility Checklist - -End with a concise checklist covering process exit, receipt schema, path -confinement, pipeline identity, index decoding, required descriptors, -descriptor schema/media compatibility, warnings and rejections, checksums or -retention, and secure handling. Compatibility should be based on published -receipt and artifact contracts rather than parsing a human version string. - -### Update Existing Navigation - -- Add a short link from `docs/consumers/subprocess.md` to the D&D-specific - workflow. Keep generic subprocess policy in the existing document. -- Add the guide to the documentation links in `README.md`. -- Extend the subprocess-consumer row in `docs/development.md` so maintainers - working on the D&D workflow are routed to the new guide and the canonical - contracts. - -### Verify Canonical Contract Documents - -Review the linked integration documents and the complete example while writing -the guide. Correct an integration document only if repository inspection finds -an actual stale contract. Do not move schema definitions, field tables, CLI -flags, or configuration semantics into the new guide. - -## Narratio Alignment - -The guide may name Narratio as the motivating consumer and identify its current -final-trimmed transcript source. It must not claim that Narratio already has a -Notarius adapter or extraction stage. Until that feature is implemented, -Narratio-specific architecture, configuration, stage behavior, manifest -records, and artifact source IDs belong in Narratio's roadmap. - -Once Narratio implements the integration, its own integration documentation -should link to this guide and the durable Notarius contracts instead of -repeating them. - -## Validation - -Documentation implementation should include: - -```sh -go run ./cmd/notarius config validate \ - --config examples/dnd-complete.config.yml \ - --pipeline dnd-session -go test ./... -``` - -Also verify all new and changed relative Markdown links, compare the artifact -inventory directly with the maintained complete configuration, and confirm -that commands and path semantics match the CLI and configuration references. -If the repository still has no automated link checker, record that fact and -perform a focused manual link review. - -## Acceptance Criteria - -- A subprocess integrator can follow one D&D-focused guide from a Seriatim - transcript through safe discovery of every artifact configured by the - complete example. -- The guide makes stdout, stderr, exit-status, receipt, and path-confinement - responsibilities unambiguous. -- The ten configured D&D lanes and both configured pipeline-wide artifacts are - listed and linked to their canonical contracts. -- The guide distinguishes process success from the caller's required-artifact - policy. -- The profile-path and reference-path resolution rules are clearly stated. -- Existing navigation makes the guide discoverable. -- No volatile contract is defined in two places, and no unimplemented Narratio - behavior is presented as current. - -## Non-Goals - -- Implementing or documenting Narratio's future adapter or stage as current - Notarius behavior. -- Adding a new Notarius command, receipt version, output format, or artifact - schema. -- Duplicating the complete configuration or individual D&D payload schemas in - prose. -- Defining a universal partial-result policy for every Notarius consumer. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 21844f02..0cb01d74 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -13,124 +13,15 @@ structural output repair within one completion. Notarius owns stage candidates, validator chains, semantic rejection policy, and whether another stage attempt is warranted. -### 1. Upgrade To PromptKit v0.8.0 +### Feedback-Aware Stage Validation Retries This item has been promoted to the standalone -[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md) roadmap. That document owns the -release-by-release compatibility review, adopted features, structured-repair -policy, target integration boundary, acceptance criteria, and settled design -decisions. +[Feedback-Aware Stage Validation Retries](validation-retries.md) roadmap. That +document owns the target validation state machine, correction protocol, retry +budgets, terminal policies, provenance requirements, settled producer +contracts, and PromptKit v0.9.0 adoption. -### 2. Feedback-Aware Stage Validation Retries - -- Model Notarius's corrective stage-retry conversation explicitly after - PromptKit v0.8.0. The first attempt sends the ordinary complete initial - prompt. If application validation rejects the resulting LLM-produced - candidate and another stage attempt is available, reconstruct that complete - initial prompt byte-for-byte and append exactly two messages: an assistant - message containing the defective response and an application-owned user - message detailing every applicable semantic validation error and requesting - one corrected, complete replacement response. This is a freshly constructed - correction request, not continuation of an accumulating conversation. -- Use the configured stage `retries` value as the one outer retry budget for - this loop. `retries: N` continues to mean at most `N` additional complete - chunk, extract, merge, or normalize attempts after the initial attempt, - whether an attempt is needed because of a producer error or semantic - rejection. Do not add a second semantic-correction count. PromptKit's - prompt-level `repair_attempts` budget is independent and internal to each - individual LLM completion, and does not consume or replenish the Notarius - stage budget. -- Extend the framework-managed validation boundary for chunk, extract, merge, - and normalize stages so a rejected LLM-produced candidate and its exact raw - model response remain available to construct the next stage attempt. - Deterministic producers cannot improve by repeating the same inputs; a - rejection from a deterministic stage is therefore terminal under the - configured rejection policy rather than consuming retries mechanically. -- Preserve the original session ID, selected profile, structured-output - contract, prompt inputs, and reusable prompt prefix. Carry only the latest - candidate and latest aggregate feedback; do not build an unbounded retry - conversation. Keep model-facing corrective guidance separate from - operator-facing diagnostics, and apply explicit size, redaction, and debug - disclosure rules to both. -- Run every applicable validator in the configured chain before deciding - whether to retry. Do not short-circuit merely because an earlier validator - rejected the candidate. Aggregate all semantic rejection reason codes and - corrective guidance into the retry message so one retry can address the - whole candidate. A validator is applicable only when its declared target and - prerequisites can be satisfied; record a deterministic skipped diagnostic - rather than invoking a validator on an input it cannot interpret. Initially - execute the chain sequentially in configured order so results, diagnostics, - costs, and feedback ordering remain deterministic; consider validator - concurrency only in response to measured latency. -- Continue running independent applicable validators after one validator - execution failure so the attempt retains as much useful diagnostic - information as practical. Do not present validator operational failures as - defects in the producer candidate and do not include them in corrective - feedback. -- Distinguish three terminal conditions and make their policies configurable - at a coherent pipeline or binding scope: - - **producer structural failure:** PromptKit could not return a usable - structured candidate after its repair budget. Default to `fail_run`; an - allowed alternative may record a terminal stage or lane rejection where - execution can safely continue, but may not accept the invalid output; - - **semantic rejection:** one or more validators completed and rejected the - candidate. Default to `fail_run` after corrective stage retries are - exhausted; allow an explicit alternative that records the existing - rejected-output outcome without advancing that output; - - **validator execution failure:** a validator could not produce a valid - decision because of generation, structural-output, transport, or internal - failure. Default to a genuine warning and an explicitly recorded - `validation_incomplete` or equivalent degraded state while allowing the - candidate to continue; allow strict configuration to fail the run instead. -- An LLM-backed validator uses the same scheduled PromptKit boundary as every - other LLM-backed module. Its own response may use PromptKit's bounded - structural repair. Distinguish its possible output states: - - output rejected by PromptKit's structural contract should consume only the - validator prompt's configured PromptKit repair budget; - - output that is structurally valid but violates a deterministically - checkable validator-result invariant should be classified as a validator - execution failure; - - output that satisfies the complete validator-result contract is the - validator's decision, even though an LLM judgment may remain imperfect. - Automatically judging that judgment would require another semantic - validator and is outside this feature. - If the validator cannot return a contract-valid decision, do not recursively - create another Notarius semantic-validation loop around it. Apply the - configured validator-failure policy. The default warning must identify the - validator and affected stage without exposing sensitive content. -- Separate validator execution retry from producer correction. A transient - validator operational failure must not automatically discard and regenerate - an otherwise usable producer candidate. Any bounded retry of the validator - itself should reuse that same immutable candidate and remain subordinate to - PromptKit and provider retry behavior. -- Preserve attempt-level provenance, cumulative token usage, validator - outcomes, aggregated correction feedback, and terminal policy decisions in - the debug and manifest models without copying raw source material into - ordinary errors or durable summaries. -- Define terminal-outcome precedence. A semantic rejection dominates a - validator execution failure for the same candidate: use the completed - rejections to correct the producer while separately recording incomplete - validation. If a later candidate has no semantic rejection but one validator - still fails, apply the configured validator-failure policy to that candidate. - Never allow a known semantic rejection to become accepted through a - warn-and-continue setting, and never accept a structurally invalid producer - response. Permissive policy may preserve a rejected-output outcome or accept - a structurally valid candidate with explicitly incomplete validation; it may - not relabel known-invalid output as approved. - -Before implementation, record the generic validation and retry state machine -in an ADR. The ADR should own the separation between PromptKit repair and -Notarius correction, use of the existing stage-retry budget, reconstruction of -correction conversations, all-applicable-validator aggregation, deterministic -validator ordering, non-recursive validator failure handling, outcome -precedence, default fail-open/fail-closed choices, configurable terminal -policies, and provenance and sensitive-data constraints. A dependency-upgrade -ADR is not needed for PromptKit v0.8.0 itself. Current behavior remains -authoritative until the validation ADR is implemented and the canonical -architecture, configuration, operations, and internal documentation are -updated. - -### 3. D&D Combat Scene Semantic Validation +### D&D Combat Scene Semantic Validation - Add an optional production LLM-backed D&D validator that determines whether proposed scene boundaries and classifications represent substantive active @@ -172,7 +63,7 @@ updated. the chunker or otherwise changes stage ownership or the durable chunk-plan contract. -### 4. Warning Signal And Presentation Reform +### Warning Signal And Presentation Reform - Audit every warning producer and representative successful runs. Ordinary success producing dozens of warnings is a failed operator experience: the diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index f3683f39..54b0600d 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,782 +1,731 @@ -# PromptKit v0.8.0 Upgrade Implementation Plan +# Feedback-Aware Validation Retry Implementation Plan -## Purpose +## Status -Implement the target state defined by the -[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md): adopt the useful PromptKit -v0.6.0, v0.7.0, and v0.8.0 changes; enable one bounded structural correction -by default; expose pipeline and binding overrides; preserve safe provider -diagnostics; and keep PromptKit behind Notarius's transport-neutral LLM -boundary. +Ready for implementation. This plan implements the target state defined by +[Feedback-Aware Stage Validation Retries](validation-retries.md). Its stages +are ordered dependencies, and each stage is intentionally scoped for one +gpt-5.6-terra implementation prompt. -This plan is ordered. Each numbered stage is one implementation prompt for a -gpt-5.6-terra coding agent. Complete and validate one stage before beginning -the next. Read `docs/development.md` and every policy under `docs/policy/` at -the start of each stage, inspect the current code and tests named by that -stage, preserve unrelated worktree changes, and update current-behavior -documentation in the same stage as the behavior it describes. +The feature roadmap owns product intent and durable policy. This document owns +implementation order, concrete boundaries, and stage-level verification. If a +conflict is discovered, preserve the feature roadmap and the repository +policies, stop the affected stage, and revise this plan rather than silently +choosing a different architecture. -Do not retire this plan or `promptkit-v0.8.md` during implementation. Keep both -until the completed work has passed a separate review. Do not implement the -future Notarius semantic-validation retry loop, D&D combat-scene validator, or -warning redesign as part of this plan. +## Settled Decisions -## Decisions Fixed For Implementation +The implementation agent must treat these decisions as fixed: -- Pin `gitea.maximumdirect.net/eric/promptkit` v0.8.0 directly, with no - `replace`, workspace dependency, or vendored source. -- Every maintained eligible production prompt defaults to exactly one - PromptKit structural repair attempt. -- Add the exact configuration key - `structured_output_repair_attempts` at pipeline scope and on LLM-backed - module and validator bindings. -- Effective precedence is binding value, then pipeline value, then the prompt's - declared `repair_attempts` value. Omission inherits; explicit zero disables - structural repair at that scope. -- Accepted values are integers from zero through three. Explicit null and - non-integer values are invalid. An explicit binding value on a deterministic - module or validator is invalid. A pipeline value is applied only to selected - LLM-backed bindings and does not make deterministic bindings invalid. -- Keep file configuration version 4. This is an additive pre-release field and - does not require parallel versioned behavior. -- Use `StructuredOutputRepairAttempts *int` for presence-aware internal Go - fields. Clone pointers at every ownership boundary. -- A configured override never replaces schema identity, output format, or - validation mode. The PromptKit adapter calls `InspectPrompt`, copies the - complete normalized prompt-owned output contract, changes only - `RepairAttempts`, and supplies the complete replacement on `RunRequest`. - Do not add an inspection cache initially. -- PromptKit repair is internal to one `CompleteStructured` call and does not - consume or replenish a binding's existing `retries` budget. -- Add `RepairAttempts int` to Notarius's structured-completion response. It is - the actual corrective-call count reported by PromptKit; token usage remains - PromptKit's cumulative usage and must not be summed again. -- A valid repaired response is successful and produces no warning solely - because repair occurred. Exhausted structural validation maps to - `ErrInvalidStructuredOutput` with the final candidate and debug material - retained. -- Add an application-owned generation-error sentinel and typed status-bearing - error. PromptKit error types must not cross `internal/framework/llm`. -- HTTP status may appear in the application-owned generation error. Provider - code, type, and message are excluded from ordinary errors, warnings, - manifests, cache, and checkpoint identity; they may appear only in an - explicitly requested debug trace after Notarius redaction. -- Profile inheritance is owned entirely by PromptKit. Notarius passes sources - through, inspects and records the resolved target, and does not parse or merge - `base_profile` itself. -- PromptKit's built-in `rakestrawhome` backend and - `rakestrawhome-gemma-4-31b` profile are available generically. Notarius does - not register, shadow, or select them by default. -- Missing optional credential environment values are allowed to reach the - provider without `Authorization`; Notarius does not recreate v0.5.0's local - failure or add provider-specific authentication logic. -- No dependency-upgrade ADR is required. Update architecture only with the - durable ownership distinction between PromptKit structural repair and - Notarius stage/semantic validation policy. +- PromptKit v0.9.0 is the minimum and exact supported PromptKit release for + this work. Notarius uses `RunRequest.AppendedMessages`; it does not create + paired correction manifests or bypass PromptKit preparation and execution. +- A correction request reconstructs the ordinary initial request and appends + exactly two messages: the latest exact producer response as `assistant`, + followed by one deterministic aggregate correction request as `user`. + Earlier correction turns never accumulate. +- A correction-capable producer supplies the exact single LLM response that + directly controlled the candidate. The initial protocol does not synthesize + a model-facing projection of a compound artifact. +- The only supported producer correction protocol is + `single_response_v1`. An empty protocol means unsupported. Configuration + rejects semantic-retry workflows for LLM-backed compound producers or any + other producer unable to satisfy `single_response_v1`. +- The existing producer binding `retries` value is the sole outer stage + budget. PromptKit repair attempts and validator execution retries are + independent budgets. +- Terminal policy is configured as pipeline defaults with field-by-field + producer-binding overrides. Validators never own candidate disposition. +- Application defaults are `fail_run` for exhausted producer structural + failure, `fail_run` for exhausted semantic rejection, and `warn_continue` + for exhausted validator execution failure. +- All applicable validators run sequentially in configured order. Rejections + and validator failures are aggregated; failures are not represented to the + producer as candidate defects. +- Deterministic producers do not consume retries after rejection. An + LLM-backed attempt that took a deterministic fast path and produced no model + response is likewise not correctable for that candidate. +- Known rejected or structurally invalid output never advances. A candidate + accepted under `validator_failure: warn_continue` advances with explicit + incomplete-validation provenance but is not checkpointed. +- The D&D combat-scene validator and broader warning-system redesign remain + out of scope. -## Stage 1: Upgrade The Dependency And Establish A Clean v0.8.0 Baseline ✅ +## Cross-Stage Implementation Rules -### Goal +Apply these rules in every stage: -Move the repository to PromptKit v0.8.0, resolve source-compatibility issues, -and establish a passing baseline before adopting new behavior. - -### Implementation - -1. Re-read the upstream v0.6.0, v0.7.0, and v0.8.0 release guides and the - v0.8.0 package consumer and format documentation. Treat the pinned v0.8.0 - tag, not the sibling checkout's moving branch, as authoritative. -2. Update `go.mod` and `go.sum` to PromptKit v0.8.0 and run `go mod tidy` with - `GOWORK=off`. -3. Compile before making compatibility edits. Correct only actual source or - behavior incompatibilities. In particular: - - convert any positional `promptkit.Profile` or - `promptkit.OpenAICompatibleProfileConfig` literals to keyed literals; - - confirm Notarius does not register the newly reserved `rakestrawhome` - backend ID; and - - preserve `PrepareExecution`/`Details`/`RunPrepared` snapshot ownership, - `Discard`, session forwarding, reasoning override, profile preflight, - and capacity adaptation. -4. Change `promptKitBuiltinProfileCatalogID` in - `internal/framework/llm/promptkit_profile_fingerprint.go` from the v0.5.0 - catalog marker to an opaque v0.8.0 marker. Do not hash PromptKit internal - files or include catalog content in manifests. -5. Update `docs/integrations/pkg-promptkit.md` to pin and link v0.8.0 and to - state that this stage still leaves the production prompt-declared repair - budget at its current value. Do not document later configuration or default - behavior before it exists. -6. Update only those existing tests whose public PromptKit types or stable - v0.8.0 behavior genuinely changed. Do not rewrite tests merely to match - upstream diagnostic wording. - -### Tests And Validation - -```sh -GOWORK=off go mod tidy -diff -GOWORK=off go test ./internal/framework/llm ./internal/cli -GOWORK=off go test ./... -GOWORK=off go vet ./... -GOWORK=off go build ./cmd/notarius -git diff --check -``` - -### Acceptance Criteria - -- `go list -m gitea.maximumdirect.net/eric/promptkit` reports v0.8.0. -- There is no PromptKit `replace`, active Go workspace dependency, or vendor - tree. -- The adapter still uses one frozen prepared execution and all existing LLM - tests pass. -- Checkpoint profile identity includes the v0.8.0 built-in catalog marker. -- Current integration documentation pins v0.8.0 without claiming that - later stages are already active. -- The full ordinary test suite, vet, and command build pass. - -## Stage 2: Verify v0.6.0 Compatibility And Hardening ✅ - -### Goal - -Audit Notarius's assets and boundary values against PromptKit v0.6.0's stricter -source, path, endpoint, JSON, and cancellation contracts, fixing only concrete -incompatibilities. - -### Implementation - -1. Inspect `internal/framework/llm/asset_registry.go`, prompt/profile source - composition, all registered asset roots, the conventional local backend, - and their focused tests. -2. Exercise every production asset registry through PromptKit engine - construction and the existing production composition tests. Confirm that: - - YAML IDs and versions, not filenames, select definitions; - - every `content_file` path is exact, relative, contained, and points to a - regular embedded file; - - every schema and JSON asset is one complete JSON value; - - every current output contract is valid under v0.8.0; and - - unrelated malformed definitions do not create a second Notarius identity - or fallback mechanism. -3. Review local endpoint parsing and validation. Retain a narrower Notarius - rule only if it has independent application value; otherwise rely on - PromptKit's absolute HTTP/HTTPS URL contract. Never accept a value that the - adapter will later reject. -4. Review conversion of Notarius variables, inputs, profile extras, and debug - values at the adapter boundary for PromptKit's bounded JSON-compatible-value - rules. Do not add a second generic JSON walker or duplicate upstream numeric - limits. -5. Verify cancellation and deadline identity through existing adapter tests. - Add or refine one focused regression only if Notarius currently destroys an - `errors.Is`-relevant context or transport error that the application owns. -6. Do not add a cross-operation schema cache, artifact cache, provider-body - reader, or duplicate JSON framing validation; v0.6.0 owns those mechanisms. - -### Tests And Validation - -Run the focused asset, profile-source, and adapter packages, then the ordinary -and race-enabled suites: - -```sh -GOWORK=off go test ./internal/framework/llm ./internal/cli -GOWORK=off go test ./... -GOWORK=off go test -race ./... -git diff --check -``` - -Tests must remain offline and should validate Notarius's assembled boundary, -not reproduce PromptKit's internal path, JSON-depth, or response-size matrices. - -### Acceptance Criteria - -- Every maintained embedded prompt, schema, and fallback profile can be loaded - through the assembled v0.8.0 engine. -- Current local endpoint and JSON-compatible values either satisfy the stricter - upstream contract or fail during preparation with safe diagnostics. -- No duplicate PromptKit-owned cache, JSON, or response-bound mechanism is - introduced. -- Cancellation and deadline behavior remains discoverable at the Notarius - boundary. -- Ordinary and race-enabled tests pass. - -## Stage 3: Adopt Profile Inheritance, Rakestrawhome, And Optional Credentials ✅ - -### Goal - -Make the useful PromptKit v0.7.0 profile and backend behavior work through -Notarius's existing generic profile boundary without adding provider-specific -composition logic. - -### Implementation - -1. Inspect `promptkit_profiles.go`, `asset_registry.go`, profile fingerprinting, - CLI profile preflight, profile provenance recording, and their tests before +1. Read the feature roadmap, `docs/development.md`, all `docs/policy/` + documents, and the focused current documentation named by the stage before editing. -2. Add an offline integration test using a temporary operator profile source - whose leaf uses `base_profile`. Prove that: - - preflight reports the leaf ID; - - the effective backend, model, reasoning, and other inherited values match - the resolved PromptKit target; - - execution uses the same resolved target as inspection; and - - a missing parent or cycle fails before provider generation with a safe - profile-load diagnostic. - Do not duplicate PromptKit's entire field-by-field merge test matrix. -3. Add a checkpoint-safety test showing that changing a parent definition in - an operator profile directory changes Notarius's profile-source fingerprint - while profile content and paths remain absent from the fingerprint value. - Retain the v0.8.0 catalog marker as coverage for built-in-parent changes. -4. Verify `rakestrawhome-gemma-4-31b` through the ordinary profile inspector. - Assert its selected backend reaches Notarius's application-owned inspection - and provenance fields. Use a fake PromptKit client or transport if execution - coverage is needed; never contact the live service or require credentials. -5. Verify that Notarius registers no `rakestrawhome` override and that the - existing `local` registration remains independent. -6. Add one `httptest`-backed adapter integration test for a filesystem profile - with a missing optional `api_key_env`. The request must reach the test server - without an `Authorization` header. Add a focused in-memory PromptKit profile - test for `APIKeyRequired` only if needed to prove Notarius preserves upstream - preflight behavior; do not expose a new operator profile API. -7. Keep `assets/dnd/profiles/dnd-extraction.yaml` standalone and unchanged. No - matching v0.8.0 built-in profile owns its `openai/gpt-5.6-luna` target. -8. Update the current profile-source, deployment, and pinned-integration - sections in `docs/config.md`, `docs/operations.md`, - `docs/internal/llm.md`, and `docs/integrations/pkg-promptkit.md`. Link to the - pinned PromptKit format rules for inheritance. Explain that filesystem - profiles cannot express PromptKit's in-memory `APIKeyRequired` field and - that an optional missing credential may result in a provider 401/403. +2. Inspect current code and tests rather than assuming paths or private helper + layouts. Preserve unrelated work and existing public ordering, + cancellation, scheduler, checkpoint, and sensitive-data invariants. +3. Use transport-neutral types outside `internal/framework/llm`. PromptKit + types must not escape the adapter boundary. +4. Keep correction material attempt-local. Do not put raw assistant responses + or correction text in ordinary errors, warnings, manifests, receipts, + checkpoints, caches, or default debug summaries. +5. Add the smallest durable tests that protect the stage's contracts. Do not + duplicate PromptKit's own prepared-execution, copying, hashing, capacity, + role-validation, or structural-repair test matrix. +6. Use `gofmt` on changed Go files. Run the focused checks listed for the + stage and fix failures before stopping. Do not proceed into the next stage + in the same prompt. +7. Do not delete this plan or the feature roadmap during implementation. They + are retired only after a final review confirms the complete target state. -### Tests And Validation +## Contract Names And Bounds -```sh -GOWORK=off go test ./internal/framework/llm ./internal/core/config ./internal/cli -GOWORK=off go test ./... -GOWORK=off go test -race ./internal/framework/llm ./internal/cli -git diff --check -``` +Use one vocabulary consistently across packages. Exact private helper names +may follow local conventions, but the following contracts and values are not +open for redesign: -### Acceptance Criteria +- `CorrectionProtocol` has only the empty unsupported value and + `single_response_v1`. +- An application-owned `SemanticCorrection` carries owned + `AssistantResponse []byte` and `UserGuidance string` values. It is exposed + on chunk, typed extract, typed merge, typed normalize, and structured + completion requests as an optional pointer. +- An application-owned `ModelCandidate` carries the owned exact response bytes + and `CorrectionProtocol`. It is exposed on the corresponding producer + results as an optional pointer and is never serialized as part of a durable + artifact. +- `ValidationResult` gains optional `CorrectionGuidance`. Operator-facing + `Message` is not copied into this field implicitly. +- Each reason code is valid UTF-8, nonblank after trimming, and at most 128 + bytes. Each validator-supplied correction guidance value is valid UTF-8, + nonblank after trimming, and at most 4 KiB. +- The exact defective assistant response may be at most 1 MiB. The aggregate + user correction message may be at most 64 KiB. The combined appended + content may therefore be at most 1,114,112 bytes. Reject invalid UTF-8, + empty/whitespace-only content, or over-limit content; never truncate it. +- Invalid producer-owned correction material is a framework contract error. + It is not a validator failure and cannot be converted to `reject_output` or + `warn_continue`. +- Aggregate correction entries are ordered by validator order, contain the + stable reason code plus guidance, and collapse only byte-identical duplicate + `(reason_code, correction_guidance)` pairs while retaining first occurrence. + A rejection without guidance receives bounded generic guidance based only on + its reason code. +- Terminal actions are closed enums: structural failure and semantic + rejection accept `fail_run` or `reject_output`; validator failure accepts + `warn_continue` or `fail_run`. -- Inherited operator profiles resolve identically during preflight and - execution, with the leaf ID and effective target kept distinct. -- Parent changes invalidate checkpoint reuse without leaking profile content or - paths. -- Rakestrawhome is available through generic PromptKit profile handling and is - not selected by default or registered by Notarius. -- Missing optional credentials omit authorization and reach the controlled - test provider; explicitly required credentials retain upstream behavior. -- Current documentation accurately describes the implemented profile and - credential behavior without duplicating PromptKit's merge algorithm. - -## Stage 4: Adapt Structured Generation Errors Safely ✅ +## Stage 1 — Adopt PromptKit v0.9.0 ### Goal -Use PromptKit v0.7.0's structured generation errors for stable status -classification and debug-only provider diagnostics without leaking PromptKit -types or sensitive provider text. +Upgrade the dependency without changing Notarius stage behavior, prove the +existing integration remains compatible, and make the new upstream request +primitive available to later stages. -### Implementation +### Work -1. In `internal/framework/contracts`, add: - - `ErrLLMGeneration` as the provider-neutral generation-failure sentinel; - - an application-owned `LLMGenerationError` with private status and safe - diagnostic fields, `Error`, `Unwrap`, and `StatusCode` methods; and - - a constructor that accepts a nonnegative status and an already-redacted - diagnostic. Status zero means no HTTP status was available. - Ordinary callers may inspect status with `errors.As` and category with - `errors.Is`, but cannot obtain provider code, type, or message from the - error. -2. Add an application-owned `LLMDebugProviderError` with `status_code`, `code`, - `type`, and `message` fields, referenced optionally from - `LLMDebugResponse`. This is debug material, not a manifest or durable public - artifact contract. -3. In `PromptKitClient.CompleteStructured`, preserve precedence in this order: - caller context cancellation/deadline, PromptKit capacity error, structured - PromptKit generation error, then other PromptKit generation failures. - Map every generation failure to `ErrLLMGeneration`; map - `*promptkit.GenerationError` to `LLMGenerationError` with its status. - Never wrap or return the PromptKit error value itself. -4. Keep the ordinary diagnostic limited to PromptKit's safe default error - formatting after bearer and known-credential redaction. Do not append - `ProviderCode`, `ProviderType`, or `ProviderMessage` to it. -5. For an explicitly requested debug path, preserve prepared prompt details and - attach the PromptKit provider code, type, and message after: - - reading only the selected prepared target's `APIKeyEnv`, if any, to obtain - the exact known credential solely for redaction; - - applying `RedactSecrets` and the existing bearer/key-pattern redaction; - - retaining PromptKit's already-normalized bounds; and - - discarding the credential value immediately rather than storing it. - Do not scan unrelated environment variables. -6. Return prompt/debug material alongside the error so the existing debug LLM - wrapper can persist it only when debug recording is enabled. Confirm that - provider fields do not appear in ordinary error text, warnings, manifests, - cache, checkpoint data, or a run without debug output. -7. Refactor error mapping into small helpers if needed to keep - `CompleteStructured` readable; do not create provider-specific policy in - modules or the pipeline runner. -8. Update the error and observability sections of `docs/internal/llm.md` and - `docs/integrations/pkg-promptkit.md`. Keep operator disclosure rules in - `docs/operations.md` concise and link to the internal boundary where useful. +- Update `go.mod` and `go.sum` to PromptKit v0.9.0 with `go get` and + `go mod tidy`. Accept the catalog modules selected transitively by PromptKit; + do not import or register them directly. +- Review every Notarius `promptkit.RunRequest` literal and retain keyed form. + Verify production prompt roles are limited to `system` and `user`. +- Update the conservative built-in-profile checkpoint marker in + `internal/framework/llm/promptkit_profile_fingerprint.go` from v0.8.0 to + v0.9.0. Do not duplicate the separately versioned catalog module versions. +- Update `docs/integrations/pkg-promptkit.md` only to describe the newly + implemented dependency version and unchanged current integration boundary. + Include the strict role vocabulary and external catalog ownership, but do + not document Notarius correction behavior yet. -### Tests And Validation +### Verification -- Use `httptest.Server` to return representative structured 400 and 503 - responses. Assert `errors.Is(ErrLLMGeneration)`, `errors.As` to the - application-owned type, and the exact status without asserting complete - human wording. -- Include a provider message containing the selected test credential and a - bearer-shaped value. Verify both are absent from the ordinary error and - debug artifact, while a non-sensitive marker appears only in the requested - debug trace. -- Retain existing capacity and context tests to prove their more specific - classifications still win. +- Run `go test ./internal/framework/llm ./internal/cli`. +- Run `go test ./...`, `go test -race ./internal/framework/llm`, `go vet ./...`, + and `go build ./cmd/notarius`. -```sh -GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline ./internal/cli -GOWORK=off go test ./... -GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline -git diff --check -``` +### Exit Criteria -### Acceptance Criteria +Notarius builds and passes its suite on PromptKit v0.9.0, its compatibility +document and checkpoint marker name v0.9.0, and no Notarius package directly +depends on either upstream catalog module. This stage is one Terra prompt. -- PromptKit generation errors never escape the adapter error chain. -- All generation failures match `ErrLLMGeneration`; structured non-success - responses expose only application-owned HTTP status to ordinary callers. -- Provider code, type, and message are available only in an explicitly - requested, redacted debug trace. -- Capacity and context classifications remain unchanged and more specific. -- Security tests prove selected credentials and bearer tokens are not leaked. - -## Stage 5: Add Adapter-Level Structured Repair Support ✅ +## Stage 2 — Record The Architecture And Add Transport-Neutral Contracts ### Goal -Teach the transport-neutral completion boundary and PromptKit adapter to apply -an optional repair override and report actual repair behavior, without yet -exposing the setting in pipeline configuration. +Record the durable decision and introduce owned correction/candidate types +without changing runtime retry behavior. -### Implementation +### Work -1. Add `StructuredOutputRepairAttempts *int` to - `contracts.StructuredCompletionRequest`. Copy the pointed-to value wherever - requests are cloned or retained. -2. Add `RepairAttempts int` to `contracts.StructuredCompletionResponse`. It is - the actual number of corrective generation calls, not the configured budget - and not the number of total candidates. -3. Validate a non-nil request value as zero through three at the adapter - boundary so programmatic callers cannot bypass later file/config validation. -4. When the request value is nil, leave `promptkit.RunRequest.Validation` nil - so the prompt's complete contract remains authoritative. -5. When the value is non-nil: - - call `Engine.InspectPrompt(ctx, promptID, promptVersion)`; - - copy `PromptInspection.OutputContract` by value; - - replace only `RepairAttempts`; - - pass the complete copied contract as `RunRequest.Validation`; and - - prepare and execute exactly as before. - Do not infer or hard-code schema paths, validation modes, or formats. Do not - cache inspection in this stage. -6. Map `result.Validation.RepairAttempts` to the response and leave - `result.Usage` cumulative values unchanged. The existing debug validation - object and prepared output contract should show actual and configured values - respectively. -7. Preserve result semantics: - - valid initial and repaired candidates decode normally; - - repair exhaustion returns the final raw candidate/debug material with an - error matching `ErrInvalidStructuredOutput`; - - explicit empty or whitespace-only content follows PromptKit validation; - - missing/null/non-string content remains a generation/provider failure; - - corrective-call generation errors use Stage 4's application-owned mapping; - and - - context cancellation wins at every error boundary. -8. Keep `CompleteStructured` and its helpers provider neutral outside this - adapter package. Do not expose PromptKit validation or inspection types. -9. Update only the adapter-owned repair behavior in `docs/internal/llm.md` and - `docs/integrations/pkg-promptkit.md`. State that public pipeline configuration - and the production default are added by later stages of this plan. +- Add ADR-0014 under `docs/adr/` using the repository ADR format. Record the + three distinct retry budgets, complete validator aggregation, the fresh + two-message correction protocol, exact-response producer capability, + non-recursive validator failure handling, terminal-policy ownership and + defaults, checkpoint conservatism, and sensitive-data constraints. +- Add `CorrectionProtocol`, `SemanticCorrection`, and `ModelCandidate` to the + domain-neutral framework contracts, with constructors/clone helpers that + validate the settled bounds and defensively copy bytes. +- Add optional correction input to `ChunkRequest`, + `TypedExtractionRequest`, `TypedMergeRequest`, and + `TypedNormalizeRequest`. +- Add optional model-candidate output to `ChunkPlanResult`, + `TypedExtractionResult`, `TypedMergeResult`, and + `TypedNormalizeResult`. +- Add `CorrectionGuidance` to `ValidationResult` and centralize validation of + reason codes and corrective guidance at the framework boundary. +- Update type-erasure adapters, stored fakes, and clone paths so the new values + retain ownership and cannot alias caller buffers. Do not declare production + module capabilities or use the fields in the runner yet. -### Tests And Validation +### Verification -Add adapter-level behavioral tests using a deterministic fake PromptKit LLM: +- Add contract-level tests for accepted values, UTF-8 and size rejection, + defensive copying, nil behavior, and typed-erasure preservation. +- Run `go test ./internal/framework/contracts ./internal/framework/pipeline`. +- Run `go test ./...`. -- nil override uses the prompt declaration; -- explicit zero overrides a positive prompt declaration without dropping its - JSON Schema contract; -- explicit one turns an invalid first candidate followed by a valid candidate - into one successful response with the final raw bytes, actual repair count - one, and cumulative usage; -- repair exhaustion returns the final candidate and validation diagnostics as - `ErrInvalidStructuredOutput`; -- explicit empty content is eligible for repair; -- a corrective generation failure maps through Stage 4; and -- invalid direct values below zero or above three fail before provider work. +### Exit Criteria -Do not assert PromptKit's exact assistant/user correction prose or copy its -full internal repair matrix. +The generic contracts can safely carry correction input and exact candidate +material across every producer stage, no PromptKit type crosses the boundary, +and existing runtime behavior is unchanged. This stage is one Terra prompt. -```sh -GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline -GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline -GOWORK=off go test ./... -git diff --check -``` - -### Acceptance Criteria - -- The adapter changes only repair count when applying a request override. -- Nil and explicit zero remain distinct. -- Repaired success returns final raw output, cumulative usage, and actual count - without a warning. -- Exhaustion, empty content, corrective generation failure, and cancellation - match the target semantics. -- No PromptKit type crosses the LLM package boundary. - -## Stage 6: Propagate Repair Policy Through Framework Requests ✅ +## Stage 3 — Add Validation Policy Configuration And Resolution ### Goal -Carry an optional effective repair budget from each resolved stage or validator -binding to its module request without changing public file configuration yet. +Implement strict configuration, inheritance, identity, and provenance for +terminal validation policy before the runner consumes it. -### Implementation +### Work -1. Add `StructuredOutputRepairAttempts *int` alongside `LLMProfile` to every - stage request that can belong to an LLM-backed binding: - - `ParseRequest`; - - `ChunkRequest`; - - `TypedExtractionRequest`; - - `TypedMergeRequest`; - - `TypedNormalizeRequest`; - - `OutputRequest`; - - `TypedValidationRequest`; - - `ChunkValidationRequest`; and - - `SerializedValidationRequest`. -2. Add the same optional field to the erased/internal request carriers used by - registry builders, preparation, runner stage attempts, validator targets, - retry closures, and debug wrappers. Copy pointer values; never share a - mutable pointer owned by configuration. -3. At every runner stage invocation, obtain the value from the exact resolved - producer binding. At every validator invocation, obtain it from that exact - resolved validator binding. Do not use the producer's value for a validator - or vice versa. -4. Ensure all retry attempts for the same binding receive the same effective - structural-repair value. Do not decrement it in Notarius; PromptKit owns the - inner budget independently on each `CompleteStructured` call. -5. Extend `semanticreconcile.Request` with the optional field and carry it into - each generic reconciliation completion. A batched reconciliation may make - several completion calls; each call receives the same effective budget. -6. Update registry erasure/adaptation code for typed merge, normalize, and - validation requests so no field is lost. Preserve input/output support even - though current production input and output modules are deterministic. -7. Add focused framework tests for one chunk producer, one extraction - producer, one normalizer, and one LLM-backed validator. Verify exact pointer - value propagation and separation between producer and validator settings. - Do not add repetitive tests for every generic adapter. +- Add optional `validation_policy` to a pipeline profile and to chunk, + extract, merge, and normalize producer bindings. Use pointer-backed override + fields so omission is distinguishable from an explicit value. +- Reject an explicitly null policy object, null policy fields, duplicate or + unknown fields, and values outside the settled enums. Keep the current file + configuration version. +- Reject `validation_policy` on input, output, and validator bindings. Reject + a binding-level explicit `producer_structural_failure` value on a + deterministic producer; a pipeline-level structural default remains valid + because a pipeline may contain LLM-backed producers. +- Resolve each policy field independently in binding, pipeline, application + default order. Store one detached concrete effective policy for chunk and + for each lane's extract, merge, and normalize stage; do not leave runtime + inheritance to the runner. +- Include both configured overrides and effective policies at their existing + appropriate configuration/provenance boundaries. Include the effective + values in resolved-pipeline digest and checkpoint identity. Preserve + deterministic serialization and redaction. +- Update `examples/dnd-minimal/config.yml` and + `examples/dnd-complete/config.yml` only if their current shape must change to + remain valid; do not add redundant explicit defaults merely to demonstrate + the feature. -### Tests And Validation +### Verification -```sh -GOWORK=off go test ./internal/framework/contracts ./internal/framework/pipeline ./internal/framework/semanticreconcile -GOWORK=off go test -race ./internal/framework/pipeline ./internal/framework/semanticreconcile -GOWORK=off go test ./... -git diff --check -``` +- Add focused parsing tests for omission versus null, unknown/duplicate + fields, invalid placements, and invalid enums. +- Add resolution tests for field-by-field inheritance and mixed overrides. +- Add digest, clone, JSON/YAML round-trip, and redacted-effective-config tests. +- Run `go test ./internal/core/config ./internal/framework/pipeline ./internal/cli`. -### Acceptance Criteria +### Exit Criteria -- Every stage and validator request can carry a detached optional repair value. -- The runner sources the value from the exact resolved binding. -- Producer and validator values cannot overwrite one another. -- Stage retries reuse but do not mutate or consume the inner repair budget. -- Semantic reconciliation forwards the budget to every one of its completion - calls. -- Existing behavior remains unchanged while all values are nil. +Every producer stage receives one immutable effective policy, invalid +configuration fails before execution, and policy changes invalidate resolved +identity. The runner still follows its old behavior until later stages. This +stage is one Terra prompt. -## Stage 7: Forward Repair Policy From Every LLM-Backed Module ✅ +## Stage 4 — Declare And Validate Producer Correction Capability ### Goal -Complete the internal end-to-end path by having every production LLM-backed -module forward its stage request value to `CompleteStructured`. +Make correction support an explicit module contract and prepare the framework +to reject impossible workflows. -### Implementation +### Work -1. Inventory every production `CompleteStructured` call with code search before - editing. The expected current owners include: - - `dnd/scenes` chunking; - - the combat-turn, enemy-event, item-occurrence, item-registry, - location-occurrence, location-registry, NPC-occurrence, NPC-registry, - scene-description, and spell extractors; and - - generic semantic reconciliation used by the item, location, and NPC - registry normalizers. - Reconcile this list with the actual repository; do not omit a newly added - production caller merely because it is not named here. -2. In each direct caller, set - `StructuredCompletionRequest.StructuredOutputRepairAttempts` from the - corresponding stage request. Clone the pointer or use a small shared helper - if that reduces repeated ownership mistakes without moving domain logic. -3. Ensure D&D registry normalizers pass their typed normalize request value into - `semanticreconcile.Request`, and that the generic engine forwards it as - established in Stage 6. -4. Update existing module prompt-mapping tests that already inspect a captured - structured-completion request to assert the new field. Do not create a new - one-test-per-module suite solely to memorialize field plumbing; rely on the - existing request-contract tests plus a final complete call-site audit. -5. Search again after editing for production `CompleteStructured` calls and - verify each either forwards the field or documents why it cannot receive a - pipeline binding. Test-only fakes need only preserve the field when their - contract test depends on it. -6. Do not set a module-specific fallback value. Nil must reach the adapter so - the prompt declaration remains authoritative. +- Extend `pipeline.ModuleSpec` with `CorrectionProtocol`; normalize, clone, + validate, and include it in relevant registry/spec fingerprints. +- Permit `single_response_v1` only for LLM-backed chunk, extract, merge, or + normalize modules. Reject it for deterministic, input, output, or validator + specs. Empty remains the default unsupported value. +- Carry the selected protocol into resolved and prepared producer metadata and + checkpoint fingerprints. +- Reject positive `retries` on deterministic validator bindings during + resolution. LLM-backed validator retries remain valid and independent of + producer capability. -### Tests And Validation +### Verification -Run focused D&D and semantic-reconciliation packages, then the full suite: +- Add spec normalization/clone tests, invalid stage/class combinations, + validator retry-class tests, and preparation tests for supported and + unsupported producer workflows. +- Run `go test ./internal/framework/pipeline ./internal/core/config`. -```sh -GOWORK=off go test ./internal/modules/dnd/... ./internal/framework/semanticreconcile -GOWORK=off go test -race ./internal/modules/dnd/... ./internal/framework/semanticreconcile -GOWORK=off go test ./... -git diff --check -``` +### Exit Criteria -### Acceptance Criteria +Capability is explicit, fingerprinted, and testable, while existing production +pipelines remain executable pending their migrations. No dormant feature gate +or unused preparation check is introduced. This stage is one Terra prompt. -- Every production LLM-backed completion receives the exact stage or validator - repair value. -- No module invents a default or imports PromptKit. -- Registry normalizers preserve the value through semantic reconciliation. -- Existing request-contract tests remain concise and pass. -- A final call-site audit finds no silent production omission. - -## Stage 8: Add The Public Repair Configuration Contract ✅ +## Stage 5 — Adapt Corrections Through PromptKit ### Goal -Add presence-aware pipeline and binding configuration for -`structured_output_repair_attempts` without yet changing runtime resolution. +Map the transport-neutral correction contract onto PromptKit v0.9.0 without +changing ordinary requests. -### Implementation +### Work -1. Add `StructuredOutputRepairAttempts *int` to - `pipeline.PipelineProfile` and `pipeline.ModuleBinding`, using - `json:"structured_output_repair_attempts,omitempty"`. -2. Add presence-aware YAML support at pipeline and object-binding scope: - - exact key `structured_output_repair_attempts`; - - integer values zero through three; - - explicit null, non-integer, and out-of-range values rejected with scoped - diagnostics; and - - scalar shorthand bindings continue to omit the binding override. - Preserve file configuration version 4. -3. Update every configuration clone, conversion, redaction, summary, and JSON - round-trip carrier. Copy pointers by value into newly allocated storage so - parsed, configured, and redacted values do not alias. -4. Preserve omission versus explicit zero through YAML parsing, profile - inheritance, module-binding object form, and JSON round trips. Keep scalar - shorthand bindings equivalent to omission. -5. Do not add a top-level `promptkit.repair_attempts` setting or CLI override. -6. Add concise parser and ownership tests. Defer execution-class checks, - effective precedence, resolved digests, and runtime forwarding to Stage 9, - where module metadata is available. +- Add optional `SemanticCorrection` to `StructuredCompletionRequest` and its + debug-safe request representation. +- In `PromptKitClient.CompleteStructured`, validate and defensively copy the + correction, then map it to exactly two `promptkit.RenderedMessage` values in + `RunRequest.AppendedMessages`, using `promptkit.RoleAssistant` followed by + `promptkit.RoleUser`. +- Leave `AppendedMessages` nil for an ordinary request. Do not allow callers to + choose other roles through the Notarius contract. +- Ensure request formatting and default debug summaries expose only safe + counts/digests. The existing explicitly requested detailed PromptKit trace + may contain complete effective messages and must retain its existing + sensitive-data treatment. +- Preserve the same prompt identity, inputs, variables, session, profile, + execution overrides, repair attempts, scheduler, and prepared-execution + path for corrected calls. -### Tests And Validation +### Verification -At the parser/config boundary, test omitted, explicit zero, positive bounds, -negative, above-three, null, non-integer, scalar shorthand, cloning, redaction, -and JSON round-trip behavior. Use relational boundary tests for the allowed -range and avoid duplicating the same cases at every layer. +- Add focused adapter tests proving an ordinary request is unchanged and a + corrected request has the ordinary rendered prefix plus exactly the two + supplied messages in order with exact content. +- Test invalid/oversized correction rejection before preparation and absence + of raw content from ordinary formatting/errors. +- Add one representative test showing PromptKit structural repair remains + available on a request with appended messages; do not reproduce upstream's + complete repair suite. +- Run `go test ./internal/framework/llm ./internal/framework/pipeline` and + `go test -race ./internal/framework/llm`. -```sh -GOWORK=off go test ./internal/core/config ./internal/cli -GOWORK=off go test -race ./internal/core/config -GOWORK=off go test ./... -go run ./cmd/notarius config validate \ - --config examples/dnd-minimal.config.yml \ - --pipeline dnd-session -go run ./cmd/notarius config validate \ - --config examples/dnd-complete.config.yml \ - --pipeline dnd-session -git diff --check -``` +### Exit Criteria -### Acceptance Criteria +Notarius has one safe, tested adapter path for fresh corrected requests, and no +stage invokes it yet. This stage is one Terra prompt. -- The exact public field parses at pipeline and object-binding scope with the - fixed range. -- Nil and explicit zero remain distinguishable through parsing, cloning, - inheritance, redaction, summaries, and round trips. -- Both maintained configurations remain valid without requiring the new field. -- No runtime or prompt default has changed prematurely. - -## Stage 9: Resolve And Apply Repair Configuration ✅ +## Stage 6 — Migrate The Scene Chunker And Foundational Extractors ### Goal -Resolve the public field against module execution classes, incorporate the -effective value into pipeline identity, and connect it to the request plumbing -completed in Stages 6 and 7. +Make the scene chunker and the foundational direct D&D extractors satisfy +`single_response_v1`. -### Implementation +### Work -1. During resolution, compute the effective value for every selected binding: - - explicit binding value wins; - - otherwise an explicit pipeline value applies to an LLM-backed binding; - - otherwise leave nil for prompt-owned policy. - Apply the pipeline value to LLM-backed validators as well as producers. -2. Reject an explicit binding value on a deterministic module or deterministic - validator using the same execution-class knowledge used for `llm_profile`. - Do not reject a pipeline-level value merely because a selected pipeline also - contains deterministic bindings; simply do not apply it to those bindings. -3. Clone every resolved pointer so the parsed pipeline, resolved profile, - redacted summaries, and runner requests have distinct ownership. -4. Include the effective field in resolved pipeline JSON and digest input. A - change between nil, zero, and a positive value must change the resolved - digest when it changes an LLM-backed selected binding. Unselected lanes must - retain the repository's existing digest and selection semantics. -5. Pass the resolved value into the Stage 6 request field for every selected - input, chunk, extract, merge, normalize, output, and validator binding. -6. Update `docs/config.md` as the canonical field, range, and precedence - contract; `docs/internal/pipeline.md` as the resolution owner; and - `docs/operations.md` for the distinction from binding `retries`. The - prompt-owned production default remains unchanged until Stage 10. -7. Add focused resolution and runner tests. Cover representative execution - classes rather than repeating the same assertion for every module type. +- Update `dnd/scenes`, `dnd/npc-registry`, `dnd/item-registry`, + `dnd/location-registry`, `dnd/scene-descriptions`, and `dnd/spells` producer + implementations to pass request correction to their structured completion + and return an owned copy of the successful response's exact validated raw + bytes as `ModelCandidate`. +- Declare `single_response_v1` in each corresponding module spec. +- Do not serialize normalized artifacts to fabricate candidate material. Keep + artifact parsing, deterministic identity attachment, evidence validation, + warnings, and durable schemas unchanged. +- Update shared D&D extraction helpers only where the behavior is genuinely + common; keep domain prompt ownership in each module. -### Tests And Validation +### Verification -Test: +- Use representative package-level tests to prove exact-response propagation, + correction forwarding, and input/result ownership. Update spec tests for the + declared protocol without multiplying the same behavior test across all six + modules. +- Run the affected module tests and `go test ./internal/modules/dnd/...`. -- binding over pipeline over nil precedence; -- inheritance into each selected LLM-backed stage and validator; -- no inheritance into deterministic bindings; -- explicit deterministic-binding rejection; -- detached pointers; -- runner forwarding for representative producer and validator bindings; and -- digest changes for execution-relevant nil, zero, and positive changes. +### Exit Criteria -```sh -GOWORK=off go test ./internal/framework/pipeline ./internal/cli -GOWORK=off go test -race ./internal/framework/pipeline -GOWORK=off go test ./... -go run ./cmd/notarius config validate \ - --config examples/dnd-minimal.config.yml \ - --pipeline dnd-session -go run ./cmd/notarius config validate \ - --config examples/dnd-complete.config.yml \ - --pipeline dnd-session -git diff --check -``` +All named producers truthfully advertise and implement the exact-response +protocol, with unchanged ordinary extraction behavior. This stage is one +Terra prompt. -### Acceptance Criteria - -- The exact public field has the fixed binding-over-pipeline-over-prompt - precedence for every selected LLM-backed producer and validator. -- Deterministic binding misuse fails during resolution before execution, while - a pipeline value coexists with deterministic bindings. -- Nil and explicit zero remain distinguishable through resolution, runtime, - summaries, and digests. -- A policy change invalidates checkpoint identity when it changes an effective - selected binding. -- Current configuration, pipeline, and operations documentation matches the - implemented behavior. - -## Stage 10: Enable The Default, Finish Documentation, And Verify The Feature ✅ +## Stage 7 — Migrate Downstream D&D Extractors ### Goal -Set the accepted production default of one repair, reconcile all canonical -documentation, and run the full repository verification pass. +Complete `single_response_v1` support for the remaining direct D&D extraction +producers. -### Implementation +### Work -1. Change `repair_attempts: 0` to `repair_attempts: 1` in every maintained - production prompt manifest that produces structured output, including the - generic semantic-reconciliation prompt and every D&D chunk, extraction, and - registry-normalization prompt. Do not mechanically change unrelated test - fixtures whose purpose is to exercise zero. -2. Inspect every production prompt output contract after the edit. Confirm that - each positive budget uses `basic`, `json`, or `json_schema`, remains no - greater than three, and retains its existing format and schema path. -3. Add or refine the smallest durable assembled-assets test that proves the - production engine can prepare the maintained prompts with the activated - contracts. Do not add a brittle test that asserts an exact prompt count, - file count, message prose, correction text, or asset length. The public - default may be tested at one canonical assembled boundary because its - literal value is an operational contract. -4. Confirm a successful repair does not create a warning and that exhausted - repair remains `ErrInvalidStructuredOutput`. Verify the debug prompt records - the configured contract, the debug response records actual repair count, - and cumulative usage is not double-counted. -5. Confirm scheduling behavior with one focused test or existing coverage: the - Notarius scheduled client admits one logical `CompleteStructured` operation - while PromptKit may make serial corrective provider calls inside it. Do not - attempt to reacquire a Notarius permit from inside PromptKit or add a second - scheduler. -6. Reconcile current-state documentation: - - `docs/integrations/pkg-promptkit.md` owns the pinned upstream boundary; - - `docs/config.md` owns field names, range, default, and precedence; - - `docs/operations.md` owns latency/cost, optional credentials, concurrency, - timeout, and the upper-bound formula; - - `docs/internal/llm.md` owns inspection-based contract replacement, - cumulative usage, actual repair count, generation errors, and debug data; - - `docs/internal/pipeline.md` owns effective policy propagation and the - separation from stage retries; and - - `docs/policy/architecture.md` adds only the durable rule that PromptKit - owns deterministic structural repair inside one completion while Notarius - owns stage attempts and semantic validation. -7. Remove current-behavior claims that PromptKit is v0.5.0, that every - production repair budget is zero, or that PromptKit is always single-pass. - Do not alter historical release notes or archived roadmaps. -8. Keep the maintained minimal and complete examples secret-free and valid. - They may omit the new field to demonstrate the default; do not add a - redundant complete profile or Rakestrawhome example merely to exercise an - upstream catalog entry. -9. Review `docs/roadmap/future.md` only for consistency. Leave the future - feedback-aware stage retry, combat-scene validator, and warning-reform work - unimplemented and clearly separate. +- Migrate `dnd/npc-occurrences`, `dnd/item-occurrences`, + `dnd/location-occurrences`, `dnd/combat-turns`, and `dnd/enemy-events` using + the same contract as Stage 6. +- Preserve their contextual-name and deterministic-ID rules. Correction + messages must never introduce opaque registry IDs or ask the model to copy + them. +- Preserve combat-scene gating, reference projections, and prompt-cache prefix + ordering. The correction pair is appended after the complete existing + request and never changes stable prompt assets. -### Tests And Validation +### Verification -Run focused tests first, then all repository checks: +- Add or adapt focused tests for correction forwarding and exact raw-response + retention at the shared boundary, plus one registry-grounded extractor case + proving no opaque IDs enter correction material. +- Run the affected module tests, `go test ./internal/modules/dnd/...`, and the + assembled D&D CLI contract tests. -```sh -GOWORK=off go test ./internal/framework/llm ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/... -GOWORK=off go test ./... -GOWORK=off go test -race ./... -GOWORK=off go vet ./... -GOWORK=off go build ./cmd/notarius -GOWORK=off go mod tidy -diff -go run ./cmd/notarius config validate \ - --config examples/dnd-minimal.config.yml \ - --pipeline dnd-session -go run ./cmd/notarius config validate \ - --config examples/dnd-complete.config.yml \ - --pipeline dnd-session -git diff --check -``` +### Exit Criteria -Also perform focused repository searches that exclude `docs/roadmap/archive/` -and historical release notes: +Every direct production D&D LLM extractor implements the same correction +protocol without changing artifact semantics. This stage is one Terra prompt. -- no active v0.5.0 PromptKit pins or links remain; -- no maintained production prompt still declares `repair_attempts: 0`; -- every production `CompleteStructured` caller forwards the repair field; and -- no provider code, type, or message is added to ordinary errors, warnings, - manifests, cache, or checkpoint schemas. +## Stage 8 — Migrate Semantic Reconciliation Normalizers -If the repository's source-release checker is available and the ordinary -checks above pass, run `./scripts/check-release-source.sh v0.0.0` as the final -integrated validation. It must not create a tag, release note, or repository -artifact. +### Goal -### Acceptance Criteria +Support correction for the current single-proposal reconciliation path and +activate capability enforcement for all eligible production producers. -- Every maintained structured prompt defaults to one corrective call and can - be overridden to zero through three at pipeline or binding scope. -- A real assembled Notarius completion follows the PromptKit v0.8.0 repair - contract without changing prompt schema identity or cacheable prefix. -- Actual repair count, cumulative usage, error classification, debug-only - provider diagnostics, scheduling, and checkpoint identity match the feature - roadmap. -- Profile inheritance, Rakestrawhome availability, optional credentials, and - v0.6.0 hardening remain covered and documented. -- All canonical documentation describes implemented v0.8.0 behavior in its - assigned home and leaves future semantic validation work in the roadmap. -- Maintained examples validate, all ordinary/race/vet/build/module checks pass, - and the worktree contains no generated or sensitive artifacts. +### Work + +- Extend `semanticreconcile.Request` to accept `SemanticCorrection` and its + result to expose the exact validated proposal response as owned + `ModelCandidate` when an LLM call actually occurred. +- Forward correction through the shared engine's ordinary structured + completion request. Preserve request-local integer candidate handles, + proposal validation, typed application policies, and fallback behavior. +- Update the NPC-, item-, and location-registry normalizers to carry the exact + proposal material through their typed result and declare + `single_response_v1`. +- Mark deterministic skip/limit/fallback outcomes as having no model candidate. + If such a candidate is later rejected, it is not correctable and must not + consume a retry merely because its module execution class is LLM-backed. +- Audit all production LLM-backed producer specs. Add the preparation rule that + an LLM-backed producer with a non-empty validator chain and `retries > 0` + must declare `single_response_v1`. Any genuine multi-response producer + remains unsupported and a configured validator-backed semantic retry for it + must fail preparation. + +### Verification + +- Add engine tests for initial and corrected requests, exact proposal bytes, + deterministic no-call outcomes, invalid proposal outcomes, and ownership. +- Add representative normalizer and preparation tests for supported, + unsupported, and no-validator/no-retry configurations. +- Run `go test ./internal/framework/semanticreconcile ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli`. + +### Exit Criteria + +All eligible production LLM producers implement the declared protocol, +unsupported workflows fail before source parsing, and ordinary operational +retries remain allowed when semantic correction cannot occur. This stage is +one Terra prompt. + +## Stage 9 — Build Complete Validator-Chain Execution + +### Goal + +Replace first-result validation with one reusable, deterministic executor that +aggregates the complete chain and isolates validator retries. + +### Work + +- Introduce an internal immutable `validationReport` model with one ordered + invocation record per validator: approved, rejected, failed, or skipped; + validator attempt count; safe reason/message metadata; warnings; and bounded + correction guidance. +- Implement one shared sequential executor around stage-specific invocation + closures. It must continue after rejection and isolated execution failure, + retry an LLM-backed failed validator against the same immutable candidate up + to its binding budget, and stop retrying after a contract-valid approval or + rejection. +- Reconstruct the same ordinary validator request for each validator retry. + Do not append feedback to validator prompts and do not regenerate the + producer candidate. +- Preserve only one final warning for an exhausted validator failure; retain + individual attempt details for debug. Keep warnings from completed validators + in configured order. +- Build aggregate correction guidance from all rejections using the settled + ordering, deduplication, generic fallback, and bounds. Validator failures, + skips, warnings, and operator messages must not enter it. +- Retain `skipped` as a framework-owned outcome for an otherwise selected + validator whose runtime prerequisites are unavailable. Module construction, + type incompatibility, candidate cloning failure, cancellation, and debug + persistence failure remain framework errors rather than skips. + +### Verification + +- Add table-driven behavior tests for all approved, multiple rejections, + rejection plus failure, failure only, skipped, retry success, retry + exhaustion, immutable candidate reuse, deterministic ordering, duplicate + guidance, and bounds. +- Prove deterministic validators cannot receive positive retry budgets and + LLM validator retry counts do not consume producer attempts. +- Run `go test ./internal/framework/pipeline` and + `go test -race ./internal/framework/pipeline`. + +### Exit Criteria + +The shared executor returns a complete ordered report without deciding whether +the producer candidate advances. Existing stage callers may still adapt the +report through their old disposition path until the next stages. This stage is +one Terra prompt. + +## Stage 10 — Implement The Generic Producer Attempt State Machine + +### Goal + +Replace the boolean retry helper with one explicit domain-neutral state +machine used by later stage integrations. + +### Work + +- Refactor `runWithRetry` into an attempt engine that accepts producer and + complete-validation closures and returns a terminal result plus ordered + attempt provenance. Keep stage-specific artifact handling outside it. +- Classify attempts as initial, operational-error retry, structural retry, + module-requested retry, or semantic correction. Preserve one total producer + budget of `retries + 1` attempts. +- On semantic rejection, correct only when another attempt exists and the + current candidate contains valid `single_response_v1` material. Construct a + fresh `SemanticCorrection` from that latest response and aggregate feedback. +- Apply outcome precedence and effective terminal policy exactly as the + feature roadmap specifies. Ordinary producer errors remain framework errors + after retries. Only `ErrInvalidStructuredOutput` uses the structural-failure + policy; `reject_output` records no usable artifact. +- A deterministic or no-model candidate applies semantic terminal policy + immediately without spending an ineffective retry. +- With rejection plus validator failure, use rejection guidance for a + correction while recording incomplete validation. With failure only, apply + validator-failure policy without regenerating the producer. +- Keep cancellation and debug-persistence failures immediately terminal. + Abandoned-attempt warnings must not be promoted. + +### Verification + +- Test the state machine through stable behavior with fake closures: budget + accounting, fresh correction history, latest-response replacement, all + terminal actions, failure precedence, deterministic candidates, + cancellation, and warning promotion. +- Do not assert private helper call choreography or exact correction prose. +- Run `go test ./internal/framework/pipeline` and its race-enabled suite. + +### Exit Criteria + +One tested state machine owns attempt budgets and disposition, but no stage is +partially migrated. This stage is one Terra prompt. + +## Stage 11 — Integrate Chunk Validation And Correction + +### Goal + +Move chunk generation, validation, cache reuse, and retry disposition onto the +new state machine. + +### Work + +- Adapt chunk candidate materialization and both chunk/serialized validator + targets to the complete-chain executor. +- Pass correction only to a generated chunker attempt and retain the exact raw + chunker response associated with the materialized plan. +- Preserve chunk-plan cache identity and mode semantics. A rejected automatic + cache hit is unusable for the invocation and falls through to generated + attempt one without pretending the cache hit has model material. Publish a + newly generated plan only after complete accepted validation. +- Apply chunk effective terminal policy. A terminal chunk rejection prevents + lane execution; a framework error still prevents output encoding. +- Preserve annotation materialization, deterministic plan canonicalization, + attempt debug paths, scheduler use, and checkpoint behavior. + +### Verification + +- Add focused tests for corrected plan acceptance, multi-validator feedback, + rejected cache hit regeneration, cache non-overwrite, incomplete-validation + non-publication, policy outcomes, cancellation, and debug failure. +- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/chunk/... ./internal/cli` and the pipeline race tests. + +### Exit Criteria + +Chunk is the first complete production stage using feedback-aware retries, with +cache and terminal behavior matching the roadmap. This stage is one Terra +prompt. + +## Stage 12 — Integrate Per-Chunk Extraction + +### Goal + +Apply the state machine independently to every concurrent extraction job. + +### Work + +- Adapt typed and serialized extract validation to the complete-chain executor + and use the extract binding's effective policy. +- Keep correction state local to one lane/chunk job. Rebuild the same initial + extraction request with correction attached only on a semantic retry. +- Preserve the run-wide worker pool, scheduled LLM client, chunk-first/lane- + second public ordering, stable error selection, cancellation, and lane + continuation rules. +- Ensure `reject_output` records the terminal chunk-scoped rejection without + advancing it to merge. `warn_continue` is available only for validator + failure with no semantic rejection and marks the accepted extract as + validation-incomplete. + +### Verification + +- Add representative extraction tests for correction success, exhausted + rejection, validator failure policy, independent concurrent jobs, stable + ordering, cancellation, and warning promotion. +- Add one assembled D&D pipeline test using fakes—not a live provider—to prove + a rejected direct extraction is corrected using the exact prior response. +- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli` and relevant race tests. + +### Exit Criteria + +Every extraction job has an isolated bounded correction conversation and +continues to obey existing concurrency and ordering contracts. This stage is +one Terra prompt. + +## Stage 13 — Integrate Merge And Normalize + +### Goal + +Complete stage coverage and reconcile semantic correction with the existing +normalizer fallback retry directive. + +### Work + +- Adapt typed and serialized merge and normalize validation to the shared + executor and their respective effective policies. +- Keep merge and normalize serial within a lane and reuse the exact same + accepted upstream artifacts, references, source input, profile, and session + on every attempt. +- Fold `NormalizeRetry` into the one producer attempt state machine. It consumes + the same remaining stage budget, retains its validated safe fallback when + exhausted, and never overrides a semantic rejection of that fallback or a + later candidate. +- Do not send correction to deterministic mergers/normalizers. A rejected + deterministic candidate applies terminal policy immediately. +- Preserve typed erasure checks, codec boundaries, generated handoffs, and + lane continuation/cancellation behavior. + +### Verification + +- Add one representative merge path and normalize paths for correction + success, deterministic rejection, `NormalizeRetry` success/exhaustion, + fallback rejection, validator failure, and terminal policies. +- Run `go test ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/... ./internal/cli` and relevant race tests. + +### Exit Criteria + +Chunk, extract, merge, and normalize all use the same attempt and validation +semantics, with no nested normalizer retry loop. This stage is one Terra +prompt. + +## Stage 14 — Finalize Provenance, Checkpoints, Debug, And Durable Results + +### Goal + +Make the new state machine auditable and safe without leaking correction +content or allowing degraded output to become reusable state. + +### Work + +- Extend internal attempt/debug records with attempt kind, PromptKit repair + count and usage, ordered validator outcomes and attempt counts, aggregate + reason codes, validation completeness, effective policy, and terminal + decision. +- Keep raw assistant responses and complete correction messages only in the + existing explicitly requested detailed trace. Default summaries contain + counts, digests, identities, and bounded safe fields. +- Add one reusable durable validation summary with the JSON fields + `status`, `rejecting_validators`, `reason_codes`, + `incomplete_validators`, `producer_attempt_count`, and `terminal_action`. + `status` is one of `complete`, `rejected`, or `incomplete`; lists preserve + configured order and omit duplicates after their first occurrence. Embed or + project this summary at the manifest, rejection, and run-result boundaries + that already expose the affected stage outcome. Preserve the existing + singular rejection fields as the first configured rejection for downstream + continuity; do not put raw guidance or responses in the summary. +- Emit one genuine, bounded, deterministically ordered warning per validator + whose execution budget is exhausted under `warn_continue`; do not emit a + warning merely because a later correction succeeded. +- Checkpoint only accepted and completely validated results. Never write or + reuse rejected, structurally invalid, or validation-incomplete producer + output. Ensure correction protocol and effective policy participate in + fingerprints. +- Confirm cache, resume, generated-reference handoff, output encoding, and CLI + result publication cannot treat a rejected or incomplete checkpoint as + accepted. + +### Verification + +- Add focused manifest, receipt, warning, debug, redaction, checkpoint reuse, + generated-handoff, and sensitive-content tests at their canonical owners. +- Test that corrected success is auditable but quiet, warn-continue output is + never checkpointed, and raw content is absent from ordinary durable files. +- Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint ./internal/framework/artifacts ./internal/cli` and relevant race tests. + +### Exit Criteria + +Every terminal path has accurate bounded provenance, no sensitive correction +content leaks by default, and only completely validated output is reusable. +This stage is one Terra prompt. + +## Stage 15 — Update Canonical Documentation And Perform Final Verification + +### Goal + +Document the implemented contracts in their canonical homes and prove the +repository is ready for review. + +### Work + +- Update `docs/policy/architecture.md` with durable validator aggregation, + retry-budget separation, producer correction capability, terminal-policy, + checkpoint, and sensitive-data invariants. Link to ADR-0014 for rationale. +- Update `docs/config.md` with the exact `validation_policy` schema, enum + values, defaults, inheritance, placement restrictions, retry-budget + meanings, and invalid combinations. +- Update `docs/operations.md` with costs, terminal outcomes, warnings, debug + sensitivity, retry exhaustion, resume/cache consequences, and recovery. +- Update `docs/internal/pipeline.md`, `docs/internal/llm.md`, and + `docs/internal/modules.md` with implemented mechanics and focused test + routing. Keep public configuration definitions in `docs/config.md`. +- Complete `docs/integrations/pkg-promptkit.md` for v0.9.0 appended messages, + supported roles, application-owned bounds, and the external catalog + boundary. Update the run-result and affected output/subprocess integration + contracts for any durable fields added in Stage 14. +- Update maintained examples only to demonstrate implemented behavior. Use a + minimal policy override in the complete D&D example if it materially aids + operators; keep the minimal example minimal. Ensure all links point to + canonical owners and remove stale v0.8.0 claims outside historical release + notes. +- Do not create a release note until an actual release is prepared. + +### Verification + +- Run `gofmt` on all changed Go files. +- Run `go test ./...`, `go test -race ./...`, `go vet ./...`, and + `go build ./cmd/notarius`. +- Run the repository's example/config validation tests and documentation link + checks. If no standalone link checker exists, verify changed relative links + and record that manual check in the implementation report. +- Use targeted searches to confirm no current documentation still claims + PromptKit v0.8.0, no production correction-capable spec is missing its + protocol, no PromptKit type escaped `internal/framework/llm`, and no raw + correction content is serialized outside explicit detailed debug data. +- Review `git diff --check` and `git status --short`; do not include generated + binaries, temporary files, or unrelated work. + +### Exit Criteria + +All roadmap acceptance criteria are satisfied, canonical documentation matches +the code, maintained examples validate, the complete ordinary and race-enabled +test suites pass, and the worktree contains only intentional implementation +changes. This stage is one Terra prompt. + +## Open Questions + +None. PromptKit transport, producer representation, terminal-policy ownership, +bounds, retry budgets, outcome precedence, and checkpoint treatment are all +settled by the feature roadmap and this plan. diff --git a/docs/roadmap/promptkit-v0.8.md b/docs/roadmap/promptkit-v0.8.md deleted file mode 100644 index c98321e7..00000000 --- a/docs/roadmap/promptkit-v0.8.md +++ /dev/null @@ -1,520 +0,0 @@ -# PromptKit v0.8.0 Upgrade - -## Status - -Proposed. - -## Purpose - -Upgrade Notarius from PromptKit v0.5.0 to v0.8.0 and deliberately adopt the -useful correctness, profile-composition, provider-diagnostic, backend, and -structured-output-repair capabilities introduced in PromptKit v0.6.0, v0.7.0, -and v0.8.0. - -The upgrade should improve structured-output reliability without confusing -PromptKit's bounded deterministic repair with Notarius's existing stage retry -budget or the future feedback-aware semantic-validation loop. PromptKit types -and provider behavior must remain behind Notarius's transport-neutral LLM -boundary. - -## Current State - -Notarius currently pins PromptKit v0.5.0. Its production adapter prepares one -frozen execution, records credential-redacted details, and runs that same -prepared value. It maps PromptKit capacity failures to an application-owned -error, maps failed structured validation to `ErrInvalidStructuredOutput`, and -returns PromptKit's raw validated bytes and usage metadata. - -Every maintained production prompt uses JSON Schema validation and currently -declares `repair_attempts: 0`. Notarius stage bindings separately expose -`retries`, which reruns a complete stage operation after an error or rejected -candidate. The two mechanisms have different ownership and must remain -independent. - -Notarius also maintains: - -- embedded prompt, schema, and fallback-profile filesystems; -- operator profile-file and profile-directory sources; -- one optional conventional `local` backend registration; -- explicit profile preflight through PromptKit inspection; -- one application-wide scheduled LLM client around the PromptKit adapter; -- PromptKit profile-source fingerprints for checkpoint safety; and -- redacted debug and manifest provenance at application-owned boundaries. - -The upgrade must preserve those established responsibilities while revising -the pinned integration contract and any behavior affected by the three -intervening releases. - -This roadmap is based on PromptKit's pinned release guides for -[v0.6.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.6.0.md), -[v0.7.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.7.0.md), -and -[v0.8.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.8.0.md), -plus the public API and format documentation at the v0.8.0 tag. - -## Target End State - -- `go.mod` and `go.sum` pin PromptKit v0.8.0 without a local replacement or - vendored copy. -- Every maintained PromptKit prompt and profile prepares successfully under - v0.8.0's stricter validation and source-loading rules. -- Eligible Notarius structured completions use one PromptKit corrective call by - default after a structurally invalid response. Operators can explicitly set - a value from zero through three for a configured pipeline, with a more local - LLM-backed binding override where needed. -- PromptKit repair remains an inner operation within one Notarius stage - attempt. It never consumes or replenishes the binding's `retries` budget. -- A successful repaired result exposes cumulative usage and the actual repair - count to Notarius's application-owned response and debug models. A repaired - success is not itself a warning. -- Exhausted PromptKit validation remains an invalid structured-output result, - preserving the final candidate and diagnostics for debug and for any - applicable outer Notarius stage policy. Invalid structured output is never - accepted merely because the repair budget was exhausted. -- Profile inheritance, the built-in Rakestrawhome backend/profile, optional - credential behavior, and structured generation errors work through the - existing Notarius PromptKit boundary and are accurately documented. -- Provider-specific PromptKit types do not escape `internal/framework/llm`. -- Checkpoint identity, effective configuration, redacted summaries, and debug - provenance reflect every execution-affecting repair or profile change. -- Current documentation pins and describes v0.8.0; future Notarius semantic - validation retries remain roadmap behavior rather than being conflated with - this dependency upgrade. - -## Release-by-Release Adoption - -### PromptKit v0.6.0: Correctness, Safety, And Efficiency - -PromptKit v0.6.0 adds no public declarations, but intentionally rejects several -formerly permissive or ambiguous inputs. The upgrade must audit Notarius's -embedded and operator-facing integration against these rules: - -- YAML `id` and `version` metadata, rather than filenames, define prompt and - profile identity. -- Prompt `content_file` paths are exact, relative, contained paths; built-in - file artifacts must resolve to regular files. -- execution controls, output contracts, and repair budgets must be finite and - within their documented ranges; -- provider endpoints must be absolute HTTP or HTTPS URLs with a host and no - user information, query, or fragment; -- JSON documents and successful provider responses contain exactly one value; -- successful provider responses are bounded to 16 MiB; and -- JSON-compatible values are bounded for depth and expansion. - -Notarius should rely on PromptKit for these rules rather than duplicate its -parsers or internal limits. Existing Notarius validation may retain a narrower -application rule where it has independent value, but overlapping validation -must agree with PromptKit and must not accept a value PromptKit will reject -later. - -The upgrade automatically receives operation-local schema-plan reuse, -artifact-text memoization, improved cancellation checks, and transport error -identity preservation. Notarius should verify these changes through its real -adapter boundary and avoid adding a second cache or response-body layer that -would duplicate PromptKit's ownership. - -### PromptKit v0.7.0: Profiles, Backend Access, And Generation Errors - -#### Profile Inheritance - -Operator profiles may use `base_profile` to alias or selectively refine a -built-in, fallback, or higher-precedence operator profile. Notarius must pass -profile sources through unchanged and let PromptKit own parent lookup, merge -rules, source precedence, cycle detection, and fully resolved prepared targets. - -Preflight inspection must resolve inherited profiles through the same source -and backend composition used at execution. The selected leaf profile ID remains -the public profile identity, while effective backend, endpoint, model, and -reasoning provenance reflect the resolved chain. Notarius must not implement a -second inheritance parser. - -The existing complete `dnd-extraction` fallback remains a standalone profile: -PromptKit v0.8.0 does not provide a built-in `openai/gpt-5.6-luna` profile that -would be an appropriate parent. Documentation should nevertheless explain how -operators can use inheritance for environment-specific workload profiles and -should link to PromptKit's pinned format contract rather than duplicate its -field-by-field merge algorithm. - -Checkpoint safety must cover inherited behavior. Operator file/directory -digests already cover changes to definitions in those sources, fallback asset -digests cover application parents, and the PromptKit built-in catalog marker -must change from its v0.5.0 identity to v0.8.0 so a changed built-in parent -cannot reuse an incompatible checkpoint. - -#### Rakestrawhome Backend And Profile - -PromptKit's reserved `rakestrawhome` backend and -`rakestrawhome-gemma-4-31b` profile become available without Notarius-specific -registration. Notarius must not register or shadow the reserved backend ID. -Profile preflight, backend-capacity reporting, scheduling, generation, and -provenance should work for it through the same generic paths used by OpenRouter -and `local`. - -The D&D default remains `dnd-extraction`; this upgrade does not silently move a -production workload to Rakestrawhome. Operator documentation should identify -the built-in profile as an available selection and link to PromptKit for its -endpoint, credential environment, model, and capacity defaults. - -#### Optional Credentials - -An absent or blank optional `APIKeyEnv` now causes PromptKit to omit the -`Authorization` header and send the request. Notarius must not restore the old -failure behavior by pre-reading provider credential environment variables or -by adding provider-specific authentication logic. - -Profile inspection may report an explicit `APIKeyRequired` policy without -reading the credential, and execution remains the boundary at which that -requirement is enforced. For optional profiles, an authentication-requiring -provider may instead return a structured 401 or 403 generation failure. The -configuration and operations documentation must explain this distinction. -Notarius does not currently expose PromptKit's in-memory profile-registration -API to operators, and PromptKit's filesystem profile format does not expose -`APIKeyRequired`; therefore Notarius must not promise that an operator profile -can force local credential preflight. Operators should provision the named -environment variable, while Notarius should preserve the provider's structured -authentication failure when it is absent. - -Notarius must continue to document mechanisms and environment-variable names, -never secret values. - -#### Structured Generation Errors - -The adapter should recognize `*promptkit.GenerationError` with `errors.As` and -translate useful information into an immutable, provider-neutral Notarius -error classification. At minimum, retain the HTTP status code so callers and -future retry policy can distinguish transport success with provider rejection -from other generation failures. - -PromptKit's provider code, type, and message accessors are bounded but remain -untrusted and potentially sensitive. They must never appear automatically in -ordinary CLI output, warnings, manifests, checkpoint identity, or cache data. -If retained for an explicitly requested debug trace, they must pass through -Notarius's known-secret and bearer redaction and remain clearly identified as -untrusted provider diagnostics. Default error formatting should continue to -use a bounded, redacted application-owned message. - -Capacity and cancellation retain their current more specific classifications -and precedence. This upgrade does not add automatic provider-error retry -classification; it only preserves safe structured data needed for diagnosis -and later policy. - -### PromptKit v0.8.0: Bounded Structured-Output Repair - -#### Default Policy - -Every maintained production prompt whose output is consumed as structured data -should declare one repair attempt. All current production prompts use eligible -JSON Schema validation, so no current prompt needs a zero default merely -because of its output mode. - -One repair means at most one corrective generation after the initial -candidate. PromptKit reconstructs the immutable original conversation and -appends only the latest invalid assistant candidate and latest deterministic -validation diagnostics. It preserves the selected target, direct session ID, -provider-native structured-output contract, and backend capacity policy. This -shape preserves the original cacheable prompt prefix and avoids accumulating -unbounded failed history. - -The default is deliberately small. A single repair captures the common case in -which a capable model can correct malformed JSON or a schema violation after -receiving an exact diagnostic, while bounding the extra latency and cost of a -single structured completion. - -#### Configuration Contract - -The public configuration is an optional, presence-aware -`structured_output_repair_attempts` integer at pipeline scope and at each -LLM-backed module or validator binding. Its effective precedence is: - -1. the binding value, when present; -2. the pipeline value, when present; and -3. the selected prompt's declared `repair_attempts` value. - -The value must be from zero through three. Explicit zero disables PromptKit -repair at that scope. A deterministic binding must reject the field because it -cannot perform structured LLM repair. Validator bindings may use it only when -the selected validator is LLM-backed. Shorthand module bindings continue to -inherit the pipeline or prompt default. - -The long, provider-neutral name is intentional: it distinguishes PromptKit's -inner structural repair from the existing binding `retries` field, which owns -complete stage attempts, without exposing a dependency name in generic -pipeline contracts. - -The effective value must survive file parsing, cloning, redacted summaries, -pipeline resolution, and pipeline digest construction without pointer aliasing -or loss of presence. It must affect checkpoint identity because it can change -the selected result, latency, token usage, and provider cost. - -#### Adapter Contract - -The transport-neutral structured-completion request should carry an optional -application-owned structural-repair budget. No `promptkit.OutputContract` or -other PromptKit type may cross the adapter boundary. - -PromptKit v0.8.0 request validation replaces the complete prompt output -contract rather than merging one field. When Notarius has a configured -override, the adapter must therefore inspect the selected prompt, copy its -normalized declared format, validation mode, and schema path, change only the -repair count, and supply that complete contract on the prepared request. A nil -override continues to use the prompt declaration directly. Inspection and -preparation must use the same immutable engine sources; a small adapter-local -cache keyed by normalized prompt ID and version is acceptable but not required -without measured need. - -This approach prevents configuration from accidentally dropping JSON Schema -validation, avoids duplicating schema paths in pipeline YAML, and keeps prompt -assets authoritative for every output-contract field other than the explicit -operator override. - -The transport-neutral structured-completion response should report the actual -number of PromptKit repair calls. PromptKit's returned token usage is already -cumulative and must be passed through without re-summing it. Debug records -should distinguish the configured budget from the actual count. Ordinary run -manifests need not gain raw prompt or response data merely to report repairs; -any durable aggregate should be added only if it has a clear consumer contract. - -#### Result And Failure Semantics - -- A valid initial candidate returns normally with zero actual repairs. -- A valid corrected candidate returns normally with cumulative usage and its - positive actual repair count. It does not emit a warning solely because a - repair occurred. -- Exhausting the repair budget returns PromptKit's final candidate and failed - validation result. The adapter maps this to - `ErrInvalidStructuredOutput`, preserves the response and debug material, and - does not decode or accept the candidate. -- An explicitly empty or whitespace-only candidate participates in the - declared structural validation and repair flow. Missing, `null`, or - non-string provider content remains a malformed provider response. -- A generation failure during a corrective call is an operational generation - failure and uses the same safe structured-error adaptation as an initial - generation failure. -- Context cancellation remains authoritative throughout the initial and - corrective calls. - -PromptKit repair happens inside one scheduled `CompleteStructured` operation. -The Notarius scheduler holds one permit for that logical operation while -PromptKit performs its initial and serial corrective calls; PromptKit -reacquires its own selected-backend capacity for each corrective generation. -Because corrective calls are serial, this cannot expand actual concurrent -provider work beyond the number of admitted Notarius operations, but -documentation must stop describing the Notarius permit as a separate admission -event for every internal repair call. - -One `CompleteStructured` invocation with effective PromptKit repair budget `R` -may make at most `R + 1` provider calls. If one stage attempt makes `C` -structured-completion invocations, a binding with `retries: N` has an upper -bound of `(N + 1) * C * (R + 1)` provider calls; `C` may itself be a bounded, -data-dependent module property, as it is for batched semantic reconciliation. -LLM-backed validators have their own corresponding invocation counts, budgets, -and costs. These formulas are upper bounds, not promises that every failure is -retryable or that every attempt reaches the provider. - -## Profile And Prompt Source Compatibility - -The upgrade must preserve Notarius's source precedence: an operator source, -then registered application fallback profiles, then PromptKit built-ins. A -selected malformed definition remains authoritative and fails rather than -falling through. Parent resolution introduced by profile inheritance observes -that same precedence. - -All embedded prompt manifests, shared content fragments, response schemas, and -fallback profiles must be prepared or inspected offline under v0.8.0. The -review should specifically catch: - -- IDs inferred accidentally from filenames; -- stale or escaping `content_file` paths; -- missing or non-regular embedded artifacts; -- repair values outside zero through three or paired with ineligible - validation; -- schemas or examples that are not exact single JSON documents; -- unsupported endpoint forms; and -- JSON-compatible variables or profile extras that exceed upstream bounds. - -No prompt prose, schema shape, durable D&D artifact contract, or default D&D -model should change merely to exercise the dependency. Prompt manifests should -change only as needed to enable the adopted repair default and satisfy v0.8.0 -contracts. - -## Provenance, Debugging, And Security - -- Update the opaque PromptKit built-in profile-catalog identity from v0.5.0 to - v0.8.0. Do not hash or publish PromptKit's internal catalog bytes. -- Ensure a prompt's repair default remains covered by its existing prompt asset - fingerprint and a configured effective override remains covered by the - resolved pipeline digest. -- Preserve selected leaf profile identity while recording the inherited - effective target already exposed by PromptKit inspection and prepared - details. -- Add actual structural-repair count and, when useful, the configured budget to - application-owned debug material. Token totals remain PromptKit's cumulative - values. -- Do not generate a warning for a successful repair. Repair exhaustion is an - invalid-output failure, while provider rejection is a generation failure. -- Never expose raw provider diagnostic fields without explicit debug capture - and application redaction. Do not place them in normal errors or durable - summaries. -- Preserve context and transport error identity sufficiently for - `errors.Is`-based cancellation and deadline handling after adapting the - external error. - -## Documentation And Examples - -Implementation should update current-state documentation only when the new -behavior lands: - -- `docs/integrations/pkg-promptkit.md` must pin v0.8.0 and define the revised - prepared-execution, repair, profile-inheritance, backend, credential, and - error-adaptation boundary. -- `docs/config.md` must own the repair configuration fields, precedence, - allowed range, explicit-zero behavior, profile inheritance availability, and - optional credential semantics. -- `docs/operations.md` must explain structural repair cost, timeout and - concurrency effects, credential failures, and its distinction from stage - retries. -- `docs/internal/llm.md` must describe adapter contract replacement, actual - repair metadata, error adaptation, source compatibility, and scheduling. -- `docs/internal/pipeline.md` must describe how effective repair configuration - is resolved and how inner repair differs from outer stage attempts. -- `docs/policy/architecture.md` should receive only the durable ownership rule: - PromptKit owns bounded deterministic structural repair within one completion, - while Notarius owns stage attempts and semantic validation policy. Detailed - fields and retry formulas belong in their canonical configuration and - operations documents. - -Update maintained configuration examples only if the public Notarius -configuration contract changes. A short inheritance illustration may remain in -the configuration reference; do not create a complete example solely to copy -PromptKit's upstream profile catalog. All upstream links must point to the -v0.8.0 tag. Historical release or archived roadmap references should remain -historical. - -No ADR is required solely to pin a newer dependency. The durable separation -between PromptKit structural repair and Notarius semantic stage retries should -be stated in architecture documentation now; the more extensive future -validation state machine still warrants the separate ADR already identified in -`future.md` when that work is promoted. - -## Validation And Acceptance Criteria - -The implementation is complete when: - -- the repository builds and tests against PromptKit v0.8.0 with no replacement - directive, workspace dependency, or vendored source; -- every maintained prompt and profile prepares or inspects successfully under - the v0.8.0 source, path, endpoint, output-contract, and JSON-value rules; -- an invalid first JSON Schema candidate followed by a valid correction returns - the valid raw output, cumulative usage, and actual repair count through the - Notarius adapter; -- repair exhaustion returns the final raw candidate and debug material with an - error matching `ErrInvalidStructuredOutput`; -- a corrective generation failure retains safe generation classification and - provider status without leaking untrusted provider detail; -- explicit empty content follows structural validation rather than being - misclassified by Notarius; -- repair configuration is presence-aware, range checked, rejected on - deterministic bindings, resolved with documented precedence, and included in - effective pipeline identity; -- inherited profiles resolve consistently during preflight and execution, and - changes to any relevant operator, fallback, or built-in parent invalidate - checkpoint reuse; -- the Rakestrawhome built-in profile reaches generic preflight, scheduling, and - provenance paths without application-specific registration; -- optional missing credentials and explicitly required credentials behave as - documented without contacting real providers in tests; -- cancellation, timeout, backend capacity, prepared-execution snapshot, - session ID, raw-output, debug-redaction, and existing profile provenance - behavior remain intact; -- maintained examples validate successfully; and -- canonical documentation contains no active v0.5.0 pin or claim that PromptKit - is always single-pass. - -Tests should follow `docs/policy/testing.md`: exercise observable Notarius -contracts with offline fake clients or `httptest` boundaries, and do not copy -PromptKit's entire internal repair test suite or assert its exact correction -message prose. The dependency's internal wording is not a Notarius contract. - -## Non-Goals - -- Implementing Notarius's future feedback-aware semantic stage-retry loop. -- Adding the D&D combat-scene semantic validator. -- Redesigning warning policy or treating successful structural repair as a - warning. -- Adding provider transport retries or deciding which HTTP statuses should - consume a stage retry. -- Exposing PromptKit request, response, profile, validation, capacity, or error - types outside the LLM adapter. -- Changing durable artifact schemas, D&D prompt semantics, the D&D default - model, or the fixed pipeline shape. -- Reimplementing PromptKit profile inheritance, schema validation, response - bounds, repair conversations, backend admission, or provider parsing inside - Notarius. - -## Decisions - -### 1. Default Structured-Output Repair Budget - -**Decision: default to one repair attempt.** Set every maintained -eligible production prompt to `repair_attempts: 1`. One corrective call is a -strong fit for Notarius because every current production LLM response has a -strict JSON Schema contract, smaller cost-effective models are a deliberate -deployment target, and a precise structural diagnostic often makes one retry -materially more successful. The budget is paid only after a structurally -invalid candidate and remains tightly bounded. - -**Alternative considered: retain zero by default.** This preserves single-pass -cost and latency and requires operators to opt in. It is preferable for an -environment where every additional request is expensive or where upstream -provider-native schema enforcement already produces negligible invalid output. -It is less suitable as the Notarius default because one malformed response can -otherwise discard substantial completed pipeline work. - -**Alternative considered: default to two.** This may improve recovery for -weak models, but it doubles the worst-case corrective cost relative to the -selected default and compounds with outer stage retries. It should be an -operator choice supported by configuration, not the initial default, unless -observational evidence shows that the second correction has a worthwhile -marginal success rate. - -### 2. Repair Override Scope - -**Decision: support both pipeline and LLM-backed binding overrides.** Use -the presence-aware `structured_output_repair_attempts` field and precedence -defined above. A pipeline value provides the convenient one-line control the -operator requested, while a binding value permits an expensive normalizer or -future LLM-backed validator to use a deliberately different budget. This -mirrors Notarius's established pipeline/binding profile inheritance and scales -without editing embedded prompts. - -**Alternative considered: support only a pipeline override.** This is smaller to -implement and document and still permits global enablement or disablement for -one pipeline. Its drawback is that one exceptional prompt cannot opt out or -request a larger budget without changing an embedded asset for every pipeline. - -**Alternative considered: expose one global value under the top-level -`promptkit` configuration.** This makes client construction simple, but applies -the same budget to unrelated pipelines and leaks an execution policy into the -dependency configuration block. It is less compositional than pipeline-owned -policy and therefore not recommended. - -### 3. Retention Of Provider-Supplied Generation Details - -**Decision: retain status in the application-owned error contract and -retain redacted provider code, type, and message only in explicitly requested -debug traces.** Status is useful for diagnosis and future retry policy without -usually containing sensitive data. The other fields can materially explain a -400 response but may echo request or schema content, so they belong only in the -already-sensitive debug surface after Notarius redaction. - -**Alternative considered: retain only HTTP status and discard all provider fields.** -This is the safest and smallest policy and still improves typed failure -handling. It sacrifices potentially decisive provider diagnostics, leaving an -operator with less information when a provider returns a terse status and the -problem cannot be reproduced easily. - -**Alternative considered: include bounded provider code and type in normal -errors while keeping message debug-only.** Codes and types are often stable and -less sensitive than messages, but PromptKit explicitly classifies every -provider field as untrusted. Promoting them to ordinary output creates a -disclosure and compatibility burden that is not currently justified. diff --git a/docs/roadmap/source-releases.md b/docs/roadmap/source-releases.md deleted file mode 100644 index 9b902461..00000000 --- a/docs/roadmap/source-releases.md +++ /dev/null @@ -1,282 +0,0 @@ -# Source-Only Releases - -## Status - -Implemented. Creating the first release under this procedure remains a -separate maintainer operation. - -## Purpose - -Define a repeatable, guarded release process for Notarius without taking on a -binary-distribution system that its current operator audience does not need. -The process should make an exact source revision, its compatibility impact, -and its validation status easy to identify while keeping installation in the -hands of technically capable operators and deployment automation. - -The model is adapted from Weatherreporter's release procedure, but its target -is deliberately narrower: an immutable source tag and checked-in release note -are the release. Notarius does not publish executable archives or support -Windows as part of this work. - -## Release Model - -Notarius releases come from commits on `main` and use stable semantic-version -tags in the form `vMAJOR.MINOR.PATCH`. Prerelease tags are not part of the -initial process. - -Every release has one nonempty, version-matched note at -`docs/releases/.md`. The note and every affected current-state document -must be present in the tagged commit. The Git tag and checked-in note together -are the durable release record; no separately editable release page is -required. - -Published tags are immutable. A maintainer must never move, reuse, or delete a -published tag. If a published candidate is defective, the correction is made -on `main` and released under a new patch version. An unpublished local tag may -be deleted when candidate inspection finds a problem before any remote push. - -Before `v1.0.0`, a minor release may intentionally change a documented CLI, -configuration, durable artifact, integration, or operating contract when its -release note explains the impact and required operator action. A patch release -must not intentionally break those documented contracts within its minor -line. - -The existing `v0.1.0`, `v0.2.0`, and `v0.3.0` tags remain unchanged. They -predate this procedure and do not need retrospective release notes. The first -release made under this process establishes the release-note series. - -## Source-Only Distribution - -Notarius does not publish release binaries, archives, installers, container -images, package-manager entries, checksum files, or signatures. A release tag -is suitable for Go-native installation and for an operator-controlled build -from an exact checkout. - -The primary installation form is: - -```sh -GOWORK=off go install \ - gitea.maximumdirect.net/eric/notarius/cmd/notarius@vMAJOR.MINOR.PATCH -``` - -Operator documentation should also describe cloning the repository, checking -out the tag in detached-head state, and building `./cmd/notarius` with the Go -version declared by `go.mod`. Private-module authentication and `GOPRIVATE` -configuration belong to the operator environment and must be documented by -mechanism rather than with real credentials. - -Consumers such as Narratio should pin the desired Notarius tag in provisioning -or deployment configuration. They must continue to decide runtime -compatibility from Notarius's published receipt and artifact schema contracts, -not merely from the executable's product version. - -Packaged binaries may be reconsidered if distribution demand, installation -friction, or a broader user audience justifies their build, signing, retention, -and platform-support costs. They are not a prerequisite for a disciplined -release process. - -## Platform Policy - -Linux is the supported deployment platform. Release validation must run the -test suite and the release build on Linux and must confirm that the command -builds with `CGO_ENABLED=0` for Linux `amd64` and `arm64`. - -macOS is a best-effort development and testing platform. Release validation -should confirm that the command cross-compiles with `CGO_ENABLED=0` for Darwin -`amd64` and `arm64`, but the project does not promise packaged artifacts or a -separate runtime test environment for those targets. - -Windows is unsupported. The release process must not require Windows builds, -Windows-specific compatibility work, or Windows documentation. Platform- -specific implementation may intentionally use Unix facilities when they are -important to Notarius's filesystem safety and operational model. Any later -decision to support Windows requires its own feature scope and validation -policy. - -## Version Reporting - -Add a root `notarius --version` interface for deployment diagnostics. It -prints exactly one line: - -```text -notarius vMAJOR.MINOR.PATCH -``` - -when the build has a valid release version, and: - -```text -notarius development -``` - -when no release version is available. - -The implementation must obtain the main-module version from Go build -information so `go install ...@vMAJOR.MINOR.PATCH` reports the selected tag. It -must also accept an optional link-time version override so controlled builds -and release CI can identify an exact tag from a checkout. The override must be -validated and must not silently turn arbitrary text into a release version. -Ordinary unversioned checkout builds remain `development`; the release process -must not modify a tracked source constant for each release. - -Version reporting is an informational product interface. It does not replace -receipt, configuration, prompt, or artifact schema versioning, and it must not -be used as the sole downstream compatibility check. - -## Release Notes - -Each new `docs/releases/.md` document has this minimum structure: - -```markdown -# Notarius vMAJOR.MINOR.PATCH - -This release ... - -## Summary - -## Compatibility - -## Upgrade - -## Changes -``` - -The note should concisely explain the release's purpose, compatibility with the -preceding release, operator actions, and material user-visible, operational, -integration, and maintainer-visible changes. It should link to canonical -current-state documentation for exact contracts rather than duplicating those -contracts. - -Release notes are durable historical summaries. They must not contain -credentials, private infrastructure detail, sensitive campaign material, or -claims that are not true of the tagged candidate. A release note does not -excuse stale current-state documentation; affected canonical documents are -updated in the same candidate. - -## Candidate Validation - -The release procedure must provide copyable POSIX-shell guards that validate -the release version, release-note filename and heading, required note sections, -repository state, and module hygiene. Validation must be run from the Notarius -repository root with Go workspace behavior disabled. - -At minimum, a candidate must pass: - -- no tracked `go.work` or `go.work.sum`, no vendored tree, and no `replace` - directive in `go.mod`; -- `GOWORK=off go test -count=1 ./...`; -- `GOWORK=off go test -race -count=1 ./...`; -- `GOWORK=off go vet ./...`; -- `GOWORK=off go build ./...`; -- `GOWORK=off go mod tidy -diff`; -- `gofmt` verification for every tracked Go file; -- `git diff --check` and `git diff --cached --check`; -- validation of both maintained D&D configuration examples with their selected - pipeline; -- Linux `amd64` and `arm64` static command builds; -- best-effort Darwin `amd64` and `arm64` static command builds; and -- a focused manual or automated check that every added or changed local - Markdown link resolves. - -The candidate review also checks for generated binaries, test output, -credentials, temporary files, module replacements, vendored dependencies, and -other unintended source-control content. Tests remain offline and do not call -an LLM provider or require live credentials. - -## Candidate Publication - -The release procedure must guard the exact commit immediately before tagging. -It requires: - -- the current branch is `main`; -- the worktree and index are clean; -- the candidate commit has been pushed and exactly matches `origin/main`; -- the matching release note exists in that commit; -- no local or remote tag already uses the selected version; and -- the substantive release checks have passed for that exact candidate. - -The maintainer records the exact candidate commit, creates a lightweight tag -bound explicitly to that commit, verifies the local tag target, and pushes only -that tag ref. The procedure must not recommend `git push --tags`. - -After publication, the maintainer verifies that the remote tag resolves to the -guarded commit and that the release note can be read from the tagged tree. A -fresh temporary checkout or `go install ...@` must build successfully, and -the resulting command must report the expected version through `--version`. - -## Validation-Only Release Automation - -Add a tag-triggered Woodpecker pipeline that validates source releases without -publishing artifacts. It should: - -- accept only stable semantic-version tags; -- require the version-matched release note; -- run the same substantive module, test, race, vet, build, formatting, and - whitespace checks as the documented local procedure; -- validate the maintained configuration examples; -- perform the supported and best-effort cross-build checks; and -- verify a release-version build's `notarius --version` output on the CI host. - -The pipeline must not upload binaries, create archives or checksums, create or -edit a Gitea release object, or require a release API token. Local guards remain -authoritative before tag publication because CI begins only after the tag is -already remote. - -If tag validation fails, preserve the published tag, fix the cause on `main`, -select a new patch version, and repeat the full process. Do not weaken tag -immutability merely because the release contains source rather than binaries. - -## Documentation Ownership - -In the target state: - -- `docs/release.md` owns the maintainer release procedure, commands, ordering, - publication checks, and failure recovery; -- `docs/releases/` owns one historical summary per release made under the new - process; -- `docs/cli.md` owns the `--version` contract; -- `README.md` owns the shortest source-installation example and links to the - release procedure where useful; -- `docs/development.md` routes release preparation, tagging, and verification - work to `docs/release.md`; -- `docs/policy/documentation.md` assigns canonical ownership to the release - procedure and release notes; -- `docs/policy/architecture.md` records Linux support, best-effort macOS - development, unsupported Windows, and source-only distribution only if those - are judged durable development invariants rather than release mechanics; and -- `docs/operations.md` describes only installation or deployment consequences - relevant to operators and links to canonical CLI and release contracts. - -Current-state documentation must not describe the new release process, -`--version`, or automated validation until the corresponding behavior exists. - -## Acceptance Criteria - -- A maintainer can prepare, validate, tag, publish, and verify a source release - by following `docs/release.md` without relying on undocumented knowledge. -- Every new release has an immutable semantic-version tag and matching - checked-in release note in the tagged commit. -- The guarded candidate is clean, synchronized with `origin/main`, and passes - the documented substantive checks before tagging. -- Tag-triggered CI independently validates the published source and never - publishes binary artifacts. -- `go install` of a tagged version succeeds and `notarius --version` reports - that version; ordinary unversioned builds report `development`. -- Linux is the documented supported deployment platform, macOS has a - best-effort development build check, and Windows is explicitly unsupported. -- Downstream compatibility remains based on durable Notarius contracts rather - than the product version alone. -- Existing pre-procedure tags remain untouched and require no invented release - history. - -## Non-Goals - -- Publishing executable archives, installers, container images, checksums, - signatures, or package-manager entries. -- Supporting or cross-compiling for Windows. -- Creating or maintaining a mutable Gitea release page. -- Supporting prerelease tag syntax in the initial procedure. -- Automating version selection, release-note authorship, commits, or tag - creation. -- Retrospectively creating release notes for `v0.1.0` through `v0.3.0`. -- Treating a product version as a substitute for receipt, configuration, - prompt, or artifact schema compatibility. diff --git a/docs/roadmap/validation-retries.md b/docs/roadmap/validation-retries.md new file mode 100644 index 00000000..552198ad --- /dev/null +++ b/docs/roadmap/validation-retries.md @@ -0,0 +1,609 @@ +# Feedback-Aware Stage Validation Retries + +## Status + +Proposed. This is the active feature roadmap for the next Notarius work set. +Its design decisions are settled. Current behavior remains authoritative until +this roadmap is implemented and the corresponding ADR and canonical +documentation are updated. + +## Purpose + +Make validation an effective corrective boundary around LLM-produced stage +candidates. When deterministic or LLM-backed validators reject a structurally +valid candidate, Notarius should give the producing model the complete, +ordered validation feedback and use the stage's existing retry budget to ask +for a corrected replacement. The feature must distinguish semantic rejection +from producer failure and validator execution failure, preserve the boundary +between PromptKit repair and Notarius stage retries, and remain safe under +concurrency, cancellation, caching, checkpoints, and sensitive input. + +This work is domain-neutral. It establishes the framework behavior required by +future LLM-backed validators such as D&D combat-scene review, but it does not +add that validator. + +## User Intent + +- A stage candidate should be evaluated by every applicable configured + validator before Notarius decides whether to retry or terminate. +- A semantic retry should be materially more useful than repeating the same + request. The producing model should see its latest defective response and + all actionable semantic feedback. +- PromptKit's bounded structural repair and Notarius's stage retry loop are + separate. Each stage attempt receives its own complete PromptKit repair + budget; PromptKit repair never consumes or replenishes the stage budget. +- Deterministic rejection, semantic rejection, producer structural failure, + and validator execution failure are different outcomes and must not be + collapsed into one generic error path. +- The default posture is strict for known-invalid producer output and tolerant + but visible when a validator itself cannot make a decision. +- Corrective prompts must not expose opaque application identifiers, secrets, + or unbounded diagnostic content merely because those values exist in an + internal artifact or operator-facing error. + +## Current State + +The current runner already provides useful foundations: + +- chunk, extract, merge, and normalize producer bindings have one `retries` + value interpreted as additional stage attempts; +- `runWithRetry` retries producer errors and semantic rejections within that + budget; +- PromptKit performs bounded structural repair inside each structured + completion; +- validator targets, execution classes, profile selection, repair policy, + attempts, debug scopes, checkpoint identity, and deterministic public + ordering are already explicit; and +- the structured-completion response retains the model's validated raw bytes + and PromptKit repair metadata. + +The current behavior is not yet the desired corrective workflow: + +- the runner repeats the ordinary producer request after rejection and does + not pass the previous model response or validator feedback; +- validation stops at the first rejection or execution failure, so later + applicable validators do not contribute findings; +- validator execution failure is immediately a framework error rather than a + configurable incomplete-validation outcome; +- `ValidationResult.Message` currently serves operator diagnostics and does + not define separately bounded model-facing guidance; +- typed stage results do not carry the exact model response needed for the + next correction attempt; +- validator-binding `retries` values participate in resolved configuration but + are not used to retry a failed validator against the same candidate; and +- Notarius still pins PromptKit v0.8, while PromptKit v0.9.0 now provides the + append-only request-message API needed for application-owned correction + attempts. + +## Target End State + +For chunk, extract, merge, and normalize stages, Notarius owns one explicit +candidate-attempt state machine: + +1. The producer creates one candidate using the ordinary request. An + LLM-backed producer may use PromptKit structural repair internally. +2. The framework establishes one immutable validation candidate and runs every + applicable validator sequentially in configured order. +3. The framework aggregates approvals, warnings, semantic rejections, + execution failures, and skipped-validator diagnostics without allowing one + validator to mutate the candidate seen by another. +4. A candidate with one or more semantic rejections is never accepted. If the + LLM-backed producer has another stage attempt available, Notarius rebuilds + the complete original prompt and appends the latest defective assistant + response followed by one application-owned correction message containing + every actionable rejection. It then requests one complete replacement + candidate. +5. A producer error consumes the same stage attempt budget under the existing + retry rules, but semantic correction material is used only when a + structurally valid candidate was actually rejected. +6. A validator execution failure is retried, when configured, against the same + immutable candidate. It never regenerates the producer candidate by itself. +7. When budgets are exhausted, the configured terminal policies decide + whether the run fails, a rejected output is recorded, or a structurally + valid candidate advances with explicitly incomplete validation. + +The first attempt remains byte-for-byte the ordinary prompt rendered from the +selected prompt definition. Every correction attempt starts from that same +ordinary prompt rather than from the prior correction conversation. It appends +exactly two messages: + +- an `assistant` message containing the producer-supplied exact defective + response for the latest candidate; and +- a `user` message containing deterministic, bounded, application-owned + correction guidance and asking for one complete replacement response. + +The session ID, prompt ID and version, selected profile, reasoning settings, +structured-output contract, repair budget, named inputs, variables, references, +and reusable prompt prefix remain unchanged across stage attempts. + +## Architectural Ownership + +### PromptKit + +PromptKit continues to own prompt loading and rendering, profile resolution, +backend admission, provider generation, structural validation, and bounded +structural repair within one completion. A PromptKit repair conversation is +private to that completion and is not exposed as a Notarius stage attempt. + +PromptKit v0.9.0 owns the mechanical operation of appending explicitly supplied +messages to a normally rendered prompt before creating the immutable prepared +execution. `RunRequest.AppendedMessages` preserves the original rendered +messages as an exact prefix, validates and defensively copies additions, +includes the complete sequence in prepared details and the rendered-prompt +hash, and runs it through the ordinary generation and structural-repair path. +PromptKit does not impose message-count, byte-size, token, or context-window +limits and permits empty content, so Notarius retains its stricter +application-owned correction validation and bounds. + +### Notarius Framework + +The framework owns stage budgets, immutable candidate preparation, complete +validator-chain execution, result aggregation, outcome precedence, correction +message construction, terminal policy, public ordering, checkpoint effects, +manifest summaries, warnings, and debug lifecycle. + +The framework must remain domain-neutral. It may format stable reason codes and +validator-supplied corrective guidance, but it must not infer D&D or other +domain rules from artifact JSON. + +### Producers And Artifact Families + +The producing module owns prompt selection, prompt inputs, typed decoding, and +the model-facing representation that corresponds to its candidate. An +LLM-backed producer that supports feedback-aware correction must return the +exact response material that the model should see as its prior assistant turn. +It must not substitute a normalized artifact containing deterministically +attached UUIDs or other opaque application identity. + +Artifact-family validators own semantic decisions and domain-specific +corrective guidance. Operator-facing explanation and model-facing correction +are separate contract fields even when their concise text happens to match. + +## Validation Outcome Model + +Each validator invocation produces one of four framework outcomes: + +| Outcome | Meaning | Effect | +| --- | --- | --- | +| Approved | The validator completed and accepted the whole candidate. | Retain its warnings and continue the chain. | +| Rejected | The validator completed and found a semantic defect in the candidate. | Record the finding, continue the chain, and make the candidate ineligible for acceptance. | +| Failed | The validator could not return a usable decision because of an internal, transport, generation, structural-output, or result-invariant failure. | Retry that validator when eligible, then record incomplete validation and continue the chain unless cancellation or framework integrity prevents it. | +| Skipped | Runtime prerequisites for an otherwise selected validator cannot be satisfied. | Record a deterministic incomplete-validation diagnostic and continue; do not invent a semantic decision. | + +Configured validator order controls invocation order and aggregate feedback +order. Execution remains sequential initially. The framework must continue +after a rejection and after an isolated validator failure when it can safely +prepare the remaining validator requests. Cancellation, inability to preserve +an immutable candidate, debug persistence failure, or another framework +integrity failure remains immediately terminal. + +### Outcome Precedence + +For one candidate, apply this precedence: + +1. A producer structural failure means no acceptable candidate exists and + cannot be converted into validator approval. +2. Any completed semantic rejection makes the candidate rejected, even when + another validator failed or was skipped. +3. With no semantic rejection, a validator failure or skip makes validation + incomplete and invokes the validator-failure policy. +4. Only a structurally valid candidate with no rejection and either complete + validation or an explicit `warn_continue` decision may advance. + +Do not turn a known rejection into acceptance through a permissive +validator-failure policy. Do not turn a structurally invalid response into a +rejected-but-usable artifact. + +## Corrective Feedback Contract + +`ValidationResult` should gain a separately bounded, optional model-facing +correction field. A rejecting production validator should provide: + +- a stable reason code suitable for aggregation and provenance; +- an operator-facing message suitable for ordinary diagnostics; and +- concise corrective guidance that explains the violated rule without asking + the model to reproduce opaque identity or leaking unrelated source data. + +The framework constructs one deterministic correction message from all +rejections in validator order. Each entry identifies the stable reason code +and corrective guidance. Duplicate identical entries may be collapsed while +preserving first occurrence; distinct findings must not be discarded merely +to shorten the message. If a validator rejects without model-facing guidance, +the framework uses a generic reason-code-based correction rather than copying +the operator message automatically. + +Warnings, validator failures, skipped diagnostics, provider messages, stack +traces, debug paths, and sensitive values are not corrective guidance. They may +be recorded through their proper diagnostic channels but must not be presented +to the producer as candidate defects. + +The framework must validate UTF-8, role, non-empty content, and +application-owned size limits before constructing the correction request. Oversized or +invalid correction material is a framework-owned inability to perform a +feedback retry; it must never be silently truncated into a misleading or +syntactically defective assistant response. + +## Producer Correction Contracts + +Introduce application-owned, defensively copied correction contracts at the +framework boundary: + +- chunk, typed extraction, typed merge, and typed normalize results can carry + optional model-facing candidate material associated with their returned + value; +- the corresponding requests can carry an optional correction containing the + latest assistant material and aggregated guidance; +- `StructuredCompletionRequest` can carry the two bounded appended messages + without importing PromptKit types into module or pipeline contracts; and +- the PromptKit adapter translates those application-owned messages into + `RunRequest.AppendedMessages` using `promptkit.RoleAssistant` and + `promptkit.RoleUser` before preparation. + +A semantic correction always supplies exactly two appended messages: the +latest defective response as `assistant`, followed by the aggregate correction +request as `user`. The framework does not expose the other PromptKit-supported +roles through this contract and does not accumulate messages from earlier +stage attempts. PromptKit preserves message content exactly, but Notarius must +reject empty content and enforce its own per-message and aggregate byte limits +before the adapter is called. + +Correction material is attempt-local sensitive data. It is not part of the +artifact schema, checkpoint value, cache key, durable output bundle, ordinary +error, or configuration summary. The policy and capability that affect +execution do participate in resolved pipeline and checkpoint identity. + +LLM-backed modules selected with both `retries > 0` and a non-empty validator +chain must declare whether they can produce and consume correction material. +Preparation must reject a pipeline that could request feedback-aware semantic +retries from an LLM-backed producer without that capability. An LLM-backed +producer with no validators may continue to use its retry budget for +operational failures without declaring semantic-correction capability. + +A correction-capable producer must supply the exact single LLM response that +directly controlled the candidate being validated. Direct D&D chunk and +extraction producers expose their exact structured response. The shared +semantic-reconciliation path exposes its exact proposal response through its +typed normalizers without turning request-local batch handles into durable +identity. Deterministic transformations after that response are permitted only +when the validated candidate remains directly traceable to it. + +A producer whose candidate combines multiple LLM responses is not +correction-capable under this initial protocol. It may continue to use ordinary +operational retries when no semantic correction can occur, but configuration +must reject a validator-backed retry workflow for it. Supporting compound +producers later requires a separately reviewed multi-response protocol; the +framework must not synthesize an assistant message by serializing the final +typed artifact. + +Deterministic producers do not receive correction material. A deterministic +candidate rejected by validation immediately applies the terminal semantic +rejection policy without consuming retries that cannot change the result. + +## Retry Budgets + +### Producer Stage Budget + +The existing producer binding `retries` field remains the sole outer stage +budget. `retries: N` means at most `N` additional complete producer attempts +after the initial attempt. Producer operational errors, producer structural +failures, module-requested normalize retries, and semantic corrections all +draw from this same budget. Do not add a separate semantic retry counter. + +Every LLM-backed producer attempt receives the configured PromptKit +`structured_output_repair_attempts` value independently. Notarius does not +decrement that value across stage attempts. + +### Validator Budget + +Use the existing `retries` field on an LLM-backed validator binding for +additional attempts to obtain a usable decision about the same immutable +candidate. A completed approval or rejection is terminal for that validator +and does not consume another validator attempt. A validator retry reconstructs +the same ordinary validator prompt; it does not append semantic feedback about +the validator's prior failed judgment and does not create a recursive +Notarius correction loop. + +Reject a positive validator `retries` value on a deterministic validator at +configuration resolution because repeating the same pure decision cannot +improve it. Validator retries do not consume the producer stage budget. + +## PromptKit v0.9.0 Adoption + +The target end state pins PromptKit v0.9.0 for correction requests. The +resolved dependency graph includes its independently versioned +OpenRouter and Rakestrawhome catalog modules through ordinary Go module +resolution; Notarius must not import or register those catalogs directly. +PromptKit continues to own their built-in backend and profile IDs, source +precedence, credentials, and capacity behavior. + +Notarius's PromptKit compatibility documentation and built-in-profile +checkpoint marker identify v0.9.0 rather than v0.8.0. The PromptKit release +identity remains the conservative checkpoint identity for the exact catalog +versions selected by that release; Notarius should not duplicate upstream +catalog module versions in a second hand-maintained marker. + +PromptKit v0.9.0 restricts text-chat roles to `developer`, `system`, `user`, and +`assistant`. Maintained Notarius prompt definitions already use only `system` +and `user`; correction requests add only `assistant` and `user`. PromptKit +`RunRequest` literals remain keyed. These compatibility conditions must remain +covered by the ordinary production-asset and adapter checks without adding a +brittle inventory test that merely counts prompt messages or literals. + +## Terminal Policy Configuration + +Add an optional `validation_policy` object at pipeline scope and on chunk, +extract, merge, and normalize producer bindings: + +```yaml +validation_policy: + producer_structural_failure: fail_run + semantic_rejection: fail_run + validator_failure: warn_continue +``` + +The binding object overrides individual pipeline values; resolution is +field-by-field in binding, pipeline, application-default order. Omitted values +inherit rather than replacing the complete object. Explicit null, unknown +fields, and unknown enum values are invalid. The effective policy is resolved +and detached before execution, appears in redacted effective configuration and +run provenance, and participates in the resolved pipeline digest and checkpoint +identity. + +The initial enum values and defaults are: + +- `producer_structural_failure`: `fail_run` by default; `reject_output` may + retain a terminal rejection and final raw candidate for debug, but may not + advance or publish an invalid artifact; +- `semantic_rejection`: `fail_run` by default after stage attempts are + exhausted; `reject_output` records the aggregate rejection and allows + unrelated work to complete without advancing that candidate; and +- `validator_failure`: `warn_continue` by default, which advances a + structurally valid and otherwise unrejected candidate with explicit + incomplete-validation provenance and one genuine warning; `fail_run` + terminates the run. + +Producer structural policy applies only to LLM-backed producers. Semantic and +validator-failure policies apply to any validated producer. Input and output +bindings do not accept `validation_policy`, and validator bindings do not own +terminal policy; they own only their decision and their own operational retry +budget. Candidate disposition belongs to the chunk, extract, merge, or +normalize producer binding after its complete validator chain has run. + +Keep the current file-configuration version. The syntax is strictly +decodable without a version change. The project is pre-v1, but the behavior +and output changes should still be called out in the next release note and +downstream documentation. + +## Stage-Specific Behavior + +### Chunk + +A generated chunk plan is structurally validated and materialized before the +validator chain runs. Semantic feedback applies to the exact raw chunker +response associated with that plan. + +When an automatically reused chunk-plan record is rejected by the current +validator chain, treat the record as unusable for this invocation and enter +ordinary generation at attempt one. A cache hit is not a new model attempt and +does not supply model-facing assistant material. Do not overwrite the cached +record until a newly generated plan is accepted. Refresh and bypass modes +retain their existing publication rules. + +### Extract + +Each chunk-scoped extraction job owns its own attempt state and correction +conversation. One rejected chunk candidate does not cancel unrelated chunks or +lanes unless terminal policy converts it into a framework error. Deterministic +public ordering remains chunk-first and lane-second regardless of concurrent +completion. + +### Merge And Normalize + +Merge and normalize remain serial within a lane. A correction attempt receives +the same accepted upstream artifacts and references as the initial attempt. +The existing safe-fallback `NormalizeRetry` mechanism must be reconciled with +the shared attempt state rather than layered into a second retry loop: it uses +the same stage budget, retains its documented fallback behavior, and cannot +override a known validator rejection. + +The initial feature supports one exact producer-supplied assistant response per +candidate attempt. Future multi-request normalization or batching must define +which response directly represents the candidate, or supply a new explicitly +reviewed correction protocol, before it can claim feedback-aware correction. + +## LLM-Backed Validators + +An LLM-backed validator uses the same scheduled PromptKit client, selected +profile, session, timeout, and structural-repair policy as other LLM-backed +modules. PromptKit may structurally repair its response inside one validator +attempt. + +- A contract-valid validator response is its decision; Notarius does not ask a + second LLM to judge that judgment. +- A structurally invalid final validator response, transport failure, or + deterministic violation of the validator-result contract is a validator + execution failure. +- Validator execution retries reuse the immutable producer candidate and do + not regenerate it. +- Exhaustion invokes `validator_failure` policy and emits a genuine warning + under `warn_continue`. + +This feature supplies the generic execution model only. It does not register a +production LLM-backed validator or change a D&D default validator chain. + +## Provenance, Diagnostics, And Sensitive Data + +Attempt debug output should make the state machine auditable. When debug is +enabled, record: + +- producer attempt number and whether it was initial, error retry, module + retry, or semantic correction; +- PromptKit repair count and cumulative usage for every completion; +- each validator's configured-order outcome and validator attempt count; +- aggregate rejection codes and the bounded correction message; +- effective terminal policy and the decision it produced; and +- whether validation was complete, rejected, or incomplete. + +Raw assistant responses and correction messages belong only in explicitly +requested detailed debug traces, following existing allowlisted content-file, +redaction, permission, and retention rules. Ordinary errors, CLI output, +warnings, manifests, checkpoints, caches, and run receipts contain identities, +counts, bounded safe summaries, and reason codes—not raw source or model +content. + +The durable run manifest and rejection summaries should record enough +structured information to distinguish: + +- the number and kinds of producer attempts; +- completed semantic rejection and all rejecting validator identities; +- incomplete validation and failed or skipped validator identities; +- the effective terminal policy and terminal result; and +- successful use of a correction attempt without treating it as a warning. + +Warnings from abandoned producer attempts must not be promoted. Warnings from +the accepted attempt remain eligible. A warn-and-continue validator failure +produces one bounded, deterministically ordered warning per affected validator +after its retry budget is exhausted; detailed repeated failures stay in debug +provenance. + +## Checkpoints, Caches, Concurrency, And Cancellation + +- Effective validation policy, producer correction capability/protocol + version, validator chain, validator retry budgets, and prompt assets must all + affect checkpoint identity. +- Only accepted, completely validated stage outputs may be checkpointed or + reused. Rejected, structurally invalid, and validation-incomplete outputs + accepted under a permissive policy must not be written as reusable stage + checkpoints. This conservative rule avoids treating a transient validator + outage as durable validation success; a future checkpoint-status contract may + revisit it explicitly. +- Correction attempts use the same run-wide scheduler and worker bounds as + initial completions. No retry path may bypass provider admission. +- A scheduled permit covers the complete PromptKit operation, including its + internal structural repair, and is reacquired normally for a later Notarius + stage attempt. +- Parent cancellation dominates producer, validator, retry, debug, cache, and + checkpoint work. Cancellation never becomes a rejection, warning, or + incomplete-validation acceptance. +- Framework errors retain deterministic selection and cancellation behavior + across concurrently executing chunks and lanes. + +## Architecture Record And Canonical Documentation + +The target end state includes an accepted ADR that records: + +- the separation between PromptKit structural repair, producer stage attempts, + and validator execution retries; +- the complete validator-chain aggregation rule and outcome precedence; +- the fresh reconstruction plus two-message correction protocol; +- module ownership of model-facing candidate material; +- deterministic producer and non-recursive validator behavior; +- default fail-closed and fail-open terminal policies; and +- provenance, cache, identity, and sensitive-data constraints. + +The canonical owners describe the implemented behavior without duplicating +one another: + +- `docs/policy/architecture.md` for durable validation and retry invariants; +- `docs/config.md` for fields, values, precedence, defaults, and validation; +- `docs/operations.md` for costs, failure behavior, warnings, debug handling, + and recovery; +- `docs/internal/pipeline.md` for the attempt state machine, aggregation, + checkpoint behavior, and concurrency; +- `docs/internal/llm.md` for appended correction messages and the distinction + from PromptKit repair; +- `docs/internal/modules.md` for producer and validator contracts; +- `docs/integrations/pkg-promptkit.md` for PromptKit v0.9.0, + `RunRequest.AppendedMessages`, supported message roles, application-owned + bounds, and the independently versioned upstream catalog boundary; and +- affected output and subprocess integration documents for durable validation + status and rejection summaries. + +Until this target state is implemented, canonical current-state documentation +continues to describe the existing behavior. + +## Testing Strategy + +Tests should protect observable state-machine behavior rather than private +helper layout or exact prose. The target test suite includes: + +- contract tests proving the first request is unchanged and a correction + request contains the same initial messages plus exactly one assistant and one + user message; +- behavioral runner tests for all-approved, multiple-rejection, + rejection-plus-failure, failure-only, skipped, retry-success, and each + terminal policy outcome; +- one representative path for chunk, extract, merge, and normalize, without + duplicating the complete state matrix at every stage; +- proof that all validators see immutable equivalent candidates and execute in + configured order after an earlier rejection or isolated failure; +- proof that validator retries reuse the candidate and do not consume producer + retries; +- proof that deterministic rejection does not repeat the producer; +- focused PromptKit-adapter tests proving that application-owned correction + messages map to the two intended PromptKit roles without content leakage; +- config parsing, precedence, invalid-placement, round-trip, redaction, and + digest tests for effective policy; +- checkpoint and chunk-cache tests for rejected, corrected, incomplete, and + accepted outcomes; +- warning, manifest, receipt, debug, and sensitive-content tests at their + canonical boundaries; and +- a small assembled D&D pipeline test proving a rejected direct extraction can + be corrected without a live provider. + +Tests remain offline and deterministic. Do not reproduce PromptKit's internal +message-copying, rendered-hash, prepared-execution, capacity, or repair suite. +One representative adapter or assembled-run test should prove that PromptKit +structural repair remains usable after Notarius appends semantic-correction +messages. Do not snapshot full prompts or error prose, assert private +constants, or multiply equivalent tests across every D&D artifact family. + +## Acceptance Criteria + +- Every applicable validator runs in configured order and contributes one + explicit outcome before candidate disposition. +- Multiple semantic rejections produce one bounded, deterministic correction + request containing all actionable findings. +- Correction attempts reconstruct the exact ordinary prompt and append only + the latest defective assistant response and one correction message. +- Notarius pins PromptKit v0.9.0 and routes correction messages through + `RunRequest.AppendedMessages`; it does not maintain paired correction prompt + manifests or bypass PromptKit's normal execution path. +- Every correction-capable LLM producer exposes the exact single response that + directly controlled its candidate. Configuration rejects semantic retries + for compound producers that cannot satisfy that contract. +- The existing producer `retries` value is the only producer-stage budget; + PromptKit structural repair and validator execution retries remain separate. +- Deterministic producers are not repeated after semantic rejection. +- Validator execution failure is never described to the producer as a + candidate defect and never creates recursive semantic validation. +- Default terminal behavior is `fail_run` for structural failure and semantic + rejection, and `warn_continue` with explicit incomplete validation for + validator failure. +- Terminal policy resolves field by field from producer-binding override to + pipeline default to application default; individual validators do not own + candidate disposition. +- Permissive policy never advances known rejected or structurally invalid + output. +- Raw model responses and correction content are confined to model requests and + explicitly requested debug traces. +- Checkpoint, cache, manifest, warning, concurrency, cancellation, and + deterministic-ordering invariants remain intact. +- Canonical architecture, configuration, operations, internal, integration, + and ADR documentation accurately describe the implemented behavior. +- Focused, full, and race-enabled Go tests; vet; builds; example validation; + formatting; link checks; and repository hygiene checks pass. + +## Non-Goals + +- Adding the D&D combat-scene semantic validator. +- Redesigning the warning taxonomy beyond the warnings required for validator + failure and retry outcomes. +- Concurrent validator execution. +- Unbounded or accumulating conversational history. +- A second semantic retry counter. +- Recursive LLM judgment of LLM-validator decisions. +- Provider-specific retry policy or bypassing PromptKit. +- General workflow graphs or new pipeline stages. +- Large-collection reconciliation batching or a generic multi-response + correction protocol.