diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 8bf79020..ad413ec6 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -5,13 +5,230 @@ configuration, operations, internal, and integration docs. This roadmap records future work only. Items are ordered roughly by current value and specificity, not as committed release dates. +## Near-Term Validation And LLM Reliability + +The following work forms one related program but should be promoted into +separate feature roadmaps and implemented in dependency order. PromptKit owns +structural output repair within one completion. Notarius owns stage candidates, +validator chains, semantic rejection policy, and whether another stage attempt +is warranted. + +### 1. Upgrade To PromptKit v0.8.0 + +- Upgrade the PromptKit dependency and follow the upstream v0.8.0 release + guide, including its now-active `repair_attempts` behavior and compatibility + checks for prompt definitions, prepared execution, validation results, + capacity accounting, error adaptation, debug data, and cumulative token + usage. +- Use PromptKit's bounded repair loop for eligible structural output failures. + A repair must resend the immutable original prompt followed by the latest + defective assistant response and one bounded user correction message. It + must not accumulate the history of every defective candidate. +- Review every maintained LLM prompt's structural repair budget. Prefer a + small positive budget where a corrected structured response can be useful, + while retaining zero where a second call would be inappropriate or where + observational evidence does not justify it. +- Keep PromptKit repair attempts distinct from Notarius stage retries. + PromptKit repairs a response that fails its deterministic `basic`, `json`, or + `json_schema` contract within one stage attempt; Notarius may later retry a + complete stage candidate after application validation. Document and test the + multiplicative maximum provider-call cost created by both configured + budgets, including PromptKit repair calls made by any LLM-backed validators. + Attribute generation count, token usage, latency, and failure provenance to + the producer, validator, PromptKit repair, and Notarius correction layers so + operators can explain the actual cost of a run. +- Treat exhaustion of PromptKit structural repair as the absence of a usable + structured candidate. The default terminal policy is to fail the run after + the applicable Notarius stage-attempt budget is also exhausted. A future + configurable alternative may reject the affected stage or lane where the + pipeline can represent that outcome, but it must never accept undecodable or + structurally invalid output merely with a warning. + +### 2. Feedback-Aware Stage Validation Retries + +- Model Notarius's corrective stage-retry conversation explicitly after + PromptKit v0.8.0. The first attempt sends the ordinary complete initial + prompt. If application validation rejects the resulting LLM-produced + candidate and another stage attempt is available, reconstruct that complete + initial prompt byte-for-byte and append exactly two messages: an assistant + message containing the defective response and an application-owned user + message detailing every applicable semantic validation error and requesting + one corrected, complete replacement response. This is a freshly constructed + correction request, not continuation of an accumulating conversation. +- Use the configured stage `retries` value as the one outer retry budget for + this loop. `retries: N` continues to mean at most `N` additional complete + chunk, extract, merge, or normalize attempts after the initial attempt, + whether an attempt is needed because of a producer error or semantic + rejection. Do not add a second semantic-correction count. PromptKit's + prompt-level `repair_attempts` budget is independent and internal to each + individual LLM completion, and does not consume or replenish the Notarius + stage budget. +- Extend the framework-managed validation boundary for chunk, extract, merge, + and normalize stages so a rejected LLM-produced candidate and its exact raw + model response remain available to construct the next stage attempt. + Deterministic producers cannot improve by repeating the same inputs; a + rejection from a deterministic stage is therefore terminal under the + configured rejection policy rather than consuming retries mechanically. +- Preserve the original session ID, selected profile, structured-output + contract, prompt inputs, and reusable prompt prefix. Carry only the latest + candidate and latest aggregate feedback; do not build an unbounded retry + conversation. Keep model-facing corrective guidance separate from + operator-facing diagnostics, and apply explicit size, redaction, and debug + disclosure rules to both. +- Run every applicable validator in the configured chain before deciding + whether to retry. Do not short-circuit merely because an earlier validator + rejected the candidate. Aggregate all semantic rejection reason codes and + corrective guidance into the retry message so one retry can address the + whole candidate. A validator is applicable only when its declared target and + prerequisites can be satisfied; record a deterministic skipped diagnostic + rather than invoking a validator on an input it cannot interpret. Initially + execute the chain sequentially in configured order so results, diagnostics, + costs, and feedback ordering remain deterministic; consider validator + concurrency only in response to measured latency. +- Continue running independent applicable validators after one validator + execution failure so the attempt retains as much useful diagnostic + information as practical. Do not present validator operational failures as + defects in the producer candidate and do not include them in corrective + feedback. +- Distinguish three terminal conditions and make their policies configurable + at a coherent pipeline or binding scope: + - **producer structural failure:** PromptKit could not return a usable + structured candidate after its repair budget. Default to `fail_run`; an + allowed alternative may record a terminal stage or lane rejection where + execution can safely continue, but may not accept the invalid output; + - **semantic rejection:** one or more validators completed and rejected the + candidate. Default to `fail_run` after corrective stage retries are + exhausted; allow an explicit alternative that records the existing + rejected-output outcome without advancing that output; + - **validator execution failure:** a validator could not produce a valid + decision because of generation, structural-output, transport, or internal + failure. Default to a genuine warning and an explicitly recorded + `validation_incomplete` or equivalent degraded state while allowing the + candidate to continue; allow strict configuration to fail the run instead. +- An LLM-backed validator uses the same scheduled PromptKit boundary as every + other LLM-backed module. Its own response may use PromptKit's bounded + structural repair. Distinguish its possible output states: + - output rejected by PromptKit's structural contract should consume only the + validator prompt's configured PromptKit repair budget; + - output that is structurally valid but violates a deterministically + checkable validator-result invariant should be classified as a validator + execution failure; + - output that satisfies the complete validator-result contract is the + validator's decision, even though an LLM judgment may remain imperfect. + Automatically judging that judgment would require another semantic + validator and is outside this feature. + If the validator cannot return a contract-valid decision, do not recursively + create another Notarius semantic-validation loop around it. Apply the + configured validator-failure policy. The default warning must identify the + validator and affected stage without exposing sensitive content. +- Separate validator execution retry from producer correction. A transient + validator operational failure must not automatically discard and regenerate + an otherwise usable producer candidate. Any bounded retry of the validator + itself should reuse that same immutable candidate and remain subordinate to + PromptKit and provider retry behavior. +- Preserve attempt-level provenance, cumulative token usage, validator + outcomes, aggregated correction feedback, and terminal policy decisions in + the debug and manifest models without copying raw source material into + ordinary errors or durable summaries. +- Define terminal-outcome precedence. A semantic rejection dominates a + validator execution failure for the same candidate: use the completed + rejections to correct the producer while separately recording incomplete + validation. If a later candidate has no semantic rejection but one validator + still fails, apply the configured validator-failure policy to that candidate. + Never allow a known semantic rejection to become accepted through a + warn-and-continue setting, and never accept a structurally invalid producer + response. Permissive policy may preserve a rejected-output outcome or accept + a structurally valid candidate with explicitly incomplete validation; it may + not relabel known-invalid output as approved. + +Before implementation, record the generic validation and retry state machine +in an ADR. The ADR should own the separation between PromptKit repair and +Notarius correction, use of the existing stage-retry budget, reconstruction of +correction conversations, all-applicable-validator aggregation, deterministic +validator ordering, non-recursive validator failure handling, outcome +precedence, default fail-open/fail-closed choices, configurable terminal +policies, and provenance and sensitive-data constraints. A dependency-upgrade +ADR is not needed for PromptKit v0.8.0 itself. Current behavior remains +authoritative until the validation ADR is implemented and the canonical +architecture, configuration, operations, and internal documentation are +updated. + +### 3. D&D Combat Scene Semantic Validation + +- Add an optional production LLM-backed D&D validator that determines whether + proposed scene boundaries and classifications represent substantive active + combat correctly. Its central quality goal is that active combat is kept in + coherent scenes classified as `combat`, rather than split incorrectly or + hidden inside scenes classified as `narrative`, `recap`, or `meta`. +- Resolve the validator's exact target before implementation. The current + `dnd/scenes` chunker owns only complete, gap-free source ranges, while the + per-chunk `dnd/scene-descriptions` extractor owns the `combat`, `narrative`, + `recap`, and `meta` classification. The preferred initial placement is + therefore an extract-stage validator for `dnd/scene-descriptions`, where it + can compare one proposed kind with the corresponding transcript chunk. +- Consider a chunk-stage LLM validator only for a distinct boundary-coherence + question that can be answered from the complete transcript and proposed + range map, such as whether one continuous combat was fragmented across + inappropriate scene boundaries. Do not duplicate the same classification + judgment at both stages. Moving classification into chunk-plan annotations + would change the deliberately minimal, annotation-free chunk contract and + requires an explicit architecture review before it is selected. +- Validate both false negatives and false positives: a non-combat kind must not + omit substantive active combat, and a combat kind must be supported by such + combat. Keep the existing deterministic downstream rule that combat-turn + extraction runs only for an exact `combat` scene classification; semantic + review improves the upstream classification but does not replace that gate. +- Run the semantic validator through PromptKit, use a minimal required-field + structured response schema, and let PromptKit repair structural validator + output within its bounded budget. A contract-invalid final validator response + is a validator execution failure, not a semantic rejection and not a reason + to recursively validate the validator. +- Evaluate the prompt and decision policy against a small human-reviewed set + containing combat setup, active turns, interruptions, multi-phase encounters, + brief rules discussion, aftermath, recalled combat, and false-positive + hostile dialogue. Measure false acceptance, false rejection, retry success, + added calls, latency, and token cost before placing it in the production + default chain. +- An ADR is not required if classification remains owned by + `dnd/scene-descriptions` and the validator follows the generic validation ADR. + Create or supersede an ADR if the work transfers scene classification into + the chunker or otherwise changes stage ownership or the durable chunk-plan + contract. + +### 4. Warning Signal And Presentation Reform + +- Audit every warning producer and representative successful runs. Ordinary + success producing dozens of warnings is a failed operator experience: the + volume obscures actionable problems and trains operators to ignore the + warning channel. +- Define a small warning taxonomy that distinguishes actionable degradation, + incomplete validation, lossy fallback, and data-quality risk from routine + normalization observations or informational diagnostics. Preserve detailed + traceability in debug or manifest data without promoting every observation + to a top-level CLI warning. +- Consider stable deduplication and aggregation by scope and reason code, + bounded samples plus omitted counts, and a concise CLI summary with a path to + detailed diagnostics. Do not suppress genuine validator execution failures + merely to reduce the count. +- Decide which warnings affect process status, rejection summaries, durable run + receipts, or only debug output. Ensure warning ordering and aggregation are + deterministic across concurrent execution. +- Establish a representative warning-volume acceptance target and human review + workflow before changing individual producers piecemeal. The intended result + is not zero warnings; it is a small set in which every surfaced warning merits + operator attention. +- This work does not require an ADR unless it changes validation acceptance, + failure, or durable contract semantics. CLI presentation and diagnostic + taxonomy otherwise belong in a feature roadmap followed by updates to their + canonical configuration, operations, integration, and internal documents. + ## Near-Term D&D Pipeline ### Evaluate Spell Extraction And Normalization - Evaluate ordinary extraction retries and the completed normalization path - against a human-reviewed transcript set before adding repair-aware retries or - an LLM-backed semantic validator. + against a human-reviewed transcript set before and after adopting the shared + PromptKit repair and Notarius validation-retry policies above. - Maintain a small set of human-reviewed transcripts and outputs for prompt, validator, and normalizer development. Treat model-quality review as an iterative human evaluation aid, not a deterministic correctness gate.