From 837efc78a7582d581401974d6f3277f190fc244e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 23 Sep 2026 18:18:45 +0000 Subject: [PATCH] Plan PromptKit adoption with structural repair enabled --- docs/roadmap/configkit.md | 11 ++-- docs/roadmap/library-adoption.md | 90 +++++++++++++++++++++----------- docs/roadmap/runtime-behavior.md | 9 ++-- 3 files changed, 73 insertions(+), 37 deletions(-) diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index b8da66c..b204926 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -121,9 +121,14 @@ ConfigKit implementation, then PromptKit implementation, followed by a combined v2.0.0 release. Revisit that sequence alongside other major changes. Keep implementation units reviewable even when the release combines them. -The separate [runtime behavior decisions](runtime-behavior.md) remain unresolved. -Coordinate the retry-count question with the new configuration semantics where -appropriate; do not silently change runtime behavior merely by changing loaders. +The separate [runtime behavior decisions](runtime-behavior.md) track remaining +questions and the selected retry direction. +The coordinated plan adopts PromptKit's explicit retry semantics and enables +structural repair from the outset. Include separate transient-retry and positive +repair budgets, plus the total-operation deadline policy, in the shared settings +design. The reviewed PromptKit baseline is v0.11.0; exact defaults and settings +ownership remain detailed-planning decisions. Do not implement the runtime change +merely as a side effect of replacing the configuration loader. Retention and transcript timestamp decisions are not automatically included. ## Open Design Decisions and Boundaries diff --git a/docs/roadmap/library-adoption.md b/docs/roadmap/library-adoption.md index b69db9d..78896c6 100644 --- a/docs/roadmap/library-adoption.md +++ b/docs/roadmap/library-adoption.md @@ -21,9 +21,12 @@ infrastructure while keeping transcript-correction responsibilities in Audita. This roadmap owns sequencing across the two adoptions. The [ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope. -PromptKit consumer requests are recorded upstream in -`../promptkit/docs/roadmap/audita-feature-requests.md`; they remain proposals until -accepted and released there. Do not treat proposed upstream capabilities as present. +PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies +transient HTTP retries, execution observations, typed failure categories, an offline +proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream +consumer requests are no longer outstanding prerequisites on those points. +Audita has not yet adopted either library; recheck the selected releases during +detailed planning. ## Migration Posture @@ -43,9 +46,10 @@ changes, especially changes to module or pipeline configuration. Define the target settings model for both libraries before implementing either migration. Decide how Audita selects proposal and validation execution profiles, -supplies credentials, selects retry behavior, and assigns per-backend concurrency -limits. Give each setting one authoritative home; avoid duplicate model, endpoint, -credential, or timeout definitions with competing precedence. +supplies credentials, selects transient-retry and structural-repair budgets, and +assigns per-backend concurrency limits. Give each setting one authoritative home; +avoid duplicate model, endpoint, credential, or timeout definitions with competing +precedence. Keep ConfigKit application configuration distinct from PromptKit execution profiles. In particular, optional ConfigKit configuration profiles and PromptKit execution @@ -66,10 +70,24 @@ call and one LLM-validator call through PromptKit using controlled offline respo Verify profile selection, credentials, embedded prompts and schemas, shared backend capacity, failed output validation, cancellation, and diagnostic mapping. -Establish how retries and content repair differ and how their budgets and execution -observations reach Audita. Keep optional content repair disabled initially unless a -specific need justifies it. Audita's domain acceptance checks remain necessary even -when generated output satisfies a JSON schema. +Enable structural repair from the outset for proposal and LLM-validator outputs. +Additional attempts to obtain valid output are desired behavior, not a deferred +optimization. Select explicit positive repair budgets during detailed planning and +exercise invalid output that becomes valid after repair, as well as exhausted +repair budgets. + +Keep transient HTTP retries and structural repair distinct. With the reviewed +library, each generation, including a repair, has its own transient-retry budget; +plan and document the combined request allowance and overall deadline. Adopt +PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy +zero-as-default behavior. The library's narrower transient retry policy does not +retry transport failures or malformed provider responses; do not restore the old +policy through an overlapping application retry loop. + +Verify that initial, retry, and repair calls share backend limits and that their +observations reach Audita without double-counting usage. Audita's domain acceptance +checks remain necessary after structural repair succeeds; valid JSON does not imply +an acceptable correction or validator decision. Use the prototype to resolve genuine boundary gaps and inform configuration design. It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing @@ -100,9 +118,10 @@ Remove the temporary bridge from the preceding stage. ### 5. Validate and Release the Combined Migration Validate the assembled configuration-to-execution workflows, including local and -API backend limits, failure/cancellation behavior, credential handling, diagnostics, -and domain outcomes. Use deterministic tests and controlled endpoints rather than -paid calls as acceptance gates. Update documentation and examples together with +API backend limits, successful and exhausted repairs, transient retries, +failure/cancellation behavior, credential handling, diagnostics, and domain outcomes. +Use deterministic tests and controlled endpoints rather than paid calls as acceptance +gates. Update documentation and examples together with implemented behavior and remove obsolete contracts and code. Complete one migration guide covering files, CLI, environment settings, credentials, @@ -110,31 +129,40 @@ execution profiles, inspection, and changed diagnostics. Release the combined breaking transition under the established [release procedure](../release.md), with no CI waiting added to release completion. -## Parallel Upstream Preparation +## Reviewed Upstream Readiness -PromptKit work can proceed alongside Audita's design and ConfigKit implementation. -Prioritize evaluation of transient-failure retries, actionable failure classification, -and execution observations, supported by a representative consumer example. Existing -per-backend capacity controls are the intended foundation; additional scheduling -features are not assumed necessary. +The v0.11.0 review covered the relevant implementation, the full test suite, +targeted race tests, and the offline proposal/assessment example; those checks +passed. Existing per-backend controls cover the requested capacity model, including +retry and repair interactions. No further upstream feature request is currently a +prerequisite to the prototype. Keep the planned adoption sequence unchanged. -Resolve the proposed LGPL 3.0 licensing change and applicable distribution terms -before selecting PromptKit for adoption. Recheck released APIs, toolchain requirements, -and dependency versions at implementation time. Pin published library releases; -production builds must not depend on sibling checkouts or local module replacements. +Remaining integration work includes selecting a total-operation deadline alongside +per-attempt timeouts, using shared backend identities with finite limits, and mapping +PromptKit observations to Audita-owned diagnostic records. Observation callbacks are +synchronous and may overlap across runs; collection must be short and safe for +concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat +observations as a complete billing ledger or a stable serialized report format. -If a required upstream capability is not ready, complete independent work and -reassess sequencing explicitly rather than quietly recreate that capability as -permanent Audita infrastructure. The prototype should identify what is genuinely -required for adoption versus useful later enhancement. +The licensing change has shipped as LGPL-3.0-or-later. Account for applicable +notices and source/rebuild distribution requirements in Audita's release design; +relicensing no longer needs to be requested upstream. Recheck released APIs, +toolchain requirements, and dependency versions at implementation time. Pin +published library releases; production builds must not depend on sibling checkouts +or local module replacements. + +If the prototype reveals a genuine remaining gap, evaluate it explicitly rather +than quietly recreate shared library functionality as permanent Audita infrastructure. ## Coordination and Open Decisions - Finalize the configuration/profile boundary, defaults, credential ownership, and - retry semantics before committing to the new file schema and CLI design. -- Define how failed validation, exhausted retries, and partial execution observations - map to Audita warnings, rejected work, and failed runs. -- Coordinate the [runtime retry-count question](runtime-behavior.md) with this design. + retry/repair budgets and deadline policy before committing to the new file schema + and CLI design. Structural repair is enabled by the agreed direction; its positive + default count remains to be selected. +- Define how exhausted repair, failed validation, exhausted retries, and partial + execution observations map to Audita warnings, rejected work, and failed runs. +- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration. Retention and transcript timestamp questions are not automatically included. - Revisit ordering when other major changes are defined. Prefer designing shared module/pipeline settings once, while keeping unrelated work independent. diff --git a/docs/roadmap/runtime-behavior.md b/docs/roadmap/runtime-behavior.md index 76f7794..9181317 100644 --- a/docs/roadmap/runtime-behavior.md +++ b/docs/roadmap/runtime-behavior.md @@ -3,13 +3,16 @@ **Last updated:** 2026-09-23 The documentation migration identified these existing behaviors for separate -product consideration. No behavior change is approved or implemented by this list. +product consideration. Implementation status and any selected direction are noted +below; this document does not claim that runtime changes have landed. - Decide whether the `never` retention selection should delete successful-run diagnostics; current behavior is documented in [operations](../operations.md#retention-behavior). -- Decide whether an explicit zero retry count should disable retries; current - adapter behavior is documented in [configuration](../config.md). +- The [library-adoption plan](library-adoption.md) selects explicit zero as disabling + transient retries, alongside separately enabled structural repair. Implement this + as part of the migration; the current adapter behavior documented in + [configuration](../config.md) remains unchanged until then. - Decide whether missing or null input timestamps should be rejected; current decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).