Plan PromptKit adoption with structural repair enabled

This commit is contained in:
2026-09-23 18:18:45 +00:00
parent 6b33254a82
commit 837efc78a7
3 changed files with 73 additions and 37 deletions

View File

@@ -121,9 +121,14 @@ ConfigKit implementation, then PromptKit implementation, followed by a combined
v2.0.0 release. Revisit that sequence alongside other major changes. Keep v2.0.0 release. Revisit that sequence alongside other major changes. Keep
implementation units reviewable even when the release combines them. implementation units reviewable even when the release combines them.
The separate [runtime behavior decisions](runtime-behavior.md) remain unresolved. The separate [runtime behavior decisions](runtime-behavior.md) track remaining
Coordinate the retry-count question with the new configuration semantics where questions and the selected retry direction.
appropriate; do not silently change runtime behavior merely by changing loaders. The coordinated plan adopts PromptKit's explicit retry semantics and enables
structural repair from the outset. Include separate transient-retry and positive
repair budgets, plus the total-operation deadline policy, in the shared settings
design. The reviewed PromptKit baseline is v0.11.0; exact defaults and settings
ownership remain detailed-planning decisions. Do not implement the runtime change
merely as a side effect of replacing the configuration loader.
Retention and transcript timestamp decisions are not automatically included. Retention and transcript timestamp decisions are not automatically included.
## Open Design Decisions and Boundaries ## Open Design Decisions and Boundaries

View File

@@ -21,9 +21,12 @@ infrastructure while keeping transcript-correction responsibilities in Audita.
This roadmap owns sequencing across the two adoptions. The This roadmap owns sequencing across the two adoptions. The
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope. [ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
PromptKit consumer requests are recorded upstream in PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
`../promptkit/docs/roadmap/audita-feature-requests.md`; they remain proposals until transient HTTP retries, execution observations, typed failure categories, an offline
accepted and released there. Do not treat proposed upstream capabilities as present. proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
consumer requests are no longer outstanding prerequisites on those points.
Audita has not yet adopted either library; recheck the selected releases during
detailed planning.
## Migration Posture ## Migration Posture
@@ -43,9 +46,10 @@ changes, especially changes to module or pipeline configuration.
Define the target settings model for both libraries before implementing either Define the target settings model for both libraries before implementing either
migration. Decide how Audita selects proposal and validation execution profiles, migration. Decide how Audita selects proposal and validation execution profiles,
supplies credentials, selects retry behavior, and assigns per-backend concurrency supplies credentials, selects transient-retry and structural-repair budgets, and
limits. Give each setting one authoritative home; avoid duplicate model, endpoint, assigns per-backend concurrency limits. Give each setting one authoritative home;
credential, or timeout definitions with competing precedence. avoid duplicate model, endpoint, credential, or timeout definitions with competing
precedence.
Keep ConfigKit application configuration distinct from PromptKit execution profiles. Keep ConfigKit application configuration distinct from PromptKit execution profiles.
In particular, optional ConfigKit configuration profiles and PromptKit execution In particular, optional ConfigKit configuration profiles and PromptKit execution
@@ -66,10 +70,24 @@ call and one LLM-validator call through PromptKit using controlled offline respo
Verify profile selection, credentials, embedded prompts and schemas, shared backend Verify profile selection, credentials, embedded prompts and schemas, shared backend
capacity, failed output validation, cancellation, and diagnostic mapping. capacity, failed output validation, cancellation, and diagnostic mapping.
Establish how retries and content repair differ and how their budgets and execution Enable structural repair from the outset for proposal and LLM-validator outputs.
observations reach Audita. Keep optional content repair disabled initially unless a Additional attempts to obtain valid output are desired behavior, not a deferred
specific need justifies it. Audita's domain acceptance checks remain necessary even optimization. Select explicit positive repair budgets during detailed planning and
when generated output satisfies a JSON schema. exercise invalid output that becomes valid after repair, as well as exhausted
repair budgets.
Keep transient HTTP retries and structural repair distinct. With the reviewed
library, each generation, including a repair, has its own transient-retry budget;
plan and document the combined request allowance and overall deadline. Adopt
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
zero-as-default behavior. The library's narrower transient retry policy does not
retry transport failures or malformed provider responses; do not restore the old
policy through an overlapping application retry loop.
Verify that initial, retry, and repair calls share backend limits and that their
observations reach Audita without double-counting usage. Audita's domain acceptance
checks remain necessary after structural repair succeeds; valid JSON does not imply
an acceptable correction or validator decision.
Use the prototype to resolve genuine boundary gaps and inform configuration design. Use the prototype to resolve genuine boundary gaps and inform configuration design.
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
@@ -100,9 +118,10 @@ Remove the temporary bridge from the preceding stage.
### 5. Validate and Release the Combined Migration ### 5. Validate and Release the Combined Migration
Validate the assembled configuration-to-execution workflows, including local and Validate the assembled configuration-to-execution workflows, including local and
API backend limits, failure/cancellation behavior, credential handling, diagnostics, API backend limits, successful and exhausted repairs, transient retries,
and domain outcomes. Use deterministic tests and controlled endpoints rather than failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
paid calls as acceptance gates. Update documentation and examples together with Use deterministic tests and controlled endpoints rather than paid calls as acceptance
gates. Update documentation and examples together with
implemented behavior and remove obsolete contracts and code. implemented behavior and remove obsolete contracts and code.
Complete one migration guide covering files, CLI, environment settings, credentials, Complete one migration guide covering files, CLI, environment settings, credentials,
@@ -110,31 +129,40 @@ execution profiles, inspection, and changed diagnostics. Release the combined
breaking transition under the established [release procedure](../release.md), with breaking transition under the established [release procedure](../release.md), with
no CI waiting added to release completion. no CI waiting added to release completion.
## Parallel Upstream Preparation ## Reviewed Upstream Readiness
PromptKit work can proceed alongside Audita's design and ConfigKit implementation. The v0.11.0 review covered the relevant implementation, the full test suite,
Prioritize evaluation of transient-failure retries, actionable failure classification, targeted race tests, and the offline proposal/assessment example; those checks
and execution observations, supported by a representative consumer example. Existing passed. Existing per-backend controls cover the requested capacity model, including
per-backend capacity controls are the intended foundation; additional scheduling retry and repair interactions. No further upstream feature request is currently a
features are not assumed necessary. prerequisite to the prototype. Keep the planned adoption sequence unchanged.
Resolve the proposed LGPL 3.0 licensing change and applicable distribution terms Remaining integration work includes selecting a total-operation deadline alongside
before selecting PromptKit for adoption. Recheck released APIs, toolchain requirements, per-attempt timeouts, using shared backend identities with finite limits, and mapping
and dependency versions at implementation time. Pin published library releases; PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
production builds must not depend on sibling checkouts or local module replacements. synchronous and may overlap across runs; collection must be short and safe for
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
observations as a complete billing ledger or a stable serialized report format.
If a required upstream capability is not ready, complete independent work and The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
reassess sequencing explicitly rather than quietly recreate that capability as notices and source/rebuild distribution requirements in Audita's release design;
permanent Audita infrastructure. The prototype should identify what is genuinely relicensing no longer needs to be requested upstream. Recheck released APIs,
required for adoption versus useful later enhancement. toolchain requirements, and dependency versions at implementation time. Pin
published library releases; production builds must not depend on sibling checkouts
or local module replacements.
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
than quietly recreate shared library functionality as permanent Audita infrastructure.
## Coordination and Open Decisions ## Coordination and Open Decisions
- Finalize the configuration/profile boundary, defaults, credential ownership, and - Finalize the configuration/profile boundary, defaults, credential ownership, and
retry semantics before committing to the new file schema and CLI design. retry/repair budgets and deadline policy before committing to the new file schema
- Define how failed validation, exhausted retries, and partial execution observations and CLI design. Structural repair is enabled by the agreed direction; its positive
map to Audita warnings, rejected work, and failed runs. default count remains to be selected.
- Coordinate the [runtime retry-count question](runtime-behavior.md) with this design. - Define how exhausted repair, failed validation, exhausted retries, and partial
execution observations map to Audita warnings, rejected work, and failed runs.
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
Retention and transcript timestamp questions are not automatically included. Retention and transcript timestamp questions are not automatically included.
- Revisit ordering when other major changes are defined. Prefer designing shared - Revisit ordering when other major changes are defined. Prefer designing shared
module/pipeline settings once, while keeping unrelated work independent. module/pipeline settings once, while keeping unrelated work independent.

View File

@@ -3,13 +3,16 @@
**Last updated:** 2026-09-23 **Last updated:** 2026-09-23
The documentation migration identified these existing behaviors for separate The documentation migration identified these existing behaviors for separate
product consideration. No behavior change is approved or implemented by this list. product consideration. Implementation status and any selected direction are noted
below; this document does not claim that runtime changes have landed.
- Decide whether the `never` retention selection should delete successful-run - Decide whether the `never` retention selection should delete successful-run
diagnostics; current behavior is documented in diagnostics; current behavior is documented in
[operations](../operations.md#retention-behavior). [operations](../operations.md#retention-behavior).
- Decide whether an explicit zero retry count should disable retries; current - The [library-adoption plan](library-adoption.md) selects explicit zero as disabling
adapter behavior is documented in [configuration](../config.md). transient retries, alongside separately enabled structural repair. Implement this
as part of the migration; the current adapter behavior documented in
[configuration](../config.md) remains unchanged until then.
- Decide whether missing or null input timestamps should be rejected; current - Decide whether missing or null input timestamps should be rejected; current
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md). decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).