Plan PromptKit adoption with structural repair enabled

This commit is contained in:
2026-09-23 18:18:45 +00:00
parent 6b33254a82
commit 837efc78a7
3 changed files with 73 additions and 37 deletions

View File

@@ -121,9 +121,14 @@ ConfigKit implementation, then PromptKit implementation, followed by a combined
v2.0.0 release. Revisit that sequence alongside other major changes. Keep
implementation units reviewable even when the release combines them.
The separate [runtime behavior decisions](runtime-behavior.md) remain unresolved.
Coordinate the retry-count question with the new configuration semantics where
appropriate; do not silently change runtime behavior merely by changing loaders.
The separate [runtime behavior decisions](runtime-behavior.md) track remaining
questions and the selected retry direction.
The coordinated plan adopts PromptKit's explicit retry semantics and enables
structural repair from the outset. Include separate transient-retry and positive
repair budgets, plus the total-operation deadline policy, in the shared settings
design. The reviewed PromptKit baseline is v0.11.0; exact defaults and settings
ownership remain detailed-planning decisions. Do not implement the runtime change
merely as a side effect of replacing the configuration loader.
Retention and transcript timestamp decisions are not automatically included.
## Open Design Decisions and Boundaries

View File

@@ -21,9 +21,12 @@ infrastructure while keeping transcript-correction responsibilities in Audita.
This roadmap owns sequencing across the two adoptions. The
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
PromptKit consumer requests are recorded upstream in
`../promptkit/docs/roadmap/audita-feature-requests.md`; they remain proposals until
accepted and released there. Do not treat proposed upstream capabilities as present.
PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
transient HTTP retries, execution observations, typed failure categories, an offline
proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
consumer requests are no longer outstanding prerequisites on those points.
Audita has not yet adopted either library; recheck the selected releases during
detailed planning.
## Migration Posture
@@ -43,9 +46,10 @@ changes, especially changes to module or pipeline configuration.
Define the target settings model for both libraries before implementing either
migration. Decide how Audita selects proposal and validation execution profiles,
supplies credentials, selects retry behavior, and assigns per-backend concurrency
limits. Give each setting one authoritative home; avoid duplicate model, endpoint,
credential, or timeout definitions with competing precedence.
supplies credentials, selects transient-retry and structural-repair budgets, and
assigns per-backend concurrency limits. Give each setting one authoritative home;
avoid duplicate model, endpoint, credential, or timeout definitions with competing
precedence.
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
In particular, optional ConfigKit configuration profiles and PromptKit execution
@@ -66,10 +70,24 @@ call and one LLM-validator call through PromptKit using controlled offline respo
Verify profile selection, credentials, embedded prompts and schemas, shared backend
capacity, failed output validation, cancellation, and diagnostic mapping.
Establish how retries and content repair differ and how their budgets and execution
observations reach Audita. Keep optional content repair disabled initially unless a
specific need justifies it. Audita's domain acceptance checks remain necessary even
when generated output satisfies a JSON schema.
Enable structural repair from the outset for proposal and LLM-validator outputs.
Additional attempts to obtain valid output are desired behavior, not a deferred
optimization. Select explicit positive repair budgets during detailed planning and
exercise invalid output that becomes valid after repair, as well as exhausted
repair budgets.
Keep transient HTTP retries and structural repair distinct. With the reviewed
library, each generation, including a repair, has its own transient-retry budget;
plan and document the combined request allowance and overall deadline. Adopt
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
zero-as-default behavior. The library's narrower transient retry policy does not
retry transport failures or malformed provider responses; do not restore the old
policy through an overlapping application retry loop.
Verify that initial, retry, and repair calls share backend limits and that their
observations reach Audita without double-counting usage. Audita's domain acceptance
checks remain necessary after structural repair succeeds; valid JSON does not imply
an acceptable correction or validator decision.
Use the prototype to resolve genuine boundary gaps and inform configuration design.
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
@@ -100,9 +118,10 @@ Remove the temporary bridge from the preceding stage.
### 5. Validate and Release the Combined Migration
Validate the assembled configuration-to-execution workflows, including local and
API backend limits, failure/cancellation behavior, credential handling, diagnostics,
and domain outcomes. Use deterministic tests and controlled endpoints rather than
paid calls as acceptance gates. Update documentation and examples together with
API backend limits, successful and exhausted repairs, transient retries,
failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
Use deterministic tests and controlled endpoints rather than paid calls as acceptance
gates. Update documentation and examples together with
implemented behavior and remove obsolete contracts and code.
Complete one migration guide covering files, CLI, environment settings, credentials,
@@ -110,31 +129,40 @@ execution profiles, inspection, and changed diagnostics. Release the combined
breaking transition under the established [release procedure](../release.md), with
no CI waiting added to release completion.
## Parallel Upstream Preparation
## Reviewed Upstream Readiness
PromptKit work can proceed alongside Audita's design and ConfigKit implementation.
Prioritize evaluation of transient-failure retries, actionable failure classification,
and execution observations, supported by a representative consumer example. Existing
per-backend capacity controls are the intended foundation; additional scheduling
features are not assumed necessary.
The v0.11.0 review covered the relevant implementation, the full test suite,
targeted race tests, and the offline proposal/assessment example; those checks
passed. Existing per-backend controls cover the requested capacity model, including
retry and repair interactions. No further upstream feature request is currently a
prerequisite to the prototype. Keep the planned adoption sequence unchanged.
Resolve the proposed LGPL 3.0 licensing change and applicable distribution terms
before selecting PromptKit for adoption. Recheck released APIs, toolchain requirements,
and dependency versions at implementation time. Pin published library releases;
production builds must not depend on sibling checkouts or local module replacements.
Remaining integration work includes selecting a total-operation deadline alongside
per-attempt timeouts, using shared backend identities with finite limits, and mapping
PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
synchronous and may overlap across runs; collection must be short and safe for
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
observations as a complete billing ledger or a stable serialized report format.
If a required upstream capability is not ready, complete independent work and
reassess sequencing explicitly rather than quietly recreate that capability as
permanent Audita infrastructure. The prototype should identify what is genuinely
required for adoption versus useful later enhancement.
The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
notices and source/rebuild distribution requirements in Audita's release design;
relicensing no longer needs to be requested upstream. Recheck released APIs,
toolchain requirements, and dependency versions at implementation time. Pin
published library releases; production builds must not depend on sibling checkouts
or local module replacements.
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
than quietly recreate shared library functionality as permanent Audita infrastructure.
## Coordination and Open Decisions
- Finalize the configuration/profile boundary, defaults, credential ownership, and
retry semantics before committing to the new file schema and CLI design.
- Define how failed validation, exhausted retries, and partial execution observations
map to Audita warnings, rejected work, and failed runs.
- Coordinate the [runtime retry-count question](runtime-behavior.md) with this design.
retry/repair budgets and deadline policy before committing to the new file schema
and CLI design. Structural repair is enabled by the agreed direction; its positive
default count remains to be selected.
- Define how exhausted repair, failed validation, exhausted retries, and partial
execution observations map to Audita warnings, rejected work, and failed runs.
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
Retention and transcript timestamp questions are not automatically included.
- Revisit ordering when other major changes are defined. Prefer designing shared
module/pipeline settings once, while keeping unrelated work independent.

View File

@@ -3,13 +3,16 @@
**Last updated:** 2026-09-23
The documentation migration identified these existing behaviors for separate
product consideration. No behavior change is approved or implemented by this list.
product consideration. Implementation status and any selected direction are noted
below; this document does not claim that runtime changes have landed.
- Decide whether the `never` retention selection should delete successful-run
diagnostics; current behavior is documented in
[operations](../operations.md#retention-behavior).
- Decide whether an explicit zero retry count should disable retries; current
adapter behavior is documented in [configuration](../config.md).
- The [library-adoption plan](library-adoption.md) selects explicit zero as disabling
transient retries, alongside separately enabled structural repair. Implement this
as part of the migration; the current adapter behavior documented in
[configuration](../config.md) remains unchanged until then.
- Decide whether missing or null input timestamps should be rejected; current
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).