Plan PromptKit adoption with structural repair enabled
This commit is contained in:
@@ -121,9 +121,14 @@ ConfigKit implementation, then PromptKit implementation, followed by a combined
|
||||
v2.0.0 release. Revisit that sequence alongside other major changes. Keep
|
||||
implementation units reviewable even when the release combines them.
|
||||
|
||||
The separate [runtime behavior decisions](runtime-behavior.md) remain unresolved.
|
||||
Coordinate the retry-count question with the new configuration semantics where
|
||||
appropriate; do not silently change runtime behavior merely by changing loaders.
|
||||
The separate [runtime behavior decisions](runtime-behavior.md) track remaining
|
||||
questions and the selected retry direction.
|
||||
The coordinated plan adopts PromptKit's explicit retry semantics and enables
|
||||
structural repair from the outset. Include separate transient-retry and positive
|
||||
repair budgets, plus the total-operation deadline policy, in the shared settings
|
||||
design. The reviewed PromptKit baseline is v0.11.0; exact defaults and settings
|
||||
ownership remain detailed-planning decisions. Do not implement the runtime change
|
||||
merely as a side effect of replacing the configuration loader.
|
||||
Retention and transcript timestamp decisions are not automatically included.
|
||||
|
||||
## Open Design Decisions and Boundaries
|
||||
|
||||
@@ -21,9 +21,12 @@ infrastructure while keeping transcript-correction responsibilities in Audita.
|
||||
|
||||
This roadmap owns sequencing across the two adoptions. The
|
||||
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
|
||||
PromptKit consumer requests are recorded upstream in
|
||||
`../promptkit/docs/roadmap/audita-feature-requests.md`; they remain proposals until
|
||||
accepted and released there. Do not treat proposed upstream capabilities as present.
|
||||
PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
|
||||
transient HTTP retries, execution observations, typed failure categories, an offline
|
||||
proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
|
||||
consumer requests are no longer outstanding prerequisites on those points.
|
||||
Audita has not yet adopted either library; recheck the selected releases during
|
||||
detailed planning.
|
||||
|
||||
## Migration Posture
|
||||
|
||||
@@ -43,9 +46,10 @@ changes, especially changes to module or pipeline configuration.
|
||||
|
||||
Define the target settings model for both libraries before implementing either
|
||||
migration. Decide how Audita selects proposal and validation execution profiles,
|
||||
supplies credentials, selects retry behavior, and assigns per-backend concurrency
|
||||
limits. Give each setting one authoritative home; avoid duplicate model, endpoint,
|
||||
credential, or timeout definitions with competing precedence.
|
||||
supplies credentials, selects transient-retry and structural-repair budgets, and
|
||||
assigns per-backend concurrency limits. Give each setting one authoritative home;
|
||||
avoid duplicate model, endpoint, credential, or timeout definitions with competing
|
||||
precedence.
|
||||
|
||||
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
|
||||
In particular, optional ConfigKit configuration profiles and PromptKit execution
|
||||
@@ -66,10 +70,24 @@ call and one LLM-validator call through PromptKit using controlled offline respo
|
||||
Verify profile selection, credentials, embedded prompts and schemas, shared backend
|
||||
capacity, failed output validation, cancellation, and diagnostic mapping.
|
||||
|
||||
Establish how retries and content repair differ and how their budgets and execution
|
||||
observations reach Audita. Keep optional content repair disabled initially unless a
|
||||
specific need justifies it. Audita's domain acceptance checks remain necessary even
|
||||
when generated output satisfies a JSON schema.
|
||||
Enable structural repair from the outset for proposal and LLM-validator outputs.
|
||||
Additional attempts to obtain valid output are desired behavior, not a deferred
|
||||
optimization. Select explicit positive repair budgets during detailed planning and
|
||||
exercise invalid output that becomes valid after repair, as well as exhausted
|
||||
repair budgets.
|
||||
|
||||
Keep transient HTTP retries and structural repair distinct. With the reviewed
|
||||
library, each generation, including a repair, has its own transient-retry budget;
|
||||
plan and document the combined request allowance and overall deadline. Adopt
|
||||
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
|
||||
zero-as-default behavior. The library's narrower transient retry policy does not
|
||||
retry transport failures or malformed provider responses; do not restore the old
|
||||
policy through an overlapping application retry loop.
|
||||
|
||||
Verify that initial, retry, and repair calls share backend limits and that their
|
||||
observations reach Audita without double-counting usage. Audita's domain acceptance
|
||||
checks remain necessary after structural repair succeeds; valid JSON does not imply
|
||||
an acceptable correction or validator decision.
|
||||
|
||||
Use the prototype to resolve genuine boundary gaps and inform configuration design.
|
||||
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
|
||||
@@ -100,9 +118,10 @@ Remove the temporary bridge from the preceding stage.
|
||||
### 5. Validate and Release the Combined Migration
|
||||
|
||||
Validate the assembled configuration-to-execution workflows, including local and
|
||||
API backend limits, failure/cancellation behavior, credential handling, diagnostics,
|
||||
and domain outcomes. Use deterministic tests and controlled endpoints rather than
|
||||
paid calls as acceptance gates. Update documentation and examples together with
|
||||
API backend limits, successful and exhausted repairs, transient retries,
|
||||
failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
|
||||
Use deterministic tests and controlled endpoints rather than paid calls as acceptance
|
||||
gates. Update documentation and examples together with
|
||||
implemented behavior and remove obsolete contracts and code.
|
||||
|
||||
Complete one migration guide covering files, CLI, environment settings, credentials,
|
||||
@@ -110,31 +129,40 @@ execution profiles, inspection, and changed diagnostics. Release the combined
|
||||
breaking transition under the established [release procedure](../release.md), with
|
||||
no CI waiting added to release completion.
|
||||
|
||||
## Parallel Upstream Preparation
|
||||
## Reviewed Upstream Readiness
|
||||
|
||||
PromptKit work can proceed alongside Audita's design and ConfigKit implementation.
|
||||
Prioritize evaluation of transient-failure retries, actionable failure classification,
|
||||
and execution observations, supported by a representative consumer example. Existing
|
||||
per-backend capacity controls are the intended foundation; additional scheduling
|
||||
features are not assumed necessary.
|
||||
The v0.11.0 review covered the relevant implementation, the full test suite,
|
||||
targeted race tests, and the offline proposal/assessment example; those checks
|
||||
passed. Existing per-backend controls cover the requested capacity model, including
|
||||
retry and repair interactions. No further upstream feature request is currently a
|
||||
prerequisite to the prototype. Keep the planned adoption sequence unchanged.
|
||||
|
||||
Resolve the proposed LGPL 3.0 licensing change and applicable distribution terms
|
||||
before selecting PromptKit for adoption. Recheck released APIs, toolchain requirements,
|
||||
and dependency versions at implementation time. Pin published library releases;
|
||||
production builds must not depend on sibling checkouts or local module replacements.
|
||||
Remaining integration work includes selecting a total-operation deadline alongside
|
||||
per-attempt timeouts, using shared backend identities with finite limits, and mapping
|
||||
PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
|
||||
synchronous and may overlap across runs; collection must be short and safe for
|
||||
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
|
||||
observations as a complete billing ledger or a stable serialized report format.
|
||||
|
||||
If a required upstream capability is not ready, complete independent work and
|
||||
reassess sequencing explicitly rather than quietly recreate that capability as
|
||||
permanent Audita infrastructure. The prototype should identify what is genuinely
|
||||
required for adoption versus useful later enhancement.
|
||||
The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
|
||||
notices and source/rebuild distribution requirements in Audita's release design;
|
||||
relicensing no longer needs to be requested upstream. Recheck released APIs,
|
||||
toolchain requirements, and dependency versions at implementation time. Pin
|
||||
published library releases; production builds must not depend on sibling checkouts
|
||||
or local module replacements.
|
||||
|
||||
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
|
||||
than quietly recreate shared library functionality as permanent Audita infrastructure.
|
||||
|
||||
## Coordination and Open Decisions
|
||||
|
||||
- Finalize the configuration/profile boundary, defaults, credential ownership, and
|
||||
retry semantics before committing to the new file schema and CLI design.
|
||||
- Define how failed validation, exhausted retries, and partial execution observations
|
||||
map to Audita warnings, rejected work, and failed runs.
|
||||
- Coordinate the [runtime retry-count question](runtime-behavior.md) with this design.
|
||||
retry/repair budgets and deadline policy before committing to the new file schema
|
||||
and CLI design. Structural repair is enabled by the agreed direction; its positive
|
||||
default count remains to be selected.
|
||||
- Define how exhausted repair, failed validation, exhausted retries, and partial
|
||||
execution observations map to Audita warnings, rejected work, and failed runs.
|
||||
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
|
||||
Retention and transcript timestamp questions are not automatically included.
|
||||
- Revisit ordering when other major changes are defined. Prefer designing shared
|
||||
module/pipeline settings once, while keeping unrelated work independent.
|
||||
|
||||
@@ -3,13 +3,16 @@
|
||||
**Last updated:** 2026-09-23
|
||||
|
||||
The documentation migration identified these existing behaviors for separate
|
||||
product consideration. No behavior change is approved or implemented by this list.
|
||||
product consideration. Implementation status and any selected direction are noted
|
||||
below; this document does not claim that runtime changes have landed.
|
||||
|
||||
- Decide whether the `never` retention selection should delete successful-run
|
||||
diagnostics; current behavior is documented in
|
||||
[operations](../operations.md#retention-behavior).
|
||||
- Decide whether an explicit zero retry count should disable retries; current
|
||||
adapter behavior is documented in [configuration](../config.md).
|
||||
- The [library-adoption plan](library-adoption.md) selects explicit zero as disabling
|
||||
transient retries, alongside separately enabled structural repair. Implement this
|
||||
as part of the migration; the current adapter behavior documented in
|
||||
[configuration](../config.md) remains unchanged until then.
|
||||
- Decide whether missing or null input timestamps should be rejected; current
|
||||
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user