Plan PromptKit adoption with structural repair enabled
This commit is contained in:
@@ -121,9 +121,14 @@ ConfigKit implementation, then PromptKit implementation, followed by a combined
|
|||||||
v2.0.0 release. Revisit that sequence alongside other major changes. Keep
|
v2.0.0 release. Revisit that sequence alongside other major changes. Keep
|
||||||
implementation units reviewable even when the release combines them.
|
implementation units reviewable even when the release combines them.
|
||||||
|
|
||||||
The separate [runtime behavior decisions](runtime-behavior.md) remain unresolved.
|
The separate [runtime behavior decisions](runtime-behavior.md) track remaining
|
||||||
Coordinate the retry-count question with the new configuration semantics where
|
questions and the selected retry direction.
|
||||||
appropriate; do not silently change runtime behavior merely by changing loaders.
|
The coordinated plan adopts PromptKit's explicit retry semantics and enables
|
||||||
|
structural repair from the outset. Include separate transient-retry and positive
|
||||||
|
repair budgets, plus the total-operation deadline policy, in the shared settings
|
||||||
|
design. The reviewed PromptKit baseline is v0.11.0; exact defaults and settings
|
||||||
|
ownership remain detailed-planning decisions. Do not implement the runtime change
|
||||||
|
merely as a side effect of replacing the configuration loader.
|
||||||
Retention and transcript timestamp decisions are not automatically included.
|
Retention and transcript timestamp decisions are not automatically included.
|
||||||
|
|
||||||
## Open Design Decisions and Boundaries
|
## Open Design Decisions and Boundaries
|
||||||
|
|||||||
@@ -21,9 +21,12 @@ infrastructure while keeping transcript-correction responsibilities in Audita.
|
|||||||
|
|
||||||
This roadmap owns sequencing across the two adoptions. The
|
This roadmap owns sequencing across the two adoptions. The
|
||||||
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
|
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
|
||||||
PromptKit consumer requests are recorded upstream in
|
PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
|
||||||
`../promptkit/docs/roadmap/audita-feature-requests.md`; they remain proposals until
|
transient HTTP retries, execution observations, typed failure categories, an offline
|
||||||
accepted and released there. Do not treat proposed upstream capabilities as present.
|
proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
|
||||||
|
consumer requests are no longer outstanding prerequisites on those points.
|
||||||
|
Audita has not yet adopted either library; recheck the selected releases during
|
||||||
|
detailed planning.
|
||||||
|
|
||||||
## Migration Posture
|
## Migration Posture
|
||||||
|
|
||||||
@@ -43,9 +46,10 @@ changes, especially changes to module or pipeline configuration.
|
|||||||
|
|
||||||
Define the target settings model for both libraries before implementing either
|
Define the target settings model for both libraries before implementing either
|
||||||
migration. Decide how Audita selects proposal and validation execution profiles,
|
migration. Decide how Audita selects proposal and validation execution profiles,
|
||||||
supplies credentials, selects retry behavior, and assigns per-backend concurrency
|
supplies credentials, selects transient-retry and structural-repair budgets, and
|
||||||
limits. Give each setting one authoritative home; avoid duplicate model, endpoint,
|
assigns per-backend concurrency limits. Give each setting one authoritative home;
|
||||||
credential, or timeout definitions with competing precedence.
|
avoid duplicate model, endpoint, credential, or timeout definitions with competing
|
||||||
|
precedence.
|
||||||
|
|
||||||
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
|
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
|
||||||
In particular, optional ConfigKit configuration profiles and PromptKit execution
|
In particular, optional ConfigKit configuration profiles and PromptKit execution
|
||||||
@@ -66,10 +70,24 @@ call and one LLM-validator call through PromptKit using controlled offline respo
|
|||||||
Verify profile selection, credentials, embedded prompts and schemas, shared backend
|
Verify profile selection, credentials, embedded prompts and schemas, shared backend
|
||||||
capacity, failed output validation, cancellation, and diagnostic mapping.
|
capacity, failed output validation, cancellation, and diagnostic mapping.
|
||||||
|
|
||||||
Establish how retries and content repair differ and how their budgets and execution
|
Enable structural repair from the outset for proposal and LLM-validator outputs.
|
||||||
observations reach Audita. Keep optional content repair disabled initially unless a
|
Additional attempts to obtain valid output are desired behavior, not a deferred
|
||||||
specific need justifies it. Audita's domain acceptance checks remain necessary even
|
optimization. Select explicit positive repair budgets during detailed planning and
|
||||||
when generated output satisfies a JSON schema.
|
exercise invalid output that becomes valid after repair, as well as exhausted
|
||||||
|
repair budgets.
|
||||||
|
|
||||||
|
Keep transient HTTP retries and structural repair distinct. With the reviewed
|
||||||
|
library, each generation, including a repair, has its own transient-retry budget;
|
||||||
|
plan and document the combined request allowance and overall deadline. Adopt
|
||||||
|
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
|
||||||
|
zero-as-default behavior. The library's narrower transient retry policy does not
|
||||||
|
retry transport failures or malformed provider responses; do not restore the old
|
||||||
|
policy through an overlapping application retry loop.
|
||||||
|
|
||||||
|
Verify that initial, retry, and repair calls share backend limits and that their
|
||||||
|
observations reach Audita without double-counting usage. Audita's domain acceptance
|
||||||
|
checks remain necessary after structural repair succeeds; valid JSON does not imply
|
||||||
|
an acceptable correction or validator decision.
|
||||||
|
|
||||||
Use the prototype to resolve genuine boundary gaps and inform configuration design.
|
Use the prototype to resolve genuine boundary gaps and inform configuration design.
|
||||||
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
|
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
|
||||||
@@ -100,9 +118,10 @@ Remove the temporary bridge from the preceding stage.
|
|||||||
### 5. Validate and Release the Combined Migration
|
### 5. Validate and Release the Combined Migration
|
||||||
|
|
||||||
Validate the assembled configuration-to-execution workflows, including local and
|
Validate the assembled configuration-to-execution workflows, including local and
|
||||||
API backend limits, failure/cancellation behavior, credential handling, diagnostics,
|
API backend limits, successful and exhausted repairs, transient retries,
|
||||||
and domain outcomes. Use deterministic tests and controlled endpoints rather than
|
failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
|
||||||
paid calls as acceptance gates. Update documentation and examples together with
|
Use deterministic tests and controlled endpoints rather than paid calls as acceptance
|
||||||
|
gates. Update documentation and examples together with
|
||||||
implemented behavior and remove obsolete contracts and code.
|
implemented behavior and remove obsolete contracts and code.
|
||||||
|
|
||||||
Complete one migration guide covering files, CLI, environment settings, credentials,
|
Complete one migration guide covering files, CLI, environment settings, credentials,
|
||||||
@@ -110,31 +129,40 @@ execution profiles, inspection, and changed diagnostics. Release the combined
|
|||||||
breaking transition under the established [release procedure](../release.md), with
|
breaking transition under the established [release procedure](../release.md), with
|
||||||
no CI waiting added to release completion.
|
no CI waiting added to release completion.
|
||||||
|
|
||||||
## Parallel Upstream Preparation
|
## Reviewed Upstream Readiness
|
||||||
|
|
||||||
PromptKit work can proceed alongside Audita's design and ConfigKit implementation.
|
The v0.11.0 review covered the relevant implementation, the full test suite,
|
||||||
Prioritize evaluation of transient-failure retries, actionable failure classification,
|
targeted race tests, and the offline proposal/assessment example; those checks
|
||||||
and execution observations, supported by a representative consumer example. Existing
|
passed. Existing per-backend controls cover the requested capacity model, including
|
||||||
per-backend capacity controls are the intended foundation; additional scheduling
|
retry and repair interactions. No further upstream feature request is currently a
|
||||||
features are not assumed necessary.
|
prerequisite to the prototype. Keep the planned adoption sequence unchanged.
|
||||||
|
|
||||||
Resolve the proposed LGPL 3.0 licensing change and applicable distribution terms
|
Remaining integration work includes selecting a total-operation deadline alongside
|
||||||
before selecting PromptKit for adoption. Recheck released APIs, toolchain requirements,
|
per-attempt timeouts, using shared backend identities with finite limits, and mapping
|
||||||
and dependency versions at implementation time. Pin published library releases;
|
PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
|
||||||
production builds must not depend on sibling checkouts or local module replacements.
|
synchronous and may overlap across runs; collection must be short and safe for
|
||||||
|
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
|
||||||
|
observations as a complete billing ledger or a stable serialized report format.
|
||||||
|
|
||||||
If a required upstream capability is not ready, complete independent work and
|
The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
|
||||||
reassess sequencing explicitly rather than quietly recreate that capability as
|
notices and source/rebuild distribution requirements in Audita's release design;
|
||||||
permanent Audita infrastructure. The prototype should identify what is genuinely
|
relicensing no longer needs to be requested upstream. Recheck released APIs,
|
||||||
required for adoption versus useful later enhancement.
|
toolchain requirements, and dependency versions at implementation time. Pin
|
||||||
|
published library releases; production builds must not depend on sibling checkouts
|
||||||
|
or local module replacements.
|
||||||
|
|
||||||
|
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
|
||||||
|
than quietly recreate shared library functionality as permanent Audita infrastructure.
|
||||||
|
|
||||||
## Coordination and Open Decisions
|
## Coordination and Open Decisions
|
||||||
|
|
||||||
- Finalize the configuration/profile boundary, defaults, credential ownership, and
|
- Finalize the configuration/profile boundary, defaults, credential ownership, and
|
||||||
retry semantics before committing to the new file schema and CLI design.
|
retry/repair budgets and deadline policy before committing to the new file schema
|
||||||
- Define how failed validation, exhausted retries, and partial execution observations
|
and CLI design. Structural repair is enabled by the agreed direction; its positive
|
||||||
map to Audita warnings, rejected work, and failed runs.
|
default count remains to be selected.
|
||||||
- Coordinate the [runtime retry-count question](runtime-behavior.md) with this design.
|
- Define how exhausted repair, failed validation, exhausted retries, and partial
|
||||||
|
execution observations map to Audita warnings, rejected work, and failed runs.
|
||||||
|
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
|
||||||
Retention and transcript timestamp questions are not automatically included.
|
Retention and transcript timestamp questions are not automatically included.
|
||||||
- Revisit ordering when other major changes are defined. Prefer designing shared
|
- Revisit ordering when other major changes are defined. Prefer designing shared
|
||||||
module/pipeline settings once, while keeping unrelated work independent.
|
module/pipeline settings once, while keeping unrelated work independent.
|
||||||
|
|||||||
@@ -3,13 +3,16 @@
|
|||||||
**Last updated:** 2026-09-23
|
**Last updated:** 2026-09-23
|
||||||
|
|
||||||
The documentation migration identified these existing behaviors for separate
|
The documentation migration identified these existing behaviors for separate
|
||||||
product consideration. No behavior change is approved or implemented by this list.
|
product consideration. Implementation status and any selected direction are noted
|
||||||
|
below; this document does not claim that runtime changes have landed.
|
||||||
|
|
||||||
- Decide whether the `never` retention selection should delete successful-run
|
- Decide whether the `never` retention selection should delete successful-run
|
||||||
diagnostics; current behavior is documented in
|
diagnostics; current behavior is documented in
|
||||||
[operations](../operations.md#retention-behavior).
|
[operations](../operations.md#retention-behavior).
|
||||||
- Decide whether an explicit zero retry count should disable retries; current
|
- The [library-adoption plan](library-adoption.md) selects explicit zero as disabling
|
||||||
adapter behavior is documented in [configuration](../config.md).
|
transient retries, alongside separately enabled structural repair. Implement this
|
||||||
|
as part of the migration; the current adapter behavior documented in
|
||||||
|
[configuration](../config.md) remains unchanged until then.
|
||||||
- Decide whether missing or null input timestamps should be rejected; current
|
- Decide whether missing or null input timestamps should be rejected; current
|
||||||
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).
|
decoding is documented in the [input contract](../integrations/transcript-glossary-files.md).
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user