175 lines
9.9 KiB
Markdown
175 lines
9.9 KiB
Markdown
# Coordinated ConfigKit and PromptKit Adoption
|
|
|
|
**Last updated:** 2026-09-23
|
|
|
|
**Status:** Agreed high-level direction; provisional roadmap. Detailed design,
|
|
implementation tasks, dependency versions, and acceptance criteria must be revisited
|
|
before migration begins. This document authorizes neither implementation nor release
|
|
publication and sets no schedule.
|
|
|
|
## Goal and Ownership
|
|
|
|
Adopt ConfigKit and PromptKit to align Audita with the application suite's shared
|
|
configuration and LLM execution conventions. Remove overlapping application
|
|
infrastructure while keeping transcript-correction responsibilities in Audita.
|
|
|
|
| Owner | Intended responsibilities |
|
|
| --- | --- |
|
|
| ConfigKit | Application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe configuration inspection |
|
|
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation, and shared execution capabilities supplied by the selected release |
|
|
| Audita | Configuration field meanings, CLI parsing and file selection, transcript normalization/chunking, module ordering, domain validators, correction application, reports, and diagnostic artifact lifecycle |
|
|
|
|
This roadmap owns sequencing across the two adoptions. The
|
|
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
|
|
PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
|
|
transient HTTP retries, execution observations, typed failure categories, an offline
|
|
proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
|
|
consumer requests are no longer outstanding prerequisites on those points.
|
|
Audita has not yet adopted either library; recheck the selected releases during
|
|
detailed planning.
|
|
|
|
## Migration Posture
|
|
|
|
Treat adoption as an intentional breaking transition. Preserve intended domain
|
|
capabilities, not legacy configuration files, flags, environment variables, or
|
|
configuration-inspection formats. Supply one migration guide rather than parallel
|
|
legacy loaders, compatibility aliases, or permanent translation layers.
|
|
|
|
Plan to ship both adoptions together as Audita v2.0.0, without an intermediate
|
|
public release. Implementation should still consist of coherent, reviewable units.
|
|
Revisit the final release scope and sequencing alongside other large planned
|
|
changes, especially changes to module or pipeline configuration.
|
|
|
|
## Proposed Sequence
|
|
|
|
The [shared-boundary and prototype roadmap](library-boundary.md) details the next
|
|
unit of work, covering stages 1 and 2 below before production migration.
|
|
|
|
### 1. Design the Shared Configuration Boundary
|
|
|
|
Define the target settings model for both libraries before implementing either
|
|
migration. Decide how Audita selects proposal and validation execution profiles,
|
|
supplies credentials, selects transient-retry and structural-repair budgets, and
|
|
assigns per-backend concurrency limits. Give each setting one authoritative home;
|
|
avoid duplicate model, endpoint, credential, or timeout definitions with competing
|
|
precedence.
|
|
|
|
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
|
|
In particular, optional ConfigKit configuration profiles and PromptKit execution
|
|
profiles are different concepts. Select an explicit profile-ownership model;
|
|
complete Audita-owned execution profiles are the initial recommendation, with
|
|
catalog-driven selection a deliberate alternative.
|
|
|
|
Capacity needs are simple: protect local backends from exceeding their available
|
|
simultaneous-query capacity and give paid API backends a reasonable finite cap.
|
|
Proposal and validation calls using the same backend share its limit. Separate
|
|
proposal/validation budgets and additional fine-grained Audita scheduling controls
|
|
are not required. Account for the engine-local scope of PromptKit's existing limits.
|
|
|
|
### 2. Prove the PromptKit Boundary
|
|
|
|
Before finalizing the new application configuration schema, exercise one proposal
|
|
call and one LLM-validator call through PromptKit using controlled offline responses.
|
|
Verify profile selection, credentials, embedded prompts and schemas, shared backend
|
|
capacity, failed output validation, cancellation, and diagnostic mapping.
|
|
|
|
Enable structural repair from the outset for proposal and LLM-validator outputs.
|
|
Additional attempts to obtain valid output are desired behavior, not a deferred
|
|
optimization. Select explicit positive repair budgets during detailed planning and
|
|
exercise invalid output that becomes valid after repair, as well as exhausted
|
|
repair budgets.
|
|
|
|
Keep transient HTTP retries and structural repair distinct. With the reviewed
|
|
library, each generation, including a repair, has its own transient-retry budget;
|
|
plan and document the combined request allowance and overall deadline. Adopt
|
|
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
|
|
zero-as-default behavior. The library's narrower transient retry policy does not
|
|
retry transport failures or malformed provider responses; do not restore the old
|
|
policy through an overlapping application retry loop.
|
|
|
|
Verify that initial, retry, and repair calls share backend limits and that their
|
|
observations reach Audita without double-counting usage. Audita's domain acceptance
|
|
checks remain necessary after structural repair succeeds; valid JSON does not imply
|
|
an acceptable correction or validator decision.
|
|
|
|
Use the prototype to resolve genuine boundary gaps and inform configuration design.
|
|
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
|
|
ConfigKit-facing settings around the old LLM runtime only to redesign them later.
|
|
|
|
### 3. Complete ConfigKit Adoption
|
|
|
|
Implement the agreed application configuration contract following the
|
|
[ConfigKit roadmap](configkit.md). Retire legacy environment overlays and obsolete
|
|
CLI/configuration conventions, update maintained examples and canonical references,
|
|
and begin the combined migration guide.
|
|
|
|
If the existing LLM runtime remains temporarily during this stage, keep translation
|
|
into it small and explicitly transitional. Do not preserve its old settings merely
|
|
to make the temporary bridge convenient. No compatibility mode is owed to users.
|
|
|
|
### 4. Complete PromptKit Adoption
|
|
|
|
Move prompt rendering, execution profiles, provider calls, and structural output
|
|
validation onto the selected PromptKit release. Remove overlapping transport,
|
|
prompt-loading, and scheduling machinery once the replacement is verified.
|
|
|
|
Retain Audita's transcript pipeline, semantic and deterministic validators,
|
|
correction application, reports, ledger, and artifact retention. Map PromptKit
|
|
results, errors, metadata, and execution observations into those responsibilities.
|
|
Remove the temporary bridge from the preceding stage.
|
|
|
|
### 5. Validate and Release the Combined Migration
|
|
|
|
Validate the assembled configuration-to-execution workflows, including local and
|
|
API backend limits, successful and exhausted repairs, transient retries,
|
|
failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
|
|
Use deterministic tests and controlled endpoints rather than paid calls as acceptance
|
|
gates. Update documentation and examples together with
|
|
implemented behavior and remove obsolete contracts and code.
|
|
|
|
Complete one migration guide covering files, CLI, environment settings, credentials,
|
|
execution profiles, inspection, and changed diagnostics. Release the combined
|
|
breaking transition under the established [release procedure](../release.md), with
|
|
no CI waiting added to release completion.
|
|
|
|
## Reviewed Upstream Readiness
|
|
|
|
The v0.11.0 review covered the relevant implementation, the full test suite,
|
|
targeted race tests, and the offline proposal/assessment example; those checks
|
|
passed. Existing per-backend controls cover the requested capacity model, including
|
|
retry and repair interactions. No further upstream feature request is currently a
|
|
prerequisite to the prototype. Keep the planned adoption sequence unchanged.
|
|
|
|
Remaining integration work includes selecting a total-operation deadline alongside
|
|
per-attempt timeouts, using shared backend identities with finite limits, and mapping
|
|
PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
|
|
synchronous and may overlap across runs; collection must be short and safe for
|
|
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
|
|
observations as a complete billing ledger or a stable serialized report format.
|
|
|
|
The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
|
|
notices and source/rebuild distribution requirements in Audita's release design;
|
|
relicensing no longer needs to be requested upstream. Recheck released APIs,
|
|
toolchain requirements, and dependency versions at implementation time. Pin
|
|
published library releases; production builds must not depend on sibling checkouts
|
|
or local module replacements.
|
|
|
|
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
|
|
than quietly recreate shared library functionality as permanent Audita infrastructure.
|
|
|
|
## Coordination and Open Decisions
|
|
|
|
- Finalize the configuration/profile boundary, defaults, credential ownership, and
|
|
retry/repair budgets and deadline policy before committing to the new file schema
|
|
and CLI design. Structural repair is enabled by the agreed direction; its positive
|
|
default count remains to be selected.
|
|
- Define how exhausted repair, failed validation, exhausted retries, and partial
|
|
execution observations map to Audita warnings, rejected work, and failed runs.
|
|
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
|
|
Retention and transcript timestamp questions are not automatically included.
|
|
- Revisit ordering when other major changes are defined. Prefer designing shared
|
|
module/pipeline settings once, while keeping unrelated work independent.
|
|
- Flesh out task-level scope, tests, and acceptance criteria before implementation.
|
|
Completion requires the intended ownership split, removed transitional machinery,
|
|
verified assembled behavior, and one coherent migration path for users.
|