Finalize the ConfigKit and PromptKit implementation plan
This commit is contained in:
@@ -2,179 +2,128 @@
|
||||
|
||||
**Last updated:** 2026-09-24
|
||||
|
||||
**Status:** Shared boundary and offline prototype completed; production migrations
|
||||
remain planned. The [v0.4.0 overlay evidence](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
|
||||
extends the completed prototype, and the [ordered ConfigKit handoff](configkit.md#ordered-production-migration-handoff)
|
||||
specifies the next implementation work. This document authorizes no release
|
||||
publication and sets no schedule.
|
||||
**Status:** Final target selected; production implementation pending. The shared
|
||||
boundary and offline prototype, including ConfigKit v0.4.0 overlays, are verified.
|
||||
[Implementation stages](implementation.md) are owned by the implementation plan.
|
||||
This roadmap defines the end state and user intent; it authorizes no publication.
|
||||
|
||||
## Goal and Ownership
|
||||
|
||||
Adopt ConfigKit and PromptKit to align Audita with the application suite's shared
|
||||
configuration and LLM execution conventions. Remove overlapping application
|
||||
infrastructure while keeping transcript-correction responsibilities in Audita.
|
||||
configuration and LLM execution conventions. Consistency is the primary benefit;
|
||||
removing overlapping application infrastructure is a secondary benefit. Preserve
|
||||
Audita's transcript-correction capabilities and domain protections.
|
||||
|
||||
| Owner | Intended responsibilities |
|
||||
| Owner | Responsibilities |
|
||||
| --- | --- |
|
||||
| ConfigKit | Application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe configuration inspection |
|
||||
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation, and shared execution capabilities supplied by the selected release |
|
||||
| Audita | Configuration field meanings, CLI parsing and file selection, transcript normalization/chunking, module ordering, domain validators, correction application, reports, and diagnostic artifact lifecycle |
|
||||
| ConfigKit | Strict application configuration loading, presence/default semantics, explicit overrides, recorded resolution, secret references, and safe inspection |
|
||||
| PromptKit | Prompt/profile formats and rendering, provider calls, structural output validation and repair, transient retries, engine-local backend admission, and execution observations |
|
||||
| Audita | Setting meanings, file/CLI selection, role/profile association, normalization/chunking, ordered modules, domain validation, correction application, reports, diagnostic redaction, and artifact lifecycle |
|
||||
|
||||
This roadmap owns sequencing across the two adoptions. The
|
||||
[ConfigKit roadmap](configkit.md) owns the configuration migration's proposed scope.
|
||||
PromptKit v0.11.0 is the reviewed baseline for the adoption prototype. It supplies
|
||||
transient HTTP retries, execution observations, typed failure categories, an offline
|
||||
proposal/assessment example, and LGPL-3.0-or-later licensing. The earlier upstream
|
||||
consumer requests are no longer outstanding prerequisites on those points.
|
||||
Audita has not yet adopted either library; recheck the selected releases during
|
||||
detailed planning.
|
||||
The [configuration roadmap](configkit.md) owns configuration and inspection policy.
|
||||
The [boundary findings](library-boundary-findings.md) own the selected defaults,
|
||||
full migration inventory, failure mapping, and prototype evidence. The selected
|
||||
production versions are ConfigKit v0.4.0 and PromptKit v0.11.0, with Go 1.25.5 as
|
||||
the minimum. Use published modules, never sibling checkouts or local replacements.
|
||||
The existing YAML dependency remains while glossary parsing needs it.
|
||||
|
||||
## Migration Posture
|
||||
|
||||
Treat adoption as an intentional breaking transition. Preserve intended domain
|
||||
capabilities, not legacy configuration files, flags, environment variables, or
|
||||
configuration-inspection formats. Supply one migration guide rather than parallel
|
||||
legacy loaders, compatibility aliases, or permanent translation layers.
|
||||
This is an intentional breaking transition targeting Audita v2.0.0. No backward
|
||||
compatibility is required for existing installations, config files, CLI flags,
|
||||
environment overlays, or inspection output. Provide one migration guide, without
|
||||
legacy loaders, aliases, dual modes, or automatic conversion tooling.
|
||||
|
||||
Plan to ship both adoptions together as Audita v2.0.0, without an intermediate
|
||||
public release. Implementation should still consist of coherent, reviewable units.
|
||||
Revisit the final release scope and sequencing alongside other large planned
|
||||
changes, especially changes to module or pipeline configuration.
|
||||
Both libraries belong to one refactor. Intermediate commits need not build or run;
|
||||
there is no supported intermediate product state. No temporary adapters, compatibility
|
||||
shims, or processing gates are needed. The complete application must work and pass
|
||||
its checks before the refactor is finished. Release compliance and publication
|
||||
come last, with no intermediate public release.
|
||||
|
||||
## Proposed Sequence
|
||||
## Configuration and Execution Target
|
||||
|
||||
The [shared-boundary and prototype roadmap](library-boundary.md) records the completed
|
||||
unit covering stages 1 and 2 below. The [findings](library-boundary-findings.md)
|
||||
select its target contract and identify the next ConfigKit implementation work.
|
||||
ConfigKit selected overlays enable a testing configuration that chooses cheaper or
|
||||
faster models. The example selects both proposal and validation profiles. ConfigKit
|
||||
application overlays and PromptKit execution profiles remain separate concepts.
|
||||
Complete operator-maintained profiles own model/generation controls; embedded
|
||||
application prompts and schemas are not operator-selected files. No catalog wiring,
|
||||
profile-inheritance feature, ConfigKit imports, or secret-directory bootstrap is
|
||||
part of this migration.
|
||||
|
||||
### 1. Design the Shared Configuration Boundary
|
||||
Capacity controls serve two needs: respecting a local service's hard simultaneous-
|
||||
query limit and bounding concurrent paid API activity. One finite limit per backend
|
||||
is shared across both roles and every initial, retry, and repair attempt. Limits
|
||||
are engine-local, not coordinated across processes. No additional global or
|
||||
per-role limiter or fine-grained scheduling policy is required. Profiles must use
|
||||
registered backends and ConfigKit-resolved credentials; endpoint-only bypass and
|
||||
competing profile credential lookup are excluded.
|
||||
|
||||
Define the target settings model for both libraries before implementing either
|
||||
migration. Decide how Audita selects proposal and validation execution profiles,
|
||||
supplies credentials, selects transient-retry and structural-repair budgets, and
|
||||
assigns per-backend concurrency limits. Give each setting one authoritative home;
|
||||
avoid duplicate model, endpoint, credential, or timeout definitions with competing
|
||||
precedence.
|
||||
Structural repair is enabled by default for both roles because additional attempts
|
||||
to obtain valid output are desired. Transient retries and structural repairs have
|
||||
separate budgets; explicit zero disables the respective behavior. Each generation,
|
||||
including a repair, receives its own transient-retry budget. The Audita logical-call
|
||||
deadline covers preparation, waiting, backoff, all generations, and repairs; profile
|
||||
HTTP timeouts apply per attempt. Exact defaults and bounds are in the
|
||||
[selected contract](library-boundary-findings.md#configuration-and-profile-contract).
|
||||
No Audita outer retry loop restores the old broader retry policy.
|
||||
|
||||
Keep ConfigKit application configuration distinct from PromptKit execution profiles.
|
||||
ConfigKit v0.4.0 calls its optional configuration layers overlays. Enable selected
|
||||
overlays for testing as specified in the [ConfigKit roadmap](configkit.md#selected-overlays-for-testing);
|
||||
these select application settings, including PromptKit execution profiles.
|
||||
Select an explicit profile-ownership model; complete Audita-owned execution profiles are the initial recommendation, with
|
||||
catalog-driven selection a deliberate alternative.
|
||||
Audita retains all four correction modules, five LLM-validator prompts,
|
||||
deterministic validators, confidence/applicability checks, glossary protection,
|
||||
validation batching/cardinality, deterministic proposal and ledger ordering,
|
||||
transcript encoders, and artifact retention. PromptKit-valid JSON still passes
|
||||
Audita's domain checks. Repair exhaustion, provider failure, and domain rejection
|
||||
follow the [selected failure table](library-boundary-findings.md#execution-and-failure-mapping).
|
||||
Retention and transcript timestamp changes remain separate work; this migration
|
||||
resolves the [retry-count follow-up](runtime-behavior.md).
|
||||
|
||||
Capacity needs are simple: protect local backends from exceeding their available
|
||||
simultaneous-query capacity and give paid API backends a reasonable finite cap.
|
||||
Proposal and validation calls using the same backend share its limit. Separate
|
||||
proposal/validation budgets and additional fine-grained Audita scheduling controls
|
||||
are not required. Account for the engine-local scope of PromptKit's existing limits.
|
||||
## Reporting and Diagnostic Target
|
||||
|
||||
### 2. Prove the PromptKit Boundary
|
||||
Keep stable Audita-owned serialized records; do not expose raw library structs.
|
||||
Use process report schema `v2` with the existing domain outcomes and a new
|
||||
`llm_summary` separating logical calls, dispatched provider attempts, transient
|
||||
retries, structural repairs, and usage. Transcript schema versions remain unchanged.
|
||||
The implementation plan specifies the concrete new fields and artifact wiring.
|
||||
|
||||
Before finalizing the new application configuration schema, exercise one proposal
|
||||
call and one LLM-validator call through PromptKit using controlled offline responses.
|
||||
Verify profile selection, credentials, embedded prompts and schemas, shared backend
|
||||
capacity, failed output validation, cancellation, and diagnostic mapping.
|
||||
Usage distinguishes known zero, unknown, and incomplete totals. Preserve observations
|
||||
from failed calls even when no final result exists. Count terminal totals once;
|
||||
attempt observations provide diagnosis without being added to those totals again.
|
||||
This is operational evidence, not a complete billing ledger.
|
||||
|
||||
Enable structural repair from the outset for proposal and LLM-validator outputs.
|
||||
Additional attempts to obtain valid output are desired behavior, not a deferred
|
||||
optimization. Select explicit positive repair budgets during detailed planning and
|
||||
exercise invalid output that becomes valid after repair, as well as exhausted
|
||||
repair budgets.
|
||||
Replace global/role scheduler utilization with per-backend configured capacity and
|
||||
observed queue/generation/retry/logical-call timing. Do not fabricate measurements
|
||||
that the event API does not provide. Keep domain timing and deterministic correction
|
||||
ledger ordering. Record prompt definition/rendered hashes and selected profile/model
|
||||
without promising the old hash values. Callbacks are short, concurrency-safe, and
|
||||
free of blocking I/O or waits on their own execution.
|
||||
|
||||
Keep transient HTTP retries and structural repair distinct. With the reviewed
|
||||
library, each generation, including a repair, has its own transient-retry budget;
|
||||
plan and document the combined request allowance and overall deadline. Adopt
|
||||
PromptKit's explicit zero-disables-retries semantics instead of Audita's legacy
|
||||
zero-as-default behavior. The library's narrower transient retry policy does not
|
||||
retry transport failures or malformed provider responses; do not restore the old
|
||||
policy through an overlapping application retry loop.
|
||||
Effective configuration and selected-profile snapshots remain distinct safe views.
|
||||
Secret values must not escape through inspection, logs, reports, runtime errors,
|
||||
or retained diagnostic files. Observation records exclude payloads, URLs, raw
|
||||
provider messages, prompts, and credentials. Existing artifact lifecycle and
|
||||
partial-failure handling remain Audita responsibilities.
|
||||
|
||||
Verify that initial, retry, and repair calls share backend limits and that their
|
||||
observations reach Audita without double-counting usage. Audita's domain acceptance
|
||||
checks remain necessary after structural repair succeeds; valid JSON does not imply
|
||||
an acceptable correction or validator decision.
|
||||
## Completion Criteria and Release Boundary
|
||||
|
||||
Use the prototype to resolve genuine boundary gaps and inform configuration design.
|
||||
It is evidence for the migration, not a permanent alternate runtime. Avoid finalizing
|
||||
ConfigKit-facing settings around the old LLM runtime only to redesign them later.
|
||||
- Both libraries own their selected responsibilities in the production application.
|
||||
Superseded loaders, transport, renderers, schedulers, aliases, and the isolated
|
||||
prototype are gone once its useful evidence is represented in production tests.
|
||||
- Offline tests prove configuration-to-execution routing for both roles, selected
|
||||
overlays and explicit overrides, shared backend capacity, retry/repair outcomes,
|
||||
cancellation/deadlines, domain decisions, redaction, and truthful partial reports.
|
||||
Tests exercise real internal collaborators and controlled HTTP boundaries.
|
||||
- Maintained examples and canonical docs describe the implemented final model.
|
||||
One migration guide covers files, flags, environment inputs, profiles, credentials,
|
||||
overlays, inspection, and changed runtime/report/diagnostic semantics.
|
||||
- The assembled application passes its required checks; Linux amd64 and arm64 build.
|
||||
macOS arm64 remains best effort. No live-provider or CI-success gate is added.
|
||||
|
||||
### 3. Complete ConfigKit Adoption
|
||||
PromptKit v0.11.0's reviewed retry, observation, and capacity APIs remove the earlier
|
||||
upstream feature prerequisites; no upstream edit is required for adoption. The
|
||||
prototype proved representative behavior, not release compliance or model quality.
|
||||
|
||||
Implement the agreed application configuration contract following the
|
||||
[ConfigKit roadmap](configkit.md). Retire legacy environment overlays and obsolete
|
||||
CLI/configuration conventions, update maintained examples and canonical references,
|
||||
and begin the combined migration guide. Include the selected testing-overlay path
|
||||
and its offline integration evidence from the ConfigKit roadmap.
|
||||
|
||||
ConfigKit and PromptKit adoption form one refactor. These sections describe
|
||||
implementation order, not separately runnable milestones. Intermediate commits
|
||||
need not build or run. Build directly toward the final integration without
|
||||
temporary adapters, compatibility shims, or an interim processing gate; validate
|
||||
the complete application before declaring the refactor finished.
|
||||
|
||||
### 4. Complete PromptKit Adoption
|
||||
|
||||
Move prompt rendering, execution profiles, provider calls, and structural output
|
||||
validation onto the selected PromptKit release. Remove overlapping transport,
|
||||
prompt-loading, and scheduling machinery once the replacement is verified.
|
||||
|
||||
Retain Audita's transcript pipeline, semantic and deterministic validators,
|
||||
correction application, reports, ledger, and artifact retention. Map PromptKit
|
||||
results, errors, metadata, and execution observations into those responsibilities.
|
||||
Connect the new configuration directly to this execution path and remove the
|
||||
superseded configuration and runtime machinery.
|
||||
|
||||
### 5. Validate and Release the Combined Migration
|
||||
|
||||
Validate the assembled configuration-to-execution workflows, including local and
|
||||
API backend limits, successful and exhausted repairs, transient retries,
|
||||
failure/cancellation behavior, credential handling, diagnostics, and domain outcomes.
|
||||
Use deterministic tests and controlled endpoints rather than paid calls as acceptance
|
||||
gates. Update documentation and examples together with
|
||||
implemented behavior and remove obsolete contracts and code.
|
||||
|
||||
Complete one migration guide covering files, CLI, environment settings, credentials,
|
||||
execution profiles, inspection, and changed diagnostics. Release the combined
|
||||
breaking transition under the established [release procedure](../release.md), with
|
||||
no CI waiting added to release completion.
|
||||
|
||||
## Reviewed Upstream Readiness
|
||||
|
||||
The v0.11.0 review covered the relevant implementation, the full test suite,
|
||||
targeted race tests, and the offline proposal/assessment example; those checks
|
||||
passed. Existing per-backend controls cover the requested capacity model, including
|
||||
retry and repair interactions. No further upstream feature request is currently a
|
||||
prerequisite to the prototype. Keep the planned adoption sequence unchanged.
|
||||
|
||||
Remaining integration work includes selecting a total-operation deadline alongside
|
||||
per-attempt timeouts, using shared backend identities with finite limits, and mapping
|
||||
PromptKit observations to Audita-owned diagnostic records. Observation callbacks are
|
||||
synchronous and may overlap across runs; collection must be short and safe for
|
||||
concurrent use. Distinguish unknown or incomplete usage from known zero and do not treat
|
||||
observations as a complete billing ledger or a stable serialized report format.
|
||||
|
||||
The licensing change has shipped as LGPL-3.0-or-later. Account for applicable
|
||||
notices and source/rebuild distribution requirements in Audita's release design;
|
||||
relicensing no longer needs to be requested upstream. Recheck released APIs,
|
||||
toolchain requirements, and dependency versions at implementation time. Pin
|
||||
published library releases; production builds must not depend on sibling checkouts
|
||||
or local module replacements.
|
||||
|
||||
If the prototype reveals a genuine remaining gap, evaluate it explicitly rather
|
||||
than quietly recreate shared library functionality as permanent Audita infrastructure.
|
||||
|
||||
## Coordination and Open Decisions
|
||||
|
||||
- Use the selected configuration/profile boundary, defaults, credential flow, and
|
||||
retry/repair/deadline contract in the [findings](library-boundary-findings.md)
|
||||
when detailing the ConfigKit production migration.
|
||||
- Implement the outcome and observation mapping recorded in the findings, retaining
|
||||
Audita domain validation and diagnostic redaction.
|
||||
- Resolve the [runtime retry-count follow-up](runtime-behavior.md) through this migration.
|
||||
Retention and transcript timestamp questions are not automatically included.
|
||||
- Revisit ordering when other major changes are defined. Prefer designing shared
|
||||
module/pipeline settings once, while keeping unrelated work independent.
|
||||
- Flesh out task-level scope, tests, and acceptance criteria before implementation.
|
||||
Completion requires the intended ownership split, removed transitional machinery,
|
||||
verified assembled behavior, and one coherent migration path for users.
|
||||
The selected PromptKit release uses LGPL-3.0-or-later. A later release unit must
|
||||
review the actual production dependency graph and distribution requirements,
|
||||
including notices and source/rebuild materials, before binary publication. This
|
||||
implementation plan does not claim that review is complete. Release publication
|
||||
uses the existing [procedure](../release.md) and completes on successful source/tag
|
||||
push, without waiting for CI or artifact delivery.
|
||||
|
||||
Reference in New Issue
Block a user