audita/docs/roadmap/promptkit.md

PromptKit v0.12.0 Adoption Roadmap

Last updated: 2026-10-03

Status: Proposed. The configuration-source scope decision at the end of this document should be resolved before preparing the implementation plan.

Purpose and Baseline

Adopt PromptKit v0.12.0 now that the independent ConfigKit v0.5.0 upgrade is complete. Align Audita with the libraries' native configuration model while retaining Audita's domain rules, application-owned backend settings, and resilient processing workflow.

At reviewed Audita revision 8cba751835ba9faad340e933a3cfbb59c03924fd, go.mod selects ConfigKit v0.5.0 and PromptKit v0.11.2, with OpenRouter and Rakestrawhome /v2 catalogs at v2.20260926.0. Go 1.25.5 remains the application minimum. The implemented contracts belong to configuration, the LLM integration, and the LLM runtime.

internal/framework/llm.NewRuntime currently assembles one engine using the old prompt/profile filesystem options and programmatic backend registration. Audita embeds seven prompt definitions and separate response schemas. Operators select proposal and validation profiles from profiles_dir; the effective backend must be registered in application configuration, its endpoint must match, and its credential comes from Audita's ConfigKit-resolved settings. Successful profile reads are recorded to protect those sources from output collisions. Both roles share the same engine's backend admission controls.

The reviewed upstream release is PromptKit v0.12.0, commit 4b8169da286917fde7875425bb32b799b12c6af3. It depends on ConfigKit v0.5.0 and both maintained /v3 catalog modules at v3.20260930.0. This is an intentional incompatible pre-1.0 minor release: it replaces construction and configuration APIs, rather than simply updating provider data. PromptKit's format reference and consumer migration guide own the upstream details.

Target End State

• Audita imports released PromptKit v0.12.0 and its /v3 catalogs, retaining ConfigKit v0.5.0 and using no local dependency replacement.
• The runtime uses Config.Sources and native ConfigKit source groups. PromptKit owns composition, inheritance, presence, final-definition validation, prompt capture, and execution. Audita does not translate old profiles or implement a second YAML loader or inheritance resolver.
• Embedded prompts and maintained operator profiles use keyed prompts and profiles mappings. Existing prompt IDs, versions, role selections, instructions, and response-schema associations remain the adoption baseline. The application configuration schema, overlay selection, CLI interface, and main-file-relative profiles_dir resolution remain in place.
• Native profile extends, field-presence semantics, and source-relative resource rules apply. Documents can contain multiple definitions, and IDs come from mapping keys rather than filenames. Operator-directory scope is subject to the open decision below.
• Application backend endpoints, configured concurrency and queue capacity, and credential ownership remain authoritative for selected roles. Preserve the existing profile-selection and prepared-target checks; do not introduce a second admission controller or implicit provider credential lookup.
• A runtime uses one captured configuration snapshot for both roles. Each CLI invocation constructs a new engine; hot reload is not part of this feature. Existing inspection remains offline and safe, and consumed host sources remain protected from output collisions.
• PromptKit retains responsibility for transport retries, schema repair, observation, and prepared execution. Preserve Audita's configured retry and repair budgets, operation deadlines, cancellation, partial-success behavior, error/recovery reporting, and debug selection.

Configuration Assembly and Ownership

Replace removed WithPromptFS, WithProfileFS, and WithBackend usage with the released source API. Discover embedded YAML through FilesystemInputs and operator YAML through DirectoryInputs, with explicit caller tiers. Keep the retained schema filesystem integration and HTTP client configuration. Use the helpers' default-disabled imports; this adoption does not require new application source-selection flags or operator underlay tiers.

PromptKit supplies catalog definitions first and its own defaults next. Caller tiers then overlay in their supplied order, followed by an optional highest priority main input. Files within one directory group compose strictly; alphabetical filenames do not establish override precedence. Use separate tiers for genuine priority differences rather than manufacturing file-order semantics.

Represent application backend settings as a final programmatic source, using BackendSource or ConfigKit mappings where explicit null is needed. Preserve configured zero queue capacity as an explicit value. Suppress backend-derived credential environment lookup explicitly where catalog layering would otherwise introduce it; omission alone can retain a lower-tier value. Continue detecting competing effective profile credential lookups rather than silently clearing operator profile fields. A missing backend credential remains valid for an unauthenticated local service unless the selected profile requires a key.

Catalog backend IDs are no longer reserved upstream. Audita should not invent a new reservation rule. Catalog selection alone must not bypass application registration, endpoint, capacity, or credential requirements. Catalog profile contents may evolve; this upgrade does not promise fixed model choices or live provider availability.

The proposed caller order is embedded Audita prompts, operator definitions, then application backend settings. Whether operator definitions may include prompts and backends is an open product decision; that order is a recommendation, not authorization to add prompt overrides before the decision is resolved.

Work Required

Runtime Construction and Source Accounting

Migrate internal/framework/llm/runtime.go to layered construction and retain the role-bound adapter boundary. Review startup inspection and failure classification because upstream now validates all final definitions, including unused ones, during engine construction. Distinguish invalid operator inputs from broken embedded assets under the existing CLI exit contract; do not treat every construction failure as an operator error merely because validation moved earlier.

Upstream permits partial profiles and defers endpoint/model completeness until request overrides are applied. Preserve Audita's established checks for its selected roles without adding a requirement that every unselected profile be runnable. Native rejection of invalid unused definitions is different from requiring all profiles to be complete.

Adapt source accounting to the new discovery/capture boundary. The existing recording filesystem wrapper may be replaceable with discovered inputs or upstream source metadata. Protect the host sources actually consumed by the runtime, including inherited and differently named definitions, without guessing filenames from selected profile IDs. If operator prompt resources are approved, account for their consumed message files as well. Keep this proportionate to the existing collision contract; do not add filesystem concurrency or power-loss guarantees.

Keep the existing safe effective-profile summaries and application explanations. Use upstream metadata where useful, but do not expose the full InspectConfiguration snapshot as ordinary CLI or report output: prompt text and arbitrary provider fields are outside those summaries' current scope.

Prompt Assets, Profiles, and Presence Semantics

Move each of the seven files in internal/prompts/assets/prompts to the keyed prompt/version format. Keep the shared instruction fragments and the complete per-call output contracts owned by Audita. This migration does not authorize another prompt-quality rewrite or changes to domain validators.

Convert the six maintained profiles in examples/profiles and test-generated profile/prompt fixtures to the new format. Convert any old base_profile usage to extends, and remove embedded identities. Migrate tests that construct engines using removed options or old Profile/Backend values.

Review presence deliberately rather than mechanically preserving old serialized values. In particular, explicit zero temperature, top-p, or max-tokens now means an outbound zero; zero timeout disables PromptKit's generation deadline while Audita's outer operation deadline still applies. Omission, blank values, and null have field-specific meanings. Empty provider maps do not erase inherited entries, and changing model/backend does not erase unrelated explicit settings.

Use the native meanings without a compatibility shim. Retaining the examples' literal temperature: 0 as an explicit provider setting is recommended; explain that omitting the field requests an unspecified value, and null suppresses an inherited control. Do not automatically turn every existing zero into omission or every omission into a zero. Demonstrate representative profile inheritance without requiring every maintained example to use it.

Documentation, Migration, and Attribution

Update the profile section of docs/config.md and maintained examples when the implementation lands. Explain keyed identities, composition versus precedence, profile inheritance, presence, eager final-definition validation, selected-root errors, and captured configuration. Link to the pinned upstream reference for generic field semantics rather than reproducing its entire contract.

Include operator migration guidance for old profile documents and behaviorally significant zeros/nulls. This is a deliberate format migration, not an automatic upgrade of existing installation files. Preserve supported application overlays, including the cheaper/faster testing-profile workflow.

Update the pinned consumer link in docs/integrations/openai-compatible-llm.md, construction/source-accounting descriptions in docs/internal/llm-runtime.md, and asset-format guidance in docs/internal/prompts.md. Update other affected references only where implemented behavior changes.

Update go.mod, go.sum, and docs/oss/attribution.md for PromptKit and the two catalog modules. Verify exact selected source revisions and upstream notices under the OSS policy; retain the applicable LGPL/GPL and MIT texts rather than duplicating unchanged license documents. Dependency adoption does not change Audita's application license.

The profile format is publicly documented and incompatible with the old one. Assess Audita's release classification under the release policy; PromptKit's own pre-1.0 minor version does not establish that this is an Audita minor update. Version selection and publication belong to later release preparation.

Verification and Completion

Use existing offline tests as the starting point. Update fixtures intentionally, and add focused behavioral coverage where the source migration creates a distinct risk. Completion means:

• The released dependency set builds and the complete Audita test suite passes.
• All registered prompts load and render from keyed assets, with the established per-call schemas and adapter-supplied shared instructions.
• Selected roles use final application backend settings and direct resolved credentials; both roles still share configured capacity, including explicit zero queue capacity. Catalog layering does not introduce a competing lookup.
• Representative keyed profiles, native inheritance, and explicit presence work through the real runtime. Directory conflicts and invalid final definitions produce useful startup errors without filename-derived identity assumptions.
• Source collision protection, safe inspection, and operator-versus-asset error classification remain effective at the changed construction boundary.
• Representative prepared execution, schema repair, transient retry, and partial-success tests retain their existing behavior. Reuse adequate coverage rather than duplicating PromptKit's complete upstream test matrix.
• Maintained examples and canonical documentation describe the migrated format; attribution identifies the selected PromptKit and catalog sources accurately.

No live provider calls, new release gates, or CI-success dependency are needed to establish adoption. Guided domain recovery and the other work recorded in future.md remain separate features. Do not expand this upgrade into new report formats, pipeline scheduling, or general configuration-inspection commands. A staged implementation plan can be prepared after the decision below.

Open Decisions

Scope of the Operator Configuration Directory

Recommendation: Accept native unified PromptKit documents in the existing profiles_dir, with embedded prompts below operator definitions and application backend settings above them. Keep the name/path setting for continuity, while documenting its broader accepted content. This follows the upstream source API without a parallel parser or arbitrary top-level filtering. It allows profiles, provider defaults, and intentional prompt overrides to use the same native composition model. Audita would still ship its domain prompts, enforce its selected-backend requirements, and supply its per-call schemas. Prompt overrides would be an intentional new operator capability and need focused documentation and tests, including resource source accounting; they should not arrive silently.

Viable alternative: Keep this directory profile-only and retain embedded prompt ownership as the existing operator boundary. This narrows the migration to profile format and engine assembly, avoids adding prompt customization during a dependency upgrade, and makes the current directory name accurate. It requires an explicit contribution-scope check using the libraries' public configuration facilities, rather than another YAML parser or inheritance engine. Do not also impose one-profile-per-file, filename/ID equality, or new profile-field restrictions. The cost is an Audita-specific constraint on a now-unified source API and continuing maintenance of that boundary. If the public API cannot express the check simply, reassess the tradeoff or request narrow upstream support rather than building a general projection layer.

Either choice can preserve the existing execution workflow. The decision is whether this adoption should deliberately open operator prompt/backend contributions; changing the source API by itself does not settle that policy.