From 79d38606fe61fe6268186c78796cbce486135ff8 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 3 Oct 2026 17:45:40 +0000 Subject: [PATCH] Document the PromptKit upgrade roadmap --- docs/roadmap/promptkit.md | 254 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 docs/roadmap/promptkit.md diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md new file mode 100644 index 0000000..0a48b23 --- /dev/null +++ b/docs/roadmap/promptkit.md @@ -0,0 +1,254 @@ +# PromptKit v0.12.0 Adoption Roadmap + +**Last updated:** 2026-10-03 + +**Status:** Proposed. The configuration-source scope decision at the end of +this document should be resolved before preparing the implementation plan. + +## Purpose and Baseline + +Adopt PromptKit v0.12.0 now that the independent +[ConfigKit v0.5.0 upgrade](configkit.md) is complete. Align Audita with the +libraries' native configuration model while retaining Audita's domain rules, +application-owned backend settings, and resilient processing workflow. + +At reviewed Audita revision `8cba751835ba9faad340e933a3cfbb59c03924fd`, +[go.mod](../../go.mod) selects ConfigKit v0.5.0 and PromptKit v0.11.2, with +OpenRouter and Rakestrawhome `/v2` catalogs at `v2.20260926.0`. Go 1.25.5 +remains the application minimum. The implemented contracts belong to +[configuration](../config.md), the +[LLM integration](../integrations/openai-compatible-llm.md), and the +[LLM runtime](../internal/llm-runtime.md). + +`internal/framework/llm.NewRuntime` currently assembles one engine using the +old prompt/profile filesystem options and programmatic backend registration. +Audita embeds seven prompt definitions and separate response schemas. Operators +select proposal and validation profiles from `profiles_dir`; the effective +backend must be registered in application configuration, its endpoint must +match, and its credential comes from Audita's ConfigKit-resolved settings. +Successful profile reads are recorded to protect those sources from output +collisions. Both roles share the same engine's backend admission controls. + +The reviewed upstream release is +[PromptKit v0.12.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/releases/v0.12.0.md), +commit `4b8169da286917fde7875425bb32b799b12c6af3`. It depends on ConfigKit +v0.5.0 and both maintained `/v3` catalog modules at `v3.20260930.0`. This is an +intentional incompatible pre-1.0 minor release: it replaces construction and +configuration APIs, rather than simply updating provider data. PromptKit's +[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md) +and [consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api) +own the upstream details. + +## Target End State + +- Audita imports released PromptKit v0.12.0 and its `/v3` catalogs, retaining + ConfigKit v0.5.0 and using no local dependency replacement. +- The runtime uses `Config.Sources` and native ConfigKit source groups. + PromptKit owns composition, inheritance, presence, final-definition + validation, prompt capture, and execution. Audita does not translate old + profiles or implement a second YAML loader or inheritance resolver. +- Embedded prompts and maintained operator profiles use keyed `prompts` and + `profiles` mappings. Existing prompt IDs, versions, role selections, + instructions, and response-schema associations remain the adoption baseline. + The application configuration schema, overlay selection, CLI interface, and + main-file-relative `profiles_dir` resolution remain in place. +- Native profile `extends`, field-presence semantics, and source-relative + resource rules apply. Documents can contain multiple definitions, and IDs + come from mapping keys rather than filenames. Operator-directory scope is + subject to the open decision below. +- Application backend endpoints, configured concurrency and queue capacity, + and credential ownership remain authoritative for selected roles. Preserve + the existing profile-selection and prepared-target checks; do not introduce + a second admission controller or implicit provider credential lookup. +- A runtime uses one captured configuration snapshot for both roles. Each CLI + invocation constructs a new engine; hot reload is not part of this feature. + Existing inspection remains offline and safe, and consumed host sources + remain protected from output collisions. +- PromptKit retains responsibility for transport retries, schema repair, + observation, and prepared execution. Preserve Audita's configured retry and + repair budgets, operation deadlines, cancellation, partial-success behavior, + error/recovery reporting, and debug selection. + +## Configuration Assembly and Ownership + +Replace removed `WithPromptFS`, `WithProfileFS`, and `WithBackend` usage with +the released source API. Discover embedded YAML through `FilesystemInputs` and +operator YAML through `DirectoryInputs`, with explicit caller tiers. Keep the +retained schema filesystem integration and HTTP client configuration. +Use the helpers' default-disabled imports; this adoption does not require new +application source-selection flags or operator underlay tiers. + +PromptKit supplies catalog definitions first and its own defaults next. Caller +tiers then overlay in their supplied order, followed by an optional highest +priority main input. Files within one directory group compose strictly; +alphabetical filenames do not establish override precedence. Use separate +tiers for genuine priority differences rather than manufacturing file-order +semantics. + +Represent application backend settings as a final programmatic source, using +`BackendSource` or ConfigKit mappings where explicit null is needed. Preserve +configured zero queue capacity as an explicit value. Suppress backend-derived +credential environment lookup explicitly where catalog layering would otherwise +introduce it; omission alone can retain a lower-tier value. Continue detecting +competing effective profile credential lookups rather than silently clearing +operator profile fields. A missing backend credential remains valid for an +unauthenticated local service unless the selected profile requires a key. + +Catalog backend IDs are no longer reserved upstream. Audita should not invent a +new reservation rule. Catalog selection alone must not bypass application +registration, endpoint, capacity, or credential requirements. Catalog profile +contents may evolve; this upgrade does not promise fixed model choices or live +provider availability. + +The proposed caller order is embedded Audita prompts, operator definitions, +then application backend settings. Whether operator definitions may include +prompts and backends is an open product decision; that order is a recommendation, +not authorization to add prompt overrides before the decision is resolved. + +## Work Required + +### Runtime Construction and Source Accounting + +Migrate `internal/framework/llm/runtime.go` to layered construction and retain +the role-bound adapter boundary. Review startup inspection and failure +classification because upstream now validates all final definitions, including +unused ones, during engine construction. Distinguish invalid operator inputs +from broken embedded assets under the existing +[CLI exit contract](../cli.md); do not treat every construction failure as an +operator error merely because validation moved earlier. + +Upstream permits partial profiles and defers endpoint/model completeness until +request overrides are applied. Preserve Audita's established checks for its +selected roles without adding a requirement that every unselected profile be +runnable. Native rejection of invalid unused definitions is different from +requiring all profiles to be complete. + +Adapt source accounting to the new discovery/capture boundary. The existing +recording filesystem wrapper may be replaceable with discovered inputs or +upstream source metadata. Protect the host sources actually consumed by the +runtime, including inherited and differently named definitions, without +guessing filenames from selected profile IDs. If operator prompt resources are +approved, account for their consumed message files as well. Keep this +proportionate to the existing collision contract; do not add filesystem +concurrency or power-loss guarantees. + +Keep the existing safe effective-profile summaries and application explanations. +Use upstream metadata where useful, but do not expose the full +`InspectConfiguration` snapshot as ordinary CLI or report output: prompt text +and arbitrary provider fields are outside those summaries' current scope. + +### Prompt Assets, Profiles, and Presence Semantics + +Move each of the seven files in `internal/prompts/assets/prompts` to the keyed +prompt/version format. Keep the shared instruction fragments and the complete +per-call output contracts owned by Audita. This migration does not authorize +another prompt-quality rewrite or changes to domain validators. + +Convert the six maintained profiles in `examples/profiles` and test-generated +profile/prompt fixtures to the new format. Convert any old `base_profile` +usage to `extends`, and remove embedded identities. Migrate tests that construct +engines using removed options or old `Profile`/`Backend` values. + +Review presence deliberately rather than mechanically preserving old serialized +values. In particular, explicit zero temperature, top-p, or max-tokens now means +an outbound zero; zero timeout disables PromptKit's generation deadline while +Audita's outer operation deadline still applies. Omission, blank values, and +null have field-specific meanings. Empty provider maps do not erase inherited +entries, and changing model/backend does not erase unrelated explicit settings. + +Use the native meanings without a compatibility shim. Retaining the examples' +literal `temperature: 0` as an explicit provider setting is recommended; explain +that omitting the field requests an unspecified value, and null suppresses an +inherited control. Do not automatically turn every existing zero into omission +or every omission into a zero. Demonstrate representative profile inheritance +without requiring every maintained example to use it. + +### Documentation, Migration, and Attribution + +Update the profile section of `docs/config.md` and maintained examples when the +implementation lands. Explain keyed identities, composition versus precedence, +profile inheritance, presence, eager final-definition validation, selected-root +errors, and captured configuration. Link to the pinned upstream reference for +generic field semantics rather than reproducing its entire contract. + +Include operator migration guidance for old profile documents and behaviorally +significant zeros/nulls. This is a deliberate format migration, not an automatic +upgrade of existing installation files. Preserve supported application overlays, +including the cheaper/faster testing-profile workflow. + +Update the pinned consumer link in `docs/integrations/openai-compatible-llm.md`, +construction/source-accounting descriptions in `docs/internal/llm-runtime.md`, +and asset-format guidance in `docs/internal/prompts.md`. Update other affected +references only where implemented behavior changes. + +Update `go.mod`, `go.sum`, and `docs/oss/attribution.md` for PromptKit and the two +catalog modules. Verify exact selected source revisions and upstream notices +under the [OSS policy](../policy/oss.md); retain the applicable LGPL/GPL and MIT +texts rather than duplicating unchanged license documents. Dependency adoption +does not change Audita's application license. + +The profile format is publicly documented and incompatible with the old one. +Assess Audita's release classification under the +[release policy](../policy/release.md#versioning-and-compatibility); PromptKit's +own pre-1.0 minor version does not establish that this is an Audita minor update. +Version selection and publication belong to later release preparation. + +## Verification and Completion + +Use existing offline tests as the starting point. Update fixtures intentionally, +and add focused behavioral coverage where the source migration creates a +distinct risk. Completion means: + +- The released dependency set builds and the complete Audita test suite passes. +- All registered prompts load and render from keyed assets, with the established + per-call schemas and adapter-supplied shared instructions. +- Selected roles use final application backend settings and direct resolved + credentials; both roles still share configured capacity, including explicit + zero queue capacity. Catalog layering does not introduce a competing lookup. +- Representative keyed profiles, native inheritance, and explicit presence work + through the real runtime. Directory conflicts and invalid final definitions + produce useful startup errors without filename-derived identity assumptions. +- Source collision protection, safe inspection, and operator-versus-asset error + classification remain effective at the changed construction boundary. +- Representative prepared execution, schema repair, transient retry, and + partial-success tests retain their existing behavior. Reuse adequate coverage + rather than duplicating PromptKit's complete upstream test matrix. +- Maintained examples and canonical documentation describe the migrated format; + attribution identifies the selected PromptKit and catalog sources accurately. + +No live provider calls, new release gates, or CI-success dependency are needed +to establish adoption. Guided domain recovery and the other work recorded in +[future.md](future.md) remain separate features. Do not expand this upgrade into +new report formats, pipeline scheduling, or general configuration-inspection +commands. A staged implementation plan can be prepared after the decision below. + +## Open Decisions + +### Scope of the Operator Configuration Directory + +**Recommendation:** Accept native unified PromptKit documents in the existing +`profiles_dir`, with embedded prompts below operator definitions and application +backend settings above them. Keep the name/path setting for continuity, while +documenting its broader accepted content. This follows the upstream source API +without a parallel parser or arbitrary top-level filtering. It allows profiles, +provider defaults, and intentional prompt overrides to use the same native +composition model. Audita would still ship its domain prompts, enforce its +selected-backend requirements, and supply its per-call schemas. Prompt overrides +would be an intentional new operator capability and need focused documentation +and tests, including resource source accounting; they should not arrive silently. + +**Viable alternative:** Keep this directory profile-only and retain embedded +prompt ownership as the existing operator boundary. This narrows the migration +to profile format and engine assembly, avoids adding prompt customization during +a dependency upgrade, and makes the current directory name accurate. It requires +an explicit contribution-scope check using the libraries' public configuration +facilities, rather than another YAML parser or inheritance engine. Do not also +impose one-profile-per-file, filename/ID equality, or new profile-field +restrictions. The cost is an Audita-specific constraint on a now-unified source +API and continuing maintenance of that boundary. If the public API cannot +express the check simply, reassess the tradeoff or request narrow upstream +support rather than building a general projection layer. + +Either choice can preserve the existing execution workflow. The decision is +whether this adoption should deliberately open operator prompt/backend +contributions; changing the source API by itself does not settle that policy.