Document the PromptKit upgrade roadmap

2026-10-03 17:45:40 +00:00
parent 8cba751835
commit 79d38606fe
1 changed files with 254 additions and 0 deletions
--- a/docs/roadmap/promptkit.md
+++ b/docs/roadmap/promptkit.md
@@ -0,0 +1,254 @@
+# PromptKit v0.12.0 Adoption Roadmap
+
+**Last updated:** 2026-10-03
+
+**Status:** Proposed. The configuration-source scope decision at the end of
+this document should be resolved before preparing the implementation plan.
+
+## Purpose and Baseline
+
+Adopt PromptKit v0.12.0 now that the independent
+[ConfigKit v0.5.0 upgrade](configkit.md) is complete. Align Audita with the
+libraries' native configuration model while retaining Audita's domain rules,
+application-owned backend settings, and resilient processing workflow.
+
+At reviewed Audita revision `8cba751835ba9faad340e933a3cfbb59c03924fd`,
+[go.mod](../../go.mod) selects ConfigKit v0.5.0 and PromptKit v0.11.2, with
+OpenRouter and Rakestrawhome `/v2` catalogs at `v2.20260926.0`. Go 1.25.5
+remains the application minimum. The implemented contracts belong to
+[configuration](../config.md), the
+[LLM integration](../integrations/openai-compatible-llm.md), and the
+[LLM runtime](../internal/llm-runtime.md).
+
+`internal/framework/llm.NewRuntime` currently assembles one engine using the
+old prompt/profile filesystem options and programmatic backend registration.
+Audita embeds seven prompt definitions and separate response schemas. Operators
+select proposal and validation profiles from `profiles_dir`; the effective
+backend must be registered in application configuration, its endpoint must
+match, and its credential comes from Audita's ConfigKit-resolved settings.
+Successful profile reads are recorded to protect those sources from output
+collisions. Both roles share the same engine's backend admission controls.
+
+The reviewed upstream release is
+[PromptKit v0.12.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/releases/v0.12.0.md),
+commit `4b8169da286917fde7875425bb32b799b12c6af3`. It depends on ConfigKit
+v0.5.0 and both maintained `/v3` catalog modules at `v3.20260930.0`. This is an
+intentional incompatible pre-1.0 minor release: it replaces construction and
+configuration APIs, rather than simply updating provider data. PromptKit's
+[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/formats.md)
+and [consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.12.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api)
+own the upstream details.
+
+## Target End State
+
+- Audita imports released PromptKit v0.12.0 and its `/v3` catalogs, retaining
+  ConfigKit v0.5.0 and using no local dependency replacement.
+- The runtime uses `Config.Sources` and native ConfigKit source groups.
+  PromptKit owns composition, inheritance, presence, final-definition
+  validation, prompt capture, and execution. Audita does not translate old
+  profiles or implement a second YAML loader or inheritance resolver.
+- Embedded prompts and maintained operator profiles use keyed `prompts` and
+  `profiles` mappings. Existing prompt IDs, versions, role selections,
+  instructions, and response-schema associations remain the adoption baseline.
+  The application configuration schema, overlay selection, CLI interface, and
+  main-file-relative `profiles_dir` resolution remain in place.
+- Native profile `extends`, field-presence semantics, and source-relative
+  resource rules apply. Documents can contain multiple definitions, and IDs
+  come from mapping keys rather than filenames. Operator-directory scope is
+  subject to the open decision below.
+- Application backend endpoints, configured concurrency and queue capacity,
+  and credential ownership remain authoritative for selected roles. Preserve
+  the existing profile-selection and prepared-target checks; do not introduce
+  a second admission controller or implicit provider credential lookup.
+- A runtime uses one captured configuration snapshot for both roles. Each CLI
+  invocation constructs a new engine; hot reload is not part of this feature.
+  Existing inspection remains offline and safe, and consumed host sources
+  remain protected from output collisions.
+- PromptKit retains responsibility for transport retries, schema repair,
+  observation, and prepared execution. Preserve Audita's configured retry and
+  repair budgets, operation deadlines, cancellation, partial-success behavior,
+  error/recovery reporting, and debug selection.
+
+## Configuration Assembly and Ownership
+
+Replace removed `WithPromptFS`, `WithProfileFS`, and `WithBackend` usage with
+the released source API. Discover embedded YAML through `FilesystemInputs` and
+operator YAML through `DirectoryInputs`, with explicit caller tiers. Keep the
+retained schema filesystem integration and HTTP client configuration.
+Use the helpers' default-disabled imports; this adoption does not require new
+application source-selection flags or operator underlay tiers.
+
+PromptKit supplies catalog definitions first and its own defaults next. Caller
+tiers then overlay in their supplied order, followed by an optional highest
+priority main input. Files within one directory group compose strictly;
+alphabetical filenames do not establish override precedence. Use separate
+tiers for genuine priority differences rather than manufacturing file-order
+semantics.
+
+Represent application backend settings as a final programmatic source, using
+`BackendSource` or ConfigKit mappings where explicit null is needed. Preserve
+configured zero queue capacity as an explicit value. Suppress backend-derived
+credential environment lookup explicitly where catalog layering would otherwise
+introduce it; omission alone can retain a lower-tier value. Continue detecting
+competing effective profile credential lookups rather than silently clearing
+operator profile fields. A missing backend credential remains valid for an
+unauthenticated local service unless the selected profile requires a key.
+
+Catalog backend IDs are no longer reserved upstream. Audita should not invent a
+new reservation rule. Catalog selection alone must not bypass application
+registration, endpoint, capacity, or credential requirements. Catalog profile
+contents may evolve; this upgrade does not promise fixed model choices or live
+provider availability.
+
+The proposed caller order is embedded Audita prompts, operator definitions,
+then application backend settings. Whether operator definitions may include
+prompts and backends is an open product decision; that order is a recommendation,
+not authorization to add prompt overrides before the decision is resolved.
+
+## Work Required
+
+### Runtime Construction and Source Accounting
+
+Migrate `internal/framework/llm/runtime.go` to layered construction and retain
+the role-bound adapter boundary. Review startup inspection and failure
+classification because upstream now validates all final definitions, including
+unused ones, during engine construction. Distinguish invalid operator inputs
+from broken embedded assets under the existing
+[CLI exit contract](../cli.md); do not treat every construction failure as an
+operator error merely because validation moved earlier.
+
+Upstream permits partial profiles and defers endpoint/model completeness until
+request overrides are applied. Preserve Audita's established checks for its
+selected roles without adding a requirement that every unselected profile be
+runnable. Native rejection of invalid unused definitions is different from
+requiring all profiles to be complete.
+
+Adapt source accounting to the new discovery/capture boundary. The existing
+recording filesystem wrapper may be replaceable with discovered inputs or
+upstream source metadata. Protect the host sources actually consumed by the
+runtime, including inherited and differently named definitions, without
+guessing filenames from selected profile IDs. If operator prompt resources are
+approved, account for their consumed message files as well. Keep this
+proportionate to the existing collision contract; do not add filesystem
+concurrency or power-loss guarantees.
+
+Keep the existing safe effective-profile summaries and application explanations.
+Use upstream metadata where useful, but do not expose the full
+`InspectConfiguration` snapshot as ordinary CLI or report output: prompt text
+and arbitrary provider fields are outside those summaries' current scope.
+
+### Prompt Assets, Profiles, and Presence Semantics
+
+Move each of the seven files in `internal/prompts/assets/prompts` to the keyed
+prompt/version format. Keep the shared instruction fragments and the complete
+per-call output contracts owned by Audita. This migration does not authorize
+another prompt-quality rewrite or changes to domain validators.
+
+Convert the six maintained profiles in `examples/profiles` and test-generated
+profile/prompt fixtures to the new format. Convert any old `base_profile`
+usage to `extends`, and remove embedded identities. Migrate tests that construct
+engines using removed options or old `Profile`/`Backend` values.
+
+Review presence deliberately rather than mechanically preserving old serialized
+values. In particular, explicit zero temperature, top-p, or max-tokens now means
+an outbound zero; zero timeout disables PromptKit's generation deadline while
+Audita's outer operation deadline still applies. Omission, blank values, and
+null have field-specific meanings. Empty provider maps do not erase inherited
+entries, and changing model/backend does not erase unrelated explicit settings.
+
+Use the native meanings without a compatibility shim. Retaining the examples'
+literal `temperature: 0` as an explicit provider setting is recommended; explain
+that omitting the field requests an unspecified value, and null suppresses an
+inherited control. Do not automatically turn every existing zero into omission
+or every omission into a zero. Demonstrate representative profile inheritance
+without requiring every maintained example to use it.
+
+### Documentation, Migration, and Attribution
+
+Update the profile section of `docs/config.md` and maintained examples when the
+implementation lands. Explain keyed identities, composition versus precedence,
+profile inheritance, presence, eager final-definition validation, selected-root
+errors, and captured configuration. Link to the pinned upstream reference for
+generic field semantics rather than reproducing its entire contract.
+
+Include operator migration guidance for old profile documents and behaviorally
+significant zeros/nulls. This is a deliberate format migration, not an automatic
+upgrade of existing installation files. Preserve supported application overlays,
+including the cheaper/faster testing-profile workflow.
+
+Update the pinned consumer link in `docs/integrations/openai-compatible-llm.md`,
+construction/source-accounting descriptions in `docs/internal/llm-runtime.md`,
+and asset-format guidance in `docs/internal/prompts.md`. Update other affected
+references only where implemented behavior changes.
+
+Update `go.mod`, `go.sum`, and `docs/oss/attribution.md` for PromptKit and the two
+catalog modules. Verify exact selected source revisions and upstream notices
+under the [OSS policy](../policy/oss.md); retain the applicable LGPL/GPL and MIT
+texts rather than duplicating unchanged license documents. Dependency adoption
+does not change Audita's application license.
+
+The profile format is publicly documented and incompatible with the old one.
+Assess Audita's release classification under the
+[release policy](../policy/release.md#versioning-and-compatibility); PromptKit's
+own pre-1.0 minor version does not establish that this is an Audita minor update.
+Version selection and publication belong to later release preparation.
+
+## Verification and Completion
+
+Use existing offline tests as the starting point. Update fixtures intentionally,
+and add focused behavioral coverage where the source migration creates a
+distinct risk. Completion means:
+
+- The released dependency set builds and the complete Audita test suite passes.
+- All registered prompts load and render from keyed assets, with the established
+  per-call schemas and adapter-supplied shared instructions.
+- Selected roles use final application backend settings and direct resolved
+  credentials; both roles still share configured capacity, including explicit
+  zero queue capacity. Catalog layering does not introduce a competing lookup.
+- Representative keyed profiles, native inheritance, and explicit presence work
+  through the real runtime. Directory conflicts and invalid final definitions
+  produce useful startup errors without filename-derived identity assumptions.
+- Source collision protection, safe inspection, and operator-versus-asset error
+  classification remain effective at the changed construction boundary.
+- Representative prepared execution, schema repair, transient retry, and
+  partial-success tests retain their existing behavior. Reuse adequate coverage
+  rather than duplicating PromptKit's complete upstream test matrix.
+- Maintained examples and canonical documentation describe the migrated format;
+  attribution identifies the selected PromptKit and catalog sources accurately.
+
+No live provider calls, new release gates, or CI-success dependency are needed
+to establish adoption. Guided domain recovery and the other work recorded in
+[future.md](future.md) remain separate features. Do not expand this upgrade into
+new report formats, pipeline scheduling, or general configuration-inspection
+commands. A staged implementation plan can be prepared after the decision below.
+
+## Open Decisions
+
+### Scope of the Operator Configuration Directory
+
+**Recommendation:** Accept native unified PromptKit documents in the existing
+`profiles_dir`, with embedded prompts below operator definitions and application
+backend settings above them. Keep the name/path setting for continuity, while
+documenting its broader accepted content. This follows the upstream source API
+without a parallel parser or arbitrary top-level filtering. It allows profiles,
+provider defaults, and intentional prompt overrides to use the same native
+composition model. Audita would still ship its domain prompts, enforce its
+selected-backend requirements, and supply its per-call schemas. Prompt overrides
+would be an intentional new operator capability and need focused documentation
+and tests, including resource source accounting; they should not arrive silently.
+
+**Viable alternative:** Keep this directory profile-only and retain embedded
+prompt ownership as the existing operator boundary. This narrows the migration
+to profile format and engine assembly, avoids adding prompt customization during
+a dependency upgrade, and makes the current directory name accurate. It requires
+an explicit contribution-scope check using the libraries' public configuration
+facilities, rather than another YAML parser or inheritance engine. Do not also
+impose one-profile-per-file, filename/ID equality, or new profile-field
+restrictions. The cost is an Audita-specific constraint on a now-unified source
+API and continuing maintenance of that boundary. If the public API cannot
+express the check simply, reassess the tradeoff or request narrow upstream
+support rather than building a general projection layer.
+
+Either choice can preserve the existing execution workflow. The decision is
+whether this adoption should deliberately open operator prompt/backend
+contributions; changing the source API by itself does not settle that policy.