diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 07f1293e..b76fd6f6 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -255,10 +255,62 @@ profile optimization is not a prerequisite for this upgrade. Raw sources, configuration history, prompt bodies, endpoints, and credential values must not be added to ordinary manifests or checkpoint metadata. The -new inspection API is useful privately for the migration and captured identity; -it does not require a new public CLI report or automatic dump into output or -debug bundles. Hash-only identity is distinct from publishing an inspection -report, and redaction is not a substitute for covering execution semantics. +new inspection API supports both private captured identity and the explicitly +requested operator reports below. Do not automatically dump reports into run +output or debug bundles. Hash-only identity is distinct from publishing an +inspection report, and redaction is not a substitute for covering execution +semantics. + +### Operator-Facing PromptKit Inspection + +Provide explicit CLI operations for three complementary views: + +- **Effective configuration:** write the captured, ConfigKit-redacted PromptKit + configuration, including effective keyed backends, profiles, and prompts. + Use `Engine.InspectConfiguration().WriteEffective` rather than reconstructing + YAML from Notarius's profile declarations or runtime manifests. +- **Setting explanation:** explain one canonical JSON Pointer in that captured + configuration, including source history, layer overrides, and inheritance. + Use the captured inspection's explanation API. Literal IDs containing `/` + or `~` use ordinary JSON Pointer escaping, not a dot-path interpretation. +- **Resolved profile:** inspect one explicit profile ID with `InspectProfile` + and present its backend-completed execution settings, including target, + effective reasoning/provider controls, credential environment name and + requirement, and selected backend identity. Endpoint-only profiles remain + identifiable as such. Do not conflate configured profile fields with values + derived from backend defaults. + +Use the same selected Notarius configuration/overlay, registered application +assets, operator sources, and local-backend contribution as execution and +preflight. A report describes the engine captured for that invocation, not a +live view of a separate running process. Constructing the engine performs +native definition/content validation, but these commands do not need transcript +input, execute pipeline stages, prepare requests, resolve credential values, +admit provider capacity, or contact a model. Required-credential profiles remain +inspectable when the named environment variable is unset. + +Keep existing Notarius `config effective` and `config explain` behavior intact; +the PromptKit view must be an explicit selection, not a replacement or an +automatic second document appended to their output. Define the concrete command +syntax in the implementation plan and maintain it in `docs/cli.md` when it +lands. Standard output contains the selected report only; ordinary diagnostics +and failures follow the existing CLI conventions. Unknown paths/profiles and +invalid pointers must be distinguishable without leaking raw configuration +through error messages. + +Reports are operator-requested application data. Preserve upstream credential +redaction and Notarius's existing secret-handling boundary; include credential +variable names, never their values. Configuration explanations can contain +overridden values and inline prompt/provider content, and effective YAML is not +a transcript-free or generally sanitized report. Document these properties and +operator-owned output handling without adding permission enforcement, new +confirmation gates, automatic report persistence, or blanket content removal. + +Effective configuration, backend-completed profile settings, and prepared +request details are different views. These commands do not render a prompt +with transcript inputs or return captured message-file bodies merely because +they appear in the configuration as `content_file` references. Keep complete +request/response inspection on the existing explicitly requested debug path. ### Existing Execution Policies Remain Unchanged @@ -289,6 +341,10 @@ not by exposing arbitrary upstream configuration content. - Adopt the unified source policy, explicit operator imports, presence/null behavior, native operator capabilities, and captured lifetime; replace live-file checkpoint fingerprinting with captured identity. +- Add explicit effective-configuration, setting-explanation, and resolved- + profile CLI inspection using the shared captured-engine construction and + public PromptKit inspection APIs. Keep upstream inspection values within + the adapter boundary and preserve ordinary Notarius configuration reports. - Update behavioral tests at the adapter, CLI preflight, asset registry, and representative assembled-module boundaries. Reuse real ConfigKit/PromptKit loading and offline injected clients or local HTTP fixtures. @@ -297,7 +353,9 @@ not by exposing arbitrary upstream configuration content. migration; `docs/operations.md` owns deployment and reload guidance; `docs/internal/llm.md` owns construction, capture, and checkpoint mechanics; `docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and - pinned links. Revise consumer/CLI examples only where actually affected. + pinned links; `docs/cli.md` owns the new inspection operations and output/error + behavior. Add a short troubleshooting workflow in `docs/operations.md` that + links to those references. Revise consumer examples only where affected. - Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs, checking their actual notices and exact source commits. PromptKit remains LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse @@ -326,6 +384,12 @@ Acceptance is behavioral, not an inventory of private helper names: - Engine/file mutation tests demonstrate captured settings and content; fingerprints stay tied to that engine and change when a new engine captures different execution-relevant settings or operator prompt content. +- CLI inspection reports effective values, setting origins, and backend- + completed profiles from the same source policy without transcript input, + credentials, or provider calls. Cover representative inheritance/null + overrides, escaped literal IDs, required-credential inspection with an unset + variable, and invalid/unknown selectors. Verify redaction and unchanged + ordinary Notarius reports without pinning incidental report formatting. - Structural repair, semantic feedback, sessions/reasoning, capacity failure, transient errors, safe diagnostics, and observations retain existing provider-neutral behavior using deterministic boundary tests.