Add operator-facing PromptKit inspection to the roadmap

This commit is contained in:
2026-10-04 11:13:33 +00:00
parent 71cf084e8b
commit aae01a59ce

View File

@@ -255,10 +255,62 @@ profile optimization is not a prerequisite for this upgrade.
Raw sources, configuration history, prompt bodies, endpoints, and credential
values must not be added to ordinary manifests or checkpoint metadata. The
new inspection API is useful privately for the migration and captured identity;
it does not require a new public CLI report or automatic dump into output or
debug bundles. Hash-only identity is distinct from publishing an inspection
report, and redaction is not a substitute for covering execution semantics.
new inspection API supports both private captured identity and the explicitly
requested operator reports below. Do not automatically dump reports into run
output or debug bundles. Hash-only identity is distinct from publishing an
inspection report, and redaction is not a substitute for covering execution
semantics.
### Operator-Facing PromptKit Inspection
Provide explicit CLI operations for three complementary views:
- **Effective configuration:** write the captured, ConfigKit-redacted PromptKit
configuration, including effective keyed backends, profiles, and prompts.
Use `Engine.InspectConfiguration().WriteEffective` rather than reconstructing
YAML from Notarius's profile declarations or runtime manifests.
- **Setting explanation:** explain one canonical JSON Pointer in that captured
configuration, including source history, layer overrides, and inheritance.
Use the captured inspection's explanation API. Literal IDs containing `/`
or `~` use ordinary JSON Pointer escaping, not a dot-path interpretation.
- **Resolved profile:** inspect one explicit profile ID with `InspectProfile`
and present its backend-completed execution settings, including target,
effective reasoning/provider controls, credential environment name and
requirement, and selected backend identity. Endpoint-only profiles remain
identifiable as such. Do not conflate configured profile fields with values
derived from backend defaults.
Use the same selected Notarius configuration/overlay, registered application
assets, operator sources, and local-backend contribution as execution and
preflight. A report describes the engine captured for that invocation, not a
live view of a separate running process. Constructing the engine performs
native definition/content validation, but these commands do not need transcript
input, execute pipeline stages, prepare requests, resolve credential values,
admit provider capacity, or contact a model. Required-credential profiles remain
inspectable when the named environment variable is unset.
Keep existing Notarius `config effective` and `config explain` behavior intact;
the PromptKit view must be an explicit selection, not a replacement or an
automatic second document appended to their output. Define the concrete command
syntax in the implementation plan and maintain it in `docs/cli.md` when it
lands. Standard output contains the selected report only; ordinary diagnostics
and failures follow the existing CLI conventions. Unknown paths/profiles and
invalid pointers must be distinguishable without leaking raw configuration
through error messages.
Reports are operator-requested application data. Preserve upstream credential
redaction and Notarius's existing secret-handling boundary; include credential
variable names, never their values. Configuration explanations can contain
overridden values and inline prompt/provider content, and effective YAML is not
a transcript-free or generally sanitized report. Document these properties and
operator-owned output handling without adding permission enforcement, new
confirmation gates, automatic report persistence, or blanket content removal.
Effective configuration, backend-completed profile settings, and prepared
request details are different views. These commands do not render a prompt
with transcript inputs or return captured message-file bodies merely because
they appear in the configuration as `content_file` references. Keep complete
request/response inspection on the existing explicitly requested debug path.
### Existing Execution Policies Remain Unchanged
@@ -289,6 +341,10 @@ not by exposing arbitrary upstream configuration content.
- Adopt the unified source policy, explicit operator imports, presence/null
behavior, native operator capabilities, and captured
lifetime; replace live-file checkpoint fingerprinting with captured identity.
- Add explicit effective-configuration, setting-explanation, and resolved-
profile CLI inspection using the shared captured-engine construction and
public PromptKit inspection APIs. Keep upstream inspection values within
the adapter boundary and preserve ordinary Notarius configuration reports.
- Update behavioral tests at the adapter, CLI preflight, asset registry, and
representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
loading and offline injected clients or local HTTP fixtures.
@@ -297,7 +353,9 @@ not by exposing arbitrary upstream configuration content.
migration; `docs/operations.md` owns deployment and reload guidance;
`docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
`docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
pinned links. Revise consumer/CLI examples only where actually affected.
pinned links; `docs/cli.md` owns the new inspection operations and output/error
behavior. Add a short troubleshooting workflow in `docs/operations.md` that
links to those references. Revise consumer examples only where affected.
- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
checking their actual notices and exact source commits. PromptKit remains
LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
@@ -326,6 +384,12 @@ Acceptance is behavioral, not an inventory of private helper names:
- Engine/file mutation tests demonstrate captured settings and content;
fingerprints stay tied to that engine and change when a new engine captures
different execution-relevant settings or operator prompt content.
- CLI inspection reports effective values, setting origins, and backend-
completed profiles from the same source policy without transcript input,
credentials, or provider calls. Cover representative inheritance/null
overrides, escaped literal IDs, required-credential inspection with an unset
variable, and invalid/unknown selectors. Verify redaction and unchanged
ordinary Notarius reports without pinning incidental report formatting.
- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
transient errors, safe diagnostics, and observations retain existing
provider-neutral behavior using deterministic boundary tests.