Add operator-facing PromptKit inspection to the roadmap
This commit is contained in:
@@ -255,10 +255,62 @@ profile optimization is not a prerequisite for this upgrade.
|
||||
|
||||
Raw sources, configuration history, prompt bodies, endpoints, and credential
|
||||
values must not be added to ordinary manifests or checkpoint metadata. The
|
||||
new inspection API is useful privately for the migration and captured identity;
|
||||
it does not require a new public CLI report or automatic dump into output or
|
||||
debug bundles. Hash-only identity is distinct from publishing an inspection
|
||||
report, and redaction is not a substitute for covering execution semantics.
|
||||
new inspection API supports both private captured identity and the explicitly
|
||||
requested operator reports below. Do not automatically dump reports into run
|
||||
output or debug bundles. Hash-only identity is distinct from publishing an
|
||||
inspection report, and redaction is not a substitute for covering execution
|
||||
semantics.
|
||||
|
||||
### Operator-Facing PromptKit Inspection
|
||||
|
||||
Provide explicit CLI operations for three complementary views:
|
||||
|
||||
- **Effective configuration:** write the captured, ConfigKit-redacted PromptKit
|
||||
configuration, including effective keyed backends, profiles, and prompts.
|
||||
Use `Engine.InspectConfiguration().WriteEffective` rather than reconstructing
|
||||
YAML from Notarius's profile declarations or runtime manifests.
|
||||
- **Setting explanation:** explain one canonical JSON Pointer in that captured
|
||||
configuration, including source history, layer overrides, and inheritance.
|
||||
Use the captured inspection's explanation API. Literal IDs containing `/`
|
||||
or `~` use ordinary JSON Pointer escaping, not a dot-path interpretation.
|
||||
- **Resolved profile:** inspect one explicit profile ID with `InspectProfile`
|
||||
and present its backend-completed execution settings, including target,
|
||||
effective reasoning/provider controls, credential environment name and
|
||||
requirement, and selected backend identity. Endpoint-only profiles remain
|
||||
identifiable as such. Do not conflate configured profile fields with values
|
||||
derived from backend defaults.
|
||||
|
||||
Use the same selected Notarius configuration/overlay, registered application
|
||||
assets, operator sources, and local-backend contribution as execution and
|
||||
preflight. A report describes the engine captured for that invocation, not a
|
||||
live view of a separate running process. Constructing the engine performs
|
||||
native definition/content validation, but these commands do not need transcript
|
||||
input, execute pipeline stages, prepare requests, resolve credential values,
|
||||
admit provider capacity, or contact a model. Required-credential profiles remain
|
||||
inspectable when the named environment variable is unset.
|
||||
|
||||
Keep existing Notarius `config effective` and `config explain` behavior intact;
|
||||
the PromptKit view must be an explicit selection, not a replacement or an
|
||||
automatic second document appended to their output. Define the concrete command
|
||||
syntax in the implementation plan and maintain it in `docs/cli.md` when it
|
||||
lands. Standard output contains the selected report only; ordinary diagnostics
|
||||
and failures follow the existing CLI conventions. Unknown paths/profiles and
|
||||
invalid pointers must be distinguishable without leaking raw configuration
|
||||
through error messages.
|
||||
|
||||
Reports are operator-requested application data. Preserve upstream credential
|
||||
redaction and Notarius's existing secret-handling boundary; include credential
|
||||
variable names, never their values. Configuration explanations can contain
|
||||
overridden values and inline prompt/provider content, and effective YAML is not
|
||||
a transcript-free or generally sanitized report. Document these properties and
|
||||
operator-owned output handling without adding permission enforcement, new
|
||||
confirmation gates, automatic report persistence, or blanket content removal.
|
||||
|
||||
Effective configuration, backend-completed profile settings, and prepared
|
||||
request details are different views. These commands do not render a prompt
|
||||
with transcript inputs or return captured message-file bodies merely because
|
||||
they appear in the configuration as `content_file` references. Keep complete
|
||||
request/response inspection on the existing explicitly requested debug path.
|
||||
|
||||
### Existing Execution Policies Remain Unchanged
|
||||
|
||||
@@ -289,6 +341,10 @@ not by exposing arbitrary upstream configuration content.
|
||||
- Adopt the unified source policy, explicit operator imports, presence/null
|
||||
behavior, native operator capabilities, and captured
|
||||
lifetime; replace live-file checkpoint fingerprinting with captured identity.
|
||||
- Add explicit effective-configuration, setting-explanation, and resolved-
|
||||
profile CLI inspection using the shared captured-engine construction and
|
||||
public PromptKit inspection APIs. Keep upstream inspection values within
|
||||
the adapter boundary and preserve ordinary Notarius configuration reports.
|
||||
- Update behavioral tests at the adapter, CLI preflight, asset registry, and
|
||||
representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
|
||||
loading and offline injected clients or local HTTP fixtures.
|
||||
@@ -297,7 +353,9 @@ not by exposing arbitrary upstream configuration content.
|
||||
migration; `docs/operations.md` owns deployment and reload guidance;
|
||||
`docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
|
||||
`docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
|
||||
pinned links. Revise consumer/CLI examples only where actually affected.
|
||||
pinned links; `docs/cli.md` owns the new inspection operations and output/error
|
||||
behavior. Add a short troubleshooting workflow in `docs/operations.md` that
|
||||
links to those references. Revise consumer examples only where affected.
|
||||
- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
|
||||
checking their actual notices and exact source commits. PromptKit remains
|
||||
LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
|
||||
@@ -326,6 +384,12 @@ Acceptance is behavioral, not an inventory of private helper names:
|
||||
- Engine/file mutation tests demonstrate captured settings and content;
|
||||
fingerprints stay tied to that engine and change when a new engine captures
|
||||
different execution-relevant settings or operator prompt content.
|
||||
- CLI inspection reports effective values, setting origins, and backend-
|
||||
completed profiles from the same source policy without transcript input,
|
||||
credentials, or provider calls. Cover representative inheritance/null
|
||||
overrides, escaped literal IDs, required-credential inspection with an unset
|
||||
variable, and invalid/unknown selectors. Verify redaction and unchanged
|
||||
ordinary Notarius reports without pinning incidental report formatting.
|
||||
- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
|
||||
transient errors, safe diagnostics, and observations retain existing
|
||||
provider-neutral behavior using deterministic boundary tests.
|
||||
|
||||
Reference in New Issue
Block a user