Add operator-facing PromptKit inspection to the roadmap
This commit is contained in:
@@ -255,10 +255,62 @@ profile optimization is not a prerequisite for this upgrade.
|
|||||||
|
|
||||||
Raw sources, configuration history, prompt bodies, endpoints, and credential
|
Raw sources, configuration history, prompt bodies, endpoints, and credential
|
||||||
values must not be added to ordinary manifests or checkpoint metadata. The
|
values must not be added to ordinary manifests or checkpoint metadata. The
|
||||||
new inspection API is useful privately for the migration and captured identity;
|
new inspection API supports both private captured identity and the explicitly
|
||||||
it does not require a new public CLI report or automatic dump into output or
|
requested operator reports below. Do not automatically dump reports into run
|
||||||
debug bundles. Hash-only identity is distinct from publishing an inspection
|
output or debug bundles. Hash-only identity is distinct from publishing an
|
||||||
report, and redaction is not a substitute for covering execution semantics.
|
inspection report, and redaction is not a substitute for covering execution
|
||||||
|
semantics.
|
||||||
|
|
||||||
|
### Operator-Facing PromptKit Inspection
|
||||||
|
|
||||||
|
Provide explicit CLI operations for three complementary views:
|
||||||
|
|
||||||
|
- **Effective configuration:** write the captured, ConfigKit-redacted PromptKit
|
||||||
|
configuration, including effective keyed backends, profiles, and prompts.
|
||||||
|
Use `Engine.InspectConfiguration().WriteEffective` rather than reconstructing
|
||||||
|
YAML from Notarius's profile declarations or runtime manifests.
|
||||||
|
- **Setting explanation:** explain one canonical JSON Pointer in that captured
|
||||||
|
configuration, including source history, layer overrides, and inheritance.
|
||||||
|
Use the captured inspection's explanation API. Literal IDs containing `/`
|
||||||
|
or `~` use ordinary JSON Pointer escaping, not a dot-path interpretation.
|
||||||
|
- **Resolved profile:** inspect one explicit profile ID with `InspectProfile`
|
||||||
|
and present its backend-completed execution settings, including target,
|
||||||
|
effective reasoning/provider controls, credential environment name and
|
||||||
|
requirement, and selected backend identity. Endpoint-only profiles remain
|
||||||
|
identifiable as such. Do not conflate configured profile fields with values
|
||||||
|
derived from backend defaults.
|
||||||
|
|
||||||
|
Use the same selected Notarius configuration/overlay, registered application
|
||||||
|
assets, operator sources, and local-backend contribution as execution and
|
||||||
|
preflight. A report describes the engine captured for that invocation, not a
|
||||||
|
live view of a separate running process. Constructing the engine performs
|
||||||
|
native definition/content validation, but these commands do not need transcript
|
||||||
|
input, execute pipeline stages, prepare requests, resolve credential values,
|
||||||
|
admit provider capacity, or contact a model. Required-credential profiles remain
|
||||||
|
inspectable when the named environment variable is unset.
|
||||||
|
|
||||||
|
Keep existing Notarius `config effective` and `config explain` behavior intact;
|
||||||
|
the PromptKit view must be an explicit selection, not a replacement or an
|
||||||
|
automatic second document appended to their output. Define the concrete command
|
||||||
|
syntax in the implementation plan and maintain it in `docs/cli.md` when it
|
||||||
|
lands. Standard output contains the selected report only; ordinary diagnostics
|
||||||
|
and failures follow the existing CLI conventions. Unknown paths/profiles and
|
||||||
|
invalid pointers must be distinguishable without leaking raw configuration
|
||||||
|
through error messages.
|
||||||
|
|
||||||
|
Reports are operator-requested application data. Preserve upstream credential
|
||||||
|
redaction and Notarius's existing secret-handling boundary; include credential
|
||||||
|
variable names, never their values. Configuration explanations can contain
|
||||||
|
overridden values and inline prompt/provider content, and effective YAML is not
|
||||||
|
a transcript-free or generally sanitized report. Document these properties and
|
||||||
|
operator-owned output handling without adding permission enforcement, new
|
||||||
|
confirmation gates, automatic report persistence, or blanket content removal.
|
||||||
|
|
||||||
|
Effective configuration, backend-completed profile settings, and prepared
|
||||||
|
request details are different views. These commands do not render a prompt
|
||||||
|
with transcript inputs or return captured message-file bodies merely because
|
||||||
|
they appear in the configuration as `content_file` references. Keep complete
|
||||||
|
request/response inspection on the existing explicitly requested debug path.
|
||||||
|
|
||||||
### Existing Execution Policies Remain Unchanged
|
### Existing Execution Policies Remain Unchanged
|
||||||
|
|
||||||
@@ -289,6 +341,10 @@ not by exposing arbitrary upstream configuration content.
|
|||||||
- Adopt the unified source policy, explicit operator imports, presence/null
|
- Adopt the unified source policy, explicit operator imports, presence/null
|
||||||
behavior, native operator capabilities, and captured
|
behavior, native operator capabilities, and captured
|
||||||
lifetime; replace live-file checkpoint fingerprinting with captured identity.
|
lifetime; replace live-file checkpoint fingerprinting with captured identity.
|
||||||
|
- Add explicit effective-configuration, setting-explanation, and resolved-
|
||||||
|
profile CLI inspection using the shared captured-engine construction and
|
||||||
|
public PromptKit inspection APIs. Keep upstream inspection values within
|
||||||
|
the adapter boundary and preserve ordinary Notarius configuration reports.
|
||||||
- Update behavioral tests at the adapter, CLI preflight, asset registry, and
|
- Update behavioral tests at the adapter, CLI preflight, asset registry, and
|
||||||
representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
|
representative assembled-module boundaries. Reuse real ConfigKit/PromptKit
|
||||||
loading and offline injected clients or local HTTP fixtures.
|
loading and offline injected clients or local HTTP fixtures.
|
||||||
@@ -297,7 +353,9 @@ not by exposing arbitrary upstream configuration content.
|
|||||||
migration; `docs/operations.md` owns deployment and reload guidance;
|
migration; `docs/operations.md` owns deployment and reload guidance;
|
||||||
`docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
|
`docs/internal/llm.md` owns construction, capture, and checkpoint mechanics;
|
||||||
`docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
|
`docs/integrations/pkg-promptkit.md` owns the supported upstream boundary and
|
||||||
pinned links. Revise consumer/CLI examples only where actually affected.
|
pinned links; `docs/cli.md` owns the new inspection operations and output/error
|
||||||
|
behavior. Add a short troubleshooting workflow in `docs/operations.md` that
|
||||||
|
links to those references. Revise consumer examples only where affected.
|
||||||
- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
|
- Update `docs/oss/attribution.md` for PromptKit and the two `/v3` catalogs,
|
||||||
checking their actual notices and exact source commits. PromptKit remains
|
checking their actual notices and exact source commits. PromptKit remains
|
||||||
LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
|
LGPL-3.0-or-later; the catalogs remain MIT at the reviewed revisions. Reuse
|
||||||
@@ -326,6 +384,12 @@ Acceptance is behavioral, not an inventory of private helper names:
|
|||||||
- Engine/file mutation tests demonstrate captured settings and content;
|
- Engine/file mutation tests demonstrate captured settings and content;
|
||||||
fingerprints stay tied to that engine and change when a new engine captures
|
fingerprints stay tied to that engine and change when a new engine captures
|
||||||
different execution-relevant settings or operator prompt content.
|
different execution-relevant settings or operator prompt content.
|
||||||
|
- CLI inspection reports effective values, setting origins, and backend-
|
||||||
|
completed profiles from the same source policy without transcript input,
|
||||||
|
credentials, or provider calls. Cover representative inheritance/null
|
||||||
|
overrides, escaped literal IDs, required-credential inspection with an unset
|
||||||
|
variable, and invalid/unknown selectors. Verify redaction and unchanged
|
||||||
|
ordinary Notarius reports without pinning incidental report formatting.
|
||||||
- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
|
- Structural repair, semantic feedback, sessions/reasoning, capacity failure,
|
||||||
transient errors, safe diagnostics, and observations retain existing
|
transient errors, safe diagnostics, and observations retain existing
|
||||||
provider-neutral behavior using deterministic boundary tests.
|
provider-neutral behavior using deterministic boundary tests.
|
||||||
|
|||||||
Reference in New Issue
Block a user