Add configuration path explanations
This commit is contained in:
15
docs/cli.md
15
docs/cli.md
@@ -28,6 +28,7 @@ go run ./cmd/scriptorium render \
|
||||
- `scriptorium config validate`: validate application configuration without building an engine.
|
||||
- `scriptorium config show`: show the resolved application configuration.
|
||||
- `scriptorium config sources`: show the ordered configuration sources.
|
||||
- `scriptorium config explain`: explain one effective or historical configuration path.
|
||||
|
||||
All commands accept `--config <path>` and `--config-overlay <name>` and reject
|
||||
positional arguments. `--config-overlay` selects one application configuration
|
||||
@@ -226,6 +227,7 @@ controls are shown as zero values when Promptkit leaves them unspecified.
|
||||
scriptorium config validate [flags]
|
||||
scriptorium config show [flags]
|
||||
scriptorium config sources [flags]
|
||||
scriptorium config explain --path JSON_POINTER [flags]
|
||||
```
|
||||
|
||||
Each accepts `--config`, `--config-overlay`, `--prompt-dir`, `--profile-dir`,
|
||||
@@ -244,10 +246,21 @@ record has `id`, `label`, and `layer`, plus `path` when available; explicit CLI
|
||||
overrides appear as synthetic sources. Defaults-only input without overrides
|
||||
produces `[]` for `sources`.
|
||||
|
||||
`explain` requires an explicitly supplied `--path` in JSON Pointer syntax.
|
||||
The empty string selects the root; `/server/addr` selects a field; `~1` and
|
||||
`~0` escape `/` and `~` inside mapping keys. Numeric tokens are interpreted
|
||||
according to the containing mapping or sequence. Dotted paths, file paths,
|
||||
and URL fragments are not accepted. ConfigKit writes a human-readable account
|
||||
of the effective value and its chronological default, base, overlay, CLI,
|
||||
and normalization history, including removed historical values. This is not a
|
||||
JSON schema or a byte-for-byte stable prose format. Unknown and malformed paths
|
||||
exit 1 without successful explanation output.
|
||||
|
||||
Diagnostics and warnings go to stderr, never into successful stdout. Parse,
|
||||
load, output, and diagnostic-write failures exit 1. Ordinary output writes may
|
||||
leave partial data on failure. The inspection output can contain paths,
|
||||
identifiers, `api_key_env` reference names, and ordinary `extra_params` values;
|
||||
identifiers, `api_key_env` reference names, and ordinary `extra_params` values,
|
||||
including displaced historical values in explanations;
|
||||
protect it accordingly. Raw API-key values are not read or emitted by these
|
||||
commands.
|
||||
|
||||
|
||||
@@ -95,7 +95,8 @@ The size fields must be zero or greater. The [HTTP API](api.md) defines how
|
||||
each limit is enforced and reported. `server.artifact_root` configures an HTTP
|
||||
deployment boundary; see [operations](operations.md) for deployment handling.
|
||||
|
||||
The `config validate`, `config show`, and `config sources` commands inspect this
|
||||
The `config validate`, `config show`, `config sources`, and `config explain`
|
||||
commands inspect this
|
||||
application configuration without creating a Promptkit engine. `show` is a
|
||||
resolved inspection snapshot, not necessarily a reloadable file. Application
|
||||
validation does not check framework sources, backend connectivity, or credential
|
||||
@@ -103,6 +104,12 @@ availability. `api_key_env` is an environment-variable *name* and remains
|
||||
visible in inspection; its environment value is never read by ConfigKit.
|
||||
Ordinary backend `extra_params` values also remain visible, so keep secrets out
|
||||
of those fields and protect inspection output as potentially sensitive.
|
||||
`config explain --path /server/addr` uses JSON Pointer syntax to show the
|
||||
selected field's current value and history from defaults, files, overlays,
|
||||
explicit CLI overrides, and normalization. `--path ''` selects the root.
|
||||
History includes replaced and removed values; `show` presents the final
|
||||
resolved snapshot. Neither renderer looks up credential bytes, and neither
|
||||
claims to identify arbitrary secrets in unclassified parameter values.
|
||||
|
||||
## Custom Backends
|
||||
|
||||
|
||||
@@ -99,6 +99,10 @@ Before changing production inputs, profiles, or schemas:
|
||||
policy.
|
||||
|
||||
Inspection output can include configuration paths, names, and ordinary backend
|
||||
parameters. `config explain --path` uses JSON Pointer paths and can show
|
||||
displaced historical values as well as the final value. Protect both the
|
||||
selected paths and the output, especially when backend parameters contain
|
||||
sensitive identifiers; ConfigKit does not infer arbitrary secrets in those
|
||||
parameters. Apply
|
||||
access and retention controls; the application stores only API-key environment
|
||||
reference names, not the resolved credential bytes.
|
||||
|
||||
@@ -390,7 +390,11 @@ selected API-key boundary without masking or changing backend parameter data.
|
||||
|
||||
## Stage 6: Add Per-Path Explanation And Verify Credential Boundaries
|
||||
|
||||
**Status: Not started.**
|
||||
**Status: Complete.** `config explain` uses required JSON Pointer paths and
|
||||
ConfigKit's snapshot renderer, including historical and normalized values.
|
||||
Tests cover root, escaped/numeric keys, unknown paths, writer failures, and
|
||||
synthetic credential-reference boundaries. Full test, race, vet, build, and
|
||||
diff checks passed.
|
||||
|
||||
Primary files: configuration command implementation/tests and CLI/config docs.
|
||||
|
||||
|
||||
@@ -23,16 +23,19 @@ func configCommand(args []string, stdout, stderr io.Writer) int {
|
||||
return ExitRuntimeError
|
||||
}
|
||||
mode := args[0]
|
||||
if mode != "validate" && mode != "show" && mode != "sources" {
|
||||
if mode != "validate" && mode != "show" && mode != "sources" && mode != "explain" {
|
||||
fmt.Fprintln(stderr, "config parse error: unknown configuration mode")
|
||||
return ExitRuntimeError
|
||||
}
|
||||
|
||||
fs := flag.NewFlagSet("config "+mode, flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var path, overlay string
|
||||
var path, overlay, pointer string
|
||||
registerConfigSelectionFlags(fs, &path, &overlay)
|
||||
registerConfigurationOverrides(fs)
|
||||
if mode == "explain" {
|
||||
fs.StringVar(&pointer, "path", "", "configuration path as a JSON Pointer")
|
||||
}
|
||||
if err := fs.Parse(args[1:]); err != nil {
|
||||
writeApplicationError(stderr, "config parse", err)
|
||||
return ExitRuntimeError
|
||||
@@ -41,6 +44,19 @@ func configCommand(args []string, stdout, stderr io.Writer) int {
|
||||
fmt.Fprintln(stderr, "config parse error: positional arguments are not accepted")
|
||||
return ExitRuntimeError
|
||||
}
|
||||
var explanationPath configkit.Path
|
||||
if mode == "explain" {
|
||||
if !flagWasSet(fs, "path") {
|
||||
fmt.Fprintln(stderr, "config parse error: --path is required")
|
||||
return ExitRuntimeError
|
||||
}
|
||||
var err error
|
||||
explanationPath, err = configkit.ParsePath(pointer)
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, "config parse error: --path must be a JSON Pointer")
|
||||
return ExitRuntimeError
|
||||
}
|
||||
}
|
||||
|
||||
result, err := appconfig.Load(appconfig.LoadOptions{
|
||||
Path: path, Explicit: flagWasSet(fs, "config"), Overlay: overlay,
|
||||
@@ -66,6 +82,8 @@ func configCommand(args []string, stdout, stderr io.Writer) int {
|
||||
})
|
||||
}
|
||||
err = json.NewEncoder(stdout).Encode(sources)
|
||||
case "explain":
|
||||
err = result.WriteExplanation(stdout, explanationPath)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "config output error: %v\n", err)
|
||||
|
||||
@@ -126,3 +126,82 @@ func TestConfigCommandFailuresDoNotProduceSuccessfulOutput(t *testing.T) {
|
||||
t.Fatalf("source writer error returned %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigExplainUsesResolvedHistoryAndKeepsParametersVisible(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := filepath.Join(dir, "config.yml")
|
||||
for path, body := range map[string]string{
|
||||
base: `prompt_dir: " ./prompts/../prompts "
|
||||
server:
|
||||
addr: :8080
|
||||
backends:
|
||||
"team/blue~1":
|
||||
endpoint: http://127.0.0.1:1/v1
|
||||
api_key_env: SCRIPTORIUM_EXPLAIN_KEY
|
||||
extra_params:
|
||||
"0": base-parameter
|
||||
`,
|
||||
filepath.Join(dir, "overlays", "local.yml"): `server:
|
||||
addr: :9090
|
||||
backends:
|
||||
"team/blue~1":
|
||||
extra_params:
|
||||
"0": overlay-parameter
|
||||
`,
|
||||
} {
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
const keyValue = "synthetic-key-bytes-must-not-appear"
|
||||
t.Setenv("SCRIPTORIUM_EXPLAIN_KEY", keyValue)
|
||||
args := []string{"--config", base, "--config-overlay", "local", "--addr", ":9090"}
|
||||
for _, tc := range []struct {
|
||||
pointer string
|
||||
want []string
|
||||
}{
|
||||
{pointer: "/server/addr", want: []string{":8080", ":9090", "--addr"}},
|
||||
{pointer: "/prompt_dir", want: []string{"normalized application path", "prompts"}},
|
||||
{pointer: "/backends/team~1blue~01/extra_params/0", want: []string{"base-parameter", "overlay-parameter"}},
|
||||
{pointer: "/backends/team~1blue~01/api_key_env", want: []string{"SCRIPTORIUM_EXPLAIN_KEY"}},
|
||||
{pointer: "", want: []string{"base-parameter", "overlay-parameter", "SCRIPTORIUM_EXPLAIN_KEY"}},
|
||||
} {
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", tc.pointer}, args...))
|
||||
if code != ExitOK || stderr != "" || strings.Contains(stdout, keyValue) {
|
||||
t.Fatalf("explain %q failed or disclosed key: code=%d out=%q err=%q", tc.pointer, code, stdout, stderr)
|
||||
}
|
||||
for _, want := range tc.want {
|
||||
if !strings.Contains(stdout, want) {
|
||||
t.Fatalf("explain %q lacks %q: %q", tc.pointer, want, stdout)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := os.Unsetenv("SCRIPTORIUM_EXPLAIN_KEY"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"explain", "--path", "/backends/team~1blue~01/api_key_env"}, args...))
|
||||
if code != ExitOK || stderr != "" || !strings.Contains(stdout, "SCRIPTORIUM_EXPLAIN_KEY") || strings.Contains(stdout, keyValue) {
|
||||
t.Fatalf("absent credential changed reference explanation: code=%d out=%q err=%q", code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigExplainRejectsUnknownOrMalformedPaths(t *testing.T) {
|
||||
path := writeAppConfigFile(t, "{}\n")
|
||||
for _, args := range [][]string{
|
||||
{"explain", "--config", path},
|
||||
{"explain", "--config", path, "--path", "not-a-pointer"},
|
||||
{"explain", "--config", path, "--path", "/missing"},
|
||||
} {
|
||||
code, stdout, stderr := runCLICommand(t, configCommand, args)
|
||||
if code != ExitRuntimeError || stdout != "" || stderr == "" {
|
||||
t.Fatalf("invalid explanation path accepted: args=%q code=%d out=%q err=%q", args, code, stdout, stderr)
|
||||
}
|
||||
}
|
||||
var stderr bytes.Buffer
|
||||
if code := configCommand([]string{"explain", "--config", path, "--path", ""}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") {
|
||||
t.Fatalf("explanation writer failure not reported: code=%d err=%q", code, stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -760,5 +760,5 @@ func printUsage(w io.Writer) {
|
||||
fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault)
|
||||
fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--prompt-version VERSION] [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--format text|json] [--out path]")
|
||||
fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--profile-dir DIR] [--format text|json] [--out path]")
|
||||
fmt.Fprintln(w, " config: scriptorium config <validate|show|sources> [--config PATH] [--config-overlay NAME] [application overrides]")
|
||||
fmt.Fprintln(w, " config: scriptorium config <validate|show|sources|explain> [--config PATH] [--config-overlay NAME] [application overrides] [--path /pointer for explain]")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user