diff --git a/README.md b/README.md index f5757c9..dba5b72 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ a model. For complete invocation and output behavior, see the - [CLI reference](docs/cli.md) - [Configuration reference](docs/config.md) +- [Application configuration inspection](docs/cli.md#application-configuration-inspection) - [HTTP API reference](docs/api.md) - [Operations guide](docs/operations.md) - [Consumer integration overview](docs/consumers/api.md) @@ -44,7 +45,8 @@ a model. For complete invocation and output behavior, see the ## Examples - [Minimal configuration](examples/config.yml) and - [complete configuration](examples/config.full.yml) + [complete configuration](examples/config.full.yml), plus a + [composed import/overlay example](examples/config.composed.yml) - [Prompt definitions](examples/prompts/), [execution profiles](examples/profiles/), [schemas](examples/schemas/), and [synthetic input fixtures](examples/fixtures/) - [Render script](examples/render-markdown-summary.sh) diff --git a/docs/cli.md b/docs/cli.md index be896ce..d85708f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -25,6 +25,9 @@ go run ./cmd/scriptorium render \ - `scriptorium serve`: start the HTTP server. - `scriptorium inspect prompt`: inspect one prompt definition without model execution. - `scriptorium inspect profile`: inspect one effective profile without model execution. +- `scriptorium config validate`: validate application configuration without building an engine. +- `scriptorium config show`: show the resolved application configuration. +- `scriptorium config sources`: show the ordered configuration sources. All commands accept `--config ` and `--config-overlay ` and reject positional arguments. `--config-overlay` selects one application configuration @@ -217,6 +220,37 @@ resolves profile inheritance and backend defaults, but never reads a credential value, loads a prompt, reserves capacity, or calls a model. Unset provider controls are shown as zero values when Promptkit leaves them unspecified. +## Application configuration inspection + +```text +scriptorium config validate [flags] +scriptorium config show [flags] +scriptorium config sources [flags] +``` + +Each accepts `--config`, `--config-overlay`, `--prompt-dir`, `--profile-dir`, +`--schema-dir`, `--addr`, `--artifact-root`, `--max-request-bytes`, +`--max-artifact-bytes`, and `--max-response-bytes`. These are application +configuration selectors and overrides; request, credential, backend-editing, +output-format, and output-file flags are not accepted. Positional arguments +are rejected. + +`validate` writes no success output; status 0 means the application configuration +loaded and passed its own validation. It does not establish that a prompt +directory exists, credentials are present, or Promptkit can construct an +engine. `show` writes ConfigKit's resolved, redacted YAML snapshot. `sources` +writes one newline-terminated JSON array of source records in load order. Each +record has `id`, `label`, and `layer`, plus `path` when available; explicit CLI +overrides appear as synthetic sources. Defaults-only input without overrides +produces `[]` for `sources`. + +Diagnostics and warnings go to stderr, never into successful stdout. Parse, +load, output, and diagnostic-write failures exit 1. Ordinary output writes may +leave partial data on failure. The inspection output can contain paths, +identifiers, `api_key_env` reference names, and ordinary `extra_params` values; +protect it accordingly. Raw API-key values are not read or emitted by these +commands. + ## Input And Variable Syntax `--input name=path` maps an input name to a local file; `--var name=value` diff --git a/docs/config.md b/docs/config.md index 463cbe2..b2de6ee 100644 --- a/docs/config.md +++ b/docs/config.md @@ -95,6 +95,15 @@ The size fields must be zero or greater. The [HTTP API](api.md) defines how each limit is enforced and reported. `server.artifact_root` configures an HTTP deployment boundary; see [operations](operations.md) for deployment handling. +The `config validate`, `config show`, and `config sources` commands inspect this +application configuration without creating a Promptkit engine. `show` is a +resolved inspection snapshot, not necessarily a reloadable file. Application +validation does not check framework sources, backend connectivity, or credential +availability. `api_key_env` is an environment-variable *name* and remains +visible in inspection; its environment value is never read by ConfigKit. +Ordinary backend `extra_params` values also remain visible, so keep secrets out +of those fields and protect inspection output as potentially sensitive. + ## Custom Backends Use `backends` when a profile selects an application-defined backend ID: diff --git a/docs/integrations/subprocess.md b/docs/integrations/subprocess.md index 6a0e3f8..530a11d 100644 --- a/docs/integrations/subprocess.md +++ b/docs/integrations/subprocess.md @@ -17,6 +17,15 @@ Pass required API-key environment variables through the child environment. Do not place raw API keys in arguments. Keep the environment limited to the values needed for the selected profile. +For application-only preflight, invoke `scriptorium config validate`; it exits +0 with empty stdout when the application configuration loads, but does not +prove that Promptkit sources or credentials are usable. `config show` emits +resolved YAML and `config sources` emits newline-terminated JSON provenance. +All three use the same source selection and explicit overrides as execution +commands and require separate stderr capture for diagnostics. Their output can +include source paths, identifiers, credential reference names, and ordinary +backend parameters; treat it as potentially sensitive. + ## Streams And Output Ownership Capture stdout and stderr separately. Stdout contains the requested artifact or diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 238c584..1f2774a 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -7,9 +7,9 @@ public behavior belongs in the linked contracts. | Component | Implemented responsibility | References | | --- | --- | --- | | `cmd/scriptorium` | Process entrypoint that delegates arguments and streams to the CLI adapter. | [CLI contract](../cli.md), [adapter internals](adapters.md) | -| `internal/adapter/cli` | Parses commands, resolves application settings, constructs Promptkit engines, maps requests, and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md) | +| `internal/adapter/cli` | Parses execution and application-configuration inspection commands, resolves settings, constructs Promptkit engines only for execution/definition inspection, maps requests, and owns process output and exit behavior. | [CLI contract](../cli.md), [adapter internals](adapters.md) | | `internal/adapter/http` | Owns routes, DTOs, strict decoding, limits, Promptkit request/result mapping, public error mapping, and restricted HTTP artifact reading. | [HTTP API](../api.md), [adapter internals](adapters.md), [source internals](sources.md) | -| `internal/config` | Discovers and strictly decodes application configuration and applies built-in and CLI precedence. | [configuration contract](../config.md), [adapter internals](adapters.md) | +| `internal/config` | Selects application sources, supplies the ConfigKit definition and explicit overrides, and converts resolved settings for runtime use. | [configuration contract](../config.md), [adapter internals](adapters.md) | | `internal/defaults` | Holds Scriptorium-owned application and HTTP defaults. | [configuration contract](../config.md) | | `internal/format` | Formats Promptkit prepared-run and prompt/profile inspection values for CLI text or JSON output. | [CLI contract](../cli.md), [adapter internals](adapters.md) | | `internal/report` | Projects one terminal Promptkit observation into Scriptorium-owned execution metadata. | [adapter internals](adapters.md) | diff --git a/docs/operations.md b/docs/operations.md index b83387d..0884bc2 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -87,14 +87,22 @@ and apply the same retention and access rules as for model input and output. Before changing production inputs, profiles, or schemas: -1. confirm the deployed configuration selects the intended sources and model +1. use `config validate`, `config show`, or `config sources` to check the + application configuration and its selected sources without constructing an + engine; this does not verify prompt files, provider access, or credentials; +2. confirm the deployed configuration selects the intended sources and model credentials; -2. use [`render`](cli.md) with the same request inputs and variables to confirm +3. use [`render`](cli.md) with the same request inputs and variables to confirm preparation without a model call; -3. use [`run`](cli.md) for generation; and -4. retain or discard validation-failed output according to the caller's +4. use [`run`](cli.md) for generation; and +5. retain or discard validation-failed output according to the caller's policy. +Inspection output can include configuration paths, names, and ordinary backend +parameters. Apply +access and retention controls; the application stores only API-key environment +reference names, not the resolved credential bytes. + The [maintained render script](../examples/render-markdown-summary.sh) is a copyable preflight example. The CLI reference owns its complete invocation and exit semantics. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 2132915..cad7e4f 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -320,7 +320,11 @@ new scanning, remote acquisition, or path policy is introduced. ## Stage 5: Add Application Validation, Effective Output, And Source Inspection -**Status: Not started.** +**Status: Complete.** Application-only validation, effective YAML, and ordered +JSON source inspection use one shared ConfigKit result without an engine. Tests +cover source composition, CLI provenance, stream failures, and the credential +reference boundary. Public and internal contracts are updated; full test, +race, vet, build, and diff checks passed. Primary files: a focused `internal/adapter/cli/config.go`, shared config definition, CLI dispatch/help/tests, and canonical CLI/config documentation. diff --git a/internal/adapter/cli/config.go b/internal/adapter/cli/config.go new file mode 100644 index 0000000..cef4635 --- /dev/null +++ b/internal/adapter/cli/config.go @@ -0,0 +1,86 @@ +package cli + +import ( + "encoding/json" + "flag" + "fmt" + "io" + + configkit "gitea.maximumdirect.net/eric/ConfigKit" + appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config" +) + +type configurationSource struct { + ID string `json:"id"` + Label string `json:"label"` + Layer configkit.Layer `json:"layer"` + Path string `json:"path,omitempty"` +} + +func configCommand(args []string, stdout, stderr io.Writer) int { + if len(args) == 0 { + fmt.Fprintln(stderr, "config parse error: configuration mode is required") + return ExitRuntimeError + } + mode := args[0] + if mode != "validate" && mode != "show" && mode != "sources" { + fmt.Fprintln(stderr, "config parse error: unknown configuration mode") + return ExitRuntimeError + } + + fs := flag.NewFlagSet("config "+mode, flag.ContinueOnError) + fs.SetOutput(io.Discard) + var path, overlay string + registerConfigSelectionFlags(fs, &path, &overlay) + registerConfigurationOverrides(fs) + if err := fs.Parse(args[1:]); err != nil { + writeApplicationError(stderr, "config parse", err) + return ExitRuntimeError + } + if fs.NArg() != 0 { + fmt.Fprintln(stderr, "config parse error: positional arguments are not accepted") + return ExitRuntimeError + } + + result, err := appconfig.Load(appconfig.LoadOptions{ + Path: path, Explicit: flagWasSet(fs, "config"), Overlay: overlay, + Overrides: applicationOverrides(fs), + }) + if err != nil { + writeApplicationError(stderr, "config", err) + return ExitRuntimeError + } + if err := writeApplicationWarnings(stderr, result.Diagnostics()); err != nil { + return ExitRuntimeError + } + switch mode { + case "validate": + return ExitOK + case "show": + err = result.WriteEffective(stdout) + case "sources": + sources := make([]configurationSource, 0, len(result.Sources())) + for _, source := range result.Sources() { + sources = append(sources, configurationSource{ + ID: source.ID, Label: source.Label, Layer: source.Layer, Path: source.Path, + }) + } + err = json.NewEncoder(stdout).Encode(sources) + } + if err != nil { + fmt.Fprintf(stderr, "config output error: %v\n", err) + return ExitRuntimeError + } + return ExitOK +} + +func registerConfigurationOverrides(fs *flag.FlagSet) { + fs.String("prompt-dir", "", "prompt-definition directory override") + fs.String("profile-dir", "", "profile directory override") + fs.String("schema-dir", "", "schema directory override") + fs.String("addr", "", "HTTP listen address override") + fs.String("artifact-root", "", "HTTP artifact root override") + fs.Int64("max-request-bytes", 0, "HTTP request byte limit override") + fs.Int64("max-artifact-bytes", 0, "HTTP artifact byte limit override") + fs.Int64("max-response-bytes", 0, "HTTP response byte limit override") +} diff --git a/internal/adapter/cli/config_test.go b/internal/adapter/cli/config_test.go new file mode 100644 index 0000000..fc9229c --- /dev/null +++ b/internal/adapter/cli/config_test.go @@ -0,0 +1,128 @@ +package cli + +import ( + "bytes" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + appconfig "gitea.maximumdirect.net/eric/scriptorium/internal/config" +) + +func TestConfigCommandsInspectApplicationWithoutEngine(t *testing.T) { + path := writeAppConfigFile(t, "{}\n") + for _, mode := range []string{"validate", "show", "sources"} { + code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path}) + if code != ExitOK || stderr != "" { + t.Fatalf("%s failed: code=%d stdout=%q stderr=%q", mode, code, stdout, stderr) + } + switch mode { + case "validate": + if stdout != "" { + t.Fatalf("validation wrote stdout: %q", stdout) + } + case "show": + if !strings.Contains(stdout, "render_format: text") || !strings.Contains(stdout, "max_request_bytes: 16777216") { + t.Fatalf("effective defaults missing: %q", stdout) + } + case "sources": + var sources []configurationSource + if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 1 || sources[0].Layer != "base" { + t.Fatalf("unexpected source records: %q (%v)", stdout, err) + } + } + } +} + +func TestConfigCommandsExposeComposedSourcesAndResolvedValues(t *testing.T) { + dir := t.TempDir() + if err := os.MkdirAll(filepath.Join(dir, "overlays"), 0o755); err != nil { + t.Fatal(err) + } + base := filepath.Join(dir, "config.yml") + for path, body := range map[string]string{ + base: "configkit:\n imports:\n - path: fragment.yml\nprompt_dir: ' ./before/../prompts '\nprofile_dir: ./profiles\n", + filepath.Join(dir, "fragment.yml"): "server:\n addr: :8081\n", + filepath.Join(dir, "overlays", "local.yml"): "server:\n addr: :9090\n", + } { + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatal(err) + } + } + args := []string{"--config", base, "--config-overlay", "local", "--profile-dir=", "--addr", ":7070"} + code, stdout, stderr := runCLICommand(t, configCommand, append([]string{"show"}, args...)) + if code != ExitOK || stderr != "" || !strings.Contains(stdout, "prompt_dir: prompts") || !strings.Contains(stdout, "profile_dir: ''") || !strings.Contains(stdout, "addr: :7070") { + t.Fatalf("resolved effective output changed: code=%d out=%q err=%q", code, stdout, stderr) + } + code, stdout, stderr = runCLICommand(t, configCommand, append([]string{"sources"}, args...)) + if code != ExitOK || stderr != "" || !strings.HasSuffix(stdout, "\n") { + t.Fatalf("source command failed: code=%d out=%q err=%q", code, stdout, stderr) + } + var sources []configurationSource + if err := json.Unmarshal([]byte(stdout), &sources); err != nil || len(sources) != 5 { + t.Fatalf("unexpected composed source records: %+v (%v)", sources, err) + } + for i, layer := range []string{"base", "base", "overlay", "cli", "cli"} { + if string(sources[i].Layer) != layer { + t.Fatalf("source %d layer = %q, want %q", i, sources[i].Layer, layer) + } + } +} + +func TestConfigInspectionKeepsReferencesAndOrdinaryParameters(t *testing.T) { + const name = "SCRIPTORIUM_CONFIG_TEST_API_KEY" + const secret = "synthetic-secret-value-should-not-appear" + path := writeAppConfigFile(t, `backends: + synthetic: + endpoint: http://127.0.0.1:1/v1 + api_key_env: SCRIPTORIUM_CONFIG_TEST_API_KEY + extra_params: + visible_parameter: synthetic-ordinary-value +`) + t.Setenv(name, secret) + for _, present := range []bool{true, false} { + if !present { + if err := os.Unsetenv(name); err != nil { + t.Fatal(err) + } + } + code, stdout, stderr := runCLICommand(t, configCommand, []string{"show", "--config", path}) + if code != ExitOK || stderr != "" || !strings.Contains(stdout, name) || !strings.Contains(stdout, "synthetic-ordinary-value") || strings.Contains(stdout, secret) { + t.Fatalf("credential boundary changed with present=%t: code=%d out=%q err=%q", present, code, stdout, stderr) + } + } + result, err := appconfig.Load(appconfig.LoadOptions{Path: path, Explicit: true}) + if err != nil { + t.Fatal(err) + } + settings := appconfig.Settings(result.Config) + if len(settings.Backends) != 1 || settings.Backends[0].ExtraParams["visible_parameter"] != "synthetic-ordinary-value" { + t.Fatalf("runtime backend parameters changed: %+v", settings.Backends) + } +} + +type errorOutputWriter struct{} + +func (errorOutputWriter) Write([]byte) (int, error) { return 0, errors.New("output unavailable") } + +func TestConfigCommandFailuresDoNotProduceSuccessfulOutput(t *testing.T) { + path := writeAppConfigFile(t, "server:\n max_request_bytes: -1\n") + for _, mode := range []string{"validate", "show", "sources"} { + code, stdout, stderr := runCLICommand(t, configCommand, []string{mode, "--config", path}) + if code != ExitRuntimeError || stdout != "" || !strings.Contains(stderr, "validate.error") { + t.Fatalf("%s misclassified invalid config: code=%d out=%q err=%q", mode, code, stdout, stderr) + } + } + path = writeAppConfigFile(t, "{}\n") + var stderr bytes.Buffer + if code := configCommand([]string{"show", "--config", path}, errorOutputWriter{}, &stderr); code != ExitRuntimeError || !strings.Contains(stderr.String(), "output error") { + t.Fatalf("writer error not reported: code=%d stderr=%q", code, stderr.String()) + } + if code := configCommand([]string{"sources", "--config", path}, errorOutputWriter{}, io.Discard); code != ExitRuntimeError { + t.Fatalf("source writer error returned %d", code) + } +} diff --git a/internal/adapter/cli/run.go b/internal/adapter/cli/run.go index bdce241..12348ce 100644 --- a/internal/adapter/cli/run.go +++ b/internal/adapter/cli/run.go @@ -137,6 +137,8 @@ func Run(args []string, stdout, stderr io.Writer) int { return serveCommand(args[1:], stderr) case "inspect": return inspectCommand(args[1:], stdout, stderr) + case "config": + return configCommand(args[1:], stdout, stderr) default: fmt.Fprintf(stderr, "unknown command %q\n", args[0]) printUsage(stderr) @@ -752,10 +754,11 @@ func runErrorMessage(err error) string { } func printUsage(w io.Writer) { - fmt.Fprintln(w, "usage: scriptorium ...") + fmt.Fprintln(w, "usage: scriptorium ...") fmt.Fprintln(w, " run: scriptorium run [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] --prompt ID [--prompt-version VERSION] [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--out path] [--execution-report path] [--timeout 10m] [--run-timeout 10m] [--transient-retries N]") fmt.Fprintln(w, " render: scriptorium render [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] --prompt ID [--prompt-version VERSION] [--input name=path] [--profile ID] [--session-id ID] [--llm-base-url URL] [--model NAME] [--api-key-env ENV] [--temperature N] [--max-tokens N] [--top-p N] [--reasoning-effort VALUE] [--var k=v] [--format text|json] [--out path] [--timeout 10m] [--transient-retries N]") fmt.Fprintf(w, " serve: scriptorium serve [--config PATH] [--config-overlay NAME] [--addr %s] [--prompt-dir DIR] [--profile-dir DIR] [--schema-dir DIR] [--artifact-root DIR] [--max-request-bytes N] [--max-artifact-bytes N] [--max-response-bytes N]\n", defaults.HTTPAddrDefault) fmt.Fprintln(w, " inspect prompt: scriptorium inspect prompt --prompt ID [--prompt-version VERSION] [--config PATH] [--config-overlay NAME] [--prompt-dir DIR] [--format text|json] [--out path]") fmt.Fprintln(w, " inspect profile: scriptorium inspect profile --profile ID [--config PATH] [--config-overlay NAME] [--profile-dir DIR] [--format text|json] [--out path]") + fmt.Fprintln(w, " config: scriptorium config [--config PATH] [--config-overlay NAME] [application overrides]") } diff --git a/internal/config/load_test.go b/internal/config/load_test.go index 783e180..3ec4cd2 100644 --- a/internal/config/load_test.go +++ b/internal/config/load_test.go @@ -93,6 +93,10 @@ func TestLoadDefaultsOnlyAndOverridesBeforeValidation(t *testing.T) { if err != nil || !reflect.DeepEqual(result.Config, documentDefaults()) || len(result.Sources()) != 0 { t.Fatalf("defaults-only load: result=%+v err=%v", result, err) } + var effective strings.Builder + if err := result.WriteEffective(&effective); err != nil || !strings.Contains(effective.String(), "render_format: text") || len(result.Sources()) != 0 { + t.Fatalf("defaults-only inspection changed: effective=%q err=%v sources=%+v", effective.String(), err, result.Sources()) + } result, err = Load(LoadOptions{Overrides: []configkit.Override{{ Path: configkit.Keys("server", "max_request_bytes"), Value: int64(0), Flag: "--max-request-bytes",