diff --git a/docs/integrations/scriptorium.md b/docs/integrations/scriptorium.md index 2dfe8aa..fbea7f8 100644 --- a/docs/integrations/scriptorium.md +++ b/docs/integrations/scriptorium.md @@ -6,7 +6,7 @@ This document describes the external Scriptorium CLI contract used by ## Purpose `weatherreporter` invokes Scriptorium as a subprocess to preflight prompt input -and generate Markdown reports. This page documents the CLI surface the adapter +and generate report artifacts. This page documents the CLI surface the adapter uses, not the full Scriptorium product. ## Commands Used @@ -29,11 +29,25 @@ scriptorium run \ --out ``` +Structured generated-text report generation uses the same command shape: + +```bash +scriptorium run \ + --prompt \ + --input data_package= \ + --out +``` + `weatherreporter` always passes prompt input as `--input data_package=`. The data package is structured YAML created by `internal/promptinput`; module snapshots remain separate JSON artifacts for inspection and Recent Changes. +For generated-text reports, Scriptorium selects the structured output schema +from the prompt configuration associated with the prompt ID. `weatherreporter` +does not pass `--format`, schema path, or JSON Schema flags for structured +generation. + ## Configured Arguments The adapter can prepend configured flags before prompt-specific arguments: @@ -68,11 +82,16 @@ Render results include: - exit code - truncation flags when applicable -Run results include the same fields plus the requested output path. +Run results include the same fields plus the requested output path. Structured +generated-text run results use the same captured fields and output-path +recording, with the output path pointing at the raw generated-text JSON +artifact. `weatherreporter` persists render preflight JSON when orchestration reaches the -preflight save point. The final Markdown artifact is written by Scriptorium to -the `--out` path. +preflight save point. Markdown report artifacts are written by Scriptorium to +the `--out` path. Generated-text raw JSON artifacts are also written by +Scriptorium to the `--out` path; later weatherreporter workflow steps validate +and render those bytes. ## Failure Behavior @@ -80,7 +99,7 @@ The adapter validates required request fields before starting Scriptorium: - prompt ID - data package path -- output path for `run` +- output path for `run` and structured generated-text `run` Nonzero exits return both the captured result and an error containing the exit code and stderr. A `run` exit code such as `2` is still treated as an error by diff --git a/docs/internal/scriptorium-adapter.md b/docs/internal/scriptorium-adapter.md index 53576ca..b7100b2 100644 --- a/docs/internal/scriptorium-adapter.md +++ b/docs/internal/scriptorium-adapter.md @@ -6,9 +6,9 @@ This document describes the subprocess adapter in ## Purpose The adapter runs `scriptorium render` for prompt preflight and `scriptorium run` -for Markdown report generation. It isolates subprocess execution, argv -construction, timeout handling, output capture, and exit-code interpretation -from app and domain packages. +for Markdown report generation or structured generated-text output. It isolates +subprocess execution, argv construction, timeout handling, output capture, and +exit-code interpretation from app and domain packages. ## Inputs And Outputs @@ -17,6 +17,7 @@ Inputs: - prompt ID - YAML prompt input data package path - report output path for `run` +- raw generated-text output path for structured `run` - configured binary, config path, profile, timeout, and extra arguments - context for cancellation @@ -27,6 +28,7 @@ Outputs: - truncation flags for captured output - exit code - report output path for `run` +- raw generated-text output path for structured `run` ## Boundaries @@ -34,9 +36,9 @@ Outputs: subprocess execution. It does not choose report types, build prompt input, fetch weather data, decide workflow order, or persist workflow metadata. -The adapter exposes request and result structs for render and run operations. -State persistence uses a state-owned preflight artifact shape; app -orchestration converts render results before saving. +The adapter exposes request and result structs for render, Markdown run, and +structured generated-text run operations. State persistence uses state-owned +artifact shapes; app orchestration converts adapter results before saving. ## Config Fields Used @@ -60,6 +62,12 @@ Report generation argv starts with: scriptorium run --prompt --input data_package= --out ``` +Structured generated-text argv uses the same `scriptorium run` form, with the +`--out` value set to the raw generated-text JSON artifact path. The adapter +does not add `--format`, schema path, or JSON Schema flags for structured +generation; Scriptorium selects the structured output schema from prompt +configuration. + Configured `--config` and `--profile` flags are inserted after the subcommand and before prompt-specific arguments. Extra arguments are appended after the built-in arguments. @@ -67,8 +75,8 @@ built-in arguments. ## Execution Behavior The adapter runs commands without shell interpolation. The same private -execution path is used by render and run after command-specific request -validation and argv construction. +execution path is used by render, Markdown run, and structured run after +command-specific request validation and argv construction. When `scriptorium.timeout` is greater than zero, each subprocess call uses a context with that timeout. Stdout and stderr are captured separately, capped at @@ -81,8 +89,8 @@ context with that timeout. Stdout and stderr are captured separately, capped at - Missing run output path returns an error before subprocess execution. - Subprocess start errors, context cancellation, and timeouts are wrapped with operation context by the caller-facing method. -- Nonzero render and run exits return the captured result plus an error - containing the exit code and stderr. +- Nonzero render, Markdown run, and structured run exits return the captured + result plus an error containing the exit code and stderr. ## Tests @@ -97,5 +105,6 @@ Inspect: - No shell interpolation is used. - The Scriptorium input name is `data_package`. - The file at the data package path is YAML produced by `internal/promptinput`. -- Render and run preserve command-specific result structs. +- Render, Markdown run, and structured run preserve command-specific result + structs. - Scriptorium-specific flags stay inside adapter and config boundaries. diff --git a/internal/adapters/scriptorium/runner.go b/internal/adapters/scriptorium/runner.go index 46d54a7..b197a8c 100644 --- a/internal/adapters/scriptorium/runner.go +++ b/internal/adapters/scriptorium/runner.go @@ -79,6 +79,12 @@ type RunRequest struct { OutputPath string } +type StructuredRunRequest struct { + PromptID string + DataPackagePath string + OutputPath string +} + type RenderResult struct { Command []string `json:"command"` Stdout string `json:"stdout"` @@ -98,6 +104,16 @@ type RunResult struct { OutputPath string `json:"outputPath"` } +type StructuredRunResult struct { + Command []string `json:"command"` + Stdout string `json:"stdout"` + Stderr string `json:"stderr"` + StdoutTruncated bool `json:"stdoutTruncated,omitempty"` + StderrTruncated bool `json:"stderrTruncated,omitempty"` + ExitCode int `json:"exitCode"` + OutputPath string `json:"outputPath"` +} + func (r Runner) Render(ctx context.Context, req RenderRequest) (*RenderResult, error) { if req.PromptID == "" { return nil, fmt.Errorf("prompt id is required") @@ -152,6 +168,35 @@ func (r Runner) Run(ctx context.Context, req RunRequest) (*RunResult, error) { return result, nil } +func (r Runner) StructuredRun(ctx context.Context, req StructuredRunRequest) (*StructuredRunResult, error) { + if req.PromptID == "" { + return nil, fmt.Errorf("prompt id is required") + } + if req.DataPackagePath == "" { + return nil, fmt.Errorf("data package path is required") + } + if req.OutputPath == "" { + return nil, fmt.Errorf("output path is required") + } + execution, err := r.execute(ctx, r.structuredRunArgs(req)) + if err != nil { + return nil, fmt.Errorf("run scriptorium structured output: %w", err) + } + result := &StructuredRunResult{ + Command: execution.argv(), + Stdout: string(execution.result.Stdout), + Stderr: string(execution.result.Stderr), + StdoutTruncated: execution.result.StdoutTruncated, + StderrTruncated: execution.result.StderrTruncated, + ExitCode: execution.result.ExitCode, + OutputPath: req.OutputPath, + } + if execution.result.ExitCode != 0 { + return result, fmt.Errorf("scriptorium structured run exited with code %d: %s", execution.result.ExitCode, result.Stderr) + } + return result, nil +} + type execution struct { binary string args []string @@ -212,6 +257,14 @@ func (r Runner) runArgs(req RunRequest) []string { return args } +func (r Runner) structuredRunArgs(req StructuredRunRequest) []string { + return r.runArgs(RunRequest{ + PromptID: req.PromptID, + DataPackagePath: req.DataPackagePath, + OutputPath: req.OutputPath, + }) +} + type limitedBuffer struct { data []byte limit int diff --git a/internal/adapters/scriptorium/runner_test.go b/internal/adapters/scriptorium/runner_test.go index e9fb0ad..de04e17 100644 --- a/internal/adapters/scriptorium/runner_test.go +++ b/internal/adapters/scriptorium/runner_test.go @@ -147,17 +147,169 @@ func TestRunReturnsResultForValidationExit(t *testing.T) { } } +func TestStructuredRunConstructsCommandWithoutSchemaFlags(t *testing.T) { + commands := &fakeCommands{result: CommandResult{ + Stdout: []byte(`{"summary":"ok"}`), + Stderr: []byte("wrote generated text"), + StdoutTruncated: true, + }} + runner := Runner{ + Binary: "/usr/local/bin/scriptorium", + ConfigPath: "/etc/scriptorium.yml", + Profile: "weather", + Timeout: 30 * time.Second, + Commands: commands, + } + + result, err := runner.StructuredRun(context.Background(), StructuredRunRequest{ + PromptID: "weather.hourly_generated_text", + DataPackagePath: "/tmp/hourly.data_package.yaml", + OutputPath: "/tmp/hourly.generated_text.raw.json", + }) + if err != nil { + t.Fatalf("StructuredRun() error = %v", err) + } + + wantArgs := []string{ + "run", + "--config", "/etc/scriptorium.yml", + "--profile", "weather", + "--prompt", "weather.hourly_generated_text", + "--input", "data_package=/tmp/hourly.data_package.yaml", + "--out", "/tmp/hourly.generated_text.raw.json", + } + if commands.name != "/usr/local/bin/scriptorium" { + t.Fatalf("command name = %q, want custom binary", commands.name) + } + if !reflect.DeepEqual(commands.args, wantArgs) { + t.Fatalf("args = %#v, want %#v", commands.args, wantArgs) + } + for _, disallowed := range []string{"--format", "--schema", "--schema-path", "--json-schema"} { + if containsArg(commands.args, disallowed) { + t.Fatalf("args = %#v, should not include %q", commands.args, disallowed) + } + } + if commands.timeout != 30*time.Second { + t.Fatalf("timeout = %s, want 30s", commands.timeout) + } + if !reflect.DeepEqual(result.Command, append([]string{"/usr/local/bin/scriptorium"}, wantArgs...)) { + t.Fatalf("result command = %#v, want full argv", result.Command) + } + if result.Stdout != `{"summary":"ok"}` || result.Stderr != "wrote generated text" || !result.StdoutTruncated { + t.Fatalf("result = %#v, want captured output and truncation flags", result) + } + if result.OutputPath != "/tmp/hourly.generated_text.raw.json" { + t.Fatalf("OutputPath = %q, want generated text raw path", result.OutputPath) + } +} + +func TestStructuredRunReturnsResultForNonzeroExit(t *testing.T) { + runner := Runner{ + Commands: &fakeCommands{ + result: CommandResult{ + Stdout: []byte(`{"summary":"partial"}`), + Stderr: []byte("structured output failed"), + ExitCode: 3, + }, + }, + } + + result, err := runner.StructuredRun(context.Background(), StructuredRunRequest{ + PromptID: "weather.hourly_generated_text", + DataPackagePath: "/tmp/hourly.data_package.yaml", + OutputPath: "/tmp/hourly.generated_text.raw.json", + }) + if err == nil { + t.Fatal("StructuredRun() error = nil, want nonzero exit error") + } + if result == nil { + t.Fatal("StructuredRun() result = nil, want captured result") + } + if result.ExitCode != 3 { + t.Fatalf("ExitCode = %d, want 3", result.ExitCode) + } + if result.Stdout != `{"summary":"partial"}` || result.OutputPath != "/tmp/hourly.generated_text.raw.json" { + t.Fatalf("result = %#v, want captured result fields", result) + } + if !strings.Contains(err.Error(), "structured output failed") { + t.Fatalf("error = %q, want stderr context", err.Error()) + } +} + +func TestStructuredRunValidatesRequiredFieldsBeforeExecution(t *testing.T) { + tests := []struct { + name string + req StructuredRunRequest + want string + }{ + { + name: "prompt id", + req: StructuredRunRequest{ + DataPackagePath: "/tmp/hourly.data_package.yaml", + OutputPath: "/tmp/hourly.generated_text.raw.json", + }, + want: "prompt id is required", + }, + { + name: "data package path", + req: StructuredRunRequest{ + PromptID: "weather.hourly_generated_text", + OutputPath: "/tmp/hourly.generated_text.raw.json", + }, + want: "data package path is required", + }, + { + name: "output path", + req: StructuredRunRequest{ + PromptID: "weather.hourly_generated_text", + DataPackagePath: "/tmp/hourly.data_package.yaml", + }, + want: "output path is required", + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + commands := &fakeCommands{} + runner := Runner{Commands: commands} + result, err := runner.StructuredRun(context.Background(), test.req) + if err == nil { + t.Fatal("StructuredRun() error = nil, want validation error") + } + if result != nil { + t.Fatalf("StructuredRun() result = %#v, want nil", result) + } + if !strings.Contains(err.Error(), test.want) { + t.Fatalf("StructuredRun() error = %v, want %q", err, test.want) + } + if commands.calls != 0 { + t.Fatalf("commands calls = %d, want no subprocess execution", commands.calls) + } + }) + } +} + type fakeCommands struct { name string args []string timeout time.Duration result CommandResult err error + calls int } func (f *fakeCommands) Run(_ context.Context, name string, args []string, timeout time.Duration) (CommandResult, error) { + f.calls++ f.name = name f.args = append([]string{}, args...) f.timeout = timeout return f.result, f.err } + +func containsArg(args []string, want string) bool { + for _, arg := range args { + if arg == want { + return true + } + } + return false +}