Enable PromptKit repair attempts
This commit is contained in:
@@ -96,7 +96,7 @@ period, prompt version, timezone, and status. Successful output has an absolute
|
|||||||
"command": "generate",
|
"command": "generate",
|
||||||
"reportId": "today",
|
"reportId": "today",
|
||||||
"promptId": "weather.today_generated_text",
|
"promptId": "weather.today_generated_text",
|
||||||
"promptVersion": "2.0.0",
|
"promptVersion": "2.1.0",
|
||||||
"runId": "20260529T120000.000000000Z_today",
|
"runId": "20260529T120000.000000000Z_today",
|
||||||
"status": "succeeded",
|
"status": "succeeded",
|
||||||
"timezone": "America/Chicago",
|
"timezone": "America/Chicago",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Promptkit Integration
|
# Promptkit Integration
|
||||||
|
|
||||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.0.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.1.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
||||||
|
|
||||||
## Logical Profile Catalog
|
## Logical Profile Catalog
|
||||||
|
|
||||||
@@ -44,6 +44,13 @@ request bodies, response bodies, and complete parameter maps.
|
|||||||
|
|
||||||
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
||||||
|
|
||||||
|
Each embedded prompt permits one Promptkit-owned corrective generation after an
|
||||||
|
eligible failed or explicitly empty result. This is not an application retry:
|
||||||
|
Weatherreporter performs no provider retry, profile fallback, or request-level
|
||||||
|
output-contract override. Promptkit reports cumulative usage and the actual
|
||||||
|
number of corrective calls; repair exhaustion remains a completed validation
|
||||||
|
failure.
|
||||||
|
|
||||||
When capture is enabled, its preparation artifact projects a provider endpoint
|
When capture is enabled, its preparation artifact projects a provider endpoint
|
||||||
to its scheme and host and retains only reviewed execution settings. Provider
|
to its scheme and host and retains only reviewed execution settings. Provider
|
||||||
extras and URL user information, paths, queries, and fragments are omitted.
|
extras and URL user information, paths, queries, and fragments are omitted.
|
||||||
|
|||||||
@@ -56,6 +56,8 @@ directly.
|
|||||||
executes them.
|
executes them.
|
||||||
- Prompt and profile validation completes before weather collection. Raw output
|
- Prompt and profile validation completes before weather collection. Raw output
|
||||||
is validated before template rendering.
|
is validated before template rendering.
|
||||||
|
- PromptKit may make at most the prompt contract's one corrective generation;
|
||||||
|
exhaustion is a validation rejection, not an application-level retry.
|
||||||
- Comparison validates every explicit profile before collection, prepares one
|
- Comparison validates every explicit profile before collection, prepares one
|
||||||
immutable report input, and delegates backend capacity to Promptkit rather
|
immutable report input, and delegates backend capacity to Promptkit rather
|
||||||
than adding an application-wide execution limit.
|
than adding an application-wide execution limit.
|
||||||
|
|||||||
@@ -332,6 +332,8 @@ GOWORK=off go test -race -count=1 ./internal/app
|
|||||||
|
|
||||||
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
|
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
Purpose: switch the operational prompts to the accepted one-correction policy
|
Purpose: switch the operational prompts to the accepted one-correction policy
|
||||||
and make ordinary generate/run/batch output report what occurred.
|
and make ordinary generate/run/batch output report what occurred.
|
||||||
|
|
||||||
|
|||||||
@@ -126,11 +126,11 @@ func (client *fakeClient) allRequests() []promptkit.GenerateRequest {
|
|||||||
|
|
||||||
func TestInspectPromptAndProfile(t *testing.T) {
|
func TestInspectPromptAndProfile(t *testing.T) {
|
||||||
adapter := newTestAdapter(t, &fakeClient{})
|
adapter := newTestAdapter(t, &fakeClient{})
|
||||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.0.0")
|
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.1.0")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("InspectPrompt() error = %v", err)
|
t.Fatalf("InspectPrompt() error = %v", err)
|
||||||
}
|
}
|
||||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != "weather-balanced" {
|
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != "weather-balanced" || inspection.Output.RepairAttempts != 1 {
|
||||||
t.Fatalf("inspection = %#v", inspection)
|
t.Fatalf("inspection = %#v", inspection)
|
||||||
}
|
}
|
||||||
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
||||||
@@ -404,7 +404,7 @@ func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
request := promptexec.ExecuteRequest{
|
request := promptexec.ExecuteRequest{
|
||||||
PromptID: "weather.hourly_generated_text",
|
PromptID: "weather.hourly_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
ProfileID: "weather-light",
|
ProfileID: "weather-light",
|
||||||
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
||||||
}
|
}
|
||||||
@@ -832,7 +832,7 @@ func writeProfileFile(t *testing.T, profile string) string {
|
|||||||
func testExecuteRequest() promptexec.ExecuteRequest {
|
func testExecuteRequest() promptexec.ExecuteRequest {
|
||||||
return promptexec.ExecuteRequest{
|
return promptexec.ExecuteRequest{
|
||||||
PromptID: "weather.daily_generated_text",
|
PromptID: "weather.daily_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
ProfileID: "test-profile",
|
ProfileID: "test-profile",
|
||||||
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -140,7 +140,7 @@ type BatchReportResult struct {
|
|||||||
ModelName string `json:"modelName,omitempty"`
|
ModelName string `json:"modelName,omitempty"`
|
||||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
||||||
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
|
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
|
||||||
RepairAttempts *int `json:"-"`
|
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||||
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
||||||
OutputPath string `json:"outputPath,omitempty"`
|
OutputPath string `json:"outputPath,omitempty"`
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -170,7 +170,8 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
|
|||||||
e.mu.Lock()
|
e.mu.Lock()
|
||||||
profile := e.profiles[req.ProfileID]
|
profile := e.profiles[req.ProfileID]
|
||||||
e.mu.Unlock()
|
e.mu.Unlock()
|
||||||
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
|
definition := generationDefinitionForPrompt(req.PromptID)
|
||||||
|
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
|
||||||
e.callbackFailures <- err
|
e.callbackFailures <- err
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -89,7 +89,7 @@ func (e *generationExecutor) InspectPrompt(_ context.Context, id, version string
|
|||||||
return promptexec.PromptInspection{}, e.inspectErr
|
return promptexec.PromptInspection{}, e.inspectErr
|
||||||
}
|
}
|
||||||
definition := generationDefinitionForPrompt(id)
|
definition := generationDefinitionForPrompt(id)
|
||||||
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}}, nil
|
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}}, nil
|
||||||
}
|
}
|
||||||
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||||
generationExecutorMu.Lock()
|
generationExecutorMu.Lock()
|
||||||
@@ -113,7 +113,8 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
|||||||
calls = 1
|
calls = 1
|
||||||
}
|
}
|
||||||
for range calls {
|
for range calls {
|
||||||
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}
|
definition := generationDefinitionForPrompt(req.PromptID)
|
||||||
|
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}
|
||||||
if prepare != nil {
|
if prepare != nil {
|
||||||
prepare(&preparation)
|
prepare(&preparation)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -350,7 +350,7 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
|
|||||||
return promptexec.PromptInspection{
|
return promptexec.PromptInspection{
|
||||||
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
|
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
|
||||||
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
|
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
|
||||||
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
|
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -355,7 +355,7 @@ func comparisonResult(outputDirectory string, results []app.ComparisonProfileRes
|
|||||||
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
result := &app.ComparisonResult{
|
result := &app.ComparisonResult{
|
||||||
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
|
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
|
||||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||||
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
||||||
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
|
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
|
||||||
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),
|
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),
|
||||||
|
|||||||
@@ -132,7 +132,7 @@ func actionConfigPath(t *testing.T) string {
|
|||||||
func generatedReportResult() *app.ReportResult {
|
func generatedReportResult() *app.ReportResult {
|
||||||
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
return &app.ReportResult{
|
return &app.ReportResult{
|
||||||
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0",
|
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0",
|
||||||
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
|
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
|
||||||
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
|
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
|
||||||
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,
|
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ type generateSummary struct {
|
|||||||
ModelName string `json:"modelName,omitempty"`
|
ModelName string `json:"modelName,omitempty"`
|
||||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
||||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||||
|
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||||
Notification *generateNotificationSummary `json:"notification,omitempty"`
|
Notification *generateNotificationSummary `json:"notification,omitempty"`
|
||||||
Error string `json:"error,omitempty"`
|
Error string `json:"error,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -129,6 +130,10 @@ func newGenerateSummary(result *app.ReportResult, err error) generateSummary {
|
|||||||
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
|
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
|
||||||
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
||||||
summary.ValidationStatus = string(result.ValidationStatus)
|
summary.ValidationStatus = string(result.ValidationStatus)
|
||||||
|
if result.RepairAttempts != nil {
|
||||||
|
value := *result.RepairAttempts
|
||||||
|
summary.RepairAttempts = &value
|
||||||
|
}
|
||||||
summary.OutputPath = result.OutputPath
|
summary.OutputPath = result.OutputPath
|
||||||
summary.LLMDebugPath = result.LLMDebugPath
|
summary.LLMDebugPath = result.LLMDebugPath
|
||||||
summary.Notification = newGenerateNotificationSummary(result.Notification)
|
summary.Notification = newGenerateNotificationSummary(result.Notification)
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ import (
|
|||||||
|
|
||||||
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
|
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
|
||||||
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
|
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
|
||||||
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
|
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
|
||||||
t.Fatalf("summary = %#v", summary)
|
t.Fatalf("summary = %#v", summary)
|
||||||
}
|
}
|
||||||
@@ -39,7 +39,7 @@ func TestComparisonSummaryUsesLockedOrderAndSafeFields(t *testing.T) {
|
|||||||
profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
|
profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
|
||||||
result := &app.ComparisonResult{
|
result := &app.ComparisonResult{
|
||||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
|
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
|
||||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||||
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
||||||
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
|
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
|
||||||
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
||||||
@@ -113,7 +113,7 @@ func TestComparisonSummaryClassifiesCompleteAndAllFailedResults(t *testing.T) {
|
|||||||
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
failure := comparison.NewSafeError("generation", "execute prompt failed")
|
failure := comparison.NewSafeError("generation", "execute prompt failed")
|
||||||
complete := &app.ComparisonResult{
|
complete := &app.ComparisonResult{
|
||||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||||
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
|
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
|
||||||
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
||||||
Total: 2, Succeeded: 2,
|
Total: 2, Succeeded: 2,
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.daily_generated_text
|
id: weather.daily_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-balanced
|
default_profile: weather-balanced
|
||||||
description: Daily weather report analysis prompt.
|
description: Daily weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: daily.generated_text.schema.json
|
schema_path: daily.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.hourly_generated_text
|
id: weather.hourly_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-light
|
default_profile: weather-light
|
||||||
description: Hourly weather report analysis prompt.
|
description: Hourly weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: hourly.generated_text.schema.json
|
schema_path: hourly.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.today_generated_text
|
id: weather.today_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-balanced
|
default_profile: weather-balanced
|
||||||
description: Today's weather report analysis prompt.
|
description: Today's weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: today.generated_text.schema.json
|
schema_path: today.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.tomorrow_generated_text
|
id: weather.tomorrow_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-balanced
|
default_profile: weather-balanced
|
||||||
description: Tomorrow's weather report analysis prompt.
|
description: Tomorrow's weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: tomorrow.generated_text.schema.json
|
schema_path: tomorrow.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -77,8 +77,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
|||||||
if err := yaml.Unmarshal(data, &definition); err != nil {
|
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||||
t.Fatalf("decode prompt definition: %v", err)
|
t.Fatalf("decode prompt definition: %v", err)
|
||||||
}
|
}
|
||||||
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile {
|
if definition.ID != tc.id || definition.Version != "2.1.0" || definition.DefaultProfile != tc.profile {
|
||||||
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile)
|
t.Fatalf("definition = %#v, want %s version 2.1.0 and profile %s", definition, tc.id, tc.profile)
|
||||||
}
|
}
|
||||||
sharedInstruction := false
|
sharedInstruction := false
|
||||||
for _, message := range definition.Messages {
|
for _, message := range definition.Messages {
|
||||||
@@ -92,8 +92,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
|||||||
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
||||||
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
||||||
}
|
}
|
||||||
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts != nil {
|
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts == nil || *definition.Output.RepairAttempts != 1 {
|
||||||
t.Fatalf("output = %#v, want JSON schema output without repair attempts", definition.Output)
|
t.Fatalf("output = %#v, want JSON schema output with one repair attempt", definition.Output)
|
||||||
}
|
}
|
||||||
if _, err := promptassets.Schema(tc.schemaID); err != nil {
|
if _, err := promptassets.Schema(tc.schemaID); err != nil {
|
||||||
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
|
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
|
||||||
@@ -286,11 +286,11 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
|
|||||||
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
||||||
} {
|
} {
|
||||||
t.Run(want.id, func(t *testing.T) {
|
t.Run(want.id, func(t *testing.T) {
|
||||||
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.0.0")
|
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.1.0")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("InspectPrompt() error = %v", err)
|
t.Fatalf("InspectPrompt() error = %v", err)
|
||||||
}
|
}
|
||||||
if inspection.PromptID != want.id || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != want.profile {
|
if inspection.PromptID != want.id || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != want.profile || inspection.OutputContract.RepairAttempts != 1 {
|
||||||
t.Fatalf("inspection = %#v", inspection)
|
t.Fatalf("inspection = %#v", inspection)
|
||||||
}
|
}
|
||||||
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
||||||
@@ -414,7 +414,7 @@ func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "repair_attempts:", "weather.daily_report"} {
|
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "weather.daily_report"} {
|
||||||
if strings.Contains(string(data), unwanted) {
|
if strings.Contains(string(data), unwanted) {
|
||||||
t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
|
t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,10 +12,10 @@ func dailyDefinition() Definition {
|
|||||||
ID: Daily,
|
ID: Daily,
|
||||||
Name: "Daily Report",
|
Name: "Daily Report",
|
||||||
PromptID: "weather.daily_generated_text",
|
PromptID: "weather.daily_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "daily",
|
TemplateID: "daily",
|
||||||
GeneratedTextSchemaID: "daily",
|
GeneratedTextSchemaID: "daily",
|
||||||
GeneratedTextRepairAttempts: 0,
|
GeneratedTextRepairAttempts: 1,
|
||||||
ArtifactGroup: "daily",
|
ArtifactGroup: "daily",
|
||||||
OutputName: "daily.md",
|
OutputName: "daily.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
|
|||||||
@@ -14,10 +14,10 @@ func hourlyDefinition() Definition {
|
|||||||
ID: Hourly,
|
ID: Hourly,
|
||||||
Name: "Hourly Report",
|
Name: "Hourly Report",
|
||||||
PromptID: "weather.hourly_generated_text",
|
PromptID: "weather.hourly_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "hourly",
|
TemplateID: "hourly",
|
||||||
GeneratedTextSchemaID: "hourly",
|
GeneratedTextSchemaID: "hourly",
|
||||||
GeneratedTextRepairAttempts: 0,
|
GeneratedTextRepairAttempts: 1,
|
||||||
ArtifactGroup: "hourly",
|
ArtifactGroup: "hourly",
|
||||||
OutputName: "hourly.md",
|
OutputName: "hourly.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
|
|||||||
@@ -82,8 +82,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, definition := range definitions {
|
for _, definition := range definitions {
|
||||||
if definition.PromptVersion != "2.0.0" {
|
if definition.PromptVersion != "2.1.0" {
|
||||||
t.Fatalf("%s PromptVersion = %q, want 2.0.0", definition.ID, definition.PromptVersion)
|
t.Fatalf("%s PromptVersion = %q, want 2.1.0", definition.ID, definition.PromptVersion)
|
||||||
}
|
}
|
||||||
if definition.PromptID == "" {
|
if definition.PromptID == "" {
|
||||||
t.Fatalf("%s PromptID is empty", definition.ID)
|
t.Fatalf("%s PromptID is empty", definition.ID)
|
||||||
@@ -91,8 +91,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
|||||||
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
||||||
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
||||||
}
|
}
|
||||||
if definition.GeneratedTextRepairAttempts != 0 {
|
if definition.GeneratedTextRepairAttempts != 1 {
|
||||||
t.Fatalf("%s repair attempts = %d, want 0", definition.ID, definition.GeneratedTextRepairAttempts)
|
t.Fatalf("%s repair attempts = %d, want 1", definition.ID, definition.GeneratedTextRepairAttempts)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -10,10 +10,10 @@ func todayDefinition() Definition {
|
|||||||
ID: Today,
|
ID: Today,
|
||||||
Name: "Today Report",
|
Name: "Today Report",
|
||||||
PromptID: "weather.today_generated_text",
|
PromptID: "weather.today_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "today",
|
TemplateID: "today",
|
||||||
GeneratedTextSchemaID: "today",
|
GeneratedTextSchemaID: "today",
|
||||||
GeneratedTextRepairAttempts: 0,
|
GeneratedTextRepairAttempts: 1,
|
||||||
ArtifactGroup: "today",
|
ArtifactGroup: "today",
|
||||||
OutputName: "today.md",
|
OutputName: "today.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
|
|||||||
@@ -10,10 +10,10 @@ func tomorrowDefinition() Definition {
|
|||||||
ID: Tomorrow,
|
ID: Tomorrow,
|
||||||
Name: "Tomorrow Report",
|
Name: "Tomorrow Report",
|
||||||
PromptID: "weather.tomorrow_generated_text",
|
PromptID: "weather.tomorrow_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "tomorrow",
|
TemplateID: "tomorrow",
|
||||||
GeneratedTextSchemaID: "tomorrow",
|
GeneratedTextSchemaID: "tomorrow",
|
||||||
GeneratedTextRepairAttempts: 0,
|
GeneratedTextRepairAttempts: 1,
|
||||||
ArtifactGroup: "tomorrow",
|
ArtifactGroup: "tomorrow",
|
||||||
OutputName: "tomorrow.md",
|
OutputName: "tomorrow.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
|
|||||||
Reference in New Issue
Block a user