eric/weatherreporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Carry repair provenance through application workflows

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-25 19:46:56 +00:00
parent 20107b0dfd
commit ce79ea92c5
16 changed files with 101 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/internal/app-orchestration.md
View File

@@ -7,7 +7,7 @@ is owned by the [CLI reference](../cli.md) and [operations guide](../operations.
## Single-Report Flow ## Single-Report Flow
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, and model are carried in the active result. `GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, model, and actual repair count (once a completed execution exists) are carried in the active result; the configured repair budget remains part of the exact prompt contract.
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run. The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.

5
docs/internal/prepared-report.md
View File

@@ -21,8 +21,9 @@ Before accepting generated JSON, the execution boundary reconciles the prepared
report definition, inspected prompt hash and selected profile identity, the one report definition, inspected prompt hash and selected profile identity, the one
preparation callback, and the completed Promptkit result. The callback and preparation callback, and the completed Promptkit result. The callback and
completion must agree on prompt, profile, backend, model, and rendered/input completion must agree on prompt, profile, backend, model, and rendered/input
hashes; the callback output and completed validation must name the prepared hashes; the callback output must also carry the prepared report's configured
report's JSON Schema. A mismatch produces no rendered Markdown and leaves repair budget, while the completed validation records the actual corrective
calls used within that budget. A mismatch produces no rendered Markdown and leaves
results with only the inspected safe identity. results with only the inspected safe identity.
Single-report generation executes one prepared profile and publishes its Single-report generation executes one prepared profile and publishes its

2
docs/roadmap/implementation.md
View File

@@ -286,6 +286,8 @@ GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
### Stage 5: Carry Repair Provenance Through Application Workflows ### Stage 5: Carry Repair Provenance Through Application Workflows
Status: Complete.
Purpose: make application orchestration understand configured and actual repair Purpose: make application orchestration understand configured and actual repair
counts before changing the embedded prompt policy. counts before changing the embedded prompt policy.

5
internal/app/app.go
View File

@@ -89,6 +89,7 @@ type ReportResult struct {
ModelName string ModelName string
SourceWarnings []weatherdata.SourceWarning SourceWarnings []weatherdata.SourceWarning
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string LLMDebugPath string
OutputPath string OutputPath string
Notification *NotificationResult Notification *NotificationResult
@@ -139,6 +140,7 @@ type BatchReportResult struct {
ModelName string `json:"modelName,omitempty"` ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"` SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"` ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"-"`
LLMDebugPath string `json:"llmDebugPath,omitempty"` LLMDebugPath string `json:"llmDebugPath,omitempty"`
OutputPath string `json:"outputPath,omitempty"` OutputPath string `json:"outputPath,omitempty"`
} }
@@ -457,6 +459,9 @@ func copyBatchReportDetails(item *BatchReportResult, result *ReportResult) {
item.Timezone = result.Timezone item.Timezone = result.Timezone
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...) item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
item.ValidationStatus = result.ValidationStatus item.ValidationStatus = result.ValidationStatus
if result.RepairAttempts != nil {
item.RepairAttempts = repairAttemptsPointer(*result.RepairAttempts)
}
} }
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) { func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {

4
internal/app/comparison_execution.go
View File

@@ -31,6 +31,7 @@ type comparisonProfileOutcome struct {
ModelName string ModelName string
Status string Status string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string ReportPath string
Markdown []byte Markdown []byte
LLMDebugPath string LLMDebugPath string
@@ -108,6 +109,9 @@ func executeComparisonProfile(ctx context.Context, req comparisonExecutionReques
}) })
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
outcome.ValidationStatus = execution.ValidationStatus outcome.ValidationStatus = execution.ValidationStatus
if execution.RepairAttempts != nil {
outcome.RepairAttempts = repairAttemptsPointer(*execution.RepairAttempts)
}
outcome.LLMDebugPath = execution.LLMDebugPath outcome.LLMDebugPath = execution.LLMDebugPath
if err != nil { if err != nil {
outcome.canceled = cancellationError(err) outcome.canceled = cancellationError(err)

4
internal/app/generation_test.go
View File

@@ -68,6 +68,7 @@ type generationExecutor struct {
beforeExecute func(promptexec.ExecuteRequest) beforeExecute func(promptexec.ExecuteRequest)
cancelBeforeReturn context.CancelFunc cancelBeforeReturn context.CancelFunc
validation promptexec.ValidationStatus validation promptexec.ValidationStatus
repairAttempts int
validations map[string]promptexec.ValidationStatus validations map[string]promptexec.ValidationStatus
rawOutput []byte rawOutput []byte
waitForCancellation map[string]bool waitForCancellation map[string]bool
@@ -128,6 +129,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
profileErr := e.executeErrors[req.ProfileID] profileErr := e.executeErrors[req.ProfileID]
executeErr := e.executeErr executeErr := e.executeErr
status := e.validation status := e.validation
repairAttempts := e.repairAttempts
if profileStatus, ok := e.validations[req.ProfileID]; ok { if profileStatus, ok := e.validations[req.ProfileID]; ok {
status = profileStatus status = profileStatus
} }
@@ -162,7 +164,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
if cancelBeforeReturn != nil { if cancelBeforeReturn != nil {
cancelBeforeReturn() cancelBeforeReturn()
} }
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", 0, nil)} execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil)}
if complete != nil { if complete != nil {
complete(execution) complete(execution)
} }

11
internal/app/profile_execution.go
View File

@@ -24,6 +24,7 @@ type profileExecutionOutcome struct {
BackendID string BackendID string
ModelName string ModelName string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string LLMDebugPath string
} }
@@ -99,6 +100,7 @@ func executePreparedProfile(ctx context.Context, req profileExecutionRequest) (p
if execution == nil { if execution == nil {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
} }
outcome.RepairAttempts = repairAttemptsPointer(execution.Validation.RepairAttempts)
if preparationCount != 1 { if preparationCount != 1 {
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()} return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
} }
@@ -178,12 +180,19 @@ func validateExecutionProvenance(req profileExecutionRequest, preparation prompt
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash || if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) || execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName || execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" { execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" ||
execution.Validation.RepairAttempts < 0 || execution.Validation.RepairAttempts > preparation.Output.RepairAttempts ||
preparation.Output.RepairAttempts != definition.GeneratedTextRepairAttempts {
return promptProvenanceError() return promptProvenanceError()
} }
return nil return nil
} }
func repairAttemptsPointer(value int) *int {
copy := value
return &copy
}
func promptProvenanceError() error { func promptProvenanceError() error {
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil) return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
} }

16
internal/app/profile_execution_test.go
View File

@@ -26,7 +26,7 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("executePreparedProfile() error = %v", err) t.Fatalf("executePreparedProfile() error = %v", err)
} }
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 { if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 0 || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls) t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
} }
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) { if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
@@ -34,6 +34,20 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
} }
} }
func TestExecutePreparedProfileRetainsCompletedRepairAttemptsOnLaterFailure(t *testing.T) {
prepared, inspection := preparedDailyProfile(t)
prepared.resolved.Definition.GeneratedTextRepairAttempts = 1
executor := &generationExecutor{repairAttempts: 1, rawOutput: []byte(`{"summary":42}`), prepare: func(value *promptexec.Preparation) { value.Output.RepairAttempts = 1 }}
outcome, _, err := executePreparedProfile(context.Background(), profileExecutionRequest{
Prepared: prepared, Prompt: inspection,
Profile: promptexec.ProfileInspection{ProfileID: inspection.ProfileID, BackendID: inspection.BackendID, ModelName: inspection.ModelName},
Executor: executor,
})
if err == nil || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 1 {
t.Fatalf("outcome/error = %#v/%v", outcome, err)
}
}
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) { func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
prepared, inspection := preparedDailyProfile(t) prepared, inspection := preparedDailyProfile(t)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir()) debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())

3
internal/app/prompt_generate.go
View File

@@ -48,6 +48,9 @@ func generatePromptReport(ctx context.Context, req promptReportRequest) (*Report
}) })
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
result.ValidationStatus = outcome.ValidationStatus result.ValidationStatus = outcome.ValidationStatus
if outcome.RepairAttempts != nil {
result.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
result.LLMDebugPath = outcome.LLMDebugPath result.LLMDebugPath = outcome.LLMDebugPath
if err != nil { if err != nil {
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err) return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)

2
internal/app/prompt_inspection.go
View File

@@ -207,7 +207,7 @@ func validPromptInput(inputs []promptexec.InputDefinition) bool {
} }
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool { func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" && output.RepairAttempts == definition.GeneratedTextRepairAttempts
} }
func promptInspectionError(operation string, err error) error { func promptInspectionError(operation string, err error) error {

17
internal/report/daily_report.go
View File

@@ -9,14 +9,15 @@ import (
func dailyDefinition() Definition { func dailyDefinition() Definition {
return Definition{ return Definition{
ID: Daily, ID: Daily,
Name: "Daily Report", Name: "Daily Report",
PromptID: "weather.daily_generated_text", PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.0.0",
TemplateID: "daily", TemplateID: "daily",
GeneratedTextSchemaID: "daily", GeneratedTextSchemaID: "daily",
ArtifactGroup: "daily", GeneratedTextRepairAttempts: 0,
OutputName: "daily.md", ArtifactGroup: "daily",
OutputName: "daily.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",

29
internal/report/definition.go
View File

@@ -27,20 +27,21 @@ const (
) )
type Definition struct { type Definition struct {
ID ID ID ID
Name string Name string
PromptID string PromptID string
PromptVersion string PromptVersion string
TemplateID string TemplateID string
GeneratedTextSchemaID string GeneratedTextSchemaID string
ArtifactGroup string GeneratedTextRepairAttempts int
OutputName string ArtifactGroup string
DistributorPathTemplates []string OutputName string
Modules []module.ConfigItem DistributorPathTemplates []string
Morning bool Modules []module.ConfigItem
Evening bool Morning bool
resolve func(ResolveRequest) (timeutil.Period, error) Evening bool
runIDDisambiguator func(Resolved) string resolve func(ResolveRequest) (timeutil.Period, error)
runIDDisambiguator func(Resolved) string
} }
func (r Resolved) OutputName() (string, error) { func (r Resolved) OutputName() (string, error) {

17
internal/report/hourly_report.go
View File

@@ -11,14 +11,15 @@ const hourlyReportHours = 6
func hourlyDefinition() Definition { func hourlyDefinition() Definition {
return Definition{ return Definition{
ID: Hourly, ID: Hourly,
Name: "Hourly Report", Name: "Hourly Report",
PromptID: "weather.hourly_generated_text", PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.0.0",
TemplateID: "hourly", TemplateID: "hourly",
GeneratedTextSchemaID: "hourly", GeneratedTextSchemaID: "hourly",
ArtifactGroup: "hourly", GeneratedTextRepairAttempts: 0,
OutputName: "hourly.md", ArtifactGroup: "hourly",
OutputName: "hourly.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"hourly/index.md", "hourly/index.md",
}, },

3
internal/report/period_test.go
View File

@@ -91,6 +91,9 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" { if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID) t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
} }
if definition.GeneratedTextRepairAttempts != 0 {
t.Fatalf("%s repair attempts = %d, want 0", definition.ID, definition.GeneratedTextRepairAttempts)
}
} }
if _, err := registry.Lookup(ID("three_day")); err == nil { if _, err := registry.Lookup(ID("three_day")); err == nil {

17
internal/report/today_report.go
View File

@@ -7,14 +7,15 @@ import (
func todayDefinition() Definition { func todayDefinition() Definition {
return Definition{ return Definition{
ID: Today, ID: Today,
Name: "Today Report", Name: "Today Report",
PromptID: "weather.today_generated_text", PromptID: "weather.today_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.0.0",
TemplateID: "today", TemplateID: "today",
GeneratedTextSchemaID: "today", GeneratedTextSchemaID: "today",
ArtifactGroup: "today", GeneratedTextRepairAttempts: 0,
OutputName: "today.md", ArtifactGroup: "today",
OutputName: "today.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",

17
internal/report/tomorrow_report.go
View File

@@ -7,14 +7,15 @@ import (
func tomorrowDefinition() Definition { func tomorrowDefinition() Definition {
return Definition{ return Definition{
ID: Tomorrow, ID: Tomorrow,
Name: "Tomorrow Report", Name: "Tomorrow Report",
PromptID: "weather.tomorrow_generated_text", PromptID: "weather.tomorrow_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.0.0",
TemplateID: "tomorrow", TemplateID: "tomorrow",
GeneratedTextSchemaID: "tomorrow", GeneratedTextSchemaID: "tomorrow",
ArtifactGroup: "tomorrow", GeneratedTextRepairAttempts: 0,
OutputName: "tomorrow.md", ArtifactGroup: "tomorrow",
OutputName: "tomorrow.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",