diff --git a/README.md b/README.md index 85cd191..b16b228 100644 --- a/README.md +++ b/README.md @@ -11,10 +11,10 @@ Framework extraction is in progress. The repository now contains the `internal/domain` model, application-neutral `internal/defaults`, and `internal/filecatalog` helpers that form the implementation foundation. It also contains internal prompt-definition and profile repositories, the embedded -built-in profile catalog, and Go-template prompt rendering. These internal -packages are not a consumer API, and the root package does not yet provide a -usable public framework API, so there is no installation or usage example at -this time. +built-in profile catalog, artifact loading, Go-template rendering, output +validation, model generation, and orchestration. These internal packages are +not a consumer API, and the root package does not yet provide a usable public +framework API, so there is no installation or usage example at this time. Contributors should start with the [development guide](docs/development.md). The [architecture policy](docs/policy/architecture.md) defines the library diff --git a/docs/internal/overview.md b/docs/internal/overview.md index ec1b608..637057f 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -22,10 +22,11 @@ contributor workflow and validation. | `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) | | `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources. | [Internal sources and validation](sources.md) | | `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests, response decoding, authentication, and deadline handling. | [Internal model client](llm.md) | +| `internal/usecase` | Coordinates preparation and execution across internal sources, rendering, artifact loading, generation, validation, and optional repair. | [Internal runner](runner.md) | These packages provide the internal model, source, rendering, validation, and -model-client foundation. Orchestration and a usable public engine are not -implemented in Promptkit yet. +model-client workflow. A usable public engine is not implemented in Promptkit +yet. ## Maintenance diff --git a/docs/internal/runner.md b/docs/internal/runner.md new file mode 100644 index 0000000..8036b77 --- /dev/null +++ b/docs/internal/runner.md @@ -0,0 +1,83 @@ +# Internal Runner + +## Purpose + +This document describes Promptkit's implemented internal orchestration. The +[architecture policy](../policy/architecture.md) owns dependency and consumer +boundaries. The [source and validation document](sources.md) owns repository, +artifact, rendering, and validation behavior, while the +[model-client document](llm.md) owns generation behavior and failure +categories. + +The runner remains under `internal/usecase`. The root package does not yet +assemble it into a usable public engine. + +## Collaborators + +`Runner` coordinates narrow internal interfaces for prompt definitions, +profiles, artifacts, rendering, model generation, and validation. Schema +documents are loaded through the validator's optional schema-loader interface. +An output repairer can be injected internally, but the ordinary runner +constructor does not enable one. + +Each invocation carries its state in request, prepared-run, and result values. +The runner has no durable run or session store. + +## Preparation Flow + +`Prepare` performs the reusable pre-generation workflow: + +1. validate the prompt selection and load the prompt definition; +2. hash the loaded definition; +3. select the request profile or the prompt's default profile; +4. resolve application-neutral defaults, profile values, and explicit request + overrides in that order; +5. validate endpoint, model, numeric overrides, and credential requirements; +6. resolve the output contract and load a structured-output schema when + required; +7. load and hash input artifacts; +8. render and hash the prompt; and +9. return the effective settings, source identities, messages, hashes, and + preparation timing. + +Pointer-based numeric overrides preserve an explicit zero. Invalid negative or +out-of-range values fail as invalid requests. A direct API key takes +precedence over environment lookup for execution; secret values remain +excluded from serialized metadata. + +## Run Flow + +`Run` calls `Prepare` rather than maintaining a second preparation path. It +performs one initial generation call, builds the named output artifact, and +validates that artifact. Invalid generated content remains a validation result; +an inability to perform validation is an operational error. + +When an internal repairer is present, a JSON or JSON Schema content failure can +trigger bounded repair attempts. Repair receives the effective execution +target, validation errors, prior output, and structured-output specification. +This capability remains internal and is not a public option. + +A successful result includes the output artifact and raw output, validation +state, prompt and rendered-prompt hashes, selected profile, effective settings, +input hashes, token usage, a generated run identifier, and UTC timing. + +## Failure Categories + +Package errors distinguish invalid requests, required profile selection, +credential failures, and prompt, profile, artifact, rendering, generation, and +validation failures. Wrapping preserves the package identities needed by the +future facade and retains collaborator identities where they are part of the +internal contract. Context cancellation propagates through the invoked +collaborator and is classified by the owning operation. + +## Test Ownership And Changes + +The [runner tests](../../internal/usecase/runner_test.go) own preparation order, +selection and override precedence, schema-before-generation behavior, hashing, +generation and validation outcomes, bounded repair, credentials and redaction, +error categories, artifact metadata, usage, and timing. + +Changes to orchestration should continue to use the existing package +interfaces, keep request state local to an invocation, and preserve `Run`'s use +of `Prepare`. Source, renderer, validator, or model-client contract changes +belong first in their owning package and document. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 3fe15bb..fe0704b 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -34,9 +34,11 @@ The implemented internal components consist of: - `internal/artifact`, which resolves ordinary inline and unrestricted caller-selected file references; - `internal/validate`, which validates basic, JSON, and JSON Schema output - using filesystem and `fs.FS` schema sources; and + using filesystem and `fs.FS` schema sources; - `internal/llm`, which defines the provider-neutral generation boundary and - implements outbound OpenAI-compatible chat requests. + implements outbound OpenAI-compatible chat requests; and +- `internal/usecase`, which coordinates preparation and execution across the + internal framework components. The defaults and renderer depend on the domain model. Prompt-definition and profile repositories use the domain model, file catalog, and YAML decoder. The @@ -44,8 +46,9 @@ built-in profile repository supplies an embedded `fs.FS` to the profile package. Artifact reading uses the domain model and application-neutral defaults. Validation uses the domain model, file catalog, and JSON Schema implementation. The model client uses the domain model, application-neutral -defaults, and an injected or standard-library HTTP client. Orchestration and -the public engine have not yet been extracted. +defaults, and an injected or standard-library HTTP client. The use-case runner +depends on the narrow interfaces owned by each internal component. The public +engine has not yet been extracted. Future framework extraction must follow this dependency direction: diff --git a/internal/usecase/repairer.go b/internal/usecase/repairer.go new file mode 100644 index 0000000..9202fc6 --- /dev/null +++ b/internal/usecase/repairer.go @@ -0,0 +1,76 @@ +package usecase + +import ( + "context" + "errors" + "fmt" + "strings" + + "gitea.maximumdirect.net/eric/promptkit/internal/domain" + "gitea.maximumdirect.net/eric/promptkit/internal/llm" +) + +type OutputRepairer interface { + Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) +} + +type RepairRequest struct { + PreviousOutput string + ValidationErrors []string + Target domain.ExecutionTarget + StructuredOutput *domain.StructuredOutputSpec + Attempt int + MaxAttempts int + Mode domain.ValidationMode +} + +type defaultOutputRepairer struct { + llm llm.Client +} + +func NewDefaultOutputRepairer(llmClient llm.Client) OutputRepairer { + return &defaultOutputRepairer{llm: llmClient} +} + +func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) { + if r.llm == nil { + return nil, errors.New("llm client is required for repair") + } + + errs := "(none provided)" + if len(req.ValidationErrors) > 0 { + errs = strings.Join(req.ValidationErrors, "\n") + } + + prompt := domain.RenderedPrompt{Messages: []domain.RenderedMessage{ + { + Role: "system", + Content: "You repair invalid JSON output. Return only corrected JSON. Do not include explanations or markdown code fences.", + }, + { + Role: "user", + Content: fmt.Sprintf( + "Repair attempt %d of %d for validation mode %s.\n\nValidation errors:\n%s\n\nPrevious output:\n%s\n\nReturn only corrected JSON.", + req.Attempt, + req.MaxAttempts, + req.Mode, + errs, + req.PreviousOutput, + ), + }, + }} + + resp, err := r.llm.Generate(ctx, domain.GenerateRequest{ + Prompt: prompt, + Target: req.Target, + StructuredOutput: req.StructuredOutput, + }) + if err != nil { + return nil, err + } + if resp == nil { + return nil, errors.New("repair llm returned nil response") + } + + return resp, nil +} diff --git a/internal/usecase/runner.go b/internal/usecase/runner.go new file mode 100644 index 0000000..4064161 --- /dev/null +++ b/internal/usecase/runner.go @@ -0,0 +1,576 @@ +package usecase + +import ( + "context" + "crypto/rand" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "strings" + "time" + "unicode" + + "gitea.maximumdirect.net/eric/promptkit/internal/artifact" + "gitea.maximumdirect.net/eric/promptkit/internal/defaults" + "gitea.maximumdirect.net/eric/promptkit/internal/domain" + "gitea.maximumdirect.net/eric/promptkit/internal/llm" + "gitea.maximumdirect.net/eric/promptkit/internal/profile" + "gitea.maximumdirect.net/eric/promptkit/internal/prompt" + "gitea.maximumdirect.net/eric/promptkit/internal/promptdef" + "gitea.maximumdirect.net/eric/promptkit/internal/validate" +) + +var ( + ErrInvalidRequest = errors.New("invalid run request") + ErrProfileRequired = errors.New("profile selection is required") + ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable") + ErrAPIKeyRequired = errors.New("api key is required") + ErrPromptLoad = errors.New("failed to load prompt definition") + ErrProfileLoad = errors.New("failed to load execution profile") + ErrArtifactLoad = errors.New("failed to load artifact") + ErrPromptRender = errors.New("failed to render prompt") + ErrLLMGenerate = errors.New("failed to generate output") + ErrValidation = errors.New("failed to validate output") +) + +// Runner executes the Promptkit core use case. +type Runner struct { + promptDefs promptdef.Repository + profiles profile.Repository + artifacts artifact.Reader + renderer prompt.Renderer + llm llm.Client + validator validate.Validator + repairer OutputRepairer +} + +func NewRunner( + promptDefs promptdef.Repository, + profiles profile.Repository, + artifacts artifact.Reader, + renderer prompt.Renderer, + llmClient llm.Client, + validator validate.Validator, +) *Runner { + return NewRunnerWithRepairer(promptDefs, profiles, artifacts, renderer, llmClient, validator, nil) +} + +func NewRunnerWithRepairer( + promptDefs promptdef.Repository, + profiles profile.Repository, + artifacts artifact.Reader, + renderer prompt.Renderer, + llmClient llm.Client, + validator validate.Validator, + repairer OutputRepairer, +) *Runner { + return &Runner{ + promptDefs: promptDefs, + profiles: profiles, + artifacts: artifacts, + renderer: renderer, + llm: llmClient, + validator: validator, + repairer: repairer, + } +} + +func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunResult, error) { + runID, err := newRunID() + if err != nil { + return nil, fmt.Errorf("failed to create run id: %w", err) + } + + start := time.Now().UTC() + + prepared, err := r.Prepare(ctx, req) + if err != nil { + return nil, err + } + + genResp, err := r.llm.Generate(ctx, domain.GenerateRequest{ + Prompt: domain.RenderedPrompt{SessionID: prepared.SessionID, Messages: prepared.Messages}, + Target: prepared.EffectiveModelParams, + TargetPresence: prepared.TargetPresence, + StructuredOutput: prepared.StructuredOutput, + }) + if err != nil { + if errors.Is(err, llm.ErrInvalidRequest) { + return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err) + } + return nil, fmt.Errorf("%w: %w", ErrLLMGenerate, err) + } + + outputArtifact := buildOutputArtifact(genResp.Content, prepared.OutputContract.Format) + validationResult, err := r.validateOutput(ctx, &outputArtifact, prepared.OutputContract, 0) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrValidation, err) + } + + if r.shouldAttemptRepair(prepared.OutputContract, validationResult) { + attemptsUsed := 0 + for attemptsUsed < prepared.OutputContract.RepairAttempts && validationResult.Status == domain.ValidationFailed { + attemptsUsed++ + + repairResp, repairErr := r.repairer.Repair(ctx, RepairRequest{ + PreviousOutput: genResp.Content, + ValidationErrors: validationResult.Errors, + Target: prepared.EffectiveModelParams, + StructuredOutput: prepared.StructuredOutput, + Attempt: attemptsUsed, + MaxAttempts: prepared.OutputContract.RepairAttempts, + Mode: prepared.OutputContract.ValidationMode, + }) + if repairErr != nil { + return nil, fmt.Errorf("%w: %w", ErrValidation, repairErr) + } + if repairResp == nil { + return nil, fmt.Errorf("%w: repairer returned nil response", ErrValidation) + } + + genResp = repairResp + outputArtifact = buildOutputArtifact(genResp.Content, prepared.OutputContract.Format) + + validationResult, err = r.validateOutput(ctx, &outputArtifact, prepared.OutputContract, attemptsUsed) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrValidation, err) + } + } + } + + end := time.Now().UTC() + + return &domain.RunResult{ + RunID: runID, + Artifact: outputArtifact, + RawOutput: genResp.Content, + Validation: validationResult, + PromptID: prepared.PromptID, + PromptVersion: prepared.PromptVersion, + PromptHash: prepared.PromptHash, + RenderedPromptHash: prepared.RenderedPromptHash, + SelectedProfileID: prepared.SelectedProfileID, + ModelName: prepared.EffectiveModelParams.Model, + Endpoint: prepared.EffectiveModelParams.Endpoint, + EffectiveModelParams: prepared.EffectiveModelParams, + InputHashes: prepared.InputHashes, + Usage: genResp.Usage, + StartTime: start, + EndTime: end, + Duration: end.Sub(start), + }, nil +} + +func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.PreparedRun, error) { + if strings.TrimSpace(req.PromptID) == "" { + return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidRequest) + } + + start := time.Now().UTC() + + def, err := r.promptDefs.GetPromptDefinition(ctx, req.PromptID, req.PromptVersion) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrPromptLoad, err) + } + promptDefinitionHash, err := hashPromptDefinition(def) + if err != nil { + return nil, fmt.Errorf("%w: failed to hash prompt definition: %v", ErrPromptLoad, err) + } + + selectedProfileID := strings.TrimSpace(req.ProfileID) + if selectedProfileID == "" { + selectedProfileID = strings.TrimSpace(def.DefaultProfile) + } + if selectedProfileID == "" { + return nil, fmt.Errorf("%w: %w: profile id is required either in request or prompt default_profile", ErrInvalidRequest, ErrProfileRequired) + } + + execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err) + } + + effectiveModel, targetPresence, err := resolveExecutionTarget(execProfile, req.Execution) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err) + } + effectiveModel.APIKey = req.APIKey + if strings.TrimSpace(effectiveModel.Endpoint) == "" { + return nil, fmt.Errorf("%w: execution endpoint is required", ErrInvalidRequest) + } + if strings.TrimSpace(effectiveModel.Model) == "" { + return nil, fmt.Errorf("%w: execution model is required", ErrInvalidRequest) + } + if err := validateAPIKey(effectiveModel.APIKeyEnv, effectiveModel.APIKey, effectiveModel.APIKeyRequired); err != nil { + return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err) + } + + effectiveContract := resolveOutputContract(def, req.Validation) + structuredOutput, err := r.resolveStructuredOutput(ctx, def, effectiveContract) + if err != nil { + return nil, err + } + + resolvedInputs := make(map[string]*domain.Artifact, len(req.Inputs)) + inputHashes := make(map[string]string, len(req.Inputs)) + for name, ref := range req.Inputs { + art, readErr := r.artifacts.Read(ctx, ref) + if readErr != nil { + return nil, fmt.Errorf("%w: input %q: %w", ErrArtifactLoad, name, readErr) + } + if art.Name == "" { + art.Name = name + } + resolvedInputs[name] = art + inputHashes[name] = art.Hash + } + + renderedPrompt, err := r.renderer.Render(ctx, def, resolvedInputs, req.Vars) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrPromptRender, err) + } + + end := time.Now().UTC() + return &domain.PreparedRun{ + PromptID: def.ID, + PromptVersion: def.Version, + PromptHash: promptDefinitionHash, + SelectedProfileID: selectedProfileID, + EffectiveModelParams: effectiveModel, + TargetPresence: targetPresence, + OutputContract: effectiveContract, + StructuredOutput: structuredOutput, + InputHashes: inputHashes, + SessionID: renderedPrompt.SessionID, + RenderedPromptHash: hashRenderedPrompt(*renderedPrompt), + Messages: renderedPrompt.Messages, + StartTime: start, + EndTime: end, + DurationMS: end.Sub(start).Milliseconds(), + }, nil +} + +func (r *Runner) resolveStructuredOutput(ctx context.Context, def *domain.PromptDefinition, contract domain.OutputContract) (*domain.StructuredOutputSpec, error) { + if contract.ValidationMode != domain.ValidationJSONSchema { + return nil, nil + } + + loader, ok := r.validator.(validate.SchemaDocumentLoader) + if !ok || loader == nil { + return nil, fmt.Errorf("%w: json_schema output requires schema document loader", ErrValidation) + } + + schemaDoc, err := loader.LoadSchemaDocument(ctx, contract.SchemaPath) + if err != nil { + return nil, fmt.Errorf("%w: failed to load json schema for structured output: %v", ErrValidation, err) + } + + return &domain.StructuredOutputSpec{ + Type: domain.StructuredOutputJSONSchema, + JSONSchema: &domain.StructuredOutputJSONSpec{ + Name: deriveStructuredSchemaName(def.ID, def.Version), + Strict: true, + Schema: schemaDoc, + }, + }, nil +} + +func deriveStructuredSchemaName(promptID string, promptVersion string) string { + raw := strings.TrimSpace(promptID) + if v := strings.TrimSpace(promptVersion); v != "" { + if raw == "" { + raw = v + } else { + raw = raw + "_" + v + } + } + + var b strings.Builder + for _, r := range raw { + if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_' || r == '-' { + b.WriteRune(r) + } else { + b.WriteRune('_') + } + } + + name := strings.Trim(b.String(), "_-") + if name == "" { + return "promptkit_schema" + } + return name +} + +func (r *Runner) validateOutput(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract, attemptsUsed int) (domain.ValidationResult, error) { + if r.validator == nil || contract.ValidationMode == domain.ValidationNone { + return domain.ValidationResult{ + Status: domain.ValidationSkipped, + Mode: contract.ValidationMode, + SchemaPath: contract.SchemaPath, + RepairAttempts: attemptsUsed, + IsValid: true, + }, nil + } + + res, err := r.validator.Validate(ctx, artifact, contract) + if err != nil { + return domain.ValidationResult{}, err + } + res.RepairAttempts = attemptsUsed + return res, nil +} + +func (r *Runner) shouldAttemptRepair(contract domain.OutputContract, validationResult domain.ValidationResult) bool { + if r.repairer == nil { + return false + } + if contract.RepairAttempts <= 0 { + return false + } + if validationResult.Status != domain.ValidationFailed { + return false + } + return contract.ValidationMode == domain.ValidationJSON || contract.ValidationMode == domain.ValidationJSONSchema +} + +func mergeExecutionTarget(base domain.ExecutionTarget, override domain.ExecutionTarget) domain.ExecutionTarget { + out := base + if override.Endpoint != "" { + out.Endpoint = override.Endpoint + } + if override.Model != "" { + out.Model = override.Model + } + if override.Temperature != 0 { + out.Temperature = override.Temperature + } + if override.MaxTokens != 0 { + out.MaxTokens = override.MaxTokens + } + if override.TopP != 0 { + out.TopP = override.TopP + } + if override.TimeoutSeconds != 0 { + out.TimeoutSeconds = override.TimeoutSeconds + } + if strings.TrimSpace(override.ServiceTier) != "" { + out.ServiceTier = override.ServiceTier + } + if strings.TrimSpace(override.ReasoningEffort) != "" { + out.ReasoningEffort = override.ReasoningEffort + } + if strings.TrimSpace(override.APIKeyEnv) != "" { + out.APIKeyEnv = override.APIKeyEnv + } + if override.APIKeyRequired { + out.APIKeyRequired = true + } + if len(override.ExtraParams) > 0 { + out.ExtraParams = copyExtraParams(override.ExtraParams) + } + return out +} + +func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) { + out := base + var presence domain.ExecutionTargetPresence + if override.Endpoint != "" { + out.Endpoint = override.Endpoint + } + if override.Model != "" { + out.Model = override.Model + } + if override.Temperature != nil { + if *override.Temperature < 0 || *override.Temperature > 2 { + return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("temperature must be between 0 and 2") + } + out.Temperature = *override.Temperature + presence.Temperature = true + } + if override.MaxTokens != nil { + if *override.MaxTokens < 0 { + return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("max_tokens must be greater than or equal to 0") + } + out.MaxTokens = *override.MaxTokens + presence.MaxTokens = true + } + if override.TopP != nil { + if *override.TopP < 0 || *override.TopP > 1 { + return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("top_p must be between 0 and 1") + } + out.TopP = *override.TopP + presence.TopP = true + } + if override.TimeoutSeconds != nil { + if *override.TimeoutSeconds < 0 { + return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("timeout_seconds must be greater than or equal to 0") + } + out.TimeoutSeconds = *override.TimeoutSeconds + presence.TimeoutSeconds = true + } + if strings.TrimSpace(override.ServiceTier) != "" { + out.ServiceTier = override.ServiceTier + } + if strings.TrimSpace(override.ReasoningEffort) != "" { + out.ReasoningEffort = override.ReasoningEffort + } + if strings.TrimSpace(override.APIKeyEnv) != "" { + out.APIKeyEnv = override.APIKeyEnv + } + if len(override.ExtraParams) > 0 { + out.ExtraParams = copyExtraParams(override.ExtraParams) + } + return out, presence, nil +} + +func resolveExecutionTarget(profileValue *domain.ExecutionProfile, override *domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) { + out := defaults.ExecutionTargetDefault() + out = mergeExecutionTarget(out, executionProfileToTarget(profileValue)) + var presence domain.ExecutionTargetPresence + if override != nil { + var err error + out, presence, err = mergeExecutionTargetOverride(out, *override) + if err != nil { + return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, err + } + } + return out, presence, nil +} + +func validateAPIKey(apiKeyEnv string, apiKey string, apiKeyRequired bool) error { + if strings.TrimSpace(apiKey) != "" { + return nil + } + envName := strings.TrimSpace(apiKeyEnv) + if envName == "" { + if apiKeyRequired { + return ErrAPIKeyRequired + } + return nil + } + if strings.TrimSpace(os.Getenv(envName)) == "" { + return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName) + } + return nil +} + +func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget { + if p == nil { + return domain.ExecutionTarget{} + } + return domain.ExecutionTarget{ + Endpoint: p.Endpoint, + Model: p.Model, + Temperature: p.Temperature, + MaxTokens: p.MaxTokens, + TopP: p.TopP, + TimeoutSeconds: p.TimeoutSeconds, + ServiceTier: p.ServiceTier, + ReasoningEffort: p.ReasoningEffort, + APIKeyEnv: p.APIKeyEnv, + APIKeyRequired: p.APIKeyRequired, + ExtraParams: copyExtraParams(p.ExtraParams), + } +} + +func copyExtraParams(src map[string]any) map[string]any { + if len(src) == 0 { + return nil + } + cp := make(map[string]any, len(src)) + for k, v := range src { + cp[k] = v + } + return cp +} + +func resolveOutputContract(def *domain.PromptDefinition, override *domain.OutputContract) domain.OutputContract { + contract := def.Validation + if contract.Format == "" { + contract.Format = def.OutputFormat + } + if override != nil { + contract = *override + } + if contract.Format == "" { + contract.Format = domain.FormatText + } + return contract +} + +func hashRenderedPrompt(p domain.RenderedPrompt) string { + var b strings.Builder + if p.SessionID != "" { + b.WriteString("session_id=") + b.WriteString(p.SessionID) + b.WriteString("\n---\n") + } + for _, msg := range p.Messages { + b.WriteString(msg.Role) + b.WriteByte('\n') + b.WriteString(msg.Content) + if msg.CacheControl != nil { + b.WriteString("\ncache_control.type=") + b.WriteString(string(msg.CacheControl.Type)) + if msg.CacheControl.TTL != "" { + b.WriteString("\ncache_control.ttl=") + b.WriteString(msg.CacheControl.TTL) + } + } + b.WriteString("\n---\n") + } + h := sha256.Sum256([]byte(b.String())) + return hex.EncodeToString(h[:]) +} + +func buildOutputArtifact(content string, format domain.OutputFormat) domain.Artifact { + body := []byte(content) + hash := sha256.Sum256(body) + + contentType := defaults.ContentTypeTextPlain + switch format { + case domain.FormatMarkdown: + contentType = defaults.ContentTypeTextMarkdown + case domain.FormatJSON: + contentType = defaults.ContentTypeApplicationJSON + } + + return domain.Artifact{ + Name: defaults.OutputArtifactName, + ContentType: contentType, + Body: body, + Size: int64(len(body)), + Hash: hex.EncodeToString(hash[:]), + } +} + +func hashPromptDefinition(def *domain.PromptDefinition) (string, error) { + b, err := json.Marshal(def) + if err != nil { + return "", err + } + sum := sha256.Sum256(b) + return hex.EncodeToString(sum[:]), nil +} + +func newRunID() (string, error) { + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + return "", err + } + + // UUID v4 (RFC 4122 variant). + b[6] = (b[6] & 0x0f) | 0x40 + b[8] = (b[8] & 0x3f) | 0x80 + + return fmt.Sprintf("%08x-%04x-%04x-%04x-%012x", + b[0:4], + b[4:6], + b[6:8], + b[8:10], + b[10:16], + ), nil +} diff --git a/internal/usecase/runner_test.go b/internal/usecase/runner_test.go new file mode 100644 index 0000000..d0b7ae6 --- /dev/null +++ b/internal/usecase/runner_test.go @@ -0,0 +1,1847 @@ +package usecase + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "path/filepath" + "reflect" + "regexp" + "strings" + "testing" + + "gitea.maximumdirect.net/eric/promptkit/internal/defaults" + "gitea.maximumdirect.net/eric/promptkit/internal/domain" + "gitea.maximumdirect.net/eric/promptkit/internal/llm" + "gitea.maximumdirect.net/eric/promptkit/internal/profile" + "gitea.maximumdirect.net/eric/promptkit/internal/prompt" + "gitea.maximumdirect.net/eric/promptkit/internal/promptdef" + "gitea.maximumdirect.net/eric/promptkit/internal/validate" +) + +type fakePromptRepo struct { + def *domain.PromptDefinition + err error + lastID string + lastVersion string +} + +type fakeExecutionProfileRepo struct { + profiles map[string]*domain.ExecutionProfile + err error + lastID string +} + +func (f *fakeExecutionProfileRepo) GetProfile(ctx context.Context, id string) (*domain.ExecutionProfile, error) { + f.lastID = id + if f.err != nil { + return nil, f.err + } + if p, ok := f.profiles[id]; ok { + cp := *p + return &cp, nil + } + return nil, errors.New("profile not found") +} + +func (f *fakePromptRepo) GetPromptDefinition(ctx context.Context, id string, version string) (*domain.PromptDefinition, error) { + f.lastID = id + f.lastVersion = version + if f.err != nil { + return nil, f.err + } + return f.def, nil +} + +type fakeArtifactReader struct { + artifactsByURI map[string]*domain.Artifact + errByURI map[string]error +} + +func (f *fakeArtifactReader) Read(ctx context.Context, ref domain.ArtifactRef) (*domain.Artifact, error) { + if err, ok := f.errByURI[ref.URI]; ok { + return nil, err + } + if art, ok := f.artifactsByURI[ref.URI]; ok { + cp := *art + return &cp, nil + } + return nil, errors.New("artifact not found") +} + +type fakeRenderer struct { + rendered *domain.RenderedPrompt + err error +} + +func (f *fakeRenderer) Render(ctx context.Context, def *domain.PromptDefinition, inputs map[string]*domain.Artifact, vars map[string]string) (*domain.RenderedPrompt, error) { + if f.err != nil { + return nil, f.err + } + return f.rendered, nil +} + +type fakeLLM struct { + resp *domain.GenerateResponse + err error + lastReq domain.GenerateRequest + calls int + forbid bool +} + +func (f *fakeLLM) Generate(ctx context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) { + f.calls++ + f.lastReq = req + if err := ctx.Err(); err != nil { + return nil, err + } + if f.forbid { + return nil, errors.New("llm should not be called") + } + if f.err != nil { + return nil, f.err + } + return f.resp, nil +} + +type fakeValidator struct { + result domain.ValidationResult + err error + schemaDoc any + schemaErr error + schemaLoadPath string + schemaLoads int +} + +func (f *fakeValidator) Validate(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract) (domain.ValidationResult, error) { + if f.err != nil { + return domain.ValidationResult{}, f.err + } + return f.result, nil +} + +func (f *fakeValidator) LoadSchemaDocument(ctx context.Context, schemaPath string) (any, error) { + f.schemaLoads++ + f.schemaLoadPath = schemaPath + if f.schemaErr != nil { + return nil, f.schemaErr + } + if f.schemaDoc != nil { + return f.schemaDoc, nil + } + return map[string]any{"type": "object"}, nil +} + +type fakeRepairer struct { + responses []*domain.GenerateResponse + err error + calls int + reqs []RepairRequest +} + +func (f *fakeRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) { + f.calls++ + f.reqs = append(f.reqs, req) + if f.err != nil { + return nil, f.err + } + if len(f.responses) == 0 { + return nil, errors.New("no repair response configured") + } + idx := f.calls - 1 + if idx >= len(f.responses) { + idx = len(f.responses) - 1 + } + return f.responses[idx], nil +} + +func TestRunnerPrepareWithExplicitProfileSelection(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}} + reader := &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{ + "a://t": {Body: []byte("transcript"), Hash: hashString("transcript")}, + "a://g": {Body: []byte("glossary"), Hash: hashString("glossary")}, + }} + renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{SessionID: "session-123", Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}} + llmClient := &fakeLLM{forbid: true} + + runner := NewRunner(promptRepo, execRepo, reader, renderer, llmClient, nil) + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + PromptVersion: "1", + ProfileID: "exec", + Inputs: map[string]domain.ArtifactRef{ + "transcript": {Type: domain.ArtifactRefFile, URI: "a://t"}, + "glossary": {Type: domain.ArtifactRefFile, URI: "a://g"}, + }, + Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "m", Temperature: float64Ptr(0.3), TimeoutSeconds: intPtr(90)}, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if prepared.PromptID != "p" || prepared.PromptVersion != "1" { + t.Fatalf("unexpected prepared prompt metadata: %+v", prepared) + } + if prepared.SelectedProfileID != "exec" { + t.Fatalf("expected selected profile exec, got %q", prepared.SelectedProfileID) + } + if prepared.PromptHash == "" || prepared.RenderedPromptHash == "" { + t.Fatal("expected prompt hashes") + } + if prepared.EffectiveModelParams.Model != "m" || prepared.EffectiveModelParams.Endpoint != "http://override/v1" { + t.Fatalf("unexpected model params: %+v", prepared.EffectiveModelParams) + } + if prepared.OutputContract.Format != domain.FormatMarkdown { + t.Fatalf("expected output format markdown, got %q", prepared.OutputContract.Format) + } + if len(prepared.InputHashes) != 2 || prepared.InputHashes["transcript"] == "" || prepared.InputHashes["glossary"] == "" { + t.Fatalf("expected input hashes, got %#v", prepared.InputHashes) + } + if len(prepared.Messages) != 2 { + t.Fatalf("expected two messages, got %d", len(prepared.Messages)) + } + if prepared.SessionID != "session-123" { + t.Fatalf("expected prepared session id, got %q", prepared.SessionID) + } + if llmClient.calls != 0 { + t.Fatalf("prepare should not call llm, calls=%d", llmClient.calls) + } +} + +func TestRunnerPrepareUsesPromptDefaultProfileWhenNoExplicitProfileID(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + promptRepo.def.DefaultProfile = "from-prompt" + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "from-prompt": {ID: "from-prompt", Endpoint: "http://llm/v1", Model: "m"}, + }} + + runner := newMinimalRunner(promptRepo, execRepo) + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if execRepo.lastID != "from-prompt" { + t.Fatalf("expected prompt default profile lookup, got %q", execRepo.lastID) + } + if prepared.SelectedProfileID != "from-prompt" { + t.Fatalf("expected selected profile from-prompt, got %q", prepared.SelectedProfileID) + } +} + +func TestRunnerPrepareMissingExplicitProfileAndMissingDefaultProfileFails(t *testing.T) { + repo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + repo.def.DefaultProfile = "" + runner := newMinimalRunner(repo, &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}) + _, err := runner.Prepare(context.Background(), domain.RunRequest{PromptID: "p", Inputs: singleInputRef()}) + if !errors.Is(err, ErrInvalidRequest) { + t.Fatalf("expected ErrInvalidRequest, got %v", err) + } + if !errors.Is(err, ErrProfileRequired) { + t.Fatalf("expected ErrProfileRequired, got %v", err) + } +} + +func TestRunnerPrepareSelectedProfileDoesNotExistFails(t *testing.T) { + repo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + repo.def.DefaultProfile = "does-not-exist" + runner := newMinimalRunner(repo, &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{}}) + _, err := runner.Prepare(context.Background(), domain.RunRequest{PromptID: "p", Inputs: singleInputRef()}) + if !errors.Is(err, ErrProfileLoad) { + t.Fatalf("expected ErrProfileLoad, got %v", err) + } +} + +func TestRunnerPreparePromptLoadFailure(t *testing.T) { + runner := NewRunner(&fakePromptRepo{err: errors.New("boom")}, &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, defaultArtifactReader(), defaultRenderer(), &fakeLLM{}, nil) + _, err := runner.Prepare(context.Background(), domain.RunRequest{PromptID: "p"}) + if !errors.Is(err, ErrPromptLoad) { + t.Fatalf("expected ErrPromptLoad, got %v", err) + } + if errors.Is(err, ErrProfileLoad) { + t.Fatalf("did not expect ErrProfileLoad, got %v", err) + } +} + +func TestRunnerPrepareRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": { + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + Temperature: 0.2, + MaxTokens: 500, + TopP: 0.9, + TimeoutSeconds: 120, + ServiceTier: "priority", + }, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil) + + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: &domain.ExecutionTargetOverride{ + Endpoint: "http://override/v1", + Model: "override-model", + Temperature: float64Ptr(0.7), + TimeoutSeconds: intPtr(30), + ServiceTier: "flex", + }, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if prepared.EffectiveModelParams.Endpoint != "http://override/v1" || prepared.EffectiveModelParams.Model != "override-model" { + t.Fatalf("expected endpoint/model override to win, got %+v", prepared.EffectiveModelParams) + } + if prepared.EffectiveModelParams.TopP != 0.9 { + t.Fatalf("expected profile top_p to remain, got %v", prepared.EffectiveModelParams.TopP) + } + if prepared.EffectiveModelParams.ServiceTier != "flex" { + t.Fatalf("expected service_tier override to win, got %q", prepared.EffectiveModelParams.ServiceTier) + } +} + +func TestRunnerPrepareRequestNumericOverridePresence(t *testing.T) { + tests := []struct { + name string + override *domain.ExecutionTargetOverride + wantTemperature float64 + wantMaxTokens int + wantTopP float64 + wantTimeoutSecs int + wantPresence domain.ExecutionTargetPresence + }{ + { + name: "omitted preserves profile values", + override: &domain.ExecutionTargetOverride{}, + wantTemperature: 0.7, + wantMaxTokens: 321, + wantTopP: 0.8, + wantTimeoutSecs: 45, + }, + { + name: "explicit zero temperature", + override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(0)}, + wantTemperature: 0, + wantMaxTokens: 321, + wantTopP: 0.8, + wantTimeoutSecs: 45, + wantPresence: domain.ExecutionTargetPresence{Temperature: true}, + }, + { + name: "explicit zero max tokens", + override: &domain.ExecutionTargetOverride{MaxTokens: intPtr(0)}, + wantTemperature: 0.7, + wantMaxTokens: 0, + wantTopP: 0.8, + wantTimeoutSecs: 45, + wantPresence: domain.ExecutionTargetPresence{MaxTokens: true}, + }, + { + name: "explicit zero top p", + override: &domain.ExecutionTargetOverride{TopP: float64Ptr(0)}, + wantTemperature: 0.7, + wantMaxTokens: 321, + wantTopP: 0, + wantTimeoutSecs: 45, + wantPresence: domain.ExecutionTargetPresence{TopP: true}, + }, + { + name: "explicit zero timeout", + override: &domain.ExecutionTargetOverride{TimeoutSeconds: intPtr(0)}, + wantTemperature: 0.7, + wantMaxTokens: 321, + wantTopP: 0.8, + wantTimeoutSecs: 0, + wantPresence: domain.ExecutionTargetPresence{TimeoutSeconds: true}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": { + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + Temperature: 0.7, + MaxTokens: 321, + TopP: 0.8, + TimeoutSeconds: 45, + }, + }}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{forbid: true}, + nil, + ) + + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: tc.override, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + got := prepared.EffectiveModelParams + if got.Temperature != tc.wantTemperature || + got.MaxTokens != tc.wantMaxTokens || + got.TopP != tc.wantTopP || + got.TimeoutSeconds != tc.wantTimeoutSecs { + t.Fatalf("unexpected effective numeric settings: %+v", got) + } + if prepared.TargetPresence != tc.wantPresence { + t.Fatalf("unexpected target presence: got %+v want %+v", prepared.TargetPresence, tc.wantPresence) + } + }) + } +} + +func TestRunnerPrepareInvalidRequestNumericOverridesFail(t *testing.T) { + tests := []struct { + name string + override *domain.ExecutionTargetOverride + }{ + {name: "temperature below range", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(-0.1)}}, + {name: "temperature above range", override: &domain.ExecutionTargetOverride{Temperature: float64Ptr(2.1)}}, + {name: "max tokens below range", override: &domain.ExecutionTargetOverride{MaxTokens: intPtr(-1)}}, + {name: "top p below range", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(-0.1)}}, + {name: "top p above range", override: &domain.ExecutionTargetOverride{TopP: float64Ptr(1.1)}}, + {name: "timeout below range", override: &domain.ExecutionTargetOverride{TimeoutSeconds: intPtr(-1)}}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{forbid: true}, + nil, + ) + + _, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: tc.override, + }) + if !errors.Is(err, ErrInvalidRequest) { + t.Fatalf("expected ErrInvalidRequest, got %v", err) + } + }) + } +} + +func TestRunnerPrepareSelectedProfileBeatsBuiltInDefault(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": { + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + TopP: 0.8, + TimeoutSeconds: 90, + ServiceTier: "priority", + }, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil) + + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if prepared.EffectiveModelParams.TopP != 0.8 { + t.Fatalf("expected profile top_p to beat default, got %v", prepared.EffectiveModelParams.TopP) + } + if prepared.EffectiveModelParams.TimeoutSeconds != 90 { + t.Fatalf("expected profile timeout to beat default, got %d", prepared.EffectiveModelParams.TimeoutSeconds) + } + if prepared.EffectiveModelParams.ServiceTier != "priority" { + t.Fatalf("expected profile service_tier to beat default, got %q", prepared.EffectiveModelParams.ServiceTier) + } +} + +func TestRunnerPrepareFileBackedPromptBodiesRenderCorrectly(t *testing.T) { + promptDir := filepath.Join("..", "promptdef", "testdata") + profileDir := filepath.Join("..", "profile", "testdata") + + reader := &fakeArtifactReader{ + artifactsByURI: map[string]*domain.Artifact{ + "a://transcript": { + Name: "transcript", + Body: []byte("Session transcript body."), + Hash: hashString("Session transcript body."), + }, + }, + } + llmClient := &fakeLLM{forbid: true} + runner := NewRunner( + promptdef.NewFilesystemRepository(promptDir), + profile.NewFilesystemRepository(profileDir), + reader, + prompt.NewGoRenderer(), + llmClient, + nil, + ) + + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "valid-file-backed", + ProfileID: "local-default", + Inputs: map[string]domain.ArtifactRef{ + "transcript": {Type: domain.ArtifactRefFile, URI: "a://transcript"}, + }, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if len(prepared.Messages) != 2 { + t.Fatalf("expected two rendered messages, got %d", len(prepared.Messages)) + } + if !strings.Contains(prepared.Messages[1].Content, "Session transcript body.") { + t.Fatalf("expected file-backed template content to render input, got %q", prepared.Messages[1].Content) + } +} + +func TestRunnerPrepareRequiredInputMissingFails(t *testing.T) { + def := promptDef(domain.FormatText, domain.ValidationNone, 0) + def.Templates = []domain.PromptMessageTemplate{{Role: "user", Content: `{{input "transcript"}}`}} + + runner := NewRunner( + &fakePromptRepo{def: def}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + prompt.NewGoRenderer(), + &fakeLLM{forbid: true}, + nil, + ) + _, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: map[string]domain.ArtifactRef{}, + }) + if !errors.Is(err, ErrPromptRender) { + t.Fatalf("expected ErrPromptRender, got %v", err) + } + if !errors.Is(err, prompt.ErrMissingRequiredInput) { + t.Fatalf("expected ErrMissingRequiredInput, got %v", err) + } +} + +func TestRunnerPrepareUnknownTemplateInputReferenceFails(t *testing.T) { + def := promptDef(domain.FormatText, domain.ValidationNone, 0) + def.Inputs = []domain.PromptInput{{Name: "transcript", Required: false}} + def.Templates = []domain.PromptMessageTemplate{{Role: "user", Content: `{{input "ghost"}}`}} + + runner := NewRunner( + &fakePromptRepo{def: def}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + prompt.NewGoRenderer(), + &fakeLLM{forbid: true}, + nil, + ) + _, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: map[string]domain.ArtifactRef{}, + }) + if !errors.Is(err, ErrPromptRender) { + t.Fatalf("expected ErrPromptRender, got %v", err) + } + if !errors.Is(err, prompt.ErrUnknownInput) { + t.Fatalf("expected ErrUnknownInput, got %v", err) + } +} + +func TestRunnerPrepareAPIKeyEnvNameIncludedButNotResolvedValue(t *testing.T) { + const envName = "PROMPTKIT_TEST_API_KEY" + const secret = "top-secret-value" + t.Setenv(envName, secret) + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: envName}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{forbid: true}, nil) + + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()}) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if prepared.EffectiveModelParams.APIKeyEnv != envName { + t.Fatalf("expected api key env name, got %q", prepared.EffectiveModelParams.APIKeyEnv) + } + metadataDump := fmt.Sprintf("%+v|%s|%s", prepared.EffectiveModelParams, prepared.PromptHash, prepared.RenderedPromptHash) + if strings.Contains(metadataDump, secret) { + t.Fatalf("unexpected api key value in prepared metadata dump: %s", metadataDump) + } +} + +func TestRunnerPrepareJSONSchemaBuildsStructuredOutputSpec(t *testing.T) { + def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0) + def.Validation.SchemaPath = "events.schema.json" + validator := &fakeValidator{ + schemaDoc: map[string]any{ + "type": "object", + "properties": map[string]any{ + "events": map[string]any{"type": "array"}, + }, + }, + } + runner := NewRunner( + &fakePromptRepo{def: def}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{forbid: true}, + validator, + ) + + prepared, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if validator.schemaLoads != 1 { + t.Fatalf("expected one schema load, got %d", validator.schemaLoads) + } + if validator.schemaLoadPath != "events.schema.json" { + t.Fatalf("expected schema path events.schema.json, got %q", validator.schemaLoadPath) + } + if prepared.StructuredOutput == nil { + t.Fatal("expected structured output spec") + } + if prepared.StructuredOutput.Type != domain.StructuredOutputJSONSchema { + t.Fatalf("expected structured output type json_schema, got %q", prepared.StructuredOutput.Type) + } + if prepared.StructuredOutput.JSONSchema == nil { + t.Fatal("expected structured output json_schema payload") + } + if prepared.StructuredOutput.JSONSchema.Name != "p_1" { + t.Fatalf("expected derived schema name p_1, got %q", prepared.StructuredOutput.JSONSchema.Name) + } + if prepared.StructuredOutput.JSONSchema.Strict != true { + t.Fatalf("expected strict=true, got %v", prepared.StructuredOutput.JSONSchema.Strict) + } +} + +func TestRunnerPrepareJSONSchemaSchemaLoadFailureReturnsValidationError(t *testing.T) { + def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0) + def.Validation.SchemaPath = "missing.schema.json" + validator := &fakeValidator{schemaErr: errors.New("schema unavailable")} + runner := NewRunner( + &fakePromptRepo{def: def}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{forbid: true}, + validator, + ) + + _, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrValidation) { + t.Fatalf("expected ErrValidation, got %v", err) + } + if validator.schemaLoads != 1 { + t.Fatalf("expected one schema load attempt, got %d", validator.schemaLoads) + } + if validator.schemaLoadPath != "missing.schema.json" { + t.Fatalf("expected schema path missing.schema.json, got %q", validator.schemaLoadPath) + } +} + +func TestRunnerRunJSONSchemaSchemaLoadFailureFailsBeforeLLM(t *testing.T) { + def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 0) + def.Validation.SchemaPath = "missing.schema.json" + llmClient := &fakeLLM{forbid: true} + validator := &fakeValidator{schemaErr: errors.New("schema unavailable")} + runner := NewRunner( + &fakePromptRepo{def: def}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + llmClient, + validator, + ) + + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrValidation) { + t.Fatalf("expected ErrValidation, got %v", err) + } + if llmClient.calls != 0 { + t.Fatalf("expected llm not called when schema loading fails, calls=%d", llmClient.calls) + } +} + +func TestDeriveStructuredSchemaName(t *testing.T) { + tests := []struct { + name string + id string + version string + want string + }{ + { + name: "sanitizes punctuation and keeps dashes", + id: "prompt.id/alpha", + version: "1.0.0-beta", + want: "prompt_id_alpha_1_0_0-beta", + }, + { + name: "fallback when empty", + id: "", + version: "", + want: "promptkit_schema", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := deriveStructuredSchemaName(tc.id, tc.version) + if got != tc.want { + t.Fatalf("expected %q, got %q", tc.want, got) + } + }) + } +} + +func TestHashRenderedPromptIncludesCacheControlWhenPresent(t *testing.T) { + uncached := domain.RenderedPrompt{Messages: []domain.RenderedMessage{ + {Role: "system", Content: "sys"}, + {Role: "user", Content: "usr"}, + }} + wantLegacyHash := hashString("system\nsys\n---\nuser\nusr\n---\n") + if got := hashRenderedPrompt(uncached); got != wantLegacyHash { + t.Fatalf("expected no-cache hash to preserve legacy input, got %q want %q", got, wantLegacyHash) + } + + withCache := domain.RenderedPrompt{Messages: []domain.RenderedMessage{ + { + Role: "system", + Content: "sys", + CacheControl: &domain.CacheControl{ + Type: domain.CacheControlEphemeral, + TTL: "1h", + }, + }, + {Role: "user", Content: "usr"}, + }} + alsoWithCache := domain.RenderedPrompt{Messages: []domain.RenderedMessage{ + { + Role: "system", + Content: "sys", + CacheControl: &domain.CacheControl{ + Type: domain.CacheControlEphemeral, + TTL: "1h", + }, + }, + {Role: "user", Content: "usr"}, + }} + withoutTTL := domain.RenderedPrompt{Messages: []domain.RenderedMessage{ + { + Role: "system", + Content: "sys", + CacheControl: &domain.CacheControl{ + Type: domain.CacheControlEphemeral, + }, + }, + {Role: "user", Content: "usr"}, + }} + + cachedHash := hashRenderedPrompt(withCache) + if cachedHash == hashRenderedPrompt(uncached) { + t.Fatal("expected cache control to change rendered prompt hash") + } + if cachedHash != hashRenderedPrompt(alsoWithCache) { + t.Fatal("expected identical cache control metadata to produce stable hash") + } + if cachedHash == hashRenderedPrompt(withoutTTL) { + t.Fatal("expected ttl changes to affect rendered prompt hash") + } +} + +func TestHashRenderedPromptIncludesSessionIDWhenPresent(t *testing.T) { + withoutSession := domain.RenderedPrompt{Messages: []domain.RenderedMessage{ + {Role: "system", Content: "sys"}, + {Role: "user", Content: "usr"}, + }} + withSession := domain.RenderedPrompt{ + SessionID: "session-123", + Messages: []domain.RenderedMessage{ + {Role: "system", Content: "sys"}, + {Role: "user", Content: "usr"}, + }, + } + alsoWithSession := domain.RenderedPrompt{ + SessionID: "session-123", + Messages: []domain.RenderedMessage{ + {Role: "system", Content: "sys"}, + {Role: "user", Content: "usr"}, + }, + } + otherSession := domain.RenderedPrompt{ + SessionID: "session-456", + Messages: []domain.RenderedMessage{ + {Role: "system", Content: "sys"}, + {Role: "user", Content: "usr"}, + }, + } + + sessionHash := hashRenderedPrompt(withSession) + if sessionHash == hashRenderedPrompt(withoutSession) { + t.Fatal("expected session_id to change rendered prompt hash") + } + if sessionHash != hashRenderedPrompt(alsoWithSession) { + t.Fatal("expected identical session_id to produce stable hash") + } + if sessionHash == hashRenderedPrompt(otherSession) { + t.Fatal("expected session_id value changes to affect rendered prompt hash") + } +} + +func TestRunnerRunSuccessful(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}} + reader := &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{ + "a://t": {Body: []byte("transcript"), Hash: hashString("transcript")}, + "a://g": {Body: []byte("glossary"), Hash: hashString("glossary")}, + }} + renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{SessionID: "session-123", Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "# recap", Usage: domain.TokenUsage{TotalTokens: 7}}} + + runner := NewRunner(promptRepo, execRepo, reader, renderer, llmClient, nil) + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + PromptVersion: "1", + ProfileID: "exec", + Inputs: map[string]domain.ArtifactRef{ + "transcript": {Type: domain.ArtifactRefFile, URI: "a://t"}, + "glossary": {Type: domain.ArtifactRefFile, URI: "a://g"}, + }, + Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "m", Temperature: float64Ptr(0.3), TimeoutSeconds: intPtr(90)}, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.PromptID != "p" || res.PromptVersion != "1" { + t.Fatalf("unexpected prompt metadata: %+v", res) + } + if res.SelectedProfileID != "exec" { + t.Fatalf("expected selected profile exec, got %q", res.SelectedProfileID) + } + if ok, _ := regexp.MatchString(`^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`, res.RunID); !ok { + t.Fatalf("invalid run id: %q", res.RunID) + } + if res.PromptHash == "" || res.RenderedPromptHash == "" { + t.Fatal("expected prompt hashes") + } + if res.EffectiveModelParams.Model != "m" || res.Endpoint != "http://override/v1" { + t.Fatalf("unexpected model params: %+v", res.EffectiveModelParams) + } + if res.Artifact.Name != defaults.OutputArtifactName { + t.Fatalf("expected default output artifact name %q, got %q", defaults.OutputArtifactName, res.Artifact.Name) + } + if res.Artifact.ContentType != defaults.ContentTypeTextMarkdown { + t.Fatalf("expected markdown content type %q, got %q", defaults.ContentTypeTextMarkdown, res.Artifact.ContentType) + } + if res.RawOutput != "# recap" { + t.Fatalf("expected raw output, got %q", res.RawOutput) + } + if res.Validation.Status != domain.ValidationSkipped { + t.Fatalf("expected skipped validation, got %q", res.Validation.Status) + } + if llmClient.lastReq.Target.TimeoutSeconds != 90 { + t.Fatalf("expected timeout propagation, got %d", llmClient.lastReq.Target.TimeoutSeconds) + } + if !llmClient.lastReq.TargetPresence.Temperature || !llmClient.lastReq.TargetPresence.TimeoutSeconds { + t.Fatalf("expected numeric override presence to be sent to llm, got %+v", llmClient.lastReq.TargetPresence) + } + if llmClient.lastReq.Prompt.SessionID != "session-123" { + t.Fatalf("expected session id to be sent to llm, got %q", llmClient.lastReq.Prompt.SessionID) + } + if res.Usage.TotalTokens != 7 { + t.Fatalf("expected token usage to be retained, got %+v", res.Usage) + } + if res.StartTime.IsZero() || res.EndTime.IsZero() || res.EndTime.Before(res.StartTime) { + t.Fatalf("unexpected run timing: start=%v end=%v", res.StartTime, res.EndTime) + } + if res.Duration != res.EndTime.Sub(res.StartTime) { + t.Fatalf("expected duration %v, got %v", res.EndTime.Sub(res.StartTime), res.Duration) + } +} + +func TestRunnerRunPassesExtraParamsToGenerateRequestTarget(t *testing.T) { + extraParams := map[string]any{ + "string_value": "enabled", + "number_value": 42, + "boolean_value": true, + "object_value": map[string]any{"nested": "value"}, + "array_value": []any{"first", 3, false}, + } + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": { + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + ExtraParams: extraParams, + }, + }} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if !reflect.DeepEqual(res.EffectiveModelParams.ExtraParams, extraParams) { + t.Fatalf("expected run result extra_params to match profile values, got %#v", res.EffectiveModelParams.ExtraParams) + } + if !reflect.DeepEqual(llmClient.lastReq.Target.ExtraParams, extraParams) { + t.Fatalf("expected generate request extra_params to match profile values, got %#v", llmClient.lastReq.Target.ExtraParams) + } +} + +func TestRunnerRunAndPrepareResolveSameProfileAndEffectiveSettings(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}} + reader := &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{ + "a://t": {Body: []byte("transcript"), Hash: hashString("transcript")}, + }} + renderer := &fakeRenderer{rendered: &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "system", Content: "sys"}, {Role: "user", Content: "usr"}}}} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "# recap"}} + runner := NewRunner(promptRepo, execRepo, reader, renderer, llmClient, nil) + + req := domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: map[string]domain.ArtifactRef{ + "transcript": {Type: domain.ArtifactRefFile, URI: "a://t"}, + }, + Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "m", Temperature: float64Ptr(0.3), TimeoutSeconds: intPtr(90)}, + } + + prepared, err := runner.Prepare(context.Background(), req) + if err != nil { + t.Fatalf("prepare should succeed, got %v", err) + } + + res, err := runner.Run(context.Background(), req) + if err != nil { + t.Fatalf("run should succeed, got %v", err) + } + + if res.SelectedProfileID != prepared.SelectedProfileID { + t.Fatalf("expected selected profile to match prepare, run=%q prepare=%q", res.SelectedProfileID, prepared.SelectedProfileID) + } + if !reflect.DeepEqual(res.EffectiveModelParams, prepared.EffectiveModelParams) { + t.Fatalf("effective model params mismatch:\nrun=%+v\nprepare=%+v", res.EffectiveModelParams, prepared.EffectiveModelParams) + } + if !reflect.DeepEqual(res.InputHashes, prepared.InputHashes) { + t.Fatalf("input hashes mismatch:\nrun=%#v\nprepare=%#v", res.InputHashes, prepared.InputHashes) + } + if res.RenderedPromptHash != prepared.RenderedPromptHash { + t.Fatalf("expected rendered prompt hash to match prepare, run=%q prepare=%q", res.RenderedPromptHash, prepared.RenderedPromptHash) + } + if !reflect.DeepEqual(llmClient.lastReq.Prompt.Messages, prepared.Messages) { + t.Fatalf("expected run to send prepare-rendered messages to llm") + } +} + +func TestRunnerRunExplicitProfileIDIsUsed(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + promptRepo.def.DefaultProfile = "default-prof" + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "explicit-prof": {ID: "explicit-prof", Endpoint: "http://explicit/v1", Model: "explicit"}, + "default-prof": {ID: "default-prof", Endpoint: "http://default/v1", Model: "default"}, + }} + + runner := newMinimalRunner(promptRepo, execRepo) + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "explicit-prof", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if execRepo.lastID != "explicit-prof" { + t.Fatalf("expected explicit profile lookup, got %q", execRepo.lastID) + } + if res.SelectedProfileID != "explicit-prof" { + t.Fatalf("expected selected profile explicit-prof, got %q", res.SelectedProfileID) + } +} + +func TestRunnerRunPromptDefaultProfileIsUsedWhenNoExplicitProfileID(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + promptRepo.def.DefaultProfile = "from-prompt" + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "from-prompt": {ID: "from-prompt", Endpoint: "http://llm/v1", Model: "m"}, + }} + + runner := newMinimalRunner(promptRepo, execRepo) + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if execRepo.lastID != "from-prompt" { + t.Fatalf("expected prompt default profile lookup, got %q", execRepo.lastID) + } + if res.SelectedProfileID != "from-prompt" { + t.Fatalf("expected selected profile from-prompt, got %q", res.SelectedProfileID) + } +} + +func TestRunnerRunMissingExplicitProfileAndMissingDefaultProfileFails(t *testing.T) { + repo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + repo.def.DefaultProfile = "" + runner := newMinimalRunner(repo, &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}) + _, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", Inputs: singleInputRef()}) + if !errors.Is(err, ErrInvalidRequest) { + t.Fatalf("expected ErrInvalidRequest, got %v", err) + } +} + +func TestRunnerRunInvalidDefaultProfileFails(t *testing.T) { + repo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + repo.def.DefaultProfile = "does-not-exist" + runner := newMinimalRunner(repo, &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{}}) + _, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", Inputs: singleInputRef()}) + if !errors.Is(err, ErrProfileLoad) { + t.Fatalf("expected ErrProfileLoad, got %v", err) + } +} + +func TestRunnerRunExecutionProfileLoadFailure(t *testing.T) { + repo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + runner := newMinimalRunner(repo, &fakeExecutionProfileRepo{err: errors.New("load failed")}) + _, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()}) + if !errors.Is(err, ErrProfileLoad) { + t.Fatalf("expected profile load failure, got %v", err) + } +} + +func TestRunnerRunExplicitRuntimeOverrideBeatsSelectedProfileValue(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": { + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + Temperature: 0.2, + MaxTokens: 500, + TopP: 0.9, + TimeoutSeconds: 120, + ServiceTier: "priority", + }, + }} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: &domain.ExecutionTargetOverride{ + Endpoint: "http://override/v1", + Model: "override-model", + Temperature: float64Ptr(0.7), + TimeoutSeconds: intPtr(30), + ServiceTier: "flex", + }, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.Endpoint != "http://override/v1" || res.ModelName != "override-model" { + t.Fatalf("expected endpoint/model override to win, got endpoint=%q model=%q", res.Endpoint, res.ModelName) + } + if res.EffectiveModelParams.Temperature != 0.7 || res.EffectiveModelParams.TimeoutSeconds != 30 { + t.Fatalf("expected numeric override to win, got %+v", res.EffectiveModelParams) + } + if res.EffectiveModelParams.TopP != 0.9 { + t.Fatalf("expected non-overridden profile top_p to remain, got %v", res.EffectiveModelParams.TopP) + } + if res.EffectiveModelParams.ServiceTier != "flex" { + t.Fatalf("expected service_tier override to win, got %q", res.EffectiveModelParams.ServiceTier) + } +} + +func TestRunnerRunSelectedProfileBeatsBuiltInDefault(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": { + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + TopP: 0.8, + TimeoutSeconds: 90, + ServiceTier: "priority", + }, + }} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.EffectiveModelParams.TopP != 0.8 { + t.Fatalf("expected profile top_p to beat default, got %v", res.EffectiveModelParams.TopP) + } + if res.EffectiveModelParams.TimeoutSeconds != 90 { + t.Fatalf("expected profile timeout to beat default, got %d", res.EffectiveModelParams.TimeoutSeconds) + } + if res.EffectiveModelParams.ServiceTier != "priority" { + t.Fatalf("expected profile service_tier to beat default, got %q", res.EffectiveModelParams.ServiceTier) + } +} + +func TestRunnerRunBuiltInDefaultsUsedWhenProfileOmitsOptionalFields(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model"}, + }} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.EffectiveModelParams.Temperature != defaults.ExecutionDefaultTemperature { + t.Fatalf("expected default temperature %v, got %v", defaults.ExecutionDefaultTemperature, res.EffectiveModelParams.Temperature) + } + if res.EffectiveModelParams.TopP != defaults.ExecutionDefaultTopP { + t.Fatalf("expected default top_p %v, got %v", defaults.ExecutionDefaultTopP, res.EffectiveModelParams.TopP) + } + if res.EffectiveModelParams.MaxTokens != defaults.ExecutionDefaultMaxTokens { + t.Fatalf("expected default max_tokens %d, got %d", defaults.ExecutionDefaultMaxTokens, res.EffectiveModelParams.MaxTokens) + } + if res.EffectiveModelParams.TimeoutSeconds != defaults.ExecutionDefaultTimeoutSeconds { + t.Fatalf("expected default timeout_seconds %d, got %d", defaults.ExecutionDefaultTimeoutSeconds, res.EffectiveModelParams.TimeoutSeconds) + } +} + +func TestRunnerRunAPIKeyEnvResolvesFromEnvironment(t *testing.T) { + t.Setenv("PROMPTKIT_TEST_API_KEY", "secret") + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: "PROMPTKIT_TEST_API_KEY"}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()}) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.EffectiveModelParams.APIKeyEnv != "PROMPTKIT_TEST_API_KEY" { + t.Fatalf("expected api_key_env name in effective params, got %q", res.EffectiveModelParams.APIKeyEnv) + } +} + +func TestRunnerRunAPIKeyEnvMissingEnvironmentValueFailsClearly(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: "PROMPTKIT_MISSING_KEY"}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil) + + _, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()}) + if !errors.Is(err, ErrInvalidRequest) { + t.Fatalf("expected ErrInvalidRequest, got %v", err) + } + if !errors.Is(err, ErrAPIKeyEnvMissing) { + t.Fatalf("expected ErrAPIKeyEnvMissing, got %v", err) + } + if !strings.Contains(err.Error(), "PROMPTKIT_MISSING_KEY") { + t.Fatalf("expected missing env name in error, got %v", err) + } +} + +func TestRunnerRunDirectAPIKeyBypassesMissingEnvAndReachesLLM(t *testing.T) { + const directKey = "direct-runner-key" + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: "PROMPTKIT_MISSING_KEY"}, + }} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil) + + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + APIKey: directKey, + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if llmClient.lastReq.Target.APIKey != directKey { + t.Fatalf("expected direct API key to reach LLM request") + } + if llmClient.lastReq.Target.APIKeyEnv != "PROMPTKIT_MISSING_KEY" { + t.Fatalf("expected api_key_env name to remain on target, got %q", llmClient.lastReq.Target.APIKeyEnv) + } +} + +func TestRunnerPrepareAPIKeyRequiredFailsWithoutDirectKey(t *testing.T) { + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyRequired: true}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil) + + _, err := runner.Prepare(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrAPIKeyRequired) { + t.Fatalf("expected ErrAPIKeyRequired, got %v", err) + } +} + +func TestRunnerRunAPIKeyRequiredSucceedsWithDirectKey(t *testing.T) { + const directKey = "direct-required-key" + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyRequired: true}, + }} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), llmClient, nil) + + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + APIKey: directKey, + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if llmClient.lastReq.Target.APIKey != directKey { + t.Fatalf("expected direct API key to reach LLM request") + } + if !llmClient.lastReq.Target.APIKeyRequired { + t.Fatalf("expected APIKeyRequired to be carried to target") + } +} + +func TestRunnerRunRuntimeAPIKeyEnvOverrideWorks(t *testing.T) { + const envName = "PROMPTKIT_RUNTIME_API_KEY" + t.Setenv(envName, "runtime-secret") + + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model"}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: &domain.ExecutionTargetOverride{APIKeyEnv: envName}, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.EffectiveModelParams.APIKeyEnv != envName { + t.Fatalf("expected runtime api_key_env override in effective params, got %q", res.EffectiveModelParams.APIKeyEnv) + } +} + +func TestRunnerRunRuntimeAPIKeyEnvOverrideBeatsProfile(t *testing.T) { + const profileEnv = "PROMPTKIT_PROFILE_API_KEY" + const runtimeEnv = "PROMPTKIT_RUNTIME_API_KEY" + t.Setenv(runtimeEnv, "runtime-secret") + + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: profileEnv}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: &domain.ExecutionTargetOverride{APIKeyEnv: runtimeEnv}, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.EffectiveModelParams.APIKeyEnv != runtimeEnv { + t.Fatalf("expected runtime override to beat profile api_key_env, got %q", res.EffectiveModelParams.APIKeyEnv) + } +} + +func TestRunnerRunAPIKeyValueNotPresentInMetadata(t *testing.T) { + const envName = "PROMPTKIT_TEST_API_KEY" + const secret = "top-secret-value" + t.Setenv(envName, secret) + promptRepo := &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)} + execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", APIKeyEnv: envName}, + }} + runner := NewRunner(promptRepo, execRepo, defaultArtifactReader(), defaultRenderer(), &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, nil) + + res, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p", ProfileID: "exec", Inputs: singleInputRef()}) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.EffectiveModelParams.APIKeyEnv != envName { + t.Fatalf("expected api key env name, got %q", res.EffectiveModelParams.APIKeyEnv) + } + metadataDump := fmt.Sprintf("%+v|%s|%s|%s|%s", res.EffectiveModelParams, res.Endpoint, res.ModelName, res.PromptHash, res.RenderedPromptHash) + if strings.Contains(metadataDump, secret) { + t.Fatalf("unexpected api key value in metadata dump: %s", metadataDump) + } +} + +func TestRunnerRunPromptLoadFailure(t *testing.T) { + runner := NewRunner(&fakePromptRepo{err: errors.New("boom")}, &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, defaultArtifactReader(), defaultRenderer(), &fakeLLM{}, nil) + _, err := runner.Run(context.Background(), domain.RunRequest{PromptID: "p"}) + if !errors.Is(err, ErrPromptLoad) { + t.Fatalf("expected ErrPromptLoad, got %v", err) + } + if errors.Is(err, ErrProfileLoad) { + t.Fatalf("did not expect ErrProfileLoad, got %v", err) + } +} + +func TestRunnerRunArtifactLoadFailure(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + &fakeArtifactReader{errByURI: map[string]error{"a://bad": errors.New("read failed")}}, + &fakeRenderer{rendered: &domain.RenderedPrompt{}}, + &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, + nil, + ) + + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://bad"}}, + }) + if !errors.Is(err, ErrArtifactLoad) { + t.Fatalf("expected ErrArtifactLoad, got %v", err) + } +} + +func TestRunnerRunPromptRenderFailure(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + &fakeRenderer{err: errors.New("render failed")}, + &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, + nil, + ) + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrPromptRender) { + t.Fatalf("expected ErrPromptRender, got %v", err) + } +} + +func TestRunnerRunLLMFailure(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{err: errors.New("llm failed")}, + nil, + ) + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrLLMGenerate) { + t.Fatalf("expected ErrLLMGenerate, got %v", err) + } +} + +func TestRunnerRunCancellationPreservesGenerationCategory(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{resp: &domain.GenerateResponse{Content: "ignored"}}, + nil, + ) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + _, err := runner.Run(ctx, domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrLLMGenerate) { + t.Fatalf("expected ErrLLMGenerate, got %v", err) + } + if !errors.Is(err, context.Canceled) { + t.Fatalf("expected context cancellation identity, got %v", err) + } +} + +func TestRunnerRunLLMInvalidRequestMapsToUsecaseInvalidRequest(t *testing.T) { + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationNone, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{err: llm.ErrInvalidRequest}, + nil, + ) + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if !errors.Is(err, ErrInvalidRequest) { + t.Fatalf("expected ErrInvalidRequest, got %v", err) + } + if errors.Is(err, ErrLLMGenerate) { + t.Fatalf("did not expect ErrLLMGenerate, got %v", err) + } +} + +func TestRunnerRunValidationStillWorks(t *testing.T) { + validator := &fakeValidator{result: domain.ValidationResult{Status: domain.ValidationFailed, Mode: domain.ValidationBasic, Errors: []string{"bad"}, IsValid: false}} + runner := NewRunner( + &fakePromptRepo{def: promptDef(domain.FormatText, domain.ValidationBasic, 0)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{resp: &domain.GenerateResponse{Content: "raw output"}}, + validator, + ) + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if res.Validation.Status != domain.ValidationFailed || res.RawOutput != "raw output" { + t.Fatalf("unexpected validation/raw output: %+v", res) + } +} + +func TestRunnerRunStructuredRepairRemainsBoundedAndUsesEffectiveModelSettings(t *testing.T) { + repairer := &fakeRepairer{responses: []*domain.GenerateResponse{{Content: `{"broken":`}, {Content: `{"still":`}}} + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: `{"initial":`}} + + runner := NewRunnerWithRepairer( + &fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{ + "exec": {ID: "exec", Endpoint: "http://profile/v1", Model: "profile-model", TimeoutSeconds: 55}, + }}, + defaultArtifactReader(), + defaultRenderer(), + llmClient, + validate.NewStandardValidator("."), + repairer, + ) + res, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + Execution: &domain.ExecutionTargetOverride{Endpoint: "http://override/v1", Model: "override-model", TimeoutSeconds: intPtr(22)}, + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if repairer.calls != 1 || res.Validation.RepairAttempts != 1 { + t.Fatalf("expected one bounded repair, calls=%d attempts=%d", repairer.calls, res.Validation.RepairAttempts) + } + if len(repairer.reqs) != 1 { + t.Fatalf("expected one repair request, got %d", len(repairer.reqs)) + } + if repairer.reqs[0].Target.Endpoint != "http://override/v1" || repairer.reqs[0].Target.Model != "override-model" { + t.Fatalf("expected repair to use effective target, got %+v", repairer.reqs[0].Target) + } + if repairer.reqs[0].Target.TimeoutSeconds != 22 { + t.Fatalf("expected repair to use effective timeout, got %d", repairer.reqs[0].Target.TimeoutSeconds) + } +} + +func TestRunnerRunJSONSchemaRepairCarriesStructuredOutputSpec(t *testing.T) { + def := promptDef(domain.FormatJSON, domain.ValidationJSONSchema, 1) + def.Validation.SchemaPath = "events.schema.json" + + validator := &fakeValidator{ + result: domain.ValidationResult{ + Status: domain.ValidationFailed, + Mode: domain.ValidationJSONSchema, + Errors: []string{"schema mismatch"}, + IsValid: false, + }, + schemaDoc: map[string]any{ + "type": "object", + "properties": map[string]any{ + "events": map[string]any{"type": "array"}, + }, + }, + } + repairer := &fakeRepairer{ + responses: []*domain.GenerateResponse{ + {Content: `{"events":[]}`}, + }, + } + llmClient := &fakeLLM{resp: &domain.GenerateResponse{Content: `{"events":[1]}`}} + runner := NewRunnerWithRepairer( + &fakePromptRepo{def: def}, + &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}, + defaultArtifactReader(), + defaultRenderer(), + llmClient, + validator, + repairer, + ) + + _, err := runner.Run(context.Background(), domain.RunRequest{ + PromptID: "p", + ProfileID: "exec", + Inputs: singleInputRef(), + }) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if llmClient.lastReq.StructuredOutput == nil || llmClient.lastReq.StructuredOutput.JSONSchema == nil { + t.Fatalf("expected initial llm request to include structured output, got %+v", llmClient.lastReq.StructuredOutput) + } + if len(repairer.reqs) != 1 { + t.Fatalf("expected one repair request, got %d", len(repairer.reqs)) + } + if repairer.reqs[0].StructuredOutput == nil || repairer.reqs[0].StructuredOutput.JSONSchema == nil { + t.Fatalf("expected repair request structured output, got %+v", repairer.reqs[0].StructuredOutput) + } + if repairer.reqs[0].StructuredOutput.JSONSchema.Name != "p_1" { + t.Fatalf("expected derived schema name p_1, got %q", repairer.reqs[0].StructuredOutput.JSONSchema.Name) + } +} + +func TestExecutionProfileToTargetPopulatesAllFieldsAndCopiesExtraParams(t *testing.T) { + src := &domain.ExecutionProfile{ + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + Temperature: 0.2, + MaxTokens: 123, + TopP: 0.75, + TimeoutSeconds: 90, + ServiceTier: "priority", + ReasoningEffort: "medium", + APIKeyEnv: "PROMPTKIT_API_KEY", + APIKeyRequired: true, + ExtraParams: map[string]any{ + "provider_option": "on", + }, + } + + target := executionProfileToTarget(src) + if target.Endpoint != src.Endpoint || + target.Model != src.Model || + target.Temperature != src.Temperature || + target.MaxTokens != src.MaxTokens || + target.TopP != src.TopP || + target.TimeoutSeconds != src.TimeoutSeconds || + target.ServiceTier != src.ServiceTier || + target.ReasoningEffort != src.ReasoningEffort || + target.APIKeyEnv != src.APIKeyEnv || + target.APIKeyRequired != src.APIKeyRequired { + t.Fatalf("expected all profile fields to populate target, got %+v", target) + } + if !reflect.DeepEqual(target.ExtraParams, src.ExtraParams) { + t.Fatalf("expected extra_params to match, got %#v", target.ExtraParams) + } + + src.ExtraParams["provider_option"] = "changed" + if target.ExtraParams["provider_option"] != "on" { + t.Fatalf("expected extra_params copy to be independent, got %#v", target.ExtraParams) + } +} + +func TestResolveExecutionTargetProfileValuesPopulateAllSupportedFields(t *testing.T) { + profileValue := &domain.ExecutionProfile{ + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + Temperature: 0.3, + MaxTokens: 222, + TopP: 0.6, + TimeoutSeconds: 77, + ServiceTier: "priority", + ReasoningEffort: "low", + APIKeyEnv: "PROFILE_KEY", + APIKeyRequired: true, + ExtraParams: map[string]any{ + "profile_option": "enabled", + }, + } + + target, presence, err := resolveExecutionTarget(profileValue, nil) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if presence != (domain.ExecutionTargetPresence{}) { + t.Fatalf("expected no request override presence, got %+v", presence) + } + if target.Endpoint != profileValue.Endpoint || + target.Model != profileValue.Model || + target.Temperature != profileValue.Temperature || + target.MaxTokens != profileValue.MaxTokens || + target.TopP != profileValue.TopP || + target.TimeoutSeconds != profileValue.TimeoutSeconds || + target.ServiceTier != profileValue.ServiceTier || + target.ReasoningEffort != profileValue.ReasoningEffort || + target.APIKeyEnv != profileValue.APIKeyEnv || + target.APIKeyRequired != profileValue.APIKeyRequired { + t.Fatalf("expected profile values to populate target, got %+v", target) + } + if !reflect.DeepEqual(target.ExtraParams, profileValue.ExtraParams) { + t.Fatalf("expected profile extra_params in target, got %#v", target.ExtraParams) + } +} + +func TestResolveExecutionTargetRuntimeOverridesBeatProfileForAllOverrideableFields(t *testing.T) { + profileValue := &domain.ExecutionProfile{ + ID: "exec", + Endpoint: "http://profile/v1", + Model: "profile-model", + Temperature: 0.2, + MaxTokens: 200, + TopP: 0.8, + TimeoutSeconds: 90, + ServiceTier: "priority", + ReasoningEffort: "medium", + APIKeyEnv: "PROFILE_KEY", + ExtraParams: map[string]any{ + "profile_only": "yes", + }, + } + override := &domain.ExecutionTargetOverride{ + Endpoint: "http://override/v1", + Model: "override-model", + Temperature: float64Ptr(0.9), + MaxTokens: intPtr(111), + TopP: float64Ptr(0.5), + TimeoutSeconds: intPtr(30), + ServiceTier: "flex", + ReasoningEffort: "high", + APIKeyEnv: "RUNTIME_KEY", + ExtraParams: map[string]any{ + "runtime_only": "yes", + }, + } + + target, presence, err := resolveExecutionTarget(profileValue, override) + if err != nil { + t.Fatalf("expected no error, got %v", err) + } + if presence != (domain.ExecutionTargetPresence{Temperature: true, MaxTokens: true, TopP: true, TimeoutSeconds: true}) { + t.Fatalf("unexpected override presence: %+v", presence) + } + if target.Endpoint != override.Endpoint || + target.Model != override.Model || + target.Temperature != *override.Temperature || + target.MaxTokens != *override.MaxTokens || + target.TopP != *override.TopP || + target.TimeoutSeconds != *override.TimeoutSeconds || + target.ServiceTier != override.ServiceTier || + target.ReasoningEffort != override.ReasoningEffort || + target.APIKeyEnv != override.APIKeyEnv { + t.Fatalf("expected runtime overrides to win for all fields, got %+v", target) + } + if !reflect.DeepEqual(target.ExtraParams, override.ExtraParams) { + t.Fatalf("expected runtime extra_params to replace profile extra_params, got %#v", target.ExtraParams) + } +} + +func TestMergeExecutionTargetEmptyStringOverridesDoNotErase(t *testing.T) { + base := domain.ExecutionTarget{ + Endpoint: "http://base/v1", + Model: "base-model", + ServiceTier: "priority", + ReasoningEffort: "medium", + APIKeyEnv: "BASE_KEY", + } + override := domain.ExecutionTarget{ + Endpoint: "http://override/v1", + Model: "override-model", + ServiceTier: " ", + ReasoningEffort: " ", + APIKeyEnv: "", + } + + merged := mergeExecutionTarget(base, override) + if merged.Endpoint != "http://override/v1" || merged.Model != "override-model" { + t.Fatalf("expected endpoint/model to override, got %+v", merged) + } + if merged.ServiceTier != "priority" { + t.Fatalf("expected empty service_tier override to be ignored, got %q", merged.ServiceTier) + } + if merged.ReasoningEffort != "medium" { + t.Fatalf("expected empty reasoning_effort override to be ignored, got %q", merged.ReasoningEffort) + } + if merged.APIKeyEnv != "BASE_KEY" { + t.Fatalf("expected empty api_key_env override to be ignored, got %q", merged.APIKeyEnv) + } +} + +func TestMergeExecutionTargetEmptyExtraParamsDoesNotErase(t *testing.T) { + base := domain.ExecutionTarget{ + ExtraParams: map[string]any{ + "keep": "value", + }, + } + override := domain.ExecutionTarget{ + ExtraParams: map[string]any{}, + } + + merged := mergeExecutionTarget(base, override) + if !reflect.DeepEqual(merged.ExtraParams, base.ExtraParams) { + t.Fatalf("expected empty extra_params override not to erase base values, got %#v", merged.ExtraParams) + } +} + +func TestBuildOutputArtifactDefaults(t *testing.T) { + tests := []struct { + name string + format domain.OutputFormat + contentType string + }{ + {name: "text", format: domain.FormatText, contentType: defaults.ContentTypeTextPlain}, + {name: "markdown", format: domain.FormatMarkdown, contentType: defaults.ContentTypeTextMarkdown}, + {name: "json", format: domain.FormatJSON, contentType: defaults.ContentTypeApplicationJSON}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + art := buildOutputArtifact("body", tc.format) + if art.Name != defaults.OutputArtifactName { + t.Fatalf("expected artifact name %q, got %q", defaults.OutputArtifactName, art.Name) + } + if art.ContentType != tc.contentType { + t.Fatalf("expected content type %q, got %q", tc.contentType, art.ContentType) + } + }) + } +} + +func promptDef(format domain.OutputFormat, mode domain.ValidationMode, attempts int) *domain.PromptDefinition { + return &domain.PromptDefinition{ + ID: "p", + Version: "1", + DefaultProfile: "exec", + Inputs: []domain.PromptInput{{Name: "transcript", Required: true}}, + Templates: []domain.PromptMessageTemplate{{Role: "user", Content: "x"}}, + OutputFormat: format, + Validation: domain.OutputContract{ + ValidationMode: mode, + RepairAttempts: attempts, + Format: format, + }, + } +} + +func hashString(s string) string { + sum := sha256.Sum256([]byte(s)) + return hex.EncodeToString(sum[:]) +} + +func defaultExecutionProfile() *domain.ExecutionProfile { + return &domain.ExecutionProfile{ + ID: "exec", + Endpoint: "http://llm/v1", + Model: "model-from-profile", + } +} + +func defaultArtifactReader() *fakeArtifactReader { + return &fakeArtifactReader{artifactsByURI: map[string]*domain.Artifact{ + "a://ok": {Body: []byte("x"), Hash: hashString("x")}, + }} +} + +func defaultRenderer() *fakeRenderer { + return &fakeRenderer{rendered: &domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hello"}}}} +} + +func singleInputRef() map[string]domain.ArtifactRef { + return map[string]domain.ArtifactRef{"transcript": {Type: domain.ArtifactRefFile, URI: "a://ok"}} +} + +func float64Ptr(v float64) *float64 { + return &v +} + +func intPtr(v int) *int { + return &v +} + +func newMinimalRunner(promptRepo *fakePromptRepo, execRepo *fakeExecutionProfileRepo) *Runner { + return NewRunner( + promptRepo, + execRepo, + defaultArtifactReader(), + defaultRenderer(), + &fakeLLM{resp: &domain.GenerateResponse{Content: "ok"}}, + nil, + ) +}