Add internal framework orchestration
This commit is contained in:
@@ -11,10 +11,10 @@ Framework extraction is in progress. The repository now contains the
|
||||
`internal/domain` model, application-neutral `internal/defaults`, and
|
||||
`internal/filecatalog` helpers that form the implementation foundation. It also
|
||||
contains internal prompt-definition and profile repositories, the embedded
|
||||
built-in profile catalog, and Go-template prompt rendering. These internal
|
||||
packages are not a consumer API, and the root package does not yet provide a
|
||||
usable public framework API, so there is no installation or usage example at
|
||||
this time.
|
||||
built-in profile catalog, artifact loading, Go-template rendering, output
|
||||
validation, model generation, and orchestration. These internal packages are
|
||||
not a consumer API, and the root package does not yet provide a usable public
|
||||
framework API, so there is no installation or usage example at this time.
|
||||
|
||||
Contributors should start with the [development guide](docs/development.md).
|
||||
The [architecture policy](docs/policy/architecture.md) defines the library
|
||||
|
||||
@@ -22,10 +22,11 @@ contributor workflow and validation.
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources. | [Internal sources and validation](sources.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests, response decoding, authentication, and deadline handling. | [Internal model client](llm.md) |
|
||||
| `internal/usecase` | Coordinates preparation and execution across internal sources, rendering, artifact loading, generation, validation, and optional repair. | [Internal runner](runner.md) |
|
||||
|
||||
These packages provide the internal model, source, rendering, validation, and
|
||||
model-client foundation. Orchestration and a usable public engine are not
|
||||
implemented in Promptkit yet.
|
||||
model-client workflow. A usable public engine is not implemented in Promptkit
|
||||
yet.
|
||||
|
||||
## Maintenance
|
||||
|
||||
|
||||
83
docs/internal/runner.md
Normal file
83
docs/internal/runner.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Internal Runner
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes Promptkit's implemented internal orchestration. The
|
||||
[architecture policy](../policy/architecture.md) owns dependency and consumer
|
||||
boundaries. The [source and validation document](sources.md) owns repository,
|
||||
artifact, rendering, and validation behavior, while the
|
||||
[model-client document](llm.md) owns generation behavior and failure
|
||||
categories.
|
||||
|
||||
The runner remains under `internal/usecase`. The root package does not yet
|
||||
assemble it into a usable public engine.
|
||||
|
||||
## Collaborators
|
||||
|
||||
`Runner` coordinates narrow internal interfaces for prompt definitions,
|
||||
profiles, artifacts, rendering, model generation, and validation. Schema
|
||||
documents are loaded through the validator's optional schema-loader interface.
|
||||
An output repairer can be injected internally, but the ordinary runner
|
||||
constructor does not enable one.
|
||||
|
||||
Each invocation carries its state in request, prepared-run, and result values.
|
||||
The runner has no durable run or session store.
|
||||
|
||||
## Preparation Flow
|
||||
|
||||
`Prepare` performs the reusable pre-generation workflow:
|
||||
|
||||
1. validate the prompt selection and load the prompt definition;
|
||||
2. hash the loaded definition;
|
||||
3. select the request profile or the prompt's default profile;
|
||||
4. resolve application-neutral defaults, profile values, and explicit request
|
||||
overrides in that order;
|
||||
5. validate endpoint, model, numeric overrides, and credential requirements;
|
||||
6. resolve the output contract and load a structured-output schema when
|
||||
required;
|
||||
7. load and hash input artifacts;
|
||||
8. render and hash the prompt; and
|
||||
9. return the effective settings, source identities, messages, hashes, and
|
||||
preparation timing.
|
||||
|
||||
Pointer-based numeric overrides preserve an explicit zero. Invalid negative or
|
||||
out-of-range values fail as invalid requests. A direct API key takes
|
||||
precedence over environment lookup for execution; secret values remain
|
||||
excluded from serialized metadata.
|
||||
|
||||
## Run Flow
|
||||
|
||||
`Run` calls `Prepare` rather than maintaining a second preparation path. It
|
||||
performs one initial generation call, builds the named output artifact, and
|
||||
validates that artifact. Invalid generated content remains a validation result;
|
||||
an inability to perform validation is an operational error.
|
||||
|
||||
When an internal repairer is present, a JSON or JSON Schema content failure can
|
||||
trigger bounded repair attempts. Repair receives the effective execution
|
||||
target, validation errors, prior output, and structured-output specification.
|
||||
This capability remains internal and is not a public option.
|
||||
|
||||
A successful result includes the output artifact and raw output, validation
|
||||
state, prompt and rendered-prompt hashes, selected profile, effective settings,
|
||||
input hashes, token usage, a generated run identifier, and UTC timing.
|
||||
|
||||
## Failure Categories
|
||||
|
||||
Package errors distinguish invalid requests, required profile selection,
|
||||
credential failures, and prompt, profile, artifact, rendering, generation, and
|
||||
validation failures. Wrapping preserves the package identities needed by the
|
||||
future facade and retains collaborator identities where they are part of the
|
||||
internal contract. Context cancellation propagates through the invoked
|
||||
collaborator and is classified by the owning operation.
|
||||
|
||||
## Test Ownership And Changes
|
||||
|
||||
The [runner tests](../../internal/usecase/runner_test.go) own preparation order,
|
||||
selection and override precedence, schema-before-generation behavior, hashing,
|
||||
generation and validation outcomes, bounded repair, credentials and redaction,
|
||||
error categories, artifact metadata, usage, and timing.
|
||||
|
||||
Changes to orchestration should continue to use the existing package
|
||||
interfaces, keep request state local to an invocation, and preserve `Run`'s use
|
||||
of `Prepare`. Source, renderer, validator, or model-client contract changes
|
||||
belong first in their owning package and document.
|
||||
@@ -34,9 +34,11 @@ The implemented internal components consist of:
|
||||
- `internal/artifact`, which resolves ordinary inline and unrestricted
|
||||
caller-selected file references;
|
||||
- `internal/validate`, which validates basic, JSON, and JSON Schema output
|
||||
using filesystem and `fs.FS` schema sources; and
|
||||
using filesystem and `fs.FS` schema sources;
|
||||
- `internal/llm`, which defines the provider-neutral generation boundary and
|
||||
implements outbound OpenAI-compatible chat requests.
|
||||
implements outbound OpenAI-compatible chat requests; and
|
||||
- `internal/usecase`, which coordinates preparation and execution across the
|
||||
internal framework components.
|
||||
|
||||
The defaults and renderer depend on the domain model. Prompt-definition and
|
||||
profile repositories use the domain model, file catalog, and YAML decoder. The
|
||||
@@ -44,8 +46,9 @@ built-in profile repository supplies an embedded `fs.FS` to the profile
|
||||
package. Artifact reading uses the domain model and application-neutral
|
||||
defaults. Validation uses the domain model, file catalog, and JSON Schema
|
||||
implementation. The model client uses the domain model, application-neutral
|
||||
defaults, and an injected or standard-library HTTP client. Orchestration and
|
||||
the public engine have not yet been extracted.
|
||||
defaults, and an injected or standard-library HTTP client. The use-case runner
|
||||
depends on the narrow interfaces owned by each internal component. The public
|
||||
engine has not yet been extracted.
|
||||
|
||||
Future framework extraction must follow this dependency direction:
|
||||
|
||||
|
||||
76
internal/usecase/repairer.go
Normal file
76
internal/usecase/repairer.go
Normal file
@@ -0,0 +1,76 @@
|
||||
package usecase
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
|
||||
)
|
||||
|
||||
type OutputRepairer interface {
|
||||
Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error)
|
||||
}
|
||||
|
||||
type RepairRequest struct {
|
||||
PreviousOutput string
|
||||
ValidationErrors []string
|
||||
Target domain.ExecutionTarget
|
||||
StructuredOutput *domain.StructuredOutputSpec
|
||||
Attempt int
|
||||
MaxAttempts int
|
||||
Mode domain.ValidationMode
|
||||
}
|
||||
|
||||
type defaultOutputRepairer struct {
|
||||
llm llm.Client
|
||||
}
|
||||
|
||||
func NewDefaultOutputRepairer(llmClient llm.Client) OutputRepairer {
|
||||
return &defaultOutputRepairer{llm: llmClient}
|
||||
}
|
||||
|
||||
func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error) {
|
||||
if r.llm == nil {
|
||||
return nil, errors.New("llm client is required for repair")
|
||||
}
|
||||
|
||||
errs := "(none provided)"
|
||||
if len(req.ValidationErrors) > 0 {
|
||||
errs = strings.Join(req.ValidationErrors, "\n")
|
||||
}
|
||||
|
||||
prompt := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
|
||||
{
|
||||
Role: "system",
|
||||
Content: "You repair invalid JSON output. Return only corrected JSON. Do not include explanations or markdown code fences.",
|
||||
},
|
||||
{
|
||||
Role: "user",
|
||||
Content: fmt.Sprintf(
|
||||
"Repair attempt %d of %d for validation mode %s.\n\nValidation errors:\n%s\n\nPrevious output:\n%s\n\nReturn only corrected JSON.",
|
||||
req.Attempt,
|
||||
req.MaxAttempts,
|
||||
req.Mode,
|
||||
errs,
|
||||
req.PreviousOutput,
|
||||
),
|
||||
},
|
||||
}}
|
||||
|
||||
resp, err := r.llm.Generate(ctx, domain.GenerateRequest{
|
||||
Prompt: prompt,
|
||||
Target: req.Target,
|
||||
StructuredOutput: req.StructuredOutput,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp == nil {
|
||||
return nil, errors.New("repair llm returned nil response")
|
||||
}
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
576
internal/usecase/runner.go
Normal file
576
internal/usecase/runner.go
Normal file
@@ -0,0 +1,576 @@
|
||||
package usecase
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/artifact"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/defaults"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/profile"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/prompt"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/promptdef"
|
||||
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrInvalidRequest = errors.New("invalid run request")
|
||||
ErrProfileRequired = errors.New("profile selection is required")
|
||||
ErrAPIKeyEnvMissing = errors.New("api_key_env points to an unset environment variable")
|
||||
ErrAPIKeyRequired = errors.New("api key is required")
|
||||
ErrPromptLoad = errors.New("failed to load prompt definition")
|
||||
ErrProfileLoad = errors.New("failed to load execution profile")
|
||||
ErrArtifactLoad = errors.New("failed to load artifact")
|
||||
ErrPromptRender = errors.New("failed to render prompt")
|
||||
ErrLLMGenerate = errors.New("failed to generate output")
|
||||
ErrValidation = errors.New("failed to validate output")
|
||||
)
|
||||
|
||||
// Runner executes the Promptkit core use case.
|
||||
type Runner struct {
|
||||
promptDefs promptdef.Repository
|
||||
profiles profile.Repository
|
||||
artifacts artifact.Reader
|
||||
renderer prompt.Renderer
|
||||
llm llm.Client
|
||||
validator validate.Validator
|
||||
repairer OutputRepairer
|
||||
}
|
||||
|
||||
func NewRunner(
|
||||
promptDefs promptdef.Repository,
|
||||
profiles profile.Repository,
|
||||
artifacts artifact.Reader,
|
||||
renderer prompt.Renderer,
|
||||
llmClient llm.Client,
|
||||
validator validate.Validator,
|
||||
) *Runner {
|
||||
return NewRunnerWithRepairer(promptDefs, profiles, artifacts, renderer, llmClient, validator, nil)
|
||||
}
|
||||
|
||||
func NewRunnerWithRepairer(
|
||||
promptDefs promptdef.Repository,
|
||||
profiles profile.Repository,
|
||||
artifacts artifact.Reader,
|
||||
renderer prompt.Renderer,
|
||||
llmClient llm.Client,
|
||||
validator validate.Validator,
|
||||
repairer OutputRepairer,
|
||||
) *Runner {
|
||||
return &Runner{
|
||||
promptDefs: promptDefs,
|
||||
profiles: profiles,
|
||||
artifacts: artifacts,
|
||||
renderer: renderer,
|
||||
llm: llmClient,
|
||||
validator: validator,
|
||||
repairer: repairer,
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Runner) Run(ctx context.Context, req domain.RunRequest) (*domain.RunResult, error) {
|
||||
runID, err := newRunID()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create run id: %w", err)
|
||||
}
|
||||
|
||||
start := time.Now().UTC()
|
||||
|
||||
prepared, err := r.Prepare(ctx, req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
genResp, err := r.llm.Generate(ctx, domain.GenerateRequest{
|
||||
Prompt: domain.RenderedPrompt{SessionID: prepared.SessionID, Messages: prepared.Messages},
|
||||
Target: prepared.EffectiveModelParams,
|
||||
TargetPresence: prepared.TargetPresence,
|
||||
StructuredOutput: prepared.StructuredOutput,
|
||||
})
|
||||
if err != nil {
|
||||
if errors.Is(err, llm.ErrInvalidRequest) {
|
||||
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
|
||||
}
|
||||
return nil, fmt.Errorf("%w: %w", ErrLLMGenerate, err)
|
||||
}
|
||||
|
||||
outputArtifact := buildOutputArtifact(genResp.Content, prepared.OutputContract.Format)
|
||||
validationResult, err := r.validateOutput(ctx, &outputArtifact, prepared.OutputContract, 0)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrValidation, err)
|
||||
}
|
||||
|
||||
if r.shouldAttemptRepair(prepared.OutputContract, validationResult) {
|
||||
attemptsUsed := 0
|
||||
for attemptsUsed < prepared.OutputContract.RepairAttempts && validationResult.Status == domain.ValidationFailed {
|
||||
attemptsUsed++
|
||||
|
||||
repairResp, repairErr := r.repairer.Repair(ctx, RepairRequest{
|
||||
PreviousOutput: genResp.Content,
|
||||
ValidationErrors: validationResult.Errors,
|
||||
Target: prepared.EffectiveModelParams,
|
||||
StructuredOutput: prepared.StructuredOutput,
|
||||
Attempt: attemptsUsed,
|
||||
MaxAttempts: prepared.OutputContract.RepairAttempts,
|
||||
Mode: prepared.OutputContract.ValidationMode,
|
||||
})
|
||||
if repairErr != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrValidation, repairErr)
|
||||
}
|
||||
if repairResp == nil {
|
||||
return nil, fmt.Errorf("%w: repairer returned nil response", ErrValidation)
|
||||
}
|
||||
|
||||
genResp = repairResp
|
||||
outputArtifact = buildOutputArtifact(genResp.Content, prepared.OutputContract.Format)
|
||||
|
||||
validationResult, err = r.validateOutput(ctx, &outputArtifact, prepared.OutputContract, attemptsUsed)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrValidation, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
end := time.Now().UTC()
|
||||
|
||||
return &domain.RunResult{
|
||||
RunID: runID,
|
||||
Artifact: outputArtifact,
|
||||
RawOutput: genResp.Content,
|
||||
Validation: validationResult,
|
||||
PromptID: prepared.PromptID,
|
||||
PromptVersion: prepared.PromptVersion,
|
||||
PromptHash: prepared.PromptHash,
|
||||
RenderedPromptHash: prepared.RenderedPromptHash,
|
||||
SelectedProfileID: prepared.SelectedProfileID,
|
||||
ModelName: prepared.EffectiveModelParams.Model,
|
||||
Endpoint: prepared.EffectiveModelParams.Endpoint,
|
||||
EffectiveModelParams: prepared.EffectiveModelParams,
|
||||
InputHashes: prepared.InputHashes,
|
||||
Usage: genResp.Usage,
|
||||
StartTime: start,
|
||||
EndTime: end,
|
||||
Duration: end.Sub(start),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *Runner) Prepare(ctx context.Context, req domain.RunRequest) (*domain.PreparedRun, error) {
|
||||
if strings.TrimSpace(req.PromptID) == "" {
|
||||
return nil, fmt.Errorf("%w: prompt id is required", ErrInvalidRequest)
|
||||
}
|
||||
|
||||
start := time.Now().UTC()
|
||||
|
||||
def, err := r.promptDefs.GetPromptDefinition(ctx, req.PromptID, req.PromptVersion)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrPromptLoad, err)
|
||||
}
|
||||
promptDefinitionHash, err := hashPromptDefinition(def)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: failed to hash prompt definition: %v", ErrPromptLoad, err)
|
||||
}
|
||||
|
||||
selectedProfileID := strings.TrimSpace(req.ProfileID)
|
||||
if selectedProfileID == "" {
|
||||
selectedProfileID = strings.TrimSpace(def.DefaultProfile)
|
||||
}
|
||||
if selectedProfileID == "" {
|
||||
return nil, fmt.Errorf("%w: %w: profile id is required either in request or prompt default_profile", ErrInvalidRequest, ErrProfileRequired)
|
||||
}
|
||||
|
||||
execProfile, err := r.profiles.GetProfile(ctx, selectedProfileID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrProfileLoad, err)
|
||||
}
|
||||
|
||||
effectiveModel, targetPresence, err := resolveExecutionTarget(execProfile, req.Execution)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
|
||||
}
|
||||
effectiveModel.APIKey = req.APIKey
|
||||
if strings.TrimSpace(effectiveModel.Endpoint) == "" {
|
||||
return nil, fmt.Errorf("%w: execution endpoint is required", ErrInvalidRequest)
|
||||
}
|
||||
if strings.TrimSpace(effectiveModel.Model) == "" {
|
||||
return nil, fmt.Errorf("%w: execution model is required", ErrInvalidRequest)
|
||||
}
|
||||
if err := validateAPIKey(effectiveModel.APIKeyEnv, effectiveModel.APIKey, effectiveModel.APIKeyRequired); err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
|
||||
}
|
||||
|
||||
effectiveContract := resolveOutputContract(def, req.Validation)
|
||||
structuredOutput, err := r.resolveStructuredOutput(ctx, def, effectiveContract)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
resolvedInputs := make(map[string]*domain.Artifact, len(req.Inputs))
|
||||
inputHashes := make(map[string]string, len(req.Inputs))
|
||||
for name, ref := range req.Inputs {
|
||||
art, readErr := r.artifacts.Read(ctx, ref)
|
||||
if readErr != nil {
|
||||
return nil, fmt.Errorf("%w: input %q: %w", ErrArtifactLoad, name, readErr)
|
||||
}
|
||||
if art.Name == "" {
|
||||
art.Name = name
|
||||
}
|
||||
resolvedInputs[name] = art
|
||||
inputHashes[name] = art.Hash
|
||||
}
|
||||
|
||||
renderedPrompt, err := r.renderer.Render(ctx, def, resolvedInputs, req.Vars)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrPromptRender, err)
|
||||
}
|
||||
|
||||
end := time.Now().UTC()
|
||||
return &domain.PreparedRun{
|
||||
PromptID: def.ID,
|
||||
PromptVersion: def.Version,
|
||||
PromptHash: promptDefinitionHash,
|
||||
SelectedProfileID: selectedProfileID,
|
||||
EffectiveModelParams: effectiveModel,
|
||||
TargetPresence: targetPresence,
|
||||
OutputContract: effectiveContract,
|
||||
StructuredOutput: structuredOutput,
|
||||
InputHashes: inputHashes,
|
||||
SessionID: renderedPrompt.SessionID,
|
||||
RenderedPromptHash: hashRenderedPrompt(*renderedPrompt),
|
||||
Messages: renderedPrompt.Messages,
|
||||
StartTime: start,
|
||||
EndTime: end,
|
||||
DurationMS: end.Sub(start).Milliseconds(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *Runner) resolveStructuredOutput(ctx context.Context, def *domain.PromptDefinition, contract domain.OutputContract) (*domain.StructuredOutputSpec, error) {
|
||||
if contract.ValidationMode != domain.ValidationJSONSchema {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
loader, ok := r.validator.(validate.SchemaDocumentLoader)
|
||||
if !ok || loader == nil {
|
||||
return nil, fmt.Errorf("%w: json_schema output requires schema document loader", ErrValidation)
|
||||
}
|
||||
|
||||
schemaDoc, err := loader.LoadSchemaDocument(ctx, contract.SchemaPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: failed to load json schema for structured output: %v", ErrValidation, err)
|
||||
}
|
||||
|
||||
return &domain.StructuredOutputSpec{
|
||||
Type: domain.StructuredOutputJSONSchema,
|
||||
JSONSchema: &domain.StructuredOutputJSONSpec{
|
||||
Name: deriveStructuredSchemaName(def.ID, def.Version),
|
||||
Strict: true,
|
||||
Schema: schemaDoc,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func deriveStructuredSchemaName(promptID string, promptVersion string) string {
|
||||
raw := strings.TrimSpace(promptID)
|
||||
if v := strings.TrimSpace(promptVersion); v != "" {
|
||||
if raw == "" {
|
||||
raw = v
|
||||
} else {
|
||||
raw = raw + "_" + v
|
||||
}
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
for _, r := range raw {
|
||||
if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_' || r == '-' {
|
||||
b.WriteRune(r)
|
||||
} else {
|
||||
b.WriteRune('_')
|
||||
}
|
||||
}
|
||||
|
||||
name := strings.Trim(b.String(), "_-")
|
||||
if name == "" {
|
||||
return "promptkit_schema"
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
func (r *Runner) validateOutput(ctx context.Context, artifact *domain.Artifact, contract domain.OutputContract, attemptsUsed int) (domain.ValidationResult, error) {
|
||||
if r.validator == nil || contract.ValidationMode == domain.ValidationNone {
|
||||
return domain.ValidationResult{
|
||||
Status: domain.ValidationSkipped,
|
||||
Mode: contract.ValidationMode,
|
||||
SchemaPath: contract.SchemaPath,
|
||||
RepairAttempts: attemptsUsed,
|
||||
IsValid: true,
|
||||
}, nil
|
||||
}
|
||||
|
||||
res, err := r.validator.Validate(ctx, artifact, contract)
|
||||
if err != nil {
|
||||
return domain.ValidationResult{}, err
|
||||
}
|
||||
res.RepairAttempts = attemptsUsed
|
||||
return res, nil
|
||||
}
|
||||
|
||||
func (r *Runner) shouldAttemptRepair(contract domain.OutputContract, validationResult domain.ValidationResult) bool {
|
||||
if r.repairer == nil {
|
||||
return false
|
||||
}
|
||||
if contract.RepairAttempts <= 0 {
|
||||
return false
|
||||
}
|
||||
if validationResult.Status != domain.ValidationFailed {
|
||||
return false
|
||||
}
|
||||
return contract.ValidationMode == domain.ValidationJSON || contract.ValidationMode == domain.ValidationJSONSchema
|
||||
}
|
||||
|
||||
func mergeExecutionTarget(base domain.ExecutionTarget, override domain.ExecutionTarget) domain.ExecutionTarget {
|
||||
out := base
|
||||
if override.Endpoint != "" {
|
||||
out.Endpoint = override.Endpoint
|
||||
}
|
||||
if override.Model != "" {
|
||||
out.Model = override.Model
|
||||
}
|
||||
if override.Temperature != 0 {
|
||||
out.Temperature = override.Temperature
|
||||
}
|
||||
if override.MaxTokens != 0 {
|
||||
out.MaxTokens = override.MaxTokens
|
||||
}
|
||||
if override.TopP != 0 {
|
||||
out.TopP = override.TopP
|
||||
}
|
||||
if override.TimeoutSeconds != 0 {
|
||||
out.TimeoutSeconds = override.TimeoutSeconds
|
||||
}
|
||||
if strings.TrimSpace(override.ServiceTier) != "" {
|
||||
out.ServiceTier = override.ServiceTier
|
||||
}
|
||||
if strings.TrimSpace(override.ReasoningEffort) != "" {
|
||||
out.ReasoningEffort = override.ReasoningEffort
|
||||
}
|
||||
if strings.TrimSpace(override.APIKeyEnv) != "" {
|
||||
out.APIKeyEnv = override.APIKeyEnv
|
||||
}
|
||||
if override.APIKeyRequired {
|
||||
out.APIKeyRequired = true
|
||||
}
|
||||
if len(override.ExtraParams) > 0 {
|
||||
out.ExtraParams = copyExtraParams(override.ExtraParams)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mergeExecutionTargetOverride(base domain.ExecutionTarget, override domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) {
|
||||
out := base
|
||||
var presence domain.ExecutionTargetPresence
|
||||
if override.Endpoint != "" {
|
||||
out.Endpoint = override.Endpoint
|
||||
}
|
||||
if override.Model != "" {
|
||||
out.Model = override.Model
|
||||
}
|
||||
if override.Temperature != nil {
|
||||
if *override.Temperature < 0 || *override.Temperature > 2 {
|
||||
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("temperature must be between 0 and 2")
|
||||
}
|
||||
out.Temperature = *override.Temperature
|
||||
presence.Temperature = true
|
||||
}
|
||||
if override.MaxTokens != nil {
|
||||
if *override.MaxTokens < 0 {
|
||||
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("max_tokens must be greater than or equal to 0")
|
||||
}
|
||||
out.MaxTokens = *override.MaxTokens
|
||||
presence.MaxTokens = true
|
||||
}
|
||||
if override.TopP != nil {
|
||||
if *override.TopP < 0 || *override.TopP > 1 {
|
||||
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("top_p must be between 0 and 1")
|
||||
}
|
||||
out.TopP = *override.TopP
|
||||
presence.TopP = true
|
||||
}
|
||||
if override.TimeoutSeconds != nil {
|
||||
if *override.TimeoutSeconds < 0 {
|
||||
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, errors.New("timeout_seconds must be greater than or equal to 0")
|
||||
}
|
||||
out.TimeoutSeconds = *override.TimeoutSeconds
|
||||
presence.TimeoutSeconds = true
|
||||
}
|
||||
if strings.TrimSpace(override.ServiceTier) != "" {
|
||||
out.ServiceTier = override.ServiceTier
|
||||
}
|
||||
if strings.TrimSpace(override.ReasoningEffort) != "" {
|
||||
out.ReasoningEffort = override.ReasoningEffort
|
||||
}
|
||||
if strings.TrimSpace(override.APIKeyEnv) != "" {
|
||||
out.APIKeyEnv = override.APIKeyEnv
|
||||
}
|
||||
if len(override.ExtraParams) > 0 {
|
||||
out.ExtraParams = copyExtraParams(override.ExtraParams)
|
||||
}
|
||||
return out, presence, nil
|
||||
}
|
||||
|
||||
func resolveExecutionTarget(profileValue *domain.ExecutionProfile, override *domain.ExecutionTargetOverride) (domain.ExecutionTarget, domain.ExecutionTargetPresence, error) {
|
||||
out := defaults.ExecutionTargetDefault()
|
||||
out = mergeExecutionTarget(out, executionProfileToTarget(profileValue))
|
||||
var presence domain.ExecutionTargetPresence
|
||||
if override != nil {
|
||||
var err error
|
||||
out, presence, err = mergeExecutionTargetOverride(out, *override)
|
||||
if err != nil {
|
||||
return domain.ExecutionTarget{}, domain.ExecutionTargetPresence{}, err
|
||||
}
|
||||
}
|
||||
return out, presence, nil
|
||||
}
|
||||
|
||||
func validateAPIKey(apiKeyEnv string, apiKey string, apiKeyRequired bool) error {
|
||||
if strings.TrimSpace(apiKey) != "" {
|
||||
return nil
|
||||
}
|
||||
envName := strings.TrimSpace(apiKeyEnv)
|
||||
if envName == "" {
|
||||
if apiKeyRequired {
|
||||
return ErrAPIKeyRequired
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if strings.TrimSpace(os.Getenv(envName)) == "" {
|
||||
return fmt.Errorf("%w: api key environment variable %q is not set", ErrAPIKeyEnvMissing, envName)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func executionProfileToTarget(p *domain.ExecutionProfile) domain.ExecutionTarget {
|
||||
if p == nil {
|
||||
return domain.ExecutionTarget{}
|
||||
}
|
||||
return domain.ExecutionTarget{
|
||||
Endpoint: p.Endpoint,
|
||||
Model: p.Model,
|
||||
Temperature: p.Temperature,
|
||||
MaxTokens: p.MaxTokens,
|
||||
TopP: p.TopP,
|
||||
TimeoutSeconds: p.TimeoutSeconds,
|
||||
ServiceTier: p.ServiceTier,
|
||||
ReasoningEffort: p.ReasoningEffort,
|
||||
APIKeyEnv: p.APIKeyEnv,
|
||||
APIKeyRequired: p.APIKeyRequired,
|
||||
ExtraParams: copyExtraParams(p.ExtraParams),
|
||||
}
|
||||
}
|
||||
|
||||
func copyExtraParams(src map[string]any) map[string]any {
|
||||
if len(src) == 0 {
|
||||
return nil
|
||||
}
|
||||
cp := make(map[string]any, len(src))
|
||||
for k, v := range src {
|
||||
cp[k] = v
|
||||
}
|
||||
return cp
|
||||
}
|
||||
|
||||
func resolveOutputContract(def *domain.PromptDefinition, override *domain.OutputContract) domain.OutputContract {
|
||||
contract := def.Validation
|
||||
if contract.Format == "" {
|
||||
contract.Format = def.OutputFormat
|
||||
}
|
||||
if override != nil {
|
||||
contract = *override
|
||||
}
|
||||
if contract.Format == "" {
|
||||
contract.Format = domain.FormatText
|
||||
}
|
||||
return contract
|
||||
}
|
||||
|
||||
func hashRenderedPrompt(p domain.RenderedPrompt) string {
|
||||
var b strings.Builder
|
||||
if p.SessionID != "" {
|
||||
b.WriteString("session_id=")
|
||||
b.WriteString(p.SessionID)
|
||||
b.WriteString("\n---\n")
|
||||
}
|
||||
for _, msg := range p.Messages {
|
||||
b.WriteString(msg.Role)
|
||||
b.WriteByte('\n')
|
||||
b.WriteString(msg.Content)
|
||||
if msg.CacheControl != nil {
|
||||
b.WriteString("\ncache_control.type=")
|
||||
b.WriteString(string(msg.CacheControl.Type))
|
||||
if msg.CacheControl.TTL != "" {
|
||||
b.WriteString("\ncache_control.ttl=")
|
||||
b.WriteString(msg.CacheControl.TTL)
|
||||
}
|
||||
}
|
||||
b.WriteString("\n---\n")
|
||||
}
|
||||
h := sha256.Sum256([]byte(b.String()))
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
|
||||
func buildOutputArtifact(content string, format domain.OutputFormat) domain.Artifact {
|
||||
body := []byte(content)
|
||||
hash := sha256.Sum256(body)
|
||||
|
||||
contentType := defaults.ContentTypeTextPlain
|
||||
switch format {
|
||||
case domain.FormatMarkdown:
|
||||
contentType = defaults.ContentTypeTextMarkdown
|
||||
case domain.FormatJSON:
|
||||
contentType = defaults.ContentTypeApplicationJSON
|
||||
}
|
||||
|
||||
return domain.Artifact{
|
||||
Name: defaults.OutputArtifactName,
|
||||
ContentType: contentType,
|
||||
Body: body,
|
||||
Size: int64(len(body)),
|
||||
Hash: hex.EncodeToString(hash[:]),
|
||||
}
|
||||
}
|
||||
|
||||
func hashPromptDefinition(def *domain.PromptDefinition) (string, error) {
|
||||
b, err := json.Marshal(def)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
sum := sha256.Sum256(b)
|
||||
return hex.EncodeToString(sum[:]), nil
|
||||
}
|
||||
|
||||
func newRunID() (string, error) {
|
||||
var b [16]byte
|
||||
if _, err := rand.Read(b[:]); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// UUID v4 (RFC 4122 variant).
|
||||
b[6] = (b[6] & 0x0f) | 0x40
|
||||
b[8] = (b[8] & 0x3f) | 0x80
|
||||
|
||||
return fmt.Sprintf("%08x-%04x-%04x-%04x-%012x",
|
||||
b[0:4],
|
||||
b[4:6],
|
||||
b[6:8],
|
||||
b[8:10],
|
||||
b[10:16],
|
||||
), nil
|
||||
}
|
||||
1847
internal/usecase/runner_test.go
Normal file
1847
internal/usecase/runner_test.go
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user