169 lines
6.2 KiB
Markdown
169 lines
6.2 KiB
Markdown
# Runner Internals
|
|
|
|
## Purpose
|
|
|
|
`internal/usecase.Runner` is the core use case orchestrator for prompt preparation and execution.
|
|
|
|
It owns request validation, prompt/profile resolution, runtime-parameter merge, artifact loading, prompt rendering, structured-output setup, LLM invocation, output validation, and result metadata.
|
|
|
|
## Inputs And Outputs
|
|
|
|
Primary input type:
|
|
|
|
- `domain.RunRequest`
|
|
|
|
Primary output types:
|
|
|
|
- `domain.PreparedRun` from `Prepare`
|
|
- `domain.RunResult` from `Run`
|
|
|
|
LLM boundary types:
|
|
|
|
- `domain.GenerateRequest`
|
|
- `domain.GenerateResponse`
|
|
|
|
## Boundaries
|
|
|
|
`Runner` coordinates the following interfaces:
|
|
|
|
- `promptdef.Repository`
|
|
- `profile.Repository`
|
|
- `artifact.Reader`
|
|
- `prompt.Renderer`
|
|
- `llm.Client`
|
|
- `validate.Validator`
|
|
- optional `usecase.OutputRepairer`
|
|
|
|
Transport concerns (CLI flags, HTTP DTO parsing, status-code mapping) stay outside runner.
|
|
|
|
## Config Fields Used
|
|
|
|
`Runner` does not read app config files directly.
|
|
|
|
It receives fully constructed repositories/readers/validators from adapters. Effective behavior depends on adapter wiring, including:
|
|
|
|
- prompt/profile directories
|
|
- schema base directory
|
|
- selected profile/runtime overrides in request
|
|
|
|
## External Adapters Used
|
|
|
|
`Runner` works with adapter implementations via interfaces. Current wiring from CLI/HTTP uses:
|
|
|
|
- filesystem prompt/profile repositories
|
|
- composite artifact reader
|
|
- Go-template prompt renderer
|
|
- OpenAI-compatible LLM client
|
|
- standard validator
|
|
|
|
## State And Resume Behavior
|
|
|
|
`Runner` is stateless across requests.
|
|
|
|
- No durable run-state storage.
|
|
- No built-in resume/skip checkpoints.
|
|
- Each `Run`/`Prepare` executes from request inputs and current repositories.
|
|
|
|
## Failure Behavior
|
|
|
|
Primary runner error classes:
|
|
|
|
- `ErrInvalidRequest`: invalid run request envelope.
|
|
- `ErrProfileRequired`: specific invalid-request reason when neither request `profile_id` nor prompt `default_profile` is available.
|
|
- `ErrAPIKeyEnvMissing`: specific invalid-request reason when `api_key_env` is set but the named environment variable is unset/empty.
|
|
- `ErrProfileLoad`: prompt/profile repository load failures.
|
|
- `ErrArtifactLoad`: artifact read failures.
|
|
- `ErrPromptRender`: template render failures.
|
|
- `ErrLLMGenerate`: outbound model request failures.
|
|
- `ErrValidation`: validation runtime failures (including structured-output schema load/compile failures).
|
|
|
|
Reason sentinel behavior:
|
|
|
|
- `ErrProfileRequired` and `ErrAPIKeyEnvMissing` are wrapped with `ErrInvalidRequest`.
|
|
- Adapters can use `errors.Is` for stable reason mapping without matching runner prose.
|
|
|
|
Validation content failures are not run errors:
|
|
|
|
- `Run` can succeed with `Validation.Status == failed`.
|
|
- CLI maps this to exit code `2`.
|
|
- HTTP returns `200` with failed validation details.
|
|
|
|
## Prepare Flow
|
|
|
|
`Prepare` performs:
|
|
|
|
1. validate request basics (prompt ID present).
|
|
2. load prompt definition by ID/version.
|
|
3. select profile ID:
|
|
- explicit request profile ID
|
|
- prompt `default_profile`
|
|
- otherwise return an invalid request with `ErrProfileRequired`
|
|
4. load execution profile.
|
|
5. merge effective runtime target:
|
|
- built-in execution defaults
|
|
- selected profile values
|
|
- request overrides
|
|
- request numeric overrides are presence-aware, so omitted values preserve the current effective value and explicit zero values override it
|
|
6. verify credentials when the effective target names `api_key_env`:
|
|
- a request-scoped direct API key satisfies the credential requirement
|
|
- otherwise a missing/empty env value returns an invalid request with `ErrAPIKeyEnvMissing`
|
|
- only the environment-variable name is returned in public output; secret values are never returned
|
|
7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
|
|
8. read input artifacts.
|
|
9. render prompt messages, including any normalized message cache-control metadata.
|
|
10. compute prompt/input/render hashes and return `PreparedRun`.
|
|
|
|
`rendered_prompt_hash` includes cache-control metadata when present because it affects the outbound provider request. Prompts without cache control keep the role/content hash behavior.
|
|
|
|
`Prepare` does not call the LLM.
|
|
|
|
Runtime target notes:
|
|
|
|
- Profile `extra_params` and request `extra_params` carry JSON-compatible values through prepared output, run metadata, and `domain.GenerateRequest.Target`.
|
|
- The OpenAI-compatible client serializes non-empty `reasoning_effort` as a top-level provider request field.
|
|
- The OpenAI-compatible client flattens `extra_params` into provider-specific top-level JSON request fields.
|
|
- Empty `extra_params` keys, reserved outbound field names, and values that cannot be JSON-encoded fail before the provider request.
|
|
- Resolved API-key values are never serialized in prepared/run output, public results, logs, or HTTP responses.
|
|
- Public direct API-key values are carried only far enough to call the configured LLM client and are excluded from JSON/YAML serialization.
|
|
|
|
## Run Flow
|
|
|
|
`Run` performs:
|
|
|
|
1. generate run ID.
|
|
2. call `Prepare`.
|
|
3. call LLM with prepared messages/effective target/structured-output spec.
|
|
4. build output artifact content type from output format.
|
|
5. validate output.
|
|
6. optionally attempt bounded repair when repairer is injected and contract allows it.
|
|
7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, token/cache usage, and timestamps.
|
|
|
|
## Repair Hook Boundary
|
|
|
|
Repair attempts occur only when all are true:
|
|
|
|
- repairer is injected
|
|
- `repair_attempts > 0`
|
|
- validation status is `failed`
|
|
- validation mode is `json` or `json_schema`
|
|
|
|
Current production wiring boundary:
|
|
|
|
- CLI and HTTP adapters call `usecase.NewRunner(...)` (no repairer argument).
|
|
- Therefore normal CLI/HTTP execution does not perform repair attempts today.
|
|
|
|
## Tests To Inspect Before Changing
|
|
|
|
- `internal/usecase/runner_test.go`
|
|
- `internal/usecase/integration_test.go`
|
|
- `internal/adapter/cli/run_test.go`
|
|
- `internal/adapter/http/handler_test.go`
|
|
|
|
## Architectural Invariants
|
|
|
|
- `Run` reuses `Prepare`; prepare logic is not duplicated.
|
|
- Effective API-key environment-variable name may appear; resolved secret value must not.
|
|
- Structured-output schema document must load before LLM call for `json_schema` mode.
|
|
- Repair loops are bounded by `repair_attempts` and repairer presence.
|
|
- Runner stays transport-agnostic.
|