214 lines
11 KiB
Markdown
214 lines
11 KiB
Markdown
# Internal Runner
|
|
|
|
## Purpose
|
|
|
|
This document describes Promptkit's implemented internal orchestration. The
|
|
[architecture policy](../policy/architecture.md) owns dependency and consumer
|
|
boundaries. The [source and validation document](sources.md) owns repository,
|
|
artifact, rendering, and validation behavior, while the
|
|
[model-client document](llm.md) owns generation behavior and failure
|
|
categories.
|
|
|
|
The runner remains under `internal/usecase` and is assembled by the root
|
|
Promptkit engine. Its concrete type is not part of the public API.
|
|
The [framework format reference](../formats.md) owns prompt, profile, schema,
|
|
and override semantics consumed by the runner.
|
|
|
|
## Collaborators
|
|
|
|
`Runner` coordinates narrow internal interfaces for prompt definitions,
|
|
profiles, backend resolution, artifacts, rendering, model generation, and
|
|
validation. The root engine supplies one immutable registry containing the
|
|
built-in backend and validated consumer additions, one engine-local run
|
|
admitter, and a model client wrapped by the same capacity manager. Validation
|
|
plans and provider-facing schema metadata come from the validator's preparation
|
|
interface.
|
|
An output repairer can be injected internally, but the ordinary runner
|
|
constructor does not enable one.
|
|
|
|
Each invocation carries its state in request, prepared-run, and result values.
|
|
The runner has no durable run or session store.
|
|
|
|
## Shared Prompt Selection
|
|
|
|
The runner uses one prompt-selection and hashing boundary for ordinary
|
|
preparation and exact prompt inspection. Preparation retains its early
|
|
request-ID check before direct-session normalization; both operations then use
|
|
the configured prompt repository to select one definition, load referenced
|
|
message content, and calculate the same prompt hash.
|
|
|
|
Inspection stops after that structural lookup. It does not parse templates or
|
|
touch profile, artifact, schema, renderer, validator, admission, or model
|
|
collaborators. The root [`Engine.InspectPrompt`](../../engine.go) GoDoc owns
|
|
the public operation's exact contract.
|
|
|
|
## Shared Profile Selection
|
|
|
|
The runner uses one profile-selection and target-resolution boundary for
|
|
ordinary preparation and exact profile inspection. Preparation first selects a
|
|
request profile or a prompt default; inspection begins with its required
|
|
explicit profile ID. Both then apply the ordinary source precedence, resolve a
|
|
named backend, and construct the effective target from framework, backend, and
|
|
profile values.
|
|
|
|
Inspection stops after the resulting endpoint and model are structurally
|
|
validated. It does not check credential availability or perform prompt,
|
|
artifact, schema, rendering, admission, or model-client work. The root
|
|
[`Engine.InspectProfile`](../../engine.go) GoDoc owns the public operation's
|
|
exact contract.
|
|
|
|
## Shared Preparation Pipeline
|
|
|
|
`Prepare` and `Run` share one private preparation pipeline split at the point
|
|
where a run can be assigned to its selected backend pool. The resolution phase
|
|
performs only the work needed to validate routing and admission:
|
|
|
|
1. validate the required prompt selection and normalize any direct session ID;
|
|
2. load the prompt definition and hash the original definition;
|
|
3. select the request profile or the prompt's default profile;
|
|
4. resolve the profile's backend ID, when present;
|
|
5. resolve application-neutral defaults, backend defaults, profile values,
|
|
and explicit request overrides in that order;
|
|
6. validate endpoint, model, numeric overrides, and credential requirements;
|
|
7. resolve and validate the effective output contract without loading its
|
|
schema; and
|
|
8. retain the definition, source identities, effective settings, output
|
|
contract, and preparation start time in invocation-local state.
|
|
|
|
The completion phase consumes that state without reloading the prompt,
|
|
profile, or backend:
|
|
|
|
1. create one operation-local validation plan and derive structured-output
|
|
schema metadata from it when required;
|
|
2. load and hash input artifacts;
|
|
3. render messages and the prompt-defined session;
|
|
4. apply any direct session ID;
|
|
5. hash the effective rendered prompt; and
|
|
6. construct the prepared value and preparation timing.
|
|
|
|
`Prepare` runs both phases consecutively and never performs capacity admission.
|
|
`Run` performs backend admission between the phases. This structure preserves
|
|
one execution-precedence and error-ordering implementation while allowing a
|
|
full backend pool to reject work before expensive schema, artifact, and
|
|
rendering operations. `Prepare` discards the plan after returning its public
|
|
metadata. `Run` retains the plan through initial and repaired-output validation
|
|
and discards it when the operation ends. Prepared execution stores the same
|
|
kind of plan only in its private payload.
|
|
|
|
Pointer-based numeric overrides preserve an explicit zero. Invalid negative or
|
|
out-of-range values fail as invalid requests. Endpoint overrides do not change
|
|
the selected backend identity. Non-empty extra-parameter maps replace whole
|
|
lower-precedence maps. A direct API key takes precedence over environment
|
|
lookup; otherwise request, profile, and backend environment-variable names
|
|
apply in that order. A profile requiring a direct key clears an inherited
|
|
backend environment name unless the request supplies its own name. Secret
|
|
values remain excluded from serialized metadata.
|
|
|
|
Reasoning overrides are tri-state: nil inherits the profile, a pointer to a
|
|
nonblank string trims and replaces it, and a pointer to a blank string clears
|
|
it. A nonblank direct session is normalized before source loading, bypasses
|
|
the prompt session template, and is applied after ordinary message rendering.
|
|
A blank direct value retains prompt-template behavior. The runner clears the
|
|
template only on a value copy of the definition, so the definition hash always
|
|
describes the original source while the rendered-prompt hash includes the
|
|
effective direct or rendered session.
|
|
|
|
The registry is read-only after engine construction. Concurrent `Prepare` and
|
|
`Run` calls resolve independent defensive backend values and keep all
|
|
invocation state local.
|
|
|
|
## Run Flow
|
|
|
|
`Run` records its start time, performs the shared resolution phase, and asks
|
|
its `RunAdmitter` to reserve capacity for the effective backend ID. A nil
|
|
admitter is an internal unlimited fallback. After successful admission, `Run`
|
|
immediately defers the returned release function, performs the completion
|
|
phase, makes one initial generation call, builds the named output artifact,
|
|
and validates that artifact with the plan compiled during completion. Invalid
|
|
generated content remains a validation result; an inability to perform
|
|
validation is an operational error.
|
|
|
|
Validation preparation and execution honor cancellation at every
|
|
Promptkit-controlled boundary and do not publish a partial plan or result.
|
|
Schema reads are bounded and context-checked between chunks; JSON decoding,
|
|
schema compilation, and schema execution are checked immediately before and
|
|
after their synchronous calls. Promptkit does not move arbitrary filesystem or
|
|
JSON Schema work to background goroutines, so an already-blocked dependency
|
|
method must return before cancellation can take precedence over its outcome.
|
|
|
|
The admission lease covers completion-phase preparation, initial generation,
|
|
validation, every repair, and every exit. It bounds accepted work without
|
|
serializing preparation or validation behind the active-generation limit.
|
|
The wrapped model client separately acquires a FIFO active permit only around
|
|
each actual generation call.
|
|
|
|
When an internal repairer is present, a JSON or JSON Schema content failure can
|
|
trigger bounded repair attempts. Repair receives the effective execution
|
|
target, explicit numeric-presence bits, credential, backend identity, session
|
|
ID, validation errors, prior output, and structured-output specification. One
|
|
request constructor supplies those common fields to initial and repair
|
|
generation while their rendered prompts remain intentionally distinct. The
|
|
default repairer uses the same wrapped client as initial generation, so each
|
|
repair reacquires the selected backend's active permit while remaining inside
|
|
its original admission lease. Repair never performs a second bounded
|
|
admission, and repaired outputs use the operation's existing validation plan.
|
|
This capability remains internal and is not a public option.
|
|
|
|
A successful result includes the output artifact and raw output, validation
|
|
state, effective session ID, prompt and rendered-prompt hashes, selected
|
|
profile and backend, effective settings, input hashes, token usage, a generated
|
|
run identifier, and UTC timing. The same effective session reaches initial
|
|
generation and any repair attempt through the rendered prompt. The same
|
|
effective target and presence metadata, including backend identity and direct
|
|
credential during execution, reaches generation and every repair attempt.
|
|
Result usage is the field-wise sum of all five usage values from the initial
|
|
response and every completed repair response. Final raw output, artifact, and
|
|
validation state still come from the last candidate. A repair error returns no
|
|
partial run result or partial usage.
|
|
|
|
## Failure Categories
|
|
|
|
Package errors distinguish invalid requests, required profile selection,
|
|
credential failures, and prompt, profile, artifact, rendering, generation, and
|
|
validation failures. Wrapping preserves the package identities mapped by the
|
|
public facade and retains collaborator identities where they are part of the
|
|
internal contract.
|
|
|
|
Admission capacity exhaustion retains the internal capacity identity. At the
|
|
use-case boundary, the runner attaches the selected backend ID in an internal
|
|
typed error, and the root facade copies that value into the public
|
|
[`CapacityError`](../../capacity_error.go) without parsing diagnostic text. It
|
|
is not recategorized as an invalid request or generation failure, and no
|
|
partial result is returned. A context already done at admission retains its
|
|
context identity directly. Cancellation while waiting for an active generation
|
|
permit prevents client invocation when it wins the grant race; the model-client
|
|
boundary then preserves the context error through the generation-failure
|
|
category. Deferred release restores the admission lease on preparation,
|
|
generation, validation, repair, and cancellation failures.
|
|
|
|
Other context cancellation propagates through the invoked collaborator and is
|
|
classified by the owning operation. In particular, cancellation observed by
|
|
validation retains the context identity through the validation error category.
|
|
An overlong direct session is an invalid request before source loading, while
|
|
an invalid or overlong prompt session template remains a prompt-render failure.
|
|
An unknown selected backend, or a selected backend with no configured resolver,
|
|
is classified as a profile-load failure.
|
|
|
|
## Test Ownership And Changes
|
|
|
|
The [runner tests](../../internal/usecase/runner_test.go) own preparation order,
|
|
selection and override precedence, the two-phase boundary, early admission,
|
|
lease lifetime and release, direct-session resolution, schema-before-generation
|
|
behavior, hashing, generation and validation outcomes, backend propagation,
|
|
bounded repair progression, initial/repair request parity, cumulative usage,
|
|
shared initial/repair capacity, credentials and redaction, error categories,
|
|
artifact metadata, and timing. The
|
|
[capacity subsystem document](capacity.md) identifies the focused pool,
|
|
waiter, and wrapped-client tests.
|
|
|
|
Changes to orchestration should continue to use the existing package
|
|
interfaces, keep request state local to an invocation, and preserve the shared
|
|
resolution and completion pipeline. Source, renderer, validator, or
|
|
model-client contract changes belong first in their owning package and
|
|
document.
|