Resolve public documentation contract questions
This commit is contained in:
@@ -20,7 +20,6 @@ func toDomainRunRequest(req RunRequest) (domain.RunRequest, error) {
|
||||
Vars: copyStringMap(req.Vars),
|
||||
Execution: execution,
|
||||
Validation: toDomainOutputContractPtr(req.Validation),
|
||||
Metadata: copyStringMap(req.Metadata),
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
250
docs/roadmap/documentation.md
Normal file
250
docs/roadmap/documentation.md
Normal file
@@ -0,0 +1,250 @@
|
||||
# Documentation Hardening Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap coordinates a focused pass over Promptkit's documentation and
|
||||
public contract. The work should close the gaps identified by the documentation
|
||||
audit, strengthen canonical ownership, and leave consumers and contributors
|
||||
with guidance that is accurate, navigable, and proportionate to their needs.
|
||||
|
||||
This is planning material, not a description of implemented behavior. Follow
|
||||
the [documentation policy](../policy/documentation.md) throughout the work and
|
||||
update current-state documents only when their claims are supported by the
|
||||
implementation and tests.
|
||||
|
||||
## Scope And Principles
|
||||
|
||||
The work covers public GoDoc, consumer guidance, maintained examples, format
|
||||
and integration references, architecture ownership, roadmap lifecycle, and
|
||||
documentation validation.
|
||||
|
||||
- Resolve ambiguous behavior before documenting it as a contract.
|
||||
- Keep exact exported API semantics in Go declarations and GoDoc.
|
||||
- Keep task-oriented guidance in consumer documents and complete runnable
|
||||
artifacts in `examples/`.
|
||||
- Put security-relevant consumer responsibilities at the public boundary, not
|
||||
only in internal contributor documents.
|
||||
- Remove duplicated ownership instead of synchronizing parallel references.
|
||||
- Make documentation checks reproducible where practical.
|
||||
- Preserve existing behavior unless a stage explicitly selects and tests an
|
||||
API change.
|
||||
|
||||
The roadmap does not add new Promptkit features, redesign framework formats,
|
||||
or implement ideas from [the future feature catalog](future.md). If resolving
|
||||
an ambiguity requires a behavioral change, treat that change as a separately
|
||||
reviewable implementation unit and update its canonical documentation in the
|
||||
same unit.
|
||||
|
||||
## Stage 1: Resolve Public Contract Questions
|
||||
|
||||
Before expanding prose, decide the intended contract for exported behavior
|
||||
that is currently ambiguous.
|
||||
|
||||
- [x] Decide whether `RunRequest.Metadata` has a supported observable purpose.
|
||||
Define its propagation and ownership, or remove or deprecate it through an
|
||||
intentional public API change.
|
||||
- [x] Decide and test whether one `Engine` supports concurrent `Prepare` and
|
||||
`Run` calls.
|
||||
- [x] Decide how repeated options of the same category behave, including
|
||||
prompt, profile, schema, model-client, and artifact-reader options.
|
||||
- [x] Define which public values have supported JSON representations.
|
||||
- [x] Define JSON time units and omission behavior, including the relationship
|
||||
between prepared-run and run-result durations.
|
||||
- [x] Decide whether run IDs and exposed hashes have stable formats or must be
|
||||
treated as opaque values.
|
||||
- [x] Confirm the intended transport-timeout default and its zero or negative
|
||||
configuration semantics.
|
||||
- [x] Confirm the supported JSON Schema dialect and reference boundaries,
|
||||
including whether remote references are allowed.
|
||||
|
||||
The resolved contract is:
|
||||
|
||||
- `RunRequest.Metadata` had no observable purpose and has been removed from the
|
||||
public and internal request values.
|
||||
- One engine supports overlapping `Prepare` and `Run` calls. Built-in
|
||||
collaborators satisfy that contract; injected collaborators may be called
|
||||
concurrently and therefore share responsibility for concurrency safety.
|
||||
- Options are applied in order. Within each prompt-source, profile-source,
|
||||
in-memory-profile, schema-source, model-client, or artifact-reader category,
|
||||
the last non-nil valid option replaces the earlier value for that category.
|
||||
An invalid earlier option still makes construction fail.
|
||||
- The supported JSON values are `PreparedRun`, `RunResult`, `Artifact`,
|
||||
`ExecutionTarget`, `OutputContract`, `ValidationResult`, `TokenUsage`,
|
||||
`RenderedPrompt`, `RenderedMessage`, `CacheControl`,
|
||||
`StructuredOutputSpec`, `StructuredOutputJSONSpec`, `GenerateRequest`,
|
||||
`GenerateResponse`, `ExecutionTargetPresence`, and the public string value
|
||||
types used by them. Construction inputs such as `Config`, `RunRequest`,
|
||||
`ArtifactRef`, `ExecutionTargetOverride`, `Profile`, and
|
||||
`OpenAICompatibleProfileConfig` do not have stable JSON representations.
|
||||
Request-scoped API keys remain excluded from JSON as a security guarantee,
|
||||
including on otherwise unsupported construction values.
|
||||
- JSON timestamps use `time.Time`'s RFC 3339 representation and are omitted
|
||||
when zero. Both prepared and completed run durations use integer
|
||||
milliseconds in `duration_ms` and are omitted when zero. A prepared duration
|
||||
measures preparation only; a run-result duration measures the complete run,
|
||||
including its preparation, generation, and validation.
|
||||
- Run IDs, prompt hashes, rendered-prompt hashes, input hashes, and artifact
|
||||
hashes are non-empty correlation or equality values where produced. Their
|
||||
spelling, length, character set, and algorithm are opaque and not stable
|
||||
formats.
|
||||
- The built-in transport timeout defaults to 10 minutes. A zero or negative
|
||||
`Config.Timeout` selects that default. A supplied HTTP client's positive
|
||||
timeout takes precedence; its zero or negative timeout inherits the positive
|
||||
configured timeout or the default. These transport semantics are independent
|
||||
of caller cancellation and per-generation timeout settings.
|
||||
- JSON Schema uses Draft 2020-12; omission of `$schema` selects that dialect
|
||||
and an explicit different dialect is rejected. Same-document fragment
|
||||
references are supported. Relative references may load other schema
|
||||
documents only within a configured directory or `fs.FS` schema root.
|
||||
A single-file schema source supports only references contained in that
|
||||
document. Absolute, escaping, and remote references are not allowed.
|
||||
|
||||
**Gate:** Each question has an explicit answer backed by existing behavior or
|
||||
by an accepted implementation change and proportionate tests. No later stage
|
||||
should invent a contract merely to fill a documentation gap.
|
||||
|
||||
## Stage 2: Make GoDoc The Canonical Public Contract
|
||||
|
||||
Strengthen the root package declarations so `go doc` is sufficient to
|
||||
understand exact public behavior without relying on internal documents.
|
||||
|
||||
- [ ] Add useful field-level GoDoc to configuration, request, profile,
|
||||
execution-target, result, artifact, validation, structured-output, and model
|
||||
client values.
|
||||
- [ ] Document required fields and nil, empty, and zero-value semantics.
|
||||
- [ ] Document override, replacement, profile-precedence, and copy-ownership
|
||||
behavior where it belongs to the exported API.
|
||||
- [ ] Document credential inputs, redaction, and the values intentionally
|
||||
excluded from serialization.
|
||||
- [ ] Give each public error sentinel an accurate comment and document the
|
||||
supported `errors.Is` relationships.
|
||||
- [ ] Document engine concurrency and option-composition behavior selected in
|
||||
Stage 1.
|
||||
- [ ] Document serialization, time, run-ID, and hash semantics selected in
|
||||
Stage 1.
|
||||
- [ ] Review constructor, option, extension-interface, `Prepare`, and `Run`
|
||||
GoDoc for complete failure and cancellation expectations.
|
||||
|
||||
Update the [consumer guide](../consumers/pkg-promptkit.md) to summarize and link
|
||||
to these contracts instead of maintaining exhaustive copies of exported names
|
||||
or exact semantics.
|
||||
|
||||
**Gate:** `go doc -all .` presents a coherent public contract, exported
|
||||
declarations have accurate comments, and contract tests protect every newly
|
||||
documented behavior whose compatibility risk warrants durable coverage.
|
||||
|
||||
## Stage 3: Improve Consumer Safety And Executable Guidance
|
||||
|
||||
Move consumer-relevant security boundaries to the places where consumers will
|
||||
encounter them and add one representative execution workflow.
|
||||
|
||||
- [ ] Explain in public GoDoc and the consumer guide that the default file
|
||||
artifact reader accepts unrestricted caller-selected paths.
|
||||
- [ ] Make clear that Promptkit does not impose an application root, inbound
|
||||
request-size policy, or untrusted-input security boundary.
|
||||
- [ ] Explain that rendered messages, artifact bodies, raw model output, and
|
||||
validation details may be sensitive even when credentials are redacted.
|
||||
- [ ] Clarify the responsibilities of injected artifact readers and model
|
||||
clients for cancellation, copying, logging, and secret handling.
|
||||
- [ ] Add a maintained offline `Run` example using an injected deterministic
|
||||
model client, without credentials, live network access, or paid calls.
|
||||
- [ ] Link the consumer guide to the execution example and keep embedded
|
||||
snippets smaller than the maintained artifact.
|
||||
- [ ] Decide whether the existing preparation example should remain separate
|
||||
or share reusable fixtures without obscuring either workflow.
|
||||
|
||||
**Gate:** Both preparation and execution have complete, secret-free,
|
||||
deterministic consumer examples, and the consumer guide exposes the important
|
||||
filesystem and data-sensitivity boundaries without leaking internal mechanics.
|
||||
|
||||
## Stage 4: Restore Canonical Ownership
|
||||
|
||||
Remove parallel definitions and make navigation follow the ownership model in
|
||||
the documentation policy.
|
||||
|
||||
- [ ] Reduce the [architecture policy](../policy/architecture.md) to durable
|
||||
boundaries, layers, dependency direction, invariants, and non-goals.
|
||||
- [ ] Keep the exact implemented package and component inventory solely in the
|
||||
[internal component overview](../internal/overview.md).
|
||||
- [ ] Review the consumer guide's public error and option lists so they remain
|
||||
task-oriented summaries rather than duplicate API references.
|
||||
- [ ] Review internal documents for public-contract statements that should be
|
||||
links to GoDoc or the format and integration owners.
|
||||
- [ ] Reconcile the documentation policy's temporary-roadmap lifecycle with
|
||||
the continuing idea-catalog role of `docs/roadmap/future.md`.
|
||||
- [ ] Rephrase or link roadmap statements that depend on exact current API
|
||||
behavior, particularly the runtime reasoning entry.
|
||||
- [ ] Confirm that every document states its audience or purpose and links to
|
||||
the canonical owner of adjacent topics.
|
||||
|
||||
**Gate:** Every authoritative fact has one clear owner, package inventory
|
||||
changes no longer require edits to the architecture policy, and roadmaps
|
||||
cannot be mistaken for current-state references.
|
||||
|
||||
## Stage 5: Refine Format And Integration References
|
||||
|
||||
Close compatibility gaps in the documents that own file formats and outbound
|
||||
wire behavior.
|
||||
|
||||
- [ ] State or canonically link the exact session-ID limit enforced by the
|
||||
OpenAI-compatible client.
|
||||
- [ ] State the configured and default transport-timeout behavior without
|
||||
referring to an unnamed internal default.
|
||||
- [ ] Document the JSON Schema dialect and local, contained, and remote
|
||||
reference behavior selected in Stage 1.
|
||||
- [ ] Clarify structured-output naming and strictness when those values are
|
||||
part of the public or integration contract.
|
||||
- [ ] Add a caveat that built-in profiles are maintained configurations, not a
|
||||
guarantee of continuing third-party model availability.
|
||||
- [ ] Recheck every prompt, profile, schema, credential, request-body, response,
|
||||
timeout, and precedence statement against its owning implementation and
|
||||
tests.
|
||||
|
||||
**Gate:** A consumer can determine the supported file and wire compatibility
|
||||
boundaries without consulting internal source code or relying on unspecified
|
||||
defaults.
|
||||
|
||||
## Stage 6: Make Documentation Validation Reproducible
|
||||
|
||||
Align contributor and release procedures around a small, consistent set of
|
||||
checks.
|
||||
|
||||
- [ ] Use one robust command for checking every tracked Go file with `gofmt`.
|
||||
- [ ] Provide a repository-local or clearly documented command that validates
|
||||
local Markdown targets and heading fragments.
|
||||
- [ ] Decide how published external links are checked without making ordinary
|
||||
validation depend on mutable network services.
|
||||
- [ ] Reconcile the validation descriptions in the
|
||||
[development guide](../development.md),
|
||||
[testing policy](../policy/testing.md), and
|
||||
[release procedure](../release.md) so one document owns each requirement.
|
||||
- [ ] Ensure example validation covers every maintained example added by this
|
||||
roadmap.
|
||||
- [ ] Keep documentation-only validation proportionate while requiring full Go
|
||||
validation when commands, examples, generated output, or checked behavior
|
||||
changes.
|
||||
|
||||
**Gate:** A maintainer can run the documented formatting, link, example, Go,
|
||||
and repository-hygiene checks exactly as written, with no hidden manual
|
||||
procedure for local documentation.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap is complete when:
|
||||
|
||||
- all Stage 1 contract questions are resolved;
|
||||
- GoDoc is the authoritative and sufficient exported API reference;
|
||||
- consumer guidance covers unrestricted file access and sensitive generated
|
||||
data;
|
||||
- maintained offline examples cover both `Prepare` and `Run`;
|
||||
- architecture, inventory, consumer, internal, format, integration, and
|
||||
roadmap documents follow their assigned ownership boundaries;
|
||||
- schema, session, timeout, structured-output, and built-in-profile
|
||||
compatibility statements are explicit;
|
||||
- documentation validation is reproducible and consistent across contributor
|
||||
and release workflows; and
|
||||
- the complete maintainer validation passes.
|
||||
|
||||
After completion, move any durable decisions to GoDoc, policy, format,
|
||||
integration, or ADR owners as appropriate. Remove this roadmap after incoming
|
||||
links are updated; do not retain it as a second current-state reference.
|
||||
122
docs/roadmap/future.md
Normal file
122
docs/roadmap/future.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Future Feature Ideas
|
||||
|
||||
## Purpose
|
||||
|
||||
This document catalogs reasonably specific ideas that may be useful in future
|
||||
Promptkit development. It is an idea pool, not a commitment, schedule, or
|
||||
description of current behavior.
|
||||
|
||||
Ideas belong here while they are worth retaining but have not been selected
|
||||
for active development. Keep each entry at the level of intended capability,
|
||||
consumer value, and important scope boundaries. Defer API design,
|
||||
implementation details, sequencing, and acceptance criteria until an idea is
|
||||
selected.
|
||||
|
||||
## Using This Catalog
|
||||
|
||||
- Add an idea when its purpose and likely value can be stated clearly.
|
||||
- Keep entries independent enough that maintainers can evaluate and select
|
||||
them individually.
|
||||
- Note significant dependencies or boundary concerns, but do not turn entries
|
||||
into implementation plans.
|
||||
- Treat inclusion as an invitation to evaluate, not as approval or priority.
|
||||
- When an idea is selected, move its active planning to a focused roadmap or,
|
||||
when it requires a durable architectural decision, an ADR. Update
|
||||
current-state documentation only when implementation lands.
|
||||
- Remove ideas that are no longer relevant. Retain a rejected idea only when
|
||||
its rationale is likely to prevent repeated reconsideration.
|
||||
|
||||
Future capabilities must continue to respect the
|
||||
[architecture policy](../policy/architecture.md), particularly Promptkit's
|
||||
role as an application-neutral library and its boundary with downstream
|
||||
consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
### Extensible LLM backend registry
|
||||
|
||||
Introduce a registry that separates backend-specific connection,
|
||||
authentication, and limited request defaults from model execution profiles.
|
||||
Initial support would cover OpenAI-compatible backends and include a small
|
||||
built-in catalog, potentially starting with OpenRouter. A profile could select
|
||||
a backend while optionally overriding its default endpoint, and each backend
|
||||
could name an optional environment variable for its API key without storing
|
||||
the credential itself. Downstream consumers could register additional,
|
||||
uniquely named backends, such as OpenAI or unauthenticated local-network
|
||||
services, but could not replace built-in IDs. Model selection and generation
|
||||
settings would remain profile concerns, and custom model clients would remain
|
||||
available for behavior outside the registry's supported protocol.
|
||||
|
||||
### Backend-specific concurrency management
|
||||
|
||||
Extend the proposed LLM backend registry with optional per-backend concurrency
|
||||
limits and bounded, buffered admission queues. Promptkit could then route
|
||||
simultaneous generation requests according to backend capacity while
|
||||
containing accidental runaway submission. Downstream consumers would continue
|
||||
invoking synchronous `Run` calls, including concurrently from multiple
|
||||
goroutines, and each admitted call would wait for and return its ordinary
|
||||
result.
|
||||
|
||||
- Scope limits to an engine instance rather than hidden process-global state.
|
||||
- Give different backend IDs independent capacity pools. A profile endpoint
|
||||
override would remain part of its selected backend's pool.
|
||||
- Configure active concurrency and waiting capacity separately. Concurrency
|
||||
protects the backend, while queue capacity protects the process from
|
||||
admitting an unbounded backlog.
|
||||
- Give queue capacity a generous, configurable bounded default intended as a
|
||||
safety ceiling for bugs or unintended loops rather than a routine
|
||||
application constraint. Select an exact default during implementation
|
||||
planning and measurement.
|
||||
- Reject a call with a recognizable capacity error when its backend queue is
|
||||
full rather than allowing it to wait outside the bounded queue.
|
||||
- Admit requests before expensive preparation and artifact copying where
|
||||
practical so queued work remains lightweight.
|
||||
- Apply a limit to each actual generation request, including repair attempts,
|
||||
without unnecessarily serializing prompt preparation.
|
||||
- Make queued and active waits respect caller cancellation and deadlines.
|
||||
- Treat concurrency as backend policy rather than a profile-level model
|
||||
setting.
|
||||
- Keep the queue ephemeral and in-process, with no survival guarantee across
|
||||
engine or process shutdown.
|
||||
- Preserve the existing execution model as far as practical. Durable jobs,
|
||||
polling, priorities, application worker lifecycle, retries, and
|
||||
cross-process coordination would be separate future capabilities.
|
||||
|
||||
### Explicit per-run session and reasoning controls
|
||||
|
||||
Allow consumers to associate a session ID with each run and to inherit,
|
||||
replace, or explicitly disable the reasoning effort configured by its selected
|
||||
profile. Prompt definitions can currently derive a session ID from a template,
|
||||
and a non-empty runtime `ReasoningEffort` can replace the profile value, but
|
||||
there is no direct request-level session ID and an empty reasoning value means
|
||||
that no override was supplied. These controls would let consumers reuse one
|
||||
prompt and model profile across sessions and reasoning levels without
|
||||
maintaining duplicate definitions.
|
||||
|
||||
- Preserve a prompt's session ID template and a profile's reasoning effort as
|
||||
reusable defaults.
|
||||
- Let a directly supplied per-run session ID take precedence over a rendered
|
||||
prompt default while retaining the existing validation limit and outbound
|
||||
representation.
|
||||
- Distinguish an omitted runtime reasoning choice from an explicit request to
|
||||
disable reasoning.
|
||||
- Ensure disabling reasoning omits the corresponding provider request setting
|
||||
rather than relying on a provider-specific magic value.
|
||||
- Keep the effective session ID and reasoning choice visible in prepared and
|
||||
run metadata and available to injected model clients without introducing
|
||||
additional prompt or profile selection mechanisms.
|
||||
|
||||
## Entry Format
|
||||
|
||||
Use a short heading followed by a concise summary. Add focused bullets when
|
||||
they help preserve important scope boundaries without becoming an
|
||||
implementation plan:
|
||||
|
||||
```markdown
|
||||
### Idea name
|
||||
|
||||
Describe the intended capability, who benefits, and the most important scope
|
||||
boundary or dependency.
|
||||
|
||||
- Optionally record an important behavior or boundary.
|
||||
```
|
||||
@@ -14,7 +14,7 @@ func (r RunRequest) GoString() string {
|
||||
|
||||
func (r RunRequest) redactedString() string {
|
||||
return fmt.Sprintf(
|
||||
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t Metadata:%d}",
|
||||
"promptkit.RunRequest{PromptID:%q PromptVersion:%q ProfileID:%q APIKeySet:%t Inputs:%d Vars:%d ExecutionSet:%t ValidationSet:%t}",
|
||||
r.PromptID,
|
||||
r.PromptVersion,
|
||||
r.ProfileID,
|
||||
@@ -23,7 +23,6 @@ func (r RunRequest) redactedString() string {
|
||||
len(r.Vars),
|
||||
r.Execution != nil,
|
||||
r.Validation != nil,
|
||||
len(r.Metadata),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -68,7 +68,6 @@ type RunRequest struct {
|
||||
Vars map[string]string
|
||||
Execution *ExecutionTargetOverride
|
||||
Validation *OutputContract
|
||||
Metadata map[string]string
|
||||
}
|
||||
|
||||
// RunResult represents the complete result of a prompt execution run.
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/url"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
@@ -16,6 +17,8 @@ import (
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
)
|
||||
|
||||
const jsonSchemaDraft2020 = "https://json-schema.org/draft/2020-12/schema"
|
||||
|
||||
// StandardValidator provides basic, JSON, and JSON Schema output validation.
|
||||
type StandardValidator struct {
|
||||
schemaBaseDir string
|
||||
@@ -121,7 +124,11 @@ func (v *StandardValidator) validateJSONSchema(instance any, schemaPath string)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
compiler := jsonschema.NewCompiler()
|
||||
schemaRoot, err := v.schemaRoot()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
compiler := newSchemaCompiler(standardSchemaLoader{root: schemaRoot})
|
||||
schema, err := compiler.Compile(resolvedSchemaPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", resolvedSchemaPath, err)
|
||||
@@ -140,7 +147,10 @@ func (v *FSValidator) validateJSONSchema(instance any, schemaPath string) ([]str
|
||||
}
|
||||
|
||||
resourceURL := fsSchemaResourceURL(schemaName)
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := validateSchemaDialect(schemaDoc); err != nil {
|
||||
return nil, fmt.Errorf("failed to compile JSON schema %q: %w", schemaName, err)
|
||||
}
|
||||
compiler := newSchemaCompiler(fsSchemaLoader{fsys: v.fsys, root: filecatalog.CleanFSRoot(v.root)})
|
||||
if err := compiler.AddResource(resourceURL, schemaDoc); err != nil {
|
||||
return nil, fmt.Errorf("failed to register JSON schema %q: %w", schemaName, err)
|
||||
}
|
||||
@@ -184,6 +194,9 @@ func (v *StandardValidator) LoadSchemaDocument(ctx context.Context, schemaPath s
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
@@ -206,12 +219,14 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
|
||||
return "", errors.New("schema path is required for json_schema validation")
|
||||
}
|
||||
|
||||
resolved := schemaPath
|
||||
if !filepath.IsAbs(schemaPath) {
|
||||
resolved = filepath.Join(v.schemaBaseDir, schemaPath)
|
||||
root, err := v.schemaRoot()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
resolved, err := containedFilesystemPath(root, schemaPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
resolved = filepath.Clean(resolved)
|
||||
if _, err := os.Stat(resolved); err != nil {
|
||||
return "", fmt.Errorf("failed to access schema file %q: %w", resolved, err)
|
||||
}
|
||||
@@ -219,6 +234,22 @@ func (v *StandardValidator) resolveSchemaPath(schemaPath string) (string, error)
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (v *StandardValidator) schemaRoot() (string, error) {
|
||||
root := v.schemaBaseDir
|
||||
if strings.TrimSpace(root) == "" {
|
||||
root = "."
|
||||
}
|
||||
absolute, err := filepath.Abs(root)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to resolve schema source %q: %w", root, err)
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(absolute)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to access schema source %q: %w", root, err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error) {
|
||||
resolved, err := v.resolveSchemaPath(schemaPath)
|
||||
if err != nil {
|
||||
@@ -234,6 +265,9 @@ func (v *FSValidator) loadSchemaDocument(schemaPath string) (string, any, error)
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return "", nil, fmt.Errorf("failed to decode JSON schema %q: %w", resolved, err)
|
||||
}
|
||||
return resolved, doc, nil
|
||||
}
|
||||
|
||||
@@ -290,3 +324,131 @@ func cleanSchemaFSPath(schemaPath string) (string, error) {
|
||||
func fsSchemaResourceURL(schemaName string) string {
|
||||
return "promptkit-schema:///" + strings.TrimPrefix(path.Clean(schemaName), "/")
|
||||
}
|
||||
|
||||
func newSchemaCompiler(loader jsonschema.URLLoader) *jsonschema.Compiler {
|
||||
compiler := jsonschema.NewCompiler()
|
||||
compiler.DefaultDraft(jsonschema.Draft2020)
|
||||
compiler.UseLoader(loader)
|
||||
return compiler
|
||||
}
|
||||
|
||||
func validateSchemaDialect(doc any) error {
|
||||
object, ok := doc.(map[string]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
value, ok := object["$schema"]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
dialect, ok := value.(string)
|
||||
if !ok {
|
||||
return errors.New("$schema must be a string")
|
||||
}
|
||||
if dialect != jsonSchemaDraft2020 && dialect != jsonSchemaDraft2020+"#" {
|
||||
return fmt.Errorf("unsupported JSON Schema dialect %q; expected %q", dialect, jsonSchemaDraft2020)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type standardSchemaLoader struct {
|
||||
root string
|
||||
}
|
||||
|
||||
func (l standardSchemaLoader) Load(resourceURL string) (any, error) {
|
||||
fileName, err := (jsonschema.FileLoader{}).ToFile(resourceURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("schema reference %q is not a contained file reference: %w", resourceURL, err)
|
||||
}
|
||||
resolved, err := containedFilesystemPath(l.root, fileName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return loadJSONSchemaFile(resolved)
|
||||
}
|
||||
|
||||
func containedFilesystemPath(root, name string) (string, error) {
|
||||
candidate := name
|
||||
if !filepath.IsAbs(candidate) {
|
||||
candidate = filepath.Join(root, candidate)
|
||||
}
|
||||
candidate, err := filepath.Abs(candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to resolve schema path %q: %w", name, err)
|
||||
}
|
||||
candidate, err = filepath.EvalSymlinks(candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to access schema file %q: %w", candidate, err)
|
||||
}
|
||||
relative, err := filepath.Rel(root, candidate)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to compare schema path %q with source root: %w", candidate, err)
|
||||
}
|
||||
if relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("schema path %q escapes source root", name)
|
||||
}
|
||||
return candidate, nil
|
||||
}
|
||||
|
||||
func loadJSONSchemaFile(name string) (any, error) {
|
||||
raw, err := os.ReadFile(name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc any
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
type fsSchemaLoader struct {
|
||||
fsys fs.FS
|
||||
root string
|
||||
}
|
||||
|
||||
func (l fsSchemaLoader) Load(resourceURL string) (any, error) {
|
||||
parsed, err := url.Parse(resourceURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
|
||||
}
|
||||
if parsed.Scheme != "promptkit-schema" || parsed.Host != "" {
|
||||
return nil, fmt.Errorf("schema reference %q is not allowed", resourceURL)
|
||||
}
|
||||
name, err := url.PathUnescape(strings.TrimPrefix(parsed.Path, "/"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid schema reference %q: %w", resourceURL, err)
|
||||
}
|
||||
name = path.Clean(name)
|
||||
if l.root == "." {
|
||||
if strings.HasPrefix(name, "../") || name == ".." {
|
||||
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
|
||||
}
|
||||
} else if name != l.root && !strings.HasPrefix(name, l.root+"/") {
|
||||
return nil, fmt.Errorf("schema reference %q escapes source root", resourceURL)
|
||||
}
|
||||
|
||||
rootInfo, err := fs.Stat(l.fsys, l.root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !rootInfo.IsDir() && name != l.root {
|
||||
return nil, fmt.Errorf("schema reference %q is outside the configured schema file", resourceURL)
|
||||
}
|
||||
|
||||
raw, err := fs.ReadFile(l.fsys, name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc any
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateSchemaDialect(doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
|
||||
@@ -2,8 +2,10 @@ package validate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
@@ -411,3 +413,138 @@ func TestFSValidatorLoadSchemaDocument(t *testing.T) {
|
||||
t.Fatalf("unexpected schema document: %#v", doc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandardValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, "child.json"), []byte(`{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"type": "string"
|
||||
}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
reference string
|
||||
wantError string
|
||||
writeOuter bool
|
||||
}{
|
||||
{name: "contained relative reference", reference: "child.json"},
|
||||
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "not a contained file reference"},
|
||||
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root", writeOuter: true},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if tc.writeOuter {
|
||||
if err := os.WriteFile(filepath.Join(filepath.Dir(root), "outside.json"), []byte(`{"type":"string"}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
schema := `{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$ref": ` + strconv.Quote(tc.reference) + `
|
||||
}`
|
||||
if err := os.WriteFile(filepath.Join(root, "root.json"), []byte(schema), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
v := NewStandardValidator(root)
|
||||
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: "root.json",
|
||||
})
|
||||
if tc.wantError == "" {
|
||||
if err != nil || !result.IsValid {
|
||||
t.Fatalf("expected contained reference to validate, got result=%#v error=%v", result, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFSValidatorJSONSchemaReferenceBoundaries(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
reference string
|
||||
wantError string
|
||||
}{
|
||||
{name: "same document fragment", reference: "#/$defs/value"},
|
||||
{name: "contained relative reference", reference: "child.json"},
|
||||
{name: "remote reference", reference: "https://example.test/schema.json", wantError: "is not allowed"},
|
||||
{name: "escaping reference", reference: "../outside.json", wantError: "escapes source root"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
rootSchema := `{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$defs": {"value": {"type": "string"}},
|
||||
"$ref": ` + strconv.Quote(tc.reference) + `
|
||||
}`
|
||||
v := NewFSValidator(fstest.MapFS{
|
||||
"schemas/root.json": &fstest.MapFile{Data: []byte(rootSchema)},
|
||||
"schemas/child.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
|
||||
"outside.json": &fstest.MapFile{Data: []byte(`{"type":"string"}`)},
|
||||
}, "schemas")
|
||||
result, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(`"value"`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: "root.json",
|
||||
})
|
||||
if tc.wantError == "" {
|
||||
if err != nil || !result.IsValid {
|
||||
t.Fatalf("expected supported reference to validate, got result=%#v error=%v", result, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err == nil || !strings.Contains(err.Error(), tc.wantError) {
|
||||
t.Fatalf("expected error containing %q, got %v", tc.wantError, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONSchemaDialectIsDraft2020(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
dialect string
|
||||
wantError bool
|
||||
}{
|
||||
{name: "omitted uses supported default"},
|
||||
{name: "draft 2020-12", dialect: "https://json-schema.org/draft/2020-12/schema"},
|
||||
{name: "draft 7 rejected", dialect: "http://json-schema.org/draft-07/schema#", wantError: true},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
schema := map[string]any{"type": "object"}
|
||||
if tc.dialect != "" {
|
||||
schema["$schema"] = tc.dialect
|
||||
}
|
||||
data, err := json.Marshal(schema)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v := NewFSValidator(fstest.MapFS{
|
||||
"schema.json": &fstest.MapFile{Data: data},
|
||||
}, ".")
|
||||
_, err = v.Validate(context.Background(), &domain.Artifact{Body: []byte(`{}`)}, domain.OutputContract{
|
||||
ValidationMode: domain.ValidationJSONSchema,
|
||||
SchemaPath: "schema.json",
|
||||
})
|
||||
if tc.wantError {
|
||||
if err == nil || !strings.Contains(err.Error(), "unsupported JSON Schema dialect") {
|
||||
t.Fatalf("expected unsupported-dialect error, got %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("expected supported dialect, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
151
json.go
Normal file
151
json.go
Normal file
@@ -0,0 +1,151 @@
|
||||
package promptkit
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MarshalJSON emits prepared-run timestamps only when they are non-zero.
|
||||
func (r PreparedRun) MarshalJSON() ([]byte, error) {
|
||||
var startTime, endTime *time.Time
|
||||
if !r.StartTime.IsZero() {
|
||||
startTime = &r.StartTime
|
||||
}
|
||||
if !r.EndTime.IsZero() {
|
||||
endTime = &r.EndTime
|
||||
}
|
||||
|
||||
var durationMS *int64
|
||||
if r.DurationMS != 0 {
|
||||
durationMS = &r.DurationMS
|
||||
}
|
||||
|
||||
return json.Marshal(struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
OutputContract OutputContract `json:"output_contract"`
|
||||
StructuredOutput *StructuredOutputSpec `json:"structured_output,omitempty"`
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
Messages []RenderedMessage `json:"messages"`
|
||||
StartTime *time.Time `json:"start_time,omitempty"`
|
||||
EndTime *time.Time `json:"end_time,omitempty"`
|
||||
DurationMS *int64 `json:"duration_ms,omitempty"`
|
||||
}{
|
||||
PromptID: r.PromptID,
|
||||
PromptVersion: r.PromptVersion,
|
||||
PromptHash: r.PromptHash,
|
||||
SelectedProfileID: r.SelectedProfileID,
|
||||
EffectiveModelParams: r.EffectiveModelParams,
|
||||
OutputContract: r.OutputContract,
|
||||
StructuredOutput: r.StructuredOutput,
|
||||
InputHashes: r.InputHashes,
|
||||
SessionID: r.SessionID,
|
||||
RenderedPromptHash: r.RenderedPromptHash,
|
||||
Messages: r.Messages,
|
||||
StartTime: startTime,
|
||||
EndTime: endTime,
|
||||
DurationMS: durationMS,
|
||||
})
|
||||
}
|
||||
|
||||
// MarshalJSON emits run duration in milliseconds and omits zero timing values.
|
||||
func (r RunResult) MarshalJSON() ([]byte, error) {
|
||||
var startTime, endTime *time.Time
|
||||
if !r.StartTime.IsZero() {
|
||||
startTime = &r.StartTime
|
||||
}
|
||||
if !r.EndTime.IsZero() {
|
||||
endTime = &r.EndTime
|
||||
}
|
||||
|
||||
var durationMS *int64
|
||||
if r.Duration != 0 {
|
||||
value := r.Duration.Milliseconds()
|
||||
durationMS = &value
|
||||
}
|
||||
|
||||
return json.Marshal(runResultJSON{
|
||||
RunID: r.RunID,
|
||||
Artifact: r.Artifact,
|
||||
RawOutput: r.RawOutput,
|
||||
Validation: r.Validation,
|
||||
PromptID: r.PromptID,
|
||||
PromptVersion: r.PromptVersion,
|
||||
PromptHash: r.PromptHash,
|
||||
RenderedPromptHash: r.RenderedPromptHash,
|
||||
SelectedProfileID: r.SelectedProfileID,
|
||||
ModelName: r.ModelName,
|
||||
Endpoint: r.Endpoint,
|
||||
EffectiveModelParams: r.EffectiveModelParams,
|
||||
InputHashes: r.InputHashes,
|
||||
Usage: r.Usage,
|
||||
StartTime: startTime,
|
||||
EndTime: endTime,
|
||||
DurationMS: durationMS,
|
||||
})
|
||||
}
|
||||
|
||||
// UnmarshalJSON decodes the supported run-result representation.
|
||||
func (r *RunResult) UnmarshalJSON(data []byte) error {
|
||||
var wire runResultJSON
|
||||
if err := json.Unmarshal(data, &wire); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
*r = RunResult{
|
||||
RunID: wire.RunID,
|
||||
Artifact: wire.Artifact,
|
||||
RawOutput: wire.RawOutput,
|
||||
Validation: wire.Validation,
|
||||
PromptID: wire.PromptID,
|
||||
PromptVersion: wire.PromptVersion,
|
||||
PromptHash: wire.PromptHash,
|
||||
RenderedPromptHash: wire.RenderedPromptHash,
|
||||
SelectedProfileID: wire.SelectedProfileID,
|
||||
ModelName: wire.ModelName,
|
||||
Endpoint: wire.Endpoint,
|
||||
EffectiveModelParams: wire.EffectiveModelParams,
|
||||
InputHashes: wire.InputHashes,
|
||||
Usage: wire.Usage,
|
||||
Duration: time.Duration(valueOrZero(wire.DurationMS)) * time.Millisecond,
|
||||
}
|
||||
if wire.StartTime != nil {
|
||||
r.StartTime = *wire.StartTime
|
||||
}
|
||||
if wire.EndTime != nil {
|
||||
r.EndTime = *wire.EndTime
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type runResultJSON struct {
|
||||
RunID string `json:"run_id"`
|
||||
Artifact Artifact `json:"artifact"`
|
||||
RawOutput string `json:"raw_output"`
|
||||
Validation ValidationResult `json:"validation"`
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
PromptHash string `json:"prompt_hash,omitempty"`
|
||||
RenderedPromptHash string `json:"rendered_prompt_hash"`
|
||||
SelectedProfileID string `json:"selected_profile_id"`
|
||||
ModelName string `json:"model_name"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
EffectiveModelParams ExecutionTarget `json:"effective_model_params"`
|
||||
InputHashes map[string]string `json:"input_hashes,omitempty"`
|
||||
Usage TokenUsage `json:"usage"`
|
||||
StartTime *time.Time `json:"start_time,omitempty"`
|
||||
EndTime *time.Time `json:"end_time,omitempty"`
|
||||
DurationMS *int64 `json:"duration_ms,omitempty"`
|
||||
}
|
||||
|
||||
func valueOrZero(value *int64) int64 {
|
||||
if value == nil {
|
||||
return 0
|
||||
}
|
||||
return *value
|
||||
}
|
||||
323
public_contract_test.go
Normal file
323
public_contract_test.go
Normal file
@@ -0,0 +1,323 @@
|
||||
package promptkit_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestPreparedRunJSONOmitsZeroTimingValues(t *testing.T) {
|
||||
payload, err := json.Marshal(promptkit.PreparedRun{})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal prepared run: %v", err)
|
||||
}
|
||||
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
|
||||
if strings.Contains(string(payload), `"`+field+`"`) {
|
||||
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunResultJSONUsesMillisecondsAndRoundTrips(t *testing.T) {
|
||||
start := time.Date(2026, time.July, 29, 12, 0, 0, 0, time.UTC)
|
||||
result := promptkit.RunResult{
|
||||
RunID: "opaque-run-id",
|
||||
Artifact: promptkit.Artifact{Name: "output", ContentType: "text/plain", Body: []byte("ok")},
|
||||
StartTime: start,
|
||||
EndTime: start.Add(1500 * time.Millisecond),
|
||||
Duration: 1500 * time.Millisecond,
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(result)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal run result: %v", err)
|
||||
}
|
||||
var object map[string]any
|
||||
if err := json.Unmarshal(payload, &object); err != nil {
|
||||
t.Fatalf("decode run result JSON: %v", err)
|
||||
}
|
||||
if got := object["duration_ms"]; got != float64(1500) {
|
||||
t.Fatalf("expected duration_ms=1500, got %#v in %s", got, payload)
|
||||
}
|
||||
if _, exists := object["duration"]; exists {
|
||||
t.Fatalf("unexpected nanosecond duration field in %s", payload)
|
||||
}
|
||||
artifact, ok := object["artifact"].(map[string]any)
|
||||
if !ok || artifact["content_type"] != "text/plain" {
|
||||
t.Fatalf("expected stable artifact JSON fields, got %#v", object["artifact"])
|
||||
}
|
||||
|
||||
var decoded promptkit.RunResult
|
||||
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal run result: %v", err)
|
||||
}
|
||||
if decoded.Duration != result.Duration || !decoded.StartTime.Equal(result.StartTime) || !decoded.EndTime.Equal(result.EndTime) {
|
||||
t.Fatalf("timing values did not round trip: got %#v, want %#v", decoded, result)
|
||||
}
|
||||
|
||||
payload, err = json.Marshal(promptkit.RunResult{})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal zero run result: %v", err)
|
||||
}
|
||||
for _, field := range []string{"start_time", "end_time", "duration_ms"} {
|
||||
if strings.Contains(string(payload), `"`+field+`"`) {
|
||||
t.Fatalf("expected zero %s to be omitted, got %s", field, payload)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
|
||||
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
|
||||
|
||||
t.Run("prompt source", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("first", "profile", "first"), "."),
|
||||
promptkit.WithPromptFS(contractPromptFS("second", "profile", "second"), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "second"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last prompt source: %v", err)
|
||||
}
|
||||
if prepared.Messages[0].Content != "second" {
|
||||
t.Fatalf("expected last prompt source, got %#v", prepared.Messages)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("profile source", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfileFS(contractProfileFS("profile", "first-model"), "."),
|
||||
promptkit.WithProfileFS(contractProfileFS("profile", "second-model"), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last profile source: %v", err)
|
||||
}
|
||||
if prepared.EffectiveModelParams.Model != "second-model" {
|
||||
t.Fatalf("expected last profile source, got %q", prepared.EffectiveModelParams.Model)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("in-memory profiles", func(t *testing.T) {
|
||||
first := profile
|
||||
first.Model = "first-model"
|
||||
second := profile
|
||||
second.Model = "second-model"
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfiles(first),
|
||||
promptkit.WithProfiles(second),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "prompt"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last in-memory profile option: %v", err)
|
||||
}
|
||||
if prepared.EffectiveModelParams.Model != "second-model" {
|
||||
t.Fatalf("expected last in-memory profiles, got %q", prepared.EffectiveModelParams.Model)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("schema source", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractSchemaPromptFS(), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
promptkit.WithSchemaFS(contractSchemaFS("first"), "."),
|
||||
promptkit.WithSchemaFS(contractSchemaFS("second"), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: "schema-prompt"})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare from last schema source: %v", err)
|
||||
}
|
||||
schema := prepared.StructuredOutput.JSONSchema.Schema.(map[string]any)
|
||||
if schema["title"] != "second" {
|
||||
t.Fatalf("expected last schema source, got %#v", schema)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("model client", func(t *testing.T) {
|
||||
var firstCalls, secondCalls atomic.Int64
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
promptkit.WithLLMClient(countingLLMClient{calls: &firstCalls}),
|
||||
promptkit.WithLLMClient(countingLLMClient{calls: &secondCalls}),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
if _, err := engine.Run(context.Background(), promptkit.RunRequest{PromptID: "prompt"}); err != nil {
|
||||
t.Fatalf("run with last model client: %v", err)
|
||||
}
|
||||
if firstCalls.Load() != 0 || secondCalls.Load() != 1 {
|
||||
t.Fatalf("expected only last client call, got first=%d second=%d", firstCalls.Load(), secondCalls.Load())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("artifact reader", func(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractInputPromptFS(), "."),
|
||||
promptkit.WithProfiles(profile),
|
||||
promptkit.WithArtifactReader(fixedArtifactReader("first")),
|
||||
promptkit.WithArtifactReader(fixedArtifactReader("second")),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: "input-prompt",
|
||||
Inputs: map[string]promptkit.ArtifactRef{"input": promptkit.Inline("ignored")},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("prepare with last artifact reader: %v", err)
|
||||
}
|
||||
if prepared.Messages[0].Content != "second" {
|
||||
t.Fatalf("expected last artifact reader, got %#v", prepared.Messages)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestEngineSupportsConcurrentPrepareAndRun(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(contractPromptFS("prompt", "profile", "message"), "."),
|
||||
promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "profile",
|
||||
Endpoint: "http://example.test/v1",
|
||||
Model: "model",
|
||||
}),
|
||||
promptkit.WithLLMClient(countingLLMClient{}),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("construct engine: %v", err)
|
||||
}
|
||||
|
||||
const calls = 40
|
||||
errs := make(chan error, calls)
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < calls; i++ {
|
||||
wg.Add(1)
|
||||
go func(run bool) {
|
||||
defer wg.Done()
|
||||
request := promptkit.RunRequest{PromptID: "prompt"}
|
||||
if run {
|
||||
_, err := engine.Run(context.Background(), request)
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
_, err := engine.Prepare(context.Background(), request)
|
||||
errs <- err
|
||||
}(i%2 == 0)
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
if err != nil {
|
||||
t.Fatalf("concurrent call failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type countingLLMClient struct {
|
||||
calls *atomic.Int64
|
||||
}
|
||||
|
||||
func (c countingLLMClient) Generate(context.Context, promptkit.GenerateRequest) (*promptkit.GenerateResponse, error) {
|
||||
if c.calls != nil {
|
||||
c.calls.Add(1)
|
||||
}
|
||||
return &promptkit.GenerateResponse{Content: "ok"}, nil
|
||||
}
|
||||
|
||||
type fixedArtifactReader string
|
||||
|
||||
func (r fixedArtifactReader) Read(context.Context, promptkit.ArtifactRef) (*promptkit.Artifact, error) {
|
||||
return &promptkit.Artifact{Body: []byte(r)}, nil
|
||||
}
|
||||
|
||||
func contractPromptFS(id, profileID, message string) fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"prompt.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
|
||||
version: "1"
|
||||
default_profile: %s
|
||||
messages:
|
||||
- role: user
|
||||
content: %q
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`, id, profileID, message))},
|
||||
}
|
||||
}
|
||||
|
||||
func contractInputPromptFS() fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: input-prompt
|
||||
version: "1"
|
||||
default_profile: profile
|
||||
inputs:
|
||||
- name: input
|
||||
required: true
|
||||
messages:
|
||||
- role: user
|
||||
content: '{{input "input"}}'
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
`)},
|
||||
}
|
||||
}
|
||||
|
||||
func contractProfileFS(id, model string) fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"profile.yaml": &fstest.MapFile{Data: []byte(fmt.Sprintf(`id: %s
|
||||
endpoint: http://example.test/v1
|
||||
model: %s
|
||||
`, id, model))},
|
||||
}
|
||||
}
|
||||
|
||||
func contractSchemaPromptFS() fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"prompt.yaml": &fstest.MapFile{Data: []byte(`id: schema-prompt
|
||||
version: "1"
|
||||
default_profile: profile
|
||||
messages:
|
||||
- role: user
|
||||
content: message
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: schema.json
|
||||
`)},
|
||||
}
|
||||
}
|
||||
|
||||
func contractSchemaFS(title string) fstest.MapFS {
|
||||
return fstest.MapFS{
|
||||
"schema.json": &fstest.MapFile{Data: []byte(fmt.Sprintf(`{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": %q,
|
||||
"type": "object"
|
||||
}`, title))},
|
||||
}
|
||||
}
|
||||
23
types.go
23
types.go
@@ -65,7 +65,6 @@ type RunRequest struct {
|
||||
Vars map[string]string
|
||||
Execution *ExecutionTargetOverride
|
||||
Validation *OutputContract
|
||||
Metadata map[string]string
|
||||
}
|
||||
|
||||
// PreparedRun contains prepared prompt execution state. It does not include
|
||||
@@ -106,7 +105,7 @@ type RunResult struct {
|
||||
Usage TokenUsage `json:"usage"`
|
||||
StartTime time.Time `json:"start_time,omitempty"`
|
||||
EndTime time.Time `json:"end_time,omitempty"`
|
||||
Duration time.Duration `json:"duration,omitempty"`
|
||||
Duration time.Duration `json:"-"`
|
||||
}
|
||||
|
||||
// ArtifactRef represents a reference to prompt input content.
|
||||
@@ -118,12 +117,12 @@ type ArtifactRef struct {
|
||||
|
||||
// Artifact represents loaded artifact content.
|
||||
type Artifact struct {
|
||||
Name string
|
||||
ContentType string
|
||||
Body []byte
|
||||
URI string
|
||||
Size int64
|
||||
Hash string
|
||||
Name string `json:"name"`
|
||||
ContentType string `json:"content_type"`
|
||||
Body []byte `json:"body"`
|
||||
URI string `json:"uri"`
|
||||
Size int64 `json:"size"`
|
||||
Hash string `json:"hash"`
|
||||
}
|
||||
|
||||
// ArtifactReader resolves a prompt input reference into its content.
|
||||
@@ -205,10 +204,10 @@ type OpenAICompatibleProfileConfig struct {
|
||||
// ExecutionTargetPresence tracks which numeric runtime settings were explicit
|
||||
// request overrides.
|
||||
type ExecutionTargetPresence struct {
|
||||
Temperature bool
|
||||
MaxTokens bool
|
||||
TopP bool
|
||||
TimeoutSeconds bool
|
||||
Temperature bool `json:"temperature"`
|
||||
MaxTokens bool `json:"max_tokens"`
|
||||
TopP bool `json:"top_p"`
|
||||
TimeoutSeconds bool `json:"timeout_seconds"`
|
||||
}
|
||||
|
||||
// OutputContract defines output and validation requirements.
|
||||
|
||||
Reference in New Issue
Block a user