diff --git a/docs/development.md b/docs/development.md index 87dd49d..bd433a7 100644 --- a/docs/development.md +++ b/docs/development.md @@ -40,7 +40,7 @@ Start with: | Model-client behavior | The [OpenAI-compatible integration contract](integrations/openai-compatible-chat.md), [internal model-client document](internal/llm.md), and owning package tests. | | Internal package implementation | The [architecture policy](policy/architecture.md), [internal component overview](internal/overview.md), and focused internal document listed for that package. | | Tests or test fixtures | The [testing policy](policy/testing.md), owning package, and focused internal document listed by the component overview. | -| Maintained example | The [example](../examples/go-library/prepare/main.go), [consumer guide](consumers/pkg-promptkit.md), [framework format reference](formats.md), and [documentation policy](policy/documentation.md). | +| Maintained example | The [preparation](../examples/go-library/prepare/main.go), [execution](../examples/go-library/run/main.go), and [observation](../examples/go-library/observe/main.go) examples, [consumer guide](consumers/pkg-promptkit.md), [framework format reference](formats.md), and [documentation policy](policy/documentation.md). | | Documentation | The [documentation policy](policy/documentation.md) and canonical owner of every affected contract. | | Release preparation or publication | The [release policy](policy/release.md) and [release procedure](release.md). | @@ -59,7 +59,7 @@ release-candidate checks have one executable entry point from the repository roo ``` The [script](../scripts/release-check.sh) owns the executable checks: -race-enabled tests, vet, both offline consumer examples with semantic output +race-enabled tests, vet, the three offline consumer examples with semantic output assertions, tracked Go formatting, local Markdown links, module/workspace hygiene, whitespace, and the credential-pattern scan. It does not publish, change files, validate release-note structure, or retire roadmaps. It requires Go, diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 1f58085..a918265 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -14,6 +14,7 @@ contributor workflow and validation. | Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, profile-inspection, and ordinary-run observation values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, and typed capacity and generation error mapping. | [Package GoDoc](../../doc.go), [observations](../../observations.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine facade](../../engine.go) | | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) | | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) | +| `examples/go-library/observe` | Demonstrates offline built-in HTTP retry and synchronous metadata observation across structured proposal and assessment runs using embedded sources, two profiles on one finite backend, and an application decision. It is not a public library package. | [Example program](../../examples/go-library/observe/main.go) | | `internal/backend` | Constructs each engine's immutable registry from maintained definitions and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) | | `internal/catalog` | Strictly validates imported immutable maintained backend and profile catalog assets before engine assembly uses them. | [Catalog adapter](../../internal/catalog/catalog.go), [internal sources](sources.md#profiles-and-built-ins) | | `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) | diff --git a/docs/policy/testing.md b/docs/policy/testing.md index 640cb7a..6a8fdf9 100644 --- a/docs/policy/testing.md +++ b/docs/policy/testing.md @@ -64,13 +64,15 @@ Introducing hosted CI later would supplement, not silently redefine, this documented validation model. Full release-candidate validation includes race-enabled package tests, explicit -static analysis, and execution with automated semantic checks of both maintained +static analysis, and execution with automated semantic checks of the three maintained offline consumer examples. The test run builds the packages; the examples also exercise ordinary compilation. Separate ordinary tests and a standalone build are not additional release gates. Ordinary tests remain useful during development. The preparation example protects assembled preparation and inspection behavior. The execution example separately protects assembled -`Run`, injected-client, validation, usage, and result behavior. +`Run`, injected-client, validation, usage, and result behavior. The observation +example protects assembled built-in retries, structured proposal and assessment +runs, and application-owned interpretation of metadata. Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs, use live network @@ -96,7 +98,7 @@ Use each test type where it protects a distinct risk: interaction, while replacing live or nondeterministic external boundaries. - External-package root tests exercise the public facade as a Go consumer, while internal package tests own focused implementation behavior. -- The maintained offline preparation and execution examples protect distinct +- The maintained offline preparation, execution, and observation examples protect distinct representative assembled consumer workflows without contacting a model provider. - Fixtures should be minimal, synthetic, versioned with the behavior they diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 3b5b964..77de6d5 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -451,6 +451,8 @@ Run `go test -race -vet=off ./...` and `go vet ./...`; commit the verified scope ## Stage 11 — Add The Offline Proposal And Assessment Example +**Completed:** 2026-09-23. + Add `examples/go-library/observe` with embedded prompt/schema assets, two application-owned profiles sharing a finite custom backend, and one engine. Use a controlled RoundTripper under the built-in client: return one eligible diff --git a/examples/go-library/observe/main.go b/examples/go-library/observe/main.go new file mode 100644 index 0000000..327b613 --- /dev/null +++ b/examples/go-library/observe/main.go @@ -0,0 +1,157 @@ +package main + +import ( + "context" + "embed" + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "strings" + + "gitea.maximumdirect.net/eric/promptkit" +) + +//go:embed prompts/*.yaml schemas/*.json +var assets embed.FS + +type scriptedTransport struct { + proposalCalls int +} + +func (t *scriptedTransport) RoundTrip(req *http.Request) (*http.Response, error) { + if err := req.Context().Err(); err != nil { + return nil, err + } + var request struct { + Model string `json:"model"` + } + if err := json.NewDecoder(req.Body).Decode(&request); err != nil { + return nil, err + } + switch request.Model { + case "proposal-model": + t.proposalCalls++ + if t.proposalCalls == 1 { + return httpResponse(http.StatusServiceUnavailable, `{"error":{}}`), nil + } + return modelResponse(`{"title":"Queue cleanup","risk":"low"}`, 10, 8), nil + case "assessment-model": + return modelResponse(`{"recommendation":"accept","confidence":0.9}`, 7, 5), nil + default: + return nil, fmt.Errorf("unexpected model in offline transport") + } +} + +func modelResponse(content string, promptTokens, completionTokens int) *http.Response { + body, _ := json.Marshal(map[string]any{ + "choices": []any{map[string]any{"message": map[string]string{"content": content}}}, + "usage": map[string]int{"prompt_tokens": promptTokens, "completion_tokens": completionTokens}, + }) + return httpResponse(http.StatusOK, string(body)) +} + +func httpResponse(status int, body string) *http.Response { + return &http.Response{StatusCode: status, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(body))} +} + +type eventLog struct{ events []promptkit.ExecutionEvent } + +func (l *eventLog) record(event promptkit.ExecutionEvent) { l.events = append(l.events, event) } + +func (l *eventLog) terminal() (promptkit.ExecutionEvent, error) { + if len(l.events) == 0 || l.events[len(l.events)-1].Kind != promptkit.ExecutionFinished { + return promptkit.ExecutionEvent{}, fmt.Errorf("execution did not finish") + } + return l.events[len(l.events)-1], nil +} + +type summary struct { + ProposalAttempts uint64 `json:"proposal_attempts"` + ProposalTransientRetries uint64 `json:"proposal_transient_retries"` + ProposalUsageComplete bool `json:"proposal_usage_complete"` + AssessmentAttempts uint64 `json:"assessment_attempts"` + AssessmentTransientRetries uint64 `json:"assessment_transient_retries"` + AssessmentOutcome string `json:"assessment_outcome"` +} + +func run() (summary, error) { + transport := &scriptedTransport{} + engine, err := promptkit.NewEngine(promptkit.Config{HTTPClient: &http.Client{Transport: transport}}, + promptkit.WithPromptFS(assets, "prompts"), + promptkit.WithSchemaFS(assets, "schemas"), + promptkit.WithBackend(promptkit.Backend{ID: "offline-backend", Endpoint: "https://example.invalid/v1", ConcurrencyLimit: 1}), + promptkit.WithProfiles( + promptkit.Profile{ID: "proposal-profile", BackendID: "offline-backend", Model: "proposal-model"}, + promptkit.Profile{ID: "assessment-profile", BackendID: "offline-backend", Model: "assessment-model"}, + )) + if err != nil { + return summary{}, err + } + + var proposalLog eventLog + proposal, err := engine.Run(context.Background(), promptkit.RunRequest{ + PromptID: "example.proposal", Inputs: map[string]promptkit.ArtifactRef{"idea": promptkit.Inline("Reduce queue cleanup toil.")}, Observer: proposalLog.record, + }) + if err != nil { + return summary{}, err + } + if !proposal.Validation.IsValid { + return summary{}, fmt.Errorf("proposal failed structural validation") + } + proposalTerminal, err := proposalLog.terminal() + if err != nil { + return summary{}, err + } + var assessmentLog eventLog + assessment, err := engine.Run(context.Background(), promptkit.RunRequest{ + PromptID: "example.assessment", Inputs: map[string]promptkit.ArtifactRef{"proposal": promptkit.Inline(proposal.RawOutput)}, Observer: assessmentLog.record, + }) + if err != nil { + return summary{}, err + } + if !assessment.Validation.IsValid { + return summary{}, fmt.Errorf("assessment failed structural validation") + } + assessmentTerminal, err := assessmentLog.terminal() + if err != nil { + return summary{}, err + } + var proposalValue struct { + Risk string `json:"risk"` + } + var assessmentValue struct { + Recommendation string `json:"recommendation"` + } + if err := json.Unmarshal([]byte(proposal.RawOutput), &proposalValue); err != nil { + return summary{}, err + } + if err := json.Unmarshal([]byte(assessment.RawOutput), &assessmentValue); err != nil { + return summary{}, err + } + decision := "rejected" + if proposalValue.Risk == "low" && assessmentValue.Recommendation == "accept" { + decision = "accepted" + } + return summary{ + ProposalAttempts: proposalTerminal.Attempts, + ProposalTransientRetries: proposalTerminal.TransientRetries, + ProposalUsageComplete: proposalTerminal.Usage.PromptTokens.Complete, + AssessmentAttempts: assessmentTerminal.Attempts, + AssessmentTransientRetries: assessmentTerminal.TransientRetries, + AssessmentOutcome: decision, + }, nil +} + +func main() { + value, err := run() + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + if err := json.NewEncoder(os.Stdout).Encode(value); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} diff --git a/examples/go-library/observe/prompts/assessment.yaml b/examples/go-library/observe/prompts/assessment.yaml new file mode 100644 index 0000000..8813801 --- /dev/null +++ b/examples/go-library/observe/prompts/assessment.yaml @@ -0,0 +1,17 @@ +id: example.assessment +version: "1.0.0" +default_profile: assessment-profile +description: Assess a structured proposal independently of schema validity. +inputs: + - name: proposal + required: true + content_type: application/json +messages: + - role: system + content: Return a concise assessment as JSON. + - role: user + content: '{{input "proposal"}}' +output: + format: json + validation_mode: json_schema + schema_path: assessment.schema.json diff --git a/examples/go-library/observe/prompts/proposal.yaml b/examples/go-library/observe/prompts/proposal.yaml new file mode 100644 index 0000000..66bcf2f --- /dev/null +++ b/examples/go-library/observe/prompts/proposal.yaml @@ -0,0 +1,17 @@ +id: example.proposal +version: "1.0.0" +default_profile: proposal-profile +description: Propose a bounded improvement from a synthetic idea. +inputs: + - name: idea + required: true + content_type: text/plain +messages: + - role: system + content: Return a concise proposal as JSON. + - role: user + content: '{{input "idea"}}' +output: + format: json + validation_mode: json_schema + schema_path: proposal.schema.json diff --git a/examples/go-library/observe/schemas/assessment.schema.json b/examples/go-library/observe/schemas/assessment.schema.json new file mode 100644 index 0000000..b74adfe --- /dev/null +++ b/examples/go-library/observe/schemas/assessment.schema.json @@ -0,0 +1,10 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "required": ["recommendation", "confidence"], + "properties": { + "recommendation": {"type": "string"}, + "confidence": {"type": "number"} + }, + "additionalProperties": false +} diff --git a/examples/go-library/observe/schemas/proposal.schema.json b/examples/go-library/observe/schemas/proposal.schema.json new file mode 100644 index 0000000..12322e7 --- /dev/null +++ b/examples/go-library/observe/schemas/proposal.schema.json @@ -0,0 +1,10 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "required": ["title", "risk"], + "properties": { + "title": {"type": "string"}, + "risk": {"type": "string"} + }, + "additionalProperties": false +} diff --git a/scripts/release-check.sh b/scripts/release-check.sh index 2ff321c..cf12d0e 100755 --- a/scripts/release-check.sh +++ b/scripts/release-check.sh @@ -24,6 +24,13 @@ assert result["is_valid"] is True, result assert result["validation_status"] == "passed", result assert result["model"] == "offline-model", result assert result["total_tokens"] == 18, result +observed = example("observe") +assert observed["proposal_attempts"] == 2, observed +assert observed["proposal_transient_retries"] == 1, observed +assert observed["proposal_usage_complete"] is False, observed +assert observed["assessment_attempts"] == 1, observed +assert observed["assessment_transient_retries"] == 0, observed +assert observed["assessment_outcome"] == "accepted", observed PY unformatted=$(