diff --git a/docs/config.md b/docs/config.md index 9d93c46..bcc43b4 100644 --- a/docs/config.md +++ b/docs/config.md @@ -93,10 +93,10 @@ promptkit: | **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. | | **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. | -Set at most one of these fields. Profile IDs used by a binding must be available -from the selected PromptKit profile source when the pipeline is resolved. -The optional local registration may coexist with either profile source or with -PromptKit's built-in profiles. +Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a +binding must be available from the selected PromptKit profile source when the +pipeline is resolved. The optional local registration may coexist with either +profile source or with PromptKit's built-in profiles. When **local_backend** is present, its endpoint is trimmed and must use HTTP or HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index f2d4a4e..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,648 +0,0 @@ -# PromptKit v0.3.0 Upgrade And Local Backend Adoption - -## Status - -Planned. - -This roadmap is an implementation specification for upgrading Notarius from -`gitea.maximumdirect.net/eric/promptkit` v0.2.0 to v0.3.0 and adopting the new -conventional local-backend helper. Implement the stages in order. Each stage -must leave its focused tests passing before the next stage begins. - -The upstream release is additive. Existing endpoint-only profiles, custom -engine options used by tests, and the built-in OpenRouter backend remain -supported. The implementation must preserve those paths. - -## Outcome - -After all stages: - -- Notarius pins PromptKit v0.3.0. -- Version 4 Notarius configuration may register one optional conventional - PromptKit `local` backend. -- A PromptKit-owned file profile may select that registration with - `backend: local`. -- The registration supplies one shared endpoint and an optional PromptKit - concurrency limit to every profile that selects it. -- CLI profile preflight and the production LLM client construct equivalent - PromptKit backend registrations. -- Run manifests and debug material continue to expose PromptKit's selected - backend ID through the existing provenance paths. -- Checkpoint identity changes when the pinned PromptKit release or configured - local endpoint changes, but not when only the local concurrency limit - changes. -- Existing configurations and endpoint-only profiles behave as before. - -## Upstream Contracts - -Use the v0.3.0 tag, rather than the moving `main` branch, as the implementation -contract: - -- release guide: - `https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/releases/v0.3.0.md` -- Go consumer guide: - `https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md` -- profile format: - `https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md` -- backend API and GoDoc: - `https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go` - -The new API used by this work is: - -```go -promptkit.WithBackend( - promptkit.LocalBackend(endpoint, concurrencyLimit), -) -``` - -`LocalBackend` creates an ordinary engine-scoped backend whose ID is -`promptkit.BackendLocal`, currently the case-sensitive string `local`. It does -not register itself, read environment variables, select a model, or add -credentials. A positive concurrency limit uses PromptKit's default waiting -capacity; zero leaves the backend unlimited inside PromptKit. - -## Decisions - -### Public configuration - -Keep `SupportedFileConfigVersion` at 4. The new field is optional and additive: - -```yaml -version: 4 - -promptkit: - profile_dir: ./profiles - # profile_file: ./profiles.yml - local_backend: - endpoint: http://localhost:8000/v1 - concurrency_limit: 2 -``` - -The selected PromptKit profile remains in the external PromptKit profile -source: - -```yaml -id: local-summary -backend: local -model: example-model -``` - -`promptkit.local_backend` has these semantics: - -- The entire object is optional. -- When the object is present, `endpoint` is required. -- File loading trims `endpoint`; an explicitly blank value is invalid. -- The effective endpoint must be an absolute HTTP or HTTPS URL with a host. - User information, query strings, and fragments are invalid. URL paths are - allowed. -- `concurrency_limit` is optional and defaults to zero. -- A negative concurrency limit is invalid. -- Zero means no PromptKit-local concurrency restriction. The existing - application-wide `concurrency.total_llm` scheduler remains in force. -- A positive value limits simultaneous generations for profiles selecting - `backend: local`. PromptKit owns its default queue capacity and admission - behavior. -- The local backend may coexist with either `profile_dir`, `profile_file`, or - built-in profiles. The existing mutual exclusion between `profile_dir` and - `profile_file` remains unchanged. -- No environment override is added for either local-backend field. -- The object contains no credential. A profile may continue to name its own - `api_key_env` according to the PromptKit profile format. - -Use pointer presence for the nested object in both the file model and effective -configuration. This distinguishes an absent registration from a configured -registration whose concurrency limit is zero. - -Add these effective configuration values in `internal/core/config/config.go`: - -```go -type PromptKitConfig struct { - ProfileDir string `json:"profile_dir,omitempty"` - ProfileFile string `json:"profile_file,omitempty"` - LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"` -} - -type PromptKitLocalBackendConfig struct { - Endpoint string `json:"endpoint"` - ConcurrencyLimit int `json:"concurrency_limit"` -} -``` - -Add a corresponding YAML file model in -`internal/core/config/file_config.go`. Use pointer fields inside the file model -where necessary to distinguish omission from an explicitly empty value: - -```go -type FilePromptKitConfig struct { - ProfileDir *string `yaml:"profile_dir,omitempty"` - ProfileFile *string `yaml:"profile_file,omitempty"` - LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"` -} - -type FilePromptKitLocalBackendConfig struct { - Endpoint *string `yaml:"endpoint,omitempty"` - ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"` -} -``` - -The existing YAML decoder's `KnownFields(true)` behavior must reject unknown -fields within `local_backend`. - -### Ownership and redaction - -`internal/core/config` owns parsing, normalization, structural validation, -copying, and the user-facing configuration contract. It must not import -PromptKit merely to validate the URL. - -The nested effective configuration pointer must be deep-copied by -`cloneConfig`. `Config.Redacted` and effective-config summaries already begin -from that clone and should preserve the endpoint and concurrency limit. Treat -the endpoint as non-secret configuration metadata, like the existing profile -source path. Do not add any credential value to this object. - -The framework LLM adapter owns conversion to `promptkit.LocalBackend` and -PromptKit engine construction. The CLI composition root maps the core -configuration type to the adapter type; the framework package must not import -`internal/core/config`. - -### Adapter representation and shared option construction - -Add an adapter-owned value in `internal/framework/llm`: - -```go -type PromptKitLocalBackendConfig struct { - Endpoint string - ConcurrencyLimit int -} -``` - -Add an optional `LocalBackend *PromptKitLocalBackendConfig` field to -`PromptKitClientConfig`. `NewPromptKitClient` must: - -- defensively copy the pointed-to value; -- add a `promptkit.WithBackend(promptkit.LocalBackend(...))` option when it is - present; -- add no backend option when it is absent; and -- retain the trimmed endpoint needed for checkpoint fingerprinting. - -Provide this small exported helper in `internal/framework/llm`: - -```go -func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option -``` - -It converts the adapter local-backend value directly to -`promptkit.WithBackend(promptkit.LocalBackend(...))`. Both -`NewPromptKitClient` and the CLI's preparation-only profile validation engine -must call this helper. The helper must be a direct conversion with no defaults -or hidden environment behavior. Its purpose is to prevent preflight and -runtime registration from drifting. - -The production CLI construction in `internal/cli/catalog.go` maps -`config.PromptKit.LocalBackend` into the adapter-owned value. The profile -validation path in `internal/cli/promptkit_profiles.go` performs the same -mapping and appends the shared option to its preparation-only engine. - -Do not remove or reinterpret `PromptKitClientConfig.EngineOptions`. It remains -the test and internal extension seam, and those options continue to be appended -after asset and profile-source options. Ensure the local backend registration -is added exactly once. - -### Scheduling and capacity - -Do not change the existing Notarius scheduled client. It remains the outer, -application-wide FIFO limit. PromptKit's local backend limit remains an inner -per-engine, per-backend limit: - -```text -Notarius total LLM scheduler - -> PromptKit local backend admission - -> local OpenAI-compatible endpoint -``` - -When the local limit is positive, the maximum active calls against the local -endpoint is effectively the smaller of `concurrency.total_llm` and -`promptkit.local_backend.concurrency_limit`. When the local limit is zero, only -the Notarius scheduler limits calls. - -Do not add another retry or translate errors in a new location. -`PromptKitClient.CompleteStructured` already maps -`promptkit.ErrCapacityExceeded` to `contracts.ErrLLMCapacityExceeded`, and the -pipeline remains responsible for retry decisions. - -### Checkpoint identity - -Keep the existing `promptkit_profile_source` fingerprint and change its -compiled-in release marker from: - -```text -promptkit:v0.2.0:builtin-profiles -``` - -to: - -```text -promptkit:v0.3.0:builtin-profiles -``` - -With the current length-prefixed hashing algorithm, the no-external-profile -fingerprint becomes: - -```text -sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276 -``` - -Add a second fingerprint only when a local backend is configured: - -- name: `promptkit_local_backend_target` -- value: a SHA-256 digest over two length-prefixed parts: - 1. the stable marker `notarius:promptkit-local-backend:v1`; - 2. the trimmed endpoint. - -Reuse the existing length-prefix helper so concatenated inputs are -unambiguous. Do not put the raw endpoint in checkpoint identity. - -Do not include `concurrency_limit` in this fingerprint. The endpoint selects -the semantic execution target; the concurrency limit changes scheduling and -admission only. Return fingerprints in deterministic order: profile source -first, optional local backend target second. - -No manifest schema change is needed. Successful calls already record -PromptKit's selected backend ID, so profiles using the new registration will -produce the existing optional `backend_id: "local"` provenance. Do not add the -endpoint to run manifests. - -### Scope exclusions - -Do not add: - -- arbitrary user-defined backend IDs; -- more than one configured local endpoint; -- backend-wide API-key fields; -- backend-wide extra request parameters; -- configurable PromptKit queue capacity; -- automatic local endpoint discovery; -- environment-variable overrides for local-backend configuration; -- an implicit local backend when the object is absent; -- a default model or an in-memory local profile; -- CLI flags for local backend settings; -- changes to existing maintained example configurations; -- a configuration version bump; -- a new ADR. - -Those capabilities require the full PromptKit `Backend` interface or a broader -product decision and are outside this release adoption. - -## Stage 1: Pin PromptKit v0.3.0 - -### Changes - -1. Run: - - ```sh - go get gitea.maximumdirect.net/eric/promptkit@v0.3.0 - go mod tidy - ``` - -2. Confirm `go.mod` directly requires exactly v0.3.0 and `go.sum` contains the - v0.3.0 module and Go module checksums with no v0.2.0 PromptKit entries. -3. Update `promptKitBuiltinProfileCatalogID` in - `internal/framework/llm/promptkit_profile_fingerprint.go`. -4. Update the exact built-in fingerprint expectation in - `internal/framework/llm/promptkit_client_test.go` to the value specified - above. -5. Do not use the new local-backend API in this stage. - -### Verification - -Run: - -```sh -go test ./internal/framework/llm -go test ./... -go vet ./... -go build ./cmd/notarius -``` - -### Completion criteria - -- The repository builds against PromptKit v0.3.0 without compatibility - shims. -- All pre-existing behavior passes unchanged. -- Checkpoints created under the v0.2.0 release identity are not considered - identical to v0.3.0 checkpoints. - -## Stage 2: Add The Version 4 Configuration Contract - -### Changes - -1. Add the effective and file-model types described in the decisions above. -2. In `Config.ApplyFileConfig`: - - recognize an absent object without changing defaults; - - require a present object to contain a nonblank endpoint; - - trim the endpoint before storing it; - - default an omitted concurrency limit to zero; - - allocate a new effective nested value so the parsed file model cannot - alias the resulting `Config`. -3. Extend `validatePromptKit`: - - preserve profile directory/file mutual exclusion; - - reject an effective local backend with a blank endpoint; - - parse the trimmed endpoint with `net/url`; - - accept `http` or `https` case-insensitively; - - require `URL.IsAbs()` and a nonempty `URL.Hostname()`; - - reject URL user information; - - reject both a nonempty `URL.RawQuery` and `URL.ForceQuery`; - - reject any endpoint text containing `#`, including an empty trailing - fragment; - - allow URL paths; - - reject a negative concurrency limit. -4. Extend `cloneConfig` to deep-copy `PromptKit.LocalBackend`. -5. Make no environment-override changes. - -Use contextual errors rooted at the public field names, for example: - -```text -promptkit.local_backend.endpoint must not be empty when set -promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment -promptkit.local_backend.concurrency_limit must not be negative -``` - -Exact wrapping may follow existing configuration conventions, but tests should -assert stable field context rather than whole error strings. - -### Tests - -Extend the current configuration contract suites rather than creating a -parallel test framework: - -- `internal/core/config/file_config_contract_test.go` - - decodes the documented object; - - trims the endpoint; - - defaults omitted concurrency to zero; - - preserves an explicit positive concurrency value; - - rejects a missing or explicitly blank endpoint; - - rejects an unknown nested field; - - confirms cloning and `Redacted` preserve values without pointer aliasing; - - confirms runtime JSON uses `local_backend`, `endpoint`, and - `concurrency_limit`. -- `internal/core/config/validation_contract_test.go` - - accepts representative HTTP and HTTPS endpoints, including a path; - - rejects relative URLs, unsupported schemes, missing hosts, user - information, queries, fragments, and negative concurrency; - - confirms the object may coexist with either valid profile-source choice. -- `internal/core/config/effective_config_contract_test.go` - - confirms resolution preserves an independently owned local-backend value; - - mutating either the input or resolved copy must not mutate the other. - -Avoid tests for `net/url` itself. Cover only the public categories in the -configuration contract. - -### Verification - -Run: - -```sh -go test ./internal/core/config -go test ./... -``` - -### Completion criteria - -- Old version 4 files load exactly as before. -- The new object is strict, validated, normalized, independently owned, and - visible through the existing safe configuration summaries. -- No runtime behavior uses the new values yet. - -## Stage 3: Register The Local Backend In Preflight And Runtime - -### Changes - -1. Add the adapter-owned configuration value and shared PromptKit option - helper described above. -2. Extend `PromptKitClientConfig` and `NewPromptKitClient` to register the - configured backend before calling `promptkit.NewEngine`. -3. In `internal/cli/catalog.go`, map the core configuration into - `PromptKitClientConfig.LocalBackend`. -4. In `internal/cli/promptkit_profiles.go`, add the identical registration to - the preparation-only engine. -5. Preserve option ordering and existing profile-source behavior: - - mounted prompt/schema assets; - - optional profile file; - - optional local backend registration; - - caller-supplied `EngineOptions`. -6. Do not change request mapping, response decoding, selected-backend - recording, scheduler construction, or capacity-error translation. - -The preflight engine must prepare a file-backed profile selecting -`backend: local` without making a provider call. If the profile selects -`local` but `promptkit.local_backend` is absent, preserve PromptKit's -profile-load failure with the existing Notarius profile-validation context; do -not silently fall back to an endpoint-only or OpenRouter profile. - -### Tests - -Add or extend focused behavioral tests: - -- In `internal/framework/llm/promptkit_client_test.go`, configure a profile - selecting `backend: local` and prove that: - - client construction succeeds when the registration is supplied; - - a structured call reaches an `httptest.Server` at the configured endpoint; - - the returned response and recorded manifest use backend ID `local`; - - omitting the registration causes preparation/execution to fail before any - provider request. -- Add `internal/cli/promptkit_profiles_test.go` if no existing CLI test gives a - narrow home for preflight behavior: - - configured local registration permits explicit-profile validation; - - absent registration rejects the same profile with profile and backend - context; - - validation remains preparation-only and does not contact the endpoint. -- Extend an existing production composition test in - `internal/cli/production_contract_test.go` to ensure the effective - configuration reaches the production client path. Exercise the constructed - client far enough to select the local-backed profile; construction alone is - insufficient because backend membership is resolved during preparation. - -Use temporary profile files and `httptest.Server`. Tests must remain offline, -deterministic, and credential-free. - -Do not duplicate PromptKit's constructor-shape tests or comprehensively retest -its backend queue. Notarius owns correct configuration handoff, equivalent -preflight/runtime construction, existing provenance, and error-boundary -behavior. - -### Verification - -Run: - -```sh -go test ./internal/framework/llm ./internal/cli -go test ./... -``` - -### Completion criteria - -- A configured `backend: local` profile passes CLI preflight and uses the - configured endpoint at runtime. -- The same profile fails clearly when the backend is not registered. -- Endpoint-only profiles and built-in profiles continue to work. -- Existing manifest/debug backend provenance records `local` without schema - changes. -- Application-wide and PromptKit-local scheduling remain layered as designed. - -## Stage 4: Add Local Backend Checkpoint Identity - -### Changes - -1. Store a defensive copy of the configured local endpoint on - `PromptKitClient`. -2. Add the optional `promptkit_local_backend_target` fingerprint using the - exact algorithm and marker specified above. -3. Extend `LLMCheckpointFingerprints` to return: - - the existing profile-source fingerprint in all cases; - - the local-target fingerprint second when configured. -4. Preserve defensive-copy behavior on the returned fingerprint slice. -5. Do not add concurrency, queue behavior, raw endpoint text, or credentials - to checkpoint identity. - -Keep this logic with the existing PromptKit profile fingerprint implementation -or in an adjacent focused file under `internal/framework/llm`; do not move -checkpoint policy into the CLI or core configuration package. - -### Tests - -Extend `internal/framework/llm/promptkit_client_test.go` to prove: - -- no local backend returns only the profile-source fingerprint; -- a configured local backend returns exactly two fingerprints in the required - order; -- changing only the endpoint changes the local-target fingerprint; -- changing only `concurrency_limit` leaves every fingerprint unchanged; -- neither fingerprint exposes the raw endpoint; -- mutating a returned slice does not affect a later result. - -The existing scheduled-client fingerprint delegation test is sufficient -unless it assumes exactly one fingerprint. If it does, update it to verify -transparent delegation of multiple values without specializing it to -PromptKit. - -### Verification - -Run: - -```sh -go test ./internal/framework/llm ./internal/cli -go test ./... -``` - -### Completion criteria - -- Checkpoint reuse is invalidated by a changed PromptKit release, profile - source, or local endpoint. -- A concurrency-only operational change does not invalidate semantic - checkpoints. -- Fingerprints remain deterministic, non-secret, and available through the - scheduled-client wrapper. - -## Stage 5: Publish Current Behavior And Complete Validation - -Documentation outside `docs/roadmap` must describe only the implemented result, -so perform these updates after Stages 1 through 4 work. - -### Documentation changes - -1. `docs/integrations/pkg-promptkit.md` - - change the pin and upstream links from v0.2.0 to v0.3.0; - - add `BackendLocal`, `LocalBackend`, and `WithBackend` to the supported - boundary; - - replace the statement that production exposes no user-defined backend - registration with the narrower truth: Notarius exposes one optional - conventional `local` registration and no general backend registry; - - retain endpoint-only profile compatibility and existing ownership - boundaries. -2. `docs/config.md` - - add the exact YAML example from this roadmap; - - document the nested fields and validation rules; - - explain that a PromptKit profile selects it with `backend: local`; - - link profile-format details to the pinned PromptKit reference; - - state that omission leaves current behavior unchanged. -3. `docs/internal/configuration.md` - - mention the independently owned nested PromptKit backend configuration - and its validation boundary. -4. `docs/internal/llm.md` - - document equivalent registration in preflight and production engines; - - describe the two scheduling layers; - - describe the optional endpoint fingerprint and why concurrency is - excluded; - - retain the existing capacity-error and provenance ownership. -5. `docs/operations.md` - - explain the effective local concurrency relationship; - - state that a zero local limit leaves only the application scheduler; - - state that PromptKit owns the positive-limit waiting capacity and emits - the already-mapped capacity failure when admission is exhausted. - -Do not add a complete local configuration under `examples/`. Existing examples -are maintained provider-neutral workflows and should not gain a dependency on -a local inference server. The concise `docs/config.md` fragments are the -canonical user guidance for this optional integration. - -Review `README.md`, `docs/cli.md`, the JSON output contract, and architecture -policy for contradictions. Do not edit them unless the implemented behavior -made a statement inaccurate. No CLI syntax, output schema, or architectural -boundary changes are intended. - -### Full verification - -Run the upstream-requested ordinary and race-enabled tests plus the repository -checks: - -```sh -go test ./... -go test -race ./... -go vet ./... -go build ./cmd/notarius -``` - -Also inspect the final diff and confirm: - -- only intended module checksums changed; -- all upstream documentation links use the v0.3.0 tag; -- no raw credential or environment value was added; -- no endpoint was added to run-manifest output; -- old endpoint-only profile fixtures remain present; -- current-behavior documentation no longer says Notarius cannot register any - production backend; -- this roadmap's completed work is reflected in canonical documentation. - -### Completion criteria - -- All focused, repository-wide, race, vet, and build checks pass. -- The public configuration, internal behavior, integration boundary, and - operational guidance agree. -- The upgrade remains backward compatible for existing Notarius - configurations and PromptKit profiles. -- The only newly supported backend configuration is the explicit conventional - `local` registration defined by this roadmap. - -## Final Acceptance Scenarios - -An implementation is complete only when all scenarios below hold: - -1. A current version 4 configuration with no `local_backend` loads and runs - without behavioral changes. -2. An endpoint-only PromptKit profile continues to run and omits backend ID - exactly as before. -3. A built-in OpenRouter profile continues to resolve through PromptKit's - built-in backend. -4. A file profile containing `backend: local` passes `config validate` when - the Notarius local backend is configured. -5. The same profile fails before provider execution when the registration is - absent. -6. A successful local-backed call reaches the configured endpoint and records - backend ID `local` through existing response, debug, and manifest fields. -7. The Notarius scheduler remains the application-wide ceiling, and a positive - local limit may narrow concurrency for local-backed profiles. -8. Existing capacity rejection mapping and pipeline retry ownership remain - unchanged. -9. Changing the local endpoint prevents checkpoint reuse; changing only its - concurrency limit does not. -10. Configuration summaries own their nested values and contain no credential.