eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clarify PromptKit configuration and retire the completed plan

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-30 15:38:31 +00:00
parent b4363b3b73
commit 9171b66a41
2 changed files with 4 additions and 652 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/config.md
View File

@@ -93,10 +93,10 @@ promptkit:
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
Set at most one of these fields. Profile IDs used by a binding must be available
from the selected PromptKit profile source when the pipeline is resolved.
The optional local registration may coexist with either profile source or with
PromptKit's built-in profiles.
Set at most one of **profile_dir** and **profile_file**. Profile IDs used by a
binding must be available from the selected PromptKit profile source when the
pipeline is resolved. The optional local registration may coexist with either
profile source or with PromptKit's built-in profiles.
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are

648
docs/roadmap/implementation.md
View File

@@ -1,648 +0,0 @@
# PromptKit v0.3.0 Upgrade And Local Backend Adoption
## Status
Planned.
This roadmap is an implementation specification for upgrading Notarius from
`gitea.maximumdirect.net/eric/promptkit` v0.2.0 to v0.3.0 and adopting the new
conventional local-backend helper. Implement the stages in order. Each stage
must leave its focused tests passing before the next stage begins.
The upstream release is additive. Existing endpoint-only profiles, custom
engine options used by tests, and the built-in OpenRouter backend remain
supported. The implementation must preserve those paths.
## Outcome
After all stages:
- Notarius pins PromptKit v0.3.0.
- Version 4 Notarius configuration may register one optional conventional
PromptKit `local` backend.
- A PromptKit-owned file profile may select that registration with
`backend: local`.
- The registration supplies one shared endpoint and an optional PromptKit
concurrency limit to every profile that selects it.
- CLI profile preflight and the production LLM client construct equivalent
PromptKit backend registrations.
- Run manifests and debug material continue to expose PromptKit's selected
backend ID through the existing provenance paths.
- Checkpoint identity changes when the pinned PromptKit release or configured
local endpoint changes, but not when only the local concurrency limit
changes.
- Existing configurations and endpoint-only profiles behave as before.
## Upstream Contracts
Use the v0.3.0 tag, rather than the moving `main` branch, as the implementation
contract:
- release guide:
`https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/releases/v0.3.0.md`
- Go consumer guide:
`https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md`
- profile format:
`https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md`
- backend API and GoDoc:
`https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go`
The new API used by this work is:
```go
promptkit.WithBackend(
promptkit.LocalBackend(endpoint, concurrencyLimit),
)
```
`LocalBackend` creates an ordinary engine-scoped backend whose ID is
`promptkit.BackendLocal`, currently the case-sensitive string `local`. It does
not register itself, read environment variables, select a model, or add
credentials. A positive concurrency limit uses PromptKit's default waiting
capacity; zero leaves the backend unlimited inside PromptKit.
## Decisions
### Public configuration
Keep `SupportedFileConfigVersion` at 4. The new field is optional and additive:
```yaml
version: 4
promptkit:
profile_dir: ./profiles
# profile_file: ./profiles.yml
local_backend:
endpoint: http://localhost:8000/v1
concurrency_limit: 2
```
The selected PromptKit profile remains in the external PromptKit profile
source:
```yaml
id: local-summary
backend: local
model: example-model
```
`promptkit.local_backend` has these semantics:
- The entire object is optional.
- When the object is present, `endpoint` is required.
- File loading trims `endpoint`; an explicitly blank value is invalid.
- The effective endpoint must be an absolute HTTP or HTTPS URL with a host.
User information, query strings, and fragments are invalid. URL paths are
allowed.
- `concurrency_limit` is optional and defaults to zero.
- A negative concurrency limit is invalid.
- Zero means no PromptKit-local concurrency restriction. The existing
application-wide `concurrency.total_llm` scheduler remains in force.
- A positive value limits simultaneous generations for profiles selecting
`backend: local`. PromptKit owns its default queue capacity and admission
behavior.
- The local backend may coexist with either `profile_dir`, `profile_file`, or
built-in profiles. The existing mutual exclusion between `profile_dir` and
`profile_file` remains unchanged.
- No environment override is added for either local-backend field.
- The object contains no credential. A profile may continue to name its own
`api_key_env` according to the PromptKit profile format.
Use pointer presence for the nested object in both the file model and effective
configuration. This distinguishes an absent registration from a configured
registration whose concurrency limit is zero.
Add these effective configuration values in `internal/core/config/config.go`:
```go
type PromptKitConfig struct {
ProfileDir string `json:"profile_dir,omitempty"`
ProfileFile string `json:"profile_file,omitempty"`
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
}
type PromptKitLocalBackendConfig struct {
Endpoint string `json:"endpoint"`
ConcurrencyLimit int `json:"concurrency_limit"`
}
```
Add a corresponding YAML file model in
`internal/core/config/file_config.go`. Use pointer fields inside the file model
where necessary to distinguish omission from an explicitly empty value:
```go
type FilePromptKitConfig struct {
ProfileDir *string `yaml:"profile_dir,omitempty"`
ProfileFile *string `yaml:"profile_file,omitempty"`
LocalBackend *FilePromptKitLocalBackendConfig `yaml:"local_backend,omitempty"`
}
type FilePromptKitLocalBackendConfig struct {
Endpoint *string `yaml:"endpoint,omitempty"`
ConcurrencyLimit *int `yaml:"concurrency_limit,omitempty"`
}
```
The existing YAML decoder's `KnownFields(true)` behavior must reject unknown
fields within `local_backend`.
### Ownership and redaction
`internal/core/config` owns parsing, normalization, structural validation,
copying, and the user-facing configuration contract. It must not import
PromptKit merely to validate the URL.
The nested effective configuration pointer must be deep-copied by
`cloneConfig`. `Config.Redacted` and effective-config summaries already begin
from that clone and should preserve the endpoint and concurrency limit. Treat
the endpoint as non-secret configuration metadata, like the existing profile
source path. Do not add any credential value to this object.
The framework LLM adapter owns conversion to `promptkit.LocalBackend` and
PromptKit engine construction. The CLI composition root maps the core
configuration type to the adapter type; the framework package must not import
`internal/core/config`.
### Adapter representation and shared option construction
Add an adapter-owned value in `internal/framework/llm`:
```go
type PromptKitLocalBackendConfig struct {
Endpoint string
ConcurrencyLimit int
}
```
Add an optional `LocalBackend *PromptKitLocalBackendConfig` field to
`PromptKitClientConfig`. `NewPromptKitClient` must:
- defensively copy the pointed-to value;
- add a `promptkit.WithBackend(promptkit.LocalBackend(...))` option when it is
present;
- add no backend option when it is absent; and
- retain the trimmed endpoint needed for checkpoint fingerprinting.
Provide this small exported helper in `internal/framework/llm`:
```go
func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option
```
It converts the adapter local-backend value directly to
`promptkit.WithBackend(promptkit.LocalBackend(...))`. Both
`NewPromptKitClient` and the CLI's preparation-only profile validation engine
must call this helper. The helper must be a direct conversion with no defaults
or hidden environment behavior. Its purpose is to prevent preflight and
runtime registration from drifting.
The production CLI construction in `internal/cli/catalog.go` maps
`config.PromptKit.LocalBackend` into the adapter-owned value. The profile
validation path in `internal/cli/promptkit_profiles.go` performs the same
mapping and appends the shared option to its preparation-only engine.
Do not remove or reinterpret `PromptKitClientConfig.EngineOptions`. It remains
the test and internal extension seam, and those options continue to be appended
after asset and profile-source options. Ensure the local backend registration
is added exactly once.
### Scheduling and capacity
Do not change the existing Notarius scheduled client. It remains the outer,
application-wide FIFO limit. PromptKit's local backend limit remains an inner
per-engine, per-backend limit:
```text
Notarius total LLM scheduler
-> PromptKit local backend admission
-> local OpenAI-compatible endpoint
```
When the local limit is positive, the maximum active calls against the local
endpoint is effectively the smaller of `concurrency.total_llm` and
`promptkit.local_backend.concurrency_limit`. When the local limit is zero, only
the Notarius scheduler limits calls.
Do not add another retry or translate errors in a new location.
`PromptKitClient.CompleteStructured` already maps
`promptkit.ErrCapacityExceeded` to `contracts.ErrLLMCapacityExceeded`, and the
pipeline remains responsible for retry decisions.
### Checkpoint identity
Keep the existing `promptkit_profile_source` fingerprint and change its
compiled-in release marker from:
```text
promptkit:v0.2.0:builtin-profiles
```
to:
```text
promptkit:v0.3.0:builtin-profiles
```
With the current length-prefixed hashing algorithm, the no-external-profile
fingerprint becomes:
```text
sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276
```
Add a second fingerprint only when a local backend is configured:
- name: `promptkit_local_backend_target`
- value: a SHA-256 digest over two length-prefixed parts:
1. the stable marker `notarius:promptkit-local-backend:v1`;
2. the trimmed endpoint.
Reuse the existing length-prefix helper so concatenated inputs are
unambiguous. Do not put the raw endpoint in checkpoint identity.
Do not include `concurrency_limit` in this fingerprint. The endpoint selects
the semantic execution target; the concurrency limit changes scheduling and
admission only. Return fingerprints in deterministic order: profile source
first, optional local backend target second.
No manifest schema change is needed. Successful calls already record
PromptKit's selected backend ID, so profiles using the new registration will
produce the existing optional `backend_id: "local"` provenance. Do not add the
endpoint to run manifests.
### Scope exclusions
Do not add:
- arbitrary user-defined backend IDs;
- more than one configured local endpoint;
- backend-wide API-key fields;
- backend-wide extra request parameters;
- configurable PromptKit queue capacity;
- automatic local endpoint discovery;
- environment-variable overrides for local-backend configuration;
- an implicit local backend when the object is absent;
- a default model or an in-memory local profile;
- CLI flags for local backend settings;
- changes to existing maintained example configurations;
- a configuration version bump;
- a new ADR.
Those capabilities require the full PromptKit `Backend` interface or a broader
product decision and are outside this release adoption.
## Stage 1: Pin PromptKit v0.3.0
### Changes
1. Run:
```sh
go get gitea.maximumdirect.net/eric/promptkit@v0.3.0
go mod tidy
```
2. Confirm `go.mod` directly requires exactly v0.3.0 and `go.sum` contains the
v0.3.0 module and Go module checksums with no v0.2.0 PromptKit entries.
3. Update `promptKitBuiltinProfileCatalogID` in
`internal/framework/llm/promptkit_profile_fingerprint.go`.
4. Update the exact built-in fingerprint expectation in
`internal/framework/llm/promptkit_client_test.go` to the value specified
above.
5. Do not use the new local-backend API in this stage.
### Verification
Run:
```sh
go test ./internal/framework/llm
go test ./...
go vet ./...
go build ./cmd/notarius
```
### Completion criteria
- The repository builds against PromptKit v0.3.0 without compatibility
shims.
- All pre-existing behavior passes unchanged.
- Checkpoints created under the v0.2.0 release identity are not considered
identical to v0.3.0 checkpoints.
## Stage 2: Add The Version 4 Configuration Contract
### Changes
1. Add the effective and file-model types described in the decisions above.
2. In `Config.ApplyFileConfig`:
- recognize an absent object without changing defaults;
- require a present object to contain a nonblank endpoint;
- trim the endpoint before storing it;
- default an omitted concurrency limit to zero;
- allocate a new effective nested value so the parsed file model cannot
alias the resulting `Config`.
3. Extend `validatePromptKit`:
- preserve profile directory/file mutual exclusion;
- reject an effective local backend with a blank endpoint;
- parse the trimmed endpoint with `net/url`;
- accept `http` or `https` case-insensitively;
- require `URL.IsAbs()` and a nonempty `URL.Hostname()`;
- reject URL user information;
- reject both a nonempty `URL.RawQuery` and `URL.ForceQuery`;
- reject any endpoint text containing `#`, including an empty trailing
fragment;
- allow URL paths;
- reject a negative concurrency limit.
4. Extend `cloneConfig` to deep-copy `PromptKit.LocalBackend`.
5. Make no environment-override changes.
Use contextual errors rooted at the public field names, for example:
```text
promptkit.local_backend.endpoint must not be empty when set
promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment
promptkit.local_backend.concurrency_limit must not be negative
```
Exact wrapping may follow existing configuration conventions, but tests should
assert stable field context rather than whole error strings.
### Tests
Extend the current configuration contract suites rather than creating a
parallel test framework:
- `internal/core/config/file_config_contract_test.go`
- decodes the documented object;
- trims the endpoint;
- defaults omitted concurrency to zero;
- preserves an explicit positive concurrency value;
- rejects a missing or explicitly blank endpoint;
- rejects an unknown nested field;
- confirms cloning and `Redacted` preserve values without pointer aliasing;
- confirms runtime JSON uses `local_backend`, `endpoint`, and
`concurrency_limit`.
- `internal/core/config/validation_contract_test.go`
- accepts representative HTTP and HTTPS endpoints, including a path;
- rejects relative URLs, unsupported schemes, missing hosts, user
information, queries, fragments, and negative concurrency;
- confirms the object may coexist with either valid profile-source choice.
- `internal/core/config/effective_config_contract_test.go`
- confirms resolution preserves an independently owned local-backend value;
- mutating either the input or resolved copy must not mutate the other.
Avoid tests for `net/url` itself. Cover only the public categories in the
configuration contract.
### Verification
Run:
```sh
go test ./internal/core/config
go test ./...
```
### Completion criteria
- Old version 4 files load exactly as before.
- The new object is strict, validated, normalized, independently owned, and
visible through the existing safe configuration summaries.
- No runtime behavior uses the new values yet.
## Stage 3: Register The Local Backend In Preflight And Runtime
### Changes
1. Add the adapter-owned configuration value and shared PromptKit option
helper described above.
2. Extend `PromptKitClientConfig` and `NewPromptKitClient` to register the
configured backend before calling `promptkit.NewEngine`.
3. In `internal/cli/catalog.go`, map the core configuration into
`PromptKitClientConfig.LocalBackend`.
4. In `internal/cli/promptkit_profiles.go`, add the identical registration to
the preparation-only engine.
5. Preserve option ordering and existing profile-source behavior:
- mounted prompt/schema assets;
- optional profile file;
- optional local backend registration;
- caller-supplied `EngineOptions`.
6. Do not change request mapping, response decoding, selected-backend
recording, scheduler construction, or capacity-error translation.
The preflight engine must prepare a file-backed profile selecting
`backend: local` without making a provider call. If the profile selects
`local` but `promptkit.local_backend` is absent, preserve PromptKit's
profile-load failure with the existing Notarius profile-validation context; do
not silently fall back to an endpoint-only or OpenRouter profile.
### Tests
Add or extend focused behavioral tests:
- In `internal/framework/llm/promptkit_client_test.go`, configure a profile
selecting `backend: local` and prove that:
- client construction succeeds when the registration is supplied;
- a structured call reaches an `httptest.Server` at the configured endpoint;
- the returned response and recorded manifest use backend ID `local`;
- omitting the registration causes preparation/execution to fail before any
provider request.
- Add `internal/cli/promptkit_profiles_test.go` if no existing CLI test gives a
narrow home for preflight behavior:
- configured local registration permits explicit-profile validation;
- absent registration rejects the same profile with profile and backend
context;
- validation remains preparation-only and does not contact the endpoint.
- Extend an existing production composition test in
`internal/cli/production_contract_test.go` to ensure the effective
configuration reaches the production client path. Exercise the constructed
client far enough to select the local-backed profile; construction alone is
insufficient because backend membership is resolved during preparation.
Use temporary profile files and `httptest.Server`. Tests must remain offline,
deterministic, and credential-free.
Do not duplicate PromptKit's constructor-shape tests or comprehensively retest
its backend queue. Notarius owns correct configuration handoff, equivalent
preflight/runtime construction, existing provenance, and error-boundary
behavior.
### Verification
Run:
```sh
go test ./internal/framework/llm ./internal/cli
go test ./...
```
### Completion criteria
- A configured `backend: local` profile passes CLI preflight and uses the
configured endpoint at runtime.
- The same profile fails clearly when the backend is not registered.
- Endpoint-only profiles and built-in profiles continue to work.
- Existing manifest/debug backend provenance records `local` without schema
changes.
- Application-wide and PromptKit-local scheduling remain layered as designed.
## Stage 4: Add Local Backend Checkpoint Identity
### Changes
1. Store a defensive copy of the configured local endpoint on
`PromptKitClient`.
2. Add the optional `promptkit_local_backend_target` fingerprint using the
exact algorithm and marker specified above.
3. Extend `LLMCheckpointFingerprints` to return:
- the existing profile-source fingerprint in all cases;
- the local-target fingerprint second when configured.
4. Preserve defensive-copy behavior on the returned fingerprint slice.
5. Do not add concurrency, queue behavior, raw endpoint text, or credentials
to checkpoint identity.
Keep this logic with the existing PromptKit profile fingerprint implementation
or in an adjacent focused file under `internal/framework/llm`; do not move
checkpoint policy into the CLI or core configuration package.
### Tests
Extend `internal/framework/llm/promptkit_client_test.go` to prove:
- no local backend returns only the profile-source fingerprint;
- a configured local backend returns exactly two fingerprints in the required
order;
- changing only the endpoint changes the local-target fingerprint;
- changing only `concurrency_limit` leaves every fingerprint unchanged;
- neither fingerprint exposes the raw endpoint;
- mutating a returned slice does not affect a later result.
The existing scheduled-client fingerprint delegation test is sufficient
unless it assumes exactly one fingerprint. If it does, update it to verify
transparent delegation of multiple values without specializing it to
PromptKit.
### Verification
Run:
```sh
go test ./internal/framework/llm ./internal/cli
go test ./...
```
### Completion criteria
- Checkpoint reuse is invalidated by a changed PromptKit release, profile
source, or local endpoint.
- A concurrency-only operational change does not invalidate semantic
checkpoints.
- Fingerprints remain deterministic, non-secret, and available through the
scheduled-client wrapper.
## Stage 5: Publish Current Behavior And Complete Validation
Documentation outside `docs/roadmap` must describe only the implemented result,
so perform these updates after Stages 1 through 4 work.
### Documentation changes
1. `docs/integrations/pkg-promptkit.md`
- change the pin and upstream links from v0.2.0 to v0.3.0;
- add `BackendLocal`, `LocalBackend`, and `WithBackend` to the supported
boundary;
- replace the statement that production exposes no user-defined backend
registration with the narrower truth: Notarius exposes one optional
conventional `local` registration and no general backend registry;
- retain endpoint-only profile compatibility and existing ownership
boundaries.
2. `docs/config.md`
- add the exact YAML example from this roadmap;
- document the nested fields and validation rules;
- explain that a PromptKit profile selects it with `backend: local`;
- link profile-format details to the pinned PromptKit reference;
- state that omission leaves current behavior unchanged.
3. `docs/internal/configuration.md`
- mention the independently owned nested PromptKit backend configuration
and its validation boundary.
4. `docs/internal/llm.md`
- document equivalent registration in preflight and production engines;
- describe the two scheduling layers;
- describe the optional endpoint fingerprint and why concurrency is
excluded;
- retain the existing capacity-error and provenance ownership.
5. `docs/operations.md`
- explain the effective local concurrency relationship;
- state that a zero local limit leaves only the application scheduler;
- state that PromptKit owns the positive-limit waiting capacity and emits
the already-mapped capacity failure when admission is exhausted.
Do not add a complete local configuration under `examples/`. Existing examples
are maintained provider-neutral workflows and should not gain a dependency on
a local inference server. The concise `docs/config.md` fragments are the
canonical user guidance for this optional integration.
Review `README.md`, `docs/cli.md`, the JSON output contract, and architecture
policy for contradictions. Do not edit them unless the implemented behavior
made a statement inaccurate. No CLI syntax, output schema, or architectural
boundary changes are intended.
### Full verification
Run the upstream-requested ordinary and race-enabled tests plus the repository
checks:
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./cmd/notarius
```
Also inspect the final diff and confirm:
- only intended module checksums changed;
- all upstream documentation links use the v0.3.0 tag;
- no raw credential or environment value was added;
- no endpoint was added to run-manifest output;
- old endpoint-only profile fixtures remain present;
- current-behavior documentation no longer says Notarius cannot register any
production backend;
- this roadmap's completed work is reflected in canonical documentation.
### Completion criteria
- All focused, repository-wide, race, vet, and build checks pass.
- The public configuration, internal behavior, integration boundary, and
operational guidance agree.
- The upgrade remains backward compatible for existing Notarius
configurations and PromptKit profiles.
- The only newly supported backend configuration is the explicit conventional
`local` registration defined by this roadmap.
## Final Acceptance Scenarios
An implementation is complete only when all scenarios below hold:
1. A current version 4 configuration with no `local_backend` loads and runs
without behavioral changes.
2. An endpoint-only PromptKit profile continues to run and omits backend ID
exactly as before.
3. A built-in OpenRouter profile continues to resolve through PromptKit's
built-in backend.
4. A file profile containing `backend: local` passes `config validate` when
the Notarius local backend is configured.
5. The same profile fails before provider execution when the registration is
absent.
6. A successful local-backed call reaches the configured endpoint and records
backend ID `local` through existing response, debug, and manifest fields.
7. The Notarius scheduler remains the application-wide ceiling, and a positive
local limit may narrow concurrency for local-backed profiles.
8. Existing capacity rejection mapping and pipeline retry ownership remain
unchanged.
9. Changing the local endpoint prevents checkpoint reuse; changing only its
concurrency limit does not.
10. Configuration summaries own their nested values and contain no credential.