notarius/docs/roadmap/implementation.md

PromptKit v0.3.0 Upgrade And Local Backend Adoption

Status

Planned.

This roadmap is an implementation specification for upgrading Notarius from gitea.maximumdirect.net/eric/promptkit v0.2.0 to v0.3.0 and adopting the new conventional local-backend helper. Implement the stages in order. Each stage must leave its focused tests passing before the next stage begins.

The upstream release is additive. Existing endpoint-only profiles, custom engine options used by tests, and the built-in OpenRouter backend remain supported. The implementation must preserve those paths.

Outcome

After all stages:

• Notarius pins PromptKit v0.3.0.
• Version 4 Notarius configuration may register one optional conventional PromptKit local backend.
• A PromptKit-owned file profile may select that registration with backend: local.
• The registration supplies one shared endpoint and an optional PromptKit concurrency limit to every profile that selects it.
• CLI profile preflight and the production LLM client construct equivalent PromptKit backend registrations.
• Run manifests and debug material continue to expose PromptKit's selected backend ID through the existing provenance paths.
• Checkpoint identity changes when the pinned PromptKit release or configured local endpoint changes, but not when only the local concurrency limit changes.
• Existing configurations and endpoint-only profiles behave as before.

Upstream Contracts

Use the v0.3.0 tag, rather than the moving main branch, as the implementation contract:

• release guide: https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/releases/v0.3.0.md
• Go consumer guide: https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/consumers/pkg-promptkit.md
• profile format: https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/docs/formats.md
• backend API and GoDoc: https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.3.0/backends.go

The new API used by this work is:

promptkit.WithBackend(
	promptkit.LocalBackend(endpoint, concurrencyLimit),
)

LocalBackend creates an ordinary engine-scoped backend whose ID is promptkit.BackendLocal, currently the case-sensitive string local. It does not register itself, read environment variables, select a model, or add credentials. A positive concurrency limit uses PromptKit's default waiting capacity; zero leaves the backend unlimited inside PromptKit.

Decisions

Public configuration

Keep SupportedFileConfigVersion at 4. The new field is optional and additive:

version: 4

promptkit:
  profile_dir: ./profiles
  # profile_file: ./profiles.yml
  local_backend:
    endpoint: http://localhost:8000/v1
    concurrency_limit: 2

The selected PromptKit profile remains in the external PromptKit profile source:

id: local-summary
backend: local
model: example-model

promptkit.local_backend has these semantics:

• The entire object is optional.
• When the object is present, endpoint is required.
• File loading trims endpoint; an explicitly blank value is invalid.
• The effective endpoint must be an absolute HTTP or HTTPS URL with a host. User information, query strings, and fragments are invalid. URL paths are allowed.
• concurrency_limit is optional and defaults to zero.
• A negative concurrency limit is invalid.
• Zero means no PromptKit-local concurrency restriction. The existing application-wide concurrency.total_llm scheduler remains in force.
• A positive value limits simultaneous generations for profiles selecting backend: local. PromptKit owns its default queue capacity and admission behavior.
• The local backend may coexist with either profile_dir, profile_file, or built-in profiles. The existing mutual exclusion between profile_dir and profile_file remains unchanged.
• No environment override is added for either local-backend field.
• The object contains no credential. A profile may continue to name its own api_key_env according to the PromptKit profile format.

Use pointer presence for the nested object in both the file model and effective configuration. This distinguishes an absent registration from a configured registration whose concurrency limit is zero.

Add these effective configuration values in internal/core/config/config.go:

type PromptKitConfig struct {
	ProfileDir   string                       `json:"profile_dir,omitempty"`
	ProfileFile  string                       `json:"profile_file,omitempty"`
	LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
}

type PromptKitLocalBackendConfig struct {
	Endpoint         string `json:"endpoint"`
	ConcurrencyLimit int    `json:"concurrency_limit"`
}

Add a corresponding YAML file model in internal/core/config/file_config.go. Use pointer fields inside the file model where necessary to distinguish omission from an explicitly empty value:

type FilePromptKitConfig struct {
	ProfileDir   *string                              `yaml:"profile_dir,omitempty"`
	ProfileFile  *string                              `yaml:"profile_file,omitempty"`
	LocalBackend *FilePromptKitLocalBackendConfig     `yaml:"local_backend,omitempty"`
}

type FilePromptKitLocalBackendConfig struct {
	Endpoint         *string `yaml:"endpoint,omitempty"`
	ConcurrencyLimit *int    `yaml:"concurrency_limit,omitempty"`
}

The existing YAML decoder's KnownFields(true) behavior must reject unknown fields within local_backend.

Ownership and redaction

internal/core/config owns parsing, normalization, structural validation, copying, and the user-facing configuration contract. It must not import PromptKit merely to validate the URL.

The nested effective configuration pointer must be deep-copied by cloneConfig. Config.Redacted and effective-config summaries already begin from that clone and should preserve the endpoint and concurrency limit. Treat the endpoint as non-secret configuration metadata, like the existing profile source path. Do not add any credential value to this object.

The framework LLM adapter owns conversion to promptkit.LocalBackend and PromptKit engine construction. The CLI composition root maps the core configuration type to the adapter type; the framework package must not import internal/core/config.

Adapter representation and shared option construction

Add an adapter-owned value in internal/framework/llm:

type PromptKitLocalBackendConfig struct {
	Endpoint         string
	ConcurrencyLimit int
}

Add an optional LocalBackend *PromptKitLocalBackendConfig field to PromptKitClientConfig. NewPromptKitClient must:

• defensively copy the pointed-to value;
• add a promptkit.WithBackend(promptkit.LocalBackend(...)) option when it is present;
• add no backend option when it is absent; and
• retain the trimmed endpoint needed for checkpoint fingerprinting.

Provide this small exported helper in internal/framework/llm:

func PromptKitLocalBackendOption(cfg PromptKitLocalBackendConfig) promptkit.Option

It converts the adapter local-backend value directly to promptkit.WithBackend(promptkit.LocalBackend(...)). Both NewPromptKitClient and the CLI's preparation-only profile validation engine must call this helper. The helper must be a direct conversion with no defaults or hidden environment behavior. Its purpose is to prevent preflight and runtime registration from drifting.

The production CLI construction in internal/cli/catalog.go maps config.PromptKit.LocalBackend into the adapter-owned value. The profile validation path in internal/cli/promptkit_profiles.go performs the same mapping and appends the shared option to its preparation-only engine.

Do not remove or reinterpret PromptKitClientConfig.EngineOptions. It remains the test and internal extension seam, and those options continue to be appended after asset and profile-source options. Ensure the local backend registration is added exactly once.

Scheduling and capacity

Do not change the existing Notarius scheduled client. It remains the outer, application-wide FIFO limit. PromptKit's local backend limit remains an inner per-engine, per-backend limit:

Notarius total LLM scheduler
    -> PromptKit local backend admission
        -> local OpenAI-compatible endpoint

When the local limit is positive, the maximum active calls against the local endpoint is effectively the smaller of concurrency.total_llm and promptkit.local_backend.concurrency_limit. When the local limit is zero, only the Notarius scheduler limits calls.

Do not add another retry or translate errors in a new location. PromptKitClient.CompleteStructured already maps promptkit.ErrCapacityExceeded to contracts.ErrLLMCapacityExceeded, and the pipeline remains responsible for retry decisions.

Checkpoint identity

Keep the existing promptkit_profile_source fingerprint and change its compiled-in release marker from:

promptkit:v0.2.0:builtin-profiles

to:

promptkit:v0.3.0:builtin-profiles

With the current length-prefixed hashing algorithm, the no-external-profile fingerprint becomes:

sha256:5218b1dec48f5fdd46836826e0b25906c33efbdf943c5f18085b8d82467e0276

Add a second fingerprint only when a local backend is configured:

• name: promptkit_local_backend_target
• value: a SHA-256 digest over two length-prefixed parts:
  1. the stable marker notarius:promptkit-local-backend:v1;
  2. the trimmed endpoint.

Reuse the existing length-prefix helper so concatenated inputs are unambiguous. Do not put the raw endpoint in checkpoint identity.

Do not include concurrency_limit in this fingerprint. The endpoint selects the semantic execution target; the concurrency limit changes scheduling and admission only. Return fingerprints in deterministic order: profile source first, optional local backend target second.

No manifest schema change is needed. Successful calls already record PromptKit's selected backend ID, so profiles using the new registration will produce the existing optional backend_id: "local" provenance. Do not add the endpoint to run manifests.

Scope exclusions

Do not add:

• arbitrary user-defined backend IDs;
• more than one configured local endpoint;
• backend-wide API-key fields;
• backend-wide extra request parameters;
• configurable PromptKit queue capacity;
• automatic local endpoint discovery;
• environment-variable overrides for local-backend configuration;
• an implicit local backend when the object is absent;
• a default model or an in-memory local profile;
• CLI flags for local backend settings;
• changes to existing maintained example configurations;
• a configuration version bump;
• a new ADR.

Those capabilities require the full PromptKit Backend interface or a broader product decision and are outside this release adoption.

Stage 1: Pin PromptKit v0.3.0

Changes

1. Run:

   go get gitea.maximumdirect.net/eric/promptkit@v0.3.0
   go mod tidy
   

2. Confirm go.mod directly requires exactly v0.3.0 and go.sum contains the v0.3.0 module and Go module checksums with no v0.2.0 PromptKit entries.

3. Update promptKitBuiltinProfileCatalogID in internal/framework/llm/promptkit_profile_fingerprint.go.

4. Update the exact built-in fingerprint expectation in internal/framework/llm/promptkit_client_test.go to the value specified above.

5. Do not use the new local-backend API in this stage.

Verification

Run:

go test ./internal/framework/llm
go test ./...
go vet ./...
go build ./cmd/notarius

Completion criteria

• The repository builds against PromptKit v0.3.0 without compatibility shims.
• All pre-existing behavior passes unchanged.
• Checkpoints created under the v0.2.0 release identity are not considered identical to v0.3.0 checkpoints.

Stage 2: Add The Version 4 Configuration Contract

Changes

1. Add the effective and file-model types described in the decisions above.
2. In Config.ApplyFileConfig:
   • recognize an absent object without changing defaults;
   • require a present object to contain a nonblank endpoint;
   • trim the endpoint before storing it;
   • default an omitted concurrency limit to zero;
   • allocate a new effective nested value so the parsed file model cannot alias the resulting Config.
3. Extend validatePromptKit:
   • preserve profile directory/file mutual exclusion;
   • reject an effective local backend with a blank endpoint;
   • parse the trimmed endpoint with net/url;
   • accept http or https case-insensitively;
   • require URL.IsAbs() and a nonempty URL.Hostname();
   • reject URL user information;
   • reject both a nonempty URL.RawQuery and URL.ForceQuery;
   • reject any endpoint text containing #, including an empty trailing fragment;
   • allow URL paths;
   • reject a negative concurrency limit.
4. Extend cloneConfig to deep-copy PromptKit.LocalBackend.
5. Make no environment-override changes.

Use contextual errors rooted at the public field names, for example:

promptkit.local_backend.endpoint must not be empty when set
promptkit.local_backend.endpoint must be an absolute HTTP or HTTPS URL with a host and no user information, query, or fragment
promptkit.local_backend.concurrency_limit must not be negative

Exact wrapping may follow existing configuration conventions, but tests should assert stable field context rather than whole error strings.

Tests

Extend the current configuration contract suites rather than creating a parallel test framework:

• internal/core/config/file_config_contract_test.go
  • decodes the documented object;
  • trims the endpoint;
  • defaults omitted concurrency to zero;
  • preserves an explicit positive concurrency value;
  • rejects a missing or explicitly blank endpoint;
  • rejects an unknown nested field;
  • confirms cloning and Redacted preserve values without pointer aliasing;
  • confirms runtime JSON uses local_backend, endpoint, and concurrency_limit.
• internal/core/config/validation_contract_test.go
  • accepts representative HTTP and HTTPS endpoints, including a path;
  • rejects relative URLs, unsupported schemes, missing hosts, user information, queries, fragments, and negative concurrency;
  • confirms the object may coexist with either valid profile-source choice.
• internal/core/config/effective_config_contract_test.go
  • confirms resolution preserves an independently owned local-backend value;
  • mutating either the input or resolved copy must not mutate the other.

Avoid tests for net/url itself. Cover only the public categories in the configuration contract.

Verification

Run:

go test ./internal/core/config
go test ./...

Completion criteria

• Old version 4 files load exactly as before.
• The new object is strict, validated, normalized, independently owned, and visible through the existing safe configuration summaries.
• No runtime behavior uses the new values yet.

Stage 3: Register The Local Backend In Preflight And Runtime

Changes

1. Add the adapter-owned configuration value and shared PromptKit option helper described above.
2. Extend PromptKitClientConfig and NewPromptKitClient to register the configured backend before calling promptkit.NewEngine.
3. In internal/cli/catalog.go, map the core configuration into PromptKitClientConfig.LocalBackend.
4. In internal/cli/promptkit_profiles.go, add the identical registration to the preparation-only engine.
5. Preserve option ordering and existing profile-source behavior:
   • mounted prompt/schema assets;
   • optional profile file;
   • optional local backend registration;
   • caller-supplied EngineOptions.
6. Do not change request mapping, response decoding, selected-backend recording, scheduler construction, or capacity-error translation.

The preflight engine must prepare a file-backed profile selecting backend: local without making a provider call. If the profile selects local but promptkit.local_backend is absent, preserve PromptKit's profile-load failure with the existing Notarius profile-validation context; do not silently fall back to an endpoint-only or OpenRouter profile.

Tests

Add or extend focused behavioral tests:

• In internal/framework/llm/promptkit_client_test.go, configure a profile selecting backend: local and prove that:
  • client construction succeeds when the registration is supplied;
  • a structured call reaches an httptest.Server at the configured endpoint;
  • the returned response and recorded manifest use backend ID local;
  • omitting the registration causes preparation/execution to fail before any provider request.
• Add internal/cli/promptkit_profiles_test.go if no existing CLI test gives a narrow home for preflight behavior:
  • configured local registration permits explicit-profile validation;
  • absent registration rejects the same profile with profile and backend context;
  • validation remains preparation-only and does not contact the endpoint.
• Extend an existing production composition test in internal/cli/production_contract_test.go to ensure the effective configuration reaches the production client path. Exercise the constructed client far enough to select the local-backed profile; construction alone is insufficient because backend membership is resolved during preparation.

Use temporary profile files and httptest.Server. Tests must remain offline, deterministic, and credential-free.

Do not duplicate PromptKit's constructor-shape tests or comprehensively retest its backend queue. Notarius owns correct configuration handoff, equivalent preflight/runtime construction, existing provenance, and error-boundary behavior.

Verification

Run:

go test ./internal/framework/llm ./internal/cli
go test ./...

Completion criteria

• A configured backend: local profile passes CLI preflight and uses the configured endpoint at runtime.
• The same profile fails clearly when the backend is not registered.
• Endpoint-only profiles and built-in profiles continue to work.
• Existing manifest/debug backend provenance records local without schema changes.
• Application-wide and PromptKit-local scheduling remain layered as designed.

Stage 4: Add Local Backend Checkpoint Identity

Changes

1. Store a defensive copy of the configured local endpoint on PromptKitClient.
2. Add the optional promptkit_local_backend_target fingerprint using the exact algorithm and marker specified above.
3. Extend LLMCheckpointFingerprints to return:
   • the existing profile-source fingerprint in all cases;
   • the local-target fingerprint second when configured.
4. Preserve defensive-copy behavior on the returned fingerprint slice.
5. Do not add concurrency, queue behavior, raw endpoint text, or credentials to checkpoint identity.

Keep this logic with the existing PromptKit profile fingerprint implementation or in an adjacent focused file under internal/framework/llm; do not move checkpoint policy into the CLI or core configuration package.

Tests

Extend internal/framework/llm/promptkit_client_test.go to prove:

• no local backend returns only the profile-source fingerprint;
• a configured local backend returns exactly two fingerprints in the required order;
• changing only the endpoint changes the local-target fingerprint;
• changing only concurrency_limit leaves every fingerprint unchanged;
• neither fingerprint exposes the raw endpoint;
• mutating a returned slice does not affect a later result.

The existing scheduled-client fingerprint delegation test is sufficient unless it assumes exactly one fingerprint. If it does, update it to verify transparent delegation of multiple values without specializing it to PromptKit.

Verification

Run:

go test ./internal/framework/llm ./internal/cli
go test ./...

Completion criteria

• Checkpoint reuse is invalidated by a changed PromptKit release, profile source, or local endpoint.
• A concurrency-only operational change does not invalidate semantic checkpoints.
• Fingerprints remain deterministic, non-secret, and available through the scheduled-client wrapper.

Stage 5: Publish Current Behavior And Complete Validation

Documentation outside docs/roadmap must describe only the implemented result, so perform these updates after Stages 1 through 4 work.

Documentation changes

1. docs/integrations/pkg-promptkit.md
   • change the pin and upstream links from v0.2.0 to v0.3.0;
   • add BackendLocal, LocalBackend, and WithBackend to the supported boundary;
   • replace the statement that production exposes no user-defined backend registration with the narrower truth: Notarius exposes one optional conventional local registration and no general backend registry;
   • retain endpoint-only profile compatibility and existing ownership boundaries.
2. docs/config.md
   • add the exact YAML example from this roadmap;
   • document the nested fields and validation rules;
   • explain that a PromptKit profile selects it with backend: local;
   • link profile-format details to the pinned PromptKit reference;
   • state that omission leaves current behavior unchanged.
3. docs/internal/configuration.md
   • mention the independently owned nested PromptKit backend configuration and its validation boundary.
4. docs/internal/llm.md
   • document equivalent registration in preflight and production engines;
   • describe the two scheduling layers;
   • describe the optional endpoint fingerprint and why concurrency is excluded;
   • retain the existing capacity-error and provenance ownership.
5. docs/operations.md
   • explain the effective local concurrency relationship;
   • state that a zero local limit leaves only the application scheduler;
   • state that PromptKit owns the positive-limit waiting capacity and emits the already-mapped capacity failure when admission is exhausted.

Do not add a complete local configuration under examples/. Existing examples are maintained provider-neutral workflows and should not gain a dependency on a local inference server. The concise docs/config.md fragments are the canonical user guidance for this optional integration.

Review README.md, docs/cli.md, the JSON output contract, and architecture policy for contradictions. Do not edit them unless the implemented behavior made a statement inaccurate. No CLI syntax, output schema, or architectural boundary changes are intended.

Full verification

Run the upstream-requested ordinary and race-enabled tests plus the repository checks:

go test ./...
go test -race ./...
go vet ./...
go build ./cmd/notarius

Also inspect the final diff and confirm:

• only intended module checksums changed;
• all upstream documentation links use the v0.3.0 tag;
• no raw credential or environment value was added;
• no endpoint was added to run-manifest output;
• old endpoint-only profile fixtures remain present;
• current-behavior documentation no longer says Notarius cannot register any production backend;
• this roadmap's completed work is reflected in canonical documentation.

Completion criteria

• All focused, repository-wide, race, vet, and build checks pass.
• The public configuration, internal behavior, integration boundary, and operational guidance agree.
• The upgrade remains backward compatible for existing Notarius configurations and PromptKit profiles.
• The only newly supported backend configuration is the explicit conventional local registration defined by this roadmap.

Final Acceptance Scenarios

An implementation is complete only when all scenarios below hold:

1. A current version 4 configuration with no local_backend loads and runs without behavioral changes.
2. An endpoint-only PromptKit profile continues to run and omits backend ID exactly as before.
3. A built-in OpenRouter profile continues to resolve through PromptKit's built-in backend.
4. A file profile containing backend: local passes config validate when the Notarius local backend is configured.
5. The same profile fails before provider execution when the registration is absent.
6. A successful local-backed call reaches the configured endpoint and records backend ID local through existing response, debug, and manifest fields.
7. The Notarius scheduler remains the application-wide ceiling, and a positive local limit may narrow concurrency for local-backed profiles.
8. Existing capacity rejection mapping and pipeline retry ownership remain unchanged.
9. Changing the local endpoint prevents checkpoint reuse; changing only its concurrency limit does not.
10. Configuration summaries own their nested values and contain no credential.