From 7cfab8ada0030fa00441abd5504e049d21d73773 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 28 Jul 2026 11:20:31 -0500 Subject: [PATCH] Add a staged implementation plan for migration to the promptkit LLM library --- docs/integrations/pkg-promptkit.md | 151 ++++++++++++ docs/roadmap/implementation.md | 373 +++++++++++++++++++++++++++++ 2 files changed, 524 insertions(+) create mode 100644 docs/integrations/pkg-promptkit.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/integrations/pkg-promptkit.md b/docs/integrations/pkg-promptkit.md new file mode 100644 index 0000000..989e887 --- /dev/null +++ b/docs/integrations/pkg-promptkit.md @@ -0,0 +1,151 @@ +# Package `promptkit` + +Import path: + +```go +import "gitea.maximumdirect.net/eric/promptkit" +``` + +Package `promptkit` is the supported Go contract for in-process prompt +preparation and execution. The declarations and their GoDoc in the +[root package](../../doc.go) own the exact API; this guide explains how the +pieces are used together. The [framework format reference](../formats.md) owns +prompt, profile, and schema file contracts. + +## Engine Construction And Sources + +Construct an engine with [`NewEngine`, `Config`, and +`Option`](../../engine.go). `PromptDir` is required unless a prompt source +option is supplied. `ProfileDir` optionally overlays built-in profiles, and an +empty `SchemaDir` uses the current directory. `Timeout` is the transport-wide +safety cap for the built-in OpenAI-compatible client. An optional `HTTPClient` +is cloned; its positive timeout takes precedence. + +Nil options are ignored. Invalid construction, including a nil injected client +or artifact reader, returns an error matching `ErrInvalidConfig`. + +The [source options](../../engine.go) replace their matching directory source: + +- `WithPromptFS` and `WithPromptFile` select prompt definitions; +- `WithProfileFS` and `WithProfileFile` overlay built-in profiles; +- `WithProfiles` adds in-memory profiles ahead of file and built-in profiles; +- `WithSchemaFS` and `WithSchemaFile` select JSON Schema documents; +- `WithLLMClient` replaces the built-in model client; and +- `WithArtifactReader` replaces the default reader for every input. + +Source selection, path resolution, strict decoding, profile overlays, and +file-to-request precedence are defined in the +[framework format reference](../formats.md). + +Per-generation timeout values from profiles or requests are independent of +the transport cap and caller context. An explicit request value of zero +disables only the per-generation deadline. The +[outbound integration contract](../integrations/openai-compatible-chat.md#timeout-and-cancellation) +defines the complete timeout layering. + +## Preparation And Execution + +[`Engine.Prepare` and `Engine.Run`](../../engine.go) accept the public +[`RunRequest`](../../types.go). `Prepare` resolves the prompt, profile, input +artifacts, validation contract, and rendered messages without calling an LLM. +`Run` performs the same preparation, calls the configured client, and validates +the generated content. The maintained +[offline preparation example](../../examples/go-library/prepare/main.go) +provides a complete runnable workflow using a prompt file, in-memory profile, +and inline input. + +[`PreparedRun` and `RunResult`](../../types.go) expose copied public values. +Preparation returns effective settings, hashes, rendered messages, selected +profile, structured-output information, and timing without resolved secrets or +model output. Execution adds the generated artifact and raw output, validation +state, model metadata, usage, run ID, and duration. + +A generated-content validation failure returns a result with +`Validation.Status == ValidationFailed`. An inability to perform validation +returns an error matching `ErrValidation`. + +## Requests, Inputs, And Overrides + +The [request and value declarations](../../types.go) own the available fields, +serialized constants, and result shapes. Use `File`, `Inline`, or +`InlineWithURI` to construct artifact references. The +[framework format reference](../formats.md) defines declared inputs, template +references, output contracts, and the relationship between file values and +request overrides. + +`ExecutionTargetOverride` uses pointers for numeric settings so an explicit +zero remains distinct from no override. `ExtraParams` accepts JSON-compatible +strings, booleans, finite numbers, string-keyed objects, arrays or slices, and +nil. Unsupported values, non-string map keys, non-finite numbers, and cycles +match `ErrInvalidConfig` in profiles or `ErrInvalidRequest` in request +overrides. + +Returned requests, profiles, prepared values, results, artifacts, maps, and +slices are isolated from internal engine state. Consumers and injected +extensions should not retain or mutate values owned by another caller. + +## Profiles And Credentials + +[`OpenAICompatibleProfile`](../../profiles.go) constructs an ordinary +in-memory profile for an OpenAI-compatible chat-completions endpoint. +`WithProfiles` rejects duplicate IDs in one call and gives in-memory profiles +precedence over explicit file sources and built-ins. + +Raw API keys do not belong in profiles. File-backed profiles may name an +environment variable, while an in-memory profile can require a request key. +A direct `RunRequest.APIKey` is request-scoped and takes precedence over an +environment lookup for the built-in client. Profile fields, ranges, built-ins, +precedence, and credential rules are owned by the +[framework format reference](../formats.md). + +API keys are excluded from JSON, prepared values, and results. The public +`String` and `GoString` methods report only whether a direct key is present. +Avoid reflection-based dumps of request structs, which can bypass that +redaction. + +## Extension Interfaces + +The [`LLMClient`, `GenerateRequest`, and +`GenerateResponse`](../../types.go) boundary lets a consumer replace model +generation. Injected clients receive copied rendered messages, effective +settings, explicit numeric-setting presence, structured-output constraints, +and the request-scoped key. They return generated content and token usage. + +The [`ArtifactReader`](../../types.go) boundary replaces the default inline and +file reader for every input. Readers provide artifact content and metadata; the +engine fills an empty artifact name from the input-map key. A reader error +matches `ErrArtifactLoad` while preserving the original identity for +`errors.Is`. A nil artifact with a nil error is also an artifact-load failure. + +Extensions should honor context cancellation and avoid logging raw prompts, +artifacts, or credentials. + +## Errors + +The [public error declarations](../../engine.go) and +[mapping](../../errors.go) preserve these sentinel checks through `errors.Is`: + +- `ErrInvalidConfig` +- `ErrInvalidRequest` +- `ErrPromptNotFound` +- `ErrProfileNotFound` +- `ErrProfileRequired` +- `ErrPromptLoad` +- `ErrProfileLoad` +- `ErrAPIKeyEnvMissing` +- `ErrArtifactLoad` +- `ErrPromptRender` +- `ErrLLMGenerate` +- `ErrValidation` + +`ErrProfileRequired` and `ErrAPIKeyEnvMissing` also match +`ErrInvalidRequest`, allowing either broad request handling or a specific +condition. Wrapped collaborator errors retain their identity where the public +contract promises it. + +## Consumer Boundary + +Promptkit is an importable library. It does not own a command, inbound HTTP +API, process configuration, or deployment policy. Scriptorium is one +downstream application that maps this root package contract into those +application concerns. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..6b831c2 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,373 @@ +# PromptKit Dependency Migration Implementation Plan + +## Status + +Planned. + +## Objective + +Replace Notarius's dependency on +`gitea.maximumdirect.net/eric/scriptorium v0.11.1` with +`gitea.maximumdirect.net/eric/promptkit v0.1.0`, and consistently adopt the +PromptKit name at the adapter, configuration, provenance, documentation, and +module-support boundaries. + +The migration is a clean break. Notarius has no production compatibility +requirement for the existing Scriptorium-named configuration or development +artifacts, so the implementation must not add legacy configuration aliases, +deprecated Go APIs, schema migrations, or compatibility shims. + +PromptKit's public engine contract is source-compatible with the subset +Notarius currently consumes. Preserve current prompt content, prompt and schema +identities, structured-completion behavior, LLM scheduling, retry behavior, +debug capture, redaction, and durable output shapes except for the intentional +configuration-version and LLM-provider provenance changes described below. + +## Governing Decisions + +- Pin PromptKit at `v0.1.0` and remove Scriptorium from `go.mod` and `go.sum`. +- Rename the top-level configuration section from `scriptorium` to + `promptkit`. +- Bump the strict file-configuration version from 3 to 4. Version 4 accepts + only the new `promptkit` key; it does not recognize `scriptorium`. +- Rename the internal adapter and its exported-within-`internal` Go API to + PromptKit terminology without transitional aliases. +- Use provider-neutral names for module-owned prompt asset helpers. Those + helpers describe Notarius assets rather than the library that loads them. +- Record `promptkit`, not `scriptorium`, as the LLM adapter/provider value in + completion responses and run manifest profile provenance. +- Do not add the PromptKit package version to checkpoint identity. A library + implementation version is not itself a semantic pipeline input. Existing + prompt, schema, module, reference, profile, runtime-override, and prepared + component fingerprints remain responsible for semantic invalidation. +- Do not adopt PromptKit's optional `ArtifactReader` extension in this work. + Notarius retains ownership of materializing and supplying prompt inputs. +- Do not add native session propagation in this work. PromptKit `v0.1.0` does + not expose the desired request-level session identifier; retain the existing + prompt-variable behavior and update the future roadmap terminology only. +- Accept PromptKit's documented timeout layering: caller context cancellation + is the outer authority, a positive generation timeout supplies an inner + request deadline, zero disables only that generation deadline, and the HTTP + client timeout remains a transport-wide cap. +- PromptKit and Notarius are both GPL-3.0 licensed, so the dependency change + requires no Notarius licensing change. + +## Non-Goals + +- Changing prompt text, message ordering, prompt IDs, schema IDs, or embedded + asset paths. +- Changing public artifact schemas, output bundle contents, run-result + receipts, checkpoint formats, or filesystem layouts. +- Adding new LLM profiles, changing profile precedence, or changing credential + handling. +- Refactoring the transport-neutral `StructuredLLMClient` contract. +- Adding PromptKit features that Notarius does not currently need. +- Preserving version 3 configuration compatibility. + +## Stage 1: Replace the Dependency and Adapter + +Update the Go dependency and the framework adapter as one compilable change. + +### Implementation + +- Replace the Scriptorium requirement with + `gitea.maximumdirect.net/eric/promptkit v0.1.0`, then run `go mod tidy` so + `go.mod` and `go.sum` contain no obsolete Scriptorium module entries. +- In `internal/framework/llm`, replace imports of + `gitea.maximumdirect.net/eric/scriptorium` with PromptKit and rename: + - `scriptorium_client.go` to `promptkit_client.go`; + - `ScriptoriumClientConfig` to `PromptKitClientConfig`; + - `ScriptoriumClient` to `PromptKitClient`; + - `NewScriptoriumClient` to `NewPromptKitClient`; + - `AssetRegistry.ScriptoriumOptions` to + `AssetRegistry.PromptKitOptions`; and + - Scriptorium-named private input, variable, metadata, debug, response, + validation, and error-redaction helpers to PromptKit terminology. +- Do not retain aliases for the old types, constructor, method, filenames, or + private helpers. +- Preserve the existing adapter sequence: + 1. validate the transport-neutral request and output target; + 2. map inputs, variables, metadata, prompt identity, profile identity, and + session prompt variable into a PromptKit `RunRequest`; + 3. call `Prepare` to capture rendered debug material; + 4. call `Run`; + 5. translate PromptKit validation failure into + `contracts.ErrInvalidStructuredOutput`; + 6. decode the structured artifact into the caller's target; and + 7. return transport-neutral response, usage, profile, and debug data. +- Continue wrapping and redacting upstream errors at the same trust boundary. + Update safe wrapper text from Scriptorium to PromptKit without exposing + prompt content, credentials, artifact content, filesystem paths, or raw + upstream payloads. +- Rename the adapter provenance constant and change its value from + `scriptorium` to `promptkit`. +- Preserve PromptKit's cancellation and timeout semantics. Do not introduce + another timeout wrapper in Notarius. +- Update the production composition root in `internal/cli/catalog.go` to + construct `PromptKitClientConfig` and `NewPromptKitClient`. + +### Tests + +- Rename and adapt the existing adapter and asset-registry tests; do not add + tests whose only purpose is to enforce private symbol or filename choices. +- Through the adapter's stable behavior, retain coverage for: + - engine and asset construction failures; + - prompt, schema, input, variable, metadata, and profile forwarding; + - structured-output success and validation failure; + - malformed or empty generated output; + - caller cancellation and configured timeout behavior; + - credential and content redaction; + - raw response and prepared-prompt debug capture; + - token usage, including cached and cache-write token fields; and + - profile manifest recording. +- Use injected PromptKit clients or local deterministic HTTP test servers. + Default tests must remain offline and must not require credentials or paid + model calls. + +### Completion Criteria + +- The framework and production CLI compile against PromptKit only. +- No Go import of the Scriptorium module remains. +- Existing adapter behavior is preserved except that response and manifest + provenance now identify `promptkit`. + +## Stage 2: Introduce Version 4 PromptKit Configuration + +Make the user-visible configuration terminology agree with the dependency and +adapter. + +### Implementation + +- Change `SupportedFileConfigVersion` from 3 to 4. +- In `internal/core/config`, rename: + - `ScriptoriumConfig` to `PromptKitConfig`; + - `FileScriptoriumConfig` to `FilePromptKitConfig`; + - `Config.Scriptorium` to `Config.PromptKit`; + - `FileConfig.Scriptorium` to `FileConfig.PromptKit`; and + - Scriptorium-named validation and application helpers to PromptKit + terminology. +- Change the runtime JSON and file YAML section name from `scriptorium` to + `promptkit`. +- Preserve the two optional fields and their meaning: + + ```yaml + version: 4 + promptkit: + profile_dir: /path/to/profiles + # profile_file: /path/to/profiles.yml + ``` + +- Preserve trimming, clone, apply, effective-configuration, and redaction + behavior for `profile_dir` and `profile_file`. +- Preserve the rule that `profile_dir` and `profile_file` are mutually + exclusive and that an explicitly supplied value must not be empty. +- Update production client construction and explicit-profile validation to + read `cfg.PromptKit`. +- Rename `internal/cli/scriptorium_profiles.go` and its functions to PromptKit + terminology. Preserve the existing profile validation timing and + `errors.Is`-based handling of PromptKit's `ErrProfileNotFound`. +- Keep YAML decoding strict. A version 4 file containing `scriptorium` must + fail as an unknown field. +- Add a targeted version 3 migration diagnostic that instructs users to: + 1. change `version: 3` to `version: 4`; and + 2. rename `scriptorium:` to `promptkit:`. + Do not attempt to decode or automatically rewrite version 3 files. +- Update maintained configuration examples to version 4 and use `promptkit` + wherever profile sources are demonstrated. + +### Tests + +- Update configuration contract tests to establish: + - a minimal version 4 file applies over defaults; + - explicit PromptKit profile directory and profile file values decode and + survive apply, clone, and effective configuration; + - the two profile sources remain mutually exclusive; + - explicit empty values remain invalid; + - unknown fields remain rejected by strict decoding; + - version 4 rejects the removed `scriptorium` key; and + - version 3 produces the actionable migration classification. +- Update CLI contract tests to demonstrate that configured and explicitly + selected PromptKit profiles are validated before pipeline preparation and + that invalid profiles retain the existing process-failure behavior. +- Test configuration behavior at the parser/configuration and CLI composition + boundaries. Do not reproduce PromptKit's own profile parser test matrix. + +### Completion Criteria + +- All runtime and file configuration code uses PromptKit terminology. +- Maintained examples are valid version 4 files. +- The obsolete section is rejected rather than silently accepted or ignored. + +## Stage 3: Make Module Prompt-Asset Support Provider-Neutral + +Remove dependency-brand terminology from module-owned prompt asset +registration without changing the assets themselves. + +### Implementation + +- Across the D&D scene chunker, extractors, NPC normalizer, and registration + support, rename each `scriptorium_assets.go` and corresponding test file to + `prompt_assets.go` and `prompt_assets_test.go`. +- Rename private helpers and values such as `scriptoriumPromptRoot`, + `scriptoriumPromptMetadata`, and equivalent schema registration names to + provider-neutral forms such as `promptAssetRoot` and + `promptAssetMetadata`. +- Retain PromptKit terminology only where code directly calls a PromptKit API, + such as producing `promptkit.Option` values at the framework asset registry + boundary. +- Do not change: + - prompt or schema contents; + - prompt, schema, or asset IDs and versions; + - embedded filesystem paths; + - message ordering or cache-control placement; + - module keys or declared prompt inputs; + - metadata values or content fingerprint algorithms; or + - extraction, normalization, or validation behavior. + +### Tests + +- Update existing module prompt-preparation and registration tests to compile + through the renamed support code. +- Retain the centralized behavioral coverage that all registered production + prompts and schemas can be mounted and prepared with their declared inputs. +- Retain prompt ordering and cache-prefix contract coverage where ordering is + semantically significant. +- Do not add word-presence tests, rename detectors, or private-helper tests. + +### Completion Criteria + +- Module-owned code no longer describes its prompt assets as Scriptorium + assets. +- Prompt and schema fingerprints remain unchanged from the pre-migration + source content. + +## Stage 4: Align Provenance and Checkpoint Behavior + +Make the intentional public provenance change explicit while leaving cache +identity tied to semantic inputs. + +### Implementation + +- Record `Provider: "promptkit"` in every newly observed + `LLMProfileManifest` and `StructuredCompletionResponse` produced by the + production adapter. +- Update manifest, debug, artifact, and assembled-run expectations that + currently identify Scriptorium. +- Do not change checkpoint schemas, layouts, or the + `CheckpointFingerprintProvider` contract. +- Do not introduce a fingerprint containing the PromptKit package name or + version. Dependency implementation identity is not a stable semantic + identity. +- Verify that no accidental prompt, schema, component, reference, profile, or + runtime-override fingerprint changes result from the provider-neutral file + and helper renames. +- If an existing test fixture contains recorded LLM provenance, update only + the adapter/provider value and preserve the profile ID, model, usage, and + remaining run data. + +### Tests + +- At the adapter boundary, assert that a successful completion and recorded + profile manifest report `promptkit`. +- At one assembled run boundary, confirm that the finalized manifest carries + the PromptKit profile provenance observed by the client. +- Retain existing checkpoint identity tests for semantic prompt, module, + profile, reference, and runtime changes. Do not add a test coupled only to + the absence of a dependency-version fingerprint. + +### Completion Criteria + +- New durable and debug provenance consistently identifies PromptKit. +- No checkpoint wire-format or layout change has been introduced. + +## Stage 5: Update Canonical Documentation + +Update documentation in the same change that implements the behavior, following +the repository's canonical ownership rules. + +### Implementation + +- Update `docs/config.md` to own: + - configuration version 4; + - the `promptkit` section and its fields; + - mutual exclusion and validation rules; and + - the version 3-to-4 migration instruction. +- Update `docs/operations.md` to describe provider retries and timeouts as + PromptKit profile behavior and accurately summarize the timeout layers. +- Update `docs/internal/llm.md` to describe: + - the PromptKit-backed adapter; + - transport-neutral request and response mapping; + - asset mounting; + - preparation, execution, validation, redaction, and debug behavior; + - profile provenance; and + - caller, generation, and transport timeout ownership. +- Update `docs/internal/cli.md` to describe PromptKit profile validation and + production client composition. +- Update `docs/development.md` so its task-reading guide refers to the + PromptKit integration. +- Update any maintained examples and nearby navigation links. Configuration + definitions and defaults remain canonical in `docs/config.md`; other + documents should summarize and link rather than repeat them. +- Replace the copied `docs/integrations/pkg-promptkit.md` guide with a concise + Notarius-owned upstream boundary document. It must: + - identify the pinned PromptKit package and supported public boundary used + by Notarius; + - link to PromptKit's canonical upstream package and format documentation; + - describe only integration facts that Notarius relies upon; + - point to `docs/internal/llm.md` for Notarius implementation behavior; and + - contain no copied relative links that resolve only inside the PromptKit + source repository. +- Update `docs/roadmap/future.md` to replace Scriptorium terminology in the + native-session item with PromptKit. Keep the work deferred and state + accurately that PromptKit `v0.1.0` does not yet provide the desired direct + request-level session field. Do not imply that existing prompt-variable + propagation is native provider session support. +- Search all maintained Go, Markdown, YAML, JSON, and module files for + remaining `Scriptorium` or `scriptorium` occurrences. Retain the old name + only where necessary to explain the one-time version 3 migration or + historical context. +- Validate affected relative documentation links. + +### Completion Criteria + +- Current-behavior documentation outside `docs/roadmap` describes the + implemented PromptKit integration only. +- Configuration and operational facts have one canonical owner. +- The integration document is Notarius-specific and does not duplicate or + impersonate upstream package documentation. + +## Stage 6: Repository-Wide Verification + +Complete the migration with focused and repository-wide validation. + +### Required Checks + +Run: + +```sh +git diff --check +go mod tidy +go test ./... +go vet ./... +go build ./cmd/notarius +go test -race ./internal/framework/llm ./internal/cli ./internal/modules/dnd/... +``` + +Also verify: + +- `go.mod` and `go.sum` contain PromptKit and no Scriptorium dependency; +- the repository contains no obsolete Scriptorium Go identifiers or imports; +- any remaining textual use of Scriptorium is limited to intentional migration + or historical explanation; +- maintained configuration examples parse successfully as version 4; +- tests remain deterministic, offline, and independent of real credentials; +- no prompt, schema, artifact, or checkpoint format changed unintentionally; + and +- no secrets, prompt payloads, source content, or private infrastructure paths + were introduced into code, errors, tests, or documentation. + +## Open Questions + +None. The dependency version, compatibility policy, configuration migration, +internal naming, provenance value, checkpoint treatment, session scope, +documentation ownership, and verification boundaries are decided above.