319 lines
15 KiB
Markdown
319 lines
15 KiB
Markdown
# PromptKit v0.5 Integration And LLM Profile Policy
|
|
|
|
## Purpose
|
|
|
|
This roadmap defines the target state for upgrading Notarius from PromptKit
|
|
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
|
|
directly improve Notarius. It also defines the application policy for stable,
|
|
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
|
|
profile validation, provider defaults, checkpoint identity, and documentation.
|
|
|
|
The ordered work needed to reach this state belongs in
|
|
[the implementation plan](implementation.md). Current behavior remains defined
|
|
by the canonical documentation outside `docs/roadmap/` until the corresponding
|
|
work is implemented.
|
|
|
|
## Background
|
|
|
|
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
|
|
for debug material and then independently runs the original request, causing
|
|
PromptKit to prepare the same logical call a second time. The CLI validates an
|
|
explicit profile by preparing a synthetic prompt. PromptKit profile selection
|
|
can be repeated on individual module bindings or replaced for one invocation
|
|
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
|
|
profile policy.
|
|
|
|
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
|
|
areas:
|
|
|
|
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
|
|
adds opaque prepared executions, exact profile and prompt inspection, and a
|
|
typed backend-capacity error;
|
|
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
|
|
adds application fallback profile filesystems and stops sending unset
|
|
optional sampling controls as framework-selected provider values; and
|
|
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
|
|
defines the resulting profile-source and execution-setting precedence.
|
|
|
|
A source-compatibility test of the current Notarius repository against
|
|
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
|
|
intentional runtime and configuration migration rather than a repair for a
|
|
breaking Go API change.
|
|
|
|
## Goals
|
|
|
|
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
|
|
- Execute the exact prepared request snapshot whose safe details are recorded
|
|
in Notarius debug material.
|
|
- Validate configured PromptKit profiles through the upstream inspection API
|
|
without synthetic prompts, provider calls, or credential-value access.
|
|
- Give Notarius an application-owned, operator-overridable
|
|
`dnd-extraction` profile fallback.
|
|
- Let a pipeline choose one default LLM profile without repeating that ID on
|
|
every LLM-backed binding.
|
|
- Apply profile inheritance and run-wide overrides only where the resolved
|
|
module or validator can use an LLM.
|
|
- Preserve accurate checkpoint invalidation, effective profile provenance,
|
|
redaction, cancellation, concurrency, and provider-neutral module contracts.
|
|
- Provide operators with one clear deployment pattern for production,
|
|
development, and local profile definitions.
|
|
|
|
## Target End State
|
|
|
|
### PromptKit Runtime Boundary
|
|
|
|
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
|
|
reimplementing source or execution resolution.
|
|
|
|
For each structured completion, the adapter:
|
|
|
|
1. builds one PromptKit run request from the provider-neutral Notarius request;
|
|
2. calls `PrepareExecution` once;
|
|
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
|
|
4. obtains credential-redacted `Details` for debug and response metadata; and
|
|
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
|
|
|
|
The debug prompt and successful result therefore describe the same selected
|
|
profile, rendered messages, input bytes, session, output contract, and effective
|
|
settings even when a filesystem-backed source changes concurrently. PromptKit
|
|
handle types remain private to `internal/framework/llm`.
|
|
|
|
PromptKit admission failures continue to match Notarius's provider-neutral
|
|
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
|
|
the adapter obtains the normalized backend ID through `errors.As` and may add it
|
|
to safe application-owned diagnostics without parsing upstream error wording.
|
|
The backend ID does not become a provider-specific module contract.
|
|
|
|
### Optional Provider Controls
|
|
|
|
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
|
|
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
|
|
requests and the provider chooses its own default. Notarius does not restore
|
|
PromptKit's former implicit `top_p: 1` value globally.
|
|
|
|
An operator who requires a particular value specifies it in the selected
|
|
PromptKit profile. The application fallback described below intentionally
|
|
leaves these controls unset. A human-reviewed D&D extraction comparison should
|
|
be performed after the upgrade, but paid or nondeterministic model output is
|
|
not part of the default automated test suite.
|
|
|
|
### Profile Inspection
|
|
|
|
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
|
|
effective explicit profile ID. It verifies that the profile exists, parses and
|
|
validates, resolves its backend and target, and is compatible with the engine's
|
|
registered backends. It does not create a synthetic prompt, load prompt inputs,
|
|
contact a provider, or require credential values to exist in the validation
|
|
process environment.
|
|
|
|
Credential availability is execution-time state. PromptKit preparation still
|
|
enforces the selected profile's credential contract before generation. This
|
|
keeps `notarius config validate` useful in build and deployment validation
|
|
environments where secrets are deliberately absent.
|
|
|
|
PromptKit construction for inspection and execution uses one shared internal
|
|
profile-source and backend-option path. The CLI does not expose PromptKit public
|
|
types across the Notarius LLM boundary merely to perform inspection.
|
|
|
|
`InspectPrompt` is not adopted merely because it exists. It remains available
|
|
for a later, separately defined module-to-prompt interface preflight if a
|
|
concrete validation requirement justifies that additional contract.
|
|
|
|
### Application And Operator Profile Sources
|
|
|
|
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
|
|
`dnd-extraction`. It is an application fallback registered through
|
|
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
|
|
preserves the current effective D&D baseline:
|
|
|
|
- backend: PromptKit's built-in `openrouter` backend;
|
|
- model: `openai/gpt-5.6-luna`;
|
|
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
|
|
`medium`;
|
|
- generation timeout: 240 seconds;
|
|
- service tier: `flex`; and
|
|
- no application-selected `temperature`, `max_tokens`, or `top_p`.
|
|
|
|
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
|
|
`default_profile`. The ID communicates workload intent rather than a provider,
|
|
model, or environment. Changing the embedded fallback is an intentional
|
|
Notarius execution-policy change and participates in checkpoint identity.
|
|
|
|
Effective profile definitions resolve in PromptKit's order:
|
|
|
|
1. programmatic in-memory profiles used by tests or explicit consumers;
|
|
2. the operator source configured by `promptkit.profile_file` or
|
|
`promptkit.profile_dir`;
|
|
3. the Notarius application fallback source; and
|
|
4. PromptKit's embedded built-in catalog.
|
|
|
|
Only an absent ID falls through to the next source. A matching profile is a
|
|
complete definition: fields are not merged with a lower-precedence definition,
|
|
and a malformed matching operator profile fails rather than silently selecting
|
|
the application fallback.
|
|
|
|
Production, development, and local deployments should normally provide
|
|
different complete definitions for the same `dnd-extraction` ID. An operator
|
|
source is optional because the application fallback keeps the maintained D&D
|
|
workflow usable, but a deployment that needs an intentional model or backend
|
|
policy should configure its own definition.
|
|
|
|
### Domain Ownership And Asset Assembly
|
|
|
|
The D&D fallback profile remains under `internal/modules/dnd` and is registered
|
|
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
|
|
to collect and flatten application fallback profile filesystems but contains no
|
|
D&D model or policy knowledge.
|
|
|
|
The shared asset registry detects invalid roots, unreadable sources, and
|
|
duplicate flattened paths. PromptKit remains responsible for strict profile
|
|
YAML parsing, duplicate profile-ID detection, source precedence, and effective
|
|
target resolution. The same assembled fallback source is supplied to runtime
|
|
execution and CLI profile inspection.
|
|
|
|
### Explicit Module Execution Metadata
|
|
|
|
Every registered input, chunk, extract, merge, normalize, and output module
|
|
declares one required execution class: `deterministic` or `llm_backed`.
|
|
Validator registrations continue to declare the same distinction through their
|
|
validator specifications.
|
|
|
|
The registered specification is authoritative for configuration resolution.
|
|
Current production classifications are:
|
|
|
|
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
|
|
LLM-backed;
|
|
- the Seriatim input adapter, generic chunker, all current mergers, all other
|
|
current normalizers, and the JSON output encoder are deterministic; and
|
|
- current validators retain their declared classifications.
|
|
|
|
Missing or unsupported execution metadata is a registration error. Explicitly
|
|
assigning `llm_profile` to a deterministic module or validator is a pipeline
|
|
resolution error. The framework does not infer execution class by inspecting
|
|
domain package names or concrete implementation types at runtime.
|
|
|
|
The module specification replaces the chunk runner's special runtime
|
|
execution-class probe. Effective resolved bindings already express the result:
|
|
only LLM-backed bindings may retain a non-empty profile.
|
|
|
|
### Pipeline-Level Profile Default
|
|
|
|
Configuration version 4 gains one optional non-empty pipeline field:
|
|
|
|
```yaml
|
|
pipelines:
|
|
dnd-session:
|
|
llm_profile: dnd-extraction
|
|
```
|
|
|
|
No configuration-version increment is required because the field is additive
|
|
and existing files remain valid. An explicitly present blank value is invalid.
|
|
|
|
For every selected LLM-backed module and validator, the effective profile uses
|
|
this precedence:
|
|
|
|
1. non-empty run-wide `--llm-profile` override;
|
|
2. binding-specific `llm_profile`;
|
|
3. pipeline-level `llm_profile`; and
|
|
4. the prompt definition's `default_profile`, represented by an empty effective
|
|
Notarius binding profile.
|
|
|
|
The run-wide override and inherited pipeline default never attach to a
|
|
deterministic binding. Binding-specific exceptions remain available when one
|
|
operation needs a different cost, latency, quality, backend, or reasoning
|
|
policy.
|
|
|
|
Inheritance is resolved after module and validator selection, including
|
|
`--only` lane filtering, but before effective-pipeline validation, digest
|
|
construction, explicit-profile inspection, checkpoint construction,
|
|
preparation, execution, or provenance capture. Only profiles used by selected
|
|
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
|
|
with no selected LLM-backed work does not require an otherwise unused profile
|
|
to exist.
|
|
|
|
The resolved pipeline contains effective binding profiles rather than a second
|
|
runtime inheritance mechanism. Two pipelines that differ only by spelling the
|
|
same effective policy once as a pipeline default and once on every LLM-backed
|
|
binding have the same semantic resolved digest. Changing an effective profile
|
|
changes the digest and applicable checkpoint identity.
|
|
|
|
### Provenance And Checkpoints
|
|
|
|
The PromptKit profile-source checkpoint fingerprint covers:
|
|
|
|
- the PromptKit v0.5.0 built-in profile catalog identity;
|
|
- exact application fallback profile asset content; and
|
|
- exact configured operator profile YAML content, when present.
|
|
|
|
The existing local-backend target fingerprint remains separate and continues
|
|
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
|
|
stable markers, not profile contents, filesystem paths, endpoints, credentials,
|
|
or other secrets.
|
|
|
|
Changing the PromptKit version, application fallback, operator profile, or
|
|
effective pipeline profile makes incompatible LLM checkpoints ineligible for
|
|
reuse. The dependency upgrade is expected to invalidate checkpoints produced
|
|
under v0.3.0.
|
|
|
|
Successful run manifests continue to record only profiles actually selected by
|
|
PromptKit, including their effective model, backend, and reasoning metadata.
|
|
Debug output reports the same effective execution snapshot used for generation.
|
|
|
|
### Operator Documentation And Examples
|
|
|
|
Canonical documentation clearly distinguishes:
|
|
|
|
- Notarius prompt and schema assets embedded in the application;
|
|
- Notarius application fallback profiles embedded in the application;
|
|
- PromptKit's own embedded built-in profiles; and
|
|
- operator profile files on the deployment filesystem.
|
|
|
|
The configuration reference owns the pipeline field, profile-source fields,
|
|
validation rules, and precedence. Operations owns deployment layout, working
|
|
directory behavior, credentials, and environment-specific profile management.
|
|
The PromptKit integration document owns the pinned upstream contract and
|
|
source-precedence boundary. Internal documents describe asset registration,
|
|
resolution, inspection, prepared execution, fingerprinting, and tests without
|
|
duplicating user-facing field definitions.
|
|
|
|
The maintained examples continue to include only the minimal and complete D&D
|
|
configurations. They use the stable `dnd-extraction` policy, and one maintained
|
|
PromptKit profile file under `examples/` demonstrates an operator override.
|
|
Examples remain secret-free and are validated without live provider calls.
|
|
|
|
## Out Of Scope
|
|
|
|
- Implementing the separate deterministic prompt-session identity roadmap
|
|
item.
|
|
- Changing the default `concurrency.total_llm` value; PromptKit's retained
|
|
OpenRouter capacity of 16 remains relevant to that separate item.
|
|
- Adding model evaluation as a deterministic or CI correctness gate.
|
|
- Automatically selecting production, development, or local environments.
|
|
Deployment configuration chooses the operator profile source.
|
|
- Profile inheritance, partial profile merging, or cross-profile aliases.
|
|
- Exposing PromptKit types to modules, validators, durable output contracts, or
|
|
public configuration structures.
|
|
- Adopting `InspectPrompt` without a separately justified prompt-interface
|
|
validation contract.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
|
|
- Every structured completion executes the exact snapshot used for safe debug
|
|
prompt details.
|
|
- Profile preflight uses profile inspection and no synthetic prompt.
|
|
- The embedded `dnd-extraction` fallback resolves without an operator source,
|
|
and a matching valid operator profile replaces it completely.
|
|
- Every production module has explicit, correct execution metadata.
|
|
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
|
|
modules and validators, while deterministic bindings remain profile-free.
|
|
- Effective profiles participate in pipeline digests, profile inspection,
|
|
checkpoint identity, debug records, and run provenance at the appropriate
|
|
boundaries.
|
|
- The dependency and application fallback changes invalidate incompatible old
|
|
checkpoints without exposing profile or credential content.
|
|
- Canonical documentation and maintained examples accurately describe and
|
|
exercise the implemented operator workflow.
|
|
- Default tests remain deterministic, offline, credential-free, and focused on
|
|
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.
|