notarius/docs/roadmap/promptkit.md

PromptKit v0.5 Integration And LLM Profile Policy

Purpose

This roadmap defines the target state for upgrading Notarius from PromptKit v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that directly improve Notarius. It also defines the application policy for stable, domain-oriented LLM profile names, operator overrides, pipeline inheritance, profile validation, provider defaults, checkpoint identity, and documentation.

The ordered work needed to reach this state belongs in the implementation plan. Current behavior remains defined by the canonical documentation outside docs/roadmap/ until the corresponding work is implemented.

Background

Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once for debug material and then independently runs the original request, causing PromptKit to prepare the same logical call a second time. The CLI validates an explicit profile by preparing a synthetic prompt. PromptKit profile selection can be repeated on individual module bindings or replaced for one invocation with --llm-profile, but a configured pipeline cannot yet declare one inherited profile policy.

PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these areas:

• v0.4.0 adds opaque prepared executions, exact profile and prompt inspection, and a typed backend-capacity error;
• v0.5.0 adds application fallback profile filesystems and stops sending unset optional sampling controls as framework-selected provider values; and
• the v0.5.0 format contract defines the resulting profile-source and execution-setting precedence.

A source-compatibility test of the current Notarius repository against PromptKit v0.5.0 completed successfully. The work is therefore primarily an intentional runtime and configuration migration rather than a repair for a breaking Go API change.

Goals

• Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
• Execute the exact prepared request snapshot whose safe details are recorded in Notarius debug material.
• Validate configured PromptKit profiles through the upstream inspection API without synthetic prompts, provider calls, or credential-value access.
• Give Notarius an application-owned, operator-overridable dnd-extraction profile fallback.
• Let a pipeline choose one default LLM profile without repeating that ID on every LLM-backed binding.
• Apply profile inheritance and run-wide overrides only where the resolved module or validator can use an LLM.
• Preserve accurate checkpoint invalidation, effective profile provenance, redaction, cancellation, concurrency, and provider-neutral module contracts.
• Provide operators with one clear deployment pattern for production, development, and local profile definitions.

Target End State

PromptKit Runtime Boundary

Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than reimplementing source or execution resolution.

For each structured completion, the adapter:

1. builds one PromptKit run request from the provider-neutral Notarius request;
2. calls PrepareExecution once;
3. immediately arranges an idempotent Discard for every unexecuted handle;
4. obtains credential-redacted Details for debug and response metadata; and
5. calls RunPrepared so generation uses that exact frozen snapshot.

The debug prompt and successful result therefore describe the same selected profile, rendered messages, input bytes, session, output contract, and effective settings even when a filesystem-backed source changes concurrently. PromptKit handle types remain private to internal/framework/llm.

PromptKit admission failures continue to match Notarius's provider-neutral ErrLLMCapacityExceeded contract. When PromptKit supplies a CapacityError, the adapter obtains the normalized backend ID through errors.As and may add it to safe application-owned diagnostics without parsing upstream error wording. The backend ID does not become a provider-specific module contract.

Optional Provider Controls

Notarius accepts PromptKit v0.5.0's new behavior for temperature, max_tokens, and top_p: an unset setting is omitted from compatible provider requests and the provider chooses its own default. Notarius does not restore PromptKit's former implicit top_p: 1 value globally.

An operator who requires a particular value specifies it in the selected PromptKit profile. The application fallback described below intentionally leaves these controls unset. A human-reviewed D&D extraction comparison should be performed after the upgrade, but paid or nondeterministic model output is not part of the default automated test suite.

Profile Inspection

Pipeline-aware configuration validation uses Engine.InspectProfile for every effective explicit profile ID. It verifies that the profile exists, parses and validates, resolves its backend and target, and is compatible with the engine's registered backends. It does not create a synthetic prompt, load prompt inputs, contact a provider, or require credential values to exist in the validation process environment.

Credential availability is execution-time state. PromptKit preparation still enforces the selected profile's credential contract before generation. This keeps notarius config validate useful in build and deployment validation environments where secrets are deliberately absent.

PromptKit construction for inspection and execution uses one shared internal profile-source and backend-option path. The CLI does not expose PromptKit public types across the Notarius LLM boundary merely to perform inspection.

InspectPrompt is not adopted merely because it exists. It remains available for a later, separately defined module-to-prompt interface preflight if a concrete validation requirement justifies that additional contract.

Application And Operator Profile Sources

Notarius embeds one ordinary PromptKit YAML profile with the stable ID dnd-extraction. It is an application fallback registered through WithFallbackProfileFS, is owned by the D&D module family, and initially preserves the current effective D&D baseline:

• backend: PromptKit's built-in openrouter backend;
• model: openai/gpt-5.6-luna;
• reasoning effort: unset, allowing OpenAI's backend to apply its default of medium;
• generation timeout: 240 seconds;
• service tier: flex; and
• no application-selected temperature, max_tokens, or top_p.

All maintained D&D LLM prompt definitions use dnd-extraction as their default_profile. The ID communicates workload intent rather than a provider, model, or environment. Changing the embedded fallback is an intentional Notarius execution-policy change and participates in checkpoint identity.

Effective profile definitions resolve in PromptKit's order:

1. programmatic in-memory profiles used by tests or explicit consumers;
2. the operator source configured by promptkit.profile_file or promptkit.profile_dir;
3. the Notarius application fallback source; and
4. PromptKit's embedded built-in catalog.

Only an absent ID falls through to the next source. A matching profile is a complete definition: fields are not merged with a lower-precedence definition, and a malformed matching operator profile fails rather than silently selecting the application fallback.

Production, development, and local deployments should normally provide different complete definitions for the same dnd-extraction ID. An operator source is optional because the application fallback keeps the maintained D&D workflow usable, but a deployment that needs an intentional model or backend policy should configure its own definition.

Domain Ownership And Asset Assembly

The D&D fallback profile remains under internal/modules/dnd and is registered by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how to collect and flatten application fallback profile filesystems but contains no D&D model or policy knowledge.

The shared asset registry detects invalid roots, unreadable sources, and duplicate flattened paths. PromptKit remains responsible for strict profile YAML parsing, duplicate profile-ID detection, source precedence, and effective target resolution. The same assembled fallback source is supplied to runtime execution and CLI profile inspection.

Explicit Module Execution Metadata

Every registered input, chunk, extract, merge, normalize, and output module declares one required execution class: deterministic or llm_backed. Validator registrations continue to declare the same distinction through their validator specifications.

The registered specification is authoritative for configuration resolution. Current production classifications are:

• the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are LLM-backed;
• the Seriatim input adapter, generic chunker, all current mergers, all other current normalizers, and the JSON output encoder are deterministic; and
• current validators retain their declared classifications.

Missing or unsupported execution metadata is a registration error. Explicitly assigning llm_profile to a deterministic module or validator is a pipeline resolution error. The framework does not infer execution class by inspecting domain package names or concrete implementation types at runtime.

The module specification replaces the chunk runner's special runtime execution-class probe. Effective resolved bindings already express the result: only LLM-backed bindings may retain a non-empty profile.

Pipeline-Level Profile Default

Configuration version 4 gains one optional non-empty pipeline field:

pipelines:
  dnd-session:
    llm_profile: dnd-extraction

No configuration-version increment is required because the field is additive and existing files remain valid. An explicitly present blank value is invalid.

For every selected LLM-backed module and validator, the effective profile uses this precedence:

1. non-empty run-wide --llm-profile override;
2. binding-specific llm_profile;
3. pipeline-level llm_profile; and
4. the prompt definition's default_profile, represented by an empty effective Notarius binding profile.

The run-wide override and inherited pipeline default never attach to a deterministic binding. Binding-specific exceptions remain available when one operation needs a different cost, latency, quality, backend, or reasoning policy.

Inheritance is resolved after module and validator selection, including --only lane filtering, but before effective-pipeline validation, digest construction, explicit-profile inspection, checkpoint construction, preparation, execution, or provenance capture. Only profiles used by selected LLM-backed bindings are inspected. An unused pipeline default in a pipeline with no selected LLM-backed work does not require an otherwise unused profile to exist.

The resolved pipeline contains effective binding profiles rather than a second runtime inheritance mechanism. Two pipelines that differ only by spelling the same effective policy once as a pipeline default and once on every LLM-backed binding have the same semantic resolved digest. Changing an effective profile changes the digest and applicable checkpoint identity.

Provenance And Checkpoints

The PromptKit profile-source checkpoint fingerprint covers:

• the PromptKit v0.5.0 built-in profile catalog identity;
• exact application fallback profile asset content; and
• exact configured operator profile YAML content, when present.

The existing local-backend target fingerprint remains separate and continues to exclude scheduling-only concurrency limits. Fingerprints contain hashes and stable markers, not profile contents, filesystem paths, endpoints, credentials, or other secrets.

Changing the PromptKit version, application fallback, operator profile, or effective pipeline profile makes incompatible LLM checkpoints ineligible for reuse. The dependency upgrade is expected to invalidate checkpoints produced under v0.3.0.

Successful run manifests continue to record only profiles actually selected by PromptKit, including their effective model, backend, and reasoning metadata. Debug output reports the same effective execution snapshot used for generation.

Operator Documentation And Examples

Canonical documentation clearly distinguishes:

• Notarius prompt and schema assets embedded in the application;
• Notarius application fallback profiles embedded in the application;
• PromptKit's own embedded built-in profiles; and
• operator profile files on the deployment filesystem.

The configuration reference owns the pipeline field, profile-source fields, validation rules, and precedence. Operations owns deployment layout, working directory behavior, credentials, and environment-specific profile management. The PromptKit integration document owns the pinned upstream contract and source-precedence boundary. Internal documents describe asset registration, resolution, inspection, prepared execution, fingerprinting, and tests without duplicating user-facing field definitions.

The maintained examples continue to include only the minimal and complete D&D configurations. They use the stable dnd-extraction policy, and one maintained PromptKit profile file under examples/ demonstrates an operator override. Examples remain secret-free and are validated without live provider calls.

Out Of Scope

• Implementing the separate deterministic prompt-session identity roadmap item.
• Changing the default concurrency.total_llm value; PromptKit's retained OpenRouter capacity of 16 remains relevant to that separate item.
• Adding model evaluation as a deterministic or CI correctness gate.
• Automatically selecting production, development, or local environments. Deployment configuration chooses the operator profile source.
• Profile inheritance, partial profile merging, or cross-profile aliases.
• Exposing PromptKit types to modules, validators, durable output contracts, or public configuration structures.
• Adopting InspectPrompt without a separately justified prompt-interface validation contract.

Acceptance Criteria

• Notarius builds and its offline test suite passes with PromptKit v0.5.0.
• Every structured completion executes the exact snapshot used for safe debug prompt details.
• Profile preflight uses profile inspection and no synthetic prompt.
• The embedded dnd-extraction fallback resolves without an operator source, and a matching valid operator profile replaces it completely.
• Every production module has explicit, correct execution metadata.
• Pipeline, binding, CLI, and prompt-default precedence behaves as defined for modules and validators, while deterministic bindings remain profile-free.
• Effective profiles participate in pipeline digests, profile inspection, checkpoint identity, debug records, and run provenance at the appropriate boundaries.
• The dependency and application fallback changes invalidate incompatible old checkpoints without exposing profile or credential content.
• Canonical documentation and maintained examples accurately describe and exercise the implemented operator workflow.
• Default tests remain deterministic, offline, credential-free, and focused on Notarius-owned behavior rather than duplicating PromptKit's upstream suite.