notarius/docs/roadmap/implementation.md

PromptKit v0.5 Implementation Plan

Objective

Implement the target state in PromptKit v0.5 Integration And LLM Profile Policy. Each numbered stage is intended to be one implementation prompt for a GPT-5.6-Terra coding agent. Complete stages in order and leave the repository buildable, tested, and internally coherent after every stage.

Follow Architecture, Testing Policy, and Documentation Policy throughout. Preserve unrelated user changes. Use apply_patch for source and documentation edits, run gofmt on changed Go files, and add only tests that protect the behaviors and risks assigned to that stage.

Do not implement the separate deterministic session-ID or default-concurrency roadmap items as part of this plan. Do not perform paid or credentialed LLM calls.

Background Summary

Notarius currently pins PromptKit v0.3.0, calls Prepare and then Run for one completion, validates profiles through a synthetic prompt, has no application fallback profile source, and accepts LLM profiles only at individual bindings or through the run-wide CLI override. PromptKit v0.5.0 is source-compatible with the current tree; a temporary v0.5.0 module override has already passed go test ./....

The implementation must nevertheless treat the upstream optional-parameter change as intentional: unset temperature, max_tokens, and top_p remain unset and are omitted from compatible provider requests. Do not restore the old implicit top_p: 1 default.

Stage 1: Upgrade The PromptKit Dependency

Goal

Establish a clean PromptKit v0.5.0 baseline before adopting its new APIs.

Work

• Update go.mod and go.sum from PromptKit v0.3.0 to v0.5.0 and run go mod tidy.
• Change the PromptKit built-in profile-catalog marker in internal/framework/llm/promptkit_profile_fingerprint.go to identify v0.5.0. This deliberately invalidates LLM checkpoints tied to the prior catalog identity.
• Review PromptKit-facing compile errors or test failures against the v0.4.0 and v0.5.0 release guides. Do not adopt prepared execution, inspection, or fallback profiles in this stage.
• Replace the existing test assertion for one exact built-in fingerprint hash with durable assertions that the fingerprint is deterministic, non-empty, non-secret, and changes when a semantic profile source changes. Do not add a new version-constant or exact-hash change detector.
• Update docs/integrations/pkg-promptkit.md to pin and link v0.5.0 and state the implemented dependency-level behavior: unset optional sampling controls are provider defaults. Do not document later stages as implemented.
• Update any other canonical text that explicitly claims the dependency is v0.3.0, but defer descriptions of unimplemented v0.5 APIs.

Tests And Validation

• go test ./internal/framework/llm ./internal/cli
• go test ./...
• go vet ./...
• go build ./cmd/notarius
• rg -n 'promptkit v0\.3\.0|promptkit@v0\.3\.0|PromptKit v0\.3\.0' .
• git diff --check

Completion Criteria

• The repository directly pins v0.5.0 and all default offline checks pass.
• The profile-source fingerprint identifies the new upstream catalog without a brittle literal-hash test.
• Current documentation no longer identifies v0.3.0 as the supported version.

Stage 2: Execute One Frozen Prepared Snapshot

Goal

Make Notarius debug details and generation use one exact PromptKit preparation.

Work

• Refactor PromptKitClient.CompleteStructured to call PrepareExecution, immediately defer Discard, obtain a caller-owned Details value, and execute with RunPrepared.
• Preserve the existing Notarius request mapping, cancellation precedence, validation classification, raw structured bytes, response decoding, profile recording, usage reporting, and credential redaction.
• Ensure every preparation, execution, validation, empty-result, and decode error retains useful Notarius prompt context without exposing prepared handle state or secrets.
• Use errors.As to obtain *promptkit.CapacityError on admission rejection. Preserve contracts.ErrLLMCapacityExceeded as the stable classification and add a nonblank backend ID only to safe application-owned diagnostic context. Do not expose promptkit.CapacityError outside the LLM adapter.
• Update docs/internal/llm.md and the implemented-mechanics portion of docs/integrations/pkg-promptkit.md to describe the single frozen execution snapshot and structured capacity adaptation.

Tests And Validation

• Adapt existing PromptKit client tests to the prepared-execution path.
• Retain or add one behavioral test proving that the debug prompt details match the request actually passed to generation when a backing prompt source could otherwise change between independent preparations. Test the resulting snapshot consistency, not a private helper call count.
• Retain capacity tests proving errors.Is reaches contracts.ErrLLMCapacityExceeded, the selected backend can appear in safe diagnostic context, and provider calls are not made after rejected admission.
• Run go test ./internal/framework/llm and go test -race ./internal/framework/llm.
• Run go test ./... and git diff --check.

Completion Criteria

• CompleteStructured no longer calls independent Prepare and Run operations for one request.
• Debug prompt material and generation result originate from the same frozen PromptKit snapshot.
• Capacity remains a provider-neutral Notarius error classification.

Stage 3: Replace Synthetic Profile Validation With Inspection

Goal

Validate profiles through PromptKit's exact profile-inspection boundary and centralize engine profile-source construction.

Work

• Introduce a small provider-adapter-owned profile inspection or validation function in internal/framework/llm. Its public internal signature must use Notarius-owned configuration and result/error types rather than returning PromptKit types to the CLI.
• Share the code that applies profile_dir, profile_file, and registered backend options between the production PromptKit engine and the inspection engine. Preserve the mutual-exclusion and local-backend rules.
• Change CLI explicit-profile preflight to use Engine.InspectProfile through that LLM boundary.
• Remove profileCheckPromptID, profileCheckPromptFS, the testing/fstest production dependency, and the synthetic Prepare request.
• Preserve distinct, useful errors for an absent profile, invalid profile, unknown backend registration, cancellation, and invalid profile source.
• Do not require api_key_env to be populated during configuration validation. Inspection may report credential requirements internally, but actual preparation remains responsible for credential availability before a model call.
• Update current-behavior sections in docs/internal/cli.md and docs/internal/llm.md. Keep field definitions in docs/config.md.

Tests And Validation

• Replace synthetic-prompt tests with profile inspection tests covering: configured local backend success; missing local backend failure; absent profile; malformed profile; and an otherwise valid profile whose credential environment variable is intentionally unset.
• Prove validation performs no provider HTTP call and remains offline.
• Run go test ./internal/framework/llm ./internal/cli and go test ./....
• Run git diff --check.

Completion Criteria

• No production synthetic profile-check prompt remains.
• Profile validation uses the same ordinary profile source and backend registrations as execution.
• Configuration validation succeeds for structurally valid profiles without reading credential values.

Stage 4: Add Application Fallback Profile Asset Plumbing

Goal

Allow module families to register application-owned fallback profile YAML without placing domain policy in generic LLM code.

Work

• Extend internal/framework/llm.AssetRegistry with a separate fallback profile source collection, registration method, flattened filesystem, and safe content digest.
• Reuse the existing asset-source path validation and flattening behavior where appropriate. Reject invalid roots, unreadable assets, and duplicate flattened paths. Do not parse PromptKit profile YAML in Notarius.
• Add promptkit.WithFallbackProfileFS to production engine options only when at least one fallback profile source is registered.
• Supply the identical assembled fallback source to the profile-inspection engine. Adjust CLI composition so pipeline-aware profile validation can use the production LLM asset registry without exposing PromptKit types.
• Extend profile-source checkpoint identity to include the exact fallback profile asset digest in addition to the PromptKit catalog marker and operator source. Keep the resulting fingerprint hash-only and path/content/credential free.
• Keep operator source precedence owned by PromptKit. Do not implement profile merging or duplicate PromptKit source resolution in Notarius.
• Update docs/internal/llm.md only for the new implemented generic asset and fingerprint mechanics. No domain fallback exists until Stage 5.

Tests And Validation

• Add focused AssetRegistry tests for successful flattening, invalid roots, duplicate paths, and hash changes when fallback bytes change.
• Add adapter-level tests showing that the fallback filesystem reaches both execution construction and inspection construction.
• Extend checkpoint tests to prove fallback content changes profile-source identity without exposing raw YAML or paths. Use relational comparisons, not a fixed hash literal.
• Run go test ./internal/framework/llm ./internal/cli and go test ./....
• Run git diff --check.

Completion Criteria

• Generic plumbing can carry application fallback profiles while remaining unaware of D&D IDs or model settings.
• Inspection, execution, and checkpoint identity use the same fallback asset source.

Stage 5: Adopt The D&D dnd-extraction Fallback

Goal

Give the D&D module family one stable embedded workload profile that operators can replace.

Work

• Add a D&D-owned embedded PromptKit profile asset with ID dnd-extraction under internal/modules/dnd. Use the exact baseline defined in promptkit.md: OpenRouter, openai/gpt-5.6-luna, no explicit reasoning effort, a 240-second timeout, flex service tier, and no selected temperature, token limit, or top_p. The omitted reasoning value intentionally allows OpenAI's backend to apply its medium default.
• Register the profile filesystem from the D&D registrar through the generic fallback profile asset boundary. Keep D&D policy out of internal/framework/llm and the CLI composition root.
• Change every maintained D&D LLM prompt definition—including scene chunking, all D&D extractors, and NPC normalization—from the model-named default to default_profile: dnd-extraction.
• Add an integration-level profile-resolution test proving that:
  • the fallback resolves when no operator source defines the ID;
  • a valid operator profile with the same ID wins completely; and
  • an invalid matching operator profile fails rather than falling through.
• Test through Notarius's assembled production assets and PromptKit boundary; do not duplicate every upstream source-precedence case.
• Update the implemented profile ownership and prompt-default behavior in docs/internal/dnd.md, docs/internal/llm.md, and docs/integrations/pkg-promptkit.md. Defer the complete operator walkthrough and examples to Stage 10.

Tests And Validation

• Run focused D&D prompt preparation tests and the production composition tests.
• Run go test ./internal/modules/dnd/... ./internal/framework/llm ./internal/cli.
• Run go test ./....
• Verify rg -n 'default_profile: gemini-2-flash' internal/modules/dnd returns no matches.
• Run git diff --check.

Completion Criteria

• All maintained D&D prompts use the application-owned logical profile ID.
• The fallback works without an operator profile and remains authoritatively overridable by a matching valid operator definition.

Stage 6: Introduce Module Execution-Class Metadata

Goal

Make each production module's ability to use an LLM statically discoverable without yet changing profile inheritance.

Work

• Add ExecutionClass contracts.ExecutionClass to pipeline.ModuleSpec and preserve it through normalization, cloning, catalogs, registries, JSON/debug views, and lookup helpers.
• In this transitional stage only, allow an omitted execution class to normalize to deterministic so existing test-only fixtures can be migrated in Stage 7 without breaking the repository midway.
• Explicitly classify every production module:
  • D&D scene chunking, every D&D extractor, and D&D NPC normalization as llm_backed;
  • all other current production input, chunk, merge, normalize, and output modules as deterministic.
• Update production module specification tests and production catalog tests to assert the semantic class alongside stage, artifact kind, and capabilities.
• Add catalog lookup support needed by later resolution to retrieve a selected module's execution class by stage and key without constructing it.
• Do not implement pipeline-level profile inheritance or reject deterministic profiles yet.
• Update docs/internal/modules.md and docs/internal/dnd.md to identify execution class as registered module metadata, while noting only implemented uses.

Tests And Validation

• Run module registration/spec tests across generic, Seriatim, and D&D families.
• Run go test ./internal/framework/pipeline ./internal/modules/....
• Run go test ./... and git diff --check.

Completion Criteria

• Every production module has an explicit correct execution class.
• Catalog consumers can retrieve that class without a concrete module instance.
• Test-only omitted classes remain the only temporary compatibility behavior.

Stage 7: Enforce Execution Metadata And Remove Runtime Probing

Goal

Finish the execution-class contract so missing metadata cannot cause future profile drift.

Work

• Update every framework, CLI, and integration test module specification to declare an explicit execution class appropriate to the fake behavior.
• Change module-spec validation so an empty or unsupported execution class is a registration error. Remove the transitional deterministic default from Stage 6.
• Replace the chunk runner's special ChunkExecutionClassProvider probe with specification-derived behavior. Remove the now-redundant provider interface, implementation methods, and tests when they have no remaining consumer.
• Ensure chunk producer provenance remains unchanged: it records a non-empty effective binding profile for an LLM-backed chunker, while a deterministic chunker records no profile. A profile selected only through the prompt default remains represented by PromptKit's actual-profile manifest rather than being invented as an explicit chunk binding.
• Review helper constructors and fixtures for opportunities to set execution class once without obscuring the class under test. Do not introduce an elaborate test-spec framework.
• Update internal documentation if the removal changes any described runtime mechanics.

Tests And Validation

• Add or retain focused registration tests for missing and invalid execution classes.
• Retain chunk-plan provenance tests for LLM-backed and deterministic chunkers.
• Run go test ./internal/framework/pipeline ./internal/modules/....
• Run go test ./..., go vet ./..., and git diff --check.

Completion Criteria

• No registered module specification relies on an implicit execution class.
• Pipeline metadata, not a concrete runtime type assertion, owns module execution classification.

Stage 8: Resolve Programmatic Pipeline Profile Defaults

Goal

Implement profile inheritance and precedence inside the pipeline resolver before exposing the field through YAML configuration.

Work

• Add an optional trimmed LLMProfile field to pipeline.PipelineProfile. Add a non-empty runtime override field to pipeline.ResolveOptions so all precedence decisions occur in the resolver rather than through pre-resolution mutation.
• After module selection, --only filtering, default validator-chain selection, and validator compatibility resolution, apply effective profiles to every selected input, chunk, extract, merge, normalize, output, and validator binding according to the precedence in promptkit.md.
• Apply profiles only when the selected module or validator execution class is llm_backed.
• Reject a binding-specific llm_profile on any deterministic module or validator. Do not reject or inspect an unused pipeline default when no selected LLM-backed binding consumes it.
• Leave an LLM-backed binding empty when no CLI, binding, or pipeline profile is selected so PromptKit can use the prompt's default_profile.
• Store the effective values on resolved bindings before digest construction. Do not add a second inheritance decision to execution.
• Ensure semantically equivalent repeated binding profiles and one inherited default produce the same resolved pipeline digest. Ensure any changed effective profile changes the digest.
• Do not modify file configuration or CLI parsing in this stage.

Tests And Validation

• Add pipeline package tests for the complete precedence matrix: runtime override; binding-specific exception; pipeline default; prompt fallback; and deterministic bindings.
• Cover default and explicitly configured validator chains, all relevant stage categories, --only lane selection, unused defaults, deterministic-profile rejection, and semantic digest equivalence.
• Prefer table-driven package-level tests over assertions on private traversal helpers.
• Run go test ./internal/framework/pipeline and go test ./....
• Run git diff --check.

Completion Criteria

• Programmatic pipelines resolve one canonical effective profile policy.
• Only LLM-backed resolved bindings can contain a profile.
• Runtime override, binding, pipeline, and prompt precedence is unambiguous and digest-stable.

Stage 9: Expose Pipeline Defaults Through Configuration And CLI

Goal

Make the profile-default workflow available to operators while preserving validation and override behavior.

Work

• Add optional pipelines.<id>.llm_profile support to the version 4 file configuration model. Use presence-aware decoding so an explicitly set blank value is rejected, while omission remains valid.
• Preserve the field through file application, configuration cloning, effective configuration, and programmatic profile copies without aliasing or trimming drift.
• Remove applyLLMProfileOverride. Pass the CLI override through the resolver's runtime-override input so deterministic bindings are never populated.
• Update effective profile-ID collection to cover every selected LLM-backed module stage and LLM-backed validator, including future LLM-backed input and output modules. Do not inspect deterministic or unselected profiles.
• Ensure run, config validate --pipeline, resume/checkpoint identity, and relevant dry preflight paths all use the same resolved effective profiles.
• Preserve --llm-profile as the highest-precedence non-empty run-wide override and preserve binding-specific profiles as exceptions when no CLI override is present.
• Do not increment the configuration version.
• Update current configuration and CLI contracts in docs/config.md and docs/cli.md in the same stage. Link to operations for the deployment workflow rather than duplicating it prematurely.

Tests And Validation

• Add file-config tests for omission, trimming, explicit blank rejection, unknown-key behavior, cloning, and round-trip application.
• Add effective-config and CLI contract tests for precedence, LLM-only application, inherited-profile inspection failure before factory execution, --only, and digest changes.
• Retain offline operation and do not require credentials for config validate --pipeline.
• Run go test ./internal/core/config ./internal/framework/pipeline ./internal/cli.
• Run go test ./..., go vet ./..., and git diff --check.

Completion Criteria

• Operators can select dnd-extraction once per pipeline.
• Configuration and CLI paths share the resolver's precedence policy.
• Unknown effective profiles fail preflight, while deterministic and unused profiles do not cause spurious inspection.

Stage 10: Complete Operator Documentation, Examples, And Decision Record

Goal

Make the implemented workflow understandable, copyable, and maintainable without duplicating canonical facts.

Work

• Create an ADR using the next sequential number for the durable decision to use workload-oriented pipeline defaults with operator-overridable application fallback profiles. Record context, decision, alternatives, and consequences; do not turn the ADR into a field reference or implementation log.
• Complete docs/config.md as the canonical owner of profile-source fields, pipelines.<id>.llm_profile, validation, and precedence.
• Complete docs/operations.md with an operator workflow that distinguishes Notarius embedded prompts, Notarius fallback profiles, PromptKit built-ins, and deployment filesystem profiles. Include production/development/local use of the same dnd-extraction ID, credential handling, absolute-path guidance, and the fact that current relative profile paths use the process working directory rather than the configuration file's directory.
• Complete docs/integrations/pkg-promptkit.md with the v0.5.0 boundary, prepared execution, inspection, fallback and ordinary source precedence, optional provider controls, capacity adaptation, and compatibility policy.
• Update docs/internal/configuration.md, docs/internal/pipeline.md, docs/internal/cli.md, docs/internal/llm.md, docs/internal/modules.md, and docs/internal/dnd.md only for their owned implementation details. Link to canonical configuration, operations, and upstream format contracts rather than restating them.
• Keep exactly the existing two D&D configuration examples. Add llm_profile: dnd-extraction to the minimal and complete pipelines and remove the now-redundant model-named binding override from the complete example.
• Add one secret-free maintained operator profile at examples/profiles/dnd-extraction.yml. It should be a complete valid profile for the same logical ID and may mirror the embedded baseline; its purpose is to demonstrate file ownership and format, not claim automatic environment detection. Link it from the configuration and operations documentation.
• If the complete example selects the external profile file, use a path that is valid for the documented repository-root invocation and explicitly note the working-directory rule. Keep the minimal example dependent only on the embedded fallback.
• Add or extend maintained-example validation so both configuration examples and the profile YAML are checked without generation or credentials.
• Remove the now-implemented Pipeline-Level LLM Profile Defaults section from docs/roadmap/future.md. Preserve the unrelated deterministic session and concurrency items.
• Do not delete promptkit.md or this implementation plan during the feature implementation; retire them only after post-implementation review.

Tests And Validation

• Run maintained example/configuration tests and relevant CLI help/parser tests.
• Run go test ./....
• Run rg -n 'gemini-2-flash' examples docs and review every remaining match for intentional model-policy or historical context.
• Run rg -n 'v0\.3\.0|profileCheckPrompt|applyLLMProfileOverride' . and resolve stale production or current-documentation matches.
• Verify all new links and git diff --check.

Completion Criteria

• Every current fact has one canonical documentation owner.
• Operators can distinguish and deploy all profile layers without reading Go source.
• Both maintained configurations and the maintained external profile are valid, secret-free, and tested offline.
• Implemented profile work no longer remains in future.md.

Stage 11: Final Verification And Quality Review

Goal

Verify the complete migration as one integrated change and correct only defects or omissions found during that review.

Work

• Review the final diff against every acceptance criterion in promptkit.md.
• Confirm provider-specific PromptKit types remain inside the LLM integration boundary and D&D policy remains inside the D&D module family.
• Confirm execution and inspection receive identical ordinary, fallback, and backend configuration.
• Confirm no paths, profile YAML, endpoints, credentials, or prepared handle state leak into fingerprints or ordinary diagnostics.
• Confirm all production module specs have explicit correct execution classes and every resolved deterministic binding is profile-free.
• Confirm prompt default, pipeline default, binding override, and CLI override behavior through representative assembled configurations.
• Review tests for redundancy and remove obsolete synthetic-prompt, runtime-probe, exact-hash, or duplicated upstream-behavior tests superseded by stronger contract tests.
• Perform an optional manual D&D quality comparison if credentials and an evaluation transcript are deliberately supplied. Record no private input or credential material, and do not make this comparison a completion gate.

Validation Commands

gofmt -w <changed-go-files>
go test ./...
go test -race ./internal/framework/llm ./internal/core/config ./internal/framework/pipeline ./internal/cli
go vet ./...
go build ./cmd/notarius
git diff --check

Also run focused stale-contract searches:

rg -n 'gitea.maximumdirect.net/eric/promptkit v0\.3\.0|PromptKit v0\.3\.0' .
rg -n 'default_profile: gemini-2-flash|profileCheckPrompt|applyLLMProfileOverride' internal docs examples

Review any matches rather than deleting intentional historical references blindly.

Completion Criteria

• All automated checks pass offline and without real credentials.
• The implemented behavior matches promptkit.md with no known architecture, provenance, checkpoint, profile-precedence, or documentation gap.
• Any optional live evaluation is clearly separate from correctness testing.

Open Questions

None. The roadmap decisions are sufficient to implement every stage without an additional product or architecture choice.