From e01b8d1b6d0f86805435260011bebeabc9e1d520 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 21 Jul 2026 13:59:28 +0000 Subject: [PATCH] Use shared assets instead of prompt prefix change detectors --- docs/roadmap/dnd.md | 38 +++++++-------- docs/roadmap/implementation.md | 85 +++++++++++++--------------------- 2 files changed, 49 insertions(+), 74 deletions(-) diff --git a/docs/roadmap/dnd.md b/docs/roadmap/dnd.md index 62c93a1..af80f01 100644 --- a/docs/roadmap/dnd.md +++ b/docs/roadmap/dnd.md @@ -56,34 +56,28 @@ module-specific message: wording differs by lane. Factor the common bytes into one message and leave the differences in later messages instead. -Tests inspect the fully rendered Scriptorium request boundary. Source-level -message identity alone is not treated as proof of cache identity. +Shared messages are embedded once and referenced by every applicable prompt. +This makes reuse of the canonical asset the easiest way to preserve cache +identity and avoids parallel module-specific copies that can drift. -### Exact Prompt And Input Identity Is Protected +### Shared Prompt Ownership And Input Behavior Are Protected -Focused tests at the narrowest stable boundary expose the fully prepared -provider-neutral request. +Focused tests protect shared asset composition and input behavior without +snapshotting the length or exact boundary of the common message prefix. -- Render each extractor prompt with the same transcript and campaign - references and assert that the intended common message prefix has identical - roles, content bytes, ordering, and cache-control metadata. -- Assert that spells and combat turns render identical NPC-registry messages - for the same bound or unbound registry. +- Verify that each applicable prompt mounts and renders the canonical shared + assets rather than module-specific copies. - Verify common input material identity, including name, media type, content, digest, origin URI, size, empty-value representation, reference ordering, and `roster` fallback behavior. -- Test both identical and intentionally different chunks and reference sets so - the test proves the cache boundary rather than merely snapshotting one - request. -- Add an explicit assertion for the length of the common prefix. A new - module-specific message inserted inside that prefix should require deliberate - review. - Test that every prompt fingerprint includes exactly the assets actually rendered by that prompt. The current grouped reference hash helper should be replaced or refined so a module does not fingerprint an unused shared asset, while no used asset is omitted. -- Prefer semantic assertions over complete prompt snapshots, except for the - common rendered prefix whose exact bytes are the behavior under protection. +- Document the expected shared message ordering and cache-control policy so + later changes are reviewed against the caching intent. +- Prefer behavioral and ownership assertions over message-count, prefix-length, + or full-prompt snapshots. ### Stable Extraction Preparation Is Centralized @@ -175,10 +169,10 @@ This roadmap is complete when: - the three lanes expose the same stable structural conventions while keeping documented domain differences local; -- shared provider-visible messages and inputs are produced from one source and - verified byte-for-byte at the rendered-request boundary; -- the common prompt prefix is deliberate, tested, and as long as extraction - quality permits; +- shared provider-visible messages are produced from canonical shared assets, + and shared inputs have one deterministic implementation; +- the common prompt ordering is deliberate, documented, and as cache-friendly + as extraction quality permits; - every validator policy that affects reusable results participates in checkpoint identity; - source-reference traversal and diagnostics no longer drift between lanes; diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 9dc1e67..8d245c4 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -33,17 +33,17 @@ The following decisions are complete and are not implementation-time choices: registration, or erase typed artifact relationships outside existing framework boundaries. - Exact cache identity means equal ordered message roles, content bytes, and - cache-control values after Scriptorium has rendered the prompt. Tests must - use `scriptorium.PreparedRun.Messages`; comparing source Markdown is - insufficient. + cache-control values after Scriptorium has rendered the prompt. Enforce this + primarily through one canonical shared asset per shared message and document + the expected order and cache-control policy. - Scriptorium and provider-specific types remain confined to the LLM runtime, prompt-asset wiring, and their tests. Production extractors continue to use only Notarius contracts. - Default tests remain offline, deterministic, and credential-free. Live model evaluation is an explicit manual acceptance activity, not part of `go test ./...`. -- Avoid broad snapshots. Exact byte assertions are warranted only for the - rendered common message prefixes because byte identity is the feature. +- Do not add message-count, common-prefix-length, or complete rendered-prompt + snapshots. They are change detectors rather than durable behavioral tests. - Update current-behavior documentation only in the stage that changes that behavior. Do not describe a partially implemented later stage as complete. @@ -100,34 +100,21 @@ spell/combat breakpoint. Use no more than these four markers so the request remains portable across the configured backends; do not add module-specific markers in this work. -## Stage 1: Characterize The Rendered Cache Boundary +## Stage 1: Protect Shared Input Behavior And Baseline Prompt Wiring ### Goal -Protect the current three-message common prefix and establish reusable test -support before changing prompt composition. +Protect deterministic shared input rendering and confirm the existing prompt +assets prepare successfully before changing prompt composition. ### Implementation -1. Add a cross-lane prompt test under - `internal/modules/dnd/shared`, using the external test package - `shared_test` so it can import all three extractor packages without an - import cycle. -2. In one `llm.AssetRegistry`, register spell, NPC, and combat prompt assets. - Build one offline Scriptorium engine with one test profile and use - `Engine.Prepare` for every prompt. Supply identical transcript, players, - party, and glossary inputs. Supply the same unbound NPC JSON to spell and - combat, plus the spell catalog required only by spells. -3. Add a small test helper that compares an expected prefix of - `[]scriptorium.RenderedMessage` field by field: role, exact content string, - and a canonical JSON representation of cache control. Do not compare prompt - IDs, rendered prompt hashes, output contracts, or schemas across modules; - those are intentionally different. -4. Assert that the first three messages are identical across all lanes and that - spell/combat NPC grounding is identical at their current corresponding - position. Assert the existing message counts separately so Stage 3 must - update them deliberately. -5. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one +1. Retain one offline prompt preparation test in each extractor package. Each + test should prove registration succeeds, the prompt selects its package-owned + response schema, required dynamic inputs render, and no provider call or + credentials are required. Do not assert total message count, shared-prefix + length, or the complete rendered prompt. +2. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one table-driven identity test covering: - identical source material and references produce deeply equal common input materials; @@ -135,11 +122,9 @@ support before changing prompt composition. - `roster` fallback produces the canonical `party` material; - an explicit non-empty `party` wins over `roster`; and - missing optional slots render the existing single-space placeholder. -6. Do not add a second full HTTP/provider test. The Scriptorium engine's - prepared messages are the stable boundary owned by this repository; - Scriptorium owns provider request serialization. Existing LLM adapter tests - continue to prove that prepared message order and content reach the engine - request. +3. Do not add a full HTTP/provider test for cross-module prompt equality. + Scriptorium owns provider request serialization, and shared asset ownership + removes the duplicated content that would otherwise need equality testing. ### Verification @@ -151,10 +136,10 @@ go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./in ### Completion Criteria -- The new cross-lane test fails if any role, content byte, cache-control value, - or order within the intended common prefix differs. -- Tests distinguish intended prompt identity from intentionally different - prompt IDs, schemas, and module-specific suffixes. +- Shared input tests protect deterministic serialization and canonical slot + handling. +- Each extractor's existing behavior-level asset test prepares successfully + without pinning message count or prefix boundaries. - No production behavior changes in this stage. ## Stage 2: Make Prompt Asset Manifests Exact @@ -233,8 +218,8 @@ go test ./internal/framework/promptfs ./internal/modules/dnd/shared ./internal/m ### Goal -Implement the exact target prompt layout defined above and maximize the -byte-identical all-lane and spell/combat prefixes. +Implement the target prompt layout defined above using canonical shared assets +and maximize the reusable all-lane and spell/combat content. ### Implementation @@ -257,17 +242,12 @@ byte-identical all-lane and spell/combat prefixes. `Return exactly one JSON object` rule through the common evidence message. 6. Keep the spell catalog input and NPC registry input byte generation unchanged. Keep the NPC extractor free of an `npcs` input. -7. Update package asset tests for new message counts and positions. Replace - content-substring assertions for moved common policy with assertions in the - cross-lane rendered-prefix test. -8. Update the cross-lane test to assert: - - indices 0 through 4 are identical for all three lanes when common inputs - are identical; - - indices 0 through 2 remain identical when transcript and references differ; - - spell and combat indices 5 and 6 are identical for the same NPC registry; - - the shared prefixes end at the exact indices in the table; and - - module-specific suffixes are not accidentally identical or reordered. -9. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same +7. Update package asset tests only for behavioral wiring: prompt preparation, + response schema selection, and required dynamic input rendering. Do not add + assertions for total message count, exact shared-prefix length, or the end + index of a shared section. The shared manifest tests own canonical asset + selection. +8. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same change to describe the implemented common-prefix composition, exact prompt manifest fingerprinting, and cache-control placement. Describe current behavior, not this staged plan. @@ -283,8 +263,9 @@ go test ./internal/framework/llm ### Completion Criteria -- Prepared messages match the target table exactly. - Shared rules exist in one embedded file and have no local paraphrased copy. +- Every applicable prompt references those shared assets in the documented + order and with the documented cache-control policy. - Prompt fingerprints change for the intentional prompt contract change and include every newly rendered asset. - No artifact schema, durable representation, module selection, or reference @@ -339,7 +320,7 @@ go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./in - Only one production implementation prepares chunk prompt material. - Extractor behavior and public errors retain useful module context. -- The rendered prompt identity tests still pass. +- Prompt preparation and shared input behavior tests still pass. ## Stage 5: Share Cited Source Traversal @@ -534,7 +515,7 @@ default test suite. the fixed decision for this plan is to leave them package-owned and separate. 4. Consolidate redundant tests created by intermediate stages. Retain: - one shared manifest test suite; - - one exact rendered-prefix suite; + - behavior-level prompt preparation tests without prefix-length snapshots; - focused package tests for artifact-specific prompts and validators; and - existing production registration contract coverage. 5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the