Use shared assets instead of prompt prefix change detectors

This commit is contained in:
2026-07-21 13:59:28 +00:00
parent 3d70920f3d
commit e01b8d1b6d
2 changed files with 49 additions and 74 deletions

View File

@@ -56,34 +56,28 @@ module-specific message:
wording differs by lane. Factor the common bytes into one message and leave wording differs by lane. Factor the common bytes into one message and leave
the differences in later messages instead. the differences in later messages instead.
Tests inspect the fully rendered Scriptorium request boundary. Source-level Shared messages are embedded once and referenced by every applicable prompt.
message identity alone is not treated as proof of cache identity. This makes reuse of the canonical asset the easiest way to preserve cache
identity and avoids parallel module-specific copies that can drift.
### Exact Prompt And Input Identity Is Protected ### Shared Prompt Ownership And Input Behavior Are Protected
Focused tests at the narrowest stable boundary expose the fully prepared Focused tests protect shared asset composition and input behavior without
provider-neutral request. snapshotting the length or exact boundary of the common message prefix.
- Render each extractor prompt with the same transcript and campaign - Verify that each applicable prompt mounts and renders the canonical shared
references and assert that the intended common message prefix has identical assets rather than module-specific copies.
roles, content bytes, ordering, and cache-control metadata.
- Assert that spells and combat turns render identical NPC-registry messages
for the same bound or unbound registry.
- Verify common input material identity, including name, media type, content, - Verify common input material identity, including name, media type, content,
digest, origin URI, size, empty-value representation, reference ordering, digest, origin URI, size, empty-value representation, reference ordering,
and `roster` fallback behavior. and `roster` fallback behavior.
- Test both identical and intentionally different chunks and reference sets so
the test proves the cache boundary rather than merely snapshotting one
request.
- Add an explicit assertion for the length of the common prefix. A new
module-specific message inserted inside that prefix should require deliberate
review.
- Test that every prompt fingerprint includes exactly the assets actually - Test that every prompt fingerprint includes exactly the assets actually
rendered by that prompt. The current grouped reference hash helper should be rendered by that prompt. The current grouped reference hash helper should be
replaced or refined so a module does not fingerprint an unused shared asset, replaced or refined so a module does not fingerprint an unused shared asset,
while no used asset is omitted. while no used asset is omitted.
- Prefer semantic assertions over complete prompt snapshots, except for the - Document the expected shared message ordering and cache-control policy so
common rendered prefix whose exact bytes are the behavior under protection. later changes are reviewed against the caching intent.
- Prefer behavioral and ownership assertions over message-count, prefix-length,
or full-prompt snapshots.
### Stable Extraction Preparation Is Centralized ### Stable Extraction Preparation Is Centralized
@@ -175,10 +169,10 @@ This roadmap is complete when:
- the three lanes expose the same stable structural conventions while keeping - the three lanes expose the same stable structural conventions while keeping
documented domain differences local; documented domain differences local;
- shared provider-visible messages and inputs are produced from one source and - shared provider-visible messages are produced from canonical shared assets,
verified byte-for-byte at the rendered-request boundary; and shared inputs have one deterministic implementation;
- the common prompt prefix is deliberate, tested, and as long as extraction - the common prompt ordering is deliberate, documented, and as cache-friendly
quality permits; as extraction quality permits;
- every validator policy that affects reusable results participates in - every validator policy that affects reusable results participates in
checkpoint identity; checkpoint identity;
- source-reference traversal and diagnostics no longer drift between lanes; - source-reference traversal and diagnostics no longer drift between lanes;

View File

@@ -33,17 +33,17 @@ The following decisions are complete and are not implementation-time choices:
registration, or erase typed artifact relationships outside existing registration, or erase typed artifact relationships outside existing
framework boundaries. framework boundaries.
- Exact cache identity means equal ordered message roles, content bytes, and - Exact cache identity means equal ordered message roles, content bytes, and
cache-control values after Scriptorium has rendered the prompt. Tests must cache-control values after Scriptorium has rendered the prompt. Enforce this
use `scriptorium.PreparedRun.Messages`; comparing source Markdown is primarily through one canonical shared asset per shared message and document
insufficient. the expected order and cache-control policy.
- Scriptorium and provider-specific types remain confined to the LLM runtime, - Scriptorium and provider-specific types remain confined to the LLM runtime,
prompt-asset wiring, and their tests. Production extractors continue to use prompt-asset wiring, and their tests. Production extractors continue to use
only Notarius contracts. only Notarius contracts.
- Default tests remain offline, deterministic, and credential-free. Live model - Default tests remain offline, deterministic, and credential-free. Live model
evaluation is an explicit manual acceptance activity, not part of evaluation is an explicit manual acceptance activity, not part of
`go test ./...`. `go test ./...`.
- Avoid broad snapshots. Exact byte assertions are warranted only for the - Do not add message-count, common-prefix-length, or complete rendered-prompt
rendered common message prefixes because byte identity is the feature. snapshots. They are change detectors rather than durable behavioral tests.
- Update current-behavior documentation only in the stage that changes that - Update current-behavior documentation only in the stage that changes that
behavior. Do not describe a partially implemented later stage as complete. behavior. Do not describe a partially implemented later stage as complete.
@@ -100,34 +100,21 @@ spell/combat breakpoint. Use no more than these four markers so the request
remains portable across the configured backends; do not add module-specific remains portable across the configured backends; do not add module-specific
markers in this work. markers in this work.
## Stage 1: Characterize The Rendered Cache Boundary ## Stage 1: Protect Shared Input Behavior And Baseline Prompt Wiring
### Goal ### Goal
Protect the current three-message common prefix and establish reusable test Protect deterministic shared input rendering and confirm the existing prompt
support before changing prompt composition. assets prepare successfully before changing prompt composition.
### Implementation ### Implementation
1. Add a cross-lane prompt test under 1. Retain one offline prompt preparation test in each extractor package. Each
`internal/modules/dnd/shared`, using the external test package test should prove registration succeeds, the prompt selects its package-owned
`shared_test` so it can import all three extractor packages without an response schema, required dynamic inputs render, and no provider call or
import cycle. credentials are required. Do not assert total message count, shared-prefix
2. In one `llm.AssetRegistry`, register spell, NPC, and combat prompt assets. length, or the complete rendered prompt.
Build one offline Scriptorium engine with one test profile and use 2. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one
`Engine.Prepare` for every prompt. Supply identical transcript, players,
party, and glossary inputs. Supply the same unbound NPC JSON to spell and
combat, plus the spell catalog required only by spells.
3. Add a small test helper that compares an expected prefix of
`[]scriptorium.RenderedMessage` field by field: role, exact content string,
and a canonical JSON representation of cache control. Do not compare prompt
IDs, rendered prompt hashes, output contracts, or schemas across modules;
those are intentionally different.
4. Assert that the first three messages are identical across all lanes and that
spell/combat NPC grounding is identical at their current corresponding
position. Assert the existing message counts separately so Stage 3 must
update them deliberately.
5. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one
table-driven identity test covering: table-driven identity test covering:
- identical source material and references produce deeply equal common input - identical source material and references produce deeply equal common input
materials; materials;
@@ -135,11 +122,9 @@ support before changing prompt composition.
- `roster` fallback produces the canonical `party` material; - `roster` fallback produces the canonical `party` material;
- an explicit non-empty `party` wins over `roster`; and - an explicit non-empty `party` wins over `roster`; and
- missing optional slots render the existing single-space placeholder. - missing optional slots render the existing single-space placeholder.
6. Do not add a second full HTTP/provider test. The Scriptorium engine's 3. Do not add a full HTTP/provider test for cross-module prompt equality.
prepared messages are the stable boundary owned by this repository; Scriptorium owns provider request serialization, and shared asset ownership
Scriptorium owns provider request serialization. Existing LLM adapter tests removes the duplicated content that would otherwise need equality testing.
continue to prove that prepared message order and content reach the engine
request.
### Verification ### Verification
@@ -151,10 +136,10 @@ go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./in
### Completion Criteria ### Completion Criteria
- The new cross-lane test fails if any role, content byte, cache-control value, - Shared input tests protect deterministic serialization and canonical slot
or order within the intended common prefix differs. handling.
- Tests distinguish intended prompt identity from intentionally different - Each extractor's existing behavior-level asset test prepares successfully
prompt IDs, schemas, and module-specific suffixes. without pinning message count or prefix boundaries.
- No production behavior changes in this stage. - No production behavior changes in this stage.
## Stage 2: Make Prompt Asset Manifests Exact ## Stage 2: Make Prompt Asset Manifests Exact
@@ -233,8 +218,8 @@ go test ./internal/framework/promptfs ./internal/modules/dnd/shared ./internal/m
### Goal ### Goal
Implement the exact target prompt layout defined above and maximize the Implement the target prompt layout defined above using canonical shared assets
byte-identical all-lane and spell/combat prefixes. and maximize the reusable all-lane and spell/combat content.
### Implementation ### Implementation
@@ -257,17 +242,12 @@ byte-identical all-lane and spell/combat prefixes.
`Return exactly one JSON object` rule through the common evidence message. `Return exactly one JSON object` rule through the common evidence message.
6. Keep the spell catalog input and NPC registry input byte generation 6. Keep the spell catalog input and NPC registry input byte generation
unchanged. Keep the NPC extractor free of an `npcs` input. unchanged. Keep the NPC extractor free of an `npcs` input.
7. Update package asset tests for new message counts and positions. Replace 7. Update package asset tests only for behavioral wiring: prompt preparation,
content-substring assertions for moved common policy with assertions in the response schema selection, and required dynamic input rendering. Do not add
cross-lane rendered-prefix test. assertions for total message count, exact shared-prefix length, or the end
8. Update the cross-lane test to assert: index of a shared section. The shared manifest tests own canonical asset
- indices 0 through 4 are identical for all three lanes when common inputs selection.
are identical; 8. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same
- indices 0 through 2 remain identical when transcript and references differ;
- spell and combat indices 5 and 6 are identical for the same NPC registry;
- the shared prefixes end at the exact indices in the table; and
- module-specific suffixes are not accidentally identical or reordered.
9. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same
change to describe the implemented common-prefix composition, exact prompt change to describe the implemented common-prefix composition, exact prompt
manifest fingerprinting, and cache-control placement. Describe current manifest fingerprinting, and cache-control placement. Describe current
behavior, not this staged plan. behavior, not this staged plan.
@@ -283,8 +263,9 @@ go test ./internal/framework/llm
### Completion Criteria ### Completion Criteria
- Prepared messages match the target table exactly.
- Shared rules exist in one embedded file and have no local paraphrased copy. - Shared rules exist in one embedded file and have no local paraphrased copy.
- Every applicable prompt references those shared assets in the documented
order and with the documented cache-control policy.
- Prompt fingerprints change for the intentional prompt contract change and - Prompt fingerprints change for the intentional prompt contract change and
include every newly rendered asset. include every newly rendered asset.
- No artifact schema, durable representation, module selection, or reference - No artifact schema, durable representation, module selection, or reference
@@ -339,7 +320,7 @@ go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./in
- Only one production implementation prepares chunk prompt material. - Only one production implementation prepares chunk prompt material.
- Extractor behavior and public errors retain useful module context. - Extractor behavior and public errors retain useful module context.
- The rendered prompt identity tests still pass. - Prompt preparation and shared input behavior tests still pass.
## Stage 5: Share Cited Source Traversal ## Stage 5: Share Cited Source Traversal
@@ -534,7 +515,7 @@ default test suite.
the fixed decision for this plan is to leave them package-owned and separate. the fixed decision for this plan is to leave them package-owned and separate.
4. Consolidate redundant tests created by intermediate stages. Retain: 4. Consolidate redundant tests created by intermediate stages. Retain:
- one shared manifest test suite; - one shared manifest test suite;
- one exact rendered-prefix suite; - behavior-level prompt preparation tests without prefix-length snapshots;
- focused package tests for artifact-specific prompts and validators; and - focused package tests for artifact-specific prompts and validators; and
- existing production registration contract coverage. - existing production registration contract coverage.
5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the 5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the