Use shared assets instead of prompt prefix change detectors

This commit is contained in:
2026-07-21 13:59:28 +00:00
parent 3d70920f3d
commit e01b8d1b6d
2 changed files with 49 additions and 74 deletions

View File

@@ -33,17 +33,17 @@ The following decisions are complete and are not implementation-time choices:
registration, or erase typed artifact relationships outside existing
framework boundaries.
- Exact cache identity means equal ordered message roles, content bytes, and
cache-control values after Scriptorium has rendered the prompt. Tests must
use `scriptorium.PreparedRun.Messages`; comparing source Markdown is
insufficient.
cache-control values after Scriptorium has rendered the prompt. Enforce this
primarily through one canonical shared asset per shared message and document
the expected order and cache-control policy.
- Scriptorium and provider-specific types remain confined to the LLM runtime,
prompt-asset wiring, and their tests. Production extractors continue to use
only Notarius contracts.
- Default tests remain offline, deterministic, and credential-free. Live model
evaluation is an explicit manual acceptance activity, not part of
`go test ./...`.
- Avoid broad snapshots. Exact byte assertions are warranted only for the
rendered common message prefixes because byte identity is the feature.
- Do not add message-count, common-prefix-length, or complete rendered-prompt
snapshots. They are change detectors rather than durable behavioral tests.
- Update current-behavior documentation only in the stage that changes that
behavior. Do not describe a partially implemented later stage as complete.
@@ -100,34 +100,21 @@ spell/combat breakpoint. Use no more than these four markers so the request
remains portable across the configured backends; do not add module-specific
markers in this work.
## Stage 1: Characterize The Rendered Cache Boundary
## Stage 1: Protect Shared Input Behavior And Baseline Prompt Wiring
### Goal
Protect the current three-message common prefix and establish reusable test
support before changing prompt composition.
Protect deterministic shared input rendering and confirm the existing prompt
assets prepare successfully before changing prompt composition.
### Implementation
1. Add a cross-lane prompt test under
`internal/modules/dnd/shared`, using the external test package
`shared_test` so it can import all three extractor packages without an
import cycle.
2. In one `llm.AssetRegistry`, register spell, NPC, and combat prompt assets.
Build one offline Scriptorium engine with one test profile and use
`Engine.Prepare` for every prompt. Supply identical transcript, players,
party, and glossary inputs. Supply the same unbound NPC JSON to spell and
combat, plus the spell catalog required only by spells.
3. Add a small test helper that compares an expected prefix of
`[]scriptorium.RenderedMessage` field by field: role, exact content string,
and a canonical JSON representation of cache control. Do not compare prompt
IDs, rendered prompt hashes, output contracts, or schemas across modules;
those are intentionally different.
4. Assert that the first three messages are identical across all lanes and that
spell/combat NPC grounding is identical at their current corresponding
position. Assert the existing message counts separately so Stage 3 must
update them deliberately.
5. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one
1. Retain one offline prompt preparation test in each extractor package. Each
test should prove registration succeeds, the prompt selects its package-owned
response schema, required dynamic inputs render, and no provider call or
credentials are required. Do not assert total message count, shared-prefix
length, or the complete rendered prompt.
2. Extend `internal/modules/dnd/shared/prompt_inputs_test.go` with one
table-driven identity test covering:
- identical source material and references produce deeply equal common input
materials;
@@ -135,11 +122,9 @@ support before changing prompt composition.
- `roster` fallback produces the canonical `party` material;
- an explicit non-empty `party` wins over `roster`; and
- missing optional slots render the existing single-space placeholder.
6. Do not add a second full HTTP/provider test. The Scriptorium engine's
prepared messages are the stable boundary owned by this repository;
Scriptorium owns provider request serialization. Existing LLM adapter tests
continue to prove that prepared message order and content reach the engine
request.
3. Do not add a full HTTP/provider test for cross-module prompt equality.
Scriptorium owns provider request serialization, and shared asset ownership
removes the duplicated content that would otherwise need equality testing.
### Verification
@@ -151,10 +136,10 @@ go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./in
### Completion Criteria
- The new cross-lane test fails if any role, content byte, cache-control value,
or order within the intended common prefix differs.
- Tests distinguish intended prompt identity from intentionally different
prompt IDs, schemas, and module-specific suffixes.
- Shared input tests protect deterministic serialization and canonical slot
handling.
- Each extractor's existing behavior-level asset test prepares successfully
without pinning message count or prefix boundaries.
- No production behavior changes in this stage.
## Stage 2: Make Prompt Asset Manifests Exact
@@ -233,8 +218,8 @@ go test ./internal/framework/promptfs ./internal/modules/dnd/shared ./internal/m
### Goal
Implement the exact target prompt layout defined above and maximize the
byte-identical all-lane and spell/combat prefixes.
Implement the target prompt layout defined above using canonical shared assets
and maximize the reusable all-lane and spell/combat content.
### Implementation
@@ -257,17 +242,12 @@ byte-identical all-lane and spell/combat prefixes.
`Return exactly one JSON object` rule through the common evidence message.
6. Keep the spell catalog input and NPC registry input byte generation
unchanged. Keep the NPC extractor free of an `npcs` input.
7. Update package asset tests for new message counts and positions. Replace
content-substring assertions for moved common policy with assertions in the
cross-lane rendered-prefix test.
8. Update the cross-lane test to assert:
- indices 0 through 4 are identical for all three lanes when common inputs
are identical;
- indices 0 through 2 remain identical when transcript and references differ;
- spell and combat indices 5 and 6 are identical for the same NPC registry;
- the shared prefixes end at the exact indices in the table; and
- module-specific suffixes are not accidentally identical or reordered.
9. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same
7. Update package asset tests only for behavioral wiring: prompt preparation,
response schema selection, and required dynamic input rendering. Do not add
assertions for total message count, exact shared-prefix length, or the end
index of a shared section. The shared manifest tests own canonical asset
selection.
8. Update `docs/internal/modules.md` and `docs/internal/llm.md` in the same
change to describe the implemented common-prefix composition, exact prompt
manifest fingerprinting, and cache-control placement. Describe current
behavior, not this staged plan.
@@ -283,8 +263,9 @@ go test ./internal/framework/llm
### Completion Criteria
- Prepared messages match the target table exactly.
- Shared rules exist in one embedded file and have no local paraphrased copy.
- Every applicable prompt references those shared assets in the documented
order and with the documented cache-control policy.
- Prompt fingerprints change for the intentional prompt contract change and
include every newly rendered asset.
- No artifact schema, durable representation, module selection, or reference
@@ -339,7 +320,7 @@ go test ./internal/modules/dnd/shared ./internal/modules/dnd/extract/spells ./in
- Only one production implementation prepares chunk prompt material.
- Extractor behavior and public errors retain useful module context.
- The rendered prompt identity tests still pass.
- Prompt preparation and shared input behavior tests still pass.
## Stage 5: Share Cited Source Traversal
@@ -534,7 +515,7 @@ default test suite.
the fixed decision for this plan is to leave them package-owned and separate.
4. Consolidate redundant tests created by intermediate stages. Retain:
- one shared manifest test suite;
- one exact rendered-prefix suite;
- behavior-level prompt preparation tests without prefix-length snapshots;
- focused package tests for artifact-specific prompts and validators; and
- existing production registration contract coverage.
5. Update `docs/internal/modules.md` and `docs/internal/llm.md` so they are the