Retire completed roadmap documents and add ideas for future feature work
This commit is contained in:
@@ -1,90 +0,0 @@
|
|||||||
# D&D Prompt Cache Ordering
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Arrange maintained D&D prompts so sibling extraction lanes share the longest
|
|
||||||
useful byte-identical prefix, including the transcript chunk, and can therefore
|
|
||||||
reuse provider-side prompt caches effectively.
|
|
||||||
|
|
||||||
This policy concerns rendered message order and cache boundaries. It does not
|
|
||||||
change extraction semantics, artifact contracts, module inputs, durable output,
|
|
||||||
or the responsibilities of deterministic validators and normalizers.
|
|
||||||
|
|
||||||
## Extraction Prompt Policy
|
|
||||||
|
|
||||||
Every D&D extraction prompt shares this rendered prefix:
|
|
||||||
|
|
||||||
1. the shared D&D system prompt;
|
|
||||||
2. the shared D&D identity prompt;
|
|
||||||
3. the shared campaign-reference prompt; and
|
|
||||||
4. the shared transcript-chunk prompt.
|
|
||||||
|
|
||||||
The messages must render identically, including roles, content, and cache
|
|
||||||
metadata, when two lanes receive the same transcript and common references.
|
|
||||||
The campaign-reference and transcript messages establish ephemeral cache
|
|
||||||
boundaries.
|
|
||||||
|
|
||||||
Only material used by every extraction lane belongs before the transcript.
|
|
||||||
Shared wording that applies to only a subset of lanes is placed after the
|
|
||||||
transcript rather than being added to unrelated prompts solely to lengthen the
|
|
||||||
common prefix. Module instructions, optional generated references, catalogs,
|
|
||||||
and other lane-specific context also follow the transcript.
|
|
||||||
|
|
||||||
The lane-specific suffixes are:
|
|
||||||
|
|
||||||
- NPCs: shared extraction-evidence policy, task, then instructions.
|
|
||||||
- Item events: shared extraction-evidence policy, task, then instructions.
|
|
||||||
- Scene descriptions: task, then instructions.
|
|
||||||
- Combat turns: shared extraction-evidence policy, NPC registry, task, then
|
|
||||||
instructions.
|
|
||||||
- NPC interactions: shared extraction-evidence policy, NPC registry, task,
|
|
||||||
then instructions.
|
|
||||||
- Spells: shared extraction-evidence policy, NPC registry, spell catalog, task,
|
|
||||||
then instructions.
|
|
||||||
|
|
||||||
The final instructions message establishes an ephemeral cache boundary.
|
|
||||||
Intermediate cache markers on the shared identity message or lane-specific
|
|
||||||
reference messages are unnecessary.
|
|
||||||
|
|
||||||
## Purpose-Specific Prompt Families
|
|
||||||
|
|
||||||
The extraction prefix is a sibling-lane policy, not a universal ordering rule
|
|
||||||
for every D&D LLM call.
|
|
||||||
|
|
||||||
Scene chunking has no sibling lane with which to share a transcript prefix. It
|
|
||||||
orders its prompt as system, common campaign references, task, instructions,
|
|
||||||
and full transcript. The campaign-reference and transcript messages establish
|
|
||||||
ephemeral cache boundaries.
|
|
||||||
|
|
||||||
NPC normalization orders its prompt as system, task, instructions, candidate
|
|
||||||
NPCs, and transcript windows. The instructions and transcript-window messages
|
|
||||||
establish ephemeral cache boundaries. Candidate artifacts remain ahead of the
|
|
||||||
evidence windows needed to evaluate them.
|
|
||||||
|
|
||||||
Future D&D prompt families should identify their actual reuse boundary rather
|
|
||||||
than mechanically copying either exception or the extraction sequence.
|
|
||||||
|
|
||||||
## Compatibility And Observability
|
|
||||||
|
|
||||||
Existing prompt IDs, prompt versions, response schemas, and module contracts
|
|
||||||
remain unchanged. Prompt-content fingerprints already make the reordered
|
|
||||||
assets part of checkpoint identity, so old development checkpoints may become
|
|
||||||
cold misses without a checkpoint-format migration.
|
|
||||||
|
|
||||||
Prompt-order verification should exercise rendered messages with unique input
|
|
||||||
sentinels. Tests must protect roles, cache metadata, input isolation, and the
|
|
||||||
shared-prefix invariant without requiring particular prose, words, or phrases
|
|
||||||
to remain in prompt assets.
|
|
||||||
|
|
||||||
## Desired End State
|
|
||||||
|
|
||||||
- All six D&D extraction prompts render the same four-message prefix for
|
|
||||||
equivalent common inputs.
|
|
||||||
- The transcript chunk is included in that common prefix and is followed only
|
|
||||||
by lane-specific material.
|
|
||||||
- Cache-control hints identify useful prefix boundaries without redundant
|
|
||||||
intermediate markers.
|
|
||||||
- Scene chunking and NPC normalization retain orderings suited to their
|
|
||||||
distinct inputs and reuse opportunities.
|
|
||||||
- Internal documentation explains both the general provider-cache principle
|
|
||||||
and the concrete D&D prompt-family policy.
|
|
||||||
@@ -7,6 +7,37 @@ not as committed release dates.
|
|||||||
|
|
||||||
## Near-Term D&D Pipeline
|
## Near-Term D&D Pipeline
|
||||||
|
|
||||||
|
### Combat Enemy Ledger
|
||||||
|
|
||||||
|
- Add a D&D artifact that identifies enemies faced during combat and supports
|
||||||
|
an end-of-session encounter ledger.
|
||||||
|
- Track each enemy's observed state using a small controlled vocabulary such as
|
||||||
|
`active`, `killed`, `fled`, `captured`, or `incapacitated`, while preserving
|
||||||
|
an explicit unresolved state when the transcript does not establish an
|
||||||
|
outcome.
|
||||||
|
- Preserve the evidence for enemy participation and state changes rather than
|
||||||
|
inferring a terminal outcome from combat ending or an enemy disappearing
|
||||||
|
from the conversation.
|
||||||
|
- Define how repeated mentions, groups of unnamed enemies, summoned or allied
|
||||||
|
creatures, and the same enemy appearing in multiple combats affect identity
|
||||||
|
and ledger entries.
|
||||||
|
- Evaluate whether the ledger should be extracted directly, derived from
|
||||||
|
combat-turn artifacts, or use a sequential pipeline that consumes combat
|
||||||
|
turns and the normalized NPC registry as grounding references.
|
||||||
|
|
||||||
|
### Location Extraction
|
||||||
|
|
||||||
|
- Add a D&D artifact for locations visited by the party or otherwise mentioned
|
||||||
|
in the transcript.
|
||||||
|
- Distinguish observed visits from references, plans, recalled places, and
|
||||||
|
uncertain or inferred locations so a mention alone is not reported as a
|
||||||
|
visit.
|
||||||
|
- Preserve transcript evidence for each visit or mention and reconcile aliases,
|
||||||
|
nested places, and repeated appearances without collapsing distinct
|
||||||
|
locations that share a generic name.
|
||||||
|
- Define how the location artifact should ground later narrative reports and
|
||||||
|
whether future event artifacts should retain canonical location identities.
|
||||||
|
|
||||||
### Evaluate Spell Extraction And Normalization
|
### Evaluate Spell Extraction And Normalization
|
||||||
|
|
||||||
- Evaluate ordinary extraction retries and the completed normalization path
|
- Evaluate ordinary extraction retries and the completed normalization path
|
||||||
|
|||||||
@@ -1,257 +0,0 @@
|
|||||||
# D&D Prompt Cache Ordering Implementation Plan
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Implement the target state in
|
|
||||||
[D&D Prompt Cache Ordering](dnd-prompt-cache-ordering.md): give all six D&D
|
|
||||||
extraction prompts a byte-identical rendered prefix through the transcript
|
|
||||||
chunk, retain purpose-specific ordering for scene chunking and NPC
|
|
||||||
normalization, consolidate tests around durable prompt-order invariants, and
|
|
||||||
replace the current transcript-last documentation.
|
|
||||||
|
|
||||||
This is a prompt-asset and documentation change. Do not change Go module APIs,
|
|
||||||
LLM request contracts, prompt IDs or versions, response schemas, artifact
|
|
||||||
schemas, reference declarations, extraction behavior, retry behavior, or
|
|
||||||
durable output. Existing prompt fingerprints must continue to provide
|
|
||||||
checkpoint invalidation; no checkpoint migration or cleanup is required.
|
|
||||||
|
|
||||||
## Stage 1: Reorder The Six Extraction Prompts
|
|
||||||
|
|
||||||
Update these manifests:
|
|
||||||
|
|
||||||
- `internal/modules/dnd/extract/npcs/assets/prompts/dnd.npcs.yaml`
|
|
||||||
- `internal/modules/dnd/extract/itemevents/assets/prompts/dnd.item_events.yaml`
|
|
||||||
- `internal/modules/dnd/extract/scenedescriptions/assets/prompts/dnd.scene_descriptions.yaml`
|
|
||||||
- `internal/modules/dnd/extract/combatturns/assets/prompts/dnd.combat_turns.yaml`
|
|
||||||
- `internal/modules/dnd/extract/npcinteractions/assets/prompts/dnd.npc_interactions.yaml`
|
|
||||||
- `internal/modules/dnd/extract/spells/assets/prompts/dnd.spells.yaml`
|
|
||||||
|
|
||||||
Give every manifest this exact leading message sequence:
|
|
||||||
|
|
||||||
1. `common-dnd-system.md`, with role `system` and no cache control;
|
|
||||||
2. `common-dnd-identity.md`, with role `user` and no cache control;
|
|
||||||
3. `common-dnd-references.md`, with role `user` and ephemeral cache control;
|
|
||||||
4. `common-dnd-transcript.md`, with role `user` and ephemeral cache control.
|
|
||||||
|
|
||||||
The common prefix must contain no extraction-evidence policy, NPC registry,
|
|
||||||
spell catalog, task, or module instructions. Those messages vary across lanes
|
|
||||||
and would prevent the transcript from participating in a shared cache prefix.
|
|
||||||
|
|
||||||
Append the following lane-specific sequences after the transcript:
|
|
||||||
|
|
||||||
| Prompt | Post-transcript sequence |
|
|
||||||
| --- | --- |
|
|
||||||
| `dnd.npcs` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.item_events` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.scene_descriptions` | `task.md`, `instructions.md` |
|
|
||||||
| `dnd.combat_turns` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.npc_interactions` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
|
||||||
| `dnd.spells` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `catalog.md`, `task.md`, `instructions.md` |
|
|
||||||
|
|
||||||
For every extraction prompt:
|
|
||||||
|
|
||||||
- apply ephemeral cache control to the final `instructions.md` message;
|
|
||||||
- do not apply cache control to the extraction-evidence, NPC-registry, catalog,
|
|
||||||
or task messages;
|
|
||||||
- remove the existing cache control from `common-dnd-identity.md` and
|
|
||||||
`common-dnd-npcs.md`; and
|
|
||||||
- retain the existing inputs, requiredness, media types, output contract,
|
|
||||||
profile, repair setting, prompt identity, and schema path.
|
|
||||||
|
|
||||||
Do not add the extraction-evidence asset to the scene-description prompt merely
|
|
||||||
to make it resemble the other lanes. It is not universal shared context and
|
|
||||||
therefore belongs outside the common-prefix contract.
|
|
||||||
|
|
||||||
### Stage 1 completion criteria
|
|
||||||
|
|
||||||
- Equivalent transcript, player, party, and glossary inputs render the same
|
|
||||||
first four messages for all six extraction prompts, including identical
|
|
||||||
roles, content, and cache metadata.
|
|
||||||
- Each transcript input is rendered exactly once.
|
|
||||||
- Every lane-specific message follows the transcript.
|
|
||||||
- Prompt preparation and schema wiring remain successful for every lane.
|
|
||||||
|
|
||||||
## Stage 2: Apply Purpose-Specific Ordering To Other D&D LLM Prompts
|
|
||||||
|
|
||||||
Update
|
|
||||||
`internal/modules/dnd/chunk/scenes/assets/prompts/dnd.scenes.yaml` to use:
|
|
||||||
|
|
||||||
1. `common-dnd-system.md`, without cache control;
|
|
||||||
2. `common-dnd-references.md`, with ephemeral cache control;
|
|
||||||
3. `task.md`, without cache control;
|
|
||||||
4. `instructions.md`, without cache control; and
|
|
||||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
|
||||||
|
|
||||||
The scene chunker consumes the full transcript once and has no sibling lanes
|
|
||||||
that can share a transcript prefix. Stable task instructions therefore remain
|
|
||||||
before the changing full transcript.
|
|
||||||
|
|
||||||
Update
|
|
||||||
`internal/modules/dnd/normalize/npcs/assets/prompts/dnd.npcs.normalize.yaml` to
|
|
||||||
use:
|
|
||||||
|
|
||||||
1. `common-dnd-system.md`, without cache control;
|
|
||||||
2. `task.md`, without cache control;
|
|
||||||
3. `instructions.md`, with ephemeral cache control;
|
|
||||||
4. `candidates.md`, without cache control; and
|
|
||||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
|
||||||
|
|
||||||
Remove the normalizer system message's current cache control. Preserve the
|
|
||||||
candidate-before-evidence ordering: the transcript windows support evaluation
|
|
||||||
of the candidate collection and are not a cross-lane transcript prefix.
|
|
||||||
|
|
||||||
Do not change input material construction, transcript windowing, prompt
|
|
||||||
fragments, module implementations, or response handling in either package.
|
|
||||||
|
|
||||||
### Stage 2 completion criteria
|
|
||||||
|
|
||||||
- Scene chunking renders references separately before the task and the full
|
|
||||||
transcript exactly once at the end.
|
|
||||||
- NPC normalization renders candidates and transcript windows in separate
|
|
||||||
messages, in that order, with the agreed cache boundaries.
|
|
||||||
- Both prompts retain their current identities, schemas, inputs, and runtime
|
|
||||||
behavior.
|
|
||||||
|
|
||||||
## Stage 3: Consolidate Prompt-Ordering Verification
|
|
||||||
|
|
||||||
Read and follow `docs/policy/testing.md` before modifying tests. The meaningful
|
|
||||||
risk is loss of the shared rendered prefix or incorrect placement of
|
|
||||||
request-specific material, not edits to prompt prose.
|
|
||||||
|
|
||||||
Add one family-level extraction invariant test in
|
|
||||||
`internal/modules/dnd/register/prompt_cache_test.go` (or an equivalently scoped
|
|
||||||
new register-package test file):
|
|
||||||
|
|
||||||
- register the complete D&D prompt asset family in a real in-memory
|
|
||||||
`llm.AssetRegistry`;
|
|
||||||
- construct a Scriptorium engine with a local, non-networked test profile;
|
|
||||||
- prepare all six extraction prompts using their exported prompt IDs and
|
|
||||||
versions;
|
|
||||||
- supply the same unique transcript, players, party, and glossary sentinels to
|
|
||||||
every prompt, plus only the lane-specific required inputs such as the NPC
|
|
||||||
registry and spell catalog;
|
|
||||||
- locate the one rendered message containing the transcript sentinel rather
|
|
||||||
than locating it through prompt prose or a hard-coded phrase;
|
|
||||||
- assert that the transcript sentinel occurs exactly once in each prepared
|
|
||||||
prompt;
|
|
||||||
- compare the rendered messages from the start through the transcript message
|
|
||||||
across all six prompts, including role, full rendered content, and complete
|
|
||||||
cache-control metadata;
|
|
||||||
- assert that the shared prefix contains exactly the four agreed messages and
|
|
||||||
that at least one lane-specific message follows it in every extraction
|
|
||||||
prompt; and
|
|
||||||
- assert that NPC-registry and spell-catalog sentinels, where supplied, occur
|
|
||||||
only after the transcript.
|
|
||||||
|
|
||||||
Use this family-level test as the single owner of the cross-lane prefix policy.
|
|
||||||
Do not duplicate the same prefix assertion in every extractor package.
|
|
||||||
|
|
||||||
Revise the existing prompt-asset tests in:
|
|
||||||
|
|
||||||
- `internal/modules/dnd/extract/npcs/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/itemevents/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/scenedescriptions/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/combatturns/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/npcinteractions/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/extract/spells/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/chunk/scenes/scriptorium_assets_test.go`
|
|
||||||
- `internal/modules/dnd/normalize/npcs/scriptorium_assets_test.go`
|
|
||||||
|
|
||||||
Remove transcript-last expectations that contradict the new extraction policy.
|
|
||||||
Keep package-local coverage only where it protects package-owned behavior, such
|
|
||||||
as prompt/schema registration, declared shared assets, isolation and placement
|
|
||||||
of package-specific input material, cache boundaries unique to the scene
|
|
||||||
chunker or NPC normalizer, and non-disclosure properties. Consolidate or remove
|
|
||||||
positional and prose-marker checks already covered by the family-level
|
|
||||||
invariant.
|
|
||||||
|
|
||||||
Tests may search for unique input sentinels created by the test. They must not
|
|
||||||
require the presence of specific words, phrases, sentences, or other prose in a
|
|
||||||
prompt asset. Do not introduce golden snapshots of complete prompts or a test
|
|
||||||
that treats harmless wording changes as failures.
|
|
||||||
|
|
||||||
### Stage 3 completion criteria
|
|
||||||
|
|
||||||
- One durable family-level test fails if any extraction lane moves
|
|
||||||
lane-specific material ahead of the transcript or changes the common
|
|
||||||
rendered prefix.
|
|
||||||
- Local asset tests protect only nonredundant package behavior and the two
|
|
||||||
purpose-specific prompt sequences.
|
|
||||||
- The test suite contains no prompt-language change detector.
|
|
||||||
- All prompt tests remain deterministic, offline, and independent of
|
|
||||||
credentials.
|
|
||||||
|
|
||||||
## Stage 4: Update Canonical Internal Documentation
|
|
||||||
|
|
||||||
Update `docs/internal/llm.md` as the generic owner of prompt-cache mechanics:
|
|
||||||
|
|
||||||
- explain that backend reuse depends on identical preceding roles, rendered
|
|
||||||
bytes, and cache metadata, not merely equivalent semantics;
|
|
||||||
- state the general sibling-prompt ordering rule: universal shared context,
|
|
||||||
request source material, then module-specific suffixes;
|
|
||||||
- explain that a cache boundary should be placed at a useful reusable prefix
|
|
||||||
and that redundant intermediate boundaries add no value;
|
|
||||||
- retain the rule that prompt-family owners may choose a different sequence
|
|
||||||
when their inputs and reuse pattern differ; and
|
|
||||||
- replace the current statement that D&D extraction puts transcripts last with
|
|
||||||
a link to the D&D-specific policy.
|
|
||||||
|
|
||||||
Update `docs/internal/dnd.md` as the concrete owner of maintained D&D prompt
|
|
||||||
composition:
|
|
||||||
|
|
||||||
- document the four-message extraction prefix and its cache boundaries;
|
|
||||||
- state that extraction-evidence policy, generated NPC registries, catalogs,
|
|
||||||
tasks, and instructions follow the transcript because they are not universal
|
|
||||||
across all extraction lanes;
|
|
||||||
- summarize the scene-chunking and NPC-normalization exceptions and their
|
|
||||||
rationale; and
|
|
||||||
- retain existing rules about shared-asset reuse, canonical reference input,
|
|
||||||
prompt fingerprints, and prompt behavior.
|
|
||||||
|
|
||||||
Do not update the README, configuration reference, architecture policy, CLI
|
|
||||||
documentation, or integration contracts. Prompt message order is an internal
|
|
||||||
runtime policy and has no user-visible configuration or wire-contract change.
|
|
||||||
|
|
||||||
### Stage 4 completion criteria
|
|
||||||
|
|
||||||
- No current-behavior document still says that D&D extraction transcripts are
|
|
||||||
final messages.
|
|
||||||
- Generic cache mechanics and D&D-specific ordering each have one canonical
|
|
||||||
owner.
|
|
||||||
- Documentation describes implemented behavior once the prompt changes land
|
|
||||||
and does not duplicate volatile manifest inventories unnecessarily.
|
|
||||||
|
|
||||||
## Stage 5: Verification
|
|
||||||
|
|
||||||
Inspect representative prepared prompts to confirm that the test inputs produce
|
|
||||||
the intended shared prefix and lane-specific suffixes. Then run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
git diff --check
|
|
||||||
go test ./internal/modules/dnd/chunk/scenes
|
|
||||||
go test ./internal/modules/dnd/extract/...
|
|
||||||
go test ./internal/modules/dnd/normalize/npcs
|
|
||||||
go test ./internal/modules/dnd/register
|
|
||||||
go test ./internal/modules/dnd/...
|
|
||||||
go test ./internal/modules/integration/...
|
|
||||||
go test ./...
|
|
||||||
go vet ./...
|
|
||||||
go build ./cmd/notarius
|
|
||||||
```
|
|
||||||
|
|
||||||
The focused commands should be used while implementing their respective
|
|
||||||
stages; the repository-wide commands are the final acceptance gate.
|
|
||||||
|
|
||||||
## Expected Compatibility Effects
|
|
||||||
|
|
||||||
- Reordered prompt content changes prompt fingerprints and may intentionally
|
|
||||||
cause one-time cold misses for affected development checkpoints.
|
|
||||||
- Prompt IDs, prompt versions, schema identities, checkpoint format, durable
|
|
||||||
artifacts, and public behavior do not change.
|
|
||||||
- Backend cache reuse remains provider-dependent, but Notarius supplies a
|
|
||||||
longer common extraction prefix that includes the transcript chunk.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
None. The message sequences, cache boundaries, test ownership, documentation
|
|
||||||
owners, and compatibility policy are specified above.
|
|
||||||
Reference in New Issue
Block a user