Retire completed roadmap documents and add ideas for future feature work
This commit is contained in:
@@ -1,90 +0,0 @@
|
||||
# D&D Prompt Cache Ordering
|
||||
|
||||
## Purpose
|
||||
|
||||
Arrange maintained D&D prompts so sibling extraction lanes share the longest
|
||||
useful byte-identical prefix, including the transcript chunk, and can therefore
|
||||
reuse provider-side prompt caches effectively.
|
||||
|
||||
This policy concerns rendered message order and cache boundaries. It does not
|
||||
change extraction semantics, artifact contracts, module inputs, durable output,
|
||||
or the responsibilities of deterministic validators and normalizers.
|
||||
|
||||
## Extraction Prompt Policy
|
||||
|
||||
Every D&D extraction prompt shares this rendered prefix:
|
||||
|
||||
1. the shared D&D system prompt;
|
||||
2. the shared D&D identity prompt;
|
||||
3. the shared campaign-reference prompt; and
|
||||
4. the shared transcript-chunk prompt.
|
||||
|
||||
The messages must render identically, including roles, content, and cache
|
||||
metadata, when two lanes receive the same transcript and common references.
|
||||
The campaign-reference and transcript messages establish ephemeral cache
|
||||
boundaries.
|
||||
|
||||
Only material used by every extraction lane belongs before the transcript.
|
||||
Shared wording that applies to only a subset of lanes is placed after the
|
||||
transcript rather than being added to unrelated prompts solely to lengthen the
|
||||
common prefix. Module instructions, optional generated references, catalogs,
|
||||
and other lane-specific context also follow the transcript.
|
||||
|
||||
The lane-specific suffixes are:
|
||||
|
||||
- NPCs: shared extraction-evidence policy, task, then instructions.
|
||||
- Item events: shared extraction-evidence policy, task, then instructions.
|
||||
- Scene descriptions: task, then instructions.
|
||||
- Combat turns: shared extraction-evidence policy, NPC registry, task, then
|
||||
instructions.
|
||||
- NPC interactions: shared extraction-evidence policy, NPC registry, task,
|
||||
then instructions.
|
||||
- Spells: shared extraction-evidence policy, NPC registry, spell catalog, task,
|
||||
then instructions.
|
||||
|
||||
The final instructions message establishes an ephemeral cache boundary.
|
||||
Intermediate cache markers on the shared identity message or lane-specific
|
||||
reference messages are unnecessary.
|
||||
|
||||
## Purpose-Specific Prompt Families
|
||||
|
||||
The extraction prefix is a sibling-lane policy, not a universal ordering rule
|
||||
for every D&D LLM call.
|
||||
|
||||
Scene chunking has no sibling lane with which to share a transcript prefix. It
|
||||
orders its prompt as system, common campaign references, task, instructions,
|
||||
and full transcript. The campaign-reference and transcript messages establish
|
||||
ephemeral cache boundaries.
|
||||
|
||||
NPC normalization orders its prompt as system, task, instructions, candidate
|
||||
NPCs, and transcript windows. The instructions and transcript-window messages
|
||||
establish ephemeral cache boundaries. Candidate artifacts remain ahead of the
|
||||
evidence windows needed to evaluate them.
|
||||
|
||||
Future D&D prompt families should identify their actual reuse boundary rather
|
||||
than mechanically copying either exception or the extraction sequence.
|
||||
|
||||
## Compatibility And Observability
|
||||
|
||||
Existing prompt IDs, prompt versions, response schemas, and module contracts
|
||||
remain unchanged. Prompt-content fingerprints already make the reordered
|
||||
assets part of checkpoint identity, so old development checkpoints may become
|
||||
cold misses without a checkpoint-format migration.
|
||||
|
||||
Prompt-order verification should exercise rendered messages with unique input
|
||||
sentinels. Tests must protect roles, cache metadata, input isolation, and the
|
||||
shared-prefix invariant without requiring particular prose, words, or phrases
|
||||
to remain in prompt assets.
|
||||
|
||||
## Desired End State
|
||||
|
||||
- All six D&D extraction prompts render the same four-message prefix for
|
||||
equivalent common inputs.
|
||||
- The transcript chunk is included in that common prefix and is followed only
|
||||
by lane-specific material.
|
||||
- Cache-control hints identify useful prefix boundaries without redundant
|
||||
intermediate markers.
|
||||
- Scene chunking and NPC normalization retain orderings suited to their
|
||||
distinct inputs and reuse opportunities.
|
||||
- Internal documentation explains both the general provider-cache principle
|
||||
and the concrete D&D prompt-family policy.
|
||||
@@ -7,6 +7,37 @@ not as committed release dates.
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
### Combat Enemy Ledger
|
||||
|
||||
- Add a D&D artifact that identifies enemies faced during combat and supports
|
||||
an end-of-session encounter ledger.
|
||||
- Track each enemy's observed state using a small controlled vocabulary such as
|
||||
`active`, `killed`, `fled`, `captured`, or `incapacitated`, while preserving
|
||||
an explicit unresolved state when the transcript does not establish an
|
||||
outcome.
|
||||
- Preserve the evidence for enemy participation and state changes rather than
|
||||
inferring a terminal outcome from combat ending or an enemy disappearing
|
||||
from the conversation.
|
||||
- Define how repeated mentions, groups of unnamed enemies, summoned or allied
|
||||
creatures, and the same enemy appearing in multiple combats affect identity
|
||||
and ledger entries.
|
||||
- Evaluate whether the ledger should be extracted directly, derived from
|
||||
combat-turn artifacts, or use a sequential pipeline that consumes combat
|
||||
turns and the normalized NPC registry as grounding references.
|
||||
|
||||
### Location Extraction
|
||||
|
||||
- Add a D&D artifact for locations visited by the party or otherwise mentioned
|
||||
in the transcript.
|
||||
- Distinguish observed visits from references, plans, recalled places, and
|
||||
uncertain or inferred locations so a mention alone is not reported as a
|
||||
visit.
|
||||
- Preserve transcript evidence for each visit or mention and reconcile aliases,
|
||||
nested places, and repeated appearances without collapsing distinct
|
||||
locations that share a generic name.
|
||||
- Define how the location artifact should ground later narrative reports and
|
||||
whether future event artifacts should retain canonical location identities.
|
||||
|
||||
### Evaluate Spell Extraction And Normalization
|
||||
|
||||
- Evaluate ordinary extraction retries and the completed normalization path
|
||||
|
||||
@@ -1,257 +0,0 @@
|
||||
# D&D Prompt Cache Ordering Implementation Plan
|
||||
|
||||
## Summary
|
||||
|
||||
Implement the target state in
|
||||
[D&D Prompt Cache Ordering](dnd-prompt-cache-ordering.md): give all six D&D
|
||||
extraction prompts a byte-identical rendered prefix through the transcript
|
||||
chunk, retain purpose-specific ordering for scene chunking and NPC
|
||||
normalization, consolidate tests around durable prompt-order invariants, and
|
||||
replace the current transcript-last documentation.
|
||||
|
||||
This is a prompt-asset and documentation change. Do not change Go module APIs,
|
||||
LLM request contracts, prompt IDs or versions, response schemas, artifact
|
||||
schemas, reference declarations, extraction behavior, retry behavior, or
|
||||
durable output. Existing prompt fingerprints must continue to provide
|
||||
checkpoint invalidation; no checkpoint migration or cleanup is required.
|
||||
|
||||
## Stage 1: Reorder The Six Extraction Prompts
|
||||
|
||||
Update these manifests:
|
||||
|
||||
- `internal/modules/dnd/extract/npcs/assets/prompts/dnd.npcs.yaml`
|
||||
- `internal/modules/dnd/extract/itemevents/assets/prompts/dnd.item_events.yaml`
|
||||
- `internal/modules/dnd/extract/scenedescriptions/assets/prompts/dnd.scene_descriptions.yaml`
|
||||
- `internal/modules/dnd/extract/combatturns/assets/prompts/dnd.combat_turns.yaml`
|
||||
- `internal/modules/dnd/extract/npcinteractions/assets/prompts/dnd.npc_interactions.yaml`
|
||||
- `internal/modules/dnd/extract/spells/assets/prompts/dnd.spells.yaml`
|
||||
|
||||
Give every manifest this exact leading message sequence:
|
||||
|
||||
1. `common-dnd-system.md`, with role `system` and no cache control;
|
||||
2. `common-dnd-identity.md`, with role `user` and no cache control;
|
||||
3. `common-dnd-references.md`, with role `user` and ephemeral cache control;
|
||||
4. `common-dnd-transcript.md`, with role `user` and ephemeral cache control.
|
||||
|
||||
The common prefix must contain no extraction-evidence policy, NPC registry,
|
||||
spell catalog, task, or module instructions. Those messages vary across lanes
|
||||
and would prevent the transcript from participating in a shared cache prefix.
|
||||
|
||||
Append the following lane-specific sequences after the transcript:
|
||||
|
||||
| Prompt | Post-transcript sequence |
|
||||
| --- | --- |
|
||||
| `dnd.npcs` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.item_events` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.scene_descriptions` | `task.md`, `instructions.md` |
|
||||
| `dnd.combat_turns` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.npc_interactions` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.spells` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `catalog.md`, `task.md`, `instructions.md` |
|
||||
|
||||
For every extraction prompt:
|
||||
|
||||
- apply ephemeral cache control to the final `instructions.md` message;
|
||||
- do not apply cache control to the extraction-evidence, NPC-registry, catalog,
|
||||
or task messages;
|
||||
- remove the existing cache control from `common-dnd-identity.md` and
|
||||
`common-dnd-npcs.md`; and
|
||||
- retain the existing inputs, requiredness, media types, output contract,
|
||||
profile, repair setting, prompt identity, and schema path.
|
||||
|
||||
Do not add the extraction-evidence asset to the scene-description prompt merely
|
||||
to make it resemble the other lanes. It is not universal shared context and
|
||||
therefore belongs outside the common-prefix contract.
|
||||
|
||||
### Stage 1 completion criteria
|
||||
|
||||
- Equivalent transcript, player, party, and glossary inputs render the same
|
||||
first four messages for all six extraction prompts, including identical
|
||||
roles, content, and cache metadata.
|
||||
- Each transcript input is rendered exactly once.
|
||||
- Every lane-specific message follows the transcript.
|
||||
- Prompt preparation and schema wiring remain successful for every lane.
|
||||
|
||||
## Stage 2: Apply Purpose-Specific Ordering To Other D&D LLM Prompts
|
||||
|
||||
Update
|
||||
`internal/modules/dnd/chunk/scenes/assets/prompts/dnd.scenes.yaml` to use:
|
||||
|
||||
1. `common-dnd-system.md`, without cache control;
|
||||
2. `common-dnd-references.md`, with ephemeral cache control;
|
||||
3. `task.md`, without cache control;
|
||||
4. `instructions.md`, without cache control; and
|
||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
||||
|
||||
The scene chunker consumes the full transcript once and has no sibling lanes
|
||||
that can share a transcript prefix. Stable task instructions therefore remain
|
||||
before the changing full transcript.
|
||||
|
||||
Update
|
||||
`internal/modules/dnd/normalize/npcs/assets/prompts/dnd.npcs.normalize.yaml` to
|
||||
use:
|
||||
|
||||
1. `common-dnd-system.md`, without cache control;
|
||||
2. `task.md`, without cache control;
|
||||
3. `instructions.md`, with ephemeral cache control;
|
||||
4. `candidates.md`, without cache control; and
|
||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
||||
|
||||
Remove the normalizer system message's current cache control. Preserve the
|
||||
candidate-before-evidence ordering: the transcript windows support evaluation
|
||||
of the candidate collection and are not a cross-lane transcript prefix.
|
||||
|
||||
Do not change input material construction, transcript windowing, prompt
|
||||
fragments, module implementations, or response handling in either package.
|
||||
|
||||
### Stage 2 completion criteria
|
||||
|
||||
- Scene chunking renders references separately before the task and the full
|
||||
transcript exactly once at the end.
|
||||
- NPC normalization renders candidates and transcript windows in separate
|
||||
messages, in that order, with the agreed cache boundaries.
|
||||
- Both prompts retain their current identities, schemas, inputs, and runtime
|
||||
behavior.
|
||||
|
||||
## Stage 3: Consolidate Prompt-Ordering Verification
|
||||
|
||||
Read and follow `docs/policy/testing.md` before modifying tests. The meaningful
|
||||
risk is loss of the shared rendered prefix or incorrect placement of
|
||||
request-specific material, not edits to prompt prose.
|
||||
|
||||
Add one family-level extraction invariant test in
|
||||
`internal/modules/dnd/register/prompt_cache_test.go` (or an equivalently scoped
|
||||
new register-package test file):
|
||||
|
||||
- register the complete D&D prompt asset family in a real in-memory
|
||||
`llm.AssetRegistry`;
|
||||
- construct a Scriptorium engine with a local, non-networked test profile;
|
||||
- prepare all six extraction prompts using their exported prompt IDs and
|
||||
versions;
|
||||
- supply the same unique transcript, players, party, and glossary sentinels to
|
||||
every prompt, plus only the lane-specific required inputs such as the NPC
|
||||
registry and spell catalog;
|
||||
- locate the one rendered message containing the transcript sentinel rather
|
||||
than locating it through prompt prose or a hard-coded phrase;
|
||||
- assert that the transcript sentinel occurs exactly once in each prepared
|
||||
prompt;
|
||||
- compare the rendered messages from the start through the transcript message
|
||||
across all six prompts, including role, full rendered content, and complete
|
||||
cache-control metadata;
|
||||
- assert that the shared prefix contains exactly the four agreed messages and
|
||||
that at least one lane-specific message follows it in every extraction
|
||||
prompt; and
|
||||
- assert that NPC-registry and spell-catalog sentinels, where supplied, occur
|
||||
only after the transcript.
|
||||
|
||||
Use this family-level test as the single owner of the cross-lane prefix policy.
|
||||
Do not duplicate the same prefix assertion in every extractor package.
|
||||
|
||||
Revise the existing prompt-asset tests in:
|
||||
|
||||
- `internal/modules/dnd/extract/npcs/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/itemevents/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/scenedescriptions/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/combatturns/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/npcinteractions/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/spells/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/chunk/scenes/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/normalize/npcs/scriptorium_assets_test.go`
|
||||
|
||||
Remove transcript-last expectations that contradict the new extraction policy.
|
||||
Keep package-local coverage only where it protects package-owned behavior, such
|
||||
as prompt/schema registration, declared shared assets, isolation and placement
|
||||
of package-specific input material, cache boundaries unique to the scene
|
||||
chunker or NPC normalizer, and non-disclosure properties. Consolidate or remove
|
||||
positional and prose-marker checks already covered by the family-level
|
||||
invariant.
|
||||
|
||||
Tests may search for unique input sentinels created by the test. They must not
|
||||
require the presence of specific words, phrases, sentences, or other prose in a
|
||||
prompt asset. Do not introduce golden snapshots of complete prompts or a test
|
||||
that treats harmless wording changes as failures.
|
||||
|
||||
### Stage 3 completion criteria
|
||||
|
||||
- One durable family-level test fails if any extraction lane moves
|
||||
lane-specific material ahead of the transcript or changes the common
|
||||
rendered prefix.
|
||||
- Local asset tests protect only nonredundant package behavior and the two
|
||||
purpose-specific prompt sequences.
|
||||
- The test suite contains no prompt-language change detector.
|
||||
- All prompt tests remain deterministic, offline, and independent of
|
||||
credentials.
|
||||
|
||||
## Stage 4: Update Canonical Internal Documentation
|
||||
|
||||
Update `docs/internal/llm.md` as the generic owner of prompt-cache mechanics:
|
||||
|
||||
- explain that backend reuse depends on identical preceding roles, rendered
|
||||
bytes, and cache metadata, not merely equivalent semantics;
|
||||
- state the general sibling-prompt ordering rule: universal shared context,
|
||||
request source material, then module-specific suffixes;
|
||||
- explain that a cache boundary should be placed at a useful reusable prefix
|
||||
and that redundant intermediate boundaries add no value;
|
||||
- retain the rule that prompt-family owners may choose a different sequence
|
||||
when their inputs and reuse pattern differ; and
|
||||
- replace the current statement that D&D extraction puts transcripts last with
|
||||
a link to the D&D-specific policy.
|
||||
|
||||
Update `docs/internal/dnd.md` as the concrete owner of maintained D&D prompt
|
||||
composition:
|
||||
|
||||
- document the four-message extraction prefix and its cache boundaries;
|
||||
- state that extraction-evidence policy, generated NPC registries, catalogs,
|
||||
tasks, and instructions follow the transcript because they are not universal
|
||||
across all extraction lanes;
|
||||
- summarize the scene-chunking and NPC-normalization exceptions and their
|
||||
rationale; and
|
||||
- retain existing rules about shared-asset reuse, canonical reference input,
|
||||
prompt fingerprints, and prompt behavior.
|
||||
|
||||
Do not update the README, configuration reference, architecture policy, CLI
|
||||
documentation, or integration contracts. Prompt message order is an internal
|
||||
runtime policy and has no user-visible configuration or wire-contract change.
|
||||
|
||||
### Stage 4 completion criteria
|
||||
|
||||
- No current-behavior document still says that D&D extraction transcripts are
|
||||
final messages.
|
||||
- Generic cache mechanics and D&D-specific ordering each have one canonical
|
||||
owner.
|
||||
- Documentation describes implemented behavior once the prompt changes land
|
||||
and does not duplicate volatile manifest inventories unnecessarily.
|
||||
|
||||
## Stage 5: Verification
|
||||
|
||||
Inspect representative prepared prompts to confirm that the test inputs produce
|
||||
the intended shared prefix and lane-specific suffixes. Then run:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
go test ./internal/modules/dnd/chunk/scenes
|
||||
go test ./internal/modules/dnd/extract/...
|
||||
go test ./internal/modules/dnd/normalize/npcs
|
||||
go test ./internal/modules/dnd/register
|
||||
go test ./internal/modules/dnd/...
|
||||
go test ./internal/modules/integration/...
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
The focused commands should be used while implementing their respective
|
||||
stages; the repository-wide commands are the final acceptance gate.
|
||||
|
||||
## Expected Compatibility Effects
|
||||
|
||||
- Reordered prompt content changes prompt fingerprints and may intentionally
|
||||
cause one-time cold misses for affected development checkpoints.
|
||||
- Prompt IDs, prompt versions, schema identities, checkpoint format, durable
|
||||
artifacts, and public behavior do not change.
|
||||
- Backend cache reuse remains provider-dependent, but Notarius supplies a
|
||||
longer common extraction prefix that includes the transcript chunk.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The message sequences, cache boundaries, test ownership, documentation
|
||||
owners, and compatibility policy are specified above.
|
||||
Reference in New Issue
Block a user