Add a new feature roadmap and implementation plan for D&D prompt ordering to improve LLM provider caching
This commit is contained in:
90
docs/roadmap/dnd-prompt-cache-ordering.md
Normal file
90
docs/roadmap/dnd-prompt-cache-ordering.md
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
# D&D Prompt Cache Ordering
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Arrange maintained D&D prompts so sibling extraction lanes share the longest
|
||||||
|
useful byte-identical prefix, including the transcript chunk, and can therefore
|
||||||
|
reuse provider-side prompt caches effectively.
|
||||||
|
|
||||||
|
This policy concerns rendered message order and cache boundaries. It does not
|
||||||
|
change extraction semantics, artifact contracts, module inputs, durable output,
|
||||||
|
or the responsibilities of deterministic validators and normalizers.
|
||||||
|
|
||||||
|
## Extraction Prompt Policy
|
||||||
|
|
||||||
|
Every D&D extraction prompt shares this rendered prefix:
|
||||||
|
|
||||||
|
1. the shared D&D system prompt;
|
||||||
|
2. the shared D&D identity prompt;
|
||||||
|
3. the shared campaign-reference prompt; and
|
||||||
|
4. the shared transcript-chunk prompt.
|
||||||
|
|
||||||
|
The messages must render identically, including roles, content, and cache
|
||||||
|
metadata, when two lanes receive the same transcript and common references.
|
||||||
|
The campaign-reference and transcript messages establish ephemeral cache
|
||||||
|
boundaries.
|
||||||
|
|
||||||
|
Only material used by every extraction lane belongs before the transcript.
|
||||||
|
Shared wording that applies to only a subset of lanes is placed after the
|
||||||
|
transcript rather than being added to unrelated prompts solely to lengthen the
|
||||||
|
common prefix. Module instructions, optional generated references, catalogs,
|
||||||
|
and other lane-specific context also follow the transcript.
|
||||||
|
|
||||||
|
The lane-specific suffixes are:
|
||||||
|
|
||||||
|
- NPCs: shared extraction-evidence policy, task, then instructions.
|
||||||
|
- Item events: shared extraction-evidence policy, task, then instructions.
|
||||||
|
- Scene descriptions: task, then instructions.
|
||||||
|
- Combat turns: shared extraction-evidence policy, NPC registry, task, then
|
||||||
|
instructions.
|
||||||
|
- NPC interactions: shared extraction-evidence policy, NPC registry, task,
|
||||||
|
then instructions.
|
||||||
|
- Spells: shared extraction-evidence policy, NPC registry, spell catalog, task,
|
||||||
|
then instructions.
|
||||||
|
|
||||||
|
The final instructions message establishes an ephemeral cache boundary.
|
||||||
|
Intermediate cache markers on the shared identity message or lane-specific
|
||||||
|
reference messages are unnecessary.
|
||||||
|
|
||||||
|
## Purpose-Specific Prompt Families
|
||||||
|
|
||||||
|
The extraction prefix is a sibling-lane policy, not a universal ordering rule
|
||||||
|
for every D&D LLM call.
|
||||||
|
|
||||||
|
Scene chunking has no sibling lane with which to share a transcript prefix. It
|
||||||
|
orders its prompt as system, common campaign references, task, instructions,
|
||||||
|
and full transcript. The campaign-reference and transcript messages establish
|
||||||
|
ephemeral cache boundaries.
|
||||||
|
|
||||||
|
NPC normalization orders its prompt as system, task, instructions, candidate
|
||||||
|
NPCs, and transcript windows. The instructions and transcript-window messages
|
||||||
|
establish ephemeral cache boundaries. Candidate artifacts remain ahead of the
|
||||||
|
evidence windows needed to evaluate them.
|
||||||
|
|
||||||
|
Future D&D prompt families should identify their actual reuse boundary rather
|
||||||
|
than mechanically copying either exception or the extraction sequence.
|
||||||
|
|
||||||
|
## Compatibility And Observability
|
||||||
|
|
||||||
|
Existing prompt IDs, prompt versions, response schemas, and module contracts
|
||||||
|
remain unchanged. Prompt-content fingerprints already make the reordered
|
||||||
|
assets part of checkpoint identity, so old development checkpoints may become
|
||||||
|
cold misses without a checkpoint-format migration.
|
||||||
|
|
||||||
|
Prompt-order verification should exercise rendered messages with unique input
|
||||||
|
sentinels. Tests must protect roles, cache metadata, input isolation, and the
|
||||||
|
shared-prefix invariant without requiring particular prose, words, or phrases
|
||||||
|
to remain in prompt assets.
|
||||||
|
|
||||||
|
## Desired End State
|
||||||
|
|
||||||
|
- All six D&D extraction prompts render the same four-message prefix for
|
||||||
|
equivalent common inputs.
|
||||||
|
- The transcript chunk is included in that common prefix and is followed only
|
||||||
|
by lane-specific material.
|
||||||
|
- Cache-control hints identify useful prefix boundaries without redundant
|
||||||
|
intermediate markers.
|
||||||
|
- Scene chunking and NPC normalization retain orderings suited to their
|
||||||
|
distinct inputs and reuse opportunities.
|
||||||
|
- Internal documentation explains both the general provider-cache principle
|
||||||
|
and the concrete D&D prompt-family policy.
|
||||||
@@ -1,226 +0,0 @@
|
|||||||
# Published Evidence Context
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Implemented.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Let downstream consumers build narrative reports from normalized artifacts
|
|
||||||
without separately parsing the original transcript or resolving source-unit
|
|
||||||
references themselves.
|
|
||||||
|
|
||||||
The production JSON output optionally publishes one deterministic, deduplicated
|
|
||||||
evidence-context artifact containing the transcript units relevant to explicitly
|
|
||||||
selected normalized lanes. Existing lane payloads remain the canonical semantic
|
|
||||||
results and retain their precise source references.
|
|
||||||
|
|
||||||
## Desired End State
|
|
||||||
|
|
||||||
When evidence-context publication is enabled, a consumer can:
|
|
||||||
|
|
||||||
1. discover one versioned evidence-context document through `index.json`;
|
|
||||||
2. obtain the union of source units needed to understand evidence cited by the
|
|
||||||
selected normalized lanes;
|
|
||||||
3. distinguish each artifact's direct evidence references from surrounding
|
|
||||||
units included only for narrative context;
|
|
||||||
4. retain speaker, timestamp, and other accepted source-unit metadata needed to
|
|
||||||
interpret the transcript; and
|
|
||||||
5. produce a narrative report without receiving duplicated transcript text in
|
|
||||||
every lane payload.
|
|
||||||
|
|
||||||
This is deterministic output projection. It does not invoke an LLM, change
|
|
||||||
normalization, or make surrounding context part of an artifact's evidence.
|
|
||||||
|
|
||||||
## Configuration Policy
|
|
||||||
|
|
||||||
Evidence publication is configured on the production JSON output module. The
|
|
||||||
intended configuration shape is:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
output:
|
|
||||||
module: json
|
|
||||||
options:
|
|
||||||
evidence_context:
|
|
||||||
enabled: true
|
|
||||||
window_units: 3
|
|
||||||
lanes:
|
|
||||||
- combat-turns
|
|
||||||
- item-events
|
|
||||||
- npc-interactions
|
|
||||||
- npcs
|
|
||||||
- spells
|
|
||||||
```
|
|
||||||
|
|
||||||
- Omitting `evidence_context` disables publication. When the object is present,
|
|
||||||
`enabled` is required.
|
|
||||||
- `enabled: false` accepts no `lanes` or `window_units` fields, preventing
|
|
||||||
silently ignored configuration.
|
|
||||||
- `lanes` is a required, non-empty allowlist of configured final lane IDs when
|
|
||||||
evidence publication is enabled. Values are trimmed, unique, and normalized
|
|
||||||
to lexical order.
|
|
||||||
- `window_units` is a non-negative integer and defaults to `3`. Zero publishes
|
|
||||||
only directly referenced units.
|
|
||||||
- Unknown lanes, duplicate lane IDs, and selected lanes whose artifact kind
|
|
||||||
cannot expose source evidence fail configuration resolution or pipeline
|
|
||||||
preparation.
|
|
||||||
- A selected lane that completes without a normalized output contributes no
|
|
||||||
evidence and does not make an otherwise successful run fail.
|
|
||||||
- Invocation-level lane filtering does not invalidate the configured allowlist.
|
|
||||||
Allowlisted lanes excluded from the effective run contribute nothing, while
|
|
||||||
the evidence document still records the configured allowlist.
|
|
||||||
|
|
||||||
The allowlist is intentional safety and stability policy. Scene descriptions
|
|
||||||
and other broad-range lanes are excluded unless named expressly. Adding a new
|
|
||||||
pipeline lane never silently increases output size or publishes more transcript
|
|
||||||
content.
|
|
||||||
|
|
||||||
## Evidence Collection Boundary
|
|
||||||
|
|
||||||
Evidence collection applies to accepted final normalized artifacts from the
|
|
||||||
selected lanes. It must not inspect arbitrary serialized JSON for fields named
|
|
||||||
`source_ref` or `source_refs`, and the generic JSON output module must not
|
|
||||||
depend on D&D artifact types.
|
|
||||||
|
|
||||||
Artifact-kind registrations expose their source references through an explicit
|
|
||||||
typed projection contract. The framework uses that contract to assemble a
|
|
||||||
domain-neutral evidence request containing:
|
|
||||||
|
|
||||||
- the accepted generic source document;
|
|
||||||
- the selected lane and artifact identities; and
|
|
||||||
- defensive copies of their direct source references.
|
|
||||||
|
|
||||||
The output stage owns publication of the resulting logical artifact. Generic
|
|
||||||
framework code owns range validation, position-based expansion, and union
|
|
||||||
logic. Domain-specific adapters own only the extraction of evidence references
|
|
||||||
from their typed artifacts.
|
|
||||||
|
|
||||||
Both plural-reference artifacts and singular-reference artifacts, such as
|
|
||||||
scene descriptions, can participate through the same projection contract.
|
|
||||||
They do so only when their configured lane is allowlisted.
|
|
||||||
|
|
||||||
## Range Expansion And Deduplication
|
|
||||||
|
|
||||||
For every valid direct source reference:
|
|
||||||
|
|
||||||
1. resolve its endpoints through source-document positions, not numeric
|
|
||||||
unit-ID arithmetic;
|
|
||||||
2. expand the range by `window_units` positions on each side;
|
|
||||||
3. clip the expanded range at document boundaries; and
|
|
||||||
4. union overlapping or contiguous expanded ranges.
|
|
||||||
|
|
||||||
Published contexts and units remain in source-document order. Each source unit
|
|
||||||
appears at most once in a merged context. Original direct references remain
|
|
||||||
unchanged and are associated with their contributing lane IDs so consumers can
|
|
||||||
tell why a context was included.
|
|
||||||
|
|
||||||
The projector must not silently omit or repair an invalid reference that
|
|
||||||
reaches this boundary. Such a value violates the accepted normalized-artifact
|
|
||||||
contract and causes output projection to fail with a content-safe error.
|
|
||||||
|
|
||||||
No implicit coverage limit truncates selected evidence. If the allowlisted
|
|
||||||
lanes collectively cite most or all of a transcript, the evidence document may
|
|
||||||
contain most or all of it. The explicit lane allowlist is the control that
|
|
||||||
prevents a broad lane such as scene descriptions from doing so accidentally.
|
|
||||||
|
|
||||||
## Durable Evidence Artifact
|
|
||||||
|
|
||||||
The JSON bundle gains one optional, non-lane artifact with these durable
|
|
||||||
identities:
|
|
||||||
|
|
||||||
| Property | Value |
|
|
||||||
| --- | --- |
|
|
||||||
| Logical file | `evidence-context.json` |
|
|
||||||
| Index descriptor | `evidence_context` |
|
|
||||||
| Artifact kind | `source/evidence-context` |
|
|
||||||
| Media type | `application/json` |
|
|
||||||
| Schema ID | `notarius.source.evidence_context` |
|
|
||||||
| Schema name | `notarius_source_evidence_context_v1` |
|
|
||||||
| Schema version | `v1` |
|
|
||||||
|
|
||||||
The descriptor in `index.json` carries the artifact and schema identities,
|
|
||||||
analogous to the existing chunk-map descriptor. The artifact is present
|
|
||||||
whenever evidence publication is enabled, including when its context collection
|
|
||||||
is empty.
|
|
||||||
|
|
||||||
The document contains:
|
|
||||||
|
|
||||||
- the source document ID and semantic digest;
|
|
||||||
- the effective window size;
|
|
||||||
- the sorted configured lane allowlist;
|
|
||||||
- an ordered context collection;
|
|
||||||
- each context's expanded start and end unit IDs;
|
|
||||||
- the original direct references and contributing lane IDs covered by that
|
|
||||||
context; and
|
|
||||||
- the ordered accepted source units, including unit ID, kind, text,
|
|
||||||
self-reference, and metadata.
|
|
||||||
|
|
||||||
Expanded context bounds are navigation aids, not citations. The original
|
|
||||||
references embedded in each context remain the authoritative direct evidence.
|
|
||||||
The evidence artifact is discovered separately from lane payloads and does not
|
|
||||||
increase the normalized-lane count reported by the runner or subprocess
|
|
||||||
receipt.
|
|
||||||
|
|
||||||
## Failure And Publication Semantics
|
|
||||||
|
|
||||||
- Evidence projection occurs only after selected normalized outputs are known
|
|
||||||
and before the output encoder returns its logical files.
|
|
||||||
- Projection or encoding failure is an output-stage framework error; the CLI
|
|
||||||
does not publish a partially assembled output bundle.
|
|
||||||
- Rejected or absent lane outputs contribute nothing. Their attempted values
|
|
||||||
and source references must not be published through this artifact.
|
|
||||||
- Context generation is deterministic for the same source document, selected
|
|
||||||
normalized outputs, lane allowlist, and window size.
|
|
||||||
- Existing output, checkpoint, warning, rejection, debug, and subprocess
|
|
||||||
success semantics remain unchanged.
|
|
||||||
|
|
||||||
## Sensitivity And Size
|
|
||||||
|
|
||||||
Unlike the current chunk map, the evidence artifact contains transcript text
|
|
||||||
and source-unit metadata. Enabling it therefore creates additional durable
|
|
||||||
sensitive data and may materially increase bundle size.
|
|
||||||
|
|
||||||
The implemented configuration, operations, integration, and consumer documents
|
|
||||||
state that:
|
|
||||||
|
|
||||||
- evidence publication is opt-in;
|
|
||||||
- output permissions and retention must be appropriate for source content;
|
|
||||||
- selecting broad or numerous lanes can publish most of the transcript; and
|
|
||||||
- the artifact must not contain raw input bytes, LLM prompts or responses,
|
|
||||||
auxiliary reference content, credentials, debug-only data, or filesystem
|
|
||||||
paths.
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
|
|
||||||
- Evidence publication is disabled by default and leaves existing bundles
|
|
||||||
unchanged.
|
|
||||||
- Enabling it requires an explicit non-empty lane allowlist.
|
|
||||||
- References from all selected successful lanes contribute to one deduplicated
|
|
||||||
document.
|
|
||||||
- Non-monotonic unit IDs are expanded and ordered correctly by document
|
|
||||||
position.
|
|
||||||
- Overlapping windows share one ordered copy of each included source unit.
|
|
||||||
- Direct references remain distinguishable from added context.
|
|
||||||
- Scene descriptions cannot contribute unless their lane is explicitly
|
|
||||||
allowlisted.
|
|
||||||
- Invalid selected lanes and unsupported artifact kinds fail before execution;
|
|
||||||
invalid accepted references fail output projection rather than being ignored.
|
|
||||||
- Empty selected-lane results produce a valid empty evidence artifact.
|
|
||||||
- Existing D&D lane schemas, normalized-output counts, and source-reference
|
|
||||||
semantics do not change.
|
|
||||||
- Generic framework and output packages do not depend on D&D types or parse
|
|
||||||
artifact JSON heuristically.
|
|
||||||
- The published contract and operational documentation clearly describe source
|
|
||||||
sensitivity, discovery, compatibility, and retention.
|
|
||||||
|
|
||||||
## Out Of Scope
|
|
||||||
|
|
||||||
- Embedding transcript units directly into each D&D record or lane payload.
|
|
||||||
- Replacing precise source references with expanded context ranges.
|
|
||||||
- Automatically including every configured lane.
|
|
||||||
- An explicit full-transcript publication mode.
|
|
||||||
- LLM summarization, retrieval, ranking, or narrative generation.
|
|
||||||
- Per-record window sizes or lane-specific window sizes.
|
|
||||||
- CLI overrides for evidence configuration.
|
|
||||||
- Reading rejected attempts, debug artifacts, auxiliary references, or prior
|
|
||||||
output bundles as evidence sources.
|
|
||||||
@@ -60,6 +60,30 @@ safety checks, and deterministic application of accepted changes.
|
|||||||
- Add media-type validators when non-JSON artifact representations are
|
- Add media-type validators when non-JSON artifact representations are
|
||||||
introduced.
|
introduced.
|
||||||
|
|
||||||
|
## LLM Runtime Evolution
|
||||||
|
|
||||||
|
### Native Session Propagation
|
||||||
|
|
||||||
|
- Once upstream Scriptorium exposes a session identifier on its run request,
|
||||||
|
propagate the existing `StructuredCompletionRequest.SessionID` through the
|
||||||
|
Scriptorium adapter's native session field.
|
||||||
|
- Preserve the current `--session-id` invocation contract and its run-wide
|
||||||
|
propagation to every prompt-facing module and validator. Do not introduce a
|
||||||
|
second session configuration surface.
|
||||||
|
- Retain session identity in checkpoint provenance so runs with different
|
||||||
|
sessions cannot reuse one another's LLM-derived checkpoints.
|
||||||
|
- Define the upstream compatibility and prompt-variable transition explicitly:
|
||||||
|
native provider session behavior must not silently remove a `session_id`
|
||||||
|
prompt variable while maintained prompts still consume it.
|
||||||
|
- Add adapter and assembled-run coverage for exact forwarding, trimming,
|
||||||
|
concurrent-run isolation, and unsupported-provider behavior once the
|
||||||
|
upstream contract is available.
|
||||||
|
|
||||||
|
This work is blocked on native session support in the upstream Scriptorium
|
||||||
|
package. Notarius already carries a run-scoped session ID through its CLI,
|
||||||
|
pipeline requests, checkpoint identity, and prompt variables; the missing
|
||||||
|
capability is native propagation across the LLM adapter boundary.
|
||||||
|
|
||||||
## Further Reference Evolution
|
## Further Reference Evolution
|
||||||
|
|
||||||
- Make prior-run artifacts easier to bind as references without changing the
|
- Make prior-run artifacts easier to bind as references without changing the
|
||||||
|
|||||||
@@ -1,486 +1,257 @@
|
|||||||
# Published Evidence Context Implementation Plan
|
# D&D Prompt Cache Ordering Implementation Plan
|
||||||
|
|
||||||
## Status
|
## Summary
|
||||||
|
|
||||||
Completed.
|
Implement the target state in
|
||||||
|
[D&D Prompt Cache Ordering](dnd-prompt-cache-ordering.md): give all six D&D
|
||||||
|
extraction prompts a byte-identical rendered prefix through the transcript
|
||||||
|
chunk, retain purpose-specific ordering for scene chunking and NPC
|
||||||
|
normalization, consolidate tests around durable prompt-order invariants, and
|
||||||
|
replace the current transcript-last documentation.
|
||||||
|
|
||||||
## Objective
|
This is a prompt-asset and documentation change. Do not change Go module APIs,
|
||||||
|
LLM request contracts, prompt IDs or versions, response schemas, artifact
|
||||||
|
schemas, reference declarations, extraction behavior, retry behavior, or
|
||||||
|
durable output. Existing prompt fingerprints must continue to provide
|
||||||
|
checkpoint invalidation; no checkpoint migration or cleanup is required.
|
||||||
|
|
||||||
Implement the accepted [Published Evidence Context](evidence.md) roadmap as an
|
## Stage 1: Reorder The Six Extraction Prompts
|
||||||
optional, deterministic extension of the production JSON output. The completed
|
|
||||||
work must publish one deduplicated source-context artifact for an explicit set
|
|
||||||
of successful normalized lanes while leaving existing lane payloads,
|
|
||||||
normalization, checkpointing, subprocess results, and disabled output bundles
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
Complete the stages below in order. Each stage must leave its affected packages
|
Update these manifests:
|
||||||
passing before the next begins. Do not implement per-record hydration, implicit
|
|
||||||
all-lane collection, a full-transcript mode, LLM processing, or any other
|
|
||||||
roadmap item marked out of scope.
|
|
||||||
|
|
||||||
## Decisions And Invariants
|
- `internal/modules/dnd/extract/npcs/assets/prompts/dnd.npcs.yaml`
|
||||||
|
- `internal/modules/dnd/extract/itemevents/assets/prompts/dnd.item_events.yaml`
|
||||||
|
- `internal/modules/dnd/extract/scenedescriptions/assets/prompts/dnd.scene_descriptions.yaml`
|
||||||
|
- `internal/modules/dnd/extract/combatturns/assets/prompts/dnd.combat_turns.yaml`
|
||||||
|
- `internal/modules/dnd/extract/npcinteractions/assets/prompts/dnd.npc_interactions.yaml`
|
||||||
|
- `internal/modules/dnd/extract/spells/assets/prompts/dnd.spells.yaml`
|
||||||
|
|
||||||
- Evidence publication is output policy. Normalizers continue to return
|
Give every manifest this exact leading message sequence:
|
||||||
semantic artifacts with precise source references and do not receive
|
|
||||||
hydration responsibilities.
|
|
||||||
- The framework operates on the accepted generic `source.SourceDocument` and
|
|
||||||
typed artifact projections. It must not inspect serialized JSON for
|
|
||||||
`source_ref` or `source_refs`, and generic packages must not depend on D&D
|
|
||||||
types.
|
|
||||||
- The selected lane allowlist is explicit, non-empty, and globally addressed by
|
|
||||||
resolved lane ID. Pipeline resolution already guarantees lane IDs are unique
|
|
||||||
across ordered steps.
|
|
||||||
- The framework decodes accepted serialized normalize outputs through their
|
|
||||||
registered artifact codecs before invoking typed evidence projectors. This
|
|
||||||
supports both fresh and checkpoint-reused normalize outputs without retaining
|
|
||||||
a second typed result channel.
|
|
||||||
- Expansion uses source-document positions. Numeric unit IDs are identities,
|
|
||||||
not sequence numbers.
|
|
||||||
- Direct source references are never widened or rewritten. Expanded ranges are
|
|
||||||
context bounds only.
|
|
||||||
- Rejected, failed, and absent normalized lane outputs contribute no evidence.
|
|
||||||
- Evidence output is sensitive durable source content, not cache or debug
|
|
||||||
state. It contains accepted source units only and never raw input bytes,
|
|
||||||
prompts, model responses, auxiliary references, paths, or credentials.
|
|
||||||
- The optional `evidence_context` index field is an additive v1 JSON-bundle
|
|
||||||
change. Existing D&D artifact schemas and the subprocess receipt do not
|
|
||||||
change.
|
|
||||||
|
|
||||||
## Stage 1: Add Typed Evidence Capability And Resolve Output Policy
|
1. `common-dnd-system.md`, with role `system` and no cache control;
|
||||||
|
2. `common-dnd-identity.md`, with role `user` and no cache control;
|
||||||
|
3. `common-dnd-references.md`, with role `user` and ephemeral cache control;
|
||||||
|
4. `common-dnd-transcript.md`, with role `user` and ephemeral cache control.
|
||||||
|
|
||||||
Add a dedicated artifact-evidence registry under the pipeline framework:
|
The common prefix must contain no extraction-evidence policy, NPC registry,
|
||||||
|
spell catalog, task, or module instructions. Those messages vary across lanes
|
||||||
|
and would prevent the transcript from participating in a shared cache prefix.
|
||||||
|
|
||||||
- `pipeline.ArtifactEvidenceRegistry` stores one typed projector per artifact
|
Append the following lane-specific sequences after the transcript:
|
||||||
kind.
|
|
||||||
- `pipeline.ArtifactEvidenceProjector[T]` is
|
|
||||||
`func(T) []source.SourceRef`.
|
|
||||||
- `pipeline.RegisterArtifactEvidence[T](registry, kind, projector)` accepts a
|
|
||||||
non-empty kind and non-nil projector, records the exact Go type for `T`, and
|
|
||||||
rejects duplicate kinds.
|
|
||||||
- The erased projection boundary checks the exact registered type, invokes the
|
|
||||||
projector, and returns a defensive copy of its references.
|
|
||||||
- The registry exposes only the discovery and projection operations required by
|
|
||||||
resolution, preparation, and execution; do not expose its mutable entries.
|
|
||||||
|
|
||||||
Add the registry to `pipeline.Registries` and `pipeline.ModuleCatalog`, including
|
| Prompt | Post-transcript sequence |
|
||||||
CLI catalog conversion, production construction, empty-set detection, and
|
| --- | --- |
|
||||||
test registry helpers. A nil evidence registry remains valid when evidence
|
| `dnd.npcs` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
||||||
publication is disabled. Production construction and the D&D registrar require
|
| `dnd.item_events` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
||||||
and populate it.
|
| `dnd.scene_descriptions` | `task.md`, `instructions.md` |
|
||||||
|
| `dnd.combat_turns` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
||||||
|
| `dnd.npc_interactions` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
||||||
|
| `dnd.spells` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `catalog.md`, `task.md`, `instructions.md` |
|
||||||
|
|
||||||
Define these framework-level output-policy contracts:
|
For every extraction prompt:
|
||||||
|
|
||||||
```go
|
- apply ephemeral cache control to the final `instructions.md` message;
|
||||||
type EvidenceContextPolicy struct {
|
- do not apply cache control to the extraction-evidence, NPC-registry, catalog,
|
||||||
Enabled bool
|
or task messages;
|
||||||
WindowUnits int
|
- remove the existing cache control from `common-dnd-identity.md` and
|
||||||
LaneIDs []string
|
`common-dnd-npcs.md`; and
|
||||||
}
|
- retain the existing inputs, requiredness, media types, output contract,
|
||||||
|
profile, repair setting, prompt identity, and schema path.
|
||||||
|
|
||||||
type EvidenceContextPolicyProvider interface {
|
Do not add the extraction-evidence asset to the scene-description prompt merely
|
||||||
EvidenceContextPolicy() EvidenceContextPolicy
|
to make it resemble the other lanes. It is not universal shared context and
|
||||||
}
|
therefore belongs outside the common-prefix contract.
|
||||||
|
|
||||||
|
### Stage 1 completion criteria
|
||||||
|
|
||||||
|
- Equivalent transcript, player, party, and glossary inputs render the same
|
||||||
|
first four messages for all six extraction prompts, including identical
|
||||||
|
roles, content, and cache metadata.
|
||||||
|
- Each transcript input is rendered exactly once.
|
||||||
|
- Every lane-specific message follows the transcript.
|
||||||
|
- Prompt preparation and schema wiring remain successful for every lane.
|
||||||
|
|
||||||
|
## Stage 2: Apply Purpose-Specific Ordering To Other D&D LLM Prompts
|
||||||
|
|
||||||
|
Update
|
||||||
|
`internal/modules/dnd/chunk/scenes/assets/prompts/dnd.scenes.yaml` to use:
|
||||||
|
|
||||||
|
1. `common-dnd-system.md`, without cache control;
|
||||||
|
2. `common-dnd-references.md`, with ephemeral cache control;
|
||||||
|
3. `task.md`, without cache control;
|
||||||
|
4. `instructions.md`, without cache control; and
|
||||||
|
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
||||||
|
|
||||||
|
The scene chunker consumes the full transcript once and has no sibling lanes
|
||||||
|
that can share a transcript prefix. Stable task instructions therefore remain
|
||||||
|
before the changing full transcript.
|
||||||
|
|
||||||
|
Update
|
||||||
|
`internal/modules/dnd/normalize/npcs/assets/prompts/dnd.npcs.normalize.yaml` to
|
||||||
|
use:
|
||||||
|
|
||||||
|
1. `common-dnd-system.md`, without cache control;
|
||||||
|
2. `task.md`, without cache control;
|
||||||
|
3. `instructions.md`, with ephemeral cache control;
|
||||||
|
4. `candidates.md`, without cache control; and
|
||||||
|
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
||||||
|
|
||||||
|
Remove the normalizer system message's current cache control. Preserve the
|
||||||
|
candidate-before-evidence ordering: the transcript windows support evaluation
|
||||||
|
of the candidate collection and are not a cross-lane transcript prefix.
|
||||||
|
|
||||||
|
Do not change input material construction, transcript windowing, prompt
|
||||||
|
fragments, module implementations, or response handling in either package.
|
||||||
|
|
||||||
|
### Stage 2 completion criteria
|
||||||
|
|
||||||
|
- Scene chunking renders references separately before the task and the full
|
||||||
|
transcript exactly once at the end.
|
||||||
|
- NPC normalization renders candidates and transcript windows in separate
|
||||||
|
messages, in that order, with the agreed cache boundaries.
|
||||||
|
- Both prompts retain their current identities, schemas, inputs, and runtime
|
||||||
|
behavior.
|
||||||
|
|
||||||
|
## Stage 3: Consolidate Prompt-Ordering Verification
|
||||||
|
|
||||||
|
Read and follow `docs/policy/testing.md` before modifying tests. The meaningful
|
||||||
|
risk is loss of the shared rendered prefix or incorrect placement of
|
||||||
|
request-specific material, not edits to prompt prose.
|
||||||
|
|
||||||
|
Add one family-level extraction invariant test in
|
||||||
|
`internal/modules/dnd/register/prompt_cache_test.go` (or an equivalently scoped
|
||||||
|
new register-package test file):
|
||||||
|
|
||||||
|
- register the complete D&D prompt asset family in a real in-memory
|
||||||
|
`llm.AssetRegistry`;
|
||||||
|
- construct a Scriptorium engine with a local, non-networked test profile;
|
||||||
|
- prepare all six extraction prompts using their exported prompt IDs and
|
||||||
|
versions;
|
||||||
|
- supply the same unique transcript, players, party, and glossary sentinels to
|
||||||
|
every prompt, plus only the lane-specific required inputs such as the NPC
|
||||||
|
registry and spell catalog;
|
||||||
|
- locate the one rendered message containing the transcript sentinel rather
|
||||||
|
than locating it through prompt prose or a hard-coded phrase;
|
||||||
|
- assert that the transcript sentinel occurs exactly once in each prepared
|
||||||
|
prompt;
|
||||||
|
- compare the rendered messages from the start through the transcript message
|
||||||
|
across all six prompts, including role, full rendered content, and complete
|
||||||
|
cache-control metadata;
|
||||||
|
- assert that the shared prefix contains exactly the four agreed messages and
|
||||||
|
that at least one lane-specific message follows it in every extraction
|
||||||
|
prompt; and
|
||||||
|
- assert that NPC-registry and spell-catalog sentinels, where supplied, occur
|
||||||
|
only after the transcript.
|
||||||
|
|
||||||
|
Use this family-level test as the single owner of the cross-lane prefix policy.
|
||||||
|
Do not duplicate the same prefix assertion in every extractor package.
|
||||||
|
|
||||||
|
Revise the existing prompt-asset tests in:
|
||||||
|
|
||||||
|
- `internal/modules/dnd/extract/npcs/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/extract/itemevents/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/extract/scenedescriptions/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/extract/combatturns/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/extract/npcinteractions/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/extract/spells/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/chunk/scenes/scriptorium_assets_test.go`
|
||||||
|
- `internal/modules/dnd/normalize/npcs/scriptorium_assets_test.go`
|
||||||
|
|
||||||
|
Remove transcript-last expectations that contradict the new extraction policy.
|
||||||
|
Keep package-local coverage only where it protects package-owned behavior, such
|
||||||
|
as prompt/schema registration, declared shared assets, isolation and placement
|
||||||
|
of package-specific input material, cache boundaries unique to the scene
|
||||||
|
chunker or NPC normalizer, and non-disclosure properties. Consolidate or remove
|
||||||
|
positional and prose-marker checks already covered by the family-level
|
||||||
|
invariant.
|
||||||
|
|
||||||
|
Tests may search for unique input sentinels created by the test. They must not
|
||||||
|
require the presence of specific words, phrases, sentences, or other prose in a
|
||||||
|
prompt asset. Do not introduce golden snapshots of complete prompts or a test
|
||||||
|
that treats harmless wording changes as failures.
|
||||||
|
|
||||||
|
### Stage 3 completion criteria
|
||||||
|
|
||||||
|
- One durable family-level test fails if any extraction lane moves
|
||||||
|
lane-specific material ahead of the transcript or changes the common
|
||||||
|
rendered prefix.
|
||||||
|
- Local asset tests protect only nonredundant package behavior and the two
|
||||||
|
purpose-specific prompt sequences.
|
||||||
|
- The test suite contains no prompt-language change detector.
|
||||||
|
- All prompt tests remain deterministic, offline, and independent of
|
||||||
|
credentials.
|
||||||
|
|
||||||
|
## Stage 4: Update Canonical Internal Documentation
|
||||||
|
|
||||||
|
Update `docs/internal/llm.md` as the generic owner of prompt-cache mechanics:
|
||||||
|
|
||||||
|
- explain that backend reuse depends on identical preceding roles, rendered
|
||||||
|
bytes, and cache metadata, not merely equivalent semantics;
|
||||||
|
- state the general sibling-prompt ordering rule: universal shared context,
|
||||||
|
request source material, then module-specific suffixes;
|
||||||
|
- explain that a cache boundary should be placed at a useful reusable prefix
|
||||||
|
and that redundant intermediate boundaries add no value;
|
||||||
|
- retain the rule that prompt-family owners may choose a different sequence
|
||||||
|
when their inputs and reuse pattern differ; and
|
||||||
|
- replace the current statement that D&D extraction puts transcripts last with
|
||||||
|
a link to the D&D-specific policy.
|
||||||
|
|
||||||
|
Update `docs/internal/dnd.md` as the concrete owner of maintained D&D prompt
|
||||||
|
composition:
|
||||||
|
|
||||||
|
- document the four-message extraction prefix and its cache boundaries;
|
||||||
|
- state that extraction-evidence policy, generated NPC registries, catalogs,
|
||||||
|
tasks, and instructions follow the transcript because they are not universal
|
||||||
|
across all extraction lanes;
|
||||||
|
- summarize the scene-chunking and NPC-normalization exceptions and their
|
||||||
|
rationale; and
|
||||||
|
- retain existing rules about shared-asset reuse, canonical reference input,
|
||||||
|
prompt fingerprints, and prompt behavior.
|
||||||
|
|
||||||
|
Do not update the README, configuration reference, architecture policy, CLI
|
||||||
|
documentation, or integration contracts. Prompt message order is an internal
|
||||||
|
runtime policy and has no user-visible configuration or wire-contract change.
|
||||||
|
|
||||||
|
### Stage 4 completion criteria
|
||||||
|
|
||||||
|
- No current-behavior document still says that D&D extraction transcripts are
|
||||||
|
final messages.
|
||||||
|
- Generic cache mechanics and D&D-specific ordering each have one canonical
|
||||||
|
owner.
|
||||||
|
- Documentation describes implemented behavior once the prompt changes land
|
||||||
|
and does not duplicate volatile manifest inventories unnecessarily.
|
||||||
|
|
||||||
|
## Stage 5: Verification
|
||||||
|
|
||||||
|
Inspect representative prepared prompts to confirm that the test inputs produce
|
||||||
|
the intended shared prefix and lane-specific suffixes. Then run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git diff --check
|
||||||
|
go test ./internal/modules/dnd/chunk/scenes
|
||||||
|
go test ./internal/modules/dnd/extract/...
|
||||||
|
go test ./internal/modules/dnd/normalize/npcs
|
||||||
|
go test ./internal/modules/dnd/register
|
||||||
|
go test ./internal/modules/dnd/...
|
||||||
|
go test ./internal/modules/integration/...
|
||||||
|
go test ./...
|
||||||
|
go vet ./...
|
||||||
|
go build ./cmd/notarius
|
||||||
```
|
```
|
||||||
|
|
||||||
Provider and prepared-pipeline boundaries defensively copy `LaneIDs`.
|
The focused commands should be used while implementing their respective
|
||||||
|
stages; the repository-wide commands are the final acceptance gate.
|
||||||
|
|
||||||
Add an optional output-profile option-validation callback to
|
## Expected Compatibility Effects
|
||||||
`OutputEncoderRegistry`:
|
|
||||||
|
|
||||||
```go
|
- Reordered prompt content changes prompt fingerprints and may intentionally
|
||||||
type OutputProfileOptionContext struct {
|
cause one-time cold misses for affected development checkpoints.
|
||||||
LaneIDs []string
|
- Prompt IDs, prompt versions, schema identities, checkpoint format, durable
|
||||||
}
|
artifacts, and public behavior do not change.
|
||||||
|
- Backend cache reuse remains provider-dependent, but Notarius supplies a
|
||||||
type OutputProfileOptionValidator func(
|
longer common extraction prefix that includes the transcript chunk.
|
||||||
OutputProfileOptionContext,
|
|
||||||
map[string]any,
|
|
||||||
) error
|
|
||||||
```
|
|
||||||
|
|
||||||
Add `RegisterBuilderWithProfileValidation(spec, validateOptions,
|
|
||||||
validateProfile, builder)` and make existing output registration methods
|
|
||||||
delegate to it with no profile callback. The registry passes defensive copies
|
|
||||||
to validation. The callback receives the complete configured lane-ID set before
|
|
||||||
invocation-level `--only` filtering. The production JSON output uses it only to
|
|
||||||
prove that every configured evidence lane exists. Keep the extension generic:
|
|
||||||
the pipeline supplies lane identities, while the output module interprets its
|
|
||||||
own options. Build that lane set from the normalized legacy-or-steps profile
|
|
||||||
before selection and reject duplicate configured lane IDs through the existing
|
|
||||||
pipeline identity rules.
|
|
||||||
|
|
||||||
Extend the production JSON output options with the nested
|
|
||||||
`evidence_context` object:
|
|
||||||
|
|
||||||
- omission disables the feature;
|
|
||||||
- `enabled` is required when the object is present;
|
|
||||||
- `enabled: false` permits no `lanes` or `window_units` fields;
|
|
||||||
- `enabled: true` requires a non-empty `lanes` array;
|
|
||||||
- lane values are strings, trimmed, non-empty, unique after trimming, and
|
|
||||||
normalized to lexical order;
|
|
||||||
- `window_units` is an optional non-negative integer with default `3`; and
|
|
||||||
- outer and nested unknown fields and incompatible YAML value types remain
|
|
||||||
strict configuration errors.
|
|
||||||
|
|
||||||
The JSON encoder implements the policy provider from its decoded immutable
|
|
||||||
options. Pipeline resolution invokes its profile validator against all
|
|
||||||
configured steps, so an unknown evidence lane fails even when another lane is
|
|
||||||
selected with `--only`.
|
|
||||||
|
|
||||||
During `pipeline.Prepare`, after constructing the output encoder:
|
|
||||||
|
|
||||||
1. obtain and defensively normalize an enabled policy;
|
|
||||||
2. intersect its configured IDs with the effective prepared lanes, treating
|
|
||||||
allowlisted lanes removed by invocation-level filtering as inactive;
|
|
||||||
3. require an artifact-evidence registration for each active lane kind;
|
|
||||||
4. prove that its projector Go type exactly matches the active lane's registered
|
|
||||||
artifact codec type; and
|
|
||||||
5. retain an immutable private evidence plan on `PreparedPipeline`.
|
|
||||||
|
|
||||||
Duplicate or empty provider values, a missing evidence registry for an active
|
|
||||||
lane, unsupported active artifact kinds, and type mismatches fail preparation
|
|
||||||
with pipeline/output/lane context. A disabled or non-participating output
|
|
||||||
encoder creates no evidence plan and preserves existing preparation behavior.
|
|
||||||
The private plan retains both the full configured allowlist for publication and
|
|
||||||
the active lane/projector intersection for execution.
|
|
||||||
|
|
||||||
Register D&D evidence projectors for all six current artifact kinds. Each
|
|
||||||
projector returns copies of the artifact's direct references in record order:
|
|
||||||
spells, NPCs, combat turns, item events, NPC interactions, and the singular
|
|
||||||
reference from each scene description. Scene descriptions gain capability but
|
|
||||||
remain excluded unless their configured lane ID is allowlisted.
|
|
||||||
|
|
||||||
Stage tests:
|
|
||||||
|
|
||||||
- Registry tests cover nil, blank, duplicate, exact-type, defensive-copy, and
|
|
||||||
deterministic discovery behavior.
|
|
||||||
- JSON option tests cover disabled, enabled/default-window, explicit zero
|
|
||||||
window, normalization, duplicates, unknown fields, and invalid types.
|
|
||||||
- Preparation tests cover selected lanes across steps, unknown lanes,
|
|
||||||
unsupported kinds, projector/codec type mismatch, disabled behavior, and
|
|
||||||
defensive policy ownership.
|
|
||||||
- Resolution/preparation tests prove a valid allowlist survives `--only`, an
|
|
||||||
excluded lane contributes no active projector, and a genuinely unknown
|
|
||||||
configured lane still fails profile resolution.
|
|
||||||
- D&D registration tests prove every production D&D artifact kind has the
|
|
||||||
expected evidence capability without testing individual field loops
|
|
||||||
redundantly.
|
|
||||||
- One table-driven D&D projector test supplies representative values for all
|
|
||||||
six artifact kinds and proves plural and singular references are copied
|
|
||||||
without aliasing or semantic rewriting.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/framework/pipeline`
|
|
||||||
- `go test ./internal/modules/generic/output/json`
|
|
||||||
- `go test ./internal/modules/dnd/register`
|
|
||||||
- `go test ./internal/cli`
|
|
||||||
|
|
||||||
## Stage 2: Define And Build The Evidence-Context Artifact
|
|
||||||
|
|
||||||
Add a domain-neutral `internal/framework/evidencecontext` package that owns the
|
|
||||||
durable model, JSON Schema, strict codec, projection algorithm, and these exact
|
|
||||||
identities:
|
|
||||||
|
|
||||||
- artifact kind `source/evidence-context`;
|
|
||||||
- media type `application/json`;
|
|
||||||
- schema ID `notarius.source.evidence_context`;
|
|
||||||
- schema name `notarius_source_evidence_context_v1`; and
|
|
||||||
- schema version `v1`.
|
|
||||||
|
|
||||||
Use these package-level model and build contracts:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Document struct {
|
|
||||||
SourceID string
|
|
||||||
SourceDigest string
|
|
||||||
WindowUnits int
|
|
||||||
SelectedLanes []string
|
|
||||||
Contexts []Context
|
|
||||||
}
|
|
||||||
|
|
||||||
type Context struct {
|
|
||||||
ContextRef source.SourceRef
|
|
||||||
EvidenceRefs []EvidenceRef
|
|
||||||
Units []source.SourceUnit
|
|
||||||
}
|
|
||||||
|
|
||||||
type EvidenceRef struct {
|
|
||||||
LaneID string
|
|
||||||
SourceRef source.SourceRef
|
|
||||||
}
|
|
||||||
|
|
||||||
type LaneEvidence struct {
|
|
||||||
LaneID string
|
|
||||||
SourceRefs []source.SourceRef
|
|
||||||
}
|
|
||||||
|
|
||||||
type BuildRequest struct {
|
|
||||||
Source *source.SourceDocument
|
|
||||||
WindowUnits int
|
|
||||||
SelectedLanes []string
|
|
||||||
LaneEvidence []LaneEvidence
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Apply the JSON field names shown below. Provide `Build(BuildRequest)`,
|
|
||||||
`Serialize(BuildRequest)`, and a `Codec` with the same identity/encode/decode
|
|
||||||
responsibilities as the chunk-map codec.
|
|
||||||
|
|
||||||
The v1 payload has this exact shape:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"source_id": "session-alpha",
|
|
||||||
"source_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
|
||||||
"window_units": 0,
|
|
||||||
"selected_lanes": ["npcs"],
|
|
||||||
"contexts": [
|
|
||||||
{
|
|
||||||
"context_ref": {
|
|
||||||
"source_id": "session-alpha",
|
|
||||||
"start_unit_id": 13,
|
|
||||||
"end_unit_id": 13
|
|
||||||
},
|
|
||||||
"evidence_refs": [
|
|
||||||
{
|
|
||||||
"lane_id": "npcs",
|
|
||||||
"source_ref": {
|
|
||||||
"source_id": "session-alpha",
|
|
||||||
"start_unit_id": 13,
|
|
||||||
"end_unit_id": 13
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"units": [
|
|
||||||
{
|
|
||||||
"id": 13,
|
|
||||||
"kind": "transcript_segment",
|
|
||||||
"text": "The party meets Rowan.",
|
|
||||||
"ref": {
|
|
||||||
"source_id": "session-alpha",
|
|
||||||
"start_unit_id": 13,
|
|
||||||
"end_unit_id": 13
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
All displayed fields are required. `selected_lanes`, `contexts`,
|
|
||||||
`evidence_refs`, and `units` encode as arrays rather than `null`; `contexts`
|
|
||||||
may be empty. Each unit uses the existing `source.SourceUnit` JSON shape with
|
|
||||||
required `id`, `kind`, `text`, and `ref`, plus optional JSON-shaped `metadata`.
|
|
||||||
Fixed objects reject unknown fields; metadata remains an arbitrary JSON object.
|
|
||||||
|
|
||||||
The builder accepts the validated source document, selected lane IDs, effective
|
|
||||||
window, and lane-attributed direct references, then:
|
|
||||||
|
|
||||||
1. requires a non-negative window and a non-empty, trimmed, unique selected
|
|
||||||
lane set, then stores that set in lexical order;
|
|
||||||
2. requires every `LaneEvidence.LaneID` to belong to the selected set;
|
|
||||||
3. validates the source document, recomputes its semantic digest, and requires
|
|
||||||
it to equal `SourceDocument.Digest`;
|
|
||||||
4. validates every direct reference against one `source.DocumentIndex`;
|
|
||||||
5. deduplicates exact `(lane_id, source_ref)` contributions;
|
|
||||||
6. resolves endpoints to document positions;
|
|
||||||
7. expands each side without integer overflow and clips at document bounds;
|
|
||||||
8. sorts by expanded document position with deterministic lane/reference
|
|
||||||
tie-breakers;
|
|
||||||
9. merges overlapping or position-contiguous expanded intervals;
|
|
||||||
10. unions and deterministically sorts each merged context's direct
|
|
||||||
contributions; and
|
|
||||||
11. deep-clones the corresponding source units and JSON-shaped metadata.
|
|
||||||
|
|
||||||
Contexts are disjoint and ordered by document position, so a source unit occurs
|
|
||||||
at most once in the document. `context_ref` identifies the first and last
|
|
||||||
included units; `evidence_refs` retains only original citations. An empty
|
|
||||||
reference collection produces the same source identity, window, sorted
|
|
||||||
allowlist, and an explicit empty contexts array.
|
|
||||||
|
|
||||||
Projection failures identify only structural scope such as lane and reference
|
|
||||||
position. They must not include source text, metadata values, raw serialized
|
|
||||||
artifacts, or unrelated paths.
|
|
||||||
|
|
||||||
The codec must validate its model before encoding, produce deterministic JSON,
|
|
||||||
strictly decode the checked-in schema, and return independently owned values.
|
|
||||||
The JSON output encoder remains responsible for pretty-printing the logical
|
|
||||||
file with its standard trailing newline. Follow the existing chunk-map
|
|
||||||
package's separation between model, builder, codec, schema asset, and contract
|
|
||||||
tests where useful, without coupling the two artifact formats.
|
|
||||||
|
|
||||||
Stage tests:
|
|
||||||
|
|
||||||
- A table-driven builder suite covers zero and nonzero windows, boundary
|
|
||||||
clipping, non-monotonic unit IDs, separate gaps, overlapping and contiguous
|
|
||||||
windows, duplicate contributions, multiple lanes, stable ordering, empty
|
|
||||||
contexts, invalid selected/contributing lanes, source-digest mismatch, and
|
|
||||||
invalid references.
|
|
||||||
- Ownership tests prove output mutation cannot affect the source document or
|
|
||||||
projector inputs, including nested metadata.
|
|
||||||
- Codec tests cover round trip, required arrays, schema identity, malformed and
|
|
||||||
trailing JSON, unknown fixed fields, invalid ordering/ranges, mismatched
|
|
||||||
source identities, and independently owned decoded metadata.
|
|
||||||
- Use structured assertions and a compact valid fixture; do not add a large
|
|
||||||
transcript golden file.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/framework/evidencecontext`
|
|
||||||
|
|
||||||
## Stage 3: Integrate Projection With Runner Output
|
|
||||||
|
|
||||||
Extend `contracts.OutputRequest` with an optional
|
|
||||||
`EvidenceContext *SerializedArtifact` field and clone it at every ownership
|
|
||||||
handoff, following the existing chunk-map pointer pattern.
|
|
||||||
|
|
||||||
After lane execution and final manifest population, but before invoking the
|
|
||||||
output encoder, the runner must:
|
|
||||||
|
|
||||||
1. skip all work when the prepared evidence plan is absent;
|
|
||||||
2. index accepted `NormalizeOutputs` by their globally unique lane IDs and fail
|
|
||||||
on an internal duplicate rather than silently overwrite it;
|
|
||||||
3. for each selected lane with an output, verify its source and artifact kind,
|
|
||||||
decode it through the prepared artifact codec registry, and invoke the
|
|
||||||
prepared typed projector;
|
|
||||||
4. build and serialize the evidence document through the evidence-context
|
|
||||||
package; and
|
|
||||||
5. pass a defensive serialized-artifact copy to the output encoder.
|
|
||||||
|
|
||||||
Selected lanes without normalized output contribute nothing. Normalize
|
|
||||||
rejections remain successful pipeline outcomes; evidence projection does not
|
|
||||||
inspect rejected candidates. An invalid accepted reference, incompatible
|
|
||||||
serialized artifact, projection type failure, or evidence serialization failure
|
|
||||||
is an output-stage framework error before logical files are returned or
|
|
||||||
physically published.
|
|
||||||
|
|
||||||
At this external-content consumption boundary, do not propagate artifact-codec
|
|
||||||
or metadata-cloning errors with `%w` when their text could contain artifact
|
|
||||||
fields or source metadata. Return fixed, actionable categories scoped by lane
|
|
||||||
and operation; detailed codec errors remain available to direct trusted
|
|
||||||
callers and their focused tests.
|
|
||||||
|
|
||||||
Add an allowlisted debug summary containing only evidence artifact identity,
|
|
||||||
selected lanes, window, context count, unit count, and source digest. Do not
|
|
||||||
duplicate transcript text or source-unit metadata into a new evidence-specific
|
|
||||||
debug envelope. Existing normalized-output debug behavior remains unchanged.
|
|
||||||
|
|
||||||
The output artifact is not a normalized lane, generated reference, checkpoint,
|
|
||||||
or manifest normalized-output entry. It does not alter normalized-output,
|
|
||||||
rejection, or warning counts. Resume continues to reuse normalized checkpoints;
|
|
||||||
evidence is deterministically rebuilt during the always-executed output stage.
|
|
||||||
|
|
||||||
Stage tests:
|
|
||||||
|
|
||||||
- Runner tests use a real codec, evidence projector, and small capturing output
|
|
||||||
encoder to prove selected-lane union, absent/rejected lane omission, invalid
|
|
||||||
accepted-reference failure, output-request defensive ownership, and no work
|
|
||||||
when disabled.
|
|
||||||
- Include one checkpoint-reused normalized-output case to prove evidence is
|
|
||||||
reconstructed identically without retaining typed normalize values.
|
|
||||||
- Confirm projection failures prevent output encoding and return a failed
|
|
||||||
manifest without changing rejection semantics.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/framework/pipeline`
|
|
||||||
|
|
||||||
## Stage 4: Publish Through The JSON Bundle
|
|
||||||
|
|
||||||
Teach the production JSON encoder to recognize the optional evidence-context
|
|
||||||
artifact, verify its exact kind, media type, schema identity, schema digest, and
|
|
||||||
payload validity through the evidence-context codec, and emit:
|
|
||||||
|
|
||||||
- logical file `evidence-context.json`; and
|
|
||||||
- optional `index.json` descriptor field `evidence_context`.
|
|
||||||
|
|
||||||
The descriptor uses the same six fields as `chunk_map`:
|
|
||||||
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
|
|
||||||
`schema_version`. Refactor the encoder's private descriptor representation only
|
|
||||||
as needed to share that shape; do not change the existing `chunk_map` wire
|
|
||||||
contract. Evidence output is ordered with the encoder's other fixed logical
|
|
||||||
files, remains a non-lane artifact, and is present with an empty contexts array
|
|
||||||
when enabled but no selected lane produces references.
|
|
||||||
|
|
||||||
The JSON encoder's validation boundary returns a fixed content-safe evidence
|
|
||||||
artifact error rather than propagating decoder or schema diagnostics that could
|
|
||||||
echo transcript text or metadata. Direct evidence-context codec tests retain
|
|
||||||
detailed structural errors.
|
|
||||||
|
|
||||||
Update the maintained complete D&D configuration to enable evidence context
|
|
||||||
with window `3` for `item-events`, `npcs`, `spells`, `combat-turns`, and
|
|
||||||
`npc-interactions`. Deliberately omit `scene-descriptions`. Keep the minimal
|
|
||||||
configuration disabled by omission.
|
|
||||||
|
|
||||||
Stage tests:
|
|
||||||
|
|
||||||
- JSON encoder tests own descriptor shape, exact logical filename, identity
|
|
||||||
checking, empty evidence publication, and disabled bundle stability.
|
|
||||||
- One assembled production D&D test uses multiple selected lanes with
|
|
||||||
overlapping references and non-monotonic unit IDs, decodes the published
|
|
||||||
artifact through its production codec, and proves union/deduplication and
|
|
||||||
scene-description exclusion.
|
|
||||||
- A second narrow case explicitly allowlists a scene-description lane to prove
|
|
||||||
capability is opt-in rather than hard-coded exclusion.
|
|
||||||
- Existing index, chunk-map, lane, manifest, warning, and rejection tests remain
|
|
||||||
the owners of their current formats; do not repeat their full matrices.
|
|
||||||
|
|
||||||
Stage completion:
|
|
||||||
|
|
||||||
- `go test ./internal/modules/generic/output/json`
|
|
||||||
- `go test ./internal/modules/dnd/...`
|
|
||||||
- `go test ./internal/modules/integration`
|
|
||||||
- `go test ./internal/cli`
|
|
||||||
|
|
||||||
## Stage 5: Publish Current-Behavior Documentation
|
|
||||||
|
|
||||||
After implementation and behavioral tests pass, update canonical documentation:
|
|
||||||
|
|
||||||
- `docs/config.md` owns the nested JSON output options, defaults, strict
|
|
||||||
validation, required lane allowlist, and a small configuration snippet.
|
|
||||||
- A new `docs/integrations/evidence-context.md` owns the complete v1 payload,
|
|
||||||
identities, direct-evidence versus context semantics, ordering,
|
|
||||||
compatibility, and a compact valid example.
|
|
||||||
- `docs/integrations/json-output.md` owns the optional logical file and
|
|
||||||
`index.json` descriptor; link to the evidence contract rather than repeating
|
|
||||||
its payload.
|
|
||||||
- `docs/operations.md` owns durable source-content sensitivity, permissions,
|
|
||||||
retention, and the possibility that selected lanes cover most of a
|
|
||||||
transcript.
|
|
||||||
- `docs/consumers/subprocess.md` explains discovery through the optional index
|
|
||||||
descriptor and requires consumers to treat `evidence_refs`, not expanded
|
|
||||||
context bounds, as citations.
|
|
||||||
- `docs/policy/architecture.md` records the generic typed evidence-projection
|
|
||||||
boundary and output ownership without adding D&D or wire-format detail.
|
|
||||||
- `docs/internal/pipeline.md` and `docs/internal/modules.md` describe the typed
|
|
||||||
evidence registry, preparation checks, reconstruction from serialized
|
|
||||||
normalize outputs, and output-stage ownership without restating public wire
|
|
||||||
fields.
|
|
||||||
|
|
||||||
Update only the smallest orientation links needed for discoverability. Do not
|
|
||||||
add a CLI flag, configuration environment override, or duplicate the complete
|
|
||||||
configuration outside `examples/`.
|
|
||||||
|
|
||||||
After all current-behavior documentation is accurate:
|
|
||||||
|
|
||||||
- set [the feature roadmap](evidence.md) status to `Implemented`;
|
|
||||||
- set this plan's status to `Completed`; and
|
|
||||||
- leave the integration and configuration documents, not either roadmap, as
|
|
||||||
the canonical implemented contract.
|
|
||||||
|
|
||||||
Final verification:
|
|
||||||
|
|
||||||
- `git diff --check`
|
|
||||||
- `go test ./...`
|
|
||||||
- `go vet ./...`
|
|
||||||
- `go build ./cmd/notarius`
|
|
||||||
- `go test -race ./internal/framework/evidencecontext ./internal/framework/pipeline ./internal/modules/generic/output/json ./internal/modules/dnd/... ./internal/modules/integration ./internal/cli`
|
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None. The plan fixes the configuration shape and defaults, typed projection
|
None. The message sequences, cache boundaries, test ownership, documentation
|
||||||
boundary, preparation timing, durable schema and identities, range-union
|
owners, and compatibility policy are specified above.
|
||||||
algorithm, failure semantics, JSON discovery, D&D coverage, documentation
|
|
||||||
ownership, and test boundaries.
|
|
||||||
|
|||||||
@@ -1,208 +0,0 @@
|
|||||||
# Subprocess Integration Contract
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
Implemented.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Make Notarius straightforward to invoke as a subprocess from an orchestrator
|
|
||||||
such as Narratio. A caller should be able to run a configured pipeline, discover
|
|
||||||
the published output bundle without parsing human prose or scanning a
|
|
||||||
directory, and hand selected structured artifacts to a later stage.
|
|
||||||
|
|
||||||
This work strengthens the public CLI boundary. It does not turn Notarius into a
|
|
||||||
Go library, embed Narratio-specific behavior, or change pipeline execution and
|
|
||||||
artifact semantics.
|
|
||||||
|
|
||||||
## Desired End State
|
|
||||||
|
|
||||||
A subprocess caller can:
|
|
||||||
|
|
||||||
1. validate a Notarius configuration and selected pipeline before execution;
|
|
||||||
2. invoke `notarius run` with explicit input, output-root, session, and
|
|
||||||
reference arguments;
|
|
||||||
3. request one versioned, machine-readable success result on standard output;
|
|
||||||
4. use that result to locate the published output bundle;
|
|
||||||
5. discover normalized lane payloads through the bundle's authoritative
|
|
||||||
`index.json`;
|
|
||||||
6. distinguish process failure from successful partial pipeline outcomes; and
|
|
||||||
7. record Notarius run provenance in its own manifest without depending on
|
|
||||||
internal packages, cache formats, debug formats, or human-readable messages.
|
|
||||||
|
|
||||||
The existing human-oriented command output remains the default for interactive
|
|
||||||
use.
|
|
||||||
|
|
||||||
## Machine-Readable Run Result
|
|
||||||
|
|
||||||
`notarius run` supports `--json`. On success, the flag makes standard output
|
|
||||||
contain exactly one JSON object followed by a newline. No human-oriented status
|
|
||||||
line is mixed into that stream.
|
|
||||||
|
|
||||||
The result uses the schema identity `notarius.run-result.v1` and contains:
|
|
||||||
|
|
||||||
| Field | Presence | Meaning |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `schema_version` | Required | Exactly `notarius.run-result.v1`. |
|
|
||||||
| `run_id` | Required | The Notarius run identifier. |
|
|
||||||
| `pipeline_id` | Required | The effective pipeline identifier. |
|
|
||||||
| `output_directory` | Required | Absolute path to the successfully published output bundle. |
|
|
||||||
| `index_file` | Required for the production JSON output | Logical bundle path `index.json`. |
|
|
||||||
| `normalized_output_count` | Required | Number of final normalized lane outputs returned by the pipeline. |
|
|
||||||
| `rejected_output_count` | Required | Number of recorded rejected outputs. |
|
|
||||||
| `warning_count` | Required | Number of final run warnings returned by the pipeline. |
|
|
||||||
| `validation_status` | Required | The run manifest's final validation status without reinterpretation. |
|
|
||||||
| `debug_directory` | Optional | Absolute debug-bundle path when debug capture was requested and completed. |
|
|
||||||
|
|
||||||
An illustrative successful result is:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": "notarius.run-result.v1",
|
|
||||||
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
|
||||||
"pipeline_id": "dnd-session",
|
|
||||||
"output_directory": "/srv/narratio/runs/session-7/notarius/run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
|
||||||
"index_file": "index.json",
|
|
||||||
"normalized_output_count": 6,
|
|
||||||
"rejected_output_count": 2,
|
|
||||||
"warning_count": 1,
|
|
||||||
"validation_status": "approved"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
The receipt is a discovery and summary document, not a duplicate output
|
|
||||||
envelope. It does not embed lane payloads, rejection entries, warnings, the run
|
|
||||||
manifest, or output-file contents. Consumers use `index_file` and the existing
|
|
||||||
published JSON output contract for those records.
|
|
||||||
|
|
||||||
The result contract must tolerate future additive optional fields. Any
|
|
||||||
incompatible field or semantic change requires a new run-result schema version.
|
|
||||||
|
|
||||||
## Stream, Publication, And Failure Semantics
|
|
||||||
|
|
||||||
Machine-readable output is emitted only after:
|
|
||||||
|
|
||||||
- the pipeline has completed without a framework error;
|
|
||||||
- all logical output files have been successfully published;
|
|
||||||
- requested debug terminal reporting has completed; and
|
|
||||||
- all result fields are known.
|
|
||||||
|
|
||||||
Writing or encoding the machine-readable result is part of successful command
|
|
||||||
completion. Failure to write it produces the existing runtime-failure exit
|
|
||||||
class.
|
|
||||||
|
|
||||||
With `--json`:
|
|
||||||
|
|
||||||
- successful stdout is exclusively the run-result JSON document;
|
|
||||||
- successful warnings remain on stderr under the existing CLI contract;
|
|
||||||
- syntax and runtime errors retain their existing exit statuses and stderr
|
|
||||||
diagnostics;
|
|
||||||
- consumers treat stdout as a valid result only when the process exits with
|
|
||||||
status 0; failures before result writing emit no result, while a failure
|
|
||||||
during the stdout write may leave incomplete bytes that must be ignored; and
|
|
||||||
- human-readable diagnostic wording is not promoted into a machine contract.
|
|
||||||
|
|
||||||
Without `--json`, current interactive stdout and stderr behavior remains
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
Successful runs may contain rejected outputs or omit some normalized lanes.
|
|
||||||
That remains a valid pipeline outcome. The run result reports counts, while
|
|
||||||
`index.json`, `rejected.json`, and `warnings.json` remain authoritative for
|
|
||||||
details. Notarius will not add a generic `--fail-on-rejection` policy as part
|
|
||||||
of this work.
|
|
||||||
|
|
||||||
## Output Discovery And Consumer Responsibilities
|
|
||||||
|
|
||||||
The production JSON encoder's `index.json` remains the authoritative mapping
|
|
||||||
from lane IDs to published payloads. A subprocess consumer should:
|
|
||||||
|
|
||||||
- resolve `index_file` beneath `output_directory` and reject path escape;
|
|
||||||
- locate expected outputs by `lane_id`, not by guessing filenames;
|
|
||||||
- check each selected descriptor's media type and schema identity;
|
|
||||||
- decode payloads according to their published integration contracts;
|
|
||||||
- decide which lanes are required or optional for its own later stages; and
|
|
||||||
- retain rejection, warning, and manifest files when they are needed for
|
|
||||||
provenance or review.
|
|
||||||
|
|
||||||
For Narratio, required report inputs and partial-success policy remain Narratio
|
|
||||||
stage configuration and orchestration concerns. Notarius does not acquire
|
|
||||||
knowledge of Narratio stages, manifests, workspace layout, publication policy,
|
|
||||||
or report formats.
|
|
||||||
|
|
||||||
## Invocation Guidance
|
|
||||||
|
|
||||||
The consumer documentation recommends that subprocess callers:
|
|
||||||
|
|
||||||
- use `notarius config validate --pipeline` as an optional preflight;
|
|
||||||
- pass explicit absolute paths for the input, configuration, output root, and
|
|
||||||
CLI-supplied references;
|
|
||||||
- use a stable, non-secret prompt session identifier when useful for provider
|
|
||||||
routing or caching;
|
|
||||||
- capture stdout and stderr separately;
|
|
||||||
- supply credentials through the configured environment mechanism rather than
|
|
||||||
command arguments or generated configuration containing secret values;
|
|
||||||
- place output, cache, debug, and subprocess logs under intentional
|
|
||||||
sensitivity and retention policies; and
|
|
||||||
- treat the Notarius manifest and run-result receipt as provenance while
|
|
||||||
leaving the caller's own manifest authoritative for its stage lifecycle.
|
|
||||||
|
|
||||||
Notarius configuration remains owned by Notarius. An orchestrator may select a
|
|
||||||
configuration and pass supported operational overrides, but should not
|
|
||||||
duplicate the complete Notarius configuration schema.
|
|
||||||
|
|
||||||
## Documentation End State
|
|
||||||
|
|
||||||
- `docs/cli.md` owns `run --json`, stream behavior, and exit semantics;
|
|
||||||
- a new `docs/integrations/run-result.md` owns the versioned run-result wire
|
|
||||||
contract and compatibility policy;
|
|
||||||
- `docs/integrations/json-output.md` remains the sole owner of output-bundle
|
|
||||||
discovery and lane publication;
|
|
||||||
- a new `docs/consumers/subprocess.md` provides the task-oriented invocation and
|
|
||||||
consumption workflow; and
|
|
||||||
- `docs/internal/cli.md` describes how the CLI constructs and emits the result
|
|
||||||
only after successful publication.
|
|
||||||
|
|
||||||
Other documents should link to these owners instead of repeating volatile
|
|
||||||
fields or command details.
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
|
|
||||||
- An ordinary successful `run` retains its existing human-readable output.
|
|
||||||
- A successful `run --json` emits one valid `notarius.run-result.v1` document
|
|
||||||
and no human prose on stdout.
|
|
||||||
- Relative configured or overridden output and debug roots are reported as
|
|
||||||
absolute bundle paths.
|
|
||||||
- The receipt identifies the production JSON bundle entry point without
|
|
||||||
copying its lane descriptors or payloads.
|
|
||||||
- Warning-bearing and rejection-bearing runs remain successful and report
|
|
||||||
accurate counts.
|
|
||||||
- Syntax, configuration, provider, pipeline, publication, debug, and result
|
|
||||||
writing failures retain the correct nonzero exit class. Consumers are
|
|
||||||
explicitly required to ignore stdout from a nonzero invocation.
|
|
||||||
- The implementation does not expose internal Go types or couple generic CLI
|
|
||||||
code to D&D or Narratio concepts.
|
|
||||||
- Public and internal documentation assigns each new contract to one canonical
|
|
||||||
owner.
|
|
||||||
- Offline behavioral tests protect the structured-output contract, default
|
|
||||||
human behavior, absolute path reporting, stream separation, and failure to
|
|
||||||
serialize or write the success result without duplicating lower-level output
|
|
||||||
encoder tests.
|
|
||||||
|
|
||||||
## Out Of Scope
|
|
||||||
|
|
||||||
The following may be useful later but are not prerequisites for the Narratio
|
|
||||||
integration:
|
|
||||||
|
|
||||||
- a result-file flag in addition to machine-readable stdout;
|
|
||||||
- a JSON failure envelope or stable machine-readable error taxonomy;
|
|
||||||
- caller-supplied Notarius run IDs or exact output-bundle paths;
|
|
||||||
- a generic `--fail-on-rejection` or required-lane CLI policy;
|
|
||||||
- a public Go client package or importable Narratio adapter;
|
|
||||||
- Narratio stage, configuration, manifest, or report-generation changes;
|
|
||||||
- `notarius version --json`;
|
|
||||||
- installable or queryable artifact JSON Schemas;
|
|
||||||
- signal-aware CLI contexts and graceful SIGINT or SIGTERM handling;
|
|
||||||
- packaged release artifacts and a broader application-versioning policy.
|
|
||||||
|
|
||||||
These items should be promoted only in response to a demonstrated integration
|
|
||||||
need rather than bundled into the initial subprocess contract.
|
|
||||||
Reference in New Issue
Block a user