Add a new feature roadmap and implementation plan for D&D prompt ordering to improve LLM provider caching
This commit is contained in:
90
docs/roadmap/dnd-prompt-cache-ordering.md
Normal file
90
docs/roadmap/dnd-prompt-cache-ordering.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# D&D Prompt Cache Ordering
|
||||
|
||||
## Purpose
|
||||
|
||||
Arrange maintained D&D prompts so sibling extraction lanes share the longest
|
||||
useful byte-identical prefix, including the transcript chunk, and can therefore
|
||||
reuse provider-side prompt caches effectively.
|
||||
|
||||
This policy concerns rendered message order and cache boundaries. It does not
|
||||
change extraction semantics, artifact contracts, module inputs, durable output,
|
||||
or the responsibilities of deterministic validators and normalizers.
|
||||
|
||||
## Extraction Prompt Policy
|
||||
|
||||
Every D&D extraction prompt shares this rendered prefix:
|
||||
|
||||
1. the shared D&D system prompt;
|
||||
2. the shared D&D identity prompt;
|
||||
3. the shared campaign-reference prompt; and
|
||||
4. the shared transcript-chunk prompt.
|
||||
|
||||
The messages must render identically, including roles, content, and cache
|
||||
metadata, when two lanes receive the same transcript and common references.
|
||||
The campaign-reference and transcript messages establish ephemeral cache
|
||||
boundaries.
|
||||
|
||||
Only material used by every extraction lane belongs before the transcript.
|
||||
Shared wording that applies to only a subset of lanes is placed after the
|
||||
transcript rather than being added to unrelated prompts solely to lengthen the
|
||||
common prefix. Module instructions, optional generated references, catalogs,
|
||||
and other lane-specific context also follow the transcript.
|
||||
|
||||
The lane-specific suffixes are:
|
||||
|
||||
- NPCs: shared extraction-evidence policy, task, then instructions.
|
||||
- Item events: shared extraction-evidence policy, task, then instructions.
|
||||
- Scene descriptions: task, then instructions.
|
||||
- Combat turns: shared extraction-evidence policy, NPC registry, task, then
|
||||
instructions.
|
||||
- NPC interactions: shared extraction-evidence policy, NPC registry, task,
|
||||
then instructions.
|
||||
- Spells: shared extraction-evidence policy, NPC registry, spell catalog, task,
|
||||
then instructions.
|
||||
|
||||
The final instructions message establishes an ephemeral cache boundary.
|
||||
Intermediate cache markers on the shared identity message or lane-specific
|
||||
reference messages are unnecessary.
|
||||
|
||||
## Purpose-Specific Prompt Families
|
||||
|
||||
The extraction prefix is a sibling-lane policy, not a universal ordering rule
|
||||
for every D&D LLM call.
|
||||
|
||||
Scene chunking has no sibling lane with which to share a transcript prefix. It
|
||||
orders its prompt as system, common campaign references, task, instructions,
|
||||
and full transcript. The campaign-reference and transcript messages establish
|
||||
ephemeral cache boundaries.
|
||||
|
||||
NPC normalization orders its prompt as system, task, instructions, candidate
|
||||
NPCs, and transcript windows. The instructions and transcript-window messages
|
||||
establish ephemeral cache boundaries. Candidate artifacts remain ahead of the
|
||||
evidence windows needed to evaluate them.
|
||||
|
||||
Future D&D prompt families should identify their actual reuse boundary rather
|
||||
than mechanically copying either exception or the extraction sequence.
|
||||
|
||||
## Compatibility And Observability
|
||||
|
||||
Existing prompt IDs, prompt versions, response schemas, and module contracts
|
||||
remain unchanged. Prompt-content fingerprints already make the reordered
|
||||
assets part of checkpoint identity, so old development checkpoints may become
|
||||
cold misses without a checkpoint-format migration.
|
||||
|
||||
Prompt-order verification should exercise rendered messages with unique input
|
||||
sentinels. Tests must protect roles, cache metadata, input isolation, and the
|
||||
shared-prefix invariant without requiring particular prose, words, or phrases
|
||||
to remain in prompt assets.
|
||||
|
||||
## Desired End State
|
||||
|
||||
- All six D&D extraction prompts render the same four-message prefix for
|
||||
equivalent common inputs.
|
||||
- The transcript chunk is included in that common prefix and is followed only
|
||||
by lane-specific material.
|
||||
- Cache-control hints identify useful prefix boundaries without redundant
|
||||
intermediate markers.
|
||||
- Scene chunking and NPC normalization retain orderings suited to their
|
||||
distinct inputs and reuse opportunities.
|
||||
- Internal documentation explains both the general provider-cache principle
|
||||
and the concrete D&D prompt-family policy.
|
||||
@@ -1,226 +0,0 @@
|
||||
# Published Evidence Context
|
||||
|
||||
## Status
|
||||
|
||||
Implemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
Let downstream consumers build narrative reports from normalized artifacts
|
||||
without separately parsing the original transcript or resolving source-unit
|
||||
references themselves.
|
||||
|
||||
The production JSON output optionally publishes one deterministic, deduplicated
|
||||
evidence-context artifact containing the transcript units relevant to explicitly
|
||||
selected normalized lanes. Existing lane payloads remain the canonical semantic
|
||||
results and retain their precise source references.
|
||||
|
||||
## Desired End State
|
||||
|
||||
When evidence-context publication is enabled, a consumer can:
|
||||
|
||||
1. discover one versioned evidence-context document through `index.json`;
|
||||
2. obtain the union of source units needed to understand evidence cited by the
|
||||
selected normalized lanes;
|
||||
3. distinguish each artifact's direct evidence references from surrounding
|
||||
units included only for narrative context;
|
||||
4. retain speaker, timestamp, and other accepted source-unit metadata needed to
|
||||
interpret the transcript; and
|
||||
5. produce a narrative report without receiving duplicated transcript text in
|
||||
every lane payload.
|
||||
|
||||
This is deterministic output projection. It does not invoke an LLM, change
|
||||
normalization, or make surrounding context part of an artifact's evidence.
|
||||
|
||||
## Configuration Policy
|
||||
|
||||
Evidence publication is configured on the production JSON output module. The
|
||||
intended configuration shape is:
|
||||
|
||||
```yaml
|
||||
output:
|
||||
module: json
|
||||
options:
|
||||
evidence_context:
|
||||
enabled: true
|
||||
window_units: 3
|
||||
lanes:
|
||||
- combat-turns
|
||||
- item-events
|
||||
- npc-interactions
|
||||
- npcs
|
||||
- spells
|
||||
```
|
||||
|
||||
- Omitting `evidence_context` disables publication. When the object is present,
|
||||
`enabled` is required.
|
||||
- `enabled: false` accepts no `lanes` or `window_units` fields, preventing
|
||||
silently ignored configuration.
|
||||
- `lanes` is a required, non-empty allowlist of configured final lane IDs when
|
||||
evidence publication is enabled. Values are trimmed, unique, and normalized
|
||||
to lexical order.
|
||||
- `window_units` is a non-negative integer and defaults to `3`. Zero publishes
|
||||
only directly referenced units.
|
||||
- Unknown lanes, duplicate lane IDs, and selected lanes whose artifact kind
|
||||
cannot expose source evidence fail configuration resolution or pipeline
|
||||
preparation.
|
||||
- A selected lane that completes without a normalized output contributes no
|
||||
evidence and does not make an otherwise successful run fail.
|
||||
- Invocation-level lane filtering does not invalidate the configured allowlist.
|
||||
Allowlisted lanes excluded from the effective run contribute nothing, while
|
||||
the evidence document still records the configured allowlist.
|
||||
|
||||
The allowlist is intentional safety and stability policy. Scene descriptions
|
||||
and other broad-range lanes are excluded unless named expressly. Adding a new
|
||||
pipeline lane never silently increases output size or publishes more transcript
|
||||
content.
|
||||
|
||||
## Evidence Collection Boundary
|
||||
|
||||
Evidence collection applies to accepted final normalized artifacts from the
|
||||
selected lanes. It must not inspect arbitrary serialized JSON for fields named
|
||||
`source_ref` or `source_refs`, and the generic JSON output module must not
|
||||
depend on D&D artifact types.
|
||||
|
||||
Artifact-kind registrations expose their source references through an explicit
|
||||
typed projection contract. The framework uses that contract to assemble a
|
||||
domain-neutral evidence request containing:
|
||||
|
||||
- the accepted generic source document;
|
||||
- the selected lane and artifact identities; and
|
||||
- defensive copies of their direct source references.
|
||||
|
||||
The output stage owns publication of the resulting logical artifact. Generic
|
||||
framework code owns range validation, position-based expansion, and union
|
||||
logic. Domain-specific adapters own only the extraction of evidence references
|
||||
from their typed artifacts.
|
||||
|
||||
Both plural-reference artifacts and singular-reference artifacts, such as
|
||||
scene descriptions, can participate through the same projection contract.
|
||||
They do so only when their configured lane is allowlisted.
|
||||
|
||||
## Range Expansion And Deduplication
|
||||
|
||||
For every valid direct source reference:
|
||||
|
||||
1. resolve its endpoints through source-document positions, not numeric
|
||||
unit-ID arithmetic;
|
||||
2. expand the range by `window_units` positions on each side;
|
||||
3. clip the expanded range at document boundaries; and
|
||||
4. union overlapping or contiguous expanded ranges.
|
||||
|
||||
Published contexts and units remain in source-document order. Each source unit
|
||||
appears at most once in a merged context. Original direct references remain
|
||||
unchanged and are associated with their contributing lane IDs so consumers can
|
||||
tell why a context was included.
|
||||
|
||||
The projector must not silently omit or repair an invalid reference that
|
||||
reaches this boundary. Such a value violates the accepted normalized-artifact
|
||||
contract and causes output projection to fail with a content-safe error.
|
||||
|
||||
No implicit coverage limit truncates selected evidence. If the allowlisted
|
||||
lanes collectively cite most or all of a transcript, the evidence document may
|
||||
contain most or all of it. The explicit lane allowlist is the control that
|
||||
prevents a broad lane such as scene descriptions from doing so accidentally.
|
||||
|
||||
## Durable Evidence Artifact
|
||||
|
||||
The JSON bundle gains one optional, non-lane artifact with these durable
|
||||
identities:
|
||||
|
||||
| Property | Value |
|
||||
| --- | --- |
|
||||
| Logical file | `evidence-context.json` |
|
||||
| Index descriptor | `evidence_context` |
|
||||
| Artifact kind | `source/evidence-context` |
|
||||
| Media type | `application/json` |
|
||||
| Schema ID | `notarius.source.evidence_context` |
|
||||
| Schema name | `notarius_source_evidence_context_v1` |
|
||||
| Schema version | `v1` |
|
||||
|
||||
The descriptor in `index.json` carries the artifact and schema identities,
|
||||
analogous to the existing chunk-map descriptor. The artifact is present
|
||||
whenever evidence publication is enabled, including when its context collection
|
||||
is empty.
|
||||
|
||||
The document contains:
|
||||
|
||||
- the source document ID and semantic digest;
|
||||
- the effective window size;
|
||||
- the sorted configured lane allowlist;
|
||||
- an ordered context collection;
|
||||
- each context's expanded start and end unit IDs;
|
||||
- the original direct references and contributing lane IDs covered by that
|
||||
context; and
|
||||
- the ordered accepted source units, including unit ID, kind, text,
|
||||
self-reference, and metadata.
|
||||
|
||||
Expanded context bounds are navigation aids, not citations. The original
|
||||
references embedded in each context remain the authoritative direct evidence.
|
||||
The evidence artifact is discovered separately from lane payloads and does not
|
||||
increase the normalized-lane count reported by the runner or subprocess
|
||||
receipt.
|
||||
|
||||
## Failure And Publication Semantics
|
||||
|
||||
- Evidence projection occurs only after selected normalized outputs are known
|
||||
and before the output encoder returns its logical files.
|
||||
- Projection or encoding failure is an output-stage framework error; the CLI
|
||||
does not publish a partially assembled output bundle.
|
||||
- Rejected or absent lane outputs contribute nothing. Their attempted values
|
||||
and source references must not be published through this artifact.
|
||||
- Context generation is deterministic for the same source document, selected
|
||||
normalized outputs, lane allowlist, and window size.
|
||||
- Existing output, checkpoint, warning, rejection, debug, and subprocess
|
||||
success semantics remain unchanged.
|
||||
|
||||
## Sensitivity And Size
|
||||
|
||||
Unlike the current chunk map, the evidence artifact contains transcript text
|
||||
and source-unit metadata. Enabling it therefore creates additional durable
|
||||
sensitive data and may materially increase bundle size.
|
||||
|
||||
The implemented configuration, operations, integration, and consumer documents
|
||||
state that:
|
||||
|
||||
- evidence publication is opt-in;
|
||||
- output permissions and retention must be appropriate for source content;
|
||||
- selecting broad or numerous lanes can publish most of the transcript; and
|
||||
- the artifact must not contain raw input bytes, LLM prompts or responses,
|
||||
auxiliary reference content, credentials, debug-only data, or filesystem
|
||||
paths.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Evidence publication is disabled by default and leaves existing bundles
|
||||
unchanged.
|
||||
- Enabling it requires an explicit non-empty lane allowlist.
|
||||
- References from all selected successful lanes contribute to one deduplicated
|
||||
document.
|
||||
- Non-monotonic unit IDs are expanded and ordered correctly by document
|
||||
position.
|
||||
- Overlapping windows share one ordered copy of each included source unit.
|
||||
- Direct references remain distinguishable from added context.
|
||||
- Scene descriptions cannot contribute unless their lane is explicitly
|
||||
allowlisted.
|
||||
- Invalid selected lanes and unsupported artifact kinds fail before execution;
|
||||
invalid accepted references fail output projection rather than being ignored.
|
||||
- Empty selected-lane results produce a valid empty evidence artifact.
|
||||
- Existing D&D lane schemas, normalized-output counts, and source-reference
|
||||
semantics do not change.
|
||||
- Generic framework and output packages do not depend on D&D types or parse
|
||||
artifact JSON heuristically.
|
||||
- The published contract and operational documentation clearly describe source
|
||||
sensitivity, discovery, compatibility, and retention.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- Embedding transcript units directly into each D&D record or lane payload.
|
||||
- Replacing precise source references with expanded context ranges.
|
||||
- Automatically including every configured lane.
|
||||
- An explicit full-transcript publication mode.
|
||||
- LLM summarization, retrieval, ranking, or narrative generation.
|
||||
- Per-record window sizes or lane-specific window sizes.
|
||||
- CLI overrides for evidence configuration.
|
||||
- Reading rejected attempts, debug artifacts, auxiliary references, or prior
|
||||
output bundles as evidence sources.
|
||||
@@ -60,6 +60,30 @@ safety checks, and deterministic application of accepted changes.
|
||||
- Add media-type validators when non-JSON artifact representations are
|
||||
introduced.
|
||||
|
||||
## LLM Runtime Evolution
|
||||
|
||||
### Native Session Propagation
|
||||
|
||||
- Once upstream Scriptorium exposes a session identifier on its run request,
|
||||
propagate the existing `StructuredCompletionRequest.SessionID` through the
|
||||
Scriptorium adapter's native session field.
|
||||
- Preserve the current `--session-id` invocation contract and its run-wide
|
||||
propagation to every prompt-facing module and validator. Do not introduce a
|
||||
second session configuration surface.
|
||||
- Retain session identity in checkpoint provenance so runs with different
|
||||
sessions cannot reuse one another's LLM-derived checkpoints.
|
||||
- Define the upstream compatibility and prompt-variable transition explicitly:
|
||||
native provider session behavior must not silently remove a `session_id`
|
||||
prompt variable while maintained prompts still consume it.
|
||||
- Add adapter and assembled-run coverage for exact forwarding, trimming,
|
||||
concurrent-run isolation, and unsupported-provider behavior once the
|
||||
upstream contract is available.
|
||||
|
||||
This work is blocked on native session support in the upstream Scriptorium
|
||||
package. Notarius already carries a run-scoped session ID through its CLI,
|
||||
pipeline requests, checkpoint identity, and prompt variables; the missing
|
||||
capability is native propagation across the LLM adapter boundary.
|
||||
|
||||
## Further Reference Evolution
|
||||
|
||||
- Make prior-run artifacts easier to bind as references without changing the
|
||||
|
||||
@@ -1,486 +1,257 @@
|
||||
# Published Evidence Context Implementation Plan
|
||||
# D&D Prompt Cache Ordering Implementation Plan
|
||||
|
||||
## Status
|
||||
## Summary
|
||||
|
||||
Completed.
|
||||
Implement the target state in
|
||||
[D&D Prompt Cache Ordering](dnd-prompt-cache-ordering.md): give all six D&D
|
||||
extraction prompts a byte-identical rendered prefix through the transcript
|
||||
chunk, retain purpose-specific ordering for scene chunking and NPC
|
||||
normalization, consolidate tests around durable prompt-order invariants, and
|
||||
replace the current transcript-last documentation.
|
||||
|
||||
## Objective
|
||||
This is a prompt-asset and documentation change. Do not change Go module APIs,
|
||||
LLM request contracts, prompt IDs or versions, response schemas, artifact
|
||||
schemas, reference declarations, extraction behavior, retry behavior, or
|
||||
durable output. Existing prompt fingerprints must continue to provide
|
||||
checkpoint invalidation; no checkpoint migration or cleanup is required.
|
||||
|
||||
Implement the accepted [Published Evidence Context](evidence.md) roadmap as an
|
||||
optional, deterministic extension of the production JSON output. The completed
|
||||
work must publish one deduplicated source-context artifact for an explicit set
|
||||
of successful normalized lanes while leaving existing lane payloads,
|
||||
normalization, checkpointing, subprocess results, and disabled output bundles
|
||||
unchanged.
|
||||
## Stage 1: Reorder The Six Extraction Prompts
|
||||
|
||||
Complete the stages below in order. Each stage must leave its affected packages
|
||||
passing before the next begins. Do not implement per-record hydration, implicit
|
||||
all-lane collection, a full-transcript mode, LLM processing, or any other
|
||||
roadmap item marked out of scope.
|
||||
Update these manifests:
|
||||
|
||||
## Decisions And Invariants
|
||||
- `internal/modules/dnd/extract/npcs/assets/prompts/dnd.npcs.yaml`
|
||||
- `internal/modules/dnd/extract/itemevents/assets/prompts/dnd.item_events.yaml`
|
||||
- `internal/modules/dnd/extract/scenedescriptions/assets/prompts/dnd.scene_descriptions.yaml`
|
||||
- `internal/modules/dnd/extract/combatturns/assets/prompts/dnd.combat_turns.yaml`
|
||||
- `internal/modules/dnd/extract/npcinteractions/assets/prompts/dnd.npc_interactions.yaml`
|
||||
- `internal/modules/dnd/extract/spells/assets/prompts/dnd.spells.yaml`
|
||||
|
||||
- Evidence publication is output policy. Normalizers continue to return
|
||||
semantic artifacts with precise source references and do not receive
|
||||
hydration responsibilities.
|
||||
- The framework operates on the accepted generic `source.SourceDocument` and
|
||||
typed artifact projections. It must not inspect serialized JSON for
|
||||
`source_ref` or `source_refs`, and generic packages must not depend on D&D
|
||||
types.
|
||||
- The selected lane allowlist is explicit, non-empty, and globally addressed by
|
||||
resolved lane ID. Pipeline resolution already guarantees lane IDs are unique
|
||||
across ordered steps.
|
||||
- The framework decodes accepted serialized normalize outputs through their
|
||||
registered artifact codecs before invoking typed evidence projectors. This
|
||||
supports both fresh and checkpoint-reused normalize outputs without retaining
|
||||
a second typed result channel.
|
||||
- Expansion uses source-document positions. Numeric unit IDs are identities,
|
||||
not sequence numbers.
|
||||
- Direct source references are never widened or rewritten. Expanded ranges are
|
||||
context bounds only.
|
||||
- Rejected, failed, and absent normalized lane outputs contribute no evidence.
|
||||
- Evidence output is sensitive durable source content, not cache or debug
|
||||
state. It contains accepted source units only and never raw input bytes,
|
||||
prompts, model responses, auxiliary references, paths, or credentials.
|
||||
- The optional `evidence_context` index field is an additive v1 JSON-bundle
|
||||
change. Existing D&D artifact schemas and the subprocess receipt do not
|
||||
change.
|
||||
Give every manifest this exact leading message sequence:
|
||||
|
||||
## Stage 1: Add Typed Evidence Capability And Resolve Output Policy
|
||||
1. `common-dnd-system.md`, with role `system` and no cache control;
|
||||
2. `common-dnd-identity.md`, with role `user` and no cache control;
|
||||
3. `common-dnd-references.md`, with role `user` and ephemeral cache control;
|
||||
4. `common-dnd-transcript.md`, with role `user` and ephemeral cache control.
|
||||
|
||||
Add a dedicated artifact-evidence registry under the pipeline framework:
|
||||
The common prefix must contain no extraction-evidence policy, NPC registry,
|
||||
spell catalog, task, or module instructions. Those messages vary across lanes
|
||||
and would prevent the transcript from participating in a shared cache prefix.
|
||||
|
||||
- `pipeline.ArtifactEvidenceRegistry` stores one typed projector per artifact
|
||||
kind.
|
||||
- `pipeline.ArtifactEvidenceProjector[T]` is
|
||||
`func(T) []source.SourceRef`.
|
||||
- `pipeline.RegisterArtifactEvidence[T](registry, kind, projector)` accepts a
|
||||
non-empty kind and non-nil projector, records the exact Go type for `T`, and
|
||||
rejects duplicate kinds.
|
||||
- The erased projection boundary checks the exact registered type, invokes the
|
||||
projector, and returns a defensive copy of its references.
|
||||
- The registry exposes only the discovery and projection operations required by
|
||||
resolution, preparation, and execution; do not expose its mutable entries.
|
||||
Append the following lane-specific sequences after the transcript:
|
||||
|
||||
Add the registry to `pipeline.Registries` and `pipeline.ModuleCatalog`, including
|
||||
CLI catalog conversion, production construction, empty-set detection, and
|
||||
test registry helpers. A nil evidence registry remains valid when evidence
|
||||
publication is disabled. Production construction and the D&D registrar require
|
||||
and populate it.
|
||||
| Prompt | Post-transcript sequence |
|
||||
| --- | --- |
|
||||
| `dnd.npcs` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.item_events` | `common-dnd-extraction-evidence.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.scene_descriptions` | `task.md`, `instructions.md` |
|
||||
| `dnd.combat_turns` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.npc_interactions` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `task.md`, `instructions.md` |
|
||||
| `dnd.spells` | `common-dnd-extraction-evidence.md`, `common-dnd-npcs.md`, `catalog.md`, `task.md`, `instructions.md` |
|
||||
|
||||
Define these framework-level output-policy contracts:
|
||||
For every extraction prompt:
|
||||
|
||||
```go
|
||||
type EvidenceContextPolicy struct {
|
||||
Enabled bool
|
||||
WindowUnits int
|
||||
LaneIDs []string
|
||||
}
|
||||
- apply ephemeral cache control to the final `instructions.md` message;
|
||||
- do not apply cache control to the extraction-evidence, NPC-registry, catalog,
|
||||
or task messages;
|
||||
- remove the existing cache control from `common-dnd-identity.md` and
|
||||
`common-dnd-npcs.md`; and
|
||||
- retain the existing inputs, requiredness, media types, output contract,
|
||||
profile, repair setting, prompt identity, and schema path.
|
||||
|
||||
type EvidenceContextPolicyProvider interface {
|
||||
EvidenceContextPolicy() EvidenceContextPolicy
|
||||
}
|
||||
Do not add the extraction-evidence asset to the scene-description prompt merely
|
||||
to make it resemble the other lanes. It is not universal shared context and
|
||||
therefore belongs outside the common-prefix contract.
|
||||
|
||||
### Stage 1 completion criteria
|
||||
|
||||
- Equivalent transcript, player, party, and glossary inputs render the same
|
||||
first four messages for all six extraction prompts, including identical
|
||||
roles, content, and cache metadata.
|
||||
- Each transcript input is rendered exactly once.
|
||||
- Every lane-specific message follows the transcript.
|
||||
- Prompt preparation and schema wiring remain successful for every lane.
|
||||
|
||||
## Stage 2: Apply Purpose-Specific Ordering To Other D&D LLM Prompts
|
||||
|
||||
Update
|
||||
`internal/modules/dnd/chunk/scenes/assets/prompts/dnd.scenes.yaml` to use:
|
||||
|
||||
1. `common-dnd-system.md`, without cache control;
|
||||
2. `common-dnd-references.md`, with ephemeral cache control;
|
||||
3. `task.md`, without cache control;
|
||||
4. `instructions.md`, without cache control; and
|
||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
||||
|
||||
The scene chunker consumes the full transcript once and has no sibling lanes
|
||||
that can share a transcript prefix. Stable task instructions therefore remain
|
||||
before the changing full transcript.
|
||||
|
||||
Update
|
||||
`internal/modules/dnd/normalize/npcs/assets/prompts/dnd.npcs.normalize.yaml` to
|
||||
use:
|
||||
|
||||
1. `common-dnd-system.md`, without cache control;
|
||||
2. `task.md`, without cache control;
|
||||
3. `instructions.md`, with ephemeral cache control;
|
||||
4. `candidates.md`, without cache control; and
|
||||
5. `common-dnd-transcript.md`, with ephemeral cache control.
|
||||
|
||||
Remove the normalizer system message's current cache control. Preserve the
|
||||
candidate-before-evidence ordering: the transcript windows support evaluation
|
||||
of the candidate collection and are not a cross-lane transcript prefix.
|
||||
|
||||
Do not change input material construction, transcript windowing, prompt
|
||||
fragments, module implementations, or response handling in either package.
|
||||
|
||||
### Stage 2 completion criteria
|
||||
|
||||
- Scene chunking renders references separately before the task and the full
|
||||
transcript exactly once at the end.
|
||||
- NPC normalization renders candidates and transcript windows in separate
|
||||
messages, in that order, with the agreed cache boundaries.
|
||||
- Both prompts retain their current identities, schemas, inputs, and runtime
|
||||
behavior.
|
||||
|
||||
## Stage 3: Consolidate Prompt-Ordering Verification
|
||||
|
||||
Read and follow `docs/policy/testing.md` before modifying tests. The meaningful
|
||||
risk is loss of the shared rendered prefix or incorrect placement of
|
||||
request-specific material, not edits to prompt prose.
|
||||
|
||||
Add one family-level extraction invariant test in
|
||||
`internal/modules/dnd/register/prompt_cache_test.go` (or an equivalently scoped
|
||||
new register-package test file):
|
||||
|
||||
- register the complete D&D prompt asset family in a real in-memory
|
||||
`llm.AssetRegistry`;
|
||||
- construct a Scriptorium engine with a local, non-networked test profile;
|
||||
- prepare all six extraction prompts using their exported prompt IDs and
|
||||
versions;
|
||||
- supply the same unique transcript, players, party, and glossary sentinels to
|
||||
every prompt, plus only the lane-specific required inputs such as the NPC
|
||||
registry and spell catalog;
|
||||
- locate the one rendered message containing the transcript sentinel rather
|
||||
than locating it through prompt prose or a hard-coded phrase;
|
||||
- assert that the transcript sentinel occurs exactly once in each prepared
|
||||
prompt;
|
||||
- compare the rendered messages from the start through the transcript message
|
||||
across all six prompts, including role, full rendered content, and complete
|
||||
cache-control metadata;
|
||||
- assert that the shared prefix contains exactly the four agreed messages and
|
||||
that at least one lane-specific message follows it in every extraction
|
||||
prompt; and
|
||||
- assert that NPC-registry and spell-catalog sentinels, where supplied, occur
|
||||
only after the transcript.
|
||||
|
||||
Use this family-level test as the single owner of the cross-lane prefix policy.
|
||||
Do not duplicate the same prefix assertion in every extractor package.
|
||||
|
||||
Revise the existing prompt-asset tests in:
|
||||
|
||||
- `internal/modules/dnd/extract/npcs/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/itemevents/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/scenedescriptions/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/combatturns/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/npcinteractions/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/extract/spells/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/chunk/scenes/scriptorium_assets_test.go`
|
||||
- `internal/modules/dnd/normalize/npcs/scriptorium_assets_test.go`
|
||||
|
||||
Remove transcript-last expectations that contradict the new extraction policy.
|
||||
Keep package-local coverage only where it protects package-owned behavior, such
|
||||
as prompt/schema registration, declared shared assets, isolation and placement
|
||||
of package-specific input material, cache boundaries unique to the scene
|
||||
chunker or NPC normalizer, and non-disclosure properties. Consolidate or remove
|
||||
positional and prose-marker checks already covered by the family-level
|
||||
invariant.
|
||||
|
||||
Tests may search for unique input sentinels created by the test. They must not
|
||||
require the presence of specific words, phrases, sentences, or other prose in a
|
||||
prompt asset. Do not introduce golden snapshots of complete prompts or a test
|
||||
that treats harmless wording changes as failures.
|
||||
|
||||
### Stage 3 completion criteria
|
||||
|
||||
- One durable family-level test fails if any extraction lane moves
|
||||
lane-specific material ahead of the transcript or changes the common
|
||||
rendered prefix.
|
||||
- Local asset tests protect only nonredundant package behavior and the two
|
||||
purpose-specific prompt sequences.
|
||||
- The test suite contains no prompt-language change detector.
|
||||
- All prompt tests remain deterministic, offline, and independent of
|
||||
credentials.
|
||||
|
||||
## Stage 4: Update Canonical Internal Documentation
|
||||
|
||||
Update `docs/internal/llm.md` as the generic owner of prompt-cache mechanics:
|
||||
|
||||
- explain that backend reuse depends on identical preceding roles, rendered
|
||||
bytes, and cache metadata, not merely equivalent semantics;
|
||||
- state the general sibling-prompt ordering rule: universal shared context,
|
||||
request source material, then module-specific suffixes;
|
||||
- explain that a cache boundary should be placed at a useful reusable prefix
|
||||
and that redundant intermediate boundaries add no value;
|
||||
- retain the rule that prompt-family owners may choose a different sequence
|
||||
when their inputs and reuse pattern differ; and
|
||||
- replace the current statement that D&D extraction puts transcripts last with
|
||||
a link to the D&D-specific policy.
|
||||
|
||||
Update `docs/internal/dnd.md` as the concrete owner of maintained D&D prompt
|
||||
composition:
|
||||
|
||||
- document the four-message extraction prefix and its cache boundaries;
|
||||
- state that extraction-evidence policy, generated NPC registries, catalogs,
|
||||
tasks, and instructions follow the transcript because they are not universal
|
||||
across all extraction lanes;
|
||||
- summarize the scene-chunking and NPC-normalization exceptions and their
|
||||
rationale; and
|
||||
- retain existing rules about shared-asset reuse, canonical reference input,
|
||||
prompt fingerprints, and prompt behavior.
|
||||
|
||||
Do not update the README, configuration reference, architecture policy, CLI
|
||||
documentation, or integration contracts. Prompt message order is an internal
|
||||
runtime policy and has no user-visible configuration or wire-contract change.
|
||||
|
||||
### Stage 4 completion criteria
|
||||
|
||||
- No current-behavior document still says that D&D extraction transcripts are
|
||||
final messages.
|
||||
- Generic cache mechanics and D&D-specific ordering each have one canonical
|
||||
owner.
|
||||
- Documentation describes implemented behavior once the prompt changes land
|
||||
and does not duplicate volatile manifest inventories unnecessarily.
|
||||
|
||||
## Stage 5: Verification
|
||||
|
||||
Inspect representative prepared prompts to confirm that the test inputs produce
|
||||
the intended shared prefix and lane-specific suffixes. Then run:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
go test ./internal/modules/dnd/chunk/scenes
|
||||
go test ./internal/modules/dnd/extract/...
|
||||
go test ./internal/modules/dnd/normalize/npcs
|
||||
go test ./internal/modules/dnd/register
|
||||
go test ./internal/modules/dnd/...
|
||||
go test ./internal/modules/integration/...
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
Provider and prepared-pipeline boundaries defensively copy `LaneIDs`.
|
||||
The focused commands should be used while implementing their respective
|
||||
stages; the repository-wide commands are the final acceptance gate.
|
||||
|
||||
Add an optional output-profile option-validation callback to
|
||||
`OutputEncoderRegistry`:
|
||||
## Expected Compatibility Effects
|
||||
|
||||
```go
|
||||
type OutputProfileOptionContext struct {
|
||||
LaneIDs []string
|
||||
}
|
||||
|
||||
type OutputProfileOptionValidator func(
|
||||
OutputProfileOptionContext,
|
||||
map[string]any,
|
||||
) error
|
||||
```
|
||||
|
||||
Add `RegisterBuilderWithProfileValidation(spec, validateOptions,
|
||||
validateProfile, builder)` and make existing output registration methods
|
||||
delegate to it with no profile callback. The registry passes defensive copies
|
||||
to validation. The callback receives the complete configured lane-ID set before
|
||||
invocation-level `--only` filtering. The production JSON output uses it only to
|
||||
prove that every configured evidence lane exists. Keep the extension generic:
|
||||
the pipeline supplies lane identities, while the output module interprets its
|
||||
own options. Build that lane set from the normalized legacy-or-steps profile
|
||||
before selection and reject duplicate configured lane IDs through the existing
|
||||
pipeline identity rules.
|
||||
|
||||
Extend the production JSON output options with the nested
|
||||
`evidence_context` object:
|
||||
|
||||
- omission disables the feature;
|
||||
- `enabled` is required when the object is present;
|
||||
- `enabled: false` permits no `lanes` or `window_units` fields;
|
||||
- `enabled: true` requires a non-empty `lanes` array;
|
||||
- lane values are strings, trimmed, non-empty, unique after trimming, and
|
||||
normalized to lexical order;
|
||||
- `window_units` is an optional non-negative integer with default `3`; and
|
||||
- outer and nested unknown fields and incompatible YAML value types remain
|
||||
strict configuration errors.
|
||||
|
||||
The JSON encoder implements the policy provider from its decoded immutable
|
||||
options. Pipeline resolution invokes its profile validator against all
|
||||
configured steps, so an unknown evidence lane fails even when another lane is
|
||||
selected with `--only`.
|
||||
|
||||
During `pipeline.Prepare`, after constructing the output encoder:
|
||||
|
||||
1. obtain and defensively normalize an enabled policy;
|
||||
2. intersect its configured IDs with the effective prepared lanes, treating
|
||||
allowlisted lanes removed by invocation-level filtering as inactive;
|
||||
3. require an artifact-evidence registration for each active lane kind;
|
||||
4. prove that its projector Go type exactly matches the active lane's registered
|
||||
artifact codec type; and
|
||||
5. retain an immutable private evidence plan on `PreparedPipeline`.
|
||||
|
||||
Duplicate or empty provider values, a missing evidence registry for an active
|
||||
lane, unsupported active artifact kinds, and type mismatches fail preparation
|
||||
with pipeline/output/lane context. A disabled or non-participating output
|
||||
encoder creates no evidence plan and preserves existing preparation behavior.
|
||||
The private plan retains both the full configured allowlist for publication and
|
||||
the active lane/projector intersection for execution.
|
||||
|
||||
Register D&D evidence projectors for all six current artifact kinds. Each
|
||||
projector returns copies of the artifact's direct references in record order:
|
||||
spells, NPCs, combat turns, item events, NPC interactions, and the singular
|
||||
reference from each scene description. Scene descriptions gain capability but
|
||||
remain excluded unless their configured lane ID is allowlisted.
|
||||
|
||||
Stage tests:
|
||||
|
||||
- Registry tests cover nil, blank, duplicate, exact-type, defensive-copy, and
|
||||
deterministic discovery behavior.
|
||||
- JSON option tests cover disabled, enabled/default-window, explicit zero
|
||||
window, normalization, duplicates, unknown fields, and invalid types.
|
||||
- Preparation tests cover selected lanes across steps, unknown lanes,
|
||||
unsupported kinds, projector/codec type mismatch, disabled behavior, and
|
||||
defensive policy ownership.
|
||||
- Resolution/preparation tests prove a valid allowlist survives `--only`, an
|
||||
excluded lane contributes no active projector, and a genuinely unknown
|
||||
configured lane still fails profile resolution.
|
||||
- D&D registration tests prove every production D&D artifact kind has the
|
||||
expected evidence capability without testing individual field loops
|
||||
redundantly.
|
||||
- One table-driven D&D projector test supplies representative values for all
|
||||
six artifact kinds and proves plural and singular references are copied
|
||||
without aliasing or semantic rewriting.
|
||||
|
||||
Stage completion:
|
||||
|
||||
- `go test ./internal/framework/pipeline`
|
||||
- `go test ./internal/modules/generic/output/json`
|
||||
- `go test ./internal/modules/dnd/register`
|
||||
- `go test ./internal/cli`
|
||||
|
||||
## Stage 2: Define And Build The Evidence-Context Artifact
|
||||
|
||||
Add a domain-neutral `internal/framework/evidencecontext` package that owns the
|
||||
durable model, JSON Schema, strict codec, projection algorithm, and these exact
|
||||
identities:
|
||||
|
||||
- artifact kind `source/evidence-context`;
|
||||
- media type `application/json`;
|
||||
- schema ID `notarius.source.evidence_context`;
|
||||
- schema name `notarius_source_evidence_context_v1`; and
|
||||
- schema version `v1`.
|
||||
|
||||
Use these package-level model and build contracts:
|
||||
|
||||
```go
|
||||
type Document struct {
|
||||
SourceID string
|
||||
SourceDigest string
|
||||
WindowUnits int
|
||||
SelectedLanes []string
|
||||
Contexts []Context
|
||||
}
|
||||
|
||||
type Context struct {
|
||||
ContextRef source.SourceRef
|
||||
EvidenceRefs []EvidenceRef
|
||||
Units []source.SourceUnit
|
||||
}
|
||||
|
||||
type EvidenceRef struct {
|
||||
LaneID string
|
||||
SourceRef source.SourceRef
|
||||
}
|
||||
|
||||
type LaneEvidence struct {
|
||||
LaneID string
|
||||
SourceRefs []source.SourceRef
|
||||
}
|
||||
|
||||
type BuildRequest struct {
|
||||
Source *source.SourceDocument
|
||||
WindowUnits int
|
||||
SelectedLanes []string
|
||||
LaneEvidence []LaneEvidence
|
||||
}
|
||||
```
|
||||
|
||||
Apply the JSON field names shown below. Provide `Build(BuildRequest)`,
|
||||
`Serialize(BuildRequest)`, and a `Codec` with the same identity/encode/decode
|
||||
responsibilities as the chunk-map codec.
|
||||
|
||||
The v1 payload has this exact shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_id": "session-alpha",
|
||||
"source_digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"window_units": 0,
|
||||
"selected_lanes": ["npcs"],
|
||||
"contexts": [
|
||||
{
|
||||
"context_ref": {
|
||||
"source_id": "session-alpha",
|
||||
"start_unit_id": 13,
|
||||
"end_unit_id": 13
|
||||
},
|
||||
"evidence_refs": [
|
||||
{
|
||||
"lane_id": "npcs",
|
||||
"source_ref": {
|
||||
"source_id": "session-alpha",
|
||||
"start_unit_id": 13,
|
||||
"end_unit_id": 13
|
||||
}
|
||||
}
|
||||
],
|
||||
"units": [
|
||||
{
|
||||
"id": 13,
|
||||
"kind": "transcript_segment",
|
||||
"text": "The party meets Rowan.",
|
||||
"ref": {
|
||||
"source_id": "session-alpha",
|
||||
"start_unit_id": 13,
|
||||
"end_unit_id": 13
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
All displayed fields are required. `selected_lanes`, `contexts`,
|
||||
`evidence_refs`, and `units` encode as arrays rather than `null`; `contexts`
|
||||
may be empty. Each unit uses the existing `source.SourceUnit` JSON shape with
|
||||
required `id`, `kind`, `text`, and `ref`, plus optional JSON-shaped `metadata`.
|
||||
Fixed objects reject unknown fields; metadata remains an arbitrary JSON object.
|
||||
|
||||
The builder accepts the validated source document, selected lane IDs, effective
|
||||
window, and lane-attributed direct references, then:
|
||||
|
||||
1. requires a non-negative window and a non-empty, trimmed, unique selected
|
||||
lane set, then stores that set in lexical order;
|
||||
2. requires every `LaneEvidence.LaneID` to belong to the selected set;
|
||||
3. validates the source document, recomputes its semantic digest, and requires
|
||||
it to equal `SourceDocument.Digest`;
|
||||
4. validates every direct reference against one `source.DocumentIndex`;
|
||||
5. deduplicates exact `(lane_id, source_ref)` contributions;
|
||||
6. resolves endpoints to document positions;
|
||||
7. expands each side without integer overflow and clips at document bounds;
|
||||
8. sorts by expanded document position with deterministic lane/reference
|
||||
tie-breakers;
|
||||
9. merges overlapping or position-contiguous expanded intervals;
|
||||
10. unions and deterministically sorts each merged context's direct
|
||||
contributions; and
|
||||
11. deep-clones the corresponding source units and JSON-shaped metadata.
|
||||
|
||||
Contexts are disjoint and ordered by document position, so a source unit occurs
|
||||
at most once in the document. `context_ref` identifies the first and last
|
||||
included units; `evidence_refs` retains only original citations. An empty
|
||||
reference collection produces the same source identity, window, sorted
|
||||
allowlist, and an explicit empty contexts array.
|
||||
|
||||
Projection failures identify only structural scope such as lane and reference
|
||||
position. They must not include source text, metadata values, raw serialized
|
||||
artifacts, or unrelated paths.
|
||||
|
||||
The codec must validate its model before encoding, produce deterministic JSON,
|
||||
strictly decode the checked-in schema, and return independently owned values.
|
||||
The JSON output encoder remains responsible for pretty-printing the logical
|
||||
file with its standard trailing newline. Follow the existing chunk-map
|
||||
package's separation between model, builder, codec, schema asset, and contract
|
||||
tests where useful, without coupling the two artifact formats.
|
||||
|
||||
Stage tests:
|
||||
|
||||
- A table-driven builder suite covers zero and nonzero windows, boundary
|
||||
clipping, non-monotonic unit IDs, separate gaps, overlapping and contiguous
|
||||
windows, duplicate contributions, multiple lanes, stable ordering, empty
|
||||
contexts, invalid selected/contributing lanes, source-digest mismatch, and
|
||||
invalid references.
|
||||
- Ownership tests prove output mutation cannot affect the source document or
|
||||
projector inputs, including nested metadata.
|
||||
- Codec tests cover round trip, required arrays, schema identity, malformed and
|
||||
trailing JSON, unknown fixed fields, invalid ordering/ranges, mismatched
|
||||
source identities, and independently owned decoded metadata.
|
||||
- Use structured assertions and a compact valid fixture; do not add a large
|
||||
transcript golden file.
|
||||
|
||||
Stage completion:
|
||||
|
||||
- `go test ./internal/framework/evidencecontext`
|
||||
|
||||
## Stage 3: Integrate Projection With Runner Output
|
||||
|
||||
Extend `contracts.OutputRequest` with an optional
|
||||
`EvidenceContext *SerializedArtifact` field and clone it at every ownership
|
||||
handoff, following the existing chunk-map pointer pattern.
|
||||
|
||||
After lane execution and final manifest population, but before invoking the
|
||||
output encoder, the runner must:
|
||||
|
||||
1. skip all work when the prepared evidence plan is absent;
|
||||
2. index accepted `NormalizeOutputs` by their globally unique lane IDs and fail
|
||||
on an internal duplicate rather than silently overwrite it;
|
||||
3. for each selected lane with an output, verify its source and artifact kind,
|
||||
decode it through the prepared artifact codec registry, and invoke the
|
||||
prepared typed projector;
|
||||
4. build and serialize the evidence document through the evidence-context
|
||||
package; and
|
||||
5. pass a defensive serialized-artifact copy to the output encoder.
|
||||
|
||||
Selected lanes without normalized output contribute nothing. Normalize
|
||||
rejections remain successful pipeline outcomes; evidence projection does not
|
||||
inspect rejected candidates. An invalid accepted reference, incompatible
|
||||
serialized artifact, projection type failure, or evidence serialization failure
|
||||
is an output-stage framework error before logical files are returned or
|
||||
physically published.
|
||||
|
||||
At this external-content consumption boundary, do not propagate artifact-codec
|
||||
or metadata-cloning errors with `%w` when their text could contain artifact
|
||||
fields or source metadata. Return fixed, actionable categories scoped by lane
|
||||
and operation; detailed codec errors remain available to direct trusted
|
||||
callers and their focused tests.
|
||||
|
||||
Add an allowlisted debug summary containing only evidence artifact identity,
|
||||
selected lanes, window, context count, unit count, and source digest. Do not
|
||||
duplicate transcript text or source-unit metadata into a new evidence-specific
|
||||
debug envelope. Existing normalized-output debug behavior remains unchanged.
|
||||
|
||||
The output artifact is not a normalized lane, generated reference, checkpoint,
|
||||
or manifest normalized-output entry. It does not alter normalized-output,
|
||||
rejection, or warning counts. Resume continues to reuse normalized checkpoints;
|
||||
evidence is deterministically rebuilt during the always-executed output stage.
|
||||
|
||||
Stage tests:
|
||||
|
||||
- Runner tests use a real codec, evidence projector, and small capturing output
|
||||
encoder to prove selected-lane union, absent/rejected lane omission, invalid
|
||||
accepted-reference failure, output-request defensive ownership, and no work
|
||||
when disabled.
|
||||
- Include one checkpoint-reused normalized-output case to prove evidence is
|
||||
reconstructed identically without retaining typed normalize values.
|
||||
- Confirm projection failures prevent output encoding and return a failed
|
||||
manifest without changing rejection semantics.
|
||||
|
||||
Stage completion:
|
||||
|
||||
- `go test ./internal/framework/pipeline`
|
||||
|
||||
## Stage 4: Publish Through The JSON Bundle
|
||||
|
||||
Teach the production JSON encoder to recognize the optional evidence-context
|
||||
artifact, verify its exact kind, media type, schema identity, schema digest, and
|
||||
payload validity through the evidence-context codec, and emit:
|
||||
|
||||
- logical file `evidence-context.json`; and
|
||||
- optional `index.json` descriptor field `evidence_context`.
|
||||
|
||||
The descriptor uses the same six fields as `chunk_map`:
|
||||
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
|
||||
`schema_version`. Refactor the encoder's private descriptor representation only
|
||||
as needed to share that shape; do not change the existing `chunk_map` wire
|
||||
contract. Evidence output is ordered with the encoder's other fixed logical
|
||||
files, remains a non-lane artifact, and is present with an empty contexts array
|
||||
when enabled but no selected lane produces references.
|
||||
|
||||
The JSON encoder's validation boundary returns a fixed content-safe evidence
|
||||
artifact error rather than propagating decoder or schema diagnostics that could
|
||||
echo transcript text or metadata. Direct evidence-context codec tests retain
|
||||
detailed structural errors.
|
||||
|
||||
Update the maintained complete D&D configuration to enable evidence context
|
||||
with window `3` for `item-events`, `npcs`, `spells`, `combat-turns`, and
|
||||
`npc-interactions`. Deliberately omit `scene-descriptions`. Keep the minimal
|
||||
configuration disabled by omission.
|
||||
|
||||
Stage tests:
|
||||
|
||||
- JSON encoder tests own descriptor shape, exact logical filename, identity
|
||||
checking, empty evidence publication, and disabled bundle stability.
|
||||
- One assembled production D&D test uses multiple selected lanes with
|
||||
overlapping references and non-monotonic unit IDs, decodes the published
|
||||
artifact through its production codec, and proves union/deduplication and
|
||||
scene-description exclusion.
|
||||
- A second narrow case explicitly allowlists a scene-description lane to prove
|
||||
capability is opt-in rather than hard-coded exclusion.
|
||||
- Existing index, chunk-map, lane, manifest, warning, and rejection tests remain
|
||||
the owners of their current formats; do not repeat their full matrices.
|
||||
|
||||
Stage completion:
|
||||
|
||||
- `go test ./internal/modules/generic/output/json`
|
||||
- `go test ./internal/modules/dnd/...`
|
||||
- `go test ./internal/modules/integration`
|
||||
- `go test ./internal/cli`
|
||||
|
||||
## Stage 5: Publish Current-Behavior Documentation
|
||||
|
||||
After implementation and behavioral tests pass, update canonical documentation:
|
||||
|
||||
- `docs/config.md` owns the nested JSON output options, defaults, strict
|
||||
validation, required lane allowlist, and a small configuration snippet.
|
||||
- A new `docs/integrations/evidence-context.md` owns the complete v1 payload,
|
||||
identities, direct-evidence versus context semantics, ordering,
|
||||
compatibility, and a compact valid example.
|
||||
- `docs/integrations/json-output.md` owns the optional logical file and
|
||||
`index.json` descriptor; link to the evidence contract rather than repeating
|
||||
its payload.
|
||||
- `docs/operations.md` owns durable source-content sensitivity, permissions,
|
||||
retention, and the possibility that selected lanes cover most of a
|
||||
transcript.
|
||||
- `docs/consumers/subprocess.md` explains discovery through the optional index
|
||||
descriptor and requires consumers to treat `evidence_refs`, not expanded
|
||||
context bounds, as citations.
|
||||
- `docs/policy/architecture.md` records the generic typed evidence-projection
|
||||
boundary and output ownership without adding D&D or wire-format detail.
|
||||
- `docs/internal/pipeline.md` and `docs/internal/modules.md` describe the typed
|
||||
evidence registry, preparation checks, reconstruction from serialized
|
||||
normalize outputs, and output-stage ownership without restating public wire
|
||||
fields.
|
||||
|
||||
Update only the smallest orientation links needed for discoverability. Do not
|
||||
add a CLI flag, configuration environment override, or duplicate the complete
|
||||
configuration outside `examples/`.
|
||||
|
||||
After all current-behavior documentation is accurate:
|
||||
|
||||
- set [the feature roadmap](evidence.md) status to `Implemented`;
|
||||
- set this plan's status to `Completed`; and
|
||||
- leave the integration and configuration documents, not either roadmap, as
|
||||
the canonical implemented contract.
|
||||
|
||||
Final verification:
|
||||
|
||||
- `git diff --check`
|
||||
- `go test ./...`
|
||||
- `go vet ./...`
|
||||
- `go build ./cmd/notarius`
|
||||
- `go test -race ./internal/framework/evidencecontext ./internal/framework/pipeline ./internal/modules/generic/output/json ./internal/modules/dnd/... ./internal/modules/integration ./internal/cli`
|
||||
- Reordered prompt content changes prompt fingerprints and may intentionally
|
||||
cause one-time cold misses for affected development checkpoints.
|
||||
- Prompt IDs, prompt versions, schema identities, checkpoint format, durable
|
||||
artifacts, and public behavior do not change.
|
||||
- Backend cache reuse remains provider-dependent, but Notarius supplies a
|
||||
longer common extraction prefix that includes the transcript chunk.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The plan fixes the configuration shape and defaults, typed projection
|
||||
boundary, preparation timing, durable schema and identities, range-union
|
||||
algorithm, failure semantics, JSON discovery, D&D coverage, documentation
|
||||
ownership, and test boundaries.
|
||||
None. The message sequences, cache boundaries, test ownership, documentation
|
||||
owners, and compatibility policy are specified above.
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
# Subprocess Integration Contract
|
||||
|
||||
## Status
|
||||
|
||||
Implemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
Make Notarius straightforward to invoke as a subprocess from an orchestrator
|
||||
such as Narratio. A caller should be able to run a configured pipeline, discover
|
||||
the published output bundle without parsing human prose or scanning a
|
||||
directory, and hand selected structured artifacts to a later stage.
|
||||
|
||||
This work strengthens the public CLI boundary. It does not turn Notarius into a
|
||||
Go library, embed Narratio-specific behavior, or change pipeline execution and
|
||||
artifact semantics.
|
||||
|
||||
## Desired End State
|
||||
|
||||
A subprocess caller can:
|
||||
|
||||
1. validate a Notarius configuration and selected pipeline before execution;
|
||||
2. invoke `notarius run` with explicit input, output-root, session, and
|
||||
reference arguments;
|
||||
3. request one versioned, machine-readable success result on standard output;
|
||||
4. use that result to locate the published output bundle;
|
||||
5. discover normalized lane payloads through the bundle's authoritative
|
||||
`index.json`;
|
||||
6. distinguish process failure from successful partial pipeline outcomes; and
|
||||
7. record Notarius run provenance in its own manifest without depending on
|
||||
internal packages, cache formats, debug formats, or human-readable messages.
|
||||
|
||||
The existing human-oriented command output remains the default for interactive
|
||||
use.
|
||||
|
||||
## Machine-Readable Run Result
|
||||
|
||||
`notarius run` supports `--json`. On success, the flag makes standard output
|
||||
contain exactly one JSON object followed by a newline. No human-oriented status
|
||||
line is mixed into that stream.
|
||||
|
||||
The result uses the schema identity `notarius.run-result.v1` and contains:
|
||||
|
||||
| Field | Presence | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `schema_version` | Required | Exactly `notarius.run-result.v1`. |
|
||||
| `run_id` | Required | The Notarius run identifier. |
|
||||
| `pipeline_id` | Required | The effective pipeline identifier. |
|
||||
| `output_directory` | Required | Absolute path to the successfully published output bundle. |
|
||||
| `index_file` | Required for the production JSON output | Logical bundle path `index.json`. |
|
||||
| `normalized_output_count` | Required | Number of final normalized lane outputs returned by the pipeline. |
|
||||
| `rejected_output_count` | Required | Number of recorded rejected outputs. |
|
||||
| `warning_count` | Required | Number of final run warnings returned by the pipeline. |
|
||||
| `validation_status` | Required | The run manifest's final validation status without reinterpretation. |
|
||||
| `debug_directory` | Optional | Absolute debug-bundle path when debug capture was requested and completed. |
|
||||
|
||||
An illustrative successful result is:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "notarius.run-result.v1",
|
||||
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
||||
"pipeline_id": "dnd-session",
|
||||
"output_directory": "/srv/narratio/runs/session-7/notarius/run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
||||
"index_file": "index.json",
|
||||
"normalized_output_count": 6,
|
||||
"rejected_output_count": 2,
|
||||
"warning_count": 1,
|
||||
"validation_status": "approved"
|
||||
}
|
||||
```
|
||||
|
||||
The receipt is a discovery and summary document, not a duplicate output
|
||||
envelope. It does not embed lane payloads, rejection entries, warnings, the run
|
||||
manifest, or output-file contents. Consumers use `index_file` and the existing
|
||||
published JSON output contract for those records.
|
||||
|
||||
The result contract must tolerate future additive optional fields. Any
|
||||
incompatible field or semantic change requires a new run-result schema version.
|
||||
|
||||
## Stream, Publication, And Failure Semantics
|
||||
|
||||
Machine-readable output is emitted only after:
|
||||
|
||||
- the pipeline has completed without a framework error;
|
||||
- all logical output files have been successfully published;
|
||||
- requested debug terminal reporting has completed; and
|
||||
- all result fields are known.
|
||||
|
||||
Writing or encoding the machine-readable result is part of successful command
|
||||
completion. Failure to write it produces the existing runtime-failure exit
|
||||
class.
|
||||
|
||||
With `--json`:
|
||||
|
||||
- successful stdout is exclusively the run-result JSON document;
|
||||
- successful warnings remain on stderr under the existing CLI contract;
|
||||
- syntax and runtime errors retain their existing exit statuses and stderr
|
||||
diagnostics;
|
||||
- consumers treat stdout as a valid result only when the process exits with
|
||||
status 0; failures before result writing emit no result, while a failure
|
||||
during the stdout write may leave incomplete bytes that must be ignored; and
|
||||
- human-readable diagnostic wording is not promoted into a machine contract.
|
||||
|
||||
Without `--json`, current interactive stdout and stderr behavior remains
|
||||
unchanged.
|
||||
|
||||
Successful runs may contain rejected outputs or omit some normalized lanes.
|
||||
That remains a valid pipeline outcome. The run result reports counts, while
|
||||
`index.json`, `rejected.json`, and `warnings.json` remain authoritative for
|
||||
details. Notarius will not add a generic `--fail-on-rejection` policy as part
|
||||
of this work.
|
||||
|
||||
## Output Discovery And Consumer Responsibilities
|
||||
|
||||
The production JSON encoder's `index.json` remains the authoritative mapping
|
||||
from lane IDs to published payloads. A subprocess consumer should:
|
||||
|
||||
- resolve `index_file` beneath `output_directory` and reject path escape;
|
||||
- locate expected outputs by `lane_id`, not by guessing filenames;
|
||||
- check each selected descriptor's media type and schema identity;
|
||||
- decode payloads according to their published integration contracts;
|
||||
- decide which lanes are required or optional for its own later stages; and
|
||||
- retain rejection, warning, and manifest files when they are needed for
|
||||
provenance or review.
|
||||
|
||||
For Narratio, required report inputs and partial-success policy remain Narratio
|
||||
stage configuration and orchestration concerns. Notarius does not acquire
|
||||
knowledge of Narratio stages, manifests, workspace layout, publication policy,
|
||||
or report formats.
|
||||
|
||||
## Invocation Guidance
|
||||
|
||||
The consumer documentation recommends that subprocess callers:
|
||||
|
||||
- use `notarius config validate --pipeline` as an optional preflight;
|
||||
- pass explicit absolute paths for the input, configuration, output root, and
|
||||
CLI-supplied references;
|
||||
- use a stable, non-secret prompt session identifier when useful for provider
|
||||
routing or caching;
|
||||
- capture stdout and stderr separately;
|
||||
- supply credentials through the configured environment mechanism rather than
|
||||
command arguments or generated configuration containing secret values;
|
||||
- place output, cache, debug, and subprocess logs under intentional
|
||||
sensitivity and retention policies; and
|
||||
- treat the Notarius manifest and run-result receipt as provenance while
|
||||
leaving the caller's own manifest authoritative for its stage lifecycle.
|
||||
|
||||
Notarius configuration remains owned by Notarius. An orchestrator may select a
|
||||
configuration and pass supported operational overrides, but should not
|
||||
duplicate the complete Notarius configuration schema.
|
||||
|
||||
## Documentation End State
|
||||
|
||||
- `docs/cli.md` owns `run --json`, stream behavior, and exit semantics;
|
||||
- a new `docs/integrations/run-result.md` owns the versioned run-result wire
|
||||
contract and compatibility policy;
|
||||
- `docs/integrations/json-output.md` remains the sole owner of output-bundle
|
||||
discovery and lane publication;
|
||||
- a new `docs/consumers/subprocess.md` provides the task-oriented invocation and
|
||||
consumption workflow; and
|
||||
- `docs/internal/cli.md` describes how the CLI constructs and emits the result
|
||||
only after successful publication.
|
||||
|
||||
Other documents should link to these owners instead of repeating volatile
|
||||
fields or command details.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- An ordinary successful `run` retains its existing human-readable output.
|
||||
- A successful `run --json` emits one valid `notarius.run-result.v1` document
|
||||
and no human prose on stdout.
|
||||
- Relative configured or overridden output and debug roots are reported as
|
||||
absolute bundle paths.
|
||||
- The receipt identifies the production JSON bundle entry point without
|
||||
copying its lane descriptors or payloads.
|
||||
- Warning-bearing and rejection-bearing runs remain successful and report
|
||||
accurate counts.
|
||||
- Syntax, configuration, provider, pipeline, publication, debug, and result
|
||||
writing failures retain the correct nonzero exit class. Consumers are
|
||||
explicitly required to ignore stdout from a nonzero invocation.
|
||||
- The implementation does not expose internal Go types or couple generic CLI
|
||||
code to D&D or Narratio concepts.
|
||||
- Public and internal documentation assigns each new contract to one canonical
|
||||
owner.
|
||||
- Offline behavioral tests protect the structured-output contract, default
|
||||
human behavior, absolute path reporting, stream separation, and failure to
|
||||
serialize or write the success result without duplicating lower-level output
|
||||
encoder tests.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
The following may be useful later but are not prerequisites for the Narratio
|
||||
integration:
|
||||
|
||||
- a result-file flag in addition to machine-readable stdout;
|
||||
- a JSON failure envelope or stable machine-readable error taxonomy;
|
||||
- caller-supplied Notarius run IDs or exact output-bundle paths;
|
||||
- a generic `--fail-on-rejection` or required-lane CLI policy;
|
||||
- a public Go client package or importable Narratio adapter;
|
||||
- Narratio stage, configuration, manifest, or report-generation changes;
|
||||
- `notarius version --json`;
|
||||
- installable or queryable artifact JSON Schemas;
|
||||
- signal-aware CLI contexts and graceful SIGINT or SIGTERM handling;
|
||||
- packaged release artifacts and a broader application-versioning policy.
|
||||
|
||||
These items should be promoted only in response to a demonstrated integration
|
||||
need rather than bundled into the initial subprocess contract.
|
||||
Reference in New Issue
Block a user