Compare commits
9 Commits
v0.1.0
...
e19cc02c4d
| Author | SHA1 | Date | |
|---|---|---|---|
| e19cc02c4d | |||
| 8a5419448f | |||
| c8217549a8 | |||
| 2130414899 | |||
| 7f83a20fa6 | |||
| 317ab0472d | |||
| e5eb0ba5c8 | |||
| b95af4f87d | |||
| 11073b613c |
@@ -1,3 +1,6 @@
|
||||
Please carefully review the documents in `docs/policy` before making any changes to this repository.
|
||||
- `architecture.md` provides the canonical high-level architecture policy for this repository.
|
||||
- `documentation.md` provides the canonical documentation policy for this repository.
|
||||
Please review `docs/internal/overview.md` for initial orientation in this repository.
|
||||
|
||||
Additionally, please carefully review the relevant documents in `docs/policy` before making any changes to this repository.
|
||||
- `development.md` defines the contributor workflow for this application.
|
||||
- `architecture.md` provides the canonical high-level architecture policy for this repository, and should be reviewed before writing or changing any code.
|
||||
- `documentation.md` provides the canonical documentation policy for this repository, and should be reviewed before writing or changing any documentation.
|
||||
|
||||
@@ -117,7 +117,7 @@ go run ./cmd/notarius pipelines list \
|
||||
The production CLI currently registers these module keys:
|
||||
|
||||
- input: `seriatim`
|
||||
- chunk: `generic`
|
||||
- chunk: `generic`, `dnd/scenes`
|
||||
- extract: `dnd/spells`
|
||||
- merge: `appendorder`
|
||||
- normalize: `noop`
|
||||
|
||||
@@ -167,6 +167,7 @@ one configured profile.
|
||||
| --- | --- | --- |
|
||||
| input | `seriatim` | Reads Seriatim transcript JSON. |
|
||||
| chunk | `generic` | Splits source units into ordered chunks. |
|
||||
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
||||
| extract | `dnd/spells` | Extracts `dnd.spell_cast` artifacts. |
|
||||
| merge | `appendorder` | Keeps candidates in append order. |
|
||||
| normalize | `noop` | Passes merged artifacts through unchanged. |
|
||||
@@ -178,6 +179,9 @@ The `generic` chunker accepts:
|
||||
- `overlap_units`: non-negative integer, default `0`, and must be less than
|
||||
`max_units`.
|
||||
|
||||
The `dnd/scenes` chunker requires transcript source capabilities, calls the
|
||||
configured structured LLM provider, and does not accept module options.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
`diagnostics` fields:
|
||||
|
||||
@@ -20,6 +20,11 @@ A production module package should provide:
|
||||
Module specs should describe capabilities accurately. Resolution uses specs to
|
||||
reject incompatible pipelines before execution.
|
||||
|
||||
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
|
||||
when they need model-backed chunking. The pipeline runner validates generic
|
||||
chunk result invariants before extraction; module-owned policies may be stricter
|
||||
but must stay within the module package.
|
||||
|
||||
## `seriatim` Input
|
||||
|
||||
Package: `internal/modules/input/seriatim`
|
||||
@@ -54,6 +59,37 @@ Provides:
|
||||
|
||||
- `chunks`
|
||||
|
||||
## `dnd/scenes` Chunker
|
||||
|
||||
Package: `internal/modules/chunk/dnd/scenes`
|
||||
|
||||
The `dnd/scenes` chunker uses the structured LLM client to divide transcript
|
||||
source units into coherent D&D scenes. It renders embedded prompts, loads the
|
||||
embedded structured response schema, validates model-authored source-unit
|
||||
boundaries, and converts each scene into a deterministic source chunk.
|
||||
|
||||
Requires:
|
||||
|
||||
- `source.transcript`
|
||||
|
||||
Provides:
|
||||
|
||||
- `chunks`
|
||||
- `chunks.scenes`
|
||||
|
||||
Options: none. Non-empty options are rejected.
|
||||
|
||||
The chunker enforces full source-unit coverage from the first source unit to the
|
||||
last, exact source-unit IDs, sequential contiguous scenes, and no overlap. It
|
||||
assigns chunk IDs such as `scene-000001` and stores scene metadata including
|
||||
title, primary mode, participants, summary, boundary note, confidence, boundary
|
||||
unit IDs, and unit count. Boundary caveats become warnings with reason code
|
||||
`scene_boundary_caveat`.
|
||||
|
||||
Malformed model output fails explicitly rather than falling back to another
|
||||
chunker. The chunker exposes prompt and response-schema provenance through its
|
||||
metadata provider without raw prompts, raw schemas, source text, or secrets.
|
||||
|
||||
## `dnd/spells` Extractor
|
||||
|
||||
Package: `internal/modules/extract/dnd/spells`
|
||||
|
||||
@@ -73,8 +73,34 @@ The runner:
|
||||
2. builds the input adapter and parses the raw input into a source document;
|
||||
3. validates the source document;
|
||||
4. builds the chunker and produces source chunks;
|
||||
5. runs each selected artifact lane in sorted resolved order;
|
||||
6. builds the output encoder and validates logical output file names.
|
||||
5. validates source chunks against framework invariants;
|
||||
6. runs each selected artifact lane in sorted resolved order;
|
||||
7. builds the output encoder and validates logical output file names.
|
||||
|
||||
## Chunk Results
|
||||
|
||||
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
|
||||
with the validated source document, the structured LLM client, the configured
|
||||
LLM profile, module options, and run metadata. Deterministic and LLM-backed
|
||||
chunkers use the same contract; provider construction stays outside chunk
|
||||
modules.
|
||||
|
||||
After `Chunk` returns, the runner appends chunker warnings before returning any
|
||||
chunker error. When chunking succeeds, the runner validates generic chunk
|
||||
invariants before running extractors:
|
||||
|
||||
- chunk IDs must be non-empty and unique in the chunk result;
|
||||
- each chunk `SourceID` must match the source document ID;
|
||||
- each chunk `Index` must match its zero-based returned order;
|
||||
- each chunk must contain at least one source unit;
|
||||
- a chunk must not repeat a source unit;
|
||||
- every chunk source unit must exist in the source document;
|
||||
- source units inside each chunk must appear in source-document order.
|
||||
|
||||
The framework does not require complete source-unit coverage and does not reject
|
||||
overlap between different chunks. Stricter policies, such as full coverage or
|
||||
non-overlap, belong to individual chunk modules when they are part of that
|
||||
module's contract.
|
||||
|
||||
Within an artifact lane, the runner:
|
||||
|
||||
|
||||
104
docs/roadmap/chunk.md
Normal file
104
docs/roadmap/chunk.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Chunk Module Roadmap
|
||||
|
||||
Current Notarius behavior is documented in the canonical README, CLI,
|
||||
configuration, operations, internal, and integration docs. This roadmap records
|
||||
future chunk-module behavior only.
|
||||
|
||||
## Goal
|
||||
|
||||
Chunk modules should be a clear module-author boundary, and LLM-backed chunking
|
||||
should be a first-class capability.
|
||||
|
||||
The immediate target is a D&D-specific scene chunker that divides transcript
|
||||
source units into coherent scenes before extraction. The broader target is that
|
||||
any chunk module can be implemented as a black box when it satisfies the
|
||||
framework chunk contract.
|
||||
|
||||
## Target Chunk Contract
|
||||
|
||||
The framework chunk contract should support deterministic and LLM-backed
|
||||
chunkers through the same module interface.
|
||||
|
||||
Chunkers should receive runtime dependencies from the runner, including the
|
||||
structured LLM client when a chunker needs model calls. Chunkers should not
|
||||
construct provider clients internally.
|
||||
|
||||
The framework should validate these result invariants for every chunk module:
|
||||
|
||||
- chunk IDs are non-empty and unique within a run;
|
||||
- each chunk references the input source document ID;
|
||||
- chunk indexes are deterministic and sequential in returned order;
|
||||
- each chunk contains at least one source unit;
|
||||
- each chunk source unit comes from the source document;
|
||||
- source units within each chunk appear in source-document order.
|
||||
|
||||
The framework should not require complete source-unit coverage and should not
|
||||
forbid overlap between chunks. Individual chunk modules may enforce stricter
|
||||
policies, such as full coverage or non-overlap, when those policies are part of
|
||||
the module's own contract.
|
||||
|
||||
Chunk metadata should remain flexible and module-owned. Framework code should
|
||||
preserve chunk metadata and pass it to downstream modules, but it should not
|
||||
adopt transcript-specific or D&D-specific metadata fields.
|
||||
|
||||
## D&D Scene Chunker Target
|
||||
|
||||
The D&D scene chunker should live under
|
||||
`internal/modules/chunk/dnd/scenes` and use the module key `dnd/scenes`.
|
||||
|
||||
It should require transcript source capabilities and provide the generic
|
||||
`chunks` capability plus a scene-specific chunk capability. D&D scene-boundary
|
||||
prompt logic, response-schema interpretation, and stricter scene policies belong
|
||||
inside the module.
|
||||
|
||||
The module should use the framework structured LLM client for scene-boundary
|
||||
detection. The model response should describe source-unit boundaries and useful
|
||||
scene metadata; the Go module should validate the response and convert it into
|
||||
`contracts.SourceChunk` values.
|
||||
|
||||
For `dnd/scenes`, the module-owned policy should be:
|
||||
|
||||
- cover the full source document from first source unit to last source unit;
|
||||
- return sequential, contiguous, non-overlapping scenes;
|
||||
- use exact source-unit IDs for boundaries;
|
||||
- fail with actionable errors for malformed model output rather than silently
|
||||
falling back to a generic chunker.
|
||||
|
||||
Scene chunk IDs and indexes should be assigned by the module, not trusted from
|
||||
model output. Useful scene information should be stored in chunk metadata, such
|
||||
as title, primary mode, participants, summary, boundary note, and boundary
|
||||
confidence. Overall boundary caveats should be surfaced as chunker warnings.
|
||||
|
||||
## Draft Asset Target
|
||||
|
||||
Initial D&D scene chunker prompt and schema drafts exist under
|
||||
`internal/modules/chunk/dnd/scenes/assets`. They should be revised before the
|
||||
module is implemented.
|
||||
|
||||
The response schema should be versioned and named consistently with existing
|
||||
module-owned response schemas, such as `dnd_scenes.v1.json`, with a schema key,
|
||||
schema ID, schema version, and OpenAI-compatible response schema name.
|
||||
|
||||
Boundary fields should use source-unit ID strings, not integer segment IDs.
|
||||
The schema should focus on boundary and metadata decisions rather than final
|
||||
framework chunk fields. Prompt terminology and schema terminology should match
|
||||
exactly, including primary mode enum values and boundary field names.
|
||||
|
||||
The user prompt should be a Go template that includes source document ID,
|
||||
chunking scope, ordered source units, and selected metadata such as speaker and
|
||||
timestamps when available. It may contain D&D-specific scene guidance, but it
|
||||
should not imply that the framework itself is transcript-specific.
|
||||
|
||||
## Documentation Target
|
||||
|
||||
Current-behavior docs should be updated only after the corresponding behavior is
|
||||
implemented.
|
||||
|
||||
Internal module-author documentation should eventually define the chunk module
|
||||
API, including `Chunker`, `ChunkRequest`, `ChunkResult`, `SourceChunk`,
|
||||
validation invariants, warning semantics, LLM-backed chunker expectations,
|
||||
module specs, capability guidance, and option parsing expectations.
|
||||
|
||||
When `dnd/scenes` becomes production behavior, configuration, CLI, internal
|
||||
module, and troubleshooting docs should describe the implemented module and its
|
||||
failure modes.
|
||||
245
docs/roadmap/implementation.md
Normal file
245
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,245 @@
|
||||
# Chunk Follow-Up Implementation Plan
|
||||
|
||||
This plan addresses review findings from the first `dnd/scenes` implementation.
|
||||
It is written for an LLM coding agent that will implement each stage in order.
|
||||
|
||||
Before beginning any stage, review:
|
||||
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- `docs/policy/documentation.md`
|
||||
- `docs/roadmap/chunk.md`
|
||||
|
||||
Do not move planned behavior into non-roadmap docs until the corresponding code
|
||||
is implemented. Do not revert unrelated user changes.
|
||||
|
||||
## Goals
|
||||
|
||||
- Record chunker prompt and response-schema provenance in run manifests.
|
||||
- Ensure downstream extractors receive canonical source units from the source
|
||||
document, not chunker-mutated unit payloads.
|
||||
- Prevent empty or whitespace-only scene caveats from becoming warnings.
|
||||
|
||||
## Stage 1: Run-Manifest Module Metadata
|
||||
|
||||
Goal: make non-lane module provenance auditable without adding
|
||||
chunker-specific fields or D&D-specific framework behavior.
|
||||
|
||||
Design decision:
|
||||
|
||||
- Add a generic top-level run manifest metadata map for singleton pipeline
|
||||
modules:
|
||||
|
||||
```go
|
||||
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
|
||||
```
|
||||
|
||||
- Use stable stage keys:
|
||||
- `input`
|
||||
- `chunker`
|
||||
- `output`
|
||||
- Keep existing artifact lane metadata under `ArtifactLaneManifest.Metadata`.
|
||||
Do not move extractor, merger, normalizer, or validator metadata into the
|
||||
top-level map in this stage.
|
||||
- Record metadata only when a built module implements
|
||||
`contracts.ManifestMetadataProvider` and returns non-empty metadata.
|
||||
- Continue to reject raw prompts, raw response schemas, source text, provider
|
||||
payloads, and secrets from manifest metadata by convention and tests.
|
||||
|
||||
Code changes:
|
||||
|
||||
- Add `ModuleMetadata` to `internal/core/artifacts.RunManifest`.
|
||||
- Add a small helper in `internal/framework/pipeline/runner.go` to attach
|
||||
top-level module metadata by stage key.
|
||||
- After building the input adapter, chunker, and output encoder, call that
|
||||
helper with keys `input`, `chunker`, and `output` respectively.
|
||||
- Keep `setLaneManifestMetadata` for lane-owned modules. If practical, share
|
||||
metadata cloning logic with the new helper.
|
||||
- Ensure failed runs that already have a manifest also retain any metadata
|
||||
collected before the failure.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add pipeline runner tests proving top-level metadata is recorded for a fake
|
||||
chunker that implements `ManifestMetadataProvider`.
|
||||
- Add a test proving the existing lane metadata behavior remains unchanged.
|
||||
- Add a CLI or output integration test proving a `dnd/scenes` run manifest
|
||||
contains `module_metadata.chunker` with prompt/schema provenance.
|
||||
- Add a negative assertion that raw prompt text, raw schema JSON, source text,
|
||||
provider payloads, and API key-like fields are not present in the scene
|
||||
chunker metadata.
|
||||
|
||||
Documentation:
|
||||
|
||||
- Update `docs/internal/pipeline.md` to describe top-level metadata for input,
|
||||
chunker, and output modules, and lane metadata for lane modules.
|
||||
- Update `docs/internal/modules.md` to say that `dnd/scenes` prompt/schema
|
||||
provenance appears under `module_metadata.chunker`.
|
||||
- Update `docs/integrations/json-output.md` and `docs/operations.md` if their
|
||||
manifest descriptions need to mention `module_metadata`.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
go test ./internal/core/artifacts
|
||||
go test ./internal/framework/pipeline
|
||||
go test ./internal/cli
|
||||
go test ./internal/modules/chunk/dnd/scenes
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- `dnd/scenes` prompt/schema provenance is visible in durable
|
||||
`manifest.json` and diagnostics `run-manifest.json`.
|
||||
- Existing artifact lane metadata remains in the same JSON location as before.
|
||||
|
||||
## Stage 2: Canonical Source Units In Chunk Results
|
||||
|
||||
Goal: preserve the chunker boundary contract while ensuring extractors always
|
||||
consume source document units, not rewritten units supplied by a chunker.
|
||||
|
||||
Design decision:
|
||||
|
||||
- Keep the chunker contract expressed in terms of `contracts.SourceChunk`.
|
||||
- Continue validating chunk IDs, source IDs, indexes, unit membership, unit
|
||||
uniqueness within each chunk, and source-ordering.
|
||||
- After validation, canonicalize chunk units by replacing each returned
|
||||
`SourceUnit` with a defensive copy of the matching source document unit.
|
||||
- Preserve `SourceChunk.Metadata` as module-owned chunk metadata.
|
||||
- Do not require full source coverage and do not reject overlap between chunks.
|
||||
- Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker
|
||||
that wants to add scene-level information must use `SourceChunk.Metadata`.
|
||||
|
||||
Code changes:
|
||||
|
||||
- Replace or extend `validateChunkResult` in
|
||||
`internal/framework/pipeline/chunk_validation.go` so it returns canonical
|
||||
chunks, for example:
|
||||
|
||||
```go
|
||||
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error)
|
||||
```
|
||||
|
||||
- Build a source-unit lookup from the validated source document.
|
||||
- For each chunk:
|
||||
- validate the existing generic invariants;
|
||||
- copy chunk ID, source ID, index, and chunk metadata;
|
||||
- replace the unit slice with cloned source units from the source document in
|
||||
the returned boundary/order.
|
||||
- Update the runner to use canonical chunks for all downstream extraction and
|
||||
merge behavior.
|
||||
- Ensure chunk metadata is cloned so later module or caller mutation cannot
|
||||
affect runner state.
|
||||
- Keep the implementation source-agnostic. Do not inspect transcript-specific
|
||||
metadata keys.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add a runner test where a fake chunker returns a valid unit ID with mutated
|
||||
text, kind, and unit metadata. Assert the extractor receives the original
|
||||
source document unit values.
|
||||
- Add a runner test proving chunk metadata survives canonicalization and is not
|
||||
aliased to the chunker-returned map.
|
||||
- Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID,
|
||||
wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order
|
||||
units, partial coverage, and overlap.
|
||||
- Add or update tests so generic chunking still behaves unchanged.
|
||||
|
||||
Documentation:
|
||||
|
||||
- Update internal chunk contract docs to state that source units in chunks are
|
||||
canonicalized from the source document by ID before extractors run.
|
||||
- Document that chunk metadata is the supported mechanism for passing
|
||||
chunker-owned context to extractors.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
go test ./internal/framework/pipeline
|
||||
go test ./internal/modules/chunk/generic
|
||||
go test ./internal/modules/chunk/dnd/scenes
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Extractors cannot observe chunker-rewritten source-unit text, kind, or unit
|
||||
metadata.
|
||||
- Chunker-owned scene metadata still reaches extractors through
|
||||
`SourceChunk.Metadata`.
|
||||
|
||||
## Stage 3: Scene Caveat Hygiene
|
||||
|
||||
Goal: ensure model caveats become useful warnings and never produce blank
|
||||
warnings that can confuse operators or affect diagnostics retention.
|
||||
|
||||
Design decision:
|
||||
|
||||
- Require caveat strings to be non-empty after trimming.
|
||||
- Treat whitespace-only caveats as malformed structured output rather than
|
||||
silently dropping them. This is consistent with the `dnd/scenes` policy of
|
||||
failing explicitly for malformed model output.
|
||||
- Store warning messages as trimmed caveat text.
|
||||
|
||||
Code changes:
|
||||
|
||||
- Update `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`
|
||||
so `boundary_caveats.items` has `minLength: 1`.
|
||||
- Update `warningsFromCaveats` or response validation in
|
||||
`internal/modules/chunk/dnd/scenes/chunker.go` to trim caveats and reject
|
||||
empty results with a module-prefixed malformed-output error.
|
||||
- Prefer validating caveats before constructing chunks so all malformed response
|
||||
checks happen together.
|
||||
|
||||
Tests:
|
||||
|
||||
- Add schema tests proving `boundary_caveats` items require non-empty strings.
|
||||
- Add chunker tests proving:
|
||||
- caveat warning messages are trimmed;
|
||||
- whitespace-only caveats fail explicitly;
|
||||
- valid caveats still produce `scene_boundary_caveat` warnings.
|
||||
- Keep existing warning tests passing.
|
||||
|
||||
Documentation:
|
||||
|
||||
- Update `docs/internal/modules.md` and `docs/troubleshooting.md` if needed to
|
||||
mention that malformed caveats are treated as malformed structured output.
|
||||
|
||||
Validation:
|
||||
|
||||
```sh
|
||||
go test ./internal/modules/chunk/dnd/scenes
|
||||
go test ./internal/cli
|
||||
```
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- No blank warnings can be emitted from `dnd/scenes` boundary caveats.
|
||||
- Valid caveats remain visible as warnings.
|
||||
|
||||
## Stage 4: Full Verification
|
||||
|
||||
Goal: verify the follow-up work across contracts, production wiring,
|
||||
documentation, and the command entry point.
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
Inspect or test representative output manifests:
|
||||
|
||||
- `manifest.json` includes `module_metadata.chunker` for a `dnd/scenes` run;
|
||||
- `module_metadata.chunker` contains prompt and response-schema provenance;
|
||||
- no raw prompt, raw schema, source text, provider payload, or secret appears in
|
||||
module metadata;
|
||||
- artifact lane metadata remains under `artifact_lanes[].metadata`;
|
||||
- `warnings.json` contains trimmed scene caveats and no blank caveat warnings.
|
||||
|
||||
Stage completion criteria:
|
||||
|
||||
- Full validation commands pass.
|
||||
- Current-behavior docs match implemented behavior.
|
||||
- Any remaining planned or deferred behavior stays under `docs/roadmap/`.
|
||||
340
docs/roadmap/references.md
Normal file
340
docs/roadmap/references.md
Normal file
@@ -0,0 +1,340 @@
|
||||
# Feature Roadmap Proposal: Extraction Reference
|
||||
|
||||
## Status
|
||||
|
||||
This document captures proposed design and implementation sequencing for the
|
||||
extraction-reference feature in Notarius. It describes planned work, not
|
||||
implemented behavior. Go snippets are conceptual sketches; the implementing
|
||||
agent should adapt names and shapes to the existing contracts, package
|
||||
boundaries, and conventions in this repository.
|
||||
|
||||
## Goal
|
||||
|
||||
Extraction quality improves significantly when the LLM receives reference
|
||||
material alongside the source input. For the initial D&D spell extractor,
|
||||
useful reference material includes a party roster (mapping players to player
|
||||
characters), a player list, and a campaign glossary.
|
||||
|
||||
Notarius should support passing this material to extractors as **named reference
|
||||
items** without introducing any domain-specific concepts into core or framework
|
||||
packages. The framework should know only that:
|
||||
|
||||
- extractors declare named reference slots they accept;
|
||||
- pipeline config and CLI flags bind content (initially files) to those slots;
|
||||
- bound content is rendered into module-owned prompt templates;
|
||||
- bound content is digested and recorded as run provenance.
|
||||
|
||||
Only extract modules should know what a "roster" or "glossary" means. All
|
||||
domain semantics live in module-owned slot declarations and prompt templates.
|
||||
|
||||
## Definitions
|
||||
|
||||
- **Reference slot**: a named, typed-by-convention input declared by an
|
||||
extractor, with a human-readable description and a required/optional flag.
|
||||
Example: extractor `dnd/spells` declares an optional slot named `roster`.
|
||||
- **Reference item**: resolved content bound to a slot for a given run: name,
|
||||
content bytes, media type, content digest, and origin (initially a file
|
||||
path).
|
||||
- **Reference binding**: the association of a slot name to a content source,
|
||||
defined in pipeline config and overridable per run via CLI.
|
||||
|
||||
## Architectural Principles
|
||||
|
||||
- Reference is opaque to the framework. Core and framework packages must not
|
||||
interpret reference content or recognize domain slot names.
|
||||
- Reference is an input. Anything that changes extraction output must be
|
||||
digested into the run manifest and participate in any cache key.
|
||||
- A Reference is not evidence. `SourceRef` values must only ever reference source
|
||||
units. Reference items must not receive unit IDs and must not be addressable
|
||||
by source references.
|
||||
- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting
|
||||
a required slot, should fail at config-load time, before any LLM call.
|
||||
- Optional slots degrade gracefully. Prompt templates should render cleanly
|
||||
whether or not an optional slot is bound.
|
||||
- Determinism. Identical input, config, prompts, and reference bytes should
|
||||
produce byte-identical rendered prompts. Reference slots should render in a
|
||||
stable, documented order (declaration order).
|
||||
|
||||
## Proposed Contracts
|
||||
|
||||
### Slot declaration (extractor contract extension)
|
||||
|
||||
Extractors should declare the reference slots they accept:
|
||||
|
||||
```go
|
||||
type ReferenceSlot struct {
|
||||
Name string
|
||||
Description string
|
||||
Required bool
|
||||
|
||||
// MVP can leave these empty/defaulted, but having the fields now
|
||||
// makes validation and future docs easier.
|
||||
AcceptedMediaTypes []string
|
||||
Multiple bool
|
||||
MaxBytes int64
|
||||
}
|
||||
```
|
||||
|
||||
The extractor interface should gain a method such as:
|
||||
|
||||
```go
|
||||
ReferenceSlots() []ReferenceSlot
|
||||
```
|
||||
|
||||
Extractors with no reference needs return an empty slice. Existing extractors
|
||||
should require no other changes.
|
||||
|
||||
### Resolved reference item
|
||||
|
||||
```go
|
||||
type ReferenceItem struct {
|
||||
SlotName string
|
||||
MediaType string
|
||||
Content []byte
|
||||
Digest string
|
||||
Origin ReferenceOrigin
|
||||
|
||||
SizeBytes int64
|
||||
TokenEstimate int
|
||||
}
|
||||
|
||||
type ReferenceOrigin struct {
|
||||
Type string // "file" for MVP
|
||||
URI string // path or future artifact URI
|
||||
}
|
||||
|
||||
type ReferenceSet struct {
|
||||
// Stable declaration order, then stable binding order within a slot.
|
||||
Slots []ResolvedReferenceSlot
|
||||
}
|
||||
|
||||
type ResolvedReferenceSlot struct {
|
||||
Name string
|
||||
Items []ReferenceItem
|
||||
}
|
||||
```
|
||||
|
||||
`ReferenceItem` is a resolved-content type, not a file path. The only MVP
|
||||
producer is "read this file," but the shape should permit future producers
|
||||
(prior-run artifacts, derived summaries, entity registries) without contract
|
||||
changes.
|
||||
|
||||
### Binding resolution
|
||||
|
||||
A resolver should, at config-load time:
|
||||
|
||||
1. Collect declared slots from every extractor selected by the active
|
||||
pipeline (respecting lane selection, e.g. `--only`).
|
||||
2. Collect bindings from pipeline config (pipeline-level and lane-level) and
|
||||
CLI overrides, applying the standard layering: config file, then CLI.
|
||||
3. Fail with a clear error if a required slot is unbound, or if a binding
|
||||
references a slot no extractor within the selected pipeline declares.
|
||||
Errors should name the pipeline, lane, slot, and the slot description.
|
||||
4. Read, digest, and materialize each bound source into a `ReferenceItem`.
|
||||
5. Enforce size guardrails (see Validation and Guardrails).
|
||||
|
||||
## Configuration and CLI
|
||||
|
||||
### Pipeline config
|
||||
|
||||
Reference bindings should live in pipeline config, because the initial use cases
|
||||
(roster, glossary) are campaign-invariant rather than run-variant. Bindings
|
||||
should be supported at two levels:
|
||||
|
||||
- pipeline level: shared by all artifact lanes;
|
||||
- lane level: additions or overrides for a single lane.
|
||||
|
||||
Illustrative shape (adapt to the existing config format):
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
references:
|
||||
roster: ./campaign/party_roster.md
|
||||
glossary: ./campaign/glossary.md
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
reference:
|
||||
npc_registry: ./campaign/npcs.md
|
||||
```
|
||||
|
||||
### CLI
|
||||
|
||||
Per-run override flag, repeatable:
|
||||
|
||||
```text
|
||||
notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md
|
||||
```
|
||||
|
||||
CLI bindings override config bindings for the same slot name. The existing
|
||||
pipeline-describe/config-validate commands (or their nearest equivalents)
|
||||
should surface declared slots, descriptions, required flags, and current
|
||||
bindings so users can discover what a pipeline accepts.
|
||||
|
||||
## Prompt Template Integration
|
||||
|
||||
Prompt templates are module-owned. Template rendering should expose:
|
||||
|
||||
- `{{ reference "roster" }}`: renders the content of the bound item;
|
||||
- `{{ hasreference "glossary" }}`: predicate for conditional sections, so
|
||||
optional slots can be included only when bound.
|
||||
|
||||
Rules:
|
||||
|
||||
- Referencing an **undeclared** slot from a template is a module bug and
|
||||
should fail at prompt registration/build time (or earliest feasible point),
|
||||
not silently at render time.
|
||||
- Referencing a declared but unbound **optional** slot should render as
|
||||
empty; templates should use `hasreference` to avoid dangling section headers.
|
||||
- Rendering must be deterministic and independent of map iteration order.
|
||||
- Prompt identity (registry hash) should be computed over the **template**,
|
||||
not the rendered prompt. Reference digests are recorded separately in the
|
||||
manifest, so a reference edit is visible as a reference change, not a prompt
|
||||
change.
|
||||
|
||||
Note: reference content is repeated in every per-chunk prompt. Diagnostics
|
||||
should record per-slot token or byte counts so reference cost is observable.
|
||||
Per-slot inclusion policies (e.g., roster in every chunk, glossary on demand)
|
||||
are explicitly out of scope until cost data justifies them.
|
||||
|
||||
## Provenance
|
||||
|
||||
The run manifest must record, for every bound slot:
|
||||
|
||||
- slot name;
|
||||
- origin (path);
|
||||
- content digest;
|
||||
- media type;
|
||||
- whether the binding came from config or CLI override.
|
||||
|
||||
Reference digests must participate in any idempotency/cache key alongside source
|
||||
digests, prompt hashes, schema versions, model, and parameters. Two runs that
|
||||
differ only in reference content must be distinguishable from the manifest
|
||||
alone.
|
||||
|
||||
Diagnostics for a run should include the resolved binding set (with digests,
|
||||
not necessarily full content) in the run directory, consistent with the
|
||||
existing redacted-effective-config pattern.
|
||||
|
||||
## Path Resolution
|
||||
|
||||
- Config-relative paths resolve relative to the pipeline config file.
|
||||
- CLI-relative paths resolve relative to the current working directory.
|
||||
- Manifest records the normalized absolute path or a redacted/display path
|
||||
according to existing diagnostics policy.
|
||||
|
||||
## Validation and Guardrails
|
||||
|
||||
### References are not evidence
|
||||
|
||||
The primary new failure mode: the model extracts facts from references rather
|
||||
than from the source input. Example: the roster lists a PC's known spells, and
|
||||
the model emits a `SpellCast` for a spell that was never cast in the session,
|
||||
with a fabricated or misattributed source reference.
|
||||
|
||||
Defenses, in priority order:
|
||||
|
||||
1. **Structural.** `SourceRef` remains the only grounding mechanism and can
|
||||
only reference source units. No contract change should make references
|
||||
addressable as evidence.
|
||||
2. **Prompt discipline.** Module templates should frame references explicitly as
|
||||
reference material, e.g. "use the roster to resolve speakers to
|
||||
characters; extract only events that occur in the transcript." This
|
||||
guidance belongs in the module prompt guidelines, not framework code.
|
||||
3. **Validator support.** The source-reference validator (or a sibling
|
||||
deterministic validator) should support checking that referenced source
|
||||
text plausibly relates to the extracted fact (e.g., spell name or a close
|
||||
variant appears in or near the referenced range). Severity should be
|
||||
`warn`, not `fail`, given paraphrase and nickname casting.
|
||||
4. **Regression fixtures.** Golden-file tests must include a fixture in which
|
||||
the bound roster mentions a spell that is never cast in the transcript,
|
||||
asserting no artifact record is produced for it. This regression is likely
|
||||
to be reintroduced by future prompt edits; the fixture is the guard.
|
||||
|
||||
### Size and sanity guardrails
|
||||
|
||||
- Fail fast, before any LLM call, if bound references plus template plus largest
|
||||
chunk exceeds the configured model context budget, with an error that names
|
||||
the offending slot(s) and sizes.
|
||||
- Empty bound files should produce a warning (probable user error).
|
||||
- MVP accepts text content only (`utf-8`); other media
|
||||
types should be rejected with a clear error.
|
||||
|
||||
## Out of Scope (MVP)
|
||||
|
||||
- Non-file reference producers (prior-run artifacts, derived summaries, entity
|
||||
registries). The `ReferenceItem` shape should permit them later.
|
||||
- Per-chunk or per-slot inclusion policies and context budgeting beyond the
|
||||
fail-fast guardrail.
|
||||
- Structured/parsed references (e.g., typed roster schemas). References are opaque
|
||||
text handed to prompts.
|
||||
- Reference caching or preprocessing (summarization, embedding, retrieval).
|
||||
- Making reference addressable as evidence, in any form.
|
||||
|
||||
## Checkpoint Sequencing
|
||||
|
||||
Each checkpoint should leave the repository compiling, with targeted tests
|
||||
covering newly introduced contracts or behavior.
|
||||
|
||||
1. **Contracts and resolution.** Add `ReferenceSlot`, `ReferenceItem`, and the
|
||||
extractor `ReferenceSlots()` method (empty default for existing extractors).
|
||||
Implement config parsing for pipeline- and lane-level bindings, CLI
|
||||
override flag, layering, and load-time validation (unknown slot, missing
|
||||
required slot, unreadable file, empty file warning). Unit tests for
|
||||
resolution and error cases.
|
||||
2. **Prompt rendering.** Add `reference`/`hasreference` template functions,
|
||||
declaration-order rendering, undeclared-slot failure at registration, and
|
||||
deterministic-render tests (byte-identical output across runs).
|
||||
3. **Provenance.** Record bindings (name, origin, digest, media type,
|
||||
binding source) in the run manifest and diagnostics; include reference
|
||||
digests in the cache/idempotency key if one exists. Tests: manifest
|
||||
round-trip; two runs differing only in reference content produce differing
|
||||
manifests.
|
||||
4. **Guardrails and validation.** Context-window fail-fast check;
|
||||
relatedness `warn` validator (or extension of the source-reference
|
||||
validator); media-type rejection.
|
||||
5. **First consumer.** Declare `roster` (optional) and `glossary` (optional)
|
||||
slots on the D&D spells extractor; update its prompt template with
|
||||
conditional reference sections and reference-material framing; add golden
|
||||
fixtures with and without references bound, including the
|
||||
roster-mentions-uncast-spell fixture. This checkpoint is the acceptance
|
||||
test for the feature: spell extraction quality with a roster bound should
|
||||
visibly improve speaker-to-character attribution in fixtures.
|
||||
|
||||
## Open Design Questions
|
||||
|
||||
The implementing agent should resolve these against existing code and record
|
||||
decisions in the implementation plan:
|
||||
|
||||
- Should slot names be namespaced per lane in config and CLI (e.g.,
|
||||
`spells.roster=...`) or flat with lane-level config as the only
|
||||
disambiguator? (Recommended default: flat names; lane-level config for
|
||||
overrides; revisit if two extractors in one pipeline want the same slot
|
||||
name with different content.)
|
||||
- Where does binding resolution live relative to the existing config and
|
||||
pipeline packages? It must run at load time, alongside existing pipeline
|
||||
validation.
|
||||
- Does the existing prompt registry hash templates or rendered prompts? If
|
||||
rendered, this feature requires moving to template hashing as described in
|
||||
Provenance.
|
||||
- Should CLI overrides be permitted to bind slots that config leaves unbound
|
||||
(yes, presumably), and to *unbind* a config-bound optional slot (e.g.,
|
||||
`--reference roster=` to clear)? Decide and test both directions.
|
||||
|
||||
## Documentation Tasks
|
||||
|
||||
Once implemented, move contracts out of this roadmap into canonical docs:
|
||||
|
||||
- `docs/cli.md`: `--reference` flag syntax, layering, and examples;
|
||||
- `docs/config.md`: pipeline- and lane-level `references` blocks;
|
||||
- `docs/internal/`: slot/item contracts, resolution flow, evidence
|
||||
exclusion rule, and template function reference for module authors;
|
||||
- module-author guidance: how to declare slots, write conditional reference
|
||||
sections, and frame reference material in prompts;
|
||||
- `examples/`: a maintained example pipeline with a roster and glossary
|
||||
bound, plus matching fixture files.
|
||||
```
|
||||
@@ -177,6 +177,31 @@ Fix:
|
||||
|
||||
Provider error messages are redacted for configured API key values.
|
||||
|
||||
## Scene Chunking Failure
|
||||
|
||||
Symptoms include:
|
||||
|
||||
- `dnd scenes chunker`
|
||||
- `malformed structured output`
|
||||
- `start_unit_id`
|
||||
- `end_unit_id`
|
||||
- `gap`
|
||||
- `overlap`
|
||||
- `final scene`
|
||||
- `complete structured output`
|
||||
|
||||
Fix:
|
||||
|
||||
- Validate the pipeline configuration and confirm the input module provides a
|
||||
transcript source when using `chunk: dnd/scenes`.
|
||||
- Confirm the LLM profile has a working OpenAI-compatible `base_url`, `model`,
|
||||
and credentials.
|
||||
- Inspect retained diagnostics for the run error and resolved pipeline.
|
||||
- If the error names malformed structured output, retry with a model that
|
||||
follows structured response schemas reliably.
|
||||
- Scene boundaries must use exact source-unit IDs, cover the full source
|
||||
document, be contiguous, and not overlap.
|
||||
|
||||
## Output Write Failure
|
||||
|
||||
Symptoms include:
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||
@@ -34,6 +35,9 @@ func productionRegistries() (pipeline.Registries, error) {
|
||||
if err := generic.Register(registries.Chunkers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
|
||||
}
|
||||
if err := scenes.Register(registries.Chunkers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register dnd scenes chunker: %w", err)
|
||||
}
|
||||
if err := spells.Register(registries.Extractors); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||
@@ -136,6 +137,11 @@ func TestProductionCatalogIncludesDefaultModules(t *testing.T) {
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(generic.Key) },
|
||||
want: generic.ModuleSpec(),
|
||||
},
|
||||
{
|
||||
name: "dnd scenes chunker",
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(scenes.Key) },
|
||||
want: scenes.ModuleSpec(),
|
||||
},
|
||||
{
|
||||
name: "dnd spells extractor",
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Extractors.Spec(spells.Key) },
|
||||
@@ -186,6 +192,21 @@ func TestRunConfigValidateUsesProductionCatalogByDefault(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunConfigValidateAcceptsDNDScenesChunker(t *testing.T) {
|
||||
configPath := writeTestConfig(t, mvpConfigYAMLWithChunk("dnd-session", scenes.Key, "dnd/spells"))
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "dnd-session"}, &stdout, &stderr, Options{})
|
||||
|
||||
if code != 0 {
|
||||
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "is valid for pipeline") {
|
||||
t.Fatalf("stdout = %q, want validation success", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunConfigValidateUnknownProductionModuleIncludesContext(t *testing.T) {
|
||||
configPath := writeTestConfig(t, mvpConfigYAML("dnd-session", "missing/extract"))
|
||||
var stdout bytes.Buffer
|
||||
@@ -1041,6 +1062,50 @@ func TestExampleFixtureRunWritesExpectedJSON(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExampleFixtureRunWithDNDScenesRecordsChunkerAndWarnings(t *testing.T) {
|
||||
configPath := writeTestConfig(t, mvpConfigYAMLWithChunk("dnd-session", scenes.Key, "dnd/spells"))
|
||||
inputPath := fixturePath(t, "examples/seriatim-minimal-transcript.json")
|
||||
outputDir := t.TempDir()
|
||||
diagnosticsDir := t.TempDir()
|
||||
client := newSceneRunLLMClient("Scene boundary was ambiguous.")
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := RunWithOptions([]string{"run", "dnd-session", "--config", configPath, "--input", inputPath, "--output-dir", outputDir, "--diagnostics-dir", diagnosticsDir}, &stdout, &stderr, Options{
|
||||
LLMClientFactory: fakeLLMFactory(client, nil),
|
||||
})
|
||||
|
||||
if code != 0 {
|
||||
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
|
||||
}
|
||||
if client.calls != 2 {
|
||||
t.Fatalf("LLM calls = %d, want chunking and extraction calls", client.calls)
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "1 warning") {
|
||||
t.Fatalf("stderr = %q, want warning count", stderr.String())
|
||||
}
|
||||
|
||||
runOutputDir := onlyChildDir(t, outputDir)
|
||||
manifestBytes := readFile(t, filepath.Join(runOutputDir, "manifest.json"))
|
||||
var manifest artifacts.RunManifest
|
||||
if err := json.Unmarshal(manifestBytes, &manifest); err != nil {
|
||||
t.Fatalf("unmarshal manifest: %v", err)
|
||||
}
|
||||
if manifest.Chunker != scenes.Key {
|
||||
t.Fatalf("manifest chunker = %q, want %q", manifest.Chunker, scenes.Key)
|
||||
}
|
||||
for _, forbidden := range []string{"Source document ID:", "Aria casts Cure Wounds.", "spell_casts", "Scene boundary was ambiguous."} {
|
||||
if strings.Contains(string(manifestBytes), forbidden) {
|
||||
t.Fatalf("manifest leaked %q: %s", forbidden, manifestBytes)
|
||||
}
|
||||
}
|
||||
|
||||
warnings := string(readFile(t, filepath.Join(runOutputDir, "warnings.json")))
|
||||
if !strings.Contains(warnings, "scene_boundary_caveat") || !strings.Contains(warnings, "Scene boundary was ambiguous.") {
|
||||
t.Fatalf("warnings output = %s, want scene boundary caveat", warnings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExampleFixtureRunOnlySpells(t *testing.T) {
|
||||
configPath := fixturePath(t, "examples/dnd-spells.config.yml")
|
||||
inputPath := fixturePath(t, "examples/seriatim-minimal-transcript.json")
|
||||
@@ -1220,6 +1285,18 @@ pipelines:
|
||||
`
|
||||
}
|
||||
|
||||
func mvpConfigYAMLWithChunk(pipelineID string, chunker string, extractor string) string {
|
||||
return `version: 1
|
||||
pipelines:
|
||||
` + pipelineID + `:
|
||||
input: seriatim
|
||||
chunk: ` + chunker + `
|
||||
artifacts:
|
||||
spells:
|
||||
extract: ` + extractor + `
|
||||
`
|
||||
}
|
||||
|
||||
func mvpConfigYAMLForLanes(pipelineID string, laneIDs ...string) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("version: 1\n")
|
||||
@@ -1287,6 +1364,7 @@ type fakeRunLLMClient struct {
|
||||
calls int
|
||||
err error
|
||||
payload map[string]any
|
||||
sceneCaveat string
|
||||
}
|
||||
|
||||
func newFakeRunLLMClient(invalidSourceRef bool) *fakeRunLLMClient {
|
||||
@@ -1301,11 +1379,43 @@ func newMalformedRunLLMClient() *fakeRunLLMClient {
|
||||
return &fakeRunLLMClient{payload: map[string]any{}}
|
||||
}
|
||||
|
||||
func newSceneRunLLMClient(caveat string) *fakeRunLLMClient {
|
||||
return &fakeRunLLMClient{sceneCaveat: caveat}
|
||||
}
|
||||
|
||||
func (client *fakeRunLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.calls++
|
||||
if client.err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, client.err
|
||||
}
|
||||
if req.StageName == scenes.Key && client.payload == nil {
|
||||
payload := map[string]any{
|
||||
"scenes": []map[string]any{
|
||||
{
|
||||
"start_unit_id": "seg-001",
|
||||
"end_unit_id": "seg-002",
|
||||
"short_title": "Opening spell",
|
||||
"primary_mode": "Narrative",
|
||||
"main_participants": []string{"Aria"},
|
||||
"summary": "Aria casts a spell.",
|
||||
"boundary_note": "The provided source units form one scene.",
|
||||
"boundary_confidence": "High",
|
||||
},
|
||||
},
|
||||
"boundary_caveats": []string{},
|
||||
}
|
||||
if client.sceneCaveat != "" {
|
||||
payload["boundary_caveats"] = []string{client.sceneCaveat}
|
||||
}
|
||||
encoded, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
if err := json.Unmarshal(encoded, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: encoded}, nil
|
||||
}
|
||||
startUnitID := "seg-001"
|
||||
if client.invalidSourceRef {
|
||||
startUnitID = "missing-segment"
|
||||
|
||||
@@ -98,6 +98,43 @@ func TestResolveSurfacesMissingCapabilityThroughCatalog(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCanBindSceneChunkerFromCatalog(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
profile := cfg.Pipelines["example"]
|
||||
profile.Chunk = pipeline.Binding("dnd/scenes")
|
||||
lane := profile.Artifacts["events"]
|
||||
profile.Artifacts = map[string]pipeline.ArtifactLaneProfile{"events": lane}
|
||||
cfg.Pipelines["example"] = profile
|
||||
|
||||
catalog := fakeCatalog(t,
|
||||
pipeline.ModuleSpec{
|
||||
Key: "fake/input",
|
||||
Stage: pipeline.StageInput,
|
||||
Provides: []string{"source.transcript"},
|
||||
},
|
||||
pipeline.ModuleSpec{
|
||||
Key: "fake/extract",
|
||||
Stage: pipeline.StageExtract,
|
||||
Requires: []string{"chunks", "source.transcript"},
|
||||
Provides: []string{"artifact"},
|
||||
},
|
||||
)
|
||||
mustRegisterChunker(t, catalog.Chunkers, pipeline.ModuleSpec{
|
||||
Key: "dnd/scenes",
|
||||
Stage: pipeline.StageChunk,
|
||||
Requires: []string{"source.transcript"},
|
||||
Provides: []string{"chunks", "chunks.scenes"},
|
||||
})
|
||||
|
||||
effective, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: catalog})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||
}
|
||||
if got := effective.ResolvedPipeline.Chunk.Module; got != "dnd/scenes" {
|
||||
t.Fatalf("Chunk.Module = %q, want dnd/scenes", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveDigestChangesWhenEffectiveConfigChanges(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
first, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
|
||||
|
||||
@@ -17,6 +17,7 @@ var _ contracts.Extractor = compositionExtractor{}
|
||||
var _ contracts.Merger = compositionMerger{}
|
||||
var _ contracts.Normalizer = compositionNormalizer{}
|
||||
var _ contracts.Validator = compositionValidator{}
|
||||
var _ contracts.StructuredLLMClient = compositionLLMClient{}
|
||||
var _ contracts.OutputEncoder = compositionOutputEncoder{}
|
||||
|
||||
func TestContractsComposeAcrossPackages(t *testing.T) {
|
||||
@@ -38,8 +39,9 @@ func TestContractsComposeAcrossPackages(t *testing.T) {
|
||||
}
|
||||
|
||||
chunking, err := chunker.Chunk(ctx, contracts.ChunkRequest{
|
||||
Source: doc,
|
||||
Metadata: map[string]any{"max_units": 2},
|
||||
Source: doc,
|
||||
LLMClient: compositionLLMClient{},
|
||||
Metadata: map[string]any{"max_units": 2},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
@@ -164,6 +166,9 @@ func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.Chunk
|
||||
if req.Source == nil {
|
||||
return contracts.ChunkResult{}, errors.New("source document is required")
|
||||
}
|
||||
if req.LLMClient == nil {
|
||||
return contracts.ChunkResult{}, errors.New("structured llm client is required")
|
||||
}
|
||||
|
||||
return contracts.ChunkResult{
|
||||
Chunks: []contracts.SourceChunk{
|
||||
@@ -178,6 +183,12 @@ func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.Chunk
|
||||
}, nil
|
||||
}
|
||||
|
||||
type compositionLLMClient struct{}
|
||||
|
||||
func (client compositionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
}
|
||||
|
||||
type compositionExtractor struct{}
|
||||
|
||||
func (extractor compositionExtractor) Key() string {
|
||||
|
||||
@@ -58,6 +58,7 @@ type SourceChunk struct {
|
||||
|
||||
type ChunkRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
|
||||
@@ -117,6 +117,27 @@ func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeChunkerReceivesLLMClient(t *testing.T) {
|
||||
doc := &source.SourceDocument{
|
||||
ID: "source-1",
|
||||
Kind: "document",
|
||||
Format: "text/plain",
|
||||
Digest: "sha256:abc123",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: "u1", Kind: "section", Text: "Source text."},
|
||||
},
|
||||
}
|
||||
client := fakeLLMClient{}
|
||||
chunker := &recordingChunker{key: "llm-chunker"}
|
||||
|
||||
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, LLMClient: client}); err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
if chunker.request.LLMClient == nil {
|
||||
t.Fatal("ChunkRequest.LLMClient = nil, want structured LLM client")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
|
||||
extractor := fakeExtractor{
|
||||
key: "generic-extractor",
|
||||
@@ -305,6 +326,20 @@ func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkRe
|
||||
}, nil
|
||||
}
|
||||
|
||||
type recordingChunker struct {
|
||||
key string
|
||||
request ChunkRequest
|
||||
}
|
||||
|
||||
func (chunker *recordingChunker) Key() string {
|
||||
return chunker.key
|
||||
}
|
||||
|
||||
func (chunker *recordingChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error) {
|
||||
chunker.request = req
|
||||
return fakeChunker{key: chunker.key}.Chunk(ctx, req)
|
||||
}
|
||||
|
||||
type fakeExtractor struct {
|
||||
key string
|
||||
artifactType string
|
||||
|
||||
60
internal/framework/pipeline/chunk_validation.go
Normal file
60
internal/framework/pipeline/chunk_validation.go
Normal file
@@ -0,0 +1,60 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func validateChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) error {
|
||||
sourceUnitIndexes := make(map[string]int, len(doc.Units))
|
||||
for index, unit := range doc.Units {
|
||||
sourceUnitIndexes[unit.ID] = index
|
||||
}
|
||||
|
||||
seenChunkIDs := make(map[string]struct{}, len(chunks))
|
||||
for chunkIndex, chunk := range chunks {
|
||||
if strings.TrimSpace(chunk.ID) == "" {
|
||||
return fmt.Errorf("chunk[%d].id must not be empty", chunkIndex)
|
||||
}
|
||||
if _, ok := seenChunkIDs[chunk.ID]; ok {
|
||||
return fmt.Errorf("chunk id %q is duplicated", chunk.ID)
|
||||
}
|
||||
seenChunkIDs[chunk.ID] = struct{}{}
|
||||
|
||||
if chunk.SourceID != doc.ID {
|
||||
return fmt.Errorf("chunk %q source_id %q does not match source document id %q", chunk.ID, chunk.SourceID, doc.ID)
|
||||
}
|
||||
if chunk.Index != chunkIndex {
|
||||
return fmt.Errorf("chunk %q index %d does not match returned order %d", chunk.ID, chunk.Index, chunkIndex)
|
||||
}
|
||||
if len(chunk.Units) == 0 {
|
||||
return fmt.Errorf("chunk %q units must not be empty", chunk.ID)
|
||||
}
|
||||
|
||||
seenUnitIDs := make(map[string]struct{}, len(chunk.Units))
|
||||
previousSourceIndex := -1
|
||||
for unitIndex, unit := range chunk.Units {
|
||||
if strings.TrimSpace(unit.ID) == "" {
|
||||
return fmt.Errorf("chunk %q unit[%d].id must not be empty", chunk.ID, unitIndex)
|
||||
}
|
||||
if _, ok := seenUnitIDs[unit.ID]; ok {
|
||||
return fmt.Errorf("chunk %q repeats source unit %q", chunk.ID, unit.ID)
|
||||
}
|
||||
seenUnitIDs[unit.ID] = struct{}{}
|
||||
|
||||
sourceIndex, ok := sourceUnitIndexes[unit.ID]
|
||||
if !ok {
|
||||
return fmt.Errorf("chunk %q source unit %q was not found in source document %q", chunk.ID, unit.ID, doc.ID)
|
||||
}
|
||||
if sourceIndex <= previousSourceIndex {
|
||||
return fmt.Errorf("chunk %q source units must appear in source document order", chunk.ID)
|
||||
}
|
||||
previousSourceIndex = sourceIndex
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -91,6 +91,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
|
||||
}
|
||||
chunkResult, err := chunker.Chunk(ctx, contracts.ChunkRequest{
|
||||
Source: doc,
|
||||
LLMClient: input.LLMClient,
|
||||
LLMProfile: input.Pipeline.Chunk.LLMProfile,
|
||||
Options: cloneOptions(input.Pipeline.Chunk.Options),
|
||||
Metadata: input.Metadata,
|
||||
@@ -102,6 +103,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
|
||||
if len(chunkResult.Chunks) == 0 {
|
||||
return failOutput(output), fmt.Errorf("chunker %q returned no chunks", chunker.Key())
|
||||
}
|
||||
if err := validateChunkResult(doc, chunkResult.Chunks); err != nil {
|
||||
return failOutput(output), fmt.Errorf("validate chunks from chunker %q: %w", chunker.Key(), err)
|
||||
}
|
||||
|
||||
nextCandidateIndex := 0
|
||||
for _, lane := range input.Pipeline.ArtifactLanes {
|
||||
|
||||
@@ -256,6 +256,91 @@ func TestRunRejectsChunkerBuildChunkAndEmptyChunkErrors(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRejectsInvalidChunks(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
chunks []contracts.SourceChunk
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "empty chunk id",
|
||||
chunks: []contracts.SourceChunk{{ID: "", SourceID: "source-1", Index: 0, Units: []source.SourceUnit{unitWithID("u1")}}},
|
||||
want: "id must not be empty",
|
||||
},
|
||||
{
|
||||
name: "duplicate chunk id",
|
||||
chunks: []contracts.SourceChunk{
|
||||
{ID: "chunk-0", SourceID: "source-1", Index: 0, Units: []source.SourceUnit{unitWithID("u1")}},
|
||||
{ID: "chunk-0", SourceID: "source-1", Index: 1, Units: []source.SourceUnit{unitWithID("u2")}},
|
||||
},
|
||||
want: "duplicated",
|
||||
},
|
||||
{
|
||||
name: "wrong source id",
|
||||
chunks: []contracts.SourceChunk{{ID: "chunk-0", SourceID: "other-source", Index: 0, Units: []source.SourceUnit{unitWithID("u1")}}},
|
||||
want: "source_id",
|
||||
},
|
||||
{
|
||||
name: "wrong index",
|
||||
chunks: []contracts.SourceChunk{{ID: "chunk-0", SourceID: "source-1", Index: 1, Units: []source.SourceUnit{unitWithID("u1")}}},
|
||||
want: "index",
|
||||
},
|
||||
{
|
||||
name: "empty units",
|
||||
chunks: []contracts.SourceChunk{{ID: "chunk-0", SourceID: "source-1", Index: 0}},
|
||||
want: "units must not be empty",
|
||||
},
|
||||
{
|
||||
name: "repeated unit inside chunk",
|
||||
chunks: []contracts.SourceChunk{{ID: "chunk-0", SourceID: "source-1", Index: 0, Units: []source.SourceUnit{unitWithID("u1"), unitWithID("u1")}}},
|
||||
want: "repeats source unit",
|
||||
},
|
||||
{
|
||||
name: "unknown unit",
|
||||
chunks: []contracts.SourceChunk{{ID: "chunk-0", SourceID: "source-1", Index: 0, Units: []source.SourceUnit{unitWithID("u9")}}},
|
||||
want: "was not found",
|
||||
},
|
||||
{
|
||||
name: "units out of source order",
|
||||
chunks: []contracts.SourceChunk{{ID: "chunk-0", SourceID: "source-1", Index: 0, Units: []source.SourceUnit{unitWithID("u2"), unitWithID("u1")}}},
|
||||
want: "source document order",
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
modules := defaultRunnerModules()
|
||||
modules.chunker.chunks = test.chunks
|
||||
|
||||
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
|
||||
|
||||
assertRunError(t, err, test.want)
|
||||
if output.Manifest.ValidationStatus != "failed" {
|
||||
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
|
||||
}
|
||||
if len(modules.extractors["extract-alpha"].requests) != 0 {
|
||||
t.Fatalf("extractor calls = %d, want none after invalid chunks", len(modules.extractors["extract-alpha"].requests))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunAllowsPartialCoverageAndOverlappingChunks(t *testing.T) {
|
||||
modules := defaultRunnerModules()
|
||||
modules.chunker.chunks = []contracts.SourceChunk{
|
||||
{ID: "chunk-0", SourceID: "source-1", Index: 0, Units: []source.SourceUnit{unitWithID("u1"), unitWithID("u2")}},
|
||||
{ID: "chunk-1", SourceID: "source-1", Index: 1, Units: []source.SourceUnit{unitWithID("u2")}},
|
||||
}
|
||||
|
||||
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
if len(output.Approved) != 2 {
|
||||
t.Fatalf("len(Approved) = %d, want one candidate per accepted chunk", len(output.Approved))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunExecutesChunksAndPassesChunkAndLLMClient(t *testing.T) {
|
||||
modules := defaultRunnerModules()
|
||||
llmClient := fakeLLMClient{}
|
||||
@@ -273,6 +358,9 @@ func TestRunExecutesChunksAndPassesChunkAndLLMClient(t *testing.T) {
|
||||
if !reflect.DeepEqual(extractor.seenChunkIDs, []string{"chunk-0", "chunk-1"}) {
|
||||
t.Fatalf("seen chunks = %#v, want both chunks", extractor.seenChunkIDs)
|
||||
}
|
||||
if len(modules.chunker.requests) != 1 || modules.chunker.requests[0].LLMClient == nil {
|
||||
t.Fatalf("chunker LLM client = %#v, want client on chunk request", modules.chunker.requests)
|
||||
}
|
||||
if len(extractor.seenLLMClients) != 2 || extractor.seenLLMClients[0] == nil || extractor.seenLLMClients[1] == nil {
|
||||
t.Fatalf("seen LLM clients = %#v, want client for each chunk", extractor.seenLLMClients)
|
||||
}
|
||||
@@ -1179,6 +1267,8 @@ func validSourceDocument() *source.SourceDocument {
|
||||
Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: "u1", Kind: "unit", Text: "Source unit."},
|
||||
{ID: "u2", Kind: "unit", Text: "Second source unit."},
|
||||
{ID: "u3", Kind: "unit", Text: "Third source unit."},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -1194,6 +1284,19 @@ func sourceChunkWithID(id string, index int) contracts.SourceChunk {
|
||||
}
|
||||
}
|
||||
|
||||
func unitWithID(id string) source.SourceUnit {
|
||||
switch id {
|
||||
case "u1":
|
||||
return source.SourceUnit{ID: "u1", Kind: "unit", Text: "Source unit."}
|
||||
case "u2":
|
||||
return source.SourceUnit{ID: "u2", Kind: "unit", Text: "Second source unit."}
|
||||
case "u3":
|
||||
return source.SourceUnit{ID: "u3", Kind: "unit", Text: "Third source unit."}
|
||||
default:
|
||||
return source.SourceUnit{ID: id, Kind: "unit", Text: "Unknown source unit."}
|
||||
}
|
||||
}
|
||||
|
||||
func warningReasons(warnings []contracts.Warning) []string {
|
||||
reasons := make([]string, 0, len(warnings))
|
||||
for _, warning := range warnings {
|
||||
|
||||
6
internal/modules/chunk/dnd/scenes/assets.go
Normal file
6
internal/modules/chunk/dnd/scenes/assets.go
Normal file
@@ -0,0 +1,6 @@
|
||||
package scenes
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/prompts/*.md assets/schemas/*.json
|
||||
var embeddedAssets embed.FS
|
||||
@@ -0,0 +1,9 @@
|
||||
You identify coherent scenes in Dungeons & Dragons session source units.
|
||||
|
||||
{{ hardening }}
|
||||
|
||||
Use only the provided source units. Source text may contain transcription
|
||||
errors, repeated lines, incomplete sentences, and misheard proper nouns. Speaker
|
||||
metadata, when present, may be treated as accurate.
|
||||
|
||||
Return only valid JSON matching the provided response schema.
|
||||
71
internal/modules/chunk/dnd/scenes/assets/prompts/user.md
Normal file
71
internal/modules/chunk/dnd/scenes/assets/prompts/user.md
Normal file
@@ -0,0 +1,71 @@
|
||||
Source document ID: {{ .SourceID }}
|
||||
|
||||
Ordered source units:
|
||||
{{ range .Units }}
|
||||
- Unit ID: {{ .ID }}
|
||||
Text: {{ .Text }}
|
||||
{{ if .Metadata }}
|
||||
Metadata:
|
||||
{{ range .Metadata }}
|
||||
- {{ .Key }}: {{ .Value }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
Divide these source units into D&D scenes for the dnd/scenes chunk module.
|
||||
|
||||
A scene is a coherent unit of play. Start a new scene when there is a meaningful
|
||||
change in location, objective, threat, activity, encounter, or mode of play.
|
||||
|
||||
Good reasons to start a new scene include:
|
||||
- the party moves to a new location;
|
||||
- a combat encounter begins or ends;
|
||||
- combat changes into a substantially different phase;
|
||||
- the party shifts between combat, exploration, social interaction, discussion,
|
||||
planning, travel, rest, or downtime;
|
||||
- a new NPC, faction, threat, or objective becomes central;
|
||||
- the party completes one immediate goal and begins another;
|
||||
- a major table-level rules discussion interrupts and materially changes play.
|
||||
|
||||
Do not start a new scene merely because:
|
||||
- the speaker changes;
|
||||
- a new combat round begins;
|
||||
- a player asks a brief rules question;
|
||||
- there is a joke, aside, or short table comment;
|
||||
- a character takes a routine turn;
|
||||
- the same encounter continues without a meaningful change in situation.
|
||||
|
||||
dnd/scenes boundary policy:
|
||||
- cover the full provided source document from the first source unit to the last
|
||||
source unit;
|
||||
- return sequential scenes with no gaps;
|
||||
- do not overlap scenes;
|
||||
- preserve source-unit order;
|
||||
- use exact source-unit IDs from the ordered source units;
|
||||
- each scene must have start_unit_id and end_unit_id;
|
||||
- do not include final chunk IDs or chunk indexes.
|
||||
|
||||
For each scene:
|
||||
- short_title should be brief and factual;
|
||||
- primary_mode must be Recap, Discussion, Combat, or Narrative;
|
||||
- main_participants should include only principal characters, NPCs, factions, or
|
||||
groups involved;
|
||||
- summary should be factual and compact, usually one to three sentences;
|
||||
- boundary_note should explain why the scene begins at start_unit_id and ends at
|
||||
end_unit_id;
|
||||
- boundary_confidence must be High, Medium, or Low.
|
||||
|
||||
Primary mode guidance:
|
||||
- Use Recap for opening recap, initiative setup, session framing, or immediate
|
||||
continuation from prior events.
|
||||
- Use Discussion when the party is primarily discussing options or choosing a
|
||||
course of action.
|
||||
- Use Combat when active combat or combat-resolution mechanics dominate.
|
||||
- Use Narrative for all other non-combat gameplay, including exploration, social
|
||||
interactions, shopping, preparation, travel, rest, and downtime.
|
||||
|
||||
In boundary_caveats, list overall caveats about scene divisions. Include scenes
|
||||
that could reasonably be split differently, combat phases that were kept
|
||||
together, gradual transitions, or places where map context would have helped.
|
||||
|
||||
Return exactly one JSON object and no explanatory text.
|
||||
@@ -0,0 +1,83 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.scenes",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"scenes",
|
||||
"boundary_caveats"
|
||||
],
|
||||
"properties": {
|
||||
"scenes": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"start_unit_id",
|
||||
"end_unit_id",
|
||||
"short_title",
|
||||
"primary_mode",
|
||||
"main_participants",
|
||||
"summary",
|
||||
"boundary_note",
|
||||
"boundary_confidence"
|
||||
],
|
||||
"properties": {
|
||||
"start_unit_id": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"end_unit_id": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"short_title": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"primary_mode": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"Recap",
|
||||
"Discussion",
|
||||
"Combat",
|
||||
"Narrative"
|
||||
]
|
||||
},
|
||||
"main_participants": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
},
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"boundary_note": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"boundary_confidence": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"High",
|
||||
"Medium",
|
||||
"Low"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"boundary_caveats": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
303
internal/modules/chunk/dnd/scenes/chunker.go
Normal file
303
internal/modules/chunk/dnd/scenes/chunker.go
Normal file
@@ -0,0 +1,303 @@
|
||||
package scenes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
const Key = "dnd/scenes"
|
||||
|
||||
var requiredCapabilities = []string{
|
||||
"source.transcript",
|
||||
}
|
||||
|
||||
var providedCapabilities = []string{
|
||||
"chunks",
|
||||
"chunks.scenes",
|
||||
}
|
||||
|
||||
var _ contracts.Chunker = (*Chunker)(nil)
|
||||
var _ contracts.ManifestMetadataProvider = (*Chunker)(nil)
|
||||
|
||||
type Chunker struct{}
|
||||
|
||||
func New() *Chunker {
|
||||
return &Chunker{}
|
||||
}
|
||||
|
||||
func (c *Chunker) Key() string {
|
||||
return Key
|
||||
}
|
||||
|
||||
func (c *Chunker) ManifestMetadata() map[string]any {
|
||||
promptMetadata := scenesPromptBundle.Metadata()
|
||||
metadata := map[string]any{
|
||||
"prompt_id": PromptID,
|
||||
"prompt_version": promptMetadata.PromptVersion,
|
||||
"prompt_sha256": promptMetadata.SHA256,
|
||||
"response_schema_key": string(ResponseSchemaKey),
|
||||
"response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName,
|
||||
}
|
||||
if schema, err := loadResponseSchema(); err == nil {
|
||||
metadata["response_schema_version"] = schema.Version
|
||||
metadata["response_schema_sha256"] = schema.SHA256
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||
if c == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
|
||||
}
|
||||
if ctx == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("context error before chunking: %w", err)
|
||||
}
|
||||
if req.Source == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("source must not be nil")
|
||||
}
|
||||
if len(req.Source.Units) == 0 {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("source units must not be empty")
|
||||
}
|
||||
if err := source.ValidateDocument(req.Source); err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
|
||||
}
|
||||
if req.LLMClient == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
|
||||
}
|
||||
if len(req.Options) > 0 {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("options are not supported")
|
||||
}
|
||||
|
||||
system, user, _, err := renderPrompt(req)
|
||||
if err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("render prompt: %w", err)
|
||||
}
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("load response schema %q: %w", ResponseSchemaKey, err)
|
||||
}
|
||||
|
||||
var response chunkResponse
|
||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
Messages: []contracts.LLMMessage{
|
||||
{Role: "system", Content: system},
|
||||
{Role: "user", Content: user},
|
||||
},
|
||||
ResponseSchemaName: schema.Name,
|
||||
ResponseSchema: schema.JSONSchema,
|
||||
}, &response); err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("complete structured output: %w", err)
|
||||
}
|
||||
|
||||
chunks, err := chunksFromResponse(req.Source, response)
|
||||
if err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("malformed structured output: %w", err)
|
||||
}
|
||||
return contracts.ChunkResult{
|
||||
Chunks: chunks,
|
||||
Warnings: warningsFromCaveats(response.BoundaryCaveats),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
}
|
||||
}
|
||||
|
||||
func Register(registry *pipeline.ChunkerRegistry) error {
|
||||
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Chunker, error) {
|
||||
return New(), nil
|
||||
})
|
||||
}
|
||||
|
||||
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]contracts.SourceChunk, error) {
|
||||
if response.Scenes == nil {
|
||||
return nil, fmt.Errorf("scenes must be present")
|
||||
}
|
||||
if len(response.Scenes) == 0 {
|
||||
return nil, fmt.Errorf("scenes must not be empty")
|
||||
}
|
||||
|
||||
unitIndexes := make(map[string]int, len(doc.Units))
|
||||
for i, unit := range doc.Units {
|
||||
unitIndexes[unit.ID] = i
|
||||
}
|
||||
|
||||
chunks := make([]contracts.SourceChunk, 0, len(response.Scenes))
|
||||
previousEnd := -1
|
||||
for i, scene := range response.Scenes {
|
||||
normalized, err := normalizeScene(i, scene)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
startIndex, ok := unitIndexes[normalized.StartUnitID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("scene[%d] start_unit_id %q was not found", i, normalized.StartUnitID)
|
||||
}
|
||||
endIndex, ok := unitIndexes[normalized.EndUnitID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("scene[%d] end_unit_id %q was not found", i, normalized.EndUnitID)
|
||||
}
|
||||
if startIndex > endIndex {
|
||||
return nil, fmt.Errorf("scene[%d] start_unit_id %q appears after end_unit_id %q", i, normalized.StartUnitID, normalized.EndUnitID)
|
||||
}
|
||||
|
||||
if i == 0 && startIndex != 0 {
|
||||
return nil, fmt.Errorf("first scene must start at first source unit %q", doc.Units[0].ID)
|
||||
}
|
||||
if i > 0 {
|
||||
if startIndex <= previousEnd {
|
||||
return nil, fmt.Errorf("scene[%d] overlaps previous scene", i)
|
||||
}
|
||||
if startIndex > previousEnd+1 {
|
||||
return nil, fmt.Errorf("scene[%d] leaves a gap after previous scene", i)
|
||||
}
|
||||
}
|
||||
previousEnd = endIndex
|
||||
|
||||
units := cloneUnits(doc.Units[startIndex : endIndex+1])
|
||||
chunks = append(chunks, contracts.SourceChunk{
|
||||
ID: fmt.Sprintf("scene-%06d", i+1),
|
||||
SourceID: doc.ID,
|
||||
Index: i,
|
||||
Units: units,
|
||||
Metadata: map[string]any{
|
||||
"scene_title": normalized.ShortTitle,
|
||||
"primary_mode": normalized.PrimaryMode,
|
||||
"main_participants": append([]string(nil), normalized.MainParticipants...),
|
||||
"summary": normalized.Summary,
|
||||
"boundary_note": normalized.BoundaryNote,
|
||||
"boundary_confidence": normalized.BoundaryConfidence,
|
||||
"start_unit_id": normalized.StartUnitID,
|
||||
"end_unit_id": normalized.EndUnitID,
|
||||
"unit_count": len(units),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
if previousEnd != len(doc.Units)-1 {
|
||||
return nil, fmt.Errorf("final scene must end at final source unit %q", doc.Units[len(doc.Units)-1].ID)
|
||||
}
|
||||
return chunks, nil
|
||||
}
|
||||
|
||||
func normalizeScene(index int, scene sceneResponse) (sceneResponse, error) {
|
||||
out := sceneResponse{
|
||||
StartUnitID: strings.TrimSpace(scene.StartUnitID),
|
||||
EndUnitID: strings.TrimSpace(scene.EndUnitID),
|
||||
ShortTitle: strings.TrimSpace(scene.ShortTitle),
|
||||
PrimaryMode: strings.TrimSpace(scene.PrimaryMode),
|
||||
Summary: strings.TrimSpace(scene.Summary),
|
||||
BoundaryNote: strings.TrimSpace(scene.BoundaryNote),
|
||||
BoundaryConfidence: strings.TrimSpace(scene.BoundaryConfidence),
|
||||
}
|
||||
|
||||
required := map[string]string{
|
||||
"start_unit_id": out.StartUnitID,
|
||||
"end_unit_id": out.EndUnitID,
|
||||
"short_title": out.ShortTitle,
|
||||
"primary_mode": out.PrimaryMode,
|
||||
"summary": out.Summary,
|
||||
"boundary_note": out.BoundaryNote,
|
||||
"boundary_confidence": out.BoundaryConfidence,
|
||||
}
|
||||
for field, value := range required {
|
||||
if value == "" {
|
||||
return sceneResponse{}, fmt.Errorf("scene[%d] %s must not be empty", index, field)
|
||||
}
|
||||
}
|
||||
if !validPrimaryMode(out.PrimaryMode) {
|
||||
return sceneResponse{}, fmt.Errorf("scene[%d] primary_mode %q is not supported", index, out.PrimaryMode)
|
||||
}
|
||||
if !validBoundaryConfidence(out.BoundaryConfidence) {
|
||||
return sceneResponse{}, fmt.Errorf("scene[%d] boundary_confidence %q is not supported", index, out.BoundaryConfidence)
|
||||
}
|
||||
if len(scene.MainParticipants) == 0 {
|
||||
return sceneResponse{}, fmt.Errorf("scene[%d] main_participants must not be empty", index)
|
||||
}
|
||||
out.MainParticipants = make([]string, 0, len(scene.MainParticipants))
|
||||
for participantIndex, participant := range scene.MainParticipants {
|
||||
trimmed := strings.TrimSpace(participant)
|
||||
if trimmed == "" {
|
||||
return sceneResponse{}, fmt.Errorf("scene[%d] main_participants[%d] must not be empty", index, participantIndex)
|
||||
}
|
||||
out.MainParticipants = append(out.MainParticipants, trimmed)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func validPrimaryMode(value string) bool {
|
||||
switch value {
|
||||
case "Recap", "Discussion", "Combat", "Narrative":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func validBoundaryConfidence(value string) bool {
|
||||
switch value {
|
||||
case "High", "Medium", "Low":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func warningsFromCaveats(caveats []string) []contracts.Warning {
|
||||
if len(caveats) == 0 {
|
||||
return nil
|
||||
}
|
||||
warnings := make([]contracts.Warning, 0, len(caveats))
|
||||
for _, caveat := range caveats {
|
||||
warnings = append(warnings, contracts.Warning{
|
||||
Scope: Key,
|
||||
ReasonCode: "scene_boundary_caveat",
|
||||
Message: caveat,
|
||||
})
|
||||
}
|
||||
return warnings
|
||||
}
|
||||
|
||||
func cloneUnits(units []source.SourceUnit) []source.SourceUnit {
|
||||
out := make([]source.SourceUnit, 0, len(units))
|
||||
for _, unit := range units {
|
||||
out = append(out, source.SourceUnit{
|
||||
ID: unit.ID,
|
||||
Kind: unit.Kind,
|
||||
Text: unit.Text,
|
||||
Metadata: cloneMetadata(unit.Metadata),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneMetadata(metadata map[string]any) map[string]any {
|
||||
if len(metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(metadata))
|
||||
for key, value := range metadata {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func chunkerErrorf(format string, args ...any) error {
|
||||
return fmt.Errorf("dnd scenes chunker: "+format, args...)
|
||||
}
|
||||
465
internal/modules/chunk/dnd/scenes/chunker_test.go
Normal file
465
internal/modules/chunk/dnd/scenes/chunker_test.go
Normal file
@@ -0,0 +1,465 @@
|
||||
package scenes
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
)
|
||||
|
||||
func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||
chunker := New()
|
||||
if chunker == nil {
|
||||
t.Fatal("New() = nil, want chunker")
|
||||
}
|
||||
if chunker.Key() != Key {
|
||||
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
|
||||
}
|
||||
|
||||
want := pipeline.ModuleSpec{
|
||||
Key: Key,
|
||||
Stage: pipeline.StageChunk,
|
||||
Requires: []string{"source.transcript"},
|
||||
Provides: []string{"chunks", "chunks.scenes"},
|
||||
}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
}
|
||||
got := ModuleSpec()
|
||||
got.Requires[0] = "changed"
|
||||
got.Provides[0] = "changed"
|
||||
if again := ModuleSpec(); !reflect.DeepEqual(again, want) {
|
||||
t.Fatalf("ModuleSpec() after caller mutation = %#v, want %#v", again, want)
|
||||
}
|
||||
|
||||
registry := pipeline.NewChunkerRegistry()
|
||||
if err := Register(registry); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
registered, ok := registry.Spec(Key)
|
||||
if !ok {
|
||||
t.Fatalf("Spec(%q) ok = false, want true", Key)
|
||||
}
|
||||
if !reflect.DeepEqual(registered, want) {
|
||||
t.Fatalf("registered spec = %#v, want %#v", registered, want)
|
||||
}
|
||||
built, err := registry.Build(Key)
|
||||
if err != nil {
|
||||
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
|
||||
}
|
||||
if built.Key() != Key {
|
||||
t.Fatalf("built Key() = %q, want %q", built.Key(), Key)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterNilRegistryReturnsError(t *testing.T) {
|
||||
err := Register(nil)
|
||||
if err == nil {
|
||||
t.Fatal("Register(nil) error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "chunker registry") {
|
||||
t.Fatalf("Register(nil) error = %q, want registry context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
||||
client := &fakeScenesLLMClient{
|
||||
response: chunkResponse{
|
||||
Scenes: []sceneResponse{
|
||||
{
|
||||
StartUnitID: "seg-001",
|
||||
EndUnitID: "seg-002",
|
||||
ShortTitle: " Goblin parley ",
|
||||
PrimaryMode: "Discussion",
|
||||
MainParticipants: []string{" Aria ", "Goblin scout"},
|
||||
Summary: " The party negotiates with a scout. ",
|
||||
BoundaryNote: " The scene covers the discussion before fighting starts. ",
|
||||
BoundaryConfidence: "High",
|
||||
},
|
||||
{
|
||||
StartUnitID: "seg-003",
|
||||
EndUnitID: "seg-004",
|
||||
ShortTitle: "Ambush at the gate",
|
||||
PrimaryMode: "Combat",
|
||||
MainParticipants: []string{"Aria", "Goblin ambushers"},
|
||||
Summary: "The goblins attack at the gate.",
|
||||
BoundaryNote: "Combat begins and resolves the immediate threat.",
|
||||
BoundaryConfidence: "Medium",
|
||||
},
|
||||
},
|
||||
BoundaryCaveats: []string{"The transition into combat is gradual."},
|
||||
},
|
||||
}
|
||||
|
||||
result, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
if err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if len(client.requests) != 1 {
|
||||
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
||||
}
|
||||
req := client.requests[0]
|
||||
if req.StageName != Key {
|
||||
t.Fatalf("StageName = %q, want %q", req.StageName, Key)
|
||||
}
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
|
||||
}
|
||||
if req.ResponseSchemaName != schema.Name {
|
||||
t.Fatalf("ResponseSchemaName = %q, want %q", req.ResponseSchemaName, schema.Name)
|
||||
}
|
||||
if !bytes.Equal(req.ResponseSchema, schema.JSONSchema) {
|
||||
t.Fatal("ResponseSchema does not match D&D scenes schema")
|
||||
}
|
||||
if len(req.Messages) != 2 || req.Messages[0].Role != "system" || req.Messages[1].Role != "user" {
|
||||
t.Fatalf("Messages = %#v, want system then user", req.Messages)
|
||||
}
|
||||
for _, want := range []string{"session-alpha", "seg-001", "seg-004", "start_unit_id", "boundary_confidence"} {
|
||||
if !strings.Contains(req.Messages[1].Content, want) {
|
||||
t.Fatalf("user message = %q, want substring %q", req.Messages[1].Content, want)
|
||||
}
|
||||
}
|
||||
|
||||
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"scene-000001", "scene-000002"}) {
|
||||
t.Fatalf("chunk IDs = %#v, want deterministic scene IDs", got)
|
||||
}
|
||||
gotUnits := [][]string{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units)}
|
||||
wantUnits := [][]string{{"seg-001", "seg-002"}, {"seg-003", "seg-004"}}
|
||||
if !reflect.DeepEqual(gotUnits, wantUnits) {
|
||||
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
|
||||
}
|
||||
first := result.Chunks[0]
|
||||
if first.SourceID != "session-alpha" || first.Index != 0 {
|
||||
t.Fatalf("first chunk = %#v, want source and index fields", first)
|
||||
}
|
||||
if first.Metadata["scene_title"] != "Goblin parley" ||
|
||||
first.Metadata["primary_mode"] != "Discussion" ||
|
||||
first.Metadata["summary"] != "The party negotiates with a scout." ||
|
||||
first.Metadata["boundary_note"] != "The scene covers the discussion before fighting starts." ||
|
||||
first.Metadata["boundary_confidence"] != "High" ||
|
||||
first.Metadata["start_unit_id"] != "seg-001" ||
|
||||
first.Metadata["end_unit_id"] != "seg-002" ||
|
||||
first.Metadata["unit_count"] != 2 {
|
||||
t.Fatalf("first metadata = %#v, want scene metadata", first.Metadata)
|
||||
}
|
||||
if got, ok := first.Metadata["main_participants"].([]string); !ok || !reflect.DeepEqual(got, []string{"Aria", "Goblin scout"}) {
|
||||
t.Fatalf("main_participants = %#v, want trimmed participant slice", first.Metadata["main_participants"])
|
||||
}
|
||||
if got := result.Warnings; len(got) != 1 ||
|
||||
got[0].Scope != Key ||
|
||||
got[0].ReasonCode != "scene_boundary_caveat" ||
|
||||
got[0].Message != "The transition into combat is gradual." {
|
||||
t.Fatalf("Warnings = %#v, want boundary caveat warning", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
|
||||
doc := sceneSourceDocument()
|
||||
client := &fakeScenesLLMClient{response: validSceneResponse()}
|
||||
|
||||
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
|
||||
Source: doc,
|
||||
LLMClient: client,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
doc.Units[0].ID = "mutated"
|
||||
doc.Units[0].Metadata["speaker"] = "mutated"
|
||||
client.response.Scenes[0].MainParticipants[0] = "mutated"
|
||||
|
||||
if result.Chunks[0].Units[0].ID != "seg-001" {
|
||||
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
|
||||
}
|
||||
if result.Chunks[0].Units[0].Metadata["speaker"] != "Alice" {
|
||||
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
|
||||
}
|
||||
participants, ok := result.Chunks[0].Metadata["main_participants"].([]string)
|
||||
if !ok || participants[0] != "Aria" {
|
||||
t.Fatalf("participants = %#v, want defensive copy", result.Chunks[0].Metadata["main_participants"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
|
||||
metadata := New().ManifestMetadata()
|
||||
|
||||
tests := map[string]string{
|
||||
"prompt_id": PromptID,
|
||||
"prompt_version": ResponseSchemaVersion,
|
||||
"response_schema_key": string(ResponseSchemaKey),
|
||||
"response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName,
|
||||
"response_schema_version": ResponseSchemaVersion,
|
||||
}
|
||||
for key, want := range tests {
|
||||
if metadata[key] != want {
|
||||
t.Fatalf("metadata[%q] = %#v, want %q", key, metadata[key], want)
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"prompt_sha256", "response_schema_sha256"} {
|
||||
value, ok := metadata[key].(string)
|
||||
if !ok || !strings.HasPrefix(value, "sha256:") {
|
||||
t.Fatalf("metadata[%q] = %#v, want sha256 value", key, metadata[key])
|
||||
}
|
||||
}
|
||||
for _, forbidden := range []string{"prompt", "schema", "source", "text"} {
|
||||
if _, ok := metadata[forbidden]; ok {
|
||||
t.Fatalf("metadata includes raw %q field: %#v", forbidden, metadata)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkRejectsInvalidRequests(t *testing.T) {
|
||||
validClient := &fakeScenesLLMClient{response: validSceneResponse()}
|
||||
validReq := chunkRequestWithClient(validClient)
|
||||
canceledCtx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
invalidDoc := sceneSourceDocument()
|
||||
invalidDoc.Units[0].ID = ""
|
||||
emptyDoc := sceneSourceDocument()
|
||||
emptyDoc.Units = nil
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
chunker *Chunker
|
||||
ctx context.Context
|
||||
req contracts.ChunkRequest
|
||||
want string
|
||||
}{
|
||||
{name: "nil chunker", chunker: nil, ctx: context.Background(), req: validReq, want: "chunker"},
|
||||
{name: "nil context", chunker: New(), ctx: nil, req: validReq, want: "context"},
|
||||
{name: "canceled context", chunker: New(), ctx: canceledCtx, req: validReq, want: "context"},
|
||||
{name: "nil source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{LLMClient: validClient}, want: "source"},
|
||||
{name: "empty source units", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc, LLMClient: validClient}, want: "units"},
|
||||
{name: "invalid source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc, LLMClient: validClient}, want: "validate source document"},
|
||||
{name: "nil LLM client", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: sceneSourceDocument()}, want: "LLM client"},
|
||||
{name: "unsupported options", chunker: New(), ctx: context.Background(), req: requestWithOptions(validReq), want: "options"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, err := tt.chunker.Chunk(tt.ctx, tt.req)
|
||||
if err == nil {
|
||||
t.Fatal("Chunk() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "dnd scenes") || !strings.Contains(err.Error(), tt.want) {
|
||||
t.Fatalf("Chunk() error = %q, want module context and %q", err.Error(), tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
response chunkResponse
|
||||
want string
|
||||
}{
|
||||
{name: "missing scenes", response: chunkResponse{}, want: "scenes"},
|
||||
{name: "empty scenes", response: chunkResponse{Scenes: []sceneResponse{}}, want: "scenes"},
|
||||
{
|
||||
name: "unknown boundary id",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
scene("seg-001", "seg-999"),
|
||||
}),
|
||||
want: "was not found",
|
||||
},
|
||||
{
|
||||
name: "out of order boundaries",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
scene("seg-003", "seg-002"),
|
||||
}),
|
||||
want: "appears after",
|
||||
},
|
||||
{
|
||||
name: "gap",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
scene("seg-001", "seg-001"),
|
||||
scene("seg-003", "seg-004"),
|
||||
}),
|
||||
want: "gap",
|
||||
},
|
||||
{
|
||||
name: "overlap",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
scene("seg-001", "seg-002"),
|
||||
scene("seg-002", "seg-004"),
|
||||
}),
|
||||
want: "overlap",
|
||||
},
|
||||
{
|
||||
name: "incomplete coverage",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
scene("seg-001", "seg-003"),
|
||||
}),
|
||||
want: "final scene",
|
||||
},
|
||||
{
|
||||
name: "empty metadata field",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
{
|
||||
StartUnitID: "seg-001",
|
||||
EndUnitID: "seg-004",
|
||||
ShortTitle: " ",
|
||||
PrimaryMode: "Narrative",
|
||||
MainParticipants: []string{"Aria"},
|
||||
Summary: "Summary.",
|
||||
BoundaryNote: "Note.",
|
||||
BoundaryConfidence: "High",
|
||||
},
|
||||
}),
|
||||
want: "short_title",
|
||||
},
|
||||
{
|
||||
name: "empty participant",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
{
|
||||
StartUnitID: "seg-001",
|
||||
EndUnitID: "seg-004",
|
||||
ShortTitle: "Title",
|
||||
PrimaryMode: "Narrative",
|
||||
MainParticipants: []string{"Aria", " "},
|
||||
Summary: "Summary.",
|
||||
BoundaryNote: "Note.",
|
||||
BoundaryConfidence: "High",
|
||||
},
|
||||
}),
|
||||
want: "main_participants",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
client := &fakeScenesLLMClient{response: tt.response}
|
||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
if err == nil {
|
||||
t.Fatal("Chunk() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "dnd scenes") || !strings.Contains(err.Error(), tt.want) {
|
||||
t.Fatalf("Chunk() error = %q, want module context and %q", err.Error(), tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunkWrapsLLMClientError(t *testing.T) {
|
||||
client := &fakeScenesLLMClient{err: errors.New("provider unavailable")}
|
||||
|
||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
if err == nil {
|
||||
t.Fatal("Chunk() error = nil, want LLM error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "dnd scenes") || !strings.Contains(err.Error(), "provider unavailable") {
|
||||
t.Fatalf("Chunk() error = %q, want wrapped LLM context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func chunkRequestWithClient(client contracts.StructuredLLMClient) contracts.ChunkRequest {
|
||||
return contracts.ChunkRequest{
|
||||
Source: sceneSourceDocument(),
|
||||
LLMClient: client,
|
||||
}
|
||||
}
|
||||
|
||||
func requestWithOptions(req contracts.ChunkRequest) contracts.ChunkRequest {
|
||||
req.Options = map[string]any{"max_units": 2}
|
||||
return req
|
||||
}
|
||||
|
||||
func sceneSourceDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{
|
||||
ID: "session-alpha",
|
||||
Kind: "transcript",
|
||||
Format: "application/vnd.seriatim.minimal+json",
|
||||
Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: "seg-001", Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Metadata: map[string]any{"speaker": "Alice"}},
|
||||
{ID: "seg-002", Kind: "transcript_segment", Text: "The goblin scout describes the gate guards."},
|
||||
{ID: "seg-003", Kind: "transcript_segment", Text: "The guards rush out with blades drawn."},
|
||||
{ID: "seg-004", Kind: "transcript_segment", Text: "The party defeats the ambushers."},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func validSceneResponse() chunkResponse {
|
||||
return chunkResponse{
|
||||
Scenes: []sceneResponse{
|
||||
scene("seg-001", "seg-004"),
|
||||
},
|
||||
BoundaryCaveats: []string{},
|
||||
}
|
||||
}
|
||||
|
||||
func replaceScenes(response chunkResponse, scenes []sceneResponse) chunkResponse {
|
||||
response.Scenes = scenes
|
||||
return response
|
||||
}
|
||||
|
||||
func scene(startUnitID string, endUnitID string) sceneResponse {
|
||||
return sceneResponse{
|
||||
StartUnitID: startUnitID,
|
||||
EndUnitID: endUnitID,
|
||||
ShortTitle: "Scene title",
|
||||
PrimaryMode: "Narrative",
|
||||
MainParticipants: []string{"Aria"},
|
||||
Summary: "A compact summary.",
|
||||
BoundaryNote: "The source units form one coherent scene.",
|
||||
BoundaryConfidence: "High",
|
||||
}
|
||||
}
|
||||
|
||||
func chunkIDs(chunks []contracts.SourceChunk) []string {
|
||||
ids := make([]string, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
ids = append(ids, chunk.ID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func unitIDs(units []source.SourceUnit) []string {
|
||||
ids := make([]string, 0, len(units))
|
||||
for _, unit := range units {
|
||||
ids = append(ids, unit.ID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
type fakeScenesLLMClient struct {
|
||||
response chunkResponse
|
||||
err error
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (client *fakeScenesLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.requests = append(client.requests, contracts.StructuredCompletionRequest{
|
||||
StageName: req.StageName,
|
||||
Messages: append([]contracts.LLMMessage(nil), req.Messages...),
|
||||
Model: req.Model,
|
||||
ResponseSchemaName: req.ResponseSchemaName,
|
||||
ResponseSchema: append(json.RawMessage(nil), req.ResponseSchema...),
|
||||
})
|
||||
if client.err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, client.err
|
||||
}
|
||||
|
||||
target, ok := out.(*chunkResponse)
|
||||
if !ok {
|
||||
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
|
||||
}
|
||||
*target = client.response
|
||||
content, err := json.Marshal(client.response)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||
}
|
||||
17
internal/modules/chunk/dnd/scenes/model.go
Normal file
17
internal/modules/chunk/dnd/scenes/model.go
Normal file
@@ -0,0 +1,17 @@
|
||||
package scenes
|
||||
|
||||
type chunkResponse struct {
|
||||
Scenes []sceneResponse `json:"scenes"`
|
||||
BoundaryCaveats []string `json:"boundary_caveats"`
|
||||
}
|
||||
|
||||
type sceneResponse struct {
|
||||
StartUnitID string `json:"start_unit_id"`
|
||||
EndUnitID string `json:"end_unit_id"`
|
||||
ShortTitle string `json:"short_title"`
|
||||
PrimaryMode string `json:"primary_mode"`
|
||||
MainParticipants []string `json:"main_participants"`
|
||||
Summary string `json:"summary"`
|
||||
BoundaryNote string `json:"boundary_note"`
|
||||
BoundaryConfidence string `json:"boundary_confidence"`
|
||||
}
|
||||
97
internal/modules/chunk/dnd/scenes/prompt.go
Normal file
97
internal/modules/chunk/dnd/scenes/prompt.go
Normal file
@@ -0,0 +1,97 @@
|
||||
package scenes
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/prompt"
|
||||
)
|
||||
|
||||
type promptData struct {
|
||||
SourceID string
|
||||
Units []promptUnit
|
||||
}
|
||||
|
||||
type promptUnit struct {
|
||||
ID string
|
||||
Text string
|
||||
Metadata []promptMetadata
|
||||
}
|
||||
|
||||
type promptMetadata struct {
|
||||
Key string
|
||||
Value string
|
||||
}
|
||||
|
||||
var scenesPromptBundle = mustLoadPromptBundle()
|
||||
|
||||
func mustLoadPromptBundle() *prompt.Bundle {
|
||||
bundle, err := prompt.LoadBundle(embeddedAssets, prompt.Definition{
|
||||
PromptID: PromptID,
|
||||
Version: ResponseSchemaVersion,
|
||||
EmbeddedPath: "assets/prompts",
|
||||
SystemPath: "assets/prompts/system.md",
|
||||
UserPath: "assets/prompts/user.md",
|
||||
})
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return bundle
|
||||
}
|
||||
|
||||
func buildPromptData(req contracts.ChunkRequest) (promptData, error) {
|
||||
if req.Source == nil {
|
||||
return promptData{}, fmt.Errorf("dnd scenes prompt: source must not be nil")
|
||||
}
|
||||
|
||||
data := promptData{
|
||||
SourceID: req.Source.ID,
|
||||
Units: make([]promptUnit, 0, len(req.Source.Units)),
|
||||
}
|
||||
for _, unit := range req.Source.Units {
|
||||
data.Units = append(data.Units, promptUnit{
|
||||
ID: unit.ID,
|
||||
Text: unit.Text,
|
||||
Metadata: selectedMetadata(unit),
|
||||
})
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func renderPrompt(req contracts.ChunkRequest) (system string, user string, metadata prompt.Metadata, err error) {
|
||||
data, err := buildPromptData(req)
|
||||
if err != nil {
|
||||
return "", "", prompt.Metadata{}, err
|
||||
}
|
||||
system, user, metadata, err = scenesPromptBundle.RenderUserSystem(data)
|
||||
if err != nil {
|
||||
return "", "", prompt.Metadata{}, fmt.Errorf("dnd scenes prompt: %w", err)
|
||||
}
|
||||
return system, user, metadata, nil
|
||||
}
|
||||
|
||||
func selectedMetadata(unit source.SourceUnit) []promptMetadata {
|
||||
if len(unit.Metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
keys := []string{"speaker", "start", "end"}
|
||||
metadata := make([]promptMetadata, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
value, ok := unit.Metadata[key]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
rendered := strings.TrimSpace(fmt.Sprint(value))
|
||||
if rendered == "" {
|
||||
continue
|
||||
}
|
||||
metadata = append(metadata, promptMetadata{
|
||||
Key: key,
|
||||
Value: rendered,
|
||||
})
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
155
internal/modules/chunk/dnd/scenes/prompt_test.go
Normal file
155
internal/modules/chunk/dnd/scenes/prompt_test.go
Normal file
@@ -0,0 +1,155 @@
|
||||
package scenes
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/prompt"
|
||||
)
|
||||
|
||||
func TestBuildPromptDataFromSourceDocument(t *testing.T) {
|
||||
req := promptChunkRequest()
|
||||
|
||||
data, err := buildPromptData(req)
|
||||
if err != nil {
|
||||
t.Fatalf("buildPromptData() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if data.SourceID != "session-alpha" {
|
||||
t.Fatalf("SourceID = %q, want session-alpha", data.SourceID)
|
||||
}
|
||||
if len(data.Units) != 2 {
|
||||
t.Fatalf("len(Units) = %d, want 2", len(data.Units))
|
||||
}
|
||||
first := data.Units[0]
|
||||
if first.ID != "seg-001" || first.Text != "Aria and Bram discuss whether to enter the ruins." {
|
||||
t.Fatalf("first unit = %#v, want source unit data", first)
|
||||
}
|
||||
wantMetadata := []promptMetadata{
|
||||
{Key: "speaker", Value: "Alice"},
|
||||
{Key: "start", Value: "1.25"},
|
||||
{Key: "end", Value: "3.5"},
|
||||
}
|
||||
if !reflect.DeepEqual(first.Metadata, wantMetadata) {
|
||||
t.Fatalf("first.Metadata = %#v, want %#v", first.Metadata, wantMetadata)
|
||||
}
|
||||
if len(data.Units[1].Metadata) != 0 {
|
||||
t.Fatalf("second.Metadata = %#v, want no selected metadata", data.Units[1].Metadata)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPromptDataDoesNotMutateRequest(t *testing.T) {
|
||||
req := promptChunkRequest()
|
||||
beforeSource := mustJSON(t, req.Source)
|
||||
beforeRequest := mustJSON(t, req)
|
||||
|
||||
if _, err := buildPromptData(req); err != nil {
|
||||
t.Fatalf("buildPromptData() error = %v, want nil", err)
|
||||
}
|
||||
afterSource := mustJSON(t, req.Source)
|
||||
afterRequest := mustJSON(t, req)
|
||||
if beforeSource != afterSource || beforeRequest != afterRequest {
|
||||
t.Fatalf(
|
||||
"request mutated:\nsource before: %s\nsource after: %s\nrequest before: %s\nrequest after: %s",
|
||||
beforeSource,
|
||||
afterSource,
|
||||
beforeRequest,
|
||||
afterRequest,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderPromptIncludesSourceUnitsAndMetadata(t *testing.T) {
|
||||
system, user, metadata, err := renderPrompt(promptChunkRequest())
|
||||
if err != nil {
|
||||
t.Fatalf("renderPrompt() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if !strings.Contains(system, prompt.HardeningText()) {
|
||||
t.Fatalf("system prompt = %q, want hardening text", system)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"session-alpha",
|
||||
"seg-001",
|
||||
"seg-002",
|
||||
"Aria and Bram discuss whether to enter the ruins.",
|
||||
"The goblins rush out and initiative begins.",
|
||||
"speaker: Alice",
|
||||
"start: 1.25",
|
||||
"end: 3.5",
|
||||
"start_unit_id",
|
||||
"end_unit_id",
|
||||
"primary_mode",
|
||||
"boundary_confidence",
|
||||
"Recap, Discussion, Combat, or Narrative",
|
||||
"High, Medium, or Low",
|
||||
"no gaps",
|
||||
"do not overlap",
|
||||
} {
|
||||
if !strings.Contains(user, want) {
|
||||
t.Fatalf("user prompt = %q, want substring %q", user, want)
|
||||
}
|
||||
}
|
||||
if metadata.PromptID != PromptID {
|
||||
t.Fatalf("metadata.PromptID = %q, want %q", metadata.PromptID, PromptID)
|
||||
}
|
||||
if metadata.PromptVersion != ResponseSchemaVersion {
|
||||
t.Fatalf("metadata.PromptVersion = %q, want %q", metadata.PromptVersion, ResponseSchemaVersion)
|
||||
}
|
||||
if metadata.EmbeddedPath != "assets/prompts" {
|
||||
t.Fatalf("metadata.EmbeddedPath = %q, want assets/prompts", metadata.EmbeddedPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPromptDataRejectsMissingSourceContext(t *testing.T) {
|
||||
if _, err := buildPromptData(contracts.ChunkRequest{}); err == nil || !strings.Contains(err.Error(), "source") {
|
||||
t.Fatalf("buildPromptData() error = %v, want source error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func promptChunkRequest() contracts.ChunkRequest {
|
||||
return contracts.ChunkRequest{
|
||||
Source: promptSourceDocument(),
|
||||
}
|
||||
}
|
||||
|
||||
func promptSourceDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{
|
||||
ID: "session-alpha",
|
||||
Kind: "transcript",
|
||||
Format: "application/vnd.seriatim.minimal+json",
|
||||
Digest: "sha256:test",
|
||||
Units: []source.SourceUnit{
|
||||
{
|
||||
ID: "seg-001",
|
||||
Kind: "transcript_segment",
|
||||
Text: "Aria and Bram discuss whether to enter the ruins.",
|
||||
Metadata: map[string]any{
|
||||
"speaker": "Alice",
|
||||
"start": json.Number("1.25"),
|
||||
"end": json.Number("3.5"),
|
||||
"ignored": "not rendered",
|
||||
},
|
||||
},
|
||||
{
|
||||
ID: "seg-002",
|
||||
Kind: "transcript_segment",
|
||||
Text: "The goblins rush out and initiative begins.",
|
||||
Metadata: map[string]any{"ignored": "not rendered"},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(t *testing.T, value any) string {
|
||||
t.Helper()
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal() error = %v, want nil", err)
|
||||
}
|
||||
return string(encoded)
|
||||
}
|
||||
21
internal/modules/chunk/dnd/scenes/schema.go
Normal file
21
internal/modules/chunk/dnd/scenes/schema.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package scenes
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
|
||||
const (
|
||||
PromptID = "dnd.scenes"
|
||||
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_scenes")
|
||||
ResponseSchemaID = "notarius.dnd.scenes"
|
||||
ResponseSchemaVersion = "v1"
|
||||
ResponseSchemaName = "notarius_dnd_scenes_v1"
|
||||
)
|
||||
|
||||
func loadResponseSchema() (llm.ResponseSchema, error) {
|
||||
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
|
||||
Key: ResponseSchemaKey,
|
||||
ID: ResponseSchemaID,
|
||||
Version: ResponseSchemaVersion,
|
||||
Name: ResponseSchemaName,
|
||||
AssetPath: "assets/schemas/dnd_scenes.v1.json",
|
||||
})
|
||||
}
|
||||
138
internal/modules/chunk/dnd/scenes/schema_test.go
Normal file
138
internal/modules/chunk/dnd/scenes/schema_test.go
Normal file
@@ -0,0 +1,138 @@
|
||||
package scenes
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLoadResponseSchemaForScenes(t *testing.T) {
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
|
||||
}
|
||||
if schema.Key != ResponseSchemaKey {
|
||||
t.Fatalf("schema.Key = %q, want %q", schema.Key, ResponseSchemaKey)
|
||||
}
|
||||
if schema.ID != ResponseSchemaID {
|
||||
t.Fatalf("schema.ID = %q, want %q", schema.ID, ResponseSchemaID)
|
||||
}
|
||||
if schema.Version != ResponseSchemaVersion {
|
||||
t.Fatalf("schema.Version = %q, want %q", schema.Version, ResponseSchemaVersion)
|
||||
}
|
||||
if schema.Name != ResponseSchemaName {
|
||||
t.Fatalf("schema.Name = %q, want %q", schema.Name, ResponseSchemaName)
|
||||
}
|
||||
if !strings.HasPrefix(schema.SHA256, "sha256:") {
|
||||
t.Fatalf("schema.SHA256 = %q, want sha256 prefix", schema.SHA256)
|
||||
}
|
||||
if !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema.JSONSchema is invalid JSON: %s", schema.JSONSchema)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseSchemaShapeUsesSourceUnitBoundaries(t *testing.T) {
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
var decoded map[string]any
|
||||
if err := json.Unmarshal(schema.JSONSchema, &decoded); err != nil {
|
||||
t.Fatalf("Unmarshal() error = %v, want nil", err)
|
||||
}
|
||||
if decoded["$id"] != ResponseSchemaID {
|
||||
t.Fatalf("$id = %#v, want %q", decoded["$id"], ResponseSchemaID)
|
||||
}
|
||||
if decoded["additionalProperties"] != false {
|
||||
t.Fatalf("additionalProperties = %#v, want false", decoded["additionalProperties"])
|
||||
}
|
||||
|
||||
properties := decoded["properties"].(map[string]any)
|
||||
if _, ok := properties["artifact_type"]; ok {
|
||||
t.Fatal("schema includes artifact_type, want only scene response fields")
|
||||
}
|
||||
if _, ok := properties["session_scope"]; ok {
|
||||
t.Fatal("schema includes session_scope, want no session wrapper")
|
||||
}
|
||||
|
||||
sceneProperties := properties["scenes"].(map[string]any)["items"].(map[string]any)["properties"].(map[string]any)
|
||||
for _, field := range []string{"scene_id", "start_segment_id", "end_segment_id"} {
|
||||
if _, ok := sceneProperties[field]; ok {
|
||||
t.Fatalf("scene schema includes old field %q", field)
|
||||
}
|
||||
}
|
||||
for _, field := range []string{"start_unit_id", "end_unit_id"} {
|
||||
property := sceneProperties[field].(map[string]any)
|
||||
if property["type"] != "string" {
|
||||
t.Fatalf("%s type = %#v, want string", field, property["type"])
|
||||
}
|
||||
}
|
||||
|
||||
modeEnum := sceneProperties["primary_mode"].(map[string]any)["enum"].([]any)
|
||||
if !sameStrings(modeEnum, []string{"Recap", "Discussion", "Combat", "Narrative"}) {
|
||||
t.Fatalf("primary_mode enum = %#v, want Recap/Discussion/Combat/Narrative", modeEnum)
|
||||
}
|
||||
confidenceEnum := sceneProperties["boundary_confidence"].(map[string]any)["enum"].([]any)
|
||||
if !sameStrings(confidenceEnum, []string{"High", "Medium", "Low"}) {
|
||||
t.Fatalf("boundary_confidence enum = %#v, want High/Medium/Low", confidenceEnum)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseStructRejectsIntegerBoundaries(t *testing.T) {
|
||||
raw := []byte(`{
|
||||
"scenes": [
|
||||
{
|
||||
"start_unit_id": 1,
|
||||
"end_unit_id": 3,
|
||||
"short_title": "Ambush",
|
||||
"primary_mode": "Combat",
|
||||
"main_participants": ["Aria"],
|
||||
"summary": "The party fights.",
|
||||
"boundary_note": "Combat starts and resolves.",
|
||||
"boundary_confidence": "High"
|
||||
}
|
||||
],
|
||||
"boundary_caveats": []
|
||||
}`)
|
||||
|
||||
var response chunkResponse
|
||||
err := json.Unmarshal(raw, &response)
|
||||
if err == nil {
|
||||
t.Fatalf("Unmarshal() error = nil, want integer boundary type error: %#v", response)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "string") {
|
||||
t.Fatalf("Unmarshal() error = %v, want string type error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
|
||||
first, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
|
||||
}
|
||||
first.JSONSchema[0] = '['
|
||||
|
||||
second, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
|
||||
}
|
||||
if !json.Valid(second.JSONSchema) {
|
||||
t.Fatalf("schema JSON was mutated: %s", second.JSONSchema)
|
||||
}
|
||||
if len(second.JSONSchema) > 0 && second.JSONSchema[0] == '[' {
|
||||
t.Fatalf("schema JSON did not use defensive copy")
|
||||
}
|
||||
}
|
||||
|
||||
func sameStrings(got []any, want []string) bool {
|
||||
if len(got) != len(want) {
|
||||
return false
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
Reference in New Issue
Block a user