Compare commits
7 Commits
f4d37f9557
...
a4d64f6f16
| Author | SHA1 | Date | |
|---|---|---|---|
| a4d64f6f16 | |||
| 39fcfba605 | |||
| 91bec3ae52 | |||
| c804cb4bca | |||
| ef8bd91d48 | |||
| 95a54505cf | |||
| 47013daa04 |
101
docs/integrations/seriatim.md
Normal file
101
docs/integrations/seriatim.md
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
# Seriatim Minimal Transcript JSON
|
||||||
|
|
||||||
|
This document describes the Seriatim input format currently accepted by the
|
||||||
|
`seriatim` input adapter.
|
||||||
|
|
||||||
|
## Adapter
|
||||||
|
|
||||||
|
- Module key: `seriatim`
|
||||||
|
- Document kind: `transcript`
|
||||||
|
- Unit kind: `transcript_segment`
|
||||||
|
- Source format: `application/vnd.seriatim.minimal+json`
|
||||||
|
|
||||||
|
The adapter parses raw Seriatim JSON into a generic `SourceDocument`. It does
|
||||||
|
not add transcript-specific fields to core source or runner contracts.
|
||||||
|
|
||||||
|
## Accepted Shape
|
||||||
|
|
||||||
|
The input must be a JSON object with top-level `metadata` and `segments` fields:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"id": "session-alpha",
|
||||||
|
"title": "Synthetic session transcript"
|
||||||
|
},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": "seg-001",
|
||||||
|
"start": 0,
|
||||||
|
"end": 4.5,
|
||||||
|
"speaker": "Narrator",
|
||||||
|
"text": "The stone door opens."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Extra compatible fields are ignored. Multiple top-level JSON values are
|
||||||
|
rejected.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
The adapter rejects:
|
||||||
|
|
||||||
|
- empty raw input;
|
||||||
|
- malformed JSON;
|
||||||
|
- missing, null, or non-object `metadata`;
|
||||||
|
- missing, null, non-array, or empty `segments`;
|
||||||
|
- empty segment IDs;
|
||||||
|
- segment IDs with leading or trailing whitespace;
|
||||||
|
- duplicate segment IDs;
|
||||||
|
- missing or empty `speaker`;
|
||||||
|
- missing, empty, invalid, non-finite, or negative `start`;
|
||||||
|
- missing, empty, invalid, non-finite, or negative `end`;
|
||||||
|
- `end` values before `start`;
|
||||||
|
- missing or empty `text`.
|
||||||
|
|
||||||
|
Segment text may keep leading or trailing whitespace, but it must not be empty
|
||||||
|
after trimming.
|
||||||
|
|
||||||
|
## Source Mapping
|
||||||
|
|
||||||
|
The adapter maps Seriatim input into the source model as follows:
|
||||||
|
|
||||||
|
- top-level `metadata` becomes `SourceDocument.Metadata`;
|
||||||
|
- `SourceDocument.Digest` is `sha256:<hex>` of the exact raw input bytes;
|
||||||
|
- `segment.id` becomes `SourceUnit.ID`;
|
||||||
|
- `segment.text` becomes `SourceUnit.Text`;
|
||||||
|
- each source unit has kind `transcript_segment`;
|
||||||
|
- segment `speaker`, `start`, and `end` are stored in source-unit metadata.
|
||||||
|
|
||||||
|
`SourceDocument.ID` is selected in this order:
|
||||||
|
|
||||||
|
1. the parse request source ID, after trimming;
|
||||||
|
2. `metadata.id`, when it is a non-empty string after trimming;
|
||||||
|
3. `metadata.source_id`, when it is a non-empty string after trimming;
|
||||||
|
4. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
|
||||||
|
|
||||||
|
## Metadata Keys
|
||||||
|
|
||||||
|
Seriatim unit metadata uses these keys:
|
||||||
|
|
||||||
|
- `speaker`: string speaker label;
|
||||||
|
- `start`: `json.Number` start value;
|
||||||
|
- `end`: `json.Number` end value.
|
||||||
|
|
||||||
|
The `internal/modules/input/seriatim` package provides typed accessors for
|
||||||
|
these metadata values.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
The module declares these provided capabilities for pipeline validation:
|
||||||
|
|
||||||
|
- `source.transcript`
|
||||||
|
- `transcript.speaker`
|
||||||
|
- `transcript.timestamps`
|
||||||
|
|
||||||
|
## Limits
|
||||||
|
|
||||||
|
Only the Seriatim minimal transcript shape described here is supported. Broader
|
||||||
|
Seriatim schema variants are not currently accepted as a compatibility contract.
|
||||||
@@ -23,7 +23,7 @@ In scope:
|
|||||||
- module metadata/capability requirements for pipeline validation;
|
- module metadata/capability requirements for pipeline validation;
|
||||||
- source-reference and schema validators in the extractor chain;
|
- source-reference and schema validators in the extractor chain;
|
||||||
- fake LLM tests;
|
- fake LLM tests;
|
||||||
- CLI-level integration test if the CLI path is ready.
|
- runner-level integration tests with fake infrastructure.
|
||||||
|
|
||||||
Out of scope:
|
Out of scope:
|
||||||
|
|
||||||
@@ -31,89 +31,71 @@ Out of scope:
|
|||||||
- NPC extraction;
|
- NPC extraction;
|
||||||
- combat extraction;
|
- combat extraction;
|
||||||
- cross-slice deduplication beyond simple deterministic merging;
|
- cross-slice deduplication beyond simple deterministic merging;
|
||||||
- broad D&D rules validation.
|
- broad D&D rules validation;
|
||||||
|
- a `notarius run` command.
|
||||||
|
|
||||||
## Proposed Stages
|
## Target End State
|
||||||
|
|
||||||
### Stage 1: Spell Artifact Schema
|
The repository should contain a real D&D spells extract-stage module at
|
||||||
|
`internal/modules/extract/dnd/spells`.
|
||||||
|
|
||||||
Define the D&D spell artifact model.
|
The spells module should be registered under the stable extractor key
|
||||||
|
`dnd/spells`. It should be selectable as a named artifact lane in pipeline
|
||||||
|
configuration, for example a lane named `spells` whose extractor module is
|
||||||
|
`dnd/spells`.
|
||||||
|
|
||||||
Initial shape:
|
The module should translate generic source chunks into spell-cast artifact
|
||||||
|
candidates with this spell payload:
|
||||||
|
|
||||||
```go
|
- `caster`;
|
||||||
type SpellCast struct {
|
- `spell`;
|
||||||
Player string `json:"player"`
|
- `effect`;
|
||||||
Spell string `json:"spell"`
|
- `narrative_description`.
|
||||||
Effect string `json:"effect"`
|
|
||||||
NarrativeDescription string `json:"narrative_description"`
|
|
||||||
SourceRefs []SourceRef `json:"source_refs"`
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Keep this schema inside the D&D spells extract module or a D&D artifact package,
|
The LLM structured response should also include source references for each
|
||||||
not inside core framework packages.
|
spell cast. The extractor should copy those references into the generic
|
||||||
|
artifact envelope rather than duplicating source references inside the durable
|
||||||
|
spell payload.
|
||||||
|
|
||||||
### Stage 2: Structured Response Schema
|
The caster is the in-world character or creature casting the spell, not the
|
||||||
|
table speaker. Speaker metadata from transcript source units may be used as
|
||||||
Add a structured response schema asset for spell extraction.
|
optional prompt context when present, but it should not be a required or durable
|
||||||
|
field in the spell payload.
|
||||||
The schema should require:
|
|
||||||
|
|
||||||
- spell-cast array;
|
|
||||||
- non-empty player, spell, effect, and narrative description fields;
|
|
||||||
- at least one source reference per spell cast.
|
|
||||||
|
|
||||||
### Stage 3: Prompt Assets
|
|
||||||
|
|
||||||
Add embedded prompt assets for D&D spell extraction.
|
|
||||||
|
|
||||||
Prompts should:
|
Prompts should:
|
||||||
|
|
||||||
- describe the generic source-unit input format;
|
- describe the generic source-unit input format;
|
||||||
- explain that source references must use source-unit IDs;
|
- explain that source references must use source-unit IDs exactly;
|
||||||
- avoid relying on transcript-specific fields except as optional metadata;
|
- avoid relying on transcript-specific fields except as optional metadata;
|
||||||
- request only spell-cast artifacts.
|
- request only D&D spell-cast artifacts.
|
||||||
|
|
||||||
### Stage 4: Process Module Implementation
|
The extractor should attach deterministic validators by default. Validation
|
||||||
|
should cover:
|
||||||
|
|
||||||
Implement `internal/modules/extract/dnd/spells`.
|
- spell payload shape;
|
||||||
|
- non-empty required spell fields;
|
||||||
|
- at least one source reference per spell cast;
|
||||||
|
- source references that validate against the source document.
|
||||||
|
|
||||||
The extractor should:
|
The module should declare flat capabilities for pipeline validation. Initial
|
||||||
|
capabilities should require chunked transcript source material and provide a
|
||||||
|
D&D spell-cast artifact capability.
|
||||||
|
|
||||||
- satisfy the framework `Extractor` contract;
|
Implementation staging belongs in
|
||||||
- declare module metadata for pipeline-profile validation;
|
[`implementation.md`](implementation.md).
|
||||||
- build LLM messages from a source document or source chunk;
|
|
||||||
- call the structured LLM client;
|
|
||||||
- return artifact candidates with source references;
|
|
||||||
- attach its validator chain.
|
|
||||||
|
|
||||||
### Stage 5: Validators And Tests
|
## Fixtures And Tests
|
||||||
|
|
||||||
Wire deterministic validators:
|
The checkpoint should add synthetic fixtures and focused tests for:
|
||||||
|
|
||||||
- schema/shape validation;
|
- successful spell extraction with a fake structured LLM client;
|
||||||
- source-reference validation;
|
- empty spell-cast results;
|
||||||
- required-field validation if not covered by schema handling.
|
|
||||||
|
|
||||||
Add tests using a fake structured LLM client:
|
|
||||||
|
|
||||||
- successful spell extraction;
|
|
||||||
- empty result;
|
|
||||||
- invalid source reference rejection;
|
|
||||||
- malformed structured output handling;
|
- malformed structured output handling;
|
||||||
- stable output ordering.
|
- invalid source-reference rejection;
|
||||||
|
- stable output ordering;
|
||||||
### Stage 6: CLI Integration
|
- pipeline-profile selection of a `spells` artifact lane;
|
||||||
|
- runner integration from Seriatim input through the spells extractor using
|
||||||
If the CLI path is ready, add an end-to-end test using:
|
fake chunk and output modules.
|
||||||
|
|
||||||
```sh
|
|
||||||
notarius run dnd-session --input ./transcript.json --only spells
|
|
||||||
```
|
|
||||||
|
|
||||||
The test should use fake LLM wiring, fixture input, and a named pipeline profile
|
|
||||||
with a `spells` artifact lane.
|
|
||||||
|
|
||||||
## Done Criteria
|
## Done Criteria
|
||||||
|
|
||||||
@@ -123,8 +105,8 @@ with a `spells` artifact lane.
|
|||||||
- D&D concepts are contained in extract module/artifact packages and docs.
|
- D&D concepts are contained in extract module/artifact packages and docs.
|
||||||
- The spells module can be selected as a named artifact lane in pipeline
|
- The spells module can be selected as a named artifact lane in pipeline
|
||||||
configuration.
|
configuration.
|
||||||
- The first meaningful vertical slice is available through tests, and through
|
- The first meaningful vertical slice is available through tests.
|
||||||
CLI if the CLI path is ready.
|
- No CLI `run` behavior is documented or implemented until the CLI path exists.
|
||||||
|
|
||||||
## Review Questions
|
## Review Questions
|
||||||
|
|
||||||
|
|||||||
@@ -1,15 +1,14 @@
|
|||||||
# Implementation Plan: Checkpoint 5 Seriatim Input Module
|
# Implementation Plan: Checkpoint 6 D&D Spells Extractor
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
This is a staged implementation plan for
|
This is a staged implementation plan for
|
||||||
[`5-seriatim-input-module.md`](5-seriatim-input-module.md). It is intended for
|
[`6-dnd-spells-extractor.md`](6-dnd-spells-extractor.md). It is intended for an
|
||||||
an LLM coding agent to follow stage by stage.
|
LLM coding agent to follow stage by stage.
|
||||||
|
|
||||||
This plan implements only checkpoint 5. Do not add D&D extraction, real domain
|
This plan implements only checkpoint 6. Do not add D&D item, NPC, combat, or
|
||||||
prompts or schemas, LLM extraction calls, a `notarius run` command, broad
|
encounter extraction; do not add broad D&D rules validation; do not add a
|
||||||
Seriatim schema support, or transcript-specific behavior in core framework
|
`notarius run` command; and do not make core framework packages D&D-specific.
|
||||||
packages in this checkpoint.
|
|
||||||
|
|
||||||
## Policy Context
|
## Policy Context
|
||||||
|
|
||||||
@@ -20,404 +19,564 @@ Follow:
|
|||||||
|
|
||||||
Required boundaries:
|
Required boundaries:
|
||||||
|
|
||||||
- keep Seriatim JSON schema details inside `internal/modules/input/seriatim`;
|
- keep D&D-specific artifact semantics, prompt data shaping, and response
|
||||||
- keep core source, runner, pipeline, extractor, validator, LLM, and config
|
interpretation inside `internal/modules/extract/dnd/spells`;
|
||||||
|
- keep source-format details inside input modules and do not depend on concrete
|
||||||
|
Seriatim package types from the spells extractor;
|
||||||
|
- keep core source, runner, pipeline, LLM, prompt, config, and validator
|
||||||
packages source-agnostic and domain-agnostic;
|
packages source-agnostic and domain-agnostic;
|
||||||
- do not add transcript-specific typed fields to `SourceDocument`,
|
- use embedded prompt and schema assets instead of inline prompt/schema strings;
|
||||||
`SourceUnit`, runner contracts, or pipeline contracts;
|
- register the extractor through the existing extractor registry;
|
||||||
- preserve transcript-specific values only as source metadata conventions;
|
|
||||||
- register the input module through the existing input adapter registry instead
|
|
||||||
of adding ad hoc conditionals;
|
|
||||||
- use flat capability strings in module metadata;
|
- use flat capability strings in module metadata;
|
||||||
- keep future or planned behavior in `docs/roadmap/` until implemented.
|
- keep future or planned behavior in `docs/roadmap/` until implemented.
|
||||||
|
|
||||||
## Global Implementation Decisions
|
## Global Implementation Decisions
|
||||||
|
|
||||||
- Add no new third-party dependency. Use `encoding/json` with
|
- Add no new third-party dependency.
|
||||||
`Decoder.UseNumber` for Seriatim JSON parsing.
|
- Use `dnd/spells` as the stable extractor module key.
|
||||||
- Use `seriatim` as the stable input adapter key.
|
- Put concrete extractor code under `internal/modules/extract/dnd/spells`.
|
||||||
- Put all concrete Seriatim input code under
|
- Use `dnd.spell_cast` as the artifact type and `v1` as the schema version.
|
||||||
`internal/modules/input/seriatim`.
|
- Use `dnd.spell_casts` as the extractor-provided capability.
|
||||||
- Expose a small module API:
|
- The extractor module spec must require `chunks` and `source.transcript`.
|
||||||
|
It must not require `transcript.speaker` or `transcript.timestamps`; those
|
||||||
|
metadata values may be included in prompts when present, but extraction must
|
||||||
|
not depend on Seriatim-specific helper APIs.
|
||||||
|
- Keep durable artifact payloads focused on spell data:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
const Key = "seriatim"
|
type SpellCast struct {
|
||||||
|
Caster string `json:"caster"`
|
||||||
func New() *Adapter
|
Spell string `json:"spell"`
|
||||||
func ModuleSpec() pipeline.ModuleSpec
|
Effect string `json:"effect"`
|
||||||
func Register(registry *pipeline.InputAdapterRegistry) error
|
NarrativeDescription string `json:"narrative_description"`
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
- `ModuleSpec()` must return stage `pipeline.StageInput`, no required
|
- The LLM response model must include source references so the extractor can
|
||||||
capabilities, and these provided capabilities:
|
populate `artifacts.ArtifactCandidate.SourceRefs`, but source references must
|
||||||
`source.transcript`, `transcript.speaker`, and `transcript.timestamps`.
|
live in the generic artifact envelope for durable pipeline output rather than
|
||||||
- The Seriatim adapter should satisfy `contracts.InputAdapter`.
|
being duplicated inside the `SpellCast` payload.
|
||||||
- Use `source.SourceDocument.Kind = "transcript"`.
|
- `caster` means the in-world character or creature casting the spell. Do not
|
||||||
- Use `source.SourceDocument.Format =
|
use table speaker/player as the durable caster field. Source-unit `speaker`
|
||||||
"application/vnd.seriatim.minimal+json"`.
|
metadata may be rendered as optional prompt context when present.
|
||||||
- Use `source.SourceUnit.Kind = "transcript_segment"`.
|
- Use `source.SourceRef` JSON field names exactly as the core type defines
|
||||||
- Compute `SourceDocument.Digest` from the exact raw input bytes as
|
them: `source_id`, `start_unit_id`, and `end_unit_id`.
|
||||||
`sha256:<hex>`.
|
- Trim text fields before marshaling candidate payloads. Do not silently trim
|
||||||
- Resolve `SourceDocument.ID` in this order:
|
source-reference IDs; invalid source refs must be rejected by validation.
|
||||||
1. trimmed `contracts.ParseRequest.SourceID`, if non-empty;
|
- `Extractor.Validators()` must return module-owned deterministic validators by
|
||||||
2. trimmed string `metadata.id`, if present and non-empty;
|
default. Do not require pipeline profiles to configure validators explicitly
|
||||||
3. trimmed string `metadata.source_id`, if present and non-empty;
|
for the default checkpoint behavior.
|
||||||
4. deterministic fallback `seriatim:<first-16-hex-chars-of-raw-sha256>`.
|
- Treat an empty `spell_casts` response array as a successful empty extraction.
|
||||||
- Segment IDs become source unit IDs exactly after validation. Reject segment
|
Treat a missing or null `spell_casts` field as malformed structured output.
|
||||||
IDs with leading or trailing whitespace rather than silently rewriting them.
|
- Use synthetic D&D transcript fixture text only. Do not include private
|
||||||
- Copy top-level Seriatim `metadata` into `SourceDocument.Metadata`.
|
campaign transcript content.
|
||||||
- Store segment `speaker`, `start`, and `end` in `SourceUnit.Metadata` under
|
|
||||||
keys with those exact names.
|
|
||||||
- Store `start` and `end` as `json.Number` values so JSON serialization remains
|
|
||||||
numeric and the original decimal representation is preserved.
|
|
||||||
- Require top-level `metadata` to be present and be an object, but do not
|
|
||||||
require any specific metadata keys in checkpoint 5.
|
|
||||||
- Require top-level `segments` to be present and contain at least one segment.
|
|
||||||
- Reject unknown or extra JSON fields only if they prevent parsing the minimal
|
|
||||||
shape. Otherwise ignore them so the module can tolerate compatible Seriatim
|
|
||||||
additions.
|
|
||||||
- Return module-specific errors prefixed with useful Seriatim context, for
|
|
||||||
example `seriatim input: segment "s1" text must not be empty`.
|
|
||||||
- Keep examples free of private transcript content. Use synthetic fixture text.
|
|
||||||
|
|
||||||
## Stage 1: Seriatim Package Skeleton And External Model
|
## Stage 1: Domain Model, Module Skeleton, And Registry Metadata
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Create the Seriatim input module package, define the module-local JSON model,
|
Create the D&D spells extractor package, define the artifact and response
|
||||||
and add registry-facing module metadata without changing framework contracts.
|
models, and make the extractor discoverable through the existing extractor
|
||||||
|
registry without calling an LLM yet.
|
||||||
|
|
||||||
### Files To Add
|
### Files To Add Or Update
|
||||||
|
|
||||||
- `internal/modules/input/seriatim/adapter.go`
|
- `internal/modules/extract/dnd/spells/extractor.go`
|
||||||
- `internal/modules/input/seriatim/model.go`
|
- `internal/modules/extract/dnd/spells/model.go`
|
||||||
- `internal/modules/input/seriatim/metadata.go`
|
- `internal/modules/extract/dnd/spells/registry_test.go`
|
||||||
- `internal/modules/input/seriatim/registry_test.go`
|
|
||||||
|
|
||||||
### Required API
|
### Required API
|
||||||
|
|
||||||
Add:
|
Add:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
package seriatim
|
package spells
|
||||||
|
|
||||||
const Key = "seriatim"
|
const Key = "dnd/spells"
|
||||||
|
const ArtifactType = "dnd.spell_cast"
|
||||||
|
const SchemaVersion = "v1"
|
||||||
|
|
||||||
const (
|
type SpellCast struct {
|
||||||
DocumentKind = "transcript"
|
Caster string `json:"caster"`
|
||||||
UnitKind = "transcript_segment"
|
Spell string `json:"spell"`
|
||||||
Format = "application/vnd.seriatim.minimal+json"
|
Effect string `json:"effect"`
|
||||||
)
|
NarrativeDescription string `json:"narrative_description"`
|
||||||
|
}
|
||||||
|
|
||||||
const (
|
type Extractor struct{}
|
||||||
MetadataSpeaker = "speaker"
|
|
||||||
MetadataStart = "start"
|
|
||||||
MetadataEnd = "end"
|
|
||||||
)
|
|
||||||
|
|
||||||
type Adapter struct{}
|
func New() *Extractor
|
||||||
|
func (e *Extractor) Key() string
|
||||||
func New() *Adapter
|
func (e *Extractor) ArtifactType() string
|
||||||
func (a *Adapter) Key() string
|
func (e *Extractor) SchemaVersion() string
|
||||||
func (a *Adapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error)
|
func (e *Extractor) Validators() []contracts.Validator
|
||||||
|
func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error)
|
||||||
func ModuleSpec() pipeline.ModuleSpec
|
func ModuleSpec() pipeline.ModuleSpec
|
||||||
func Register(registry *pipeline.InputAdapterRegistry) error
|
func Register(registry *pipeline.ExtractorRegistry) error
|
||||||
```
|
```
|
||||||
|
|
||||||
Add typed metadata helpers:
|
Add unexported response structs in the same package:
|
||||||
|
|
||||||
```go
|
```go
|
||||||
func Speaker(unit source.SourceUnit) (string, bool)
|
type extractionResponse struct {
|
||||||
func Start(unit source.SourceUnit) (json.Number, bool)
|
SpellCasts []spellCastResponse `json:"spell_casts"`
|
||||||
func End(unit source.SourceUnit) (json.Number, bool)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Seriatim JSON Shape
|
|
||||||
|
|
||||||
Define module-local structs for the minimal external shape:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type transcript struct {
|
|
||||||
Metadata map[string]any `json:"metadata"`
|
|
||||||
Segments []segment `json:"segments"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type segment struct {
|
type spellCastResponse struct {
|
||||||
ID string `json:"id"`
|
Caster string `json:"caster"`
|
||||||
Start json.Number `json:"start"`
|
Spell string `json:"spell"`
|
||||||
End json.Number `json:"end"`
|
Effect string `json:"effect"`
|
||||||
Speaker string `json:"speaker"`
|
NarrativeDescription string `json:"narrative_description"`
|
||||||
Text string `json:"text"`
|
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Use an internal decode helper based on `json.NewDecoder(bytes.NewReader(raw))`
|
|
||||||
and `UseNumber`.
|
|
||||||
|
|
||||||
### Required Behavior
|
### Required Behavior
|
||||||
|
|
||||||
- `New()` returns a non-nil adapter.
|
- `New()` returns a non-nil extractor.
|
||||||
- `Adapter.Key()` returns `Key`.
|
- `ModuleSpec()` returns defensive slices with stage `pipeline.StageExtract`,
|
||||||
- `ModuleSpec()` returns defensive slices and the capability set listed in the
|
key `dnd/spells`, requires `chunks` and `source.transcript`, and provides
|
||||||
global decisions.
|
`dnd.spell_casts`.
|
||||||
- `Register()` calls `InputAdapterRegistry.RegisterWithSpec(ModuleSpec(), ...)`.
|
- `Register()` calls `ExtractorRegistry.RegisterWithSpec(ModuleSpec(), ...)`.
|
||||||
- `Register(nil)` returns an error from the registry path rather than panicking.
|
- `Register(nil)` returns an error from the registry path rather than panicking.
|
||||||
- Keep external JSON structs unexported.
|
- `Extract()` may return a clear not-yet-implemented error in this stage only.
|
||||||
|
- `Validators()` returns nil in this stage only; later stages must replace it
|
||||||
|
with the real validator chain.
|
||||||
|
|
||||||
### Required Tests
|
### Required Tests
|
||||||
|
|
||||||
- `New()` returns an adapter whose key is `seriatim`.
|
- `New()` returns an extractor whose key, artifact type, and schema version
|
||||||
- `ModuleSpec()` uses input stage and declares the required provided
|
match the constants.
|
||||||
capabilities.
|
- `ModuleSpec()` uses extract stage and declares the required capabilities.
|
||||||
- `Register()` makes the adapter buildable from an `InputAdapterRegistry`.
|
- Caller mutation of `ModuleSpec().Provides` or `ModuleSpec().Requires` does not
|
||||||
- Registry lookup returns the Seriatim module spec.
|
affect later calls.
|
||||||
|
- `Register()` makes the extractor buildable from an `ExtractorRegistry`.
|
||||||
|
- Registry lookup returns the spells module spec.
|
||||||
|
- `Register(nil)` returns an error containing extractor registry context.
|
||||||
|
|
||||||
### Validation
|
### Validation
|
||||||
|
|
||||||
Run:
|
Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -w internal/modules/input/seriatim
|
gofmt -w internal/modules/extract/dnd/spells
|
||||||
go test ./internal/modules/input/seriatim
|
go test ./internal/modules/extract/dnd/spells
|
||||||
go test ./...
|
go test ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
## Stage 2: Parse, Validate, And Map To SourceDocument
|
## Stage 2: Structured Response Schema Asset
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Implement the Seriatim parser and mapper from minimal Seriatim JSON into the
|
Add the structured response schema used by the D&D spells extractor and register
|
||||||
generic source model.
|
it through the existing LLM schema registry.
|
||||||
|
|
||||||
### Files To Add Or Update
|
### Files To Add Or Update
|
||||||
|
|
||||||
- `internal/modules/input/seriatim/adapter.go`
|
- `internal/framework/llm/assets/schemas/dnd_spells.v1.json`
|
||||||
- `internal/modules/input/seriatim/model.go`
|
- `internal/framework/llm/schema_registry.go`
|
||||||
- `internal/modules/input/seriatim/adapter_test.go`
|
- `internal/framework/llm/schema_registry_test.go`
|
||||||
- `internal/modules/input/seriatim/testdata/valid_minimal.json`
|
- `internal/modules/extract/dnd/spells/schema_test.go`
|
||||||
- `internal/modules/input/seriatim/testdata/duplicate_segment_id.json`
|
|
||||||
|
|
||||||
### Required Validation
|
### Schema Decisions
|
||||||
|
|
||||||
Reject:
|
Register:
|
||||||
|
|
||||||
- nil or canceled context before parsing;
|
- response schema key: `dnd_spells`
|
||||||
- empty raw input;
|
- schema ID: `notarius.dnd.spells`
|
||||||
- malformed JSON;
|
- schema version: `v1`
|
||||||
- valid JSON with trailing non-whitespace data;
|
- response schema name: `notarius_dnd_spells_v1`
|
||||||
- missing, null, or non-object top-level `metadata`;
|
- asset path: `assets/schemas/dnd_spells.v1.json`
|
||||||
- missing, null, empty, or non-array top-level `segments`;
|
|
||||||
- segment IDs that are empty after trimming;
|
|
||||||
- segment IDs with leading or trailing whitespace;
|
|
||||||
- duplicate segment IDs;
|
|
||||||
- missing or empty `speaker`;
|
|
||||||
- missing, empty, non-numeric, negative, or non-finite `start`;
|
|
||||||
- missing, empty, non-numeric, negative, or non-finite `end`;
|
|
||||||
- segments where `end < start`;
|
|
||||||
- missing or empty `text`.
|
|
||||||
|
|
||||||
The parser may preserve leading and trailing whitespace in segment text as long
|
The JSON schema must require a top-level object:
|
||||||
as the text is not empty after trimming.
|
|
||||||
|
|
||||||
### Mapping Rules
|
```json
|
||||||
|
{
|
||||||
|
"spell_casts": [
|
||||||
|
{
|
||||||
|
"caster": "Aria",
|
||||||
|
"spell": "Cure Wounds",
|
||||||
|
"effect": "Heals an injured ally.",
|
||||||
|
"narrative_description": "Aria casts Cure Wounds after the fight.",
|
||||||
|
"source_refs": [
|
||||||
|
{
|
||||||
|
"source_id": "session-alpha",
|
||||||
|
"start_unit_id": "seg-001",
|
||||||
|
"end_unit_id": "seg-002"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
- `segment.id` becomes `SourceUnit.ID`.
|
Required schema constraints:
|
||||||
- `segment.text` becomes `SourceUnit.Text`.
|
|
||||||
- `speaker`, `start`, and `end` become unit metadata under the exact keys
|
- `additionalProperties: false` at every object level.
|
||||||
defined in stage 1.
|
- top-level `spell_casts` is required and must be an array.
|
||||||
- The document metadata is a shallow copy of top-level Seriatim metadata.
|
- `spell_casts` may be empty.
|
||||||
- The document digest is based on raw input bytes, not normalized JSON.
|
- each spell cast requires non-empty `caster`, `spell`, `effect`, and
|
||||||
- Call `source.ValidateDocument` before returning the document and wrap any
|
`narrative_description`.
|
||||||
validation failure with Seriatim context.
|
- each spell cast requires `source_refs` with `minItems: 1`.
|
||||||
|
- each source ref requires non-empty `source_id`, `start_unit_id`, and
|
||||||
|
`end_unit_id`.
|
||||||
|
|
||||||
### Required Tests
|
### Required Tests
|
||||||
|
|
||||||
Add tests for:
|
- `llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)` succeeds.
|
||||||
|
- Registered schema list remains sorted and now includes the D&D spells schema.
|
||||||
- valid minimal transcript parses to a source document with expected ID, kind,
|
- The schema content is valid JSON and mutation-safe through registry lookups.
|
||||||
format, digest, units, and metadata;
|
- The spells package can look up the schema key it will use during extraction.
|
||||||
- `ParseRequest.SourceID` overrides metadata-derived IDs;
|
- The schema diagnostics map omits raw schema content.
|
||||||
- fallback document ID is deterministic and has prefix `seriatim:`;
|
|
||||||
- malformed JSON returns an actionable Seriatim parse error;
|
|
||||||
- missing metadata is rejected;
|
|
||||||
- missing or empty segments is rejected;
|
|
||||||
- duplicate segment IDs are rejected;
|
|
||||||
- empty segment text is rejected;
|
|
||||||
- missing speaker is rejected;
|
|
||||||
- invalid timestamp values are rejected;
|
|
||||||
- `end < start` is rejected;
|
|
||||||
- typed metadata helpers return the expected speaker and timestamp values;
|
|
||||||
- a `source.SourceRef` using the first and last generated unit IDs validates
|
|
||||||
with `source.ValidateRef`.
|
|
||||||
|
|
||||||
### Validation
|
### Validation
|
||||||
|
|
||||||
Run:
|
Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -w internal/modules/input/seriatim
|
gofmt -w internal/framework/llm internal/modules/extract/dnd/spells
|
||||||
go test ./internal/modules/input/seriatim
|
go test ./internal/framework/llm
|
||||||
|
go test ./internal/modules/extract/dnd/spells
|
||||||
go test ./...
|
go test ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
## Stage 3: Pipeline Resolution And Config Compatibility
|
## Stage 3: Prompt Assets And Prompt Rendering
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Prove the Seriatim input module participates in pipeline-profile resolution and
|
Add embedded prompt assets for D&D spell extraction and render deterministic
|
||||||
capability validation through existing registries and config loading.
|
system/user messages from generic source chunks.
|
||||||
|
|
||||||
### Files To Add Or Update
|
### Files To Add Or Update
|
||||||
|
|
||||||
- `internal/modules/input/seriatim/config_test.go`
|
- `internal/framework/prompt/assets/dnd/spells/system.md`
|
||||||
- `internal/modules/input/seriatim/testdata/pipeline.yml`
|
- `internal/framework/prompt/assets/dnd/spells/user.md`
|
||||||
|
- `internal/framework/prompt/registry.go`
|
||||||
|
- `internal/framework/prompt/registry_test.go`
|
||||||
|
- `internal/framework/prompt/render_test.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/prompt.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/prompt_test.go`
|
||||||
|
|
||||||
|
### Prompt Decisions
|
||||||
|
|
||||||
|
Register:
|
||||||
|
|
||||||
|
- prompt ID: `dnd.spells`
|
||||||
|
- prompt constant: `DNDSpellsPromptID`
|
||||||
|
- prompt version: `v1`
|
||||||
|
- embedded directory: `assets/dnd/spells`
|
||||||
|
|
||||||
|
The system prompt must:
|
||||||
|
|
||||||
|
- include the shared prompt hardening text via `{{ hardening }}`;
|
||||||
|
- identify the task as extracting D&D spell casts only;
|
||||||
|
- state that the model must not infer spells not supported by the provided
|
||||||
|
source text;
|
||||||
|
- state that source references must use source-unit IDs exactly as provided.
|
||||||
|
|
||||||
|
The user prompt must render:
|
||||||
|
|
||||||
|
- source document ID;
|
||||||
|
- chunk ID and chunk index when a chunk is present;
|
||||||
|
- source units in their existing order;
|
||||||
|
- each source unit's ID and text;
|
||||||
|
- selected metadata only as optional context, using generic labels. Include
|
||||||
|
`speaker`, `start`, and `end` when present without importing Seriatim helper
|
||||||
|
APIs.
|
||||||
|
|
||||||
|
Add a module-local prompt data builder that accepts
|
||||||
|
`contracts.ExtractionRequest` and returns template data. It must not mutate the
|
||||||
|
request, source document, chunk, units, or metadata maps.
|
||||||
|
|
||||||
|
### Required Tests
|
||||||
|
|
||||||
|
- Prompt metadata lookup succeeds for `dnd.spells`.
|
||||||
|
- Registered prompt metadata remains sorted.
|
||||||
|
- Rendering includes hardening text.
|
||||||
|
- Rendering includes source ID, chunk ID, unit IDs, unit text, and optional
|
||||||
|
speaker/timestamp metadata when present.
|
||||||
|
- Rendering fails clearly if required template data is missing.
|
||||||
|
- Module-local prompt data construction works with a generic source chunk and
|
||||||
|
does not depend on concrete Seriatim package helpers.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w internal/framework/prompt internal/modules/extract/dnd/spells
|
||||||
|
go test ./internal/framework/prompt
|
||||||
|
go test ./internal/modules/extract/dnd/spells
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage 4: Extractor LLM Call And Candidate Mapping
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Implement `Extractor.Extract` so a source chunk can flow through prompt
|
||||||
|
rendering, structured LLM completion, and candidate creation.
|
||||||
|
|
||||||
|
### Files To Add Or Update
|
||||||
|
|
||||||
|
- `internal/modules/extract/dnd/spells/extractor.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/model.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/extractor_test.go`
|
||||||
|
|
||||||
|
### Required Behavior
|
||||||
|
|
||||||
|
`Extract()` must:
|
||||||
|
|
||||||
|
- reject nil extractor, nil context, canceled context, nil source, nil chunk,
|
||||||
|
empty chunk units, and nil LLM client with clear `dnd spells` error context;
|
||||||
|
- render the `dnd.spells` prompt using the stage 3 prompt data builder;
|
||||||
|
- look up the stage 2 response schema;
|
||||||
|
- call `req.LLMClient.CompleteStructured` with:
|
||||||
|
- `StageName: Key`;
|
||||||
|
- two messages, system then user;
|
||||||
|
- `ResponseSchemaName: schema.Name`;
|
||||||
|
- `ResponseSchema: schema.JSONSchema`;
|
||||||
|
- decode into `extractionResponse`;
|
||||||
|
- reject a nil `SpellCasts` slice as malformed structured output;
|
||||||
|
- return no candidates for an empty `SpellCasts` slice;
|
||||||
|
- preserve response order when creating candidates;
|
||||||
|
- trim `caster`, `spell`, `effect`, and `narrative_description` before
|
||||||
|
marshaling the `SpellCast` payload;
|
||||||
|
- copy response source refs into `ArtifactCandidate.SourceRefs`;
|
||||||
|
- leave `ExtractorKey`, `ArtifactType`, `SchemaVersion`, and `Index` empty or
|
||||||
|
zero so the runner's existing candidate normalization remains authoritative;
|
||||||
|
- wrap LLM, prompt, schema lookup, response validation, and JSON marshal errors
|
||||||
|
with useful `dnd spells` context.
|
||||||
|
|
||||||
|
Do not call validators from `Extract()`. Validation belongs to the runner's
|
||||||
|
validator phase.
|
||||||
|
|
||||||
|
### Required Tests
|
||||||
|
|
||||||
|
Use a fake `contracts.StructuredLLMClient`.
|
||||||
|
|
||||||
|
- Successful extraction returns one candidate with expected payload fields,
|
||||||
|
source refs, response schema name, schema JSON, prompt messages, and stage
|
||||||
|
name.
|
||||||
|
- Empty `spell_casts: []` returns no candidates and no error.
|
||||||
|
- Missing or null `spell_casts` is rejected as malformed structured output.
|
||||||
|
- LLM client errors are wrapped with D&D spells context.
|
||||||
|
- Nil source, nil chunk, empty chunk units, nil LLM client, nil context, and
|
||||||
|
canceled context are rejected.
|
||||||
|
- Multiple response spell casts produce candidates in response order.
|
||||||
|
- Candidate source refs are copied so later mutation of the fake response does
|
||||||
|
not mutate returned candidates.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w internal/modules/extract/dnd/spells
|
||||||
|
go test ./internal/modules/extract/dnd/spells
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage 5: Deterministic Validator Chain
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Add module-owned deterministic validators for spell payload shape, required
|
||||||
|
fields, and source-reference grounding.
|
||||||
|
|
||||||
|
### Files To Add Or Update
|
||||||
|
|
||||||
|
- `internal/modules/extract/dnd/spells/validator.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/validator_test.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/extractor.go`
|
||||||
|
|
||||||
|
### Validator Decisions
|
||||||
|
|
||||||
|
Add two validators:
|
||||||
|
|
||||||
|
1. `ShapeValidator`
|
||||||
|
- `Name()` returns `dnd/spells/shape`.
|
||||||
|
- Rejects malformed JSON payloads with reason code `invalid_payload`.
|
||||||
|
- Rejects blank `caster`, `spell`, `effect`, or `narrative_description`
|
||||||
|
with reason code `missing_required_field`.
|
||||||
|
- Approves candidates with valid payload shape and required fields.
|
||||||
|
|
||||||
|
2. `SourceRefValidator`
|
||||||
|
- `Name()` returns `dnd/spells/source_refs`.
|
||||||
|
- Rejects candidates with no source refs using reason code
|
||||||
|
`missing_source_ref`.
|
||||||
|
- Rejects any invalid source ref using reason code `invalid_source_ref` and
|
||||||
|
the `source.ValidateRef` error message.
|
||||||
|
- Approves candidates whose source refs all validate against the request
|
||||||
|
source document.
|
||||||
|
|
||||||
|
`Extractor.Validators()` must return `ShapeValidator{}` followed by
|
||||||
|
`SourceRefValidator{}`. Return a fresh slice each time.
|
||||||
|
|
||||||
|
Both validators must:
|
||||||
|
|
||||||
|
- satisfy `contracts.Validator`;
|
||||||
|
- return one decision for every candidate;
|
||||||
|
- preserve candidate indexes in decisions;
|
||||||
|
- return an error, not rejection decisions, when called with nil source only if
|
||||||
|
source-reference validation cannot run. Shape validation does not require a
|
||||||
|
source document.
|
||||||
|
|
||||||
|
### Required Tests
|
||||||
|
|
||||||
|
- `Extractor.Validators()` returns the two validators in the required order and
|
||||||
|
is mutation-safe.
|
||||||
|
- Each validator approves a valid candidate.
|
||||||
|
- Shape validator rejects malformed payload JSON.
|
||||||
|
- Shape validator rejects each blank required field.
|
||||||
|
- Source ref validator rejects missing refs.
|
||||||
|
- Source ref validator rejects unknown source IDs, unknown unit IDs, and
|
||||||
|
reversed unit ranges.
|
||||||
|
- Both validators return one decision per candidate and preserve indexes.
|
||||||
|
- Approved decisions use `validate.Approved`; rejected decisions use the reason
|
||||||
|
codes listed above.
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w internal/modules/extract/dnd/spells
|
||||||
|
go test ./internal/modules/extract/dnd/spells
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage 6: Pipeline Resolution And Runner Integration
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Prove Seriatim input can flow through the existing runner into the real D&D
|
||||||
|
spells extractor, fake downstream infrastructure, and the extractor-owned
|
||||||
|
validators.
|
||||||
|
|
||||||
|
### Files To Add Or Update
|
||||||
|
|
||||||
|
- `internal/modules/extract/dnd/spells/config_test.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/runner_test.go`
|
||||||
|
- `internal/modules/extract/dnd/spells/testdata/seriatim_spell_session.json`
|
||||||
|
- `internal/modules/extract/dnd/spells/testdata/pipeline.yml`
|
||||||
|
|
||||||
### Required Test Catalog
|
### Required Test Catalog
|
||||||
|
|
||||||
Build a test-only module catalog with:
|
Build test-only catalogs and registries with:
|
||||||
|
|
||||||
- Seriatim input registered through `seriatim.Register`;
|
- Seriatim input registered through `seriatim.Register`;
|
||||||
- a fake chunker requiring `source.transcript` and providing `chunks`;
|
- a fake chunker registered as `fake/chunk`, requiring `source.transcript` and
|
||||||
- a fake extractor requiring `chunks`, `transcript.speaker`, and
|
providing `chunks`;
|
||||||
`transcript.timestamps`, and providing `fake.artifacts`;
|
- D&D spells extractor registered through `spells.Register`;
|
||||||
- `pipeline.AppendOrderMerger` registered as `appendorder`, requiring
|
- `pipeline.AppendOrderMerger` registered as `appendorder`, requiring
|
||||||
`fake.artifacts`;
|
`dnd.spell_casts`;
|
||||||
- `pipeline.NoopNormalizer` registered as `noop`;
|
- `pipeline.NoopNormalizer` registered as `noop`;
|
||||||
- a fake `json` output encoder registered as output stage.
|
- a fake `json` output encoder registered as output stage.
|
||||||
|
|
||||||
Do not add real extract, chunk, normalize, or output modules for this checkpoint.
|
Do not add real chunk or output modules for this checkpoint.
|
||||||
|
|
||||||
### YAML Fixture
|
### YAML Fixture
|
||||||
|
|
||||||
Use a synthetic pipeline fixture shaped like:
|
Use a synthetic pipeline fixture:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
version: 1
|
version: 1
|
||||||
pipelines:
|
pipelines:
|
||||||
seriatim-fixture:
|
dnd-spells-fixture:
|
||||||
input: seriatim
|
input: seriatim
|
||||||
chunk: fake/chunk
|
chunk: fake/chunk
|
||||||
artifacts:
|
artifacts:
|
||||||
events:
|
spells:
|
||||||
extract: fake/extract
|
extract: dnd/spells
|
||||||
merge: appendorder
|
merge: appendorder
|
||||||
normalize: noop
|
normalize: noop
|
||||||
output: json
|
output: json
|
||||||
```
|
```
|
||||||
|
|
||||||
The default LLM profile supplied by `config.Default()` is sufficient. Do not
|
|
||||||
add real provider settings to this fixture.
|
|
||||||
|
|
||||||
### Required Tests
|
### Required Tests
|
||||||
|
|
||||||
- `config.ParseFileConfigYAML` and `Config.ApplyFileConfig` load the fixture.
|
- `config.ParseFileConfigYAML` and `Config.ApplyFileConfig` load the fixture.
|
||||||
- `Config.Resolve` succeeds with pipeline ID `seriatim-fixture` and the
|
- `Config.Resolve` succeeds with pipeline ID `dnd-spells-fixture` and the
|
||||||
test-only catalog.
|
test-only catalog.
|
||||||
- The resolved pipeline input module is `seriatim`.
|
- The resolved artifact lane ID is `spells` and extractor module is
|
||||||
|
`dnd/spells`.
|
||||||
- The resolved pipeline digest is non-empty and stable across repeated
|
- The resolved pipeline digest is non-empty and stable across repeated
|
||||||
resolution.
|
resolution.
|
||||||
- Removing `transcript.timestamps` from the Seriatim module spec in the
|
- Removing `source.transcript` from the Seriatim module spec causes resolution
|
||||||
test-only catalog causes resolution to fail with a missing capability error.
|
to fail with a missing capability error for `dnd/spells`.
|
||||||
|
- Removing `dnd.spell_casts` from the extractor spec causes resolution to fail
|
||||||
|
with a missing capability error for `appendorder`.
|
||||||
- Selecting an unknown `--only` lane still fails through existing resolution
|
- Selecting an unknown `--only` lane still fails through existing resolution
|
||||||
behavior.
|
behavior.
|
||||||
|
- Runner success path:
|
||||||
|
- parse the Seriatim fixture;
|
||||||
|
- use a fake LLM that returns at least two valid spell casts;
|
||||||
|
- assert approved artifacts preserve response order;
|
||||||
|
- assert approved payloads contain spell data;
|
||||||
|
- assert approved artifact source refs validate with `source.ValidateRef`;
|
||||||
|
- assert the manifest records input module `seriatim`, extractor
|
||||||
|
`dnd/spells`, lane `spells`, and validation status `approved`.
|
||||||
|
- Runner rejection path:
|
||||||
|
- fake LLM returns a spell cast with an invalid source ref;
|
||||||
|
- runner completes with rejected artifacts and validation status `rejected`;
|
||||||
|
- rejection contains validator `dnd/spells/source_refs` and reason code
|
||||||
|
`invalid_source_ref`.
|
||||||
|
- Runner malformed-output path:
|
||||||
|
- fake LLM returns or reports malformed structured output;
|
||||||
|
- runner returns an extraction error with D&D spells context and failed
|
||||||
|
validation status.
|
||||||
|
|
||||||
### Validation
|
### Validation
|
||||||
|
|
||||||
Run:
|
Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -w internal/modules/input/seriatim
|
gofmt -w internal/modules/extract/dnd/spells
|
||||||
go test ./internal/modules/input/seriatim
|
go test ./internal/modules/extract/dnd/spells
|
||||||
go test ./internal/core/config
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Stage 4: Runner Integration With Fake Downstream Stages
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Prove real Seriatim input can flow through the existing runner into fake
|
|
||||||
downstream stages while preserving source-unit IDs and metadata.
|
|
||||||
|
|
||||||
### Files To Add Or Update
|
|
||||||
|
|
||||||
- `internal/modules/input/seriatim/runner_test.go`
|
|
||||||
|
|
||||||
### Required Behavior
|
|
||||||
|
|
||||||
Use the same Seriatim fixture from stage 2 and a resolved pipeline from stage 3.
|
|
||||||
Register fake downstream stages only inside the test.
|
|
||||||
|
|
||||||
The fake extractor should:
|
|
||||||
|
|
||||||
- inspect the received `SourceDocument` and `SourceChunk`;
|
|
||||||
- assert that unit IDs match Seriatim segment IDs;
|
|
||||||
- assert that speaker and timestamp metadata are present;
|
|
||||||
- return one generic artifact candidate with a source reference pointing at
|
|
||||||
existing Seriatim-derived unit IDs.
|
|
||||||
|
|
||||||
The test should then assert:
|
|
||||||
|
|
||||||
- `Runner.Run` succeeds;
|
|
||||||
- the manifest records input module `seriatim`;
|
|
||||||
- the manifest source digest equals the parsed document digest;
|
|
||||||
- approved artifacts preserve valid source references;
|
|
||||||
- no transcript-specific type has been added outside the module.
|
|
||||||
|
|
||||||
### Required Tests
|
|
||||||
|
|
||||||
- successful runner execution from Seriatim JSON through fake chunk, extract,
|
|
||||||
merge, normalize, and output stages;
|
|
||||||
- runner failure when the Seriatim adapter returns an invalid source document,
|
|
||||||
using a malformed fixture or test input;
|
|
||||||
- validation of the fake candidate's source reference with
|
|
||||||
`source.ValidateRef`.
|
|
||||||
|
|
||||||
### Validation
|
|
||||||
|
|
||||||
Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
gofmt -w internal/modules/input/seriatim
|
|
||||||
go test ./internal/modules/input/seriatim
|
go test ./internal/modules/input/seriatim
|
||||||
go test ./internal/framework/pipeline
|
go test ./internal/framework/pipeline
|
||||||
go test ./...
|
go test ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
## Stage 5: Documentation And Final Verification
|
## Stage 7: Documentation And Final Verification
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Document the implemented Seriatim integration contract once the module exists,
|
Document the implemented D&D spells extraction contract without describing
|
||||||
without describing unimplemented D&D extraction or run-command behavior.
|
unimplemented extractors or a `notarius run` command.
|
||||||
|
|
||||||
### Files To Add Or Update
|
### Files To Add Or Update
|
||||||
|
|
||||||
- `docs/integrations/seriatim.md`
|
- `docs/integrations/dnd-spells.md`
|
||||||
- `docs/roadmap/5-seriatim-input-module.md`
|
- `docs/roadmap/6-dnd-spells-extractor.md`
|
||||||
|
|
||||||
### Required Documentation
|
### Required Documentation
|
||||||
|
|
||||||
Create `docs/integrations/seriatim.md` as implemented-behavior documentation
|
Create `docs/integrations/dnd-spells.md` as implemented-behavior documentation
|
||||||
with:
|
with:
|
||||||
|
|
||||||
- accepted minimal JSON shape;
|
- module key, artifact type, schema version, prompt ID, and response schema key;
|
||||||
- required fields and validation rules;
|
- accepted source expectations: generic source document/chunk with transcript
|
||||||
- mapping from Seriatim fields to `SourceDocument` and `SourceUnit`;
|
capability supplied by pipeline resolution;
|
||||||
- metadata key conventions for `speaker`, `start`, and `end`;
|
- spell payload fields;
|
||||||
- capability strings declared by the module;
|
- source-reference behavior: LLM response includes refs, durable artifact
|
||||||
- note that broader Seriatim schema variants are not yet supported.
|
output carries refs in the generic artifact envelope;
|
||||||
|
- default deterministic validators and rejection reason codes;
|
||||||
|
- declared required/provided capabilities;
|
||||||
|
- note that item, NPC, combat, encounter, broad D&D rules, and CLI `run`
|
||||||
|
workflows are not implemented by this checkpoint.
|
||||||
|
|
||||||
Update `docs/roadmap/5-seriatim-input-module.md` only if implementation
|
Update `docs/roadmap/6-dnd-spells-extractor.md` only if implementation reveals
|
||||||
reveals a real scope or policy correction. Keep future D&D extraction behavior
|
a real scope or policy correction. Keep implementation staging in this file,
|
||||||
out of the integration doc.
|
not in the feature roadmap.
|
||||||
|
|
||||||
### Final Validation
|
### Final Validation
|
||||||
|
|
||||||
Run:
|
Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -w internal/modules/input/seriatim
|
gofmt -w internal/modules/extract/dnd/spells internal/framework/llm internal/framework/prompt
|
||||||
go test ./...
|
go test ./...
|
||||||
go build ./cmd/notarius
|
go build ./cmd/notarius
|
||||||
rm -f ./notarius
|
rm -f ./notarius
|
||||||
@@ -427,17 +586,19 @@ rm -f ./notarius
|
|||||||
|
|
||||||
- `go test ./...` passes.
|
- `go test ./...` passes.
|
||||||
- `go build ./cmd/notarius` passes.
|
- `go build ./cmd/notarius` passes.
|
||||||
- Seriatim minimal transcript JSON maps into `SourceDocument`.
|
- The D&D spells extractor is registered through the extractor registry.
|
||||||
- Unit IDs are stable and validate in source references.
|
- Pipeline-profile resolution can select a `spells` artifact lane using
|
||||||
- Transcript fields do not appear in core runner contracts.
|
extractor module `dnd/spells`.
|
||||||
- The input module is selectable through the input registry and pipeline-profile
|
- Seriatim minimal transcript JSON can flow through the runner into the D&D
|
||||||
resolution.
|
spells extractor in tests.
|
||||||
- The input module declares transcript-oriented flat capabilities for pipeline
|
- Spell artifacts include valid generic source references in the artifact
|
||||||
validation.
|
envelope.
|
||||||
- Tests prove transcript-specific assumptions are isolated to
|
- Invalid source references are rejected by the extractor-owned validator chain.
|
||||||
`internal/modules/input/seriatim`.
|
- D&D concepts do not appear in core runner, source, pipeline, config, prompt,
|
||||||
|
LLM, or validator contracts.
|
||||||
|
- No CLI `run` behavior is documented or implemented in this checkpoint.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None. This plan chooses the checkpoint-5 behavior needed to implement the
|
None. This plan chooses the checkpoint-6 behavior needed to implement the
|
||||||
feature without requiring additional product decisions.
|
feature without requiring additional product decisions.
|
||||||
|
|||||||
209
internal/modules/input/seriatim/adapter.go
Normal file
209
internal/modules/input/seriatim/adapter.go
Normal file
@@ -0,0 +1,209 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"math"
|
||||||
|
"math/big"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "seriatim"
|
||||||
|
|
||||||
|
const (
|
||||||
|
DocumentKind = "transcript"
|
||||||
|
UnitKind = "transcript_segment"
|
||||||
|
Format = "application/vnd.seriatim.minimal+json"
|
||||||
|
)
|
||||||
|
|
||||||
|
var providedCapabilities = []string{
|
||||||
|
"source.transcript",
|
||||||
|
"transcript.speaker",
|
||||||
|
"transcript.timestamps",
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ contracts.InputAdapter = (*Adapter)(nil)
|
||||||
|
|
||||||
|
type Adapter struct{}
|
||||||
|
|
||||||
|
func New() *Adapter {
|
||||||
|
return &Adapter{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Adapter) Key() string {
|
||||||
|
return Key
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Adapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||||
|
if ctx == nil {
|
||||||
|
return nil, inputErrorf("context must not be nil")
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, inputErrorf("context error before parsing: %w", err)
|
||||||
|
}
|
||||||
|
if len(req.Raw) == 0 {
|
||||||
|
return nil, inputErrorf("raw input must not be empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
parsed, err := decodeTranscript(req.Raw)
|
||||||
|
if err != nil {
|
||||||
|
return nil, inputErrorf("parse JSON: %w", err)
|
||||||
|
}
|
||||||
|
if len(parsed.Segments) == 0 {
|
||||||
|
return nil, inputErrorf("segments must not be empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
rawDigest := digest(req.Raw)
|
||||||
|
doc := &source.SourceDocument{
|
||||||
|
ID: documentID(req.SourceID, parsed.Metadata, rawDigest),
|
||||||
|
Kind: DocumentKind,
|
||||||
|
Format: Format,
|
||||||
|
Digest: rawDigest,
|
||||||
|
Metadata: copyMetadata(parsed.Metadata),
|
||||||
|
}
|
||||||
|
|
||||||
|
seenSegmentIDs := make(map[string]struct{}, len(parsed.Segments))
|
||||||
|
for i, segment := range parsed.Segments {
|
||||||
|
unit, err := sourceUnit(segment, i, seenSegmentIDs)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
doc.Units = append(doc.Units, unit)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := source.ValidateDocument(doc); err != nil {
|
||||||
|
return nil, inputErrorf("validate source document: %w", err)
|
||||||
|
}
|
||||||
|
return doc, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
|
return pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageInput,
|
||||||
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.InputAdapterRegistry) error {
|
||||||
|
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.InputAdapter, error) {
|
||||||
|
return New(), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func sourceUnit(segment segment, index int, seen map[string]struct{}) (source.SourceUnit, error) {
|
||||||
|
segmentLabel := fmt.Sprintf("segment[%d]", index)
|
||||||
|
segmentID := strings.TrimSpace(segment.ID)
|
||||||
|
if segmentID == "" {
|
||||||
|
return source.SourceUnit{}, inputErrorf("%s id must not be empty", segmentLabel)
|
||||||
|
}
|
||||||
|
if segmentID != segment.ID {
|
||||||
|
return source.SourceUnit{}, inputErrorf("%s id %q must not contain leading or trailing whitespace", segmentLabel, segment.ID)
|
||||||
|
}
|
||||||
|
if _, ok := seen[segment.ID]; ok {
|
||||||
|
return source.SourceUnit{}, inputErrorf("segment id %q is duplicated", segment.ID)
|
||||||
|
}
|
||||||
|
seen[segment.ID] = struct{}{}
|
||||||
|
|
||||||
|
speaker := strings.TrimSpace(segment.Speaker)
|
||||||
|
if speaker == "" {
|
||||||
|
return source.SourceUnit{}, inputErrorf("segment %q speaker must not be empty", segment.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
start, err := validTimestamp(segment.Start, fmt.Sprintf("segment %q start", segment.ID))
|
||||||
|
if err != nil {
|
||||||
|
return source.SourceUnit{}, err
|
||||||
|
}
|
||||||
|
end, err := validTimestamp(segment.End, fmt.Sprintf("segment %q end", segment.ID))
|
||||||
|
if err != nil {
|
||||||
|
return source.SourceUnit{}, err
|
||||||
|
}
|
||||||
|
if end.Cmp(start) < 0 {
|
||||||
|
return source.SourceUnit{}, inputErrorf("segment %q end must be greater than or equal to start", segment.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.TrimSpace(segment.Text) == "" {
|
||||||
|
return source.SourceUnit{}, inputErrorf("segment %q text must not be empty", segment.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
return source.SourceUnit{
|
||||||
|
ID: segment.ID,
|
||||||
|
Kind: UnitKind,
|
||||||
|
Text: segment.Text,
|
||||||
|
Metadata: map[string]any{
|
||||||
|
MetadataSpeaker: segment.Speaker,
|
||||||
|
MetadataStart: segment.Start,
|
||||||
|
MetadataEnd: segment.End,
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validTimestamp(value fmt.Stringer, label string) (*big.Rat, error) {
|
||||||
|
raw := strings.TrimSpace(value.String())
|
||||||
|
if raw == "" {
|
||||||
|
return nil, inputErrorf("%s must not be empty", label)
|
||||||
|
}
|
||||||
|
parsed, err := strconv.ParseFloat(raw, 64)
|
||||||
|
if err != nil {
|
||||||
|
return nil, inputErrorf("%s must be a valid number: %w", label, err)
|
||||||
|
}
|
||||||
|
if math.IsInf(parsed, 0) || math.IsNaN(parsed) {
|
||||||
|
return nil, inputErrorf("%s must be finite", label)
|
||||||
|
}
|
||||||
|
if parsed < 0 {
|
||||||
|
return nil, inputErrorf("%s must not be negative", label)
|
||||||
|
}
|
||||||
|
rat, ok := new(big.Rat).SetString(raw)
|
||||||
|
if !ok {
|
||||||
|
return nil, inputErrorf("%s must be a valid number", label)
|
||||||
|
}
|
||||||
|
return rat, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func documentID(requestedID string, metadata map[string]any, rawDigest string) string {
|
||||||
|
if id := strings.TrimSpace(requestedID); id != "" {
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
if id := stringMetadata(metadata, "id"); id != "" {
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
if id := stringMetadata(metadata, "source_id"); id != "" {
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
return "seriatim:" + strings.TrimPrefix(rawDigest, "sha256:")[:16]
|
||||||
|
}
|
||||||
|
|
||||||
|
func stringMetadata(metadata map[string]any, key string) string {
|
||||||
|
value, ok := metadata[key].(string)
|
||||||
|
if !ok {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyMetadata(metadata map[string]any) map[string]any {
|
||||||
|
if len(metadata) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
copied := make(map[string]any, len(metadata))
|
||||||
|
for key, value := range metadata {
|
||||||
|
copied[key] = value
|
||||||
|
}
|
||||||
|
return copied
|
||||||
|
}
|
||||||
|
|
||||||
|
func digest(raw []byte) string {
|
||||||
|
sum := sha256.Sum256(raw)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func inputErrorf(format string, args ...any) error {
|
||||||
|
return fmt.Errorf("seriatim input: "+format, args...)
|
||||||
|
}
|
||||||
290
internal/modules/input/seriatim/adapter_test.go
Normal file
290
internal/modules/input/seriatim/adapter_test.go
Normal file
@@ -0,0 +1,290 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseValidMinimalTranscript(t *testing.T) {
|
||||||
|
raw := readFixture(t, "testdata/valid_minimal.json")
|
||||||
|
|
||||||
|
doc, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if doc.ID != "session-alpha" {
|
||||||
|
t.Fatalf("doc.ID = %q, want session-alpha", doc.ID)
|
||||||
|
}
|
||||||
|
if doc.Kind != DocumentKind {
|
||||||
|
t.Fatalf("doc.Kind = %q, want %q", doc.Kind, DocumentKind)
|
||||||
|
}
|
||||||
|
if doc.Format != Format {
|
||||||
|
t.Fatalf("doc.Format = %q, want %q", doc.Format, Format)
|
||||||
|
}
|
||||||
|
if doc.Digest != testDigest(raw) {
|
||||||
|
t.Fatalf("doc.Digest = %q, want %q", doc.Digest, testDigest(raw))
|
||||||
|
}
|
||||||
|
if got := doc.Metadata["title"]; got != "Synthetic session transcript" {
|
||||||
|
t.Fatalf("doc.Metadata[title] = %#v, want Synthetic session transcript", got)
|
||||||
|
}
|
||||||
|
if len(doc.Units) != 2 {
|
||||||
|
t.Fatalf("len(doc.Units) = %d, want 2", len(doc.Units))
|
||||||
|
}
|
||||||
|
|
||||||
|
first := doc.Units[0]
|
||||||
|
if first.ID != "seg-001" {
|
||||||
|
t.Fatalf("first.ID = %q, want seg-001", first.ID)
|
||||||
|
}
|
||||||
|
if first.Kind != UnitKind {
|
||||||
|
t.Fatalf("first.Kind = %q, want %q", first.Kind, UnitKind)
|
||||||
|
}
|
||||||
|
if first.Text != "The stone door opens." {
|
||||||
|
t.Fatalf("first.Text = %q, want fixture text", first.Text)
|
||||||
|
}
|
||||||
|
if speaker, ok := Speaker(first); !ok || speaker != "Narrator" {
|
||||||
|
t.Fatalf("Speaker(first) = %q, %v; want Narrator, true", speaker, ok)
|
||||||
|
}
|
||||||
|
if start, ok := Start(first); !ok || start != json.Number("0") {
|
||||||
|
t.Fatalf("Start(first) = %q, %v; want 0, true", start, ok)
|
||||||
|
}
|
||||||
|
if end, ok := End(first); !ok || end != json.Number("4.5") {
|
||||||
|
t.Fatalf("End(first) = %q, %v; want 4.5, true", end, ok)
|
||||||
|
}
|
||||||
|
|
||||||
|
ref := source.SourceRef{
|
||||||
|
SourceID: doc.ID,
|
||||||
|
StartUnitID: doc.Units[0].ID,
|
||||||
|
EndUnitID: doc.Units[len(doc.Units)-1].ID,
|
||||||
|
}
|
||||||
|
if err := source.ValidateRef(doc, ref); err != nil {
|
||||||
|
t.Fatalf("ValidateRef() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRequestSourceIDOverridesMetadataIDs(t *testing.T) {
|
||||||
|
raw := readFixture(t, "testdata/valid_minimal.json")
|
||||||
|
|
||||||
|
doc, err := New().Parse(context.Background(), contracts.ParseRequest{
|
||||||
|
SourceID: " requested-source ",
|
||||||
|
Raw: raw,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if doc.ID != "requested-source" {
|
||||||
|
t.Fatalf("doc.ID = %q, want requested-source", doc.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseFallbackDocumentIDIsDeterministic(t *testing.T) {
|
||||||
|
raw := []byte(`{"metadata":{},"segments":[{"id":"s1","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."}]}`)
|
||||||
|
|
||||||
|
first, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
second, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if first.ID != second.ID {
|
||||||
|
t.Fatalf("fallback IDs differ: %q vs %q", first.ID, second.ID)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(first.ID, "seriatim:") {
|
||||||
|
t.Fatalf("fallback ID = %q, want seriatim prefix", first.ID)
|
||||||
|
}
|
||||||
|
if first.ID != "seriatim:"+strings.TrimPrefix(testDigest(raw), "sha256:")[:16] {
|
||||||
|
t.Fatalf("fallback ID = %q, want digest-derived ID", first.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseUsesMetadataSourceIDWhenMetadataIDIsAbsent(t *testing.T) {
|
||||||
|
raw := []byte(`{"metadata":{"source_id":" source-from-metadata "},"segments":[{"id":"s1","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."}]}`)
|
||||||
|
|
||||||
|
doc, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if doc.ID != "source-from-metadata" {
|
||||||
|
t.Fatalf("doc.ID = %q, want source-from-metadata", doc.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRejectsInvalidInput(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
raw []byte
|
||||||
|
wantErr []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "malformed JSON",
|
||||||
|
raw: []byte(`{"metadata":`),
|
||||||
|
wantErr: []string{"seriatim input", "parse JSON"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "trailing JSON",
|
||||||
|
raw: []byte(`{"metadata":{},"segments":[]} {}`),
|
||||||
|
wantErr: []string{"seriatim input", "trailing"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing metadata",
|
||||||
|
raw: []byte(`{"segments":[]}`),
|
||||||
|
wantErr: []string{"metadata"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "null metadata",
|
||||||
|
raw: []byte(`{"metadata":null,"segments":[]}`),
|
||||||
|
wantErr: []string{"metadata", "object"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "metadata wrong type",
|
||||||
|
raw: []byte(`{"metadata":[],"segments":[]}`),
|
||||||
|
wantErr: []string{"metadata", "object"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing segments",
|
||||||
|
raw: []byte(`{"metadata":{}}`),
|
||||||
|
wantErr: []string{"segments"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "null segments",
|
||||||
|
raw: []byte(`{"metadata":{},"segments":null}`),
|
||||||
|
wantErr: []string{"segments", "array"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "segments wrong type",
|
||||||
|
raw: []byte(`{"metadata":{},"segments":{}}`),
|
||||||
|
wantErr: []string{"segments", "array"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty segments",
|
||||||
|
raw: []byte(`{"metadata":{},"segments":[]}`),
|
||||||
|
wantErr: []string{"segments", "empty"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing segment id",
|
||||||
|
raw: validJSONWithSegment(`"start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"id", "empty"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "whitespace segment id",
|
||||||
|
raw: validJSONWithSegment(`"id":" s1 ","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"id", "whitespace"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty text",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":0,"end":1,"speaker":"Narrator","text":" "`),
|
||||||
|
wantErr: []string{"text", "empty"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing speaker",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":0,"end":1,"text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"speaker", "empty"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing start",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","end":1,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"start", "empty"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing end",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":0,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"end", "empty"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "negative start",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":-1,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"start", "negative"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "non-numeric end",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":0,"end":"late","speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"segment[0]", "end", "number"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "non-finite timestamp",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":1e10000,"end":1e10000,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"start", "valid number"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "end before start",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":2,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"end", "start"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "end before start beyond float precision",
|
||||||
|
raw: validJSONWithSegment(`"id":"s1","start":9007199254740993,"end":9007199254740992,"speaker":"Narrator","text":"Synthetic text."`),
|
||||||
|
wantErr: []string{"end", "start"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
_, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: tt.raw})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Parse() error = nil, want error")
|
||||||
|
}
|
||||||
|
for _, want := range tt.wantErr {
|
||||||
|
if !strings.Contains(err.Error(), want) {
|
||||||
|
t.Fatalf("Parse() error = %q, want substring %q", err.Error(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRejectsDuplicateSegmentIDs(t *testing.T) {
|
||||||
|
raw := readFixture(t, "testdata/duplicate_segment_id.json")
|
||||||
|
|
||||||
|
_, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Parse() error = nil, want duplicate ID error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "duplicated") || !strings.Contains(err.Error(), "seg-001") {
|
||||||
|
t.Fatalf("Parse() error = %q, want duplicate segment context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRejectsInvalidContextOrEmptyInput(t *testing.T) {
|
||||||
|
if _, err := New().Parse(nil, contracts.ParseRequest{Raw: []byte(`{}`)}); err == nil {
|
||||||
|
t.Fatal("Parse(nil context) error = nil, want error")
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
if _, err := New().Parse(ctx, contracts.ParseRequest{Raw: []byte(`{}`)}); err == nil {
|
||||||
|
t.Fatal("Parse(canceled context) error = nil, want error")
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := New().Parse(context.Background(), contracts.ParseRequest{}); err == nil {
|
||||||
|
t.Fatal("Parse(empty input) error = nil, want error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func readFixture(t *testing.T, path string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(%q) error = %v, want nil", path, err)
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
|
||||||
|
func validJSONWithSegment(segmentFields string) []byte {
|
||||||
|
return []byte(`{"metadata":{"id":"fixture"},"segments":[{` + segmentFields + `}]}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testDigest(raw []byte) string {
|
||||||
|
sum := sha256.Sum256(raw)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
250
internal/modules/input/seriatim/config_test.go
Normal file
250
internal/modules/input/seriatim/config_test.go
Normal file
@@ -0,0 +1,250 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPipelineConfigLoadsAndResolvesWithSeriatimInput(t *testing.T) {
|
||||||
|
cfg := loadPipelineConfig(t)
|
||||||
|
|
||||||
|
resolved, err := cfg.Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "seriatim-fixture",
|
||||||
|
Catalog: seriatimTestCatalog(t, ModuleSpec()),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if resolved.ResolvedPipeline.Input.Module != Key {
|
||||||
|
t.Fatalf("resolved input module = %q, want %q", resolved.ResolvedPipeline.Input.Module, Key)
|
||||||
|
}
|
||||||
|
if resolved.ResolvedPipeline.Digest == "" {
|
||||||
|
t.Fatal("resolved digest is empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
again, err := cfg.Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "seriatim-fixture",
|
||||||
|
Catalog: seriatimTestCatalog(t, ModuleSpec()),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if resolved.ResolvedPipeline.Digest != again.ResolvedPipeline.Digest {
|
||||||
|
t.Fatalf("resolved digest = %q, second digest = %q; want stable digest", resolved.ResolvedPipeline.Digest, again.ResolvedPipeline.Digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPipelineConfigRejectsMissingSeriatimCapability(t *testing.T) {
|
||||||
|
spec := ModuleSpec()
|
||||||
|
spec.Provides = withoutCapability(spec.Provides, "transcript.timestamps")
|
||||||
|
cfg := loadPipelineConfig(t)
|
||||||
|
|
||||||
|
_, err := cfg.Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "seriatim-fixture",
|
||||||
|
Catalog: seriatimTestCatalog(t, spec),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Resolve() error = nil, want missing capability error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "missing capability") || !strings.Contains(err.Error(), "transcript.timestamps") {
|
||||||
|
t.Fatalf("Resolve() error = %q, want missing transcript.timestamps capability", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPipelineConfigRejectsUnknownLaneSelection(t *testing.T) {
|
||||||
|
cfg := loadPipelineConfig(t)
|
||||||
|
|
||||||
|
_, err := cfg.Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "seriatim-fixture",
|
||||||
|
Only: []string{"missing"},
|
||||||
|
Catalog: seriatimTestCatalog(t, ModuleSpec()),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Resolve() error = nil, want unknown lane error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "selected artifact lane") || !strings.Contains(err.Error(), "missing") {
|
||||||
|
t.Fatalf("Resolve() error = %q, want unknown lane context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadPipelineConfig(t *testing.T) config.Config {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
data, err := os.ReadFile("testdata/pipeline.yml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(pipeline.yml) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
fileCfg, err := config.ParseFileConfigYAML(data)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
if err := cfg.ApplyFileConfig(fileCfg); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
profile, ok := cfg.Pipelines["seriatim-fixture"]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("pipeline seriatim-fixture was not loaded")
|
||||||
|
}
|
||||||
|
if profile.Input.Module != Key {
|
||||||
|
t.Fatalf("loaded input module = %q, want %q", profile.Input.Module, Key)
|
||||||
|
}
|
||||||
|
return cfg
|
||||||
|
}
|
||||||
|
|
||||||
|
func seriatimTestCatalog(t *testing.T, inputSpec pipeline.ModuleSpec) pipeline.ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
inputs := pipeline.NewInputAdapterRegistry()
|
||||||
|
chunkers := pipeline.NewChunkerRegistry()
|
||||||
|
extractors := pipeline.NewExtractorRegistry()
|
||||||
|
mergers := pipeline.NewMergerRegistry()
|
||||||
|
normalizers := pipeline.NewNormalizerRegistry()
|
||||||
|
outputs := pipeline.NewOutputEncoderRegistry()
|
||||||
|
|
||||||
|
if reflect.DeepEqual(inputSpec, ModuleSpec()) {
|
||||||
|
if err := Register(inputs); err != nil {
|
||||||
|
t.Fatalf("register seriatim input: %v", err)
|
||||||
|
}
|
||||||
|
} else if err := inputs.RegisterWithSpec(inputSpec, func() (contracts.InputAdapter, error) {
|
||||||
|
return New(), nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register seriatim input override: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
mustRegisterChunker(t, chunkers, pipeline.ModuleSpec{
|
||||||
|
Key: "fake/chunk",
|
||||||
|
Stage: pipeline.StageChunk,
|
||||||
|
Requires: []string{"source.transcript"},
|
||||||
|
Provides: []string{"chunks"},
|
||||||
|
})
|
||||||
|
mustRegisterExtractor(t, extractors, pipeline.ModuleSpec{
|
||||||
|
Key: "fake/extract",
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
Requires: []string{"chunks", "transcript.speaker", "transcript.timestamps"},
|
||||||
|
Provides: []string{"fake.artifacts"},
|
||||||
|
})
|
||||||
|
mustRegisterMerger(t, mergers, pipeline.ModuleSpec{
|
||||||
|
Key: pipeline.DefaultMergeModule,
|
||||||
|
Stage: pipeline.StageMerge,
|
||||||
|
Requires: []string{"fake.artifacts"},
|
||||||
|
})
|
||||||
|
mustRegisterNormalizer(t, normalizers, pipeline.ModuleSpec{
|
||||||
|
Key: pipeline.DefaultNormalizeModule,
|
||||||
|
Stage: pipeline.StageNormalize,
|
||||||
|
})
|
||||||
|
mustRegisterOutput(t, outputs, pipeline.ModuleSpec{
|
||||||
|
Key: pipeline.DefaultOutputModule,
|
||||||
|
Stage: pipeline.StageOutput,
|
||||||
|
})
|
||||||
|
|
||||||
|
return pipeline.ModuleCatalog{
|
||||||
|
Inputs: inputs,
|
||||||
|
Chunkers: chunkers,
|
||||||
|
Extractors: extractors,
|
||||||
|
Mergers: mergers,
|
||||||
|
Normalizers: normalizers,
|
||||||
|
Outputs: outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec pipeline.ModuleSpec) {
|
||||||
|
t.Helper()
|
||||||
|
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) {
|
||||||
|
return fakeChunker{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunker: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
||||||
|
t.Helper()
|
||||||
|
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) {
|
||||||
|
return fakeExtractor{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register extractor: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
||||||
|
t.Helper()
|
||||||
|
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) {
|
||||||
|
return pipeline.AppendOrderMerger{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register merger: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
||||||
|
t.Helper()
|
||||||
|
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) {
|
||||||
|
return pipeline.NoopNormalizer{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register normalizer: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
||||||
|
t.Helper()
|
||||||
|
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) {
|
||||||
|
return fakeOutput{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register output: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeChunker struct{}
|
||||||
|
|
||||||
|
func (fakeChunker) Key() string { return "fake/chunk" }
|
||||||
|
|
||||||
|
func (fakeChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||||
|
return contracts.ChunkResult{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeExtractor struct{}
|
||||||
|
|
||||||
|
func (fakeExtractor) Key() string { return "fake/extract" }
|
||||||
|
|
||||||
|
func (fakeExtractor) ArtifactType() string { return "fake" }
|
||||||
|
|
||||||
|
func (fakeExtractor) SchemaVersion() string { return "v1" }
|
||||||
|
|
||||||
|
func (fakeExtractor) Validators() []contracts.Validator { return nil }
|
||||||
|
|
||||||
|
func (fakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||||
|
return contracts.ExtractionResult{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeOutput struct{}
|
||||||
|
|
||||||
|
func (fakeOutput) Key() string { return pipeline.DefaultOutputModule }
|
||||||
|
|
||||||
|
func (fakeOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
|
return contracts.OutputResult{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func withoutCapability(capabilities []string, capability string) []string {
|
||||||
|
filtered := make([]string, 0, len(capabilities))
|
||||||
|
for _, candidate := range capabilities {
|
||||||
|
if candidate != capability {
|
||||||
|
filtered = append(filtered, candidate)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return filtered
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ contracts.Chunker = fakeChunker{}
|
||||||
|
_ contracts.Extractor = fakeExtractor{}
|
||||||
|
_ contracts.OutputEncoder = fakeOutput{}
|
||||||
|
)
|
||||||
28
internal/modules/input/seriatim/metadata.go
Normal file
28
internal/modules/input/seriatim/metadata.go
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
MetadataSpeaker = "speaker"
|
||||||
|
MetadataStart = "start"
|
||||||
|
MetadataEnd = "end"
|
||||||
|
)
|
||||||
|
|
||||||
|
func Speaker(unit source.SourceUnit) (string, bool) {
|
||||||
|
value, ok := unit.Metadata[MetadataSpeaker].(string)
|
||||||
|
return value, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func Start(unit source.SourceUnit) (json.Number, bool) {
|
||||||
|
value, ok := unit.Metadata[MetadataStart].(json.Number)
|
||||||
|
return value, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func End(unit source.SourceUnit) (json.Number, bool) {
|
||||||
|
value, ok := unit.Metadata[MetadataEnd].(json.Number)
|
||||||
|
return value, ok
|
||||||
|
}
|
||||||
127
internal/modules/input/seriatim/model.go
Normal file
127
internal/modules/input/seriatim/model.go
Normal file
@@ -0,0 +1,127 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
)
|
||||||
|
|
||||||
|
type transcript struct {
|
||||||
|
Metadata map[string]any `json:"metadata"`
|
||||||
|
Segments []segment `json:"segments"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type segment struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Start json.Number `json:"start"`
|
||||||
|
End json.Number `json:"end"`
|
||||||
|
Speaker string `json:"speaker"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeTranscript(raw []byte) (transcript, error) {
|
||||||
|
var fields map[string]json.RawMessage
|
||||||
|
if err := decodeJSON(raw, &fields); err != nil {
|
||||||
|
return transcript{}, err
|
||||||
|
}
|
||||||
|
if fields == nil {
|
||||||
|
return transcript{}, fmt.Errorf("top-level value must be an object")
|
||||||
|
}
|
||||||
|
|
||||||
|
metadataRaw, ok := fields["metadata"]
|
||||||
|
if !ok {
|
||||||
|
return transcript{}, fmt.Errorf("metadata is required")
|
||||||
|
}
|
||||||
|
var metadata map[string]any
|
||||||
|
if err := decodeJSON(metadataRaw, &metadata); err != nil {
|
||||||
|
return transcript{}, fmt.Errorf("metadata must be an object: %w", err)
|
||||||
|
}
|
||||||
|
if metadata == nil {
|
||||||
|
return transcript{}, fmt.Errorf("metadata must be an object")
|
||||||
|
}
|
||||||
|
|
||||||
|
segmentsRaw, ok := fields["segments"]
|
||||||
|
if !ok {
|
||||||
|
return transcript{}, fmt.Errorf("segments are required")
|
||||||
|
}
|
||||||
|
var segmentValues []json.RawMessage
|
||||||
|
if err := decodeJSON(segmentsRaw, &segmentValues); err != nil {
|
||||||
|
return transcript{}, fmt.Errorf("segments must be an array: %w", err)
|
||||||
|
}
|
||||||
|
if segmentValues == nil {
|
||||||
|
return transcript{}, fmt.Errorf("segments must be an array")
|
||||||
|
}
|
||||||
|
segments := make([]segment, 0, len(segmentValues))
|
||||||
|
for i, rawSegment := range segmentValues {
|
||||||
|
segment, err := decodeSegment(rawSegment, i)
|
||||||
|
if err != nil {
|
||||||
|
return transcript{}, err
|
||||||
|
}
|
||||||
|
segments = append(segments, segment)
|
||||||
|
}
|
||||||
|
|
||||||
|
return transcript{
|
||||||
|
Metadata: metadata,
|
||||||
|
Segments: segments,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeSegment(raw []byte, index int) (segment, error) {
|
||||||
|
var fields map[string]json.RawMessage
|
||||||
|
if err := decodeJSON(raw, &fields); err != nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] must be an object: %w", index, err)
|
||||||
|
}
|
||||||
|
if fields == nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] must be an object", index)
|
||||||
|
}
|
||||||
|
|
||||||
|
var decoded segment
|
||||||
|
if err := decodeOptionalString(fields, "id", &decoded.ID); err != nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] id must be a string: %w", index, err)
|
||||||
|
}
|
||||||
|
if err := decodeOptionalNumber(fields, "start", &decoded.Start); err != nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] start must be a number: %w", index, err)
|
||||||
|
}
|
||||||
|
if err := decodeOptionalNumber(fields, "end", &decoded.End); err != nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] end must be a number: %w", index, err)
|
||||||
|
}
|
||||||
|
if err := decodeOptionalString(fields, "speaker", &decoded.Speaker); err != nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] speaker must be a string: %w", index, err)
|
||||||
|
}
|
||||||
|
if err := decodeOptionalString(fields, "text", &decoded.Text); err != nil {
|
||||||
|
return segment{}, fmt.Errorf("segment[%d] text must be a string: %w", index, err)
|
||||||
|
}
|
||||||
|
return decoded, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeOptionalString(fields map[string]json.RawMessage, key string, out *string) error {
|
||||||
|
raw, ok := fields[key]
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return decodeJSON(raw, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeOptionalNumber(fields map[string]json.RawMessage, key string, out *json.Number) error {
|
||||||
|
raw, ok := fields[key]
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return decodeJSON(raw, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeJSON(raw []byte, out any) error {
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(raw))
|
||||||
|
decoder.UseNumber()
|
||||||
|
if err := decoder.Decode(out); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := decoder.Decode(&struct{}{}); err != io.EOF {
|
||||||
|
if err == nil {
|
||||||
|
return fmt.Errorf("unexpected trailing JSON value")
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
106
internal/modules/input/seriatim/registry_test.go
Normal file
106
internal/modules/input/seriatim/registry_test.go
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNewReturnsAdapterWithKey(t *testing.T) {
|
||||||
|
adapter := New()
|
||||||
|
if adapter == nil {
|
||||||
|
t.Fatal("New() = nil, want adapter")
|
||||||
|
}
|
||||||
|
if adapter.Key() != Key {
|
||||||
|
t.Fatalf("adapter.Key() = %q, want %q", adapter.Key(), Key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestModuleSpec(t *testing.T) {
|
||||||
|
got := ModuleSpec()
|
||||||
|
want := pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageInput,
|
||||||
|
Provides: []string{
|
||||||
|
"source.transcript",
|
||||||
|
"transcript.speaker",
|
||||||
|
"transcript.timestamps",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
got.Provides[0] = "changed"
|
||||||
|
again := ModuleSpec()
|
||||||
|
if !reflect.DeepEqual(again, want) {
|
||||||
|
t.Fatalf("ModuleSpec() after caller mutation = %#v, want %#v", again, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterMakesAdapterBuildable(t *testing.T) {
|
||||||
|
registry := pipeline.NewInputAdapterRegistry()
|
||||||
|
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
adapter, err := registry.Build(Key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Build() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if adapter.Key() != Key {
|
||||||
|
t.Fatalf("adapter.Key() = %q, want %q", adapter.Key(), Key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterStoresModuleSpec(t *testing.T) {
|
||||||
|
registry := pipeline.NewInputAdapterRegistry()
|
||||||
|
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, ok := registry.Spec(Key)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("Spec() ok = false, want true")
|
||||||
|
}
|
||||||
|
want := ModuleSpec()
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterNilRegistryReturnsError(t *testing.T) {
|
||||||
|
err := Register(nil)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Register(nil) error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "input adapter registry") {
|
||||||
|
t.Fatalf("Register(nil) error = %q, want registry context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMetadataHelpers(t *testing.T) {
|
||||||
|
unit := source.SourceUnit{
|
||||||
|
Metadata: map[string]any{
|
||||||
|
MetadataSpeaker: "Narrator",
|
||||||
|
MetadataStart: json.Number("1.25"),
|
||||||
|
MetadataEnd: json.Number("2.5"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
if got, ok := Speaker(unit); !ok || got != "Narrator" {
|
||||||
|
t.Fatalf("Speaker() = %q, %v; want Narrator, true", got, ok)
|
||||||
|
}
|
||||||
|
if got, ok := Start(unit); !ok || got != json.Number("1.25") {
|
||||||
|
t.Fatalf("Start() = %q, %v; want 1.25, true", got, ok)
|
||||||
|
}
|
||||||
|
if got, ok := End(unit); !ok || got != json.Number("2.5") {
|
||||||
|
t.Fatalf("End() = %q, %v; want 2.5, true", got, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
267
internal/modules/input/seriatim/runner_test.go
Normal file
267
internal/modules/input/seriatim/runner_test.go
Normal file
@@ -0,0 +1,267 @@
|
|||||||
|
package seriatim
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRunnerProcessesSeriatimInputWithFakeModules(t *testing.T) {
|
||||||
|
raw := readFixture(t, "testdata/valid_minimal.json")
|
||||||
|
expectedDoc, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
resolved, err := loadPipelineConfig(t).Resolve(configResolveInput(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
extractor := &runnerSeriatimExtractor{}
|
||||||
|
output, err := pipeline.New(seriatimRunnerRegistries(t, extractor)).Run(context.Background(), pipeline.RunInput{
|
||||||
|
Pipeline: resolved.ResolvedPipeline,
|
||||||
|
RawInput: raw,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if output.Manifest.InputModule != Key {
|
||||||
|
t.Fatalf("manifest input module = %q, want %q", output.Manifest.InputModule, Key)
|
||||||
|
}
|
||||||
|
if got := output.Manifest.SourceDigests; len(got) != 1 || got[0] != expectedDoc.Digest {
|
||||||
|
t.Fatalf("manifest source digests = %#v, want %q", got, expectedDoc.Digest)
|
||||||
|
}
|
||||||
|
if len(output.Approved) != 1 {
|
||||||
|
t.Fatalf("len(Approved) = %d, want 1", len(output.Approved))
|
||||||
|
}
|
||||||
|
|
||||||
|
artifact := output.Approved[0]
|
||||||
|
if artifact.ExtractorKey != "fake/extract" || artifact.ArtifactType != "fake.event" || artifact.SchemaVersion != "v1" {
|
||||||
|
t.Fatalf("approved artifact envelope = %#v, want fake extractor envelope", artifact)
|
||||||
|
}
|
||||||
|
if len(artifact.SourceRefs) != 1 {
|
||||||
|
t.Fatalf("len(SourceRefs) = %d, want 1", len(artifact.SourceRefs))
|
||||||
|
}
|
||||||
|
if err := source.ValidateRef(expectedDoc, artifact.SourceRefs[0]); err != nil {
|
||||||
|
t.Fatalf("ValidateRef() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if artifact.SourceRefs[0].StartUnitID != "seg-001" || artifact.SourceRefs[0].EndUnitID != "seg-002" {
|
||||||
|
t.Fatalf("SourceRefs[0] = %#v, want Seriatim unit IDs", artifact.SourceRefs[0])
|
||||||
|
}
|
||||||
|
if extractor.calls != 1 {
|
||||||
|
t.Fatalf("extractor calls = %d, want 1", extractor.calls)
|
||||||
|
}
|
||||||
|
if output.ContentType != "application/json" {
|
||||||
|
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerFailsOnInvalidSeriatimInput(t *testing.T) {
|
||||||
|
resolved, err := loadPipelineConfig(t).Resolve(configResolveInput(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := pipeline.New(seriatimRunnerRegistries(t, &runnerSeriatimExtractor{})).Run(context.Background(), pipeline.RunInput{
|
||||||
|
Pipeline: resolved.ResolvedPipeline,
|
||||||
|
RawInput: []byte(`{"metadata":{},"segments":[]}`),
|
||||||
|
SourceID: "invalid-source",
|
||||||
|
LLMClient: nil,
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Run() error = nil, want invalid input error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "parse input with adapter") || !strings.Contains(err.Error(), "seriatim input") {
|
||||||
|
t.Fatalf("Run() error = %q, want Seriatim parse context", err.Error())
|
||||||
|
}
|
||||||
|
if output.Manifest.ValidationStatus != "failed" {
|
||||||
|
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func configResolveInput(t *testing.T) config.ResolveInput {
|
||||||
|
t.Helper()
|
||||||
|
return config.ResolveInput{
|
||||||
|
PipelineID: "seriatim-fixture",
|
||||||
|
Catalog: seriatimTestCatalog(t, ModuleSpec()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func seriatimRunnerRegistries(t *testing.T, extractor contracts.Extractor) pipeline.Registries {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
inputs := pipeline.NewInputAdapterRegistry()
|
||||||
|
chunkers := pipeline.NewChunkerRegistry()
|
||||||
|
extractors := pipeline.NewExtractorRegistry()
|
||||||
|
mergers := pipeline.NewMergerRegistry()
|
||||||
|
normalizers := pipeline.NewNormalizerRegistry()
|
||||||
|
outputs := pipeline.NewOutputEncoderRegistry()
|
||||||
|
|
||||||
|
if err := Register(inputs); err != nil {
|
||||||
|
t.Fatalf("register seriatim input: %v", err)
|
||||||
|
}
|
||||||
|
if err := chunkers.Register("fake/chunk", func() (contracts.Chunker, error) {
|
||||||
|
return runnerSeriatimChunker{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunker: %v", err)
|
||||||
|
}
|
||||||
|
if err := extractors.Register("fake/extract", func() (contracts.Extractor, error) {
|
||||||
|
return extractor, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register extractor: %v", err)
|
||||||
|
}
|
||||||
|
if err := mergers.Register(pipeline.DefaultMergeModule, func() (contracts.Merger, error) {
|
||||||
|
return pipeline.AppendOrderMerger{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register merger: %v", err)
|
||||||
|
}
|
||||||
|
if err := normalizers.Register(pipeline.DefaultNormalizeModule, func() (contracts.Normalizer, error) {
|
||||||
|
return pipeline.NoopNormalizer{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register normalizer: %v", err)
|
||||||
|
}
|
||||||
|
if err := outputs.Register(pipeline.DefaultOutputModule, func() (contracts.OutputEncoder, error) {
|
||||||
|
return runnerSeriatimOutput{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register output: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return pipeline.Registries{
|
||||||
|
Inputs: inputs,
|
||||||
|
Chunkers: chunkers,
|
||||||
|
Extractors: extractors,
|
||||||
|
Mergers: mergers,
|
||||||
|
Normalizers: normalizers,
|
||||||
|
Outputs: outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type runnerSeriatimChunker struct{}
|
||||||
|
|
||||||
|
func (runnerSeriatimChunker) Key() string {
|
||||||
|
return "fake/chunk"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (runnerSeriatimChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||||
|
return contracts.ChunkResult{
|
||||||
|
Chunks: []contracts.SourceChunk{
|
||||||
|
{
|
||||||
|
ID: req.Source.ID + ":chunk:0",
|
||||||
|
SourceID: req.Source.ID,
|
||||||
|
Index: 0,
|
||||||
|
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type runnerSeriatimExtractor struct {
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *runnerSeriatimExtractor) Key() string {
|
||||||
|
return "fake/extract"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *runnerSeriatimExtractor) ArtifactType() string {
|
||||||
|
return "fake.event"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *runnerSeriatimExtractor) SchemaVersion() string {
|
||||||
|
return "v1"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *runnerSeriatimExtractor) Validators() []contracts.Validator {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *runnerSeriatimExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||||
|
e.calls++
|
||||||
|
if req.Source == nil {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("source must not be nil")
|
||||||
|
}
|
||||||
|
if req.Chunk == nil {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("chunk must not be nil")
|
||||||
|
}
|
||||||
|
if got := unitIDs(req.Source.Units); !equalStrings(got, []string{"seg-001", "seg-002"}) {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("source unit IDs = %#v, want Seriatim segment IDs", got)
|
||||||
|
}
|
||||||
|
if got := unitIDs(req.Chunk.Units); !equalStrings(got, []string{"seg-001", "seg-002"}) {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("chunk unit IDs = %#v, want Seriatim segment IDs", got)
|
||||||
|
}
|
||||||
|
for _, unit := range req.Chunk.Units {
|
||||||
|
if speaker, ok := Speaker(unit); !ok || speaker == "" {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("unit %q missing speaker metadata", unit.ID)
|
||||||
|
}
|
||||||
|
if _, ok := Start(unit); !ok {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("unit %q missing start metadata", unit.ID)
|
||||||
|
}
|
||||||
|
if _, ok := End(unit); !ok {
|
||||||
|
return contracts.ExtractionResult{}, fmt.Errorf("unit %q missing end metadata", unit.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return contracts.ExtractionResult{
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{
|
||||||
|
{
|
||||||
|
Payload: json.RawMessage(`{"value":"seriatim-source-ref"}`),
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{
|
||||||
|
SourceID: req.Source.ID,
|
||||||
|
StartUnitID: req.Chunk.Units[0].ID,
|
||||||
|
EndUnitID: req.Chunk.Units[len(req.Chunk.Units)-1].ID,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type runnerSeriatimOutput struct{}
|
||||||
|
|
||||||
|
func (runnerSeriatimOutput) Key() string {
|
||||||
|
return pipeline.DefaultOutputModule
|
||||||
|
}
|
||||||
|
|
||||||
|
func (runnerSeriatimOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
|
return contracts.OutputResult{
|
||||||
|
Bytes: []byte(`{"encoded":true}`),
|
||||||
|
ContentType: "application/json",
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func unitIDs(units []source.SourceUnit) []string {
|
||||||
|
ids := make([]string, 0, len(units))
|
||||||
|
for _, unit := range units {
|
||||||
|
ids = append(ids, unit.ID)
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
func equalStrings(a, b []string) bool {
|
||||||
|
if len(a) != len(b) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for i := range a {
|
||||||
|
if a[i] != b[i] {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ contracts.Chunker = runnerSeriatimChunker{}
|
||||||
|
_ contracts.Extractor = (*runnerSeriatimExtractor)(nil)
|
||||||
|
_ contracts.OutputEncoder = runnerSeriatimOutput{}
|
||||||
|
)
|
||||||
21
internal/modules/input/seriatim/testdata/duplicate_segment_id.json
vendored
Normal file
21
internal/modules/input/seriatim/testdata/duplicate_segment_id.json
vendored
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"id": "duplicate-segment-fixture"
|
||||||
|
},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": "seg-001",
|
||||||
|
"start": 0,
|
||||||
|
"end": 1,
|
||||||
|
"speaker": "Narrator",
|
||||||
|
"text": "First segment."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "seg-001",
|
||||||
|
"start": 1,
|
||||||
|
"end": 2,
|
||||||
|
"speaker": "Player",
|
||||||
|
"text": "Duplicate segment."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
11
internal/modules/input/seriatim/testdata/pipeline.yml
vendored
Normal file
11
internal/modules/input/seriatim/testdata/pipeline.yml
vendored
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
version: 1
|
||||||
|
pipelines:
|
||||||
|
seriatim-fixture:
|
||||||
|
input: seriatim
|
||||||
|
chunk: fake/chunk
|
||||||
|
artifacts:
|
||||||
|
events:
|
||||||
|
extract: fake/extract
|
||||||
|
merge: appendorder
|
||||||
|
normalize: noop
|
||||||
|
output: json
|
||||||
23
internal/modules/input/seriatim/testdata/valid_minimal.json
vendored
Normal file
23
internal/modules/input/seriatim/testdata/valid_minimal.json
vendored
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"id": "session-alpha",
|
||||||
|
"source_id": "fallback-session",
|
||||||
|
"title": "Synthetic session transcript"
|
||||||
|
},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": "seg-001",
|
||||||
|
"start": 0,
|
||||||
|
"end": 4.5,
|
||||||
|
"speaker": "Narrator",
|
||||||
|
"text": "The stone door opens."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "seg-002",
|
||||||
|
"start": 4.5,
|
||||||
|
"end": 8,
|
||||||
|
"speaker": "Player",
|
||||||
|
"text": "I cast light."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user