eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revise the architecture plan to reflect an input -> chunk -> process -> merge -> normalize -> output workflow

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-03 08:54:23 -05:00
parent 32be4ee85e
commit 88042174b3
10 changed files with 426 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files

120
docs/policy/architecture.md
View File

@@ -25,17 +25,27 @@ contracts that can be exercised by tests and real modules.
The core framework must remain source-agnostic and domain-agnostic.
Source-format details belong in input adapters. Transcript-specific concepts
Source-format details belong in input modules. Transcript-specific concepts
such as segments, speakers, timestamps, and transcript schemas must not spread
into runner, extractor, or validator framework code.
Extraction-domain details belong in extractor packages. D&D-specific concepts
Extraction-domain details belong in process modules. D&D-specific concepts
such as spells, NPCs, items, combat turns, and encounters must not spread into
core source, runner, or LLM framework packages.
Extracted facts should be grounded with source references. Source references
should point to generic source units, not to transcript-only structures.
The application workflow is:
```text
input -> chunk -> process -> merge -> normalize -> output
```
These stages should remain explicit in the architecture. Chunking, merging, and
normalization must not be hidden inside domain process modules when they represent
general pipeline behavior.
## Dependency Policy
Prefer the Go standard library where practical.
@@ -69,16 +79,15 @@ Core deterministic model and policy:
- `internal/core/inputcatalog`: known input adapter keys and metadata.
- `internal/core/extractorcatalog`: known extractor keys and metadata.
External source and provider adapters:
- `internal/adapters/input/<name>`: source-format adapters that parse external input into core source documents.
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
Reusable framework plumbing:
- `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
- `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output.
- `internal/framework/pipeline`: shared pipeline-stage orchestration types, when needed.
- `internal/framework/extraction`: shared extraction helper code.
- `internal/framework/merge`: shared merge-stage behavior.
- `internal/framework/normalize`: shared normalization-stage behavior.
- `internal/framework/output`: output encoding contracts and shared helpers.
- `internal/framework/validators`: shared validator runtime behavior and decision checks.
- `internal/framework/llm`: LLM runtime, scheduling, and provider adapters.
- `internal/framework/responseschema`: embedded structured-output schema registry.
@@ -88,35 +97,62 @@ Reusable framework plumbing:
Domain implementations:
- `internal/extractors/<domain>/<extractor>`: domain-specific extractor packages.
- `internal/modules/input/<name>`: input-stage modules that parse external input into core source documents.
- `internal/modules/chunk/<name>`: chunk-stage modules.
- `internal/modules/process/<domain>/<name>`: process-stage extractor modules.
- `internal/modules/merge/<name>` or `internal/modules/merge/<domain>/<name>`: merge-stage modules.
- `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
- `internal/modules/output/<name>`: output-stage modules.
- `internal/validators/<validator>`: built-in validator implementations.
- `internal/prompts`: embedded prompt assets and prompt metadata registry.
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
Package-private implementation constants may live near the package that owns
them, preferably in `constants.go` when useful.
## Input Adapters
## Stage Modules
Concrete business logic should live under `internal/modules/<stage>/...`.
Stage-oriented module layout is preferred because it makes the application
workflow visible in the filesystem:
```text
internal/modules/input/...
internal/modules/chunk/...
internal/modules/process/...
internal/modules/merge/...
internal/modules/normalize/...
internal/modules/output/...
```
Use short, lowercase, idiomatic Go package names. Prefer names such as
`dndtranscript`, `appendorder`, and `spells` over names like `dnd_transcript`,
`serial_merge`, or `spell_extractor` that repeat parent-stage context.
## Input Modules
Use a hexagonal architecture style for source input.
Input adapters translate external source formats into the core source model.
Adapters may know about external schema details, source-specific metadata, and
Input modules translate external source formats into the core source model.
They may know about external schema details, source-specific metadata, and
format-specific validation rules. They should not own extraction-domain
decisions.
Other packages should interact with source input through adapter contracts and
core source types. Adapter implementation details and external dependency types
must not leak into framework or extractor packages.
core source types. Input module implementation details and external dependency
types must not leak into framework or process module packages.
Adapter metadata may preserve source-specific facts such as transcript speaker,
Input module metadata may preserve source-specific facts such as transcript speaker,
timestamps, Markdown heading path, page number, or block ID. Framework code may
carry metadata through, but should not require a specific adapter's metadata
shape.
## Extractors
Extractors are independent modules that produce one kind of structured artifact.
Each extractor package owns:
Extractors are independent modules that process source chunks or whole source
documents and produce one kind of structured artifact candidate.
Each process module owns:
- its artifact semantics;
- its prompt usage;
@@ -124,8 +160,13 @@ Each extractor package owns:
- its validator chain;
- any domain-specific mapping or interpretation.
Extractors should depend on framework contracts and core source/artifact types.
They should not depend on concrete input adapter packages.
Process modules should depend on framework contracts and core source/artifact
types. They should not depend on concrete input module packages.
Extractors should not be the only place where chunking, merging, or
normalization happens. They may choose processing mode or provide domain-specific
merge/normalization behavior when generic behavior is insufficient, but the
pipeline stages themselves are framework concepts.
The runner should be able to compose, skip, resume, or run individual extractors
when their prerequisites are satisfied. Ordering should be explicit through
@@ -134,6 +175,31 @@ configuration, a default sequence, or documented orchestration rules.
Extractor selection must go through a registry or equivalent mechanism rather
than scattered conditionals.
## Pipeline Stages
The pipeline has six conceptual stages:
1. input: external source material becomes a `SourceDocument`;
2. chunk: a `SourceDocument` becomes ordered source chunks;
3. process: extractors produce artifact candidates from chunks or whole documents;
4. merge: per-chunk candidates become a merged candidate collection;
5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
6. output: final artifacts are serialized.
Chunking is first-class because source documents may be too large for a single
LLM pass. Chunkers should preserve source-unit order and produce stable chunk
metadata.
Merge and normalize are separate concerns. Merge combines per-chunk results into
a deterministic collection. Normalize performs semantic reconciliation after
merge. Generic append-in-chunk-order merge and no-op normalization should be
available for simple artifact types, while domain-specific behavior can be
provided where needed.
The framework should allow serial and parallel chunk processing. The first
implementation may execute chunks serially for determinism, but contracts should
not prevent later parallel execution.
## Validators
Validators should be independently testable and composable.
@@ -181,8 +247,8 @@ Configuration files should not contain raw secrets unless the application is
explicitly designed for that. Prefer environment variables or secret files for
secrets.
Adapter-specific and extractor-specific configuration should remain grouped by
the adapter or extractor that owns it.
Stage-module-specific configuration should remain grouped by the module that
owns it.
## Embedded Assets
@@ -242,12 +308,12 @@ focused on implemented behavior. Put future, planned, or aspirational work only
under `docs/roadmap/`.
Core documentation should use generic terms such as source document, source
unit, source reference, input adapter, extractor, artifact, validator, and run
manifest.
unit, source reference, input adapter, extractor, chunker, merger, normalizer,
artifact, validator, and run manifest.
Source-format details belong in adapter or integration docs. Domain-specific
extraction details belong in extractor or artifact docs.
Source-format details belong in input module or integration docs.
Domain-specific extraction details belong in process module or artifact docs.
When changing architecture, config, CLI behavior, adapters, extractor contracts,
validator contracts, LLM runtime behavior, or artifact schemas, update the
relevant docs and examples in the same change.
When changing architecture, config, CLI behavior, stage modules, extractor
contracts, validator contracts, LLM runtime behavior, or artifact schemas, update
the relevant docs and examples in the same change.

6
docs/roadmap/1-core-contracts-and-skeleton.md
View File

@@ -25,8 +25,8 @@ In scope:
Out of scope:
- real input adapters;
- real extractors;
- real input modules;
- real process modules;
- LLM provider calls;
- prompt or response-schema assets;
- diagnostics run directory;
@@ -47,7 +47,7 @@ contract packages:
structured LLM interfaces used by later checkpoints.
The contracts should be proven with fake implementations in tests. Those tests
should demonstrate composition without real source adapters, real extractors,
should demonstrate composition without real input modules, real process modules,
LLM provider calls, prompt assets, or diagnostics infrastructure.
Implementation staging belongs in

12
docs/roadmap/2-framework-composition.md
View File

@@ -6,8 +6,8 @@ This document describes planned work, not implemented behavior.
## Goal
Prove the core contracts compose before adding real adapters, real extractors,
or portable Audita infrastructure.
Prove the core contracts compose before adding real stage modules or portable
Audita infrastructure.
This checkpoint should produce a minimal runner that can execute fake registered
components from source input to artifact output in tests.
@@ -26,6 +26,8 @@ In scope:
Out of scope:
- real input parsing;
- real input modules;
- real process modules;
- real LLM calls;
- prompt assets;
- response schema assets;
@@ -47,8 +49,8 @@ The repository should contain a minimal framework composition layer:
artifacts.
The runner should operate on already parsed source documents in this checkpoint.
Raw input parsing and concrete input adapter behavior remain deferred to the
Seriatim adapter checkpoint.
Raw input parsing and concrete input module behavior remain deferred to the
Seriatim input module checkpoint.
Implementation staging belongs in
[`implementation.md`](implementation.md).
@@ -58,7 +60,7 @@ Implementation staging belongs in
- `go test ./...` passes.
- Fake adapter/extractor/validator registrations work in tests.
- The runner operates on `SourceDocument`, not transcript-specific structures.
- The runner does not import concrete D&D extractor packages.
- The runner does not import concrete D&D process module packages.
## Review Questions

132
docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md Normal file
View File

@@ -0,0 +1,132 @@
# Checkpoint 3: Pipeline Stages, Chunking, Merge, And Normalize
## Status
This document describes planned work, not implemented behavior.
## Goal
Make Notarius's application workflow first-class before adding real input
modules or process modules.
The workflow should be:
```text
input -> chunk -> process -> merge -> normalize -> output
```
Checkpoint 3 should define the contracts and minimal fake-tested framework
behavior for chunking, per-chunk processing, merging, and normalization. It
should not add real input modules, real domain process modules, LLM provider code,
or output encoders.
## Scope
In scope:
- source chunk model;
- chunker contract;
- process-stage contract for extractors operating on chunks;
- merge-stage contract;
- normalize-stage contract;
- output-stage contract shape, if useful for pipeline completeness;
- runner/pipeline updates that exercise these stages with fake components;
- generic append/chronological merge behavior for artifact candidates when
appropriate.
Out of scope:
- Seriatim parsing;
- D&D spell extraction;
- LLM provider calls;
- prompt assets;
- response schema assets;
- diagnostics run directory;
- production output serialization.
## Target End State
The repository should contain explicit pipeline-stage contracts:
- `InputAdapter`: external source input to `SourceDocument`.
- `Chunker`: `SourceDocument` to ordered `SourceChunk` values.
- `Extractor` or processor: `SourceChunk` to artifact candidates.
- `Merger`: per-chunk candidates to merged candidates.
- `Normalizer`: merged candidates to normalized candidates.
- `OutputEncoder`: final artifact bundle to bytes, if introduced in this
checkpoint.
The runner should orchestrate fake implementations through chunk, process,
merge, normalize, and approval/validation behavior in tests.
## Design Intent
Chunking is a core application concern because many source documents, especially
transcripts, will be too large for a single LLM extraction pass.
Chunk processing may be serial or parallel depending on extractor needs. The
architecture should support both, but the first implementation can execute
deterministically in series until a later checkpoint introduces concurrency.
Merge and normalize are separate stages:
- merge combines per-chunk extracted candidates into one stream or collection;
- normalize reconciles the merged output by checking duplicates, consistency,
ordering, identity resolution, or other cross-chunk concerns.
For some artifact types, merge may be generic append-in-source-order behavior.
For other artifact types, merge may be domain-specific. Normalization is where
deduplication and consistency checks should live.
## Processing Modes
The architecture should leave room for extractor-level processing modes:
- whole-document processing;
- serial chunk processing;
- parallel chunk processing.
The first implementation may model these modes without implementing parallel
execution. It should not bake in a single-pass assumption.
## Generic Merge Behavior
A generic merger should be able to concatenate candidates in deterministic
chunk order and candidate order. This is likely sufficient for early spell-cast
extraction, where chronological serialization is useful.
Domain-specific mergers may be added later when generic ordering is not enough.
## Generic Normalize Behavior
A no-op normalizer should be available as the default.
Domain-specific normalizers may later:
- deduplicate repeated extracted facts;
- resolve aliases;
- reconcile conflicting candidate fields;
- enforce chronological or source-reference consistency;
- attach normalization warnings.
## Done Criteria
- `go test ./...` passes.
- Pipeline-stage contracts are explicit and source/domain agnostic.
- Fake tests prove input source documents can be chunked, processed, merged, and
normalized.
- Merge and normalize are distinct concepts in code and tests.
- The runner no longer implies whole-document-only extraction as the core
application model.
- No concrete input module, process module, LLM provider, prompt, response schema,
diagnostics, config, or D&D artifact code is added.
## Review Questions
- Is the workflow clearly represented as input, chunk, process, merge,
normalize, and output?
- Are merge and normalize cleanly separated?
- Can a generic merger handle simple chronological artifact streams?
- Can a later domain-specific normalizer handle duplicates and consistency
without changing core runner contracts?
- Does the design allow serial and parallel chunk processing later?

6
docs/roadmap/3-portable-audita-infrastructure.md → docs/roadmap/4-portable-audita-infrastructure.md
View File

@@ -1,4 +1,4 @@
# Checkpoint 3: Portable Audita Infrastructure
# Checkpoint 4: Portable Audita Infrastructure
## Status
@@ -72,7 +72,7 @@ The registry should track:
- prompt hash.
Do not add D&D prompt assets here unless the implementation naturally overlaps
with checkpoint 5. Test prompts are acceptable for registry tests.
with checkpoint 6. Test prompts are acceptable for registry tests.
### Stage 4: Diagnostics Run Directory
@@ -94,7 +94,7 @@ Add config structs and defaults only for infrastructure that now exists.
Initial config areas:
- input adapter key;
- input module key;
- extractor keys;
- primary LLM settings;
- validation LLM settings if needed;

40
docs/roadmap/4-seriatim-input-adapter.md → docs/roadmap/5-seriatim-input-module.md
View File

@@ -1,4 +1,4 @@
# Checkpoint 4: Seriatim Input Adapter
# Checkpoint 5: Seriatim Input Module
## Status
@@ -7,7 +7,7 @@ This document describes planned work, not implemented behavior.
## Goal
Add the first real input source while keeping transcript-specific behavior
isolated inside an input adapter.
isolated inside an input-stage module.
This checkpoint should allow Seriatim minimal transcript JSON to become a
generic `SourceDocument`.
@@ -16,13 +16,13 @@ generic `SourceDocument`.
In scope:
- `internal/adapters/input/seriatim`;
- `internal/modules/input/seriatim`;
- parser for Seriatim minimal output JSON;
- mapping into `SourceDocument` and `SourceUnit`;
- source-document validation;
- adapter registry wiring;
- input adapter registry wiring;
- fixtures and tests;
- CLI/config path to select the adapter if the CLI shell exists.
- CLI/config path to select the input module if the CLI shell exists.
Out of scope:
@@ -33,9 +33,9 @@ Out of scope:
## Proposed Stages
### Stage 1: Seriatim Source Model
### Seriatim Source Model
Define adapter-local structs for the Seriatim minimal output schema.
Define module-local structs for the Seriatim minimal output schema.
Expected external shape:
@@ -47,9 +47,9 @@ Expected external shape:
- segment `speaker`;
- segment `text`.
Keep these structs in the adapter package.
Keep these structs in the Seriatim input module package.
### Stage 2: Parse And Validate
### Parse And Validate
Implement parser and validation behavior.
@@ -62,9 +62,9 @@ Validation should cover:
- non-empty segment text;
- valid start/end values as appropriate.
Prefer clear adapter-specific errors.
Prefer clear module-specific errors.
### Stage 3: Map To SourceDocument
### Map To SourceDocument
Map Seriatim data into the generic source model:
@@ -77,9 +77,9 @@ Map Seriatim data into the generic source model:
The resulting `SourceDocument` should pass core source validation.
### Stage 4: Registry And CLI Wiring
### Registry And CLI Wiring
Register the adapter under a stable key, likely `seriatim`.
Register the module under a stable input adapter key, likely `seriatim`.
If CLI support exists, add provisional selection:
@@ -87,9 +87,9 @@ If CLI support exists, add provisional selection:
notarius extract ./transcript.json --input seriatim
```
The command may still use fake extractors until checkpoint 5.
The command may still use fake extractors until checkpoint 6.
### Stage 5: Fixtures And Tests
### Fixtures And Tests
Add fixtures and tests for:
@@ -105,12 +105,12 @@ Add fixtures and tests for:
- `go test ./...` passes.
- Seriatim minimal transcript JSON maps into `SourceDocument`.
- Transcript fields do not appear in core runner contracts.
- The adapter is selectable through the registry.
- Tests prove transcript-specific assumptions are isolated to the adapter.
- The input module is selectable through the registry.
- Tests prove transcript-specific assumptions are isolated to the input module.
## Review Questions
- Are segment, speaker, and timestamp assumptions contained inside the adapter?
- Are segment, speaker, and timestamp assumptions contained inside the input module?
- Are unit IDs stable and suitable for source references?
- Does the adapter preserve enough metadata for transcript-oriented output later?
- Should the adapter accept only Seriatim minimal output for now?
- Does the input module preserve enough metadata for transcript-oriented output later?
- Should the input module accept only Seriatim minimal output for now?

22
docs/roadmap/5-dnd-spells-extractor.md → docs/roadmap/6-dnd-spells-extractor.md
View File

@@ -1,4 +1,4 @@
# Checkpoint 5: D&D Spells Extractor
# Checkpoint 6: D&D Spells Extractor
## Status
@@ -6,8 +6,8 @@ This document describes planned work, not implemented behavior.
## Goal
Implement the first useful extraction module: D&D spell casts from a Seriatim
transcript source document.
Implement the first useful process-stage module: D&D spell casts from a
Seriatim transcript source document.
This checkpoint should produce the first meaningful vertical slice from real
source input to validated artifact output.
@@ -19,7 +19,7 @@ In scope:
- D&D spell artifact schema and Go structs;
- structured response schema asset;
- prompt assets;
- `internal/extractors/dnd/spells`;
- `internal/modules/process/dnd/spells`;
- source-reference and schema validators in the extractor chain;
- fake LLM tests;
- CLI-level integration test if the CLI path is ready.
@@ -50,8 +50,8 @@ type SpellCast struct {
}
```
Keep this schema inside the D&D spells extractor or a D&D artifact package, not
inside core framework packages.
Keep this schema inside the D&D spells process module or a D&D artifact package,
not inside core framework packages.
### Stage 2: Structured Response Schema
@@ -74,14 +74,14 @@ Prompts should:
- avoid relying on transcript-specific fields except as optional metadata;
- request only spell-cast artifacts.
### Stage 4: Extractor Implementation
### Stage 4: Process Module Implementation
Implement `internal/extractors/dnd/spells`.
Implement `internal/modules/process/dnd/spells`.
The extractor should:
- satisfy the framework `Extractor` contract;
- build LLM messages from a source document or source slice;
- build LLM messages from a source document or source chunk;
- call the structured LLM client;
- return artifact candidates with source references;
- attach its validator chain.
@@ -117,13 +117,13 @@ The test should use fake LLM wiring and fixture input.
- `go test ./...` passes.
- Seriatim input can flow through the runner into the D&D spells extractor.
- Spell artifacts include valid source references.
- D&D concepts are contained in extractor/artifact packages and docs.
- D&D concepts are contained in process module/artifact packages and docs.
- The first meaningful vertical slice is available through tests, and through
CLI if the CLI path is ready.
## Review Questions
- Is the spell extractor domain-specific without making the framework
- Is the spell process module domain-specific without making the framework
D&D-specific?
- Are source references valid and useful for downstream validation?
- Is prompt/schema ownership clear?

72
docs/roadmap/documentation.md
View File

@@ -10,8 +10,8 @@ not describe implemented behavior.
Notarius documentation should make three boundaries obvious:
- source-format support belongs to input adapters;
- extraction-domain behavior belongs to extractor packages;
- source-format support belongs to input-stage modules;
- extraction-domain behavior belongs to process-stage modules;
- core framework behavior is source-agnostic and domain-agnostic.
Documentation should avoid making the MVP look more transcript-specific or
@@ -35,6 +35,10 @@ Core architecture docs should prefer:
- source reference;
- input adapter;
- extractor;
- chunker;
- merger;
- normalizer;
- output encoder;
- artifact;
- validator;
- run manifest.
@@ -43,11 +47,12 @@ Core docs should avoid transcript-specific terms such as segment, speaker,
timestamp, and transcript range unless discussing an input adapter or an example.
Core docs should avoid D&D-specific terms such as spell, NPC, item, combat, and
encounter unless discussing extractor packages or examples.
encounter unless discussing process modules, artifact docs, or examples.
### Adapter Docs Own Source Formats
### Input Module Docs Own Source Formats
Each implemented input adapter should have a canonical integration document.
Each implemented input-stage module should have a canonical integration
document.
Likely future files:
@@ -56,32 +61,33 @@ docs/integrations/seriatim-transcript.md
docs/integrations/markdown-source.md
```
Adapter docs should cover:
Input module docs should cover:
- accepted external schema or file shape;
- mapping into `SourceDocument` and `SourceUnit`;
- metadata preserved by the adapter;
- metadata preserved by the module;
- validation rules and failure behavior;
- examples.
The Seriatim adapter doc should reference the Seriatim schema it supports and
The Seriatim input module doc should reference the Seriatim schema it supports and
explain how transcript segment IDs become source-unit IDs.
### Extractor Docs Own Domains
### Stage Module Docs Own Business Logic
Each implemented extractor family should have canonical internal or integration
docs.
Each implemented stage-module family should have canonical internal or
integration docs.
Likely future files:
```text
docs/internal/extractors.md
docs/internal/stage-modules.md
docs/integrations/artifacts-dnd.md
```
Extractor docs should cover:
Stage module docs should cover:
- extractor key;
- module key;
- stage;
- artifact type;
- schema version;
- required source-reference behavior;
@@ -89,13 +95,14 @@ Extractor docs should cover:
- prompt and response-schema ownership;
- examples.
D&D concepts should be documented in D&D extractor docs, not in generic runner
or framework docs.
D&D concepts should be documented in D&D process-module or artifact docs, not in
generic runner or framework docs.
### CLI Docs Should Reflect Extensibility
The CLI reference should present input adapters and extractors as selectable
components.
The CLI reference should present input modules, chunk modules, process modules,
merge modules, normalize modules, and output modules as selectable or
configurable components as they become user-facing.
Provisional command shape:
@@ -106,8 +113,9 @@ notarius extract ./source.json --input seriatim --extractors dnd.spells --output
Once implemented, `docs/cli.md` should document:
- positional source input path;
- input adapter selection;
- extractor selection;
- input module selection;
- process module selection;
- chunk/merge/normalize/output selection when configurable;
- config path behavior;
- output path behavior;
- diagnostics and report behavior;
@@ -117,15 +125,17 @@ Once implemented, `docs/cli.md` should document:
`docs/config.md` should group fields by responsibility:
- input adapter selection and adapter-specific options;
- extractor selection and extractor-specific options;
- input module selection and module-specific options;
- chunk module selection and module-specific options;
- process module selection and module-specific options;
- merge module selection and module-specific options;
- normalize module selection and module-specific options;
- output module selection and module-specific options;
- LLM runtime;
- validation runtime;
- source chunking;
- output and diagnostics.
- diagnostics.
Adapter-specific and extractor-specific config should not leak into unrelated
core config sections.
Module-specific config should not leak into unrelated core config sections.
### Examples Should Stay Real
@@ -154,8 +164,8 @@ When the first vertical slice is implemented, add or update:
- `docs/troubleshooting.md`: common failures.
- `docs/internal/overview.md`: implemented package map.
- `docs/internal/pipeline.md`: implemented extraction flow.
- `docs/internal/adapters.md`: adapter contract and implemented adapters.
- `docs/internal/extractors.md`: extractor contract and built-ins.
- `docs/internal/stage-modules.md`: stage contracts and implemented modules.
- `docs/internal/input-modules.md`: input adapter contract and implemented input modules.
- `docs/internal/validators.md`: validator contract and built-ins.
- `docs/integrations/seriatim-transcript.md`: Seriatim input contract.
- `docs/integrations/artifacts.md`: output artifact envelope.
@@ -165,10 +175,10 @@ When the first vertical slice is implemented, add or update:
Before merging docs, check:
- Does the document describe implemented behavior outside `docs/roadmap/`?
- Are source-format details isolated to adapter docs?
- Are D&D details isolated to extractor or artifact docs?
- Are source-format details isolated to input module or integration docs?
- Are D&D details isolated to process module or artifact docs?
- Is there one canonical home for the topic?
- Do command examples match implemented CLI syntax?
- Are examples valid, maintained, and free of secrets?
- Did any architecture, config, CLI, adapter, extractor, validator, or artifact
- Did any architecture, config, CLI, stage module, validator, or artifact
contract change require a docs update?

23
docs/roadmap/implementation.md
View File

@@ -6,10 +6,10 @@ This is a staged implementation plan for
[`2-framework-composition.md`](2-framework-composition.md). It is intended for
an LLM coding agent to follow stage by stage.
This plan implements only checkpoint 2. Do not implement real input adapters,
real extractors, source chunking, LLM provider clients, prompt assets, response
schema registries, diagnostics run directories, production config loading, or
D&D artifact schemas in this pass.
This plan implements only checkpoint 2. Do not implement real input modules,
real process modules, source chunking, LLM provider clients, prompt assets,
response schema registries, diagnostics run directories, production config
loading, or D&D artifact schemas in this pass.
## Policy Context
@@ -21,8 +21,8 @@ Follow:
Required boundaries:
- framework packages must stay source-agnostic and domain-agnostic;
- input adapter registry code must not import concrete adapter packages;
- extractor registry and runner code must not import concrete extractor
- input adapter registry code must not import concrete input module packages;
- extractor registry and runner code must not import concrete process module
packages;
- runner code should operate on `SourceDocument`, not transcript-specific
structures;
@@ -130,7 +130,7 @@ Cover:
- sorted `RegisteredKeys`;
- nil registry behavior.
Use fake adapters only. Do not add concrete adapter packages.
Use fake adapters only. Do not add concrete input module packages.
### Validation
@@ -196,7 +196,7 @@ Cover the same cases as the input adapter registry:
- sorted `RegisteredKeys`;
- nil registry behavior.
Use fake extractors only. Do not add concrete extractor packages.
Use fake extractors only. Do not add concrete process module packages.
### Validation
@@ -478,7 +478,8 @@ go test ./...
### Goal
Prove the extractor registry and runner compose without adding real extractors.
Prove the extractor registry and runner compose without adding real process
modules.
### Files To Add
@@ -501,7 +502,7 @@ Assertions:
- extractor execution follows configured key order;
- approved artifacts are in deterministic order;
- rejected artifacts are in deterministic order;
- no concrete adapter or extractor packages are imported.
- no concrete input module or process module packages are imported.
### Validation
@@ -527,7 +528,7 @@ Check:
- no package names mention transcripts, Seriatim, D&D, spells, NPCs, items, or
combat;
- no concrete adapter or extractor package exists;
- no concrete input module or process module package exists;
- no LLM provider code exists;
- no prompt, response schema, diagnostics, config, or source chunking package
was added;

153
docs/roadmap/initial-architecture.md
View File

@@ -11,21 +11,21 @@ Notarius should extract structured JSON artifacts from primary source inputs
using modular, LLM-backed extractors.
The first MVP should target audio transcripts generated by Seriatim. That
choice should be implemented as an input adapter, not as a transcript-specific
assumption in the application core. Later input sources, such as unstructured
Markdown notes or Obsidian documents, should be addable through new adapters and
extractors without reshaping the framework.
choice should be implemented as an input-stage module, not as a
transcript-specific assumption in the application core. Later input sources,
such as unstructured Markdown notes or Obsidian documents, should be addable
through new input and process modules without reshaping the framework.
The first extraction domain should be D&D session analysis, starting with spell
casts. That domain should live in extractor packages and related schemas, not in
core framework packages.
casts. That domain should live in process-stage modules and related schemas, not
in core framework packages.
The application should follow the same broad architecture as Audita:
- deterministic core packages for config, source documents, artifacts, diagnostics, and reporting;
- input adapters that translate external source formats into a small internal source model;
- input-stage modules that translate external source formats into a small internal source model;
- reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation;
- independent extractor packages that own domain-specific behavior;
- independent process-stage modules that own domain-specific behavior;
- independent validator packages;
- embedded prompt and JSON schema assets;
- CLI orchestration that wires the pieces together without owning domain logic.
@@ -36,8 +36,8 @@ artifacts rather than proposing and applying transcript corrections.
## Architectural Principles
- Keep the core input model generic: ordered text units plus metadata.
- Keep source-format details in hexagonal input adapters.
- Keep extraction-domain details in extractor packages.
- Keep source-format details in hexagonal input modules.
- Keep extraction-domain details in process modules.
- Treat evidence as source references, not transcript references.
- Prefer narrow, useful abstractions over a universal document model.
- Preserve enough provenance for validation, replay, and downstream inspection.
@@ -57,12 +57,13 @@ internal/core/reporting
internal/core/extractorcatalog
internal/core/inputcatalog
internal/adapters/input/seriatim
internal/adapters/input/markdown
internal/framework/contracts
internal/framework/extraction
internal/framework/runner
internal/framework/pipeline
internal/framework/merge
internal/framework/normalize
internal/framework/output
internal/framework/validators
internal/framework/llm
internal/framework/responseschema
@@ -70,10 +71,24 @@ internal/framework/structuredoutput
internal/framework/promptcontext
internal/framework/warnings
internal/extractors/dnd/spells
internal/extractors/dnd/items
internal/extractors/dnd/npcs
internal/extractors/dnd/combat
internal/modules/input/seriatim
internal/modules/input/markdown
internal/modules/chunk/generic
internal/modules/chunk/dndtranscript
internal/modules/process/dnd/spells
internal/modules/process/dnd/items
internal/modules/process/dnd/npcs
internal/modules/process/dnd/combat
internal/modules/merge/appendorder
internal/modules/merge/dnd/spells
internal/modules/normalize/noop
internal/modules/normalize/dnd/spells
internal/modules/output/json
internal/validators/source_refs
internal/validators/schema_validity
@@ -85,9 +100,9 @@ examples
docs/internal
```
The `markdown` adapter is listed as a likely future package. The MVP should only
implement the Seriatim adapter unless a second adapter is needed to test the
boundary.
The `markdown` input module and D&D-specific chunk, merge, normalize, and
output modules are listed as likely future packages. The MVP should implement
only the stage modules needed by the checkpoint sequence.
## Core Concepts
@@ -123,7 +138,7 @@ Initial source-unit assumptions:
- adapter-specific metadata may carry speaker, timestamps, heading paths, page
numbers, or other source details.
### Input Adapter
### Input Module / Adapter Contract
Hexagonal boundary for external source formats.
@@ -134,7 +149,7 @@ type InputAdapter interface {
}
```
The MVP adapter should target Seriatim minimal transcript JSON. Seriatim segment
The MVP input module should target Seriatim minimal transcript JSON. Seriatim segment
fields should map as follows:
- `id` becomes `SourceUnit.ID`;
@@ -178,17 +193,44 @@ type Extractor interface {
ArtifactType() string
SchemaVersion() string
Validators() []Validator
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
Process(ctx context.Context, req ProcessRequest) (ProcessResult, error)
}
```
An extractor should receive either a whole source document or a source slice,
depending on runner configuration. It should return typed artifact candidates
plus warnings. It should not mutate the source document.
An extractor should receive either a whole source document or a source chunk,
depending on processing mode. It should return typed artifact candidates plus
warnings. It should not mutate the source document.
Extractor packages own domain concepts. For example, D&D spell extraction should
live under `internal/extractors/dnd/spells`; a future to-do extractor for notes
should live under a different domain path and use the same framework contract.
Process modules own domain concepts. For example, D&D spell extraction should
live under `internal/modules/process/dnd/spells`; a future to-do extractor for
notes should live under a different process-module path and use the same
framework contract.
### Chunker
Reusable stage contract for splitting a source document into ordered source
chunks.
Chunking is a first-class pipeline concern because source documents may exceed a
single LLM extraction pass. Chunkers should preserve source-unit order and
produce stable chunk metadata suitable for diagnostics and replay.
### Merger
Reusable stage contract for combining per-chunk artifact candidates into one
merged candidate collection.
Merge should combine outputs without doing semantic reconciliation. A generic
append-in-chunk-order merger should be sufficient for many artifact streams,
including the likely first D&D spell-cast extractor.
### Normalizer
Reusable stage contract for reconciling merged artifact candidates.
Normalize is distinct from merge. Normalizers may deduplicate repeated facts,
resolve aliases, reconcile conflicting fields, check cross-chunk consistency,
or attach normalization warnings.
### Validator
@@ -303,27 +345,37 @@ The architecture should support extractors outside the D&D domain. Examples:
- decisions and action items from meeting transcripts;
- named people, places, and dates from research notes.
These should be addable as extractor packages without changing runner,
These should be addable as process modules without changing runner,
validator, source-reference, or LLM framework contracts.
## Proposed Runner Flow
## Proposed Pipeline Flow
The application workflow should be first-class:
```text
input -> chunk -> process -> merge -> normalize -> output
```
Proposed runner flow:
1. Load effective config.
2. Create diagnostics run directory.
3. Resolve the configured input adapter.
3. Resolve the configured input module through the input adapter registry.
4. Read source input.
5. Parse source input into a `SourceDocument`.
6. Validate source-document invariants.
7. Chunk source units into deterministic source slices.
8. Resolve configured extractor instances through a registry.
9. Execute extractor instances in configured order.
10. Run deterministic validators before LLM-backed validators.
11. Retain approved artifacts and rejected-artifact diagnostics.
12. Merge approved slice artifacts deterministically.
13. Serialize final output JSON.
14. Write run manifest, diagnostics, and optional report JSON.
7. Resolve the configured chunker.
8. Chunk source units into deterministic source chunks.
9. Resolve configured extractor instances through a registry.
10. Process chunks in extractor-defined mode.
11. Merge per-chunk artifact candidates deterministically.
12. Normalize merged artifact candidates.
13. Run deterministic validators before LLM-backed validators.
14. Retain approved artifacts and rejected-artifact diagnostics.
15. Serialize final output JSON.
16. Write run manifest, diagnostics, and optional report JSON.
The runner should operate on source documents and source slices only. Any
The runner should operate on source documents and source chunks only. Any
transcript-specific behavior should happen before the runner, inside the input
adapter, or after the runner, inside output rendering that understands source
metadata.
@@ -335,6 +387,7 @@ Reuse these architectural patterns:
- deterministic parsing and schema validation style;
- deterministic chunking of ordered source units;
- explicit extractor registry;
- explicit pipeline stage contracts;
- `contracts` package for transport-neutral interfaces;
- OpenAI-compatible structured LLM client;
- scheduler for bounded LLM concurrency;
@@ -358,17 +411,18 @@ extraction-report concepts.
## Checkpoint Roadmap
The initial implementation should proceed through five coherent checkpoints.
The initial implementation should proceed through six coherent checkpoints.
Each checkpoint should leave the repository in a reviewable state, with the code
compiling and targeted tests covering the newly introduced contracts or behavior.
1. [Core Contracts And Skeleton](1-core-contracts-and-skeleton.md)
2. [Framework Composition](2-framework-composition.md)
3. [Portable Audita Infrastructure](3-portable-audita-infrastructure.md)
4. [Seriatim Input Adapter](4-seriatim-input-adapter.md)
5. [D&D Spells Extractor](5-dnd-spells-extractor.md)
3. [Pipeline Stages, Chunking, Merge, And Normalize](3-pipeline-stages-chunking-merge-normalize.md)
4. [Portable Audita Infrastructure](4-portable-audita-infrastructure.md)
5. [Seriatim Input Module](5-seriatim-input-module.md)
6. [D&D Spells Extractor](6-dnd-spells-extractor.md)
The first useful vertical slice should arrive at checkpoint 5: Seriatim
The first useful vertical slice should arrive at checkpoint 6: Seriatim
transcript input to validated D&D spell artifact output. Earlier checkpoints are
intentionally contract-first and may not produce useful user output yet.
@@ -383,10 +437,11 @@ intentionally contract-first and may not produce useful user output yet.
artifact metadata be allowed without source references?
- Should overlapping source-reference ranges be merged, preserved exactly, or
both?
- Should extraction run independently per source slice only, or should some
- Should extraction run independently per source chunk only, or should some
extractors receive whole-document context?
- Should a later reconciliation stage deduplicate entities and events across
source slices?
- Which artifact types can use a generic append-in-chunk-order merger?
- Which artifact types need domain-specific normalization for deduplication,
identity resolution, or consistency?
- Should LLM review be part of each extractor's validator chain or a separate
review phase?
- Should the Seriatim adapter accept only its minimal schema initially or also