eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update the architecture plan to standardize on input -> chunk -> extract -> merge -> normalize -> output naming conventions

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-03 09:03:45 -05:00
parent 88042174b3
commit 5a6e82f599
18 changed files with 121 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
docs/policy/architecture.md
View File

@@ -29,7 +29,7 @@ Source-format details belong in input modules. Transcript-specific concepts
such as segments, speakers, timestamps, and transcript schemas must not spread
into runner, extractor, or validator framework code.
Extraction-domain details belong in process modules. D&D-specific concepts
Extraction-domain details belong in extract modules. D&D-specific concepts
such as spells, NPCs, items, combat turns, and encounters must not spread into
core source, runner, or LLM framework packages.
@@ -39,11 +39,11 @@ should point to generic source units, not to transcript-only structures.
The application workflow is:
```text
input -> chunk -> process -> merge -> normalize -> output
input -> chunk -> extract -> merge -> normalize -> output
```
These stages should remain explicit in the architecture. Chunking, merging, and
normalization must not be hidden inside domain process modules when they represent
normalization must not be hidden inside domain extract modules when they represent
general pipeline behavior.
## Dependency Policy
@@ -99,7 +99,7 @@ Domain implementations:
- `internal/modules/input/<name>`: input-stage modules that parse external input into core source documents.
- `internal/modules/chunk/<name>`: chunk-stage modules.
- `internal/modules/process/<domain>/<name>`: process-stage extractor modules.
- `internal/modules/extract/<domain>/<name>`: extract-stage extractor modules.
- `internal/modules/merge/<name>` or `internal/modules/merge/<domain>/<name>`: merge-stage modules.
- `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
- `internal/modules/output/<name>`: output-stage modules.
@@ -120,7 +120,7 @@ workflow visible in the filesystem:
```text
internal/modules/input/...
internal/modules/chunk/...
internal/modules/process/...
internal/modules/extract/...
internal/modules/merge/...
internal/modules/normalize/...
internal/modules/output/...
@@ -141,7 +141,7 @@ decisions.
Other packages should interact with source input through adapter contracts and
core source types. Input module implementation details and external dependency
types must not leak into framework or process module packages.
types must not leak into framework or extract module packages.
Input module metadata may preserve source-specific facts such as transcript speaker,
timestamps, Markdown heading path, page number, or block ID. Framework code may
@@ -152,7 +152,7 @@ shape.
Extractors are independent modules that process source chunks or whole source
documents and produce one kind of structured artifact candidate.
Each process module owns:
Each extract module owns:
- its artifact semantics;
- its prompt usage;
@@ -160,7 +160,7 @@ Each process module owns:
- its validator chain;
- any domain-specific mapping or interpretation.
Process modules should depend on framework contracts and core source/artifact
Extract modules should depend on framework contracts and core source/artifact
types. They should not depend on concrete input module packages.
Extractors should not be the only place where chunking, merging, or
@@ -181,7 +181,7 @@ The pipeline has six conceptual stages:
1. input: external source material becomes a `SourceDocument`;
2. chunk: a `SourceDocument` becomes ordered source chunks;
3. process: extractors produce artifact candidates from chunks or whole documents;
3. extract: extractors produce artifact candidates from chunks or whole documents;
4. merge: per-chunk candidates become a merged candidate collection;
5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
6. output: final artifacts are serialized.
@@ -290,11 +290,11 @@ be opt-in.
## Testing
Core logic should be testable without real external services. Use fakes,
fixtures, or local test doubles for adapters, extractors, validators, and LLM
clients where practical.
fixtures, or local test doubles for input modules, extract modules, validators,
and LLM clients where practical.
Contract-first work should include fake implementations that prove interfaces
compose before real adapters or extractors depend on them.
compose before real input modules or extract modules depend on them.
Config examples should be load-tested once config files exist. Important CLI
workflows should have parser or command tests. Adapter, extractor, and validator
@@ -312,7 +312,7 @@ unit, source reference, input adapter, extractor, chunker, merger, normalizer,
artifact, validator, and run manifest.
Source-format details belong in input module or integration docs.
Domain-specific extraction details belong in process module or artifact docs.
Domain-specific extraction details belong in extract module or artifact docs.
When changing architecture, config, CLI behavior, stage modules, extractor
contracts, validator contracts, LLM runtime behavior, or artifact schemas, update

4
docs/roadmap/1-core-contracts-and-skeleton.md
View File

@@ -26,7 +26,7 @@ In scope:
Out of scope:
- real input modules;
- real process modules;
- real extract modules;
- LLM provider calls;
- prompt or response-schema assets;
- diagnostics run directory;
@@ -47,7 +47,7 @@ contract packages:
structured LLM interfaces used by later checkpoints.
The contracts should be proven with fake implementations in tests. Those tests
should demonstrate composition without real input modules, real process modules,
should demonstrate composition without real input modules, real extract modules,
LLM provider calls, prompt assets, or diagnostics infrastructure.
Implementation staging belongs in

4
docs/roadmap/2-framework-composition.md
View File

@@ -27,7 +27,7 @@ Out of scope:
- real input parsing;
- real input modules;
- real process modules;
- real extract modules;
- real LLM calls;
- prompt assets;
- response schema assets;
@@ -60,7 +60,7 @@ Implementation staging belongs in
- `go test ./...` passes.
- Fake adapter/extractor/validator registrations work in tests.
- The runner operates on `SourceDocument`, not transcript-specific structures.
- The runner does not import concrete D&D process module packages.
- The runner does not import concrete D&D extract module packages.
## Review Questions

18
docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md
View File

@@ -7,17 +7,17 @@ This document describes planned work, not implemented behavior.
## Goal
Make Notarius's application workflow first-class before adding real input
modules or process modules.
modules or extract modules.
The workflow should be:
```text
input -> chunk -> process -> merge -> normalize -> output
input -> chunk -> extract -> merge -> normalize -> output
```
Checkpoint 3 should define the contracts and minimal fake-tested framework
behavior for chunking, per-chunk processing, merging, and normalization. It
should not add real input modules, real domain process modules, LLM provider code,
should not add real input modules, real domain extract modules, LLM provider code,
or output encoders.
## Scope
@@ -26,7 +26,7 @@ In scope:
- source chunk model;
- chunker contract;
- process-stage contract for extractors operating on chunks;
- extract-stage contract for extractors operating on chunks;
- merge-stage contract;
- normalize-stage contract;
- output-stage contract shape, if useful for pipeline completeness;
@@ -50,13 +50,13 @@ The repository should contain explicit pipeline-stage contracts:
- `InputAdapter`: external source input to `SourceDocument`.
- `Chunker`: `SourceDocument` to ordered `SourceChunk` values.
- `Extractor` or processor: `SourceChunk` to artifact candidates.
- `Extractor`: `SourceChunk` to artifact candidates.
- `Merger`: per-chunk candidates to merged candidates.
- `Normalizer`: merged candidates to normalized candidates.
- `OutputEncoder`: final artifact bundle to bytes, if introduced in this
checkpoint.
The runner should orchestrate fake implementations through chunk, process,
The runner should orchestrate fake implementations through chunk, extract,
merge, normalize, and approval/validation behavior in tests.
## Design Intent
@@ -113,17 +113,17 @@ Domain-specific normalizers may later:
- `go test ./...` passes.
- Pipeline-stage contracts are explicit and source/domain agnostic.
- Fake tests prove input source documents can be chunked, processed, merged, and
- Fake tests prove input source documents can be chunked, extracted, merged, and
normalized.
- Merge and normalize are distinct concepts in code and tests.
- The runner no longer implies whole-document-only extraction as the core
application model.
- No concrete input module, process module, LLM provider, prompt, response schema,
- No concrete input module, extract module, LLM provider, prompt, response schema,
diagnostics, config, or D&D artifact code is added.
## Review Questions
- Is the workflow clearly represented as input, chunk, process, merge,
- Is the workflow clearly represented as input, chunk, extract, merge,
normalize, and output?
- Are merge and normalize cleanly separated?
- Can a generic merger handle simple chronological artifact streams?

12
docs/roadmap/6-dnd-spells-extractor.md
View File

@@ -6,7 +6,7 @@ This document describes planned work, not implemented behavior.
## Goal
Implement the first useful process-stage module: D&D spell casts from a
Implement the first useful extract-stage module: D&D spell casts from a
Seriatim transcript source document.
This checkpoint should produce the first meaningful vertical slice from real
@@ -19,7 +19,7 @@ In scope:
- D&D spell artifact schema and Go structs;
- structured response schema asset;
- prompt assets;
- `internal/modules/process/dnd/spells`;
- `internal/modules/extract/dnd/spells`;
- source-reference and schema validators in the extractor chain;
- fake LLM tests;
- CLI-level integration test if the CLI path is ready.
@@ -50,7 +50,7 @@ type SpellCast struct {
}
```
Keep this schema inside the D&D spells process module or a D&D artifact package,
Keep this schema inside the D&D spells extract module or a D&D artifact package,
not inside core framework packages.
### Stage 2: Structured Response Schema
@@ -76,7 +76,7 @@ Prompts should:
### Stage 4: Process Module Implementation
Implement `internal/modules/process/dnd/spells`.
Implement `internal/modules/extract/dnd/spells`.
The extractor should:
@@ -117,13 +117,13 @@ The test should use fake LLM wiring and fixture input.
- `go test ./...` passes.
- Seriatim input can flow through the runner into the D&D spells extractor.
- Spell artifacts include valid source references.
- D&D concepts are contained in process module/artifact packages and docs.
- D&D concepts are contained in extract module/artifact packages and docs.
- The first meaningful vertical slice is available through tests, and through
CLI if the CLI path is ready.
## Review Questions
- Is the spell process module domain-specific without making the framework
- Is the spell extract module domain-specific without making the framework
D&D-specific?
- Are source references valid and useful for downstream validation?
- Is prompt/schema ownership clear?

14
docs/roadmap/documentation.md
View File

@@ -11,7 +11,7 @@ not describe implemented behavior.
Notarius documentation should make three boundaries obvious:
- source-format support belongs to input-stage modules;
- extraction-domain behavior belongs to process-stage modules;
- extraction-domain behavior belongs to extract-stage modules;
- core framework behavior is source-agnostic and domain-agnostic.
Documentation should avoid making the MVP look more transcript-specific or
@@ -47,7 +47,7 @@ Core docs should avoid transcript-specific terms such as segment, speaker,
timestamp, and transcript range unless discussing an input adapter or an example.
Core docs should avoid D&D-specific terms such as spell, NPC, item, combat, and
encounter unless discussing process modules, artifact docs, or examples.
encounter unless discussing extract modules, artifact docs, or examples.
### Input Module Docs Own Source Formats
@@ -95,12 +95,12 @@ Stage module docs should cover:
- prompt and response-schema ownership;
- examples.
D&D concepts should be documented in D&D process-module or artifact docs, not in
D&D concepts should be documented in D&D extract-module or artifact docs, not in
generic runner or framework docs.
### CLI Docs Should Reflect Extensibility
The CLI reference should present input modules, chunk modules, process modules,
The CLI reference should present input modules, chunk modules, extract modules,
merge modules, normalize modules, and output modules as selectable or
configurable components as they become user-facing.
@@ -114,7 +114,7 @@ Once implemented, `docs/cli.md` should document:
- positional source input path;
- input module selection;
- process module selection;
- extract module selection;
- chunk/merge/normalize/output selection when configurable;
- config path behavior;
- output path behavior;
@@ -127,7 +127,7 @@ Once implemented, `docs/cli.md` should document:
- input module selection and module-specific options;
- chunk module selection and module-specific options;
- process module selection and module-specific options;
- extract module selection and module-specific options;
- merge module selection and module-specific options;
- normalize module selection and module-specific options;
- output module selection and module-specific options;
@@ -176,7 +176,7 @@ Before merging docs, check:
- Does the document describe implemented behavior outside `docs/roadmap/`?
- Are source-format details isolated to input module or integration docs?
- Are D&D details isolated to process module or artifact docs?
- Are D&D details isolated to extract module or artifact docs?
- Is there one canonical home for the topic?
- Do command examples match implemented CLI syntax?
- Are examples valid, maintained, and free of secrets?

22
docs/roadmap/implementation.md
View File

@@ -7,7 +7,7 @@ This is a staged implementation plan for
an LLM coding agent to follow stage by stage.
This plan implements only checkpoint 2. Do not implement real input modules,
real process modules, source chunking, LLM provider clients, prompt assets,
real extract modules, source chunking, LLM provider clients, prompt assets,
response schema registries, diagnostics run directories, production config
loading, or D&D artifact schemas in this pass.
@@ -22,7 +22,7 @@ Required boundaries:
- framework packages must stay source-agnostic and domain-agnostic;
- input adapter registry code must not import concrete input module packages;
- extractor registry and runner code must not import concrete process module
- extractor registry and runner code must not import concrete extract module
packages;
- runner code should operate on `SourceDocument`, not transcript-specific
structures;
@@ -196,7 +196,7 @@ Cover the same cases as the input adapter registry:
- sorted `RegisteredKeys`;
- nil registry behavior.
Use fake extractors only. Do not add concrete process module packages.
Use fake extractors only. Do not add concrete extract module packages.
### Validation
@@ -236,7 +236,7 @@ Define helpers:
```go
func Approved(candidateIndex int) contracts.ValidationDecision
func Rejected(candidateIndex int, reasonCode string, message string) contracts.ValidationDecision
func EnforceDecisionCardinality(candidates []artifacts.Candidate, decisions []contracts.ValidationDecision) error
func EnforceDecisionCardinality(candidates []artifacts.ArtifactCandidate, decisions []contracts.ValidationDecision) error
```
### Required Behavior
@@ -264,8 +264,8 @@ func EnforceDecisionCardinality(candidates []artifacts.Candidate, decisions []co
- work with candidate indices, not slice positions;
- allow an empty candidate slice only when decisions are also empty.
Candidate indices are expected to be unique by the time validators run. If the
candidate slice itself contains duplicate indices, return an error.
Artifact candidate indices are expected to be unique by the time validators run.
If the candidate slice itself contains duplicate indices, return an error.
### Required Tests
@@ -396,7 +396,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error)
1. reject nil runner or nil extractor factory;
2. validate `input.Source` with `source.ValidateDocument`;
3. reject an empty `ExtractorKeys` list;
4. process extractor keys in input order;
4. iterate extractor keys in input order;
5. build each extractor through the factory;
6. call `Extractor.Extract` with the source, LLM client, and metadata;
7. append extractor warnings to `RunOutput.Warnings`;
@@ -406,9 +406,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error)
11. reject candidates denied by validators;
12. return approved/rejected artifacts in deterministic order.
Candidate normalization must:
Artifact candidate normalization must:
- assign `Candidate.Index` using a global monotonically increasing counter;
- assign `ArtifactCandidate.Index` using a global monotonically increasing counter;
- fill empty `ExtractorKey`, `ArtifactType`, and `SchemaVersion` from the
owning extractor;
- return an error if a candidate's non-empty `ExtractorKey` differs from
@@ -502,7 +502,7 @@ Assertions:
- extractor execution follows configured key order;
- approved artifacts are in deterministic order;
- rejected artifacts are in deterministic order;
- no concrete input module or process module packages are imported.
- no concrete input module or extract module packages are imported.
### Validation
@@ -528,7 +528,7 @@ Check:
- no package names mention transcripts, Seriatim, D&D, spells, NPCs, items, or
combat;
- no concrete input module or process module package exists;
- no concrete input module or extract module package exists;
- no LLM provider code exists;
- no prompt, response schema, diagnostics, config, or source chunking package
was added;

34
docs/roadmap/initial-architecture.md
View File

@@ -14,10 +14,10 @@ The first MVP should target audio transcripts generated by Seriatim. That
choice should be implemented as an input-stage module, not as a
transcript-specific assumption in the application core. Later input sources,
such as unstructured Markdown notes or Obsidian documents, should be addable
through new input and process modules without reshaping the framework.
through new input and extract modules without reshaping the framework.
The first extraction domain should be D&D session analysis, starting with spell
casts. That domain should live in process-stage modules and related schemas, not
casts. That domain should live in extract-stage modules and related schemas, not
in core framework packages.
The application should follow the same broad architecture as Audita:
@@ -25,7 +25,7 @@ The application should follow the same broad architecture as Audita:
- deterministic core packages for config, source documents, artifacts, diagnostics, and reporting;
- input-stage modules that translate external source formats into a small internal source model;
- reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation;
- independent process-stage modules that own domain-specific behavior;
- independent extract-stage modules that own domain-specific behavior;
- independent validator packages;
- embedded prompt and JSON schema assets;
- CLI orchestration that wires the pieces together without owning domain logic.
@@ -37,7 +37,7 @@ artifacts rather than proposing and applying transcript corrections.
- Keep the core input model generic: ordered text units plus metadata.
- Keep source-format details in hexagonal input modules.
- Keep extraction-domain details in process modules.
- Keep extraction-domain details in extract modules.
- Treat evidence as source references, not transcript references.
- Prefer narrow, useful abstractions over a universal document model.
- Preserve enough provenance for validation, replay, and downstream inspection.
@@ -77,10 +77,10 @@ internal/modules/input/markdown
internal/modules/chunk/generic
internal/modules/chunk/dndtranscript
internal/modules/process/dnd/spells
internal/modules/process/dnd/items
internal/modules/process/dnd/npcs
internal/modules/process/dnd/combat
internal/modules/extract/dnd/spells
internal/modules/extract/dnd/items
internal/modules/extract/dnd/npcs
internal/modules/extract/dnd/combat
internal/modules/merge/appendorder
internal/modules/merge/dnd/spells
@@ -193,7 +193,7 @@ type Extractor interface {
ArtifactType() string
SchemaVersion() string
Validators() []Validator
Process(ctx context.Context, req ProcessRequest) (ProcessResult, error)
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
```
@@ -201,9 +201,9 @@ An extractor should receive either a whole source document or a source chunk,
depending on processing mode. It should return typed artifact candidates plus
warnings. It should not mutate the source document.
Process modules own domain concepts. For example, D&D spell extraction should
live under `internal/modules/process/dnd/spells`; a future to-do extractor for
notes should live under a different process-module path and use the same
Extract modules own domain concepts. For example, D&D spell extraction should
live under `internal/modules/extract/dnd/spells`; a future to-do extractor for
notes should live under a different extract-module path and use the same
framework contract.
### Chunker
@@ -262,9 +262,13 @@ Per-run provenance record.
```go
type RunManifest struct {
InputAdapter string `json:"input_adapter"`
InputModule string `json:"input_module"`
Chunker string `json:"chunker"`
SourceDigests []string `json:"source_digests"`
Extractors []string `json:"extractors"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
OutputEncoder string `json:"output_encoder"`
SchemaVersion string `json:"schema_version"`
ValidationStatus string `json:"validation_status"`
}
@@ -345,7 +349,7 @@ The architecture should support extractors outside the D&D domain. Examples:
- decisions and action items from meeting transcripts;
- named people, places, and dates from research notes.
These should be addable as process modules without changing runner,
These should be addable as extract modules without changing runner,
validator, source-reference, or LLM framework contracts.
## Proposed Pipeline Flow
@@ -353,7 +357,7 @@ validator, source-reference, or LLM framework contracts.
The application workflow should be first-class:
```text
input -> chunk -> process -> merge -> normalize -> output
input -> chunk -> extract -> merge -> normalize -> output
```
Proposed runner flow: