diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 7f2bb79..10018f9 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -25,17 +25,27 @@ contracts that can be exercised by tests and real modules. The core framework must remain source-agnostic and domain-agnostic. -Source-format details belong in input adapters. Transcript-specific concepts +Source-format details belong in input modules. Transcript-specific concepts such as segments, speakers, timestamps, and transcript schemas must not spread into runner, extractor, or validator framework code. -Extraction-domain details belong in extractor packages. D&D-specific concepts +Extraction-domain details belong in process modules. D&D-specific concepts such as spells, NPCs, items, combat turns, and encounters must not spread into core source, runner, or LLM framework packages. Extracted facts should be grounded with source references. Source references should point to generic source units, not to transcript-only structures. +The application workflow is: + +```text +input -> chunk -> process -> merge -> normalize -> output +``` + +These stages should remain explicit in the architecture. Chunking, merging, and +normalization must not be hidden inside domain process modules when they represent +general pipeline behavior. + ## Dependency Policy Prefer the Go standard library where practical. @@ -69,16 +79,15 @@ Core deterministic model and policy: - `internal/core/inputcatalog`: known input adapter keys and metadata. - `internal/core/extractorcatalog`: known extractor keys and metadata. -External source and provider adapters: - -- `internal/adapters/input/`: source-format adapters that parse external input into core source documents. -- `internal/transport/http`: shared HTTP client code, if needed by provider integrations. - Reusable framework plumbing: - `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts. - `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output. +- `internal/framework/pipeline`: shared pipeline-stage orchestration types, when needed. - `internal/framework/extraction`: shared extraction helper code. +- `internal/framework/merge`: shared merge-stage behavior. +- `internal/framework/normalize`: shared normalization-stage behavior. +- `internal/framework/output`: output encoding contracts and shared helpers. - `internal/framework/validators`: shared validator runtime behavior and decision checks. - `internal/framework/llm`: LLM runtime, scheduling, and provider adapters. - `internal/framework/responseschema`: embedded structured-output schema registry. @@ -88,35 +97,62 @@ Reusable framework plumbing: Domain implementations: -- `internal/extractors//`: domain-specific extractor packages. +- `internal/modules/input/`: input-stage modules that parse external input into core source documents. +- `internal/modules/chunk/`: chunk-stage modules. +- `internal/modules/process//`: process-stage extractor modules. +- `internal/modules/merge/` or `internal/modules/merge//`: merge-stage modules. +- `internal/modules/normalize/` or `internal/modules/normalize//`: normalize-stage modules. +- `internal/modules/output/`: output-stage modules. - `internal/validators/`: built-in validator implementations. - `internal/prompts`: embedded prompt assets and prompt metadata registry. +- `internal/transport/http`: shared HTTP client code, if needed by provider integrations. Package-private implementation constants may live near the package that owns them, preferably in `constants.go` when useful. -## Input Adapters +## Stage Modules + +Concrete business logic should live under `internal/modules//...`. + +Stage-oriented module layout is preferred because it makes the application +workflow visible in the filesystem: + +```text +internal/modules/input/... +internal/modules/chunk/... +internal/modules/process/... +internal/modules/merge/... +internal/modules/normalize/... +internal/modules/output/... +``` + +Use short, lowercase, idiomatic Go package names. Prefer names such as +`dndtranscript`, `appendorder`, and `spells` over names like `dnd_transcript`, +`serial_merge`, or `spell_extractor` that repeat parent-stage context. + +## Input Modules Use a hexagonal architecture style for source input. -Input adapters translate external source formats into the core source model. -Adapters may know about external schema details, source-specific metadata, and +Input modules translate external source formats into the core source model. +They may know about external schema details, source-specific metadata, and format-specific validation rules. They should not own extraction-domain decisions. Other packages should interact with source input through adapter contracts and -core source types. Adapter implementation details and external dependency types -must not leak into framework or extractor packages. +core source types. Input module implementation details and external dependency +types must not leak into framework or process module packages. -Adapter metadata may preserve source-specific facts such as transcript speaker, +Input module metadata may preserve source-specific facts such as transcript speaker, timestamps, Markdown heading path, page number, or block ID. Framework code may carry metadata through, but should not require a specific adapter's metadata shape. ## Extractors -Extractors are independent modules that produce one kind of structured artifact. -Each extractor package owns: +Extractors are independent modules that process source chunks or whole source +documents and produce one kind of structured artifact candidate. +Each process module owns: - its artifact semantics; - its prompt usage; @@ -124,8 +160,13 @@ Each extractor package owns: - its validator chain; - any domain-specific mapping or interpretation. -Extractors should depend on framework contracts and core source/artifact types. -They should not depend on concrete input adapter packages. +Process modules should depend on framework contracts and core source/artifact +types. They should not depend on concrete input module packages. + +Extractors should not be the only place where chunking, merging, or +normalization happens. They may choose processing mode or provide domain-specific +merge/normalization behavior when generic behavior is insufficient, but the +pipeline stages themselves are framework concepts. The runner should be able to compose, skip, resume, or run individual extractors when their prerequisites are satisfied. Ordering should be explicit through @@ -134,6 +175,31 @@ configuration, a default sequence, or documented orchestration rules. Extractor selection must go through a registry or equivalent mechanism rather than scattered conditionals. +## Pipeline Stages + +The pipeline has six conceptual stages: + +1. input: external source material becomes a `SourceDocument`; +2. chunk: a `SourceDocument` becomes ordered source chunks; +3. process: extractors produce artifact candidates from chunks or whole documents; +4. merge: per-chunk candidates become a merged candidate collection; +5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues; +6. output: final artifacts are serialized. + +Chunking is first-class because source documents may be too large for a single +LLM pass. Chunkers should preserve source-unit order and produce stable chunk +metadata. + +Merge and normalize are separate concerns. Merge combines per-chunk results into +a deterministic collection. Normalize performs semantic reconciliation after +merge. Generic append-in-chunk-order merge and no-op normalization should be +available for simple artifact types, while domain-specific behavior can be +provided where needed. + +The framework should allow serial and parallel chunk processing. The first +implementation may execute chunks serially for determinism, but contracts should +not prevent later parallel execution. + ## Validators Validators should be independently testable and composable. @@ -181,8 +247,8 @@ Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets. -Adapter-specific and extractor-specific configuration should remain grouped by -the adapter or extractor that owns it. +Stage-module-specific configuration should remain grouped by the module that +owns it. ## Embedded Assets @@ -242,12 +308,12 @@ focused on implemented behavior. Put future, planned, or aspirational work only under `docs/roadmap/`. Core documentation should use generic terms such as source document, source -unit, source reference, input adapter, extractor, artifact, validator, and run -manifest. +unit, source reference, input adapter, extractor, chunker, merger, normalizer, +artifact, validator, and run manifest. -Source-format details belong in adapter or integration docs. Domain-specific -extraction details belong in extractor or artifact docs. +Source-format details belong in input module or integration docs. +Domain-specific extraction details belong in process module or artifact docs. -When changing architecture, config, CLI behavior, adapters, extractor contracts, -validator contracts, LLM runtime behavior, or artifact schemas, update the -relevant docs and examples in the same change. +When changing architecture, config, CLI behavior, stage modules, extractor +contracts, validator contracts, LLM runtime behavior, or artifact schemas, update +the relevant docs and examples in the same change. diff --git a/docs/roadmap/1-core-contracts-and-skeleton.md b/docs/roadmap/1-core-contracts-and-skeleton.md index b7430fd..cab47f2 100644 --- a/docs/roadmap/1-core-contracts-and-skeleton.md +++ b/docs/roadmap/1-core-contracts-and-skeleton.md @@ -25,8 +25,8 @@ In scope: Out of scope: -- real input adapters; -- real extractors; +- real input modules; +- real process modules; - LLM provider calls; - prompt or response-schema assets; - diagnostics run directory; @@ -47,7 +47,7 @@ contract packages: structured LLM interfaces used by later checkpoints. The contracts should be proven with fake implementations in tests. Those tests -should demonstrate composition without real source adapters, real extractors, +should demonstrate composition without real input modules, real process modules, LLM provider calls, prompt assets, or diagnostics infrastructure. Implementation staging belongs in diff --git a/docs/roadmap/2-framework-composition.md b/docs/roadmap/2-framework-composition.md index 83e8824..370f3d8 100644 --- a/docs/roadmap/2-framework-composition.md +++ b/docs/roadmap/2-framework-composition.md @@ -6,8 +6,8 @@ This document describes planned work, not implemented behavior. ## Goal -Prove the core contracts compose before adding real adapters, real extractors, -or portable Audita infrastructure. +Prove the core contracts compose before adding real stage modules or portable +Audita infrastructure. This checkpoint should produce a minimal runner that can execute fake registered components from source input to artifact output in tests. @@ -26,6 +26,8 @@ In scope: Out of scope: - real input parsing; +- real input modules; +- real process modules; - real LLM calls; - prompt assets; - response schema assets; @@ -47,8 +49,8 @@ The repository should contain a minimal framework composition layer: artifacts. The runner should operate on already parsed source documents in this checkpoint. -Raw input parsing and concrete input adapter behavior remain deferred to the -Seriatim adapter checkpoint. +Raw input parsing and concrete input module behavior remain deferred to the +Seriatim input module checkpoint. Implementation staging belongs in [`implementation.md`](implementation.md). @@ -58,7 +60,7 @@ Implementation staging belongs in - `go test ./...` passes. - Fake adapter/extractor/validator registrations work in tests. - The runner operates on `SourceDocument`, not transcript-specific structures. -- The runner does not import concrete D&D extractor packages. +- The runner does not import concrete D&D process module packages. ## Review Questions diff --git a/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md b/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md new file mode 100644 index 0000000..c6a443e --- /dev/null +++ b/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md @@ -0,0 +1,132 @@ +# Checkpoint 3: Pipeline Stages, Chunking, Merge, And Normalize + +## Status + +This document describes planned work, not implemented behavior. + +## Goal + +Make Notarius's application workflow first-class before adding real input +modules or process modules. + +The workflow should be: + +```text +input -> chunk -> process -> merge -> normalize -> output +``` + +Checkpoint 3 should define the contracts and minimal fake-tested framework +behavior for chunking, per-chunk processing, merging, and normalization. It +should not add real input modules, real domain process modules, LLM provider code, +or output encoders. + +## Scope + +In scope: + +- source chunk model; +- chunker contract; +- process-stage contract for extractors operating on chunks; +- merge-stage contract; +- normalize-stage contract; +- output-stage contract shape, if useful for pipeline completeness; +- runner/pipeline updates that exercise these stages with fake components; +- generic append/chronological merge behavior for artifact candidates when + appropriate. + +Out of scope: + +- Seriatim parsing; +- D&D spell extraction; +- LLM provider calls; +- prompt assets; +- response schema assets; +- diagnostics run directory; +- production output serialization. + +## Target End State + +The repository should contain explicit pipeline-stage contracts: + +- `InputAdapter`: external source input to `SourceDocument`. +- `Chunker`: `SourceDocument` to ordered `SourceChunk` values. +- `Extractor` or processor: `SourceChunk` to artifact candidates. +- `Merger`: per-chunk candidates to merged candidates. +- `Normalizer`: merged candidates to normalized candidates. +- `OutputEncoder`: final artifact bundle to bytes, if introduced in this + checkpoint. + +The runner should orchestrate fake implementations through chunk, process, +merge, normalize, and approval/validation behavior in tests. + +## Design Intent + +Chunking is a core application concern because many source documents, especially +transcripts, will be too large for a single LLM extraction pass. + +Chunk processing may be serial or parallel depending on extractor needs. The +architecture should support both, but the first implementation can execute +deterministically in series until a later checkpoint introduces concurrency. + +Merge and normalize are separate stages: + +- merge combines per-chunk extracted candidates into one stream or collection; +- normalize reconciles the merged output by checking duplicates, consistency, + ordering, identity resolution, or other cross-chunk concerns. + +For some artifact types, merge may be generic append-in-source-order behavior. +For other artifact types, merge may be domain-specific. Normalization is where +deduplication and consistency checks should live. + +## Processing Modes + +The architecture should leave room for extractor-level processing modes: + +- whole-document processing; +- serial chunk processing; +- parallel chunk processing. + +The first implementation may model these modes without implementing parallel +execution. It should not bake in a single-pass assumption. + +## Generic Merge Behavior + +A generic merger should be able to concatenate candidates in deterministic +chunk order and candidate order. This is likely sufficient for early spell-cast +extraction, where chronological serialization is useful. + +Domain-specific mergers may be added later when generic ordering is not enough. + +## Generic Normalize Behavior + +A no-op normalizer should be available as the default. + +Domain-specific normalizers may later: + +- deduplicate repeated extracted facts; +- resolve aliases; +- reconcile conflicting candidate fields; +- enforce chronological or source-reference consistency; +- attach normalization warnings. + +## Done Criteria + +- `go test ./...` passes. +- Pipeline-stage contracts are explicit and source/domain agnostic. +- Fake tests prove input source documents can be chunked, processed, merged, and + normalized. +- Merge and normalize are distinct concepts in code and tests. +- The runner no longer implies whole-document-only extraction as the core + application model. +- No concrete input module, process module, LLM provider, prompt, response schema, + diagnostics, config, or D&D artifact code is added. + +## Review Questions + +- Is the workflow clearly represented as input, chunk, process, merge, + normalize, and output? +- Are merge and normalize cleanly separated? +- Can a generic merger handle simple chronological artifact streams? +- Can a later domain-specific normalizer handle duplicates and consistency + without changing core runner contracts? +- Does the design allow serial and parallel chunk processing later? diff --git a/docs/roadmap/3-portable-audita-infrastructure.md b/docs/roadmap/4-portable-audita-infrastructure.md similarity index 95% rename from docs/roadmap/3-portable-audita-infrastructure.md rename to docs/roadmap/4-portable-audita-infrastructure.md index cf06cc0..50fe6c5 100644 --- a/docs/roadmap/3-portable-audita-infrastructure.md +++ b/docs/roadmap/4-portable-audita-infrastructure.md @@ -1,4 +1,4 @@ -# Checkpoint 3: Portable Audita Infrastructure +# Checkpoint 4: Portable Audita Infrastructure ## Status @@ -72,7 +72,7 @@ The registry should track: - prompt hash. Do not add D&D prompt assets here unless the implementation naturally overlaps -with checkpoint 5. Test prompts are acceptable for registry tests. +with checkpoint 6. Test prompts are acceptable for registry tests. ### Stage 4: Diagnostics Run Directory @@ -94,7 +94,7 @@ Add config structs and defaults only for infrastructure that now exists. Initial config areas: -- input adapter key; +- input module key; - extractor keys; - primary LLM settings; - validation LLM settings if needed; diff --git a/docs/roadmap/4-seriatim-input-adapter.md b/docs/roadmap/5-seriatim-input-module.md similarity index 70% rename from docs/roadmap/4-seriatim-input-adapter.md rename to docs/roadmap/5-seriatim-input-module.md index ca67c09..c70db9d 100644 --- a/docs/roadmap/4-seriatim-input-adapter.md +++ b/docs/roadmap/5-seriatim-input-module.md @@ -1,4 +1,4 @@ -# Checkpoint 4: Seriatim Input Adapter +# Checkpoint 5: Seriatim Input Module ## Status @@ -7,7 +7,7 @@ This document describes planned work, not implemented behavior. ## Goal Add the first real input source while keeping transcript-specific behavior -isolated inside an input adapter. +isolated inside an input-stage module. This checkpoint should allow Seriatim minimal transcript JSON to become a generic `SourceDocument`. @@ -16,13 +16,13 @@ generic `SourceDocument`. In scope: -- `internal/adapters/input/seriatim`; +- `internal/modules/input/seriatim`; - parser for Seriatim minimal output JSON; - mapping into `SourceDocument` and `SourceUnit`; - source-document validation; -- adapter registry wiring; +- input adapter registry wiring; - fixtures and tests; -- CLI/config path to select the adapter if the CLI shell exists. +- CLI/config path to select the input module if the CLI shell exists. Out of scope: @@ -33,9 +33,9 @@ Out of scope: ## Proposed Stages -### Stage 1: Seriatim Source Model +### Seriatim Source Model -Define adapter-local structs for the Seriatim minimal output schema. +Define module-local structs for the Seriatim minimal output schema. Expected external shape: @@ -47,9 +47,9 @@ Expected external shape: - segment `speaker`; - segment `text`. -Keep these structs in the adapter package. +Keep these structs in the Seriatim input module package. -### Stage 2: Parse And Validate +### Parse And Validate Implement parser and validation behavior. @@ -62,9 +62,9 @@ Validation should cover: - non-empty segment text; - valid start/end values as appropriate. -Prefer clear adapter-specific errors. +Prefer clear module-specific errors. -### Stage 3: Map To SourceDocument +### Map To SourceDocument Map Seriatim data into the generic source model: @@ -77,9 +77,9 @@ Map Seriatim data into the generic source model: The resulting `SourceDocument` should pass core source validation. -### Stage 4: Registry And CLI Wiring +### Registry And CLI Wiring -Register the adapter under a stable key, likely `seriatim`. +Register the module under a stable input adapter key, likely `seriatim`. If CLI support exists, add provisional selection: @@ -87,9 +87,9 @@ If CLI support exists, add provisional selection: notarius extract ./transcript.json --input seriatim ``` -The command may still use fake extractors until checkpoint 5. +The command may still use fake extractors until checkpoint 6. -### Stage 5: Fixtures And Tests +### Fixtures And Tests Add fixtures and tests for: @@ -105,12 +105,12 @@ Add fixtures and tests for: - `go test ./...` passes. - Seriatim minimal transcript JSON maps into `SourceDocument`. - Transcript fields do not appear in core runner contracts. -- The adapter is selectable through the registry. -- Tests prove transcript-specific assumptions are isolated to the adapter. +- The input module is selectable through the registry. +- Tests prove transcript-specific assumptions are isolated to the input module. ## Review Questions -- Are segment, speaker, and timestamp assumptions contained inside the adapter? +- Are segment, speaker, and timestamp assumptions contained inside the input module? - Are unit IDs stable and suitable for source references? -- Does the adapter preserve enough metadata for transcript-oriented output later? -- Should the adapter accept only Seriatim minimal output for now? +- Does the input module preserve enough metadata for transcript-oriented output later? +- Should the input module accept only Seriatim minimal output for now? diff --git a/docs/roadmap/5-dnd-spells-extractor.md b/docs/roadmap/6-dnd-spells-extractor.md similarity index 83% rename from docs/roadmap/5-dnd-spells-extractor.md rename to docs/roadmap/6-dnd-spells-extractor.md index 2f500e8..c0c15a3 100644 --- a/docs/roadmap/5-dnd-spells-extractor.md +++ b/docs/roadmap/6-dnd-spells-extractor.md @@ -1,4 +1,4 @@ -# Checkpoint 5: D&D Spells Extractor +# Checkpoint 6: D&D Spells Extractor ## Status @@ -6,8 +6,8 @@ This document describes planned work, not implemented behavior. ## Goal -Implement the first useful extraction module: D&D spell casts from a Seriatim -transcript source document. +Implement the first useful process-stage module: D&D spell casts from a +Seriatim transcript source document. This checkpoint should produce the first meaningful vertical slice from real source input to validated artifact output. @@ -19,7 +19,7 @@ In scope: - D&D spell artifact schema and Go structs; - structured response schema asset; - prompt assets; -- `internal/extractors/dnd/spells`; +- `internal/modules/process/dnd/spells`; - source-reference and schema validators in the extractor chain; - fake LLM tests; - CLI-level integration test if the CLI path is ready. @@ -50,8 +50,8 @@ type SpellCast struct { } ``` -Keep this schema inside the D&D spells extractor or a D&D artifact package, not -inside core framework packages. +Keep this schema inside the D&D spells process module or a D&D artifact package, +not inside core framework packages. ### Stage 2: Structured Response Schema @@ -74,14 +74,14 @@ Prompts should: - avoid relying on transcript-specific fields except as optional metadata; - request only spell-cast artifacts. -### Stage 4: Extractor Implementation +### Stage 4: Process Module Implementation -Implement `internal/extractors/dnd/spells`. +Implement `internal/modules/process/dnd/spells`. The extractor should: - satisfy the framework `Extractor` contract; -- build LLM messages from a source document or source slice; +- build LLM messages from a source document or source chunk; - call the structured LLM client; - return artifact candidates with source references; - attach its validator chain. @@ -117,13 +117,13 @@ The test should use fake LLM wiring and fixture input. - `go test ./...` passes. - Seriatim input can flow through the runner into the D&D spells extractor. - Spell artifacts include valid source references. -- D&D concepts are contained in extractor/artifact packages and docs. +- D&D concepts are contained in process module/artifact packages and docs. - The first meaningful vertical slice is available through tests, and through CLI if the CLI path is ready. ## Review Questions -- Is the spell extractor domain-specific without making the framework +- Is the spell process module domain-specific without making the framework D&D-specific? - Are source references valid and useful for downstream validation? - Is prompt/schema ownership clear? diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md index 34bcf42..916cc91 100644 --- a/docs/roadmap/documentation.md +++ b/docs/roadmap/documentation.md @@ -10,8 +10,8 @@ not describe implemented behavior. Notarius documentation should make three boundaries obvious: -- source-format support belongs to input adapters; -- extraction-domain behavior belongs to extractor packages; +- source-format support belongs to input-stage modules; +- extraction-domain behavior belongs to process-stage modules; - core framework behavior is source-agnostic and domain-agnostic. Documentation should avoid making the MVP look more transcript-specific or @@ -35,6 +35,10 @@ Core architecture docs should prefer: - source reference; - input adapter; - extractor; +- chunker; +- merger; +- normalizer; +- output encoder; - artifact; - validator; - run manifest. @@ -43,11 +47,12 @@ Core docs should avoid transcript-specific terms such as segment, speaker, timestamp, and transcript range unless discussing an input adapter or an example. Core docs should avoid D&D-specific terms such as spell, NPC, item, combat, and -encounter unless discussing extractor packages or examples. +encounter unless discussing process modules, artifact docs, or examples. -### Adapter Docs Own Source Formats +### Input Module Docs Own Source Formats -Each implemented input adapter should have a canonical integration document. +Each implemented input-stage module should have a canonical integration +document. Likely future files: @@ -56,32 +61,33 @@ docs/integrations/seriatim-transcript.md docs/integrations/markdown-source.md ``` -Adapter docs should cover: +Input module docs should cover: - accepted external schema or file shape; - mapping into `SourceDocument` and `SourceUnit`; -- metadata preserved by the adapter; +- metadata preserved by the module; - validation rules and failure behavior; - examples. -The Seriatim adapter doc should reference the Seriatim schema it supports and +The Seriatim input module doc should reference the Seriatim schema it supports and explain how transcript segment IDs become source-unit IDs. -### Extractor Docs Own Domains +### Stage Module Docs Own Business Logic -Each implemented extractor family should have canonical internal or integration -docs. +Each implemented stage-module family should have canonical internal or +integration docs. Likely future files: ```text -docs/internal/extractors.md +docs/internal/stage-modules.md docs/integrations/artifacts-dnd.md ``` -Extractor docs should cover: +Stage module docs should cover: -- extractor key; +- module key; +- stage; - artifact type; - schema version; - required source-reference behavior; @@ -89,13 +95,14 @@ Extractor docs should cover: - prompt and response-schema ownership; - examples. -D&D concepts should be documented in D&D extractor docs, not in generic runner -or framework docs. +D&D concepts should be documented in D&D process-module or artifact docs, not in +generic runner or framework docs. ### CLI Docs Should Reflect Extensibility -The CLI reference should present input adapters and extractors as selectable -components. +The CLI reference should present input modules, chunk modules, process modules, +merge modules, normalize modules, and output modules as selectable or +configurable components as they become user-facing. Provisional command shape: @@ -106,8 +113,9 @@ notarius extract ./source.json --input seriatim --extractors dnd.spells --output Once implemented, `docs/cli.md` should document: - positional source input path; -- input adapter selection; -- extractor selection; +- input module selection; +- process module selection; +- chunk/merge/normalize/output selection when configurable; - config path behavior; - output path behavior; - diagnostics and report behavior; @@ -117,15 +125,17 @@ Once implemented, `docs/cli.md` should document: `docs/config.md` should group fields by responsibility: -- input adapter selection and adapter-specific options; -- extractor selection and extractor-specific options; +- input module selection and module-specific options; +- chunk module selection and module-specific options; +- process module selection and module-specific options; +- merge module selection and module-specific options; +- normalize module selection and module-specific options; +- output module selection and module-specific options; - LLM runtime; - validation runtime; -- source chunking; -- output and diagnostics. +- diagnostics. -Adapter-specific and extractor-specific config should not leak into unrelated -core config sections. +Module-specific config should not leak into unrelated core config sections. ### Examples Should Stay Real @@ -154,8 +164,8 @@ When the first vertical slice is implemented, add or update: - `docs/troubleshooting.md`: common failures. - `docs/internal/overview.md`: implemented package map. - `docs/internal/pipeline.md`: implemented extraction flow. -- `docs/internal/adapters.md`: adapter contract and implemented adapters. -- `docs/internal/extractors.md`: extractor contract and built-ins. +- `docs/internal/stage-modules.md`: stage contracts and implemented modules. +- `docs/internal/input-modules.md`: input adapter contract and implemented input modules. - `docs/internal/validators.md`: validator contract and built-ins. - `docs/integrations/seriatim-transcript.md`: Seriatim input contract. - `docs/integrations/artifacts.md`: output artifact envelope. @@ -165,10 +175,10 @@ When the first vertical slice is implemented, add or update: Before merging docs, check: - Does the document describe implemented behavior outside `docs/roadmap/`? -- Are source-format details isolated to adapter docs? -- Are D&D details isolated to extractor or artifact docs? +- Are source-format details isolated to input module or integration docs? +- Are D&D details isolated to process module or artifact docs? - Is there one canonical home for the topic? - Do command examples match implemented CLI syntax? - Are examples valid, maintained, and free of secrets? -- Did any architecture, config, CLI, adapter, extractor, validator, or artifact +- Did any architecture, config, CLI, stage module, validator, or artifact contract change require a docs update? diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 5279968..694ad19 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -6,10 +6,10 @@ This is a staged implementation plan for [`2-framework-composition.md`](2-framework-composition.md). It is intended for an LLM coding agent to follow stage by stage. -This plan implements only checkpoint 2. Do not implement real input adapters, -real extractors, source chunking, LLM provider clients, prompt assets, response -schema registries, diagnostics run directories, production config loading, or -D&D artifact schemas in this pass. +This plan implements only checkpoint 2. Do not implement real input modules, +real process modules, source chunking, LLM provider clients, prompt assets, +response schema registries, diagnostics run directories, production config +loading, or D&D artifact schemas in this pass. ## Policy Context @@ -21,8 +21,8 @@ Follow: Required boundaries: - framework packages must stay source-agnostic and domain-agnostic; -- input adapter registry code must not import concrete adapter packages; -- extractor registry and runner code must not import concrete extractor +- input adapter registry code must not import concrete input module packages; +- extractor registry and runner code must not import concrete process module packages; - runner code should operate on `SourceDocument`, not transcript-specific structures; @@ -130,7 +130,7 @@ Cover: - sorted `RegisteredKeys`; - nil registry behavior. -Use fake adapters only. Do not add concrete adapter packages. +Use fake adapters only. Do not add concrete input module packages. ### Validation @@ -196,7 +196,7 @@ Cover the same cases as the input adapter registry: - sorted `RegisteredKeys`; - nil registry behavior. -Use fake extractors only. Do not add concrete extractor packages. +Use fake extractors only. Do not add concrete process module packages. ### Validation @@ -478,7 +478,8 @@ go test ./... ### Goal -Prove the extractor registry and runner compose without adding real extractors. +Prove the extractor registry and runner compose without adding real process +modules. ### Files To Add @@ -501,7 +502,7 @@ Assertions: - extractor execution follows configured key order; - approved artifacts are in deterministic order; - rejected artifacts are in deterministic order; -- no concrete adapter or extractor packages are imported. +- no concrete input module or process module packages are imported. ### Validation @@ -527,7 +528,7 @@ Check: - no package names mention transcripts, Seriatim, D&D, spells, NPCs, items, or combat; -- no concrete adapter or extractor package exists; +- no concrete input module or process module package exists; - no LLM provider code exists; - no prompt, response schema, diagnostics, config, or source chunking package was added; diff --git a/docs/roadmap/initial-architecture.md b/docs/roadmap/initial-architecture.md index 836404d..26e0cdd 100644 --- a/docs/roadmap/initial-architecture.md +++ b/docs/roadmap/initial-architecture.md @@ -11,21 +11,21 @@ Notarius should extract structured JSON artifacts from primary source inputs using modular, LLM-backed extractors. The first MVP should target audio transcripts generated by Seriatim. That -choice should be implemented as an input adapter, not as a transcript-specific -assumption in the application core. Later input sources, such as unstructured -Markdown notes or Obsidian documents, should be addable through new adapters and -extractors without reshaping the framework. +choice should be implemented as an input-stage module, not as a +transcript-specific assumption in the application core. Later input sources, +such as unstructured Markdown notes or Obsidian documents, should be addable +through new input and process modules without reshaping the framework. The first extraction domain should be D&D session analysis, starting with spell -casts. That domain should live in extractor packages and related schemas, not in -core framework packages. +casts. That domain should live in process-stage modules and related schemas, not +in core framework packages. The application should follow the same broad architecture as Audita: - deterministic core packages for config, source documents, artifacts, diagnostics, and reporting; -- input adapters that translate external source formats into a small internal source model; +- input-stage modules that translate external source formats into a small internal source model; - reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation; -- independent extractor packages that own domain-specific behavior; +- independent process-stage modules that own domain-specific behavior; - independent validator packages; - embedded prompt and JSON schema assets; - CLI orchestration that wires the pieces together without owning domain logic. @@ -36,8 +36,8 @@ artifacts rather than proposing and applying transcript corrections. ## Architectural Principles - Keep the core input model generic: ordered text units plus metadata. -- Keep source-format details in hexagonal input adapters. -- Keep extraction-domain details in extractor packages. +- Keep source-format details in hexagonal input modules. +- Keep extraction-domain details in process modules. - Treat evidence as source references, not transcript references. - Prefer narrow, useful abstractions over a universal document model. - Preserve enough provenance for validation, replay, and downstream inspection. @@ -57,12 +57,13 @@ internal/core/reporting internal/core/extractorcatalog internal/core/inputcatalog -internal/adapters/input/seriatim -internal/adapters/input/markdown - internal/framework/contracts internal/framework/extraction internal/framework/runner +internal/framework/pipeline +internal/framework/merge +internal/framework/normalize +internal/framework/output internal/framework/validators internal/framework/llm internal/framework/responseschema @@ -70,10 +71,24 @@ internal/framework/structuredoutput internal/framework/promptcontext internal/framework/warnings -internal/extractors/dnd/spells -internal/extractors/dnd/items -internal/extractors/dnd/npcs -internal/extractors/dnd/combat +internal/modules/input/seriatim +internal/modules/input/markdown + +internal/modules/chunk/generic +internal/modules/chunk/dndtranscript + +internal/modules/process/dnd/spells +internal/modules/process/dnd/items +internal/modules/process/dnd/npcs +internal/modules/process/dnd/combat + +internal/modules/merge/appendorder +internal/modules/merge/dnd/spells + +internal/modules/normalize/noop +internal/modules/normalize/dnd/spells + +internal/modules/output/json internal/validators/source_refs internal/validators/schema_validity @@ -85,9 +100,9 @@ examples docs/internal ``` -The `markdown` adapter is listed as a likely future package. The MVP should only -implement the Seriatim adapter unless a second adapter is needed to test the -boundary. +The `markdown` input module and D&D-specific chunk, merge, normalize, and +output modules are listed as likely future packages. The MVP should implement +only the stage modules needed by the checkpoint sequence. ## Core Concepts @@ -123,7 +138,7 @@ Initial source-unit assumptions: - adapter-specific metadata may carry speaker, timestamps, heading paths, page numbers, or other source details. -### Input Adapter +### Input Module / Adapter Contract Hexagonal boundary for external source formats. @@ -134,7 +149,7 @@ type InputAdapter interface { } ``` -The MVP adapter should target Seriatim minimal transcript JSON. Seriatim segment +The MVP input module should target Seriatim minimal transcript JSON. Seriatim segment fields should map as follows: - `id` becomes `SourceUnit.ID`; @@ -178,17 +193,44 @@ type Extractor interface { ArtifactType() string SchemaVersion() string Validators() []Validator - Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error) + Process(ctx context.Context, req ProcessRequest) (ProcessResult, error) } ``` -An extractor should receive either a whole source document or a source slice, -depending on runner configuration. It should return typed artifact candidates -plus warnings. It should not mutate the source document. +An extractor should receive either a whole source document or a source chunk, +depending on processing mode. It should return typed artifact candidates plus +warnings. It should not mutate the source document. -Extractor packages own domain concepts. For example, D&D spell extraction should -live under `internal/extractors/dnd/spells`; a future to-do extractor for notes -should live under a different domain path and use the same framework contract. +Process modules own domain concepts. For example, D&D spell extraction should +live under `internal/modules/process/dnd/spells`; a future to-do extractor for +notes should live under a different process-module path and use the same +framework contract. + +### Chunker + +Reusable stage contract for splitting a source document into ordered source +chunks. + +Chunking is a first-class pipeline concern because source documents may exceed a +single LLM extraction pass. Chunkers should preserve source-unit order and +produce stable chunk metadata suitable for diagnostics and replay. + +### Merger + +Reusable stage contract for combining per-chunk artifact candidates into one +merged candidate collection. + +Merge should combine outputs without doing semantic reconciliation. A generic +append-in-chunk-order merger should be sufficient for many artifact streams, +including the likely first D&D spell-cast extractor. + +### Normalizer + +Reusable stage contract for reconciling merged artifact candidates. + +Normalize is distinct from merge. Normalizers may deduplicate repeated facts, +resolve aliases, reconcile conflicting fields, check cross-chunk consistency, +or attach normalization warnings. ### Validator @@ -303,27 +345,37 @@ The architecture should support extractors outside the D&D domain. Examples: - decisions and action items from meeting transcripts; - named people, places, and dates from research notes. -These should be addable as extractor packages without changing runner, +These should be addable as process modules without changing runner, validator, source-reference, or LLM framework contracts. -## Proposed Runner Flow +## Proposed Pipeline Flow + +The application workflow should be first-class: + +```text +input -> chunk -> process -> merge -> normalize -> output +``` + +Proposed runner flow: 1. Load effective config. 2. Create diagnostics run directory. -3. Resolve the configured input adapter. +3. Resolve the configured input module through the input adapter registry. 4. Read source input. 5. Parse source input into a `SourceDocument`. 6. Validate source-document invariants. -7. Chunk source units into deterministic source slices. -8. Resolve configured extractor instances through a registry. -9. Execute extractor instances in configured order. -10. Run deterministic validators before LLM-backed validators. -11. Retain approved artifacts and rejected-artifact diagnostics. -12. Merge approved slice artifacts deterministically. -13. Serialize final output JSON. -14. Write run manifest, diagnostics, and optional report JSON. +7. Resolve the configured chunker. +8. Chunk source units into deterministic source chunks. +9. Resolve configured extractor instances through a registry. +10. Process chunks in extractor-defined mode. +11. Merge per-chunk artifact candidates deterministically. +12. Normalize merged artifact candidates. +13. Run deterministic validators before LLM-backed validators. +14. Retain approved artifacts and rejected-artifact diagnostics. +15. Serialize final output JSON. +16. Write run manifest, diagnostics, and optional report JSON. -The runner should operate on source documents and source slices only. Any +The runner should operate on source documents and source chunks only. Any transcript-specific behavior should happen before the runner, inside the input adapter, or after the runner, inside output rendering that understands source metadata. @@ -335,6 +387,7 @@ Reuse these architectural patterns: - deterministic parsing and schema validation style; - deterministic chunking of ordered source units; - explicit extractor registry; +- explicit pipeline stage contracts; - `contracts` package for transport-neutral interfaces; - OpenAI-compatible structured LLM client; - scheduler for bounded LLM concurrency; @@ -358,17 +411,18 @@ extraction-report concepts. ## Checkpoint Roadmap -The initial implementation should proceed through five coherent checkpoints. +The initial implementation should proceed through six coherent checkpoints. Each checkpoint should leave the repository in a reviewable state, with the code compiling and targeted tests covering the newly introduced contracts or behavior. 1. [Core Contracts And Skeleton](1-core-contracts-and-skeleton.md) 2. [Framework Composition](2-framework-composition.md) -3. [Portable Audita Infrastructure](3-portable-audita-infrastructure.md) -4. [Seriatim Input Adapter](4-seriatim-input-adapter.md) -5. [D&D Spells Extractor](5-dnd-spells-extractor.md) +3. [Pipeline Stages, Chunking, Merge, And Normalize](3-pipeline-stages-chunking-merge-normalize.md) +4. [Portable Audita Infrastructure](4-portable-audita-infrastructure.md) +5. [Seriatim Input Module](5-seriatim-input-module.md) +6. [D&D Spells Extractor](6-dnd-spells-extractor.md) -The first useful vertical slice should arrive at checkpoint 5: Seriatim +The first useful vertical slice should arrive at checkpoint 6: Seriatim transcript input to validated D&D spell artifact output. Earlier checkpoints are intentionally contract-first and may not produce useful user output yet. @@ -383,10 +437,11 @@ intentionally contract-first and may not produce useful user output yet. artifact metadata be allowed without source references? - Should overlapping source-reference ranges be merged, preserved exactly, or both? -- Should extraction run independently per source slice only, or should some +- Should extraction run independently per source chunk only, or should some extractors receive whole-document context? -- Should a later reconciliation stage deduplicate entities and events across - source slices? +- Which artifact types can use a generic append-in-chunk-order merger? +- Which artifact types need domain-specific normalization for deduplication, + identity resolution, or consistency? - Should LLM review be part of each extractor's validator chain or a separate review phase? - Should the Seriatim adapter accept only its minimal schema initially or also