Revise the architecture plan to reflect an input -> chunk -> process -> merge -> normalize -> output workflow

docs/policy/architecture.md

@@ -25,17 +25,27 @@ contracts that can be exercised by tests and real modules.
 
 The core framework must remain source-agnostic and domain-agnostic.
 
-Source-format details belong in input adapters. Transcript-specific concepts
+Source-format details belong in input modules. Transcript-specific concepts
 such as segments, speakers, timestamps, and transcript schemas must not spread
 into runner, extractor, or validator framework code.
 
-Extraction-domain details belong in extractor packages. D&D-specific concepts
+Extraction-domain details belong in process modules. D&D-specific concepts
 such as spells, NPCs, items, combat turns, and encounters must not spread into
 core source, runner, or LLM framework packages.
 
 Extracted facts should be grounded with source references. Source references
 should point to generic source units, not to transcript-only structures.
 
+The application workflow is:
+
+```text
+input -> chunk -> process -> merge -> normalize -> output
+```
+
+These stages should remain explicit in the architecture. Chunking, merging, and
+normalization must not be hidden inside domain process modules when they represent
+general pipeline behavior.
+
 ## Dependency Policy
 
 Prefer the Go standard library where practical.
@@ -69,16 +79,15 @@ Core deterministic model and policy:
 - `internal/core/inputcatalog`: known input adapter keys and metadata.
 - `internal/core/extractorcatalog`: known extractor keys and metadata.
 
-External source and provider adapters:
-
-- `internal/adapters/input/<name>`: source-format adapters that parse external input into core source documents.
-- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
-
 Reusable framework plumbing:
 
 - `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
 - `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output.
+- `internal/framework/pipeline`: shared pipeline-stage orchestration types, when needed.
 - `internal/framework/extraction`: shared extraction helper code.
+- `internal/framework/merge`: shared merge-stage behavior.
+- `internal/framework/normalize`: shared normalization-stage behavior.
+- `internal/framework/output`: output encoding contracts and shared helpers.
 - `internal/framework/validators`: shared validator runtime behavior and decision checks.
 - `internal/framework/llm`: LLM runtime, scheduling, and provider adapters.
 - `internal/framework/responseschema`: embedded structured-output schema registry.
@@ -88,35 +97,62 @@ Reusable framework plumbing:
 
 Domain implementations:
 
-- `internal/extractors/<domain>/<extractor>`: domain-specific extractor packages.
+- `internal/modules/input/<name>`: input-stage modules that parse external input into core source documents.
+- `internal/modules/chunk/<name>`: chunk-stage modules.
+- `internal/modules/process/<domain>/<name>`: process-stage extractor modules.
+- `internal/modules/merge/<name>` or `internal/modules/merge/<domain>/<name>`: merge-stage modules.
+- `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
+- `internal/modules/output/<name>`: output-stage modules.
 - `internal/validators/<validator>`: built-in validator implementations.
 - `internal/prompts`: embedded prompt assets and prompt metadata registry.
+- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
 
 Package-private implementation constants may live near the package that owns
 them, preferably in `constants.go` when useful.
 
-## Input Adapters
+## Stage Modules
+
+Concrete business logic should live under `internal/modules/<stage>/...`.
+
+Stage-oriented module layout is preferred because it makes the application
+workflow visible in the filesystem:
+
+```text
+internal/modules/input/...
+internal/modules/chunk/...
+internal/modules/process/...
+internal/modules/merge/...
+internal/modules/normalize/...
+internal/modules/output/...
+```
+
+Use short, lowercase, idiomatic Go package names. Prefer names such as
+`dndtranscript`, `appendorder`, and `spells` over names like `dnd_transcript`,
+`serial_merge`, or `spell_extractor` that repeat parent-stage context.
+
+## Input Modules
 
 Use a hexagonal architecture style for source input.
 
-Input adapters translate external source formats into the core source model.
-Adapters may know about external schema details, source-specific metadata, and
+Input modules translate external source formats into the core source model.
+They may know about external schema details, source-specific metadata, and
 format-specific validation rules. They should not own extraction-domain
 decisions.
 
 Other packages should interact with source input through adapter contracts and
-core source types. Adapter implementation details and external dependency types
-must not leak into framework or extractor packages.
+core source types. Input module implementation details and external dependency
+types must not leak into framework or process module packages.
 
-Adapter metadata may preserve source-specific facts such as transcript speaker,
+Input module metadata may preserve source-specific facts such as transcript speaker,
 timestamps, Markdown heading path, page number, or block ID. Framework code may
 carry metadata through, but should not require a specific adapter's metadata
 shape.
 
 ## Extractors
 
-Extractors are independent modules that produce one kind of structured artifact.
-Each extractor package owns:
+Extractors are independent modules that process source chunks or whole source
+documents and produce one kind of structured artifact candidate.
+Each process module owns:
 
 - its artifact semantics;
 - its prompt usage;
@@ -124,8 +160,13 @@ Each extractor package owns:
 - its validator chain;
 - any domain-specific mapping or interpretation.
 
-Extractors should depend on framework contracts and core source/artifact types.
-They should not depend on concrete input adapter packages.
+Process modules should depend on framework contracts and core source/artifact
+types. They should not depend on concrete input module packages.
+
+Extractors should not be the only place where chunking, merging, or
+normalization happens. They may choose processing mode or provide domain-specific
+merge/normalization behavior when generic behavior is insufficient, but the
+pipeline stages themselves are framework concepts.
 
 The runner should be able to compose, skip, resume, or run individual extractors
 when their prerequisites are satisfied. Ordering should be explicit through
@@ -134,6 +175,31 @@ configuration, a default sequence, or documented orchestration rules.
 Extractor selection must go through a registry or equivalent mechanism rather
 than scattered conditionals.
 
+## Pipeline Stages
+
+The pipeline has six conceptual stages:
+
+1. input: external source material becomes a `SourceDocument`;
+2. chunk: a `SourceDocument` becomes ordered source chunks;
+3. process: extractors produce artifact candidates from chunks or whole documents;
+4. merge: per-chunk candidates become a merged candidate collection;
+5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
+6. output: final artifacts are serialized.
+
+Chunking is first-class because source documents may be too large for a single
+LLM pass. Chunkers should preserve source-unit order and produce stable chunk
+metadata.
+
+Merge and normalize are separate concerns. Merge combines per-chunk results into
+a deterministic collection. Normalize performs semantic reconciliation after
+merge. Generic append-in-chunk-order merge and no-op normalization should be
+available for simple artifact types, while domain-specific behavior can be
+provided where needed.
+
+The framework should allow serial and parallel chunk processing. The first
+implementation may execute chunks serially for determinism, but contracts should
+not prevent later parallel execution.
+
 ## Validators
 
 Validators should be independently testable and composable.
@@ -181,8 +247,8 @@ Configuration files should not contain raw secrets unless the application is
 explicitly designed for that. Prefer environment variables or secret files for
 secrets.
 
-Adapter-specific and extractor-specific configuration should remain grouped by
-the adapter or extractor that owns it.
+Stage-module-specific configuration should remain grouped by the module that
+owns it.
 
 ## Embedded Assets
 
@@ -242,12 +308,12 @@ focused on implemented behavior. Put future, planned, or aspirational work only
 under `docs/roadmap/`.
 
 Core documentation should use generic terms such as source document, source
-unit, source reference, input adapter, extractor, artifact, validator, and run
-manifest.
+unit, source reference, input adapter, extractor, chunker, merger, normalizer,
+artifact, validator, and run manifest.
 
-Source-format details belong in adapter or integration docs. Domain-specific
-extraction details belong in extractor or artifact docs.
+Source-format details belong in input module or integration docs.
+Domain-specific extraction details belong in process module or artifact docs.
 
-When changing architecture, config, CLI behavior, adapters, extractor contracts,
-validator contracts, LLM runtime behavior, or artifact schemas, update the
-relevant docs and examples in the same change.
+When changing architecture, config, CLI behavior, stage modules, extractor
+contracts, validator contracts, LLM runtime behavior, or artifact schemas, update
+the relevant docs and examples in the same change.