Consolidate the architecture plan into fewer packages

docs/policy/architecture.md

@@ -35,6 +35,10 @@ core source, runner, or LLM framework packages.
 
 Extracted facts should be grounded with source references. Source references
 should point to generic source units, not to transcript-only structures.
+Artifact records should require source references by default unless their schema
+explicitly opts into ungrounded fields. Generic pipeline code should preserve
+source-reference ranges exactly and should not merge or rewrite overlapping
+ranges.
 
 The application workflow is:
 
@@ -70,30 +74,17 @@ CLI and executable entrypoint:
 
 Core deterministic model and policy:
 
-- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation.
 - `internal/core/source`: source document, source unit, and source reference types.
-- `internal/core/sourcechunking`: deterministic chunking of ordered source units.
 - `internal/core/artifacts`: artifact envelope, artifact candidates, rejected artifacts, and manifests.
-- `internal/core/diagnostics`: run directories and diagnostics artifact paths.
-- `internal/core/reporting`: process reports and report serialization.
-- `internal/core/inputcatalog`: known input adapter keys and metadata.
-- `internal/core/extractorcatalog`: known extractor keys and metadata.
+- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation, once config exists.
 
 Reusable framework plumbing:
 
 - `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
-- `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output.
-- `internal/framework/pipeline`: shared pipeline-stage orchestration types, when needed.
-- `internal/framework/extraction`: shared extraction helper code.
-- `internal/framework/merge`: shared merge-stage behavior.
-- `internal/framework/normalize`: shared normalization-stage behavior.
-- `internal/framework/output`: output encoding contracts and shared helpers.
-- `internal/framework/validators`: shared validator runtime behavior and decision checks.
-- `internal/framework/llm`: LLM runtime, scheduling, and provider adapters.
-- `internal/framework/responseschema`: embedded structured-output schema registry.
-- `internal/framework/structuredoutput`: structured-output parsing and malformed-response handling.
-- `internal/framework/promptcontext`: source-document prompt rendering helpers.
-- `internal/framework/warnings`: shared warning records.
+- `internal/framework/pipeline`: runner, pipeline-stage orchestration, registries, and small shared stage helpers.
+- `internal/framework/validate`: shared validator runtime behavior and decision checks.
+- `internal/framework/llm`: LLM clients, scheduling, structured-output parsing, and response-schema registry, once LLM runtime exists.
+- `internal/framework/prompt`: embedded prompt assets, prompt registry, and prompt rendering helpers, once prompt assets exist.
 
 Domain implementations:
 
@@ -104,12 +95,17 @@ Domain implementations:
 - `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
 - `internal/modules/output/<name>`: output-stage modules.
 - `internal/validators/<validator>`: built-in validator implementations.
-- `internal/prompts`: embedded prompt assets and prompt metadata registry.
 - `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
 
 Package-private implementation constants may live near the package that owns
 them, preferably in `constants.go` when useful.
 
+Start with fewer, larger framework packages. Split a package only when a real
+boundary proves itself through import direction, ownership, test seams, or
+substantial file size. Do not create catalog, diagnostics, reporting,
+structured-output, response-schema, output, merge, normalize, extraction, or
+warnings packages merely because the concepts exist in the architecture.
+
 ## Stage Modules
 
 Concrete business logic should live under `internal/modules/<stage>/...`.
@@ -143,10 +139,14 @@ Other packages should interact with source input through adapter contracts and
 core source types. Input module implementation details and external dependency
 types must not leak into framework or extract module packages.
 
-Input module metadata may preserve source-specific facts such as transcript speaker,
-timestamps, Markdown heading path, page number, or block ID. Framework code may
-carry metadata through, but should not require a specific adapter's metadata
-shape.
+Input module metadata may preserve source-specific facts such as transcript
+speaker, timestamps, Markdown heading path, page number, or block ID. Framework
+code may carry metadata through, but should not require a specific adapter's
+metadata shape.
+
+Core source metadata should remain `map[string]any`. Document well-known keys
+as conventions, and let input modules expose typed accessor helpers for their
+own metadata when useful.
 
 ## Extractors
 
@@ -163,6 +163,11 @@ Each extract module owns:
 Extract modules should depend on framework contracts and core source/artifact
 types. They should not depend on concrete input module packages.
 
+Extraction requests should carry the active source chunk plus optional ambient
+context, such as a document synopsis, prior-chunk summaries, known entities, or
+other module-provided state. The context may be empty for simple modules, but
+the contract should not assume extraction is always chunk-local.
+
 Extractors should not be the only place where chunking, merging, or
 normalization happens. They may choose processing mode or provide domain-specific
 merge/normalization behavior when generic behavior is insufficient, but the
@@ -200,6 +205,15 @@ The framework should allow serial and parallel chunk processing. The first
 implementation may execute chunks serially for determinism, but contracts should
 not prevent later parallel execution.
 
+Final durable output should use one artifact file per artifact type plus a
+run-level manifest/index file. Framework artifact flow should use a generic
+envelope with `json.RawMessage` payloads; extract modules should own typed Go
+structs at their boundaries and encode into that generic envelope before
+returning to framework code.
+
+Schemas should be versioned per extractor, with a separate envelope/manifest
+format version.
+
 ## Validators
 
 Validators should be independently testable and composable.
@@ -209,7 +223,11 @@ present. Validator decision semantics should be explicit: each candidate
 artifact should receive exactly one decision from each validator that evaluates
 it.
 
-Shared validator runtime mechanics belong under `internal/framework/validators`.
+LLM-backed review belongs in module-owned validator chains, not in a separate
+global review phase. Extract modules and normalize modules may both use
+deterministic and LLM-backed validators.
+
+Shared validator runtime mechanics belong under `internal/framework/validate`.
 Concrete validator behavior belongs under `internal/validators/<validator>`.
 
 ## LLM Runtime
@@ -296,6 +314,12 @@ and LLM clients where practical.
 Contract-first work should include fake implementations that prove interfaces
 compose before real input modules or extract modules depend on them.
 
+Once the pipeline-stage contracts exist, maintain a fixture-driven walking
+skeleton that exercises input, chunk, extract, merge, normalize, and output
+stages with fake modules and fake external clients. This test should protect
+stage composition continuously while real modules are introduced over later
+checkpoints.
+
 Config examples should be load-tested once config files exist. Important CLI
 workflows should have parser or command tests. Adapter, extractor, and validator
 contracts should have focused tests that do not require running the full