Add developer and internal architecture documentation

docs/internal/pipeline.md

@@ -0,0 +1,127 @@
+# Pipeline Internals
+
+The implemented pipeline runner lives in `internal/framework/pipeline`. It
+executes the fixed workflow defined by the architecture policy:
+
+```text
+input -> chunk -> extract -> merge -> normalize -> output
+```
+
+Pipeline execution is serial. The runner executes the resolved lanes one after
+another in the fixed workflow order.
+
+## Profile Resolution
+
+Config loading produces `pipeline.PipelineProfile` values. Resolution happens
+before execution:
+
+1. `internal/core/config.Config.Resolve` validates config and finds the named
+   pipeline.
+2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
+3. Module bindings are defaulted:
+   - chunk: `generic`
+   - merge: `appendorder`
+   - normalize: `noop`
+   - output: `json`
+   - LLM profile: `default`
+4. The module catalog is checked for each bound module key.
+5. Module capabilities are checked in workflow order.
+6. A digest is calculated from the resolved pipeline without the digest field.
+
+The CLI writes the resolved pipeline and digest to diagnostics.
+
+## Registries And Module Specs
+
+`pipeline.Registries` holds concrete constructors for execution. A
+`pipeline.ModuleCatalog` exposes module specs for config validation and
+resolution.
+
+Every production module registers a `ModuleSpec` with:
+
+- `Key`: module key used in config;
+- `Stage`: module kind such as input, chunk, extract, merge, normalize,
+  validate, or output;
+- `Provides`: capabilities added after that module runs;
+- `Requires`: capabilities that must already be available.
+
+Capability checks prevent incompatible pipeline composition before a run starts.
+
+## Runner Input And Output
+
+`pipeline.RunInput` carries:
+
+- a `ResolvedPipeline`;
+- optional source ID, input path, and raw input bytes;
+- a structured LLM client;
+- run ID, start time, LLM profile manifest metadata, and CLI metadata.
+
+`pipeline.RunOutput` carries:
+
+- run manifest;
+- approved artifacts;
+- rejected artifacts;
+- warnings;
+- logical output files returned by the output encoder.
+
+The CLI owns durable file writes and diagnostics writes after the runner returns.
+
+## Execution
+
+The runner:
+
+1. validates run input and registries;
+2. builds the input adapter and parses the raw input into a source document;
+3. validates the source document;
+4. builds the chunker and produces source chunks;
+5. runs each selected artifact lane in sorted resolved order;
+6. builds the output encoder and validates logical output file names.
+
+Within an artifact lane, the runner:
+
+1. builds the extractor, merger, and normalizer;
+2. records module manifest metadata when modules provide it;
+3. extracts candidates from each chunk;
+4. normalizes candidate envelope fields such as index, extractor key, artifact
+   type, and schema version;
+5. merges candidates;
+6. normalizes merged candidates;
+7. validates candidate envelope consistency;
+8. runs validators;
+9. converts approved candidates to artifacts.
+
+## Validators
+
+If a lane declares validators in config, the runner builds those validators from
+the validator registry. Otherwise it uses validators returned by the extractor.
+
+Each validator must return exactly one decision for each eligible candidate. The
+runner enforces decision cardinality with `internal/framework/validate`.
+Rejected candidates are removed before the next validator runs. Approved
+candidates continue through the chain.
+
+The production CLI currently registers no standalone validator modules. The
+current D&D spell extractor supplies deterministic shape and source-reference
+validators.
+
+## Warnings And Failures
+
+Warnings from chunking, extraction, merging, normalization, validation, and
+output encoding are accumulated in `RunOutput.Warnings`.
+
+Errors wrap the operation and module key or lane context. If execution fails
+after a manifest exists, the returned manifest is marked `failed` and receives a
+completion timestamp.
+
+On successful execution, the manifest validation status is:
+
+- `approved` when no candidates were rejected;
+- `rejected` when at least one candidate was rejected.
+
+## Manifest Population
+
+The manifest records run ID, pipeline ID, pipeline digest, module keys, artifact
+lanes, LLM profile metadata, source digest, validation status, and timing.
+
+Modules can add non-secret manifest metadata by implementing
+`contracts.ManifestMetadataProvider`. The D&D spell extractor uses this for
+prompt and response-schema provenance.