Update feature roadmap to include a named pipeline profile configuration model

docs/policy/architecture.md

@@ -250,12 +250,61 @@ The goal is to make configuration discoverable and avoid implicit or hidden
 operational values. User-visible defaults and cross-package operational defaults
 should be defined in config code.
 
-Unless documented otherwise, precedence is:
+Configuration should be organized around named pipeline profiles. A pipeline is
+a fixed-shape template for the application workflow, not a free-form DAG or
+general workflow program. The six-stage flow remains fixed:
 
-1. CLI flags
-2. environment variables
-3. configuration file
-4. built-in defaults
+```text
+input -> chunk -> extract -> merge -> normalize -> output
+```
+
+A pipeline profile should bind registered modules to those stage slots:
+
+- one shared input module;
+- one shared chunk module by default;
+- one or more artifact lanes, each with extract, merge, normalize, and
+  validator behavior;
+- one output module.
+
+The MVP should use one shared chunk module per pipeline. Per-lane chunk
+overrides are a future extension and should be added only if a real artifact
+lane needs different chunking.
+
+The CLI should select a named pipeline by ID, such as
+`notarius run dnd-session --input session.json`. Structural module selection
+should come from configuration, not ad hoc CLI flags. CLI flags may select a
+subset of configured artifact lanes, such as `--only spells,npcs`, and may
+override operational settings such as model, concurrency, output directory, or
+diagnostics directory.
+
+Pipeline definitions should support compact defaults:
+
+- `chunk`: `generic`;
+- lane `merge`: `appendorder`;
+- lane `normalize`: `noop`;
+- `output`: `json`;
+- `llm_profile`: `default` where an LLM profile is needed.
+
+Module bindings should support both string shorthand and object form. For
+example, `extract: dnd/spells` and
+`extract: {module: dnd/spells, llm_profile: fast}` should normalize to the same
+internal binding type.
+
+Module registries should expose module metadata, including flat string
+capabilities, without requiring module construction. Config validation should
+fail fast on unknown module keys, unknown pipeline IDs, missing required slots,
+missing capabilities, unknown LLM profiles, empty artifact-lane sets, or
+`--only` lane names that do not exist in the selected pipeline.
+
+Keep capabilities as a flat string set. Do not evolve capabilities into a type
+system unless real module interactions prove the need.
+
+Unless documented otherwise, precedence from lowest to highest is:
+
+1. built-in defaults
+2. configuration file
+3. environment variables
+4. CLI flags
 
 Prefer YAML configuration unless the project has a strong reason to use another
 format. Config files should be discoverable at
@@ -265,8 +314,15 @@ Configuration files should not contain raw secrets unless the application is
 explicitly designed for that. Prefer environment variables or secret files for
 secrets.
 
-Stage-module-specific configuration should remain grouped by the module that
-owns it.
+Stage-module-specific configuration should remain inline with the pipeline slot
+that owns it. Do not add named module instances until repeated inline settings
+create real drift or duplication. LLM profiles are the justified top-level
+exception because model settings are cross-cutting.
+
+The run manifest should record the selected `pipeline_id` and a digest of the
+resolved pipeline definition after defaults and lane selection are applied.
+`pipeline_id` alone is not sufficient provenance because a named pipeline can
+change over time.
 
 ## Embedded Assets
 

docs/roadmap/2-framework-composition.md

@@ -28,6 +28,8 @@ Out of scope:
 - real input parsing;
 - real input modules;
 - real extract modules;
+- pipeline-profile config loading;
+- module capability validation;
 - real LLM calls;
 - prompt assets;
 - response schema assets;
@@ -50,6 +52,10 @@ The runner should operate on already parsed source documents in this checkpoint.
 Raw input parsing and concrete input module behavior remain deferred to the
 Seriatim input module checkpoint.
 
+Pipeline-profile resolution, module metadata, and capability validation are
+deferred to checkpoint 3. This checkpoint only needs constructor registries and
+minimal runner composition.
+
 Implementation staging belongs in
 [`implementation.md`](implementation.md).
 

docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md

@@ -32,6 +32,11 @@ In scope:
 - merge-stage contract;
 - normalize-stage contract;
 - output-stage contract and fake output encoder for pipeline completeness;
+- resolved pipeline definition types for a fixed-shape pipeline template;
+- module binding and module metadata types, including flat capability strings;
+- default application for `chunk`, lane `merge`, lane `normalize`, `output`,
+  and `llm_profile`;
+- lane selection behavior equivalent to future `--only`;
 - runner/pipeline updates that exercise these stages with fake components;
 - generic append/chronological merge behavior for artifact candidates when
   appropriate;
@@ -47,6 +52,7 @@ Out of scope:
 - prompt assets;
 - response schema assets;
 - diagnostics run directory;
+- production config file loading;
 - real CLI command behavior;
 - production output serialization or durable output writing.
 
@@ -61,15 +67,26 @@ The repository should contain explicit pipeline-stage contracts:
 - `Normalizer`: merged candidates to normalized candidates.
 - `OutputEncoder`: final artifact bundle to bytes.
 
+The repository should also contain a resolved pipeline model that represents:
+
+- `pipeline_id`;
+- shared input binding;
+- shared chunk binding;
+- selected artifact lanes;
+- lane extract, merge, normalize, and validator bindings;
+- output binding;
+- resolved defaults;
+- resolved pipeline digest input.
+
 The runner should orchestrate fake implementations through chunk, extract,
 merge, normalize, and approval/validation behavior in tests.
 
 The checkpoint should include one fixture-driven integration test that starts
 from fixture input bytes and ends at encoded output bytes. The fixture should
-use a fake input adapter, deterministic chunker, trivial extractor, fake
-structured LLM client, generic merger, no-op normalizer, and fake or minimal
-JSON output encoder. This is a contract exercise, not a useful user-facing
-workflow.
+use an in-memory pipeline profile with fake input adapter, deterministic
+chunker, trivial extractor, fake structured LLM client, generic merger, no-op
+normalizer, and fake or minimal JSON output encoder. This is a contract
+exercise, not a useful user-facing workflow.
 
 The walking skeleton should live in `internal/framework/pipeline`, with
 fixtures under that package's `testdata/`. If the CLI has an extract command by
@@ -146,10 +163,17 @@ The fake LLM client should be part of the test setup so the contract is
 exercised without introducing provider code, prompt assets, or response schema
 assets.
 
+The walking skeleton should validate module keys and flat capability
+requirements before execution. This should use registry metadata rather than
+constructing modules. Keep capability values as simple strings.
+
 ## Done Criteria
 
 - `go test ./...` passes.
 - Pipeline-stage contracts are explicit and source/domain agnostic.
+- A resolved pipeline profile model exists for fixed-shape pipeline templates.
+- Pipeline defaults and lane selection are covered by fake tests.
+- Module capability validation is covered by fake tests.
 - Fake tests prove input source documents can be chunked, extracted, merged, and
   normalized.
 - A fixture-driven walking skeleton proves fake input, chunk, extract, merge,
@@ -165,6 +189,10 @@ assets.
 - Is the workflow clearly represented as input, chunk, extract, merge,
   normalize, and output?
 - Are merge and normalize cleanly separated?
+- Does the resolved pipeline model avoid becoming a general-purpose workflow
+  engine?
+- Are module capabilities simple flat strings?
+- Does lane selection avoid creating ad hoc pipelines?
 - Can a generic merger handle simple chronological artifact streams?
 - Can a later domain-specific normalizer handle duplicates and consistency
   without changing core runner contracts?

docs/roadmap/4-portable-audita-infrastructure.md

@@ -21,7 +21,8 @@ In scope:
 - prompt registry pattern;
 - response-schema registry pattern;
 - diagnostics run directory pattern;
-- minimal config structs and defaults for implemented runtime pieces.
+- config loading and validation for named pipeline profiles and implemented
+  runtime pieces.
 
 Out of scope:
 
@@ -31,6 +32,7 @@ Out of scope:
 - correction ledger terminology;
 - Audita module or validator behavior;
 - real D&D prompts or schemas unless needed as inert registry tests.
+- embedded built-in pipeline profiles.
 
 ## Proposed Stages
 
@@ -82,28 +84,55 @@ Initial diagnostics should cover:
 
 - invocation metadata;
 - redacted effective config;
+- selected pipeline ID;
+- resolved pipeline definition and digest;
 - source document artifact;
 - run report placeholder;
 - error log on failure.
 
 Avoid Audita-specific artifact names such as correction ledger.
 
-### Stage 5: Minimal Runtime Config
+### Stage 5: Pipeline Profile Config
 
-Add config structs and defaults only for infrastructure that now exists.
+Add config structs, loading, defaults, and validation for named pipeline
+profiles.
 
 Initial config areas:
 
-- input module key;
-- extractor keys;
-- primary LLM settings;
-- validation LLM settings if needed;
+- `llm_profiles`;
+- `pipelines.<pipeline_id>.input`;
+- `pipelines.<pipeline_id>.chunk`;
+- `pipelines.<pipeline_id>.artifacts.<lane>.extract`;
+- lane `merge`, `normalize`, and validator settings;
+- output module selection;
+- inline module-binding object form and string shorthand;
+- default `chunk`, `merge`, `normalize`, `output`, and `llm_profile`;
+- selected pipeline ID and lane filtering for runtime use;
 - concurrency;
 - work directory;
 - diagnostics retention.
 
-Config loading can remain minimal unless the implementation needs full file/env
-precedence at this checkpoint.
+Config loading should support the standard precedence model:
+
+1. built-in defaults
+2. configuration file
+3. environment variables
+4. CLI flags
+
+Structural module selection should come from pipeline config. CLI flags may
+override operational settings and artifact lane selection, but should not offer
+ad hoc `--extractor` or `--chunker` wiring.
+
+Add validation for unknown pipeline IDs, unknown module keys, missing required
+slots, missing capabilities, unknown LLM profiles, empty artifact-lane sets, and
+invalid lane selections.
+
+If the CLI shell is ready, add:
+
+```sh
+notarius config validate
+notarius pipelines list
+```
 
 ## Done Criteria
 
@@ -113,10 +142,13 @@ precedence at this checkpoint.
   ledger code has been copied.
 - Runtime tests cover secret redaction, schema registry lookup, prompt metadata,
   and scheduler behavior where applicable.
+- Config tests cover named pipeline profiles, defaults, lane selection,
+  capability validation, and resolved pipeline digesting.
 
 ## Review Questions
 
 - Did we copy only reusable infrastructure?
 - Do provider-specific types stay behind adapter/runtime boundaries?
 - Are diagnostics names and report concepts extraction-oriented?
-- Is config limited to implemented behavior?
+- Is config limited to named pipeline profiles and implemented behavior?
+- Are structural pipeline changes kept out of ad hoc CLI flags?

docs/roadmap/5-seriatim-input-module.md

@@ -21,8 +21,10 @@ In scope:
 - mapping into `SourceDocument` and `SourceUnit`;
 - source-document validation;
 - input adapter registry wiring;
+- module metadata/capabilities for pipeline validation;
 - fixtures and tests;
-- CLI/config path to select the input module if the CLI shell exists.
+- config path to use the input module through a named pipeline profile if
+  config loading exists.
 
 Out of scope:
 
@@ -81,13 +83,19 @@ The resulting `SourceDocument` should pass core source validation.
 
 Register the module under a stable input adapter key, likely `seriatim`.
 
-If CLI support exists, add provisional selection:
+Declare module metadata for pipeline validation. Initial provided capabilities
+should include transcript-oriented metadata such as `speaker` and `timestamps`
+if those fields are preserved from Seriatim input.
+
+If config and CLI support exist, add a minimal pipeline-profile fixture or test
+config using the Seriatim input module:
 
 ```sh
-notarius extract ./transcript.json --input seriatim
+notarius run dnd-session --input ./transcript.json --only spells
 ```
 
-The command may still use fake extractors until checkpoint 6.
+The command may still use fake extract, merge, normalize, and output modules
+until checkpoint 6.
 
 ### Fixtures And Tests
 
@@ -105,7 +113,9 @@ Add fixtures and tests for:
 - `go test ./...` passes.
 - Seriatim minimal transcript JSON maps into `SourceDocument`.
 - Transcript fields do not appear in core runner contracts.
-- The input module is selectable through the registry.
+- The input module is selectable through the registry and pipeline-profile
+  configuration when config support exists.
+- The input module declares capabilities needed for pipeline validation.
 - Tests prove transcript-specific assumptions are isolated to the input module.
 
 ## Review Questions

docs/roadmap/6-dnd-spells-extractor.md

@@ -20,6 +20,7 @@ In scope:
 - structured response schema asset;
 - prompt assets;
 - `internal/modules/extract/dnd/spells`;
+- module metadata/capability requirements for pipeline validation;
 - source-reference and schema validators in the extractor chain;
 - fake LLM tests;
 - CLI-level integration test if the CLI path is ready.
@@ -81,6 +82,7 @@ Implement `internal/modules/extract/dnd/spells`.
 The extractor should:
 
 - satisfy the framework `Extractor` contract;
+- declare module metadata for pipeline-profile validation;
 - build LLM messages from a source document or source chunk;
 - call the structured LLM client;
 - return artifact candidates with source references;
@@ -107,10 +109,11 @@ Add tests using a fake structured LLM client:
 If the CLI path is ready, add an end-to-end test using:
 
 ```sh
-notarius extract ./transcript.json --input seriatim --extractors dnd.spells --output ./artifacts.json
+notarius run dnd-session --input ./transcript.json --only spells
 ```
 
-The test should use fake LLM wiring and fixture input.
+The test should use fake LLM wiring, fixture input, and a named pipeline profile
+with a `spells` artifact lane.
 
 ## Done Criteria
 
@@ -118,6 +121,8 @@ The test should use fake LLM wiring and fixture input.
 - Seriatim input can flow through the runner into the D&D spells extractor.
 - Spell artifacts include valid source references.
 - D&D concepts are contained in extract module/artifact packages and docs.
+- The spells module can be selected as a named artifact lane in pipeline
+  configuration.
 - The first meaningful vertical slice is available through tests, and through
   CLI if the CLI path is ready.
 

docs/roadmap/documentation.md

@@ -100,42 +100,54 @@ generic runner or framework docs.
 
 ### CLI Docs Should Reflect Extensibility
 
-The CLI reference should present input modules, chunk modules, extract modules,
-merge modules, normalize modules, and output modules as selectable or
-configurable components as they become user-facing.
+The CLI reference should present named pipeline profiles as the primary
+user-facing abstraction. Individual stage modules should be visible through
+pipeline configuration and discovery commands, not through ad hoc structural
+run flags.
 
 Provisional command shape:
 
 ```sh
-notarius extract ./source.json --input seriatim --extractors dnd.spells --output ./artifacts.json
+notarius run dnd-session --input ./source.json
+notarius run dnd-session --input ./source.json --only spells,npcs
+notarius config validate
+notarius pipelines list
 ```
 
 Once implemented, `docs/cli.md` should document:
 
-- positional source input path;
-- input module selection;
-- extract module selection;
-- chunk/merge/normalize/output selection when configurable;
+- pipeline ID selection;
+- required input path flags;
+- `--only` artifact-lane selection;
 - config path behavior;
+- operational overrides such as output path, model, concurrency, and diagnostics
+  directory;
 - output path behavior;
 - diagnostics and report behavior;
 - exit codes.
 
 ### Config Docs Should Separate Framework And Plugin-Like Options
 
-`docs/config.md` should group fields by responsibility:
+`docs/config.md` should describe named pipeline profiles and the resolved
+pipeline model.
 
-- input module selection and module-specific options;
-- chunk module selection and module-specific options;
-- extract module selection and module-specific options;
-- merge module selection and module-specific options;
-- normalize module selection and module-specific options;
-- output module selection and module-specific options;
-- LLM runtime;
-- validation runtime;
-- diagnostics.
+It should cover:
 
-Module-specific config should not leak into unrelated core config sections.
+- config file locations and precedence;
+- `llm_profiles`;
+- `pipelines.<pipeline_id>.input`;
+- `pipelines.<pipeline_id>.chunk`;
+- `pipelines.<pipeline_id>.artifacts.<lane>.extract`;
+- lane `merge`, `normalize`, and validator settings;
+- output module selection;
+- string shorthand versus inline module-binding object form;
+- defaults for omitted slots;
+- capability validation;
+- pipeline digest and manifest provenance.
+
+Module-specific config should stay inline with the pipeline slot that owns it.
+Top-level named module instances should not be introduced until repeated inline
+settings create real drift. `llm_profiles` are the cross-cutting exception.
 
 ### Examples Should Stay Real
 
@@ -148,6 +160,7 @@ Likely future examples:
 examples/seriatim-minimal-transcript.json
 examples/minimal-config.yml
 examples/dnd-spells.artifacts.json
+examples/dnd-session.config.yml
 ```
 
 Examples should be secret-free and should use the same command shapes documented
@@ -179,6 +192,8 @@ Before merging docs, check:
 - Are D&D details isolated to extract module or artifact docs?
 - Is there one canonical home for the topic?
 - Do command examples match implemented CLI syntax?
+- Do config examples use named pipeline profiles rather than ad hoc module
+  flags?
 - Are examples valid, maintained, and free of secrets?
 - Did any architecture, config, CLI, stage module, validator, or artifact
   contract change require a docs update?

docs/roadmap/initial-architecture.md

@@ -303,6 +303,8 @@ Per-run provenance record.
 ```go
 type RunManifest struct {
     EnvelopeVersion  string   `json:"envelope_version"`
+    PipelineID       string   `json:"pipeline_id"`
+    PipelineDigest   string   `json:"pipeline_digest"`
     InputModule      string   `json:"input_module"`
     Chunker          string   `json:"chunker"`
     SourceDigests    []string `json:"source_digests"`
@@ -316,8 +318,8 @@ type RunManifest struct {
 ```
 
 The manifest should eventually include model names, prompt IDs, prompt hashes,
-response schema versions, config source, started/completed timestamps, and
-diagnostics paths.
+response schema versions, config source, redacted resolved config digest,
+started/completed timestamps, and diagnostics paths.
 
 ## Initial Extractor Targets
 
@@ -393,6 +395,107 @@ The architecture should support extractors outside the D&D domain. Examples:
 These should be addable as extract modules without changing runner,
 validator, source-reference, or LLM framework contracts.
 
+## Configuration Model
+
+Notarius should use named pipeline profiles selected by ID at the CLI. A
+pipeline is a fixed-shape template for the known application workflow, not a
+free-form list of steps:
+
+```text
+input -> chunk -> extract -> merge -> normalize -> output
+```
+
+A pipeline profile should define one shared front end and one or more artifact
+lanes:
+
+- shared input module;
+- shared chunk module by default;
+- artifact lanes containing extract, merge, normalize, and validator behavior;
+- shared output module.
+
+The MVP should use one shared chunk module per pipeline. Per-lane chunk
+overrides can be added later if an artifact lane, such as combat, proves it
+needs a different chunking strategy.
+
+Example shape:
+
+```yaml
+llm_profiles:
+  default:
+    model: example-model
+    max_concurrency: 4
+
+pipelines:
+  dnd-session:
+    input: seriatim
+    chunk: dnd/transcript
+    artifacts:
+      spells:
+        extract: dnd/spells
+        normalize: dnd/spells
+      npcs:
+        extract: dnd/npcs
+      items:
+        extract: dnd/items
+```
+
+The CLI should run named pipelines:
+
+```sh
+notarius run dnd-session --input session-014.json
+notarius run dnd-session --input session-014.json --only spells,npcs
+```
+
+`--only` should select configured artifact lanes. It should not create an
+ad hoc pipeline. Structural module selection should come from config, while CLI
+flags may override operational knobs such as model, concurrency, output
+directory, and diagnostics directory.
+
+Initial defaults:
+
+- `chunk`: `generic`;
+- lane `merge`: `appendorder`;
+- lane `normalize`: `noop`;
+- `output`: `json`;
+- `llm_profile`: `default` where an LLM profile is needed.
+
+Module bindings should support both string shorthand and object form:
+
+```yaml
+extract: dnd/spells
+```
+
+```yaml
+extract:
+  module: dnd/spells
+  llm_profile: fast
+  prompt_version: v1
+```
+
+Both forms should normalize into a single internal `ModuleBinding` shape before
+validation and manifest hashing.
+
+Pipeline validation should use module metadata declared through registries.
+Modules should expose flat string capability metadata, such as `speaker` or
+`timestamps`, without requiring module construction. Config validation should
+fail fast for:
+
+- unknown pipeline IDs;
+- unknown module keys;
+- missing required slots;
+- missing required capabilities;
+- unknown LLM profiles;
+- empty artifact-lane sets;
+- `--only` lane names that do not exist in the selected pipeline.
+
+The MVP should keep pipelines config-file-only. Built-in pipeline profiles can
+be added later if the project needs embedded defaults, but that introduces
+merge/override semantics that the MVP does not need.
+
+The resolved pipeline definition should be hashed after defaults and lane
+selection are applied. The run manifest should record both `pipeline_id` and
+`pipeline_digest`; a pipeline ID alone is not stable provenance.
+
 ## Proposed Pipeline Flow
 
 The application workflow should be first-class:
@@ -404,21 +507,22 @@ input -> chunk -> extract -> merge -> normalize -> output
 Proposed runner flow:
 
 1. Load effective config.
-2. Create diagnostics run directory.
-3. Resolve the configured input module through the input adapter registry.
-4. Read source input.
-5. Parse source input into a `SourceDocument`.
-6. Validate source-document invariants.
-7. Resolve the configured chunker.
-8. Chunk source units into deterministic source chunks.
-9. Resolve configured extractor instances through a registry.
-10. Extract from chunks in extractor-defined mode.
-11. Merge per-chunk artifact candidates deterministically.
-12. Normalize merged artifact candidates.
-13. Run deterministic validators before LLM-backed validators.
-14. Retain approved artifacts and rejected-artifact diagnostics.
-15. Serialize final output JSON.
-16. Write run manifest, diagnostics, and optional report JSON.
+2. Resolve the selected pipeline profile by ID.
+3. Apply defaults and `--only` lane selection.
+4. Validate module keys, lane definitions, LLM profiles, and capabilities.
+5. Hash the resolved pipeline definition.
+6. Create diagnostics run directory.
+7. Resolve the configured input module through the input adapter registry.
+8. Read source input.
+9. Parse source input into a `SourceDocument`.
+10. Validate source-document invariants.
+11. Resolve the configured chunker.
+12. Chunk source units into deterministic source chunks.
+13. Resolve configured artifact lanes through registries.
+14. Extract, merge, normalize, and validate each selected artifact lane.
+15. Retain approved artifacts and rejected-artifact diagnostics.
+16. Serialize output files and run-level manifest/index.
+17. Write diagnostics and optional report JSON.
 
 The runner should operate on source documents and source chunks only. Any
 transcript-specific behavior should happen before the runner, inside the input
@@ -504,6 +608,17 @@ yet.
 - Core source metadata should remain `map[string]any`. Well-known metadata keys
   should be documented as conventions, and input modules may expose typed
   accessor helpers for their own metadata.
+- Configuration should use named pipeline profiles selected by ID at the CLI.
+- A pipeline profile should be a fixed template, not a free-form DAG: shared
+  input and chunk stages, one or more artifact lanes, and shared output.
+- `--only` should select configured artifact lanes without creating ad hoc
+  pipelines.
+- Module bindings should support string shorthand and inline object settings,
+  normalized into one internal binding shape.
+- Registries should expose flat capability metadata so config can fail fast on
+  invalid module combinations.
+- The run manifest should record both `pipeline_id` and a digest of the resolved
+  pipeline definition after defaults and lane selection.
 
 ## Open Design Questions
 
@@ -511,8 +626,8 @@ yet.
   should use domain-specific merge?
 - Which artifact types need domain-specific normalization for deduplication,
   identity resolution, or consistency?
-- What should the configuration model look like for selecting input, chunk,
-  extract, merge, normalize, output, and validator modules?
+- Which operational settings should be allowed as CLI/environment overrides
+  without weakening pipeline provenance?
 
 ## Near-Term Documentation Tasks