From c4da7bea1ad7f7df5c3e81492906e907933a99cc Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 3 Jul 2026 10:09:02 -0500 Subject: [PATCH] Update feature roadmap to include a named pipeline profile configuration model --- docs/policy/architecture.md | 70 +++++++- docs/roadmap/2-framework-composition.md | 6 + ...ipeline-stages-chunking-merge-normalize.md | 36 ++++- .../4-portable-audita-infrastructure.md | 52 ++++-- docs/roadmap/5-seriatim-input-module.md | 20 ++- docs/roadmap/6-dnd-spells-extractor.md | 9 +- docs/roadmap/documentation.md | 53 +++--- docs/roadmap/initial-architecture.md | 153 +++++++++++++++--- 8 files changed, 333 insertions(+), 66 deletions(-) diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 4fe4855..33c7af7 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -250,12 +250,61 @@ The goal is to make configuration discoverable and avoid implicit or hidden operational values. User-visible defaults and cross-package operational defaults should be defined in config code. -Unless documented otherwise, precedence is: +Configuration should be organized around named pipeline profiles. A pipeline is +a fixed-shape template for the application workflow, not a free-form DAG or +general workflow program. The six-stage flow remains fixed: -1. CLI flags -2. environment variables -3. configuration file -4. built-in defaults +```text +input -> chunk -> extract -> merge -> normalize -> output +``` + +A pipeline profile should bind registered modules to those stage slots: + +- one shared input module; +- one shared chunk module by default; +- one or more artifact lanes, each with extract, merge, normalize, and + validator behavior; +- one output module. + +The MVP should use one shared chunk module per pipeline. Per-lane chunk +overrides are a future extension and should be added only if a real artifact +lane needs different chunking. + +The CLI should select a named pipeline by ID, such as +`notarius run dnd-session --input session.json`. Structural module selection +should come from configuration, not ad hoc CLI flags. CLI flags may select a +subset of configured artifact lanes, such as `--only spells,npcs`, and may +override operational settings such as model, concurrency, output directory, or +diagnostics directory. + +Pipeline definitions should support compact defaults: + +- `chunk`: `generic`; +- lane `merge`: `appendorder`; +- lane `normalize`: `noop`; +- `output`: `json`; +- `llm_profile`: `default` where an LLM profile is needed. + +Module bindings should support both string shorthand and object form. For +example, `extract: dnd/spells` and +`extract: {module: dnd/spells, llm_profile: fast}` should normalize to the same +internal binding type. + +Module registries should expose module metadata, including flat string +capabilities, without requiring module construction. Config validation should +fail fast on unknown module keys, unknown pipeline IDs, missing required slots, +missing capabilities, unknown LLM profiles, empty artifact-lane sets, or +`--only` lane names that do not exist in the selected pipeline. + +Keep capabilities as a flat string set. Do not evolve capabilities into a type +system unless real module interactions prove the need. + +Unless documented otherwise, precedence from lowest to highest is: + +1. built-in defaults +2. configuration file +3. environment variables +4. CLI flags Prefer YAML configuration unless the project has a strong reason to use another format. Config files should be discoverable at @@ -265,8 +314,15 @@ Configuration files should not contain raw secrets unless the application is explicitly designed for that. Prefer environment variables or secret files for secrets. -Stage-module-specific configuration should remain grouped by the module that -owns it. +Stage-module-specific configuration should remain inline with the pipeline slot +that owns it. Do not add named module instances until repeated inline settings +create real drift or duplication. LLM profiles are the justified top-level +exception because model settings are cross-cutting. + +The run manifest should record the selected `pipeline_id` and a digest of the +resolved pipeline definition after defaults and lane selection are applied. +`pipeline_id` alone is not sufficient provenance because a named pipeline can +change over time. ## Embedded Assets diff --git a/docs/roadmap/2-framework-composition.md b/docs/roadmap/2-framework-composition.md index 4342ed5..85a4acc 100644 --- a/docs/roadmap/2-framework-composition.md +++ b/docs/roadmap/2-framework-composition.md @@ -28,6 +28,8 @@ Out of scope: - real input parsing; - real input modules; - real extract modules; +- pipeline-profile config loading; +- module capability validation; - real LLM calls; - prompt assets; - response schema assets; @@ -50,6 +52,10 @@ The runner should operate on already parsed source documents in this checkpoint. Raw input parsing and concrete input module behavior remain deferred to the Seriatim input module checkpoint. +Pipeline-profile resolution, module metadata, and capability validation are +deferred to checkpoint 3. This checkpoint only needs constructor registries and +minimal runner composition. + Implementation staging belongs in [`implementation.md`](implementation.md). diff --git a/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md b/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md index 6de11e1..55becfa 100644 --- a/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md +++ b/docs/roadmap/3-pipeline-stages-chunking-merge-normalize.md @@ -32,6 +32,11 @@ In scope: - merge-stage contract; - normalize-stage contract; - output-stage contract and fake output encoder for pipeline completeness; +- resolved pipeline definition types for a fixed-shape pipeline template; +- module binding and module metadata types, including flat capability strings; +- default application for `chunk`, lane `merge`, lane `normalize`, `output`, + and `llm_profile`; +- lane selection behavior equivalent to future `--only`; - runner/pipeline updates that exercise these stages with fake components; - generic append/chronological merge behavior for artifact candidates when appropriate; @@ -47,6 +52,7 @@ Out of scope: - prompt assets; - response schema assets; - diagnostics run directory; +- production config file loading; - real CLI command behavior; - production output serialization or durable output writing. @@ -61,15 +67,26 @@ The repository should contain explicit pipeline-stage contracts: - `Normalizer`: merged candidates to normalized candidates. - `OutputEncoder`: final artifact bundle to bytes. +The repository should also contain a resolved pipeline model that represents: + +- `pipeline_id`; +- shared input binding; +- shared chunk binding; +- selected artifact lanes; +- lane extract, merge, normalize, and validator bindings; +- output binding; +- resolved defaults; +- resolved pipeline digest input. + The runner should orchestrate fake implementations through chunk, extract, merge, normalize, and approval/validation behavior in tests. The checkpoint should include one fixture-driven integration test that starts from fixture input bytes and ends at encoded output bytes. The fixture should -use a fake input adapter, deterministic chunker, trivial extractor, fake -structured LLM client, generic merger, no-op normalizer, and fake or minimal -JSON output encoder. This is a contract exercise, not a useful user-facing -workflow. +use an in-memory pipeline profile with fake input adapter, deterministic +chunker, trivial extractor, fake structured LLM client, generic merger, no-op +normalizer, and fake or minimal JSON output encoder. This is a contract +exercise, not a useful user-facing workflow. The walking skeleton should live in `internal/framework/pipeline`, with fixtures under that package's `testdata/`. If the CLI has an extract command by @@ -146,10 +163,17 @@ The fake LLM client should be part of the test setup so the contract is exercised without introducing provider code, prompt assets, or response schema assets. +The walking skeleton should validate module keys and flat capability +requirements before execution. This should use registry metadata rather than +constructing modules. Keep capability values as simple strings. + ## Done Criteria - `go test ./...` passes. - Pipeline-stage contracts are explicit and source/domain agnostic. +- A resolved pipeline profile model exists for fixed-shape pipeline templates. +- Pipeline defaults and lane selection are covered by fake tests. +- Module capability validation is covered by fake tests. - Fake tests prove input source documents can be chunked, extracted, merged, and normalized. - A fixture-driven walking skeleton proves fake input, chunk, extract, merge, @@ -165,6 +189,10 @@ assets. - Is the workflow clearly represented as input, chunk, extract, merge, normalize, and output? - Are merge and normalize cleanly separated? +- Does the resolved pipeline model avoid becoming a general-purpose workflow + engine? +- Are module capabilities simple flat strings? +- Does lane selection avoid creating ad hoc pipelines? - Can a generic merger handle simple chronological artifact streams? - Can a later domain-specific normalizer handle duplicates and consistency without changing core runner contracts? diff --git a/docs/roadmap/4-portable-audita-infrastructure.md b/docs/roadmap/4-portable-audita-infrastructure.md index 50fe6c5..155d050 100644 --- a/docs/roadmap/4-portable-audita-infrastructure.md +++ b/docs/roadmap/4-portable-audita-infrastructure.md @@ -21,7 +21,8 @@ In scope: - prompt registry pattern; - response-schema registry pattern; - diagnostics run directory pattern; -- minimal config structs and defaults for implemented runtime pieces. +- config loading and validation for named pipeline profiles and implemented + runtime pieces. Out of scope: @@ -31,6 +32,7 @@ Out of scope: - correction ledger terminology; - Audita module or validator behavior; - real D&D prompts or schemas unless needed as inert registry tests. +- embedded built-in pipeline profiles. ## Proposed Stages @@ -82,28 +84,55 @@ Initial diagnostics should cover: - invocation metadata; - redacted effective config; +- selected pipeline ID; +- resolved pipeline definition and digest; - source document artifact; - run report placeholder; - error log on failure. Avoid Audita-specific artifact names such as correction ledger. -### Stage 5: Minimal Runtime Config +### Stage 5: Pipeline Profile Config -Add config structs and defaults only for infrastructure that now exists. +Add config structs, loading, defaults, and validation for named pipeline +profiles. Initial config areas: -- input module key; -- extractor keys; -- primary LLM settings; -- validation LLM settings if needed; +- `llm_profiles`; +- `pipelines..input`; +- `pipelines..chunk`; +- `pipelines..artifacts..extract`; +- lane `merge`, `normalize`, and validator settings; +- output module selection; +- inline module-binding object form and string shorthand; +- default `chunk`, `merge`, `normalize`, `output`, and `llm_profile`; +- selected pipeline ID and lane filtering for runtime use; - concurrency; - work directory; - diagnostics retention. -Config loading can remain minimal unless the implementation needs full file/env -precedence at this checkpoint. +Config loading should support the standard precedence model: + +1. built-in defaults +2. configuration file +3. environment variables +4. CLI flags + +Structural module selection should come from pipeline config. CLI flags may +override operational settings and artifact lane selection, but should not offer +ad hoc `--extractor` or `--chunker` wiring. + +Add validation for unknown pipeline IDs, unknown module keys, missing required +slots, missing capabilities, unknown LLM profiles, empty artifact-lane sets, and +invalid lane selections. + +If the CLI shell is ready, add: + +```sh +notarius config validate +notarius pipelines list +``` ## Done Criteria @@ -113,10 +142,13 @@ precedence at this checkpoint. ledger code has been copied. - Runtime tests cover secret redaction, schema registry lookup, prompt metadata, and scheduler behavior where applicable. +- Config tests cover named pipeline profiles, defaults, lane selection, + capability validation, and resolved pipeline digesting. ## Review Questions - Did we copy only reusable infrastructure? - Do provider-specific types stay behind adapter/runtime boundaries? - Are diagnostics names and report concepts extraction-oriented? -- Is config limited to implemented behavior? +- Is config limited to named pipeline profiles and implemented behavior? +- Are structural pipeline changes kept out of ad hoc CLI flags? diff --git a/docs/roadmap/5-seriatim-input-module.md b/docs/roadmap/5-seriatim-input-module.md index c70db9d..d561452 100644 --- a/docs/roadmap/5-seriatim-input-module.md +++ b/docs/roadmap/5-seriatim-input-module.md @@ -21,8 +21,10 @@ In scope: - mapping into `SourceDocument` and `SourceUnit`; - source-document validation; - input adapter registry wiring; +- module metadata/capabilities for pipeline validation; - fixtures and tests; -- CLI/config path to select the input module if the CLI shell exists. +- config path to use the input module through a named pipeline profile if + config loading exists. Out of scope: @@ -81,13 +83,19 @@ The resulting `SourceDocument` should pass core source validation. Register the module under a stable input adapter key, likely `seriatim`. -If CLI support exists, add provisional selection: +Declare module metadata for pipeline validation. Initial provided capabilities +should include transcript-oriented metadata such as `speaker` and `timestamps` +if those fields are preserved from Seriatim input. + +If config and CLI support exist, add a minimal pipeline-profile fixture or test +config using the Seriatim input module: ```sh -notarius extract ./transcript.json --input seriatim +notarius run dnd-session --input ./transcript.json --only spells ``` -The command may still use fake extractors until checkpoint 6. +The command may still use fake extract, merge, normalize, and output modules +until checkpoint 6. ### Fixtures And Tests @@ -105,7 +113,9 @@ Add fixtures and tests for: - `go test ./...` passes. - Seriatim minimal transcript JSON maps into `SourceDocument`. - Transcript fields do not appear in core runner contracts. -- The input module is selectable through the registry. +- The input module is selectable through the registry and pipeline-profile + configuration when config support exists. +- The input module declares capabilities needed for pipeline validation. - Tests prove transcript-specific assumptions are isolated to the input module. ## Review Questions diff --git a/docs/roadmap/6-dnd-spells-extractor.md b/docs/roadmap/6-dnd-spells-extractor.md index 038c94a..0d5f68e 100644 --- a/docs/roadmap/6-dnd-spells-extractor.md +++ b/docs/roadmap/6-dnd-spells-extractor.md @@ -20,6 +20,7 @@ In scope: - structured response schema asset; - prompt assets; - `internal/modules/extract/dnd/spells`; +- module metadata/capability requirements for pipeline validation; - source-reference and schema validators in the extractor chain; - fake LLM tests; - CLI-level integration test if the CLI path is ready. @@ -81,6 +82,7 @@ Implement `internal/modules/extract/dnd/spells`. The extractor should: - satisfy the framework `Extractor` contract; +- declare module metadata for pipeline-profile validation; - build LLM messages from a source document or source chunk; - call the structured LLM client; - return artifact candidates with source references; @@ -107,10 +109,11 @@ Add tests using a fake structured LLM client: If the CLI path is ready, add an end-to-end test using: ```sh -notarius extract ./transcript.json --input seriatim --extractors dnd.spells --output ./artifacts.json +notarius run dnd-session --input ./transcript.json --only spells ``` -The test should use fake LLM wiring and fixture input. +The test should use fake LLM wiring, fixture input, and a named pipeline profile +with a `spells` artifact lane. ## Done Criteria @@ -118,6 +121,8 @@ The test should use fake LLM wiring and fixture input. - Seriatim input can flow through the runner into the D&D spells extractor. - Spell artifacts include valid source references. - D&D concepts are contained in extract module/artifact packages and docs. +- The spells module can be selected as a named artifact lane in pipeline + configuration. - The first meaningful vertical slice is available through tests, and through CLI if the CLI path is ready. diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md index cc4cff8..a77b108 100644 --- a/docs/roadmap/documentation.md +++ b/docs/roadmap/documentation.md @@ -100,42 +100,54 @@ generic runner or framework docs. ### CLI Docs Should Reflect Extensibility -The CLI reference should present input modules, chunk modules, extract modules, -merge modules, normalize modules, and output modules as selectable or -configurable components as they become user-facing. +The CLI reference should present named pipeline profiles as the primary +user-facing abstraction. Individual stage modules should be visible through +pipeline configuration and discovery commands, not through ad hoc structural +run flags. Provisional command shape: ```sh -notarius extract ./source.json --input seriatim --extractors dnd.spells --output ./artifacts.json +notarius run dnd-session --input ./source.json +notarius run dnd-session --input ./source.json --only spells,npcs +notarius config validate +notarius pipelines list ``` Once implemented, `docs/cli.md` should document: -- positional source input path; -- input module selection; -- extract module selection; -- chunk/merge/normalize/output selection when configurable; +- pipeline ID selection; +- required input path flags; +- `--only` artifact-lane selection; - config path behavior; +- operational overrides such as output path, model, concurrency, and diagnostics + directory; - output path behavior; - diagnostics and report behavior; - exit codes. ### Config Docs Should Separate Framework And Plugin-Like Options -`docs/config.md` should group fields by responsibility: +`docs/config.md` should describe named pipeline profiles and the resolved +pipeline model. -- input module selection and module-specific options; -- chunk module selection and module-specific options; -- extract module selection and module-specific options; -- merge module selection and module-specific options; -- normalize module selection and module-specific options; -- output module selection and module-specific options; -- LLM runtime; -- validation runtime; -- diagnostics. +It should cover: -Module-specific config should not leak into unrelated core config sections. +- config file locations and precedence; +- `llm_profiles`; +- `pipelines..input`; +- `pipelines..chunk`; +- `pipelines..artifacts..extract`; +- lane `merge`, `normalize`, and validator settings; +- output module selection; +- string shorthand versus inline module-binding object form; +- defaults for omitted slots; +- capability validation; +- pipeline digest and manifest provenance. + +Module-specific config should stay inline with the pipeline slot that owns it. +Top-level named module instances should not be introduced until repeated inline +settings create real drift. `llm_profiles` are the cross-cutting exception. ### Examples Should Stay Real @@ -148,6 +160,7 @@ Likely future examples: examples/seriatim-minimal-transcript.json examples/minimal-config.yml examples/dnd-spells.artifacts.json +examples/dnd-session.config.yml ``` Examples should be secret-free and should use the same command shapes documented @@ -179,6 +192,8 @@ Before merging docs, check: - Are D&D details isolated to extract module or artifact docs? - Is there one canonical home for the topic? - Do command examples match implemented CLI syntax? +- Do config examples use named pipeline profiles rather than ad hoc module + flags? - Are examples valid, maintained, and free of secrets? - Did any architecture, config, CLI, stage module, validator, or artifact contract change require a docs update? diff --git a/docs/roadmap/initial-architecture.md b/docs/roadmap/initial-architecture.md index 863e3b2..7a95705 100644 --- a/docs/roadmap/initial-architecture.md +++ b/docs/roadmap/initial-architecture.md @@ -303,6 +303,8 @@ Per-run provenance record. ```go type RunManifest struct { EnvelopeVersion string `json:"envelope_version"` + PipelineID string `json:"pipeline_id"` + PipelineDigest string `json:"pipeline_digest"` InputModule string `json:"input_module"` Chunker string `json:"chunker"` SourceDigests []string `json:"source_digests"` @@ -316,8 +318,8 @@ type RunManifest struct { ``` The manifest should eventually include model names, prompt IDs, prompt hashes, -response schema versions, config source, started/completed timestamps, and -diagnostics paths. +response schema versions, config source, redacted resolved config digest, +started/completed timestamps, and diagnostics paths. ## Initial Extractor Targets @@ -393,6 +395,107 @@ The architecture should support extractors outside the D&D domain. Examples: These should be addable as extract modules without changing runner, validator, source-reference, or LLM framework contracts. +## Configuration Model + +Notarius should use named pipeline profiles selected by ID at the CLI. A +pipeline is a fixed-shape template for the known application workflow, not a +free-form list of steps: + +```text +input -> chunk -> extract -> merge -> normalize -> output +``` + +A pipeline profile should define one shared front end and one or more artifact +lanes: + +- shared input module; +- shared chunk module by default; +- artifact lanes containing extract, merge, normalize, and validator behavior; +- shared output module. + +The MVP should use one shared chunk module per pipeline. Per-lane chunk +overrides can be added later if an artifact lane, such as combat, proves it +needs a different chunking strategy. + +Example shape: + +```yaml +llm_profiles: + default: + model: example-model + max_concurrency: 4 + +pipelines: + dnd-session: + input: seriatim + chunk: dnd/transcript + artifacts: + spells: + extract: dnd/spells + normalize: dnd/spells + npcs: + extract: dnd/npcs + items: + extract: dnd/items +``` + +The CLI should run named pipelines: + +```sh +notarius run dnd-session --input session-014.json +notarius run dnd-session --input session-014.json --only spells,npcs +``` + +`--only` should select configured artifact lanes. It should not create an +ad hoc pipeline. Structural module selection should come from config, while CLI +flags may override operational knobs such as model, concurrency, output +directory, and diagnostics directory. + +Initial defaults: + +- `chunk`: `generic`; +- lane `merge`: `appendorder`; +- lane `normalize`: `noop`; +- `output`: `json`; +- `llm_profile`: `default` where an LLM profile is needed. + +Module bindings should support both string shorthand and object form: + +```yaml +extract: dnd/spells +``` + +```yaml +extract: + module: dnd/spells + llm_profile: fast + prompt_version: v1 +``` + +Both forms should normalize into a single internal `ModuleBinding` shape before +validation and manifest hashing. + +Pipeline validation should use module metadata declared through registries. +Modules should expose flat string capability metadata, such as `speaker` or +`timestamps`, without requiring module construction. Config validation should +fail fast for: + +- unknown pipeline IDs; +- unknown module keys; +- missing required slots; +- missing required capabilities; +- unknown LLM profiles; +- empty artifact-lane sets; +- `--only` lane names that do not exist in the selected pipeline. + +The MVP should keep pipelines config-file-only. Built-in pipeline profiles can +be added later if the project needs embedded defaults, but that introduces +merge/override semantics that the MVP does not need. + +The resolved pipeline definition should be hashed after defaults and lane +selection are applied. The run manifest should record both `pipeline_id` and +`pipeline_digest`; a pipeline ID alone is not stable provenance. + ## Proposed Pipeline Flow The application workflow should be first-class: @@ -404,21 +507,22 @@ input -> chunk -> extract -> merge -> normalize -> output Proposed runner flow: 1. Load effective config. -2. Create diagnostics run directory. -3. Resolve the configured input module through the input adapter registry. -4. Read source input. -5. Parse source input into a `SourceDocument`. -6. Validate source-document invariants. -7. Resolve the configured chunker. -8. Chunk source units into deterministic source chunks. -9. Resolve configured extractor instances through a registry. -10. Extract from chunks in extractor-defined mode. -11. Merge per-chunk artifact candidates deterministically. -12. Normalize merged artifact candidates. -13. Run deterministic validators before LLM-backed validators. -14. Retain approved artifacts and rejected-artifact diagnostics. -15. Serialize final output JSON. -16. Write run manifest, diagnostics, and optional report JSON. +2. Resolve the selected pipeline profile by ID. +3. Apply defaults and `--only` lane selection. +4. Validate module keys, lane definitions, LLM profiles, and capabilities. +5. Hash the resolved pipeline definition. +6. Create diagnostics run directory. +7. Resolve the configured input module through the input adapter registry. +8. Read source input. +9. Parse source input into a `SourceDocument`. +10. Validate source-document invariants. +11. Resolve the configured chunker. +12. Chunk source units into deterministic source chunks. +13. Resolve configured artifact lanes through registries. +14. Extract, merge, normalize, and validate each selected artifact lane. +15. Retain approved artifacts and rejected-artifact diagnostics. +16. Serialize output files and run-level manifest/index. +17. Write diagnostics and optional report JSON. The runner should operate on source documents and source chunks only. Any transcript-specific behavior should happen before the runner, inside the input @@ -504,6 +608,17 @@ yet. - Core source metadata should remain `map[string]any`. Well-known metadata keys should be documented as conventions, and input modules may expose typed accessor helpers for their own metadata. +- Configuration should use named pipeline profiles selected by ID at the CLI. +- A pipeline profile should be a fixed template, not a free-form DAG: shared + input and chunk stages, one or more artifact lanes, and shared output. +- `--only` should select configured artifact lanes without creating ad hoc + pipelines. +- Module bindings should support string shorthand and inline object settings, + normalized into one internal binding shape. +- Registries should expose flat capability metadata so config can fail fast on + invalid module combinations. +- The run manifest should record both `pipeline_id` and a digest of the resolved + pipeline definition after defaults and lane selection. ## Open Design Questions @@ -511,8 +626,8 @@ yet. should use domain-specific merge? - Which artifact types need domain-specific normalization for deduplication, identity resolution, or consistency? -- What should the configuration model look like for selecting input, chunk, - extract, merge, normalize, output, and validator modules? +- Which operational settings should be allowed as CLI/environment overrides + without weakening pipeline provenance? ## Near-Term Documentation Tasks