Add a staged implementation plan for the pipeline stages

This commit is contained in:
2026-07-03 10:19:57 -05:00
parent c4da7bea1a
commit 3cf2ac577f
2 changed files with 805 additions and 499 deletions

View File

@@ -81,17 +81,11 @@ The repository should also contain a resolved pipeline model that represents:
The runner should orchestrate fake implementations through chunk, extract, The runner should orchestrate fake implementations through chunk, extract,
merge, normalize, and approval/validation behavior in tests. merge, normalize, and approval/validation behavior in tests.
The checkpoint should include one fixture-driven integration test that starts The checkpoint should include a fixture-driven walking skeleton that starts from
from fixture input bytes and ends at encoded output bytes. The fixture should fixture input bytes and ends at encoded output bytes. The walking skeleton should
use an in-memory pipeline profile with fake input adapter, deterministic exercise the stage contracts, resolved pipeline model, module metadata,
chunker, trivial extractor, fake structured LLM client, generic merger, no-op capability validation, defaults, lane selection, and fake LLM client wiring. It
normalizer, and fake or minimal JSON output encoder. This is a contract is contract coverage, not useful user-facing behavior.
exercise, not a useful user-facing workflow.
The walking skeleton should live in `internal/framework/pipeline`, with
fixtures under that package's `testdata/`. If the CLI has an extract command by
then, the same fixture may also be exercised through CLI wiring; if not, CLI
coverage remains deferred.
## Design Intent ## Design Intent
@@ -99,7 +93,7 @@ Chunking is a core application concern because many source documents, especially
transcripts, will be too large for a single LLM extraction pass. transcripts, will be too large for a single LLM extraction pass.
Chunk processing may be serial or parallel depending on extractor needs. The Chunk processing may be serial or parallel depending on extractor needs. The
architecture should support both, but the first implementation can execute architecture should support both, while checkpoint 3 may execute
deterministically in series until a later checkpoint introduces concurrency. deterministically in series until a later checkpoint introduces concurrency.
Merge and normalize are separate stages: Merge and normalize are separate stages:
@@ -120,8 +114,9 @@ The architecture should leave room for extractor-level processing modes:
- serial chunk processing; - serial chunk processing;
- parallel chunk processing. - parallel chunk processing.
The first implementation may model these modes without implementing parallel Checkpoint 3 may execute chunks serially for deterministic behavior.
execution. It should not bake in a single-pass assumption. The contracts should not bake in a single-pass assumption or prevent later
parallel execution.
## Generic Merge Behavior ## Generic Merge Behavior
@@ -143,29 +138,16 @@ Domain-specific normalizers may later:
- enforce chronological or source-reference consistency; - enforce chronological or source-reference consistency;
- attach normalization warnings. - attach normalization warnings.
## Walking Skeleton Fixture ## Walking Skeleton
The fixture-driven skeleton should prove the staged architecture continuously as The fixture-driven skeleton should prove the staged architecture continuously as
new contracts are added. It should be deliberately small: new contracts are added. It should remain deliberately small and use fake modules
only. It should validate module keys and flat capability requirements before
execution, using registry metadata rather than constructing modules. Capability
values should remain simple strings.
- a fixture source document or raw source input with two or three source units; Implementation staging for the walking skeleton belongs in
- a fake input adapter that parses the fixture into `SourceDocument`; [`implementation.md`](implementation.md).
- a deterministic chunker that produces multiple chunks;
- a trivial extractor that calls the fake structured LLM client and emits one
artifact candidate per chunk;
- a generic append-in-chunk-order merger;
- a no-op normalizer;
- a fake or minimal JSON output encoder;
- an expected output fixture checked byte-for-byte or by unmarshaled structural
equality.
The fake LLM client should be part of the test setup so the contract is
exercised without introducing provider code, prompt assets, or response schema
assets.
The walking skeleton should validate module keys and flat capability
requirements before execution. This should use registry metadata rather than
constructing modules. Keep capability values as simple strings.
## Done Criteria ## Done Criteria

File diff suppressed because it is too large Load Diff