Add a staged implementation plan for the pipeline stages

This commit is contained in:
2026-07-03 10:19:57 -05:00
parent c4da7bea1a
commit 3cf2ac577f
2 changed files with 805 additions and 499 deletions

View File

@@ -81,17 +81,11 @@ The repository should also contain a resolved pipeline model that represents:
The runner should orchestrate fake implementations through chunk, extract,
merge, normalize, and approval/validation behavior in tests.
The checkpoint should include one fixture-driven integration test that starts
from fixture input bytes and ends at encoded output bytes. The fixture should
use an in-memory pipeline profile with fake input adapter, deterministic
chunker, trivial extractor, fake structured LLM client, generic merger, no-op
normalizer, and fake or minimal JSON output encoder. This is a contract
exercise, not a useful user-facing workflow.
The walking skeleton should live in `internal/framework/pipeline`, with
fixtures under that package's `testdata/`. If the CLI has an extract command by
then, the same fixture may also be exercised through CLI wiring; if not, CLI
coverage remains deferred.
The checkpoint should include a fixture-driven walking skeleton that starts from
fixture input bytes and ends at encoded output bytes. The walking skeleton should
exercise the stage contracts, resolved pipeline model, module metadata,
capability validation, defaults, lane selection, and fake LLM client wiring. It
is contract coverage, not useful user-facing behavior.
## Design Intent
@@ -99,7 +93,7 @@ Chunking is a core application concern because many source documents, especially
transcripts, will be too large for a single LLM extraction pass.
Chunk processing may be serial or parallel depending on extractor needs. The
architecture should support both, but the first implementation can execute
architecture should support both, while checkpoint 3 may execute
deterministically in series until a later checkpoint introduces concurrency.
Merge and normalize are separate stages:
@@ -120,8 +114,9 @@ The architecture should leave room for extractor-level processing modes:
- serial chunk processing;
- parallel chunk processing.
The first implementation may model these modes without implementing parallel
execution. It should not bake in a single-pass assumption.
Checkpoint 3 may execute chunks serially for deterministic behavior.
The contracts should not bake in a single-pass assumption or prevent later
parallel execution.
## Generic Merge Behavior
@@ -143,29 +138,16 @@ Domain-specific normalizers may later:
- enforce chronological or source-reference consistency;
- attach normalization warnings.
## Walking Skeleton Fixture
## Walking Skeleton
The fixture-driven skeleton should prove the staged architecture continuously as
new contracts are added. It should be deliberately small:
new contracts are added. It should remain deliberately small and use fake modules
only. It should validate module keys and flat capability requirements before
execution, using registry metadata rather than constructing modules. Capability
values should remain simple strings.
- a fixture source document or raw source input with two or three source units;
- a fake input adapter that parses the fixture into `SourceDocument`;
- a deterministic chunker that produces multiple chunks;
- a trivial extractor that calls the fake structured LLM client and emits one
artifact candidate per chunk;
- a generic append-in-chunk-order merger;
- a no-op normalizer;
- a fake or minimal JSON output encoder;
- an expected output fixture checked byte-for-byte or by unmarshaled structural
equality.
The fake LLM client should be part of the test setup so the contract is
exercised without introducing provider code, prompt assets, or response schema
assets.
The walking skeleton should validate module keys and flat capability
requirements before execution. This should use registry metadata rather than
constructing modules. Keep capability values as simple strings.
Implementation staging for the walking skeleton belongs in
[`implementation.md`](implementation.md).
## Done Criteria

File diff suppressed because it is too large Load Diff