Add a staged implementation plan for the initial framework and module composition architecture
This commit is contained in:
@@ -32,65 +32,26 @@ Out of scope:
|
||||
- diagnostics run directory;
|
||||
- real D&D artifact schemas.
|
||||
|
||||
## Proposed Stages
|
||||
## Target End State
|
||||
|
||||
### Stage 1: Input Adapter Registry
|
||||
The repository should contain a minimal framework composition layer:
|
||||
|
||||
Add a registry for `InputAdapter` constructors or instances.
|
||||
- `internal/framework/inputregistry` registers and builds input adapter
|
||||
constructors by stable key.
|
||||
- `internal/framework/extractorregistry` registers and builds extractor
|
||||
constructors by stable key.
|
||||
- `internal/framework/validators` provides shared validator decision helpers and
|
||||
cardinality checks.
|
||||
- `internal/framework/runner` executes configured extractors against a
|
||||
`SourceDocument`, applies validator chains, and returns approved and rejected
|
||||
artifacts.
|
||||
|
||||
The registry should:
|
||||
The runner should operate on already parsed source documents in this checkpoint.
|
||||
Raw input parsing and concrete input adapter behavior remain deferred to the
|
||||
Seriatim adapter checkpoint.
|
||||
|
||||
- reject empty keys;
|
||||
- reject duplicate registrations;
|
||||
- return clear errors for unknown keys;
|
||||
- avoid importing concrete adapter packages from core contracts.
|
||||
|
||||
### Stage 2: Extractor Registry
|
||||
|
||||
Add a registry for extractor constructors or instances.
|
||||
|
||||
The registry should:
|
||||
|
||||
- support stable extractor keys;
|
||||
- support repeated extractor instances if needed later;
|
||||
- return clear errors for unknown keys;
|
||||
- avoid domain-specific logic.
|
||||
|
||||
### Stage 3: Validator Decisions
|
||||
|
||||
Add validator decision types and cardinality checks.
|
||||
|
||||
Each validator should return exactly one decision for each candidate artifact it
|
||||
receives.
|
||||
|
||||
Decision fields should include:
|
||||
|
||||
- candidate index;
|
||||
- approved flag;
|
||||
- reason code;
|
||||
- message;
|
||||
- optional diagnostics path.
|
||||
|
||||
### Stage 4: Minimal Runner
|
||||
|
||||
Add a runner that can:
|
||||
|
||||
1. receive a `SourceDocument`;
|
||||
2. execute configured extractors;
|
||||
3. validate candidate artifacts;
|
||||
4. return approved and rejected artifacts.
|
||||
|
||||
Keep source chunking optional or stubbed at this checkpoint. The runner may
|
||||
operate on whole documents only until source-unit chunking is added later.
|
||||
|
||||
### Stage 5: Runner Tests With Fakes
|
||||
|
||||
Add tests with fake components that prove:
|
||||
|
||||
- registered fake extractors run in configured order;
|
||||
- validators filter candidates deterministically;
|
||||
- decision cardinality failures are surfaced;
|
||||
- approved and rejected artifacts are returned in stable order.
|
||||
Implementation staging belongs in
|
||||
[`implementation.md`](implementation.md).
|
||||
|
||||
## Done Criteria
|
||||
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
# Implementation Plan: Checkpoint 1 Core Contracts And Skeleton
|
||||
# Implementation Plan: Checkpoint 2 Framework Composition
|
||||
|
||||
## Status
|
||||
|
||||
This is a staged implementation plan for
|
||||
[`1-core-contracts-and-skeleton.md`](1-core-contracts-and-skeleton.md). It is
|
||||
intended for an LLM coding agent to follow stage by stage.
|
||||
[`2-framework-composition.md`](2-framework-composition.md). It is intended for
|
||||
an LLM coding agent to follow stage by stage.
|
||||
|
||||
This plan implements only checkpoint 1. Do not implement real input adapters,
|
||||
real extractors, LLM provider clients, prompt assets, response schema registries,
|
||||
diagnostics run directories, or production config loading in this pass.
|
||||
This plan implements only checkpoint 2. Do not implement real input adapters,
|
||||
real extractors, source chunking, LLM provider clients, prompt assets, response
|
||||
schema registries, diagnostics run directories, production config loading, or
|
||||
D&D artifact schemas in this pass.
|
||||
|
||||
## Policy Context
|
||||
|
||||
@@ -19,481 +20,505 @@ Follow:
|
||||
|
||||
Required boundaries:
|
||||
|
||||
- core packages must stay source-agnostic and domain-agnostic;
|
||||
- transcript-specific concepts must not appear in core or framework contracts;
|
||||
- D&D concepts must not appear in core or framework contracts;
|
||||
- source references must target generic source units;
|
||||
- framework packages must stay source-agnostic and domain-agnostic;
|
||||
- input adapter registry code must not import concrete adapter packages;
|
||||
- extractor registry and runner code must not import concrete extractor
|
||||
packages;
|
||||
- runner code should operate on `SourceDocument`, not transcript-specific
|
||||
structures;
|
||||
- validators should be independently testable and composable;
|
||||
- planned behavior outside this checkpoint must remain in roadmap docs.
|
||||
|
||||
## Global Implementation Decisions
|
||||
|
||||
- Use module path `gitea.maximumdirect.net/eric/notarius`.
|
||||
- Use Go `1.24.0`, matching the nearby Audita project.
|
||||
- Add no third-party dependencies in checkpoint 1.
|
||||
- Keep CLI behavior minimal: root help/usage and unknown-command handling only.
|
||||
- Use package names exactly as listed in this plan unless implementation reveals
|
||||
a compile-time conflict.
|
||||
- Use `encoding/json.RawMessage` for generic artifact payloads.
|
||||
- Store contract tests close to the contracts they exercise.
|
||||
- Run `gofmt` on all Go files before validation.
|
||||
- Add no third-party dependencies in checkpoint 2.
|
||||
- Do not change public core source/artifact/contract types unless a stage cannot
|
||||
compile without a small compatibility adjustment.
|
||||
- Use constructor-based registries. Constructors can close over future
|
||||
dependencies without changing registry APIs.
|
||||
- Registries normalize keys with `strings.TrimSpace`.
|
||||
- Registries reject empty keys, duplicate keys, nil constructors, nil built
|
||||
instances, and key/name mismatches.
|
||||
- The minimal runner receives an already parsed `*source.SourceDocument`.
|
||||
- The minimal runner does not use input adapters yet; input adapter registry is
|
||||
tested directly.
|
||||
- The runner assigns global candidate indices in extractor execution order.
|
||||
- If a candidate supplies non-empty extractor metadata that conflicts with its
|
||||
owning extractor, the runner returns an error.
|
||||
- If a candidate leaves extractor metadata empty, the runner fills
|
||||
`ExtractorKey`, `ArtifactType`, and `SchemaVersion` from the owning extractor.
|
||||
- Validator chains run in the order returned by `Extractor.Validators()`.
|
||||
- A validator result must use the same `ValidatorName` as `Validator.Name()`.
|
||||
- Each validator must return exactly one decision for each currently eligible
|
||||
candidate.
|
||||
- If an extractor has no validators, its candidates are approved.
|
||||
- On setup, extraction, or validation errors, return the current partial
|
||||
`RunOutput` with a wrapped error.
|
||||
- Run `gofmt` on all touched Go files before validation.
|
||||
|
||||
## Stage 1: Bootstrap Go Project And CLI Shell
|
||||
## Stage 1: Input Adapter Registry
|
||||
|
||||
### Goal
|
||||
|
||||
Create a compileable Go application shell with a minimal CLI entrypoint.
|
||||
Add a constructor-based registry for `contracts.InputAdapter`.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `go.mod`
|
||||
- `cmd/notarius/main.go`
|
||||
- `internal/cli/run.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `internal/framework/inputregistry/registry.go`
|
||||
- `internal/framework/inputregistry/registry_test.go`
|
||||
|
||||
### Required Implementation
|
||||
### Required API
|
||||
|
||||
Create `go.mod`:
|
||||
|
||||
```go
|
||||
module gitea.maximumdirect.net/eric/notarius
|
||||
|
||||
go 1.24.0
|
||||
```
|
||||
|
||||
Create `cmd/notarius/main.go`:
|
||||
|
||||
- import `os`;
|
||||
- import `gitea.maximumdirect.net/eric/notarius/internal/cli`;
|
||||
- call `os.Exit(cli.Run(os.Args[1:], os.Stdout, os.Stderr))`.
|
||||
|
||||
Create `internal/cli.Run(args []string, stdout, stderr io.Writer) int`.
|
||||
|
||||
Initial behavior:
|
||||
|
||||
- `notarius`, `notarius help`, `notarius --help`, and `notarius -h` write root
|
||||
usage to `stdout` and return `0`;
|
||||
- unknown commands write `notarius: unknown command "<cmd>"` plus usage to
|
||||
`stderr` and return `2`;
|
||||
- no `extract` command is implemented in checkpoint 1.
|
||||
|
||||
Root usage should be short and should not document unimplemented behavior as if
|
||||
it exists. Acceptable text:
|
||||
|
||||
```text
|
||||
Usage:
|
||||
notarius help
|
||||
```
|
||||
|
||||
### Required Tests
|
||||
|
||||
In `internal/cli/run_test.go`, cover:
|
||||
|
||||
- no args returns `0` and writes usage to stdout;
|
||||
- `help`, `--help`, and `-h` return `0`;
|
||||
- unknown command returns `2`, writes to stderr, and does not write meaningful
|
||||
stdout.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w cmd internal
|
||||
go test ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
## Stage 2: Core Source Model
|
||||
|
||||
### Goal
|
||||
|
||||
Add the generic source-document model and validation helpers.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/core/source/source.go`
|
||||
- `internal/core/source/validation.go`
|
||||
- `internal/core/source/source_test.go`
|
||||
|
||||
### Required Types
|
||||
|
||||
Create package `source`.
|
||||
Create package `inputregistry`.
|
||||
|
||||
Define:
|
||||
|
||||
```go
|
||||
type SourceDocument struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Format string `json:"format"`
|
||||
Digest string `json:"digest"`
|
||||
Units []SourceUnit `json:"units"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
type Constructor func() (contracts.InputAdapter, error)
|
||||
|
||||
type Registry struct {
|
||||
// unexported fields
|
||||
}
|
||||
|
||||
type SourceUnit struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Text string `json:"text"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type SourceRef struct {
|
||||
SourceID string `json:"source_id"`
|
||||
StartUnitID string `json:"start_unit_id"`
|
||||
EndUnitID string `json:"end_unit_id"`
|
||||
}
|
||||
func New() *Registry
|
||||
func (r *Registry) Register(key string, constructor Constructor) error
|
||||
func (r *Registry) Build(key string) (contracts.InputAdapter, error)
|
||||
func (r *Registry) RegisteredKeys() []string
|
||||
```
|
||||
|
||||
### Required Helpers
|
||||
### Required Behavior
|
||||
|
||||
Implement:
|
||||
`Register` must:
|
||||
|
||||
```go
|
||||
func ValidateDocument(doc *SourceDocument) error
|
||||
func ValidateRef(doc *SourceDocument, ref SourceRef) error
|
||||
func UnitIndex(doc *SourceDocument, unitID string) (int, bool)
|
||||
```
|
||||
- return an error if the registry is nil;
|
||||
- trim `key`;
|
||||
- reject empty keys;
|
||||
- reject nil constructors;
|
||||
- reject duplicate keys;
|
||||
- store constructors by normalized key.
|
||||
|
||||
Validation rules:
|
||||
`Build` must:
|
||||
|
||||
- document must be non-nil;
|
||||
- document `ID`, `Kind`, `Format`, and `Digest` must be non-empty after
|
||||
trimming whitespace;
|
||||
- document must contain at least one unit;
|
||||
- each unit `ID`, `Kind`, and `Text` must be non-empty after trimming
|
||||
whitespace;
|
||||
- unit IDs must be unique within a document;
|
||||
- `ValidateRef` requires non-empty `SourceID`, `StartUnitID`, and `EndUnitID`;
|
||||
- `ValidateRef` requires `ref.SourceID == doc.ID`;
|
||||
- start and end unit IDs must exist;
|
||||
- start unit index must be less than or equal to end unit index.
|
||||
- return an error if the registry is nil;
|
||||
- trim `key`;
|
||||
- reject empty keys;
|
||||
- return a clear unknown-key error for unregistered keys;
|
||||
- call the constructor;
|
||||
- wrap constructor errors with key context;
|
||||
- reject nil adapters;
|
||||
- reject adapters whose `Key()` does not match the normalized key.
|
||||
|
||||
Do not add transcript-specific fields or helpers.
|
||||
`RegisteredKeys` must:
|
||||
|
||||
### Error Style
|
||||
|
||||
Return ordinary Go errors with actionable field context, such as:
|
||||
|
||||
- `source document id must not be empty`;
|
||||
- `source unit[2].id must not be empty`;
|
||||
- `source unit id "u1" is duplicated`;
|
||||
- `source ref start_unit_id "u9" was not found`.
|
||||
- return nil for a nil registry;
|
||||
- return registered keys in sorted order;
|
||||
- return a copy that callers cannot mutate to affect registry state.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Cover:
|
||||
|
||||
- valid document;
|
||||
- nil document;
|
||||
- missing document fields;
|
||||
- empty units;
|
||||
- missing unit fields;
|
||||
- duplicate unit IDs;
|
||||
- valid source reference;
|
||||
- source ID mismatch;
|
||||
- missing source-ref unit ID;
|
||||
- reversed source-ref unit order.
|
||||
- successful registration and build;
|
||||
- key trimming;
|
||||
- empty key rejection;
|
||||
- duplicate key rejection;
|
||||
- nil constructor rejection;
|
||||
- unknown key build error;
|
||||
- constructor error wrapping;
|
||||
- nil adapter rejection;
|
||||
- adapter key mismatch rejection;
|
||||
- sorted `RegisteredKeys`;
|
||||
- nil registry behavior.
|
||||
|
||||
Use fake adapters only. Do not add concrete adapter packages.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/core/source
|
||||
go test ./internal/core/source
|
||||
gofmt -w internal/framework/inputregistry
|
||||
go test ./internal/framework/inputregistry
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 3: Core Artifact Model
|
||||
## Stage 2: Extractor Registry
|
||||
|
||||
### Goal
|
||||
|
||||
Add extractor-neutral artifact and manifest types.
|
||||
Add a constructor-based registry for `contracts.Extractor`.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/core/artifacts/artifacts.go`
|
||||
- `internal/core/artifacts/artifacts_test.go`
|
||||
- `internal/framework/extractorregistry/registry.go`
|
||||
- `internal/framework/extractorregistry/registry_test.go`
|
||||
|
||||
### Required Types
|
||||
### Required API
|
||||
|
||||
Create package `artifacts`.
|
||||
Create package `extractorregistry`.
|
||||
|
||||
Define:
|
||||
|
||||
```go
|
||||
type Candidate struct {
|
||||
Index int `json:"index"`
|
||||
ExtractorKey string `json:"extractor_key"`
|
||||
ArtifactType string `json:"artifact_type"`
|
||||
SchemaVersion string `json:"schema_version"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
SourceRefs []source.SourceRef `json:"source_refs,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
type Constructor func() (contracts.Extractor, error)
|
||||
|
||||
type Registry struct {
|
||||
// unexported fields
|
||||
}
|
||||
|
||||
type Artifact struct {
|
||||
ExtractorKey string `json:"extractor_key"`
|
||||
ArtifactType string `json:"artifact_type"`
|
||||
SchemaVersion string `json:"schema_version"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
SourceRefs []source.SourceRef `json:"source_refs,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
func New() *Registry
|
||||
func (r *Registry) Register(key string, constructor Constructor) error
|
||||
func (r *Registry) Build(key string) (contracts.Extractor, error)
|
||||
func (r *Registry) RegisteredKeys() []string
|
||||
```
|
||||
|
||||
### Required Behavior
|
||||
|
||||
Mirror `inputregistry` behavior, but for `contracts.Extractor`.
|
||||
|
||||
`Build` must reject extractors whose `Key()` does not match the normalized key.
|
||||
It should not validate artifact type, schema version, or validator chain. Those
|
||||
are extractor/runtime concerns.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Cover the same cases as the input adapter registry:
|
||||
|
||||
- successful registration and build;
|
||||
- key trimming;
|
||||
- empty key rejection;
|
||||
- duplicate key rejection;
|
||||
- nil constructor rejection;
|
||||
- unknown key build error;
|
||||
- constructor error wrapping;
|
||||
- nil extractor rejection;
|
||||
- extractor key mismatch rejection;
|
||||
- sorted `RegisteredKeys`;
|
||||
- nil registry behavior.
|
||||
|
||||
Use fake extractors only. Do not add concrete extractor packages.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/extractorregistry
|
||||
go test ./internal/framework/extractorregistry
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 3: Validator Runtime Helpers
|
||||
|
||||
### Goal
|
||||
|
||||
Add shared validator decision helpers and decision-cardinality enforcement.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/validators/validators.go`
|
||||
- `internal/framework/validators/validators_test.go`
|
||||
|
||||
### Required API
|
||||
|
||||
Create package `validators`.
|
||||
|
||||
Define reason constants:
|
||||
|
||||
```go
|
||||
const (
|
||||
ReasonApproved = "approved"
|
||||
)
|
||||
```
|
||||
|
||||
Define helpers:
|
||||
|
||||
```go
|
||||
func Approved(candidateIndex int) contracts.ValidationDecision
|
||||
func Rejected(candidateIndex int, reasonCode string, message string) contracts.ValidationDecision
|
||||
func EnforceDecisionCardinality(candidates []artifacts.Candidate, decisions []contracts.ValidationDecision) error
|
||||
```
|
||||
|
||||
### Required Behavior
|
||||
|
||||
`Approved` returns an approved decision with:
|
||||
|
||||
- `CandidateIndex` set to the input;
|
||||
- `Approved` set to true;
|
||||
- `ReasonCode` set to `ReasonApproved`;
|
||||
- `Message` set to `approved`.
|
||||
|
||||
`Rejected` returns a rejected decision with:
|
||||
|
||||
- `CandidateIndex` set to the input;
|
||||
- `Approved` set to false;
|
||||
- `ReasonCode` set to the trimmed reason code;
|
||||
- `Message` set to the trimmed message.
|
||||
|
||||
`EnforceDecisionCardinality` must:
|
||||
|
||||
- require exactly one decision per candidate index;
|
||||
- reject decisions for unknown candidate indices;
|
||||
- reject duplicate decisions for the same candidate index;
|
||||
- reject missing decisions;
|
||||
- work with candidate indices, not slice positions;
|
||||
- allow an empty candidate slice only when decisions are also empty.
|
||||
|
||||
Candidate indices are expected to be unique by the time validators run. If the
|
||||
candidate slice itself contains duplicate indices, return an error.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Cover:
|
||||
|
||||
- approved helper shape;
|
||||
- rejected helper trims reason/message;
|
||||
- cardinality success with non-zero candidate indices;
|
||||
- empty candidates and empty decisions success;
|
||||
- unknown decision index error;
|
||||
- duplicate decision index error;
|
||||
- missing decision index error;
|
||||
- duplicate candidate index error.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/validators
|
||||
go test ./internal/framework/validators
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 4: Minimal Runner Types And Constructor
|
||||
|
||||
### Goal
|
||||
|
||||
Add runner package types and constructor without implementing the full run loop
|
||||
yet.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/runner/runner.go`
|
||||
- `internal/framework/runner/runner_test.go`
|
||||
|
||||
### Required API
|
||||
|
||||
Create package `runner`.
|
||||
|
||||
Define:
|
||||
|
||||
```go
|
||||
type ExtractorFactory interface {
|
||||
Build(key string) (contracts.Extractor, error)
|
||||
}
|
||||
|
||||
type RejectedArtifact struct {
|
||||
Candidate Candidate `json:"candidate"`
|
||||
ValidatorName string `json:"validator_name"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
Message string `json:"message"`
|
||||
type Runner struct {
|
||||
// unexported fields
|
||||
}
|
||||
|
||||
type RunManifest struct {
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
InputAdapter string `json:"input_adapter,omitempty"`
|
||||
SourceDigests []string `json:"source_digests,omitempty"`
|
||||
Extractors []string `json:"extractors,omitempty"`
|
||||
SchemaVersion string `json:"schema_version,omitempty"`
|
||||
ValidationStatus string `json:"validation_status,omitempty"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
func New(extractors ExtractorFactory) *Runner
|
||||
```
|
||||
|
||||
Define input/output structs:
|
||||
|
||||
```go
|
||||
type RunInput struct {
|
||||
Source *source.SourceDocument
|
||||
ExtractorKeys []string
|
||||
LLMClient contracts.StructuredLLMClient
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type RunOutput struct {
|
||||
Approved []artifacts.Artifact `json:"approved,omitempty"`
|
||||
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
Imports should include:
|
||||
Define package errors as ordinary returned errors. Do not introduce custom error
|
||||
types in this checkpoint unless tests require behavior that ordinary errors
|
||||
cannot express.
|
||||
|
||||
- `encoding/json`;
|
||||
- `time`;
|
||||
- `gitea.maximumdirect.net/eric/notarius/internal/core/source`.
|
||||
### Required Behavior
|
||||
|
||||
At this stage, `New` should only store the extractor factory.
|
||||
|
||||
If you add `Run` in this stage as a stub, it must return an explicit
|
||||
`runner is not implemented` error and must be completed in stage 5. Prefer
|
||||
adding the real `Run` method only in stage 5.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Cover:
|
||||
|
||||
- `New` accepts a fake extractor factory and returns a non-nil runner;
|
||||
- `RunInput` and `RunOutput` can be constructed with the expected fields.
|
||||
|
||||
Do not add registry imports to runner tests unless needed for interface
|
||||
assertions. Runner should depend only on framework contracts and core packages.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/runner
|
||||
go test ./internal/framework/runner
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 5: Minimal Runner Execution
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the minimal runner flow from `SourceDocument` to approved/rejected
|
||||
artifacts.
|
||||
|
||||
### Files To Update
|
||||
|
||||
- `internal/framework/runner/runner.go`
|
||||
- `internal/framework/runner/runner_test.go`
|
||||
|
||||
### Required API
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
func ArtifactFromCandidate(candidate Candidate) Artifact
|
||||
func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error)
|
||||
```
|
||||
|
||||
This helper should copy candidate fields into an approved artifact. It may share
|
||||
immutable `json.RawMessage` bytes at this checkpoint, but copying slices is
|
||||
preferred where cheap.
|
||||
### Required Behavior
|
||||
|
||||
`Run` must:
|
||||
|
||||
1. reject nil runner or nil extractor factory;
|
||||
2. validate `input.Source` with `source.ValidateDocument`;
|
||||
3. reject an empty `ExtractorKeys` list;
|
||||
4. process extractor keys in input order;
|
||||
5. build each extractor through the factory;
|
||||
6. call `Extractor.Extract` with the source, LLM client, and metadata;
|
||||
7. append extractor warnings to `RunOutput.Warnings`;
|
||||
8. normalize candidate metadata for each returned candidate;
|
||||
9. run validators in `Extractor.Validators()` order;
|
||||
10. approve candidates that survive all validators;
|
||||
11. reject candidates denied by validators;
|
||||
12. return approved/rejected artifacts in deterministic order.
|
||||
|
||||
Candidate normalization must:
|
||||
|
||||
- assign `Candidate.Index` using a global monotonically increasing counter;
|
||||
- fill empty `ExtractorKey`, `ArtifactType`, and `SchemaVersion` from the
|
||||
owning extractor;
|
||||
- return an error if a candidate's non-empty `ExtractorKey` differs from
|
||||
`Extractor.Key()`;
|
||||
- return an error if a candidate's non-empty `ArtifactType` differs from
|
||||
`Extractor.ArtifactType()`;
|
||||
- return an error if a candidate's non-empty `SchemaVersion` differs from
|
||||
`Extractor.SchemaVersion()`.
|
||||
|
||||
Validation behavior:
|
||||
|
||||
- if an extractor has no validators, all normalized candidates are approved;
|
||||
- each validator receives only currently eligible candidates;
|
||||
- each validator result must have `ValidatorName == Validator.Name()`;
|
||||
- call `validators.EnforceDecisionCardinality` for every validator result;
|
||||
- approved decisions keep candidates eligible for the next validator;
|
||||
- rejected decisions create `artifacts.RejectedArtifact` records using the
|
||||
candidate, validator name, reason code, and message;
|
||||
- candidates rejected by one validator are not passed to later validators;
|
||||
- append validator warnings to `RunOutput.Warnings`.
|
||||
|
||||
Error behavior:
|
||||
|
||||
- return current partial `RunOutput` plus an error for setup, extraction,
|
||||
validation, cardinality, or metadata mismatch failures;
|
||||
- wrap errors with enough context to identify the extractor key or validator
|
||||
name.
|
||||
|
||||
Do not validate source references inside the runner in checkpoint 2. Source
|
||||
reference validation will be a deterministic validator in a later checkpoint.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Cover:
|
||||
|
||||
- `ArtifactFromCandidate` preserves extractor key, artifact type, schema
|
||||
version, payload, source refs, and metadata;
|
||||
- JSON marshaling uses the expected field names;
|
||||
- empty optional manifest fields are omitted.
|
||||
- nil runner/factory error;
|
||||
- invalid source document error;
|
||||
- empty extractor list error;
|
||||
- extractor keys run in configured order;
|
||||
- runner assigns global candidate indices across extractors;
|
||||
- runner fills empty candidate extractor metadata;
|
||||
- candidate extractor-key mismatch errors;
|
||||
- candidate artifact-type mismatch errors;
|
||||
- candidate schema-version mismatch errors;
|
||||
- no validators approves all candidates;
|
||||
- validator approval produces approved artifacts;
|
||||
- validator rejection produces rejected artifacts and removes candidate from
|
||||
later validators;
|
||||
- validator name mismatch errors;
|
||||
- validator cardinality error is surfaced;
|
||||
- extractor warnings and validator warnings are collected;
|
||||
- partial output is returned when a later extractor or validator fails.
|
||||
|
||||
Do not add D&D-specific payload structs.
|
||||
Use fake extractors, validators, and factories only.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/core/artifacts
|
||||
go test ./internal/core/artifacts
|
||||
gofmt -w internal/framework/runner
|
||||
go test ./internal/framework/runner
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 4: Framework Contract Types
|
||||
## Stage 6: Registry And Runner Integration Tests
|
||||
|
||||
### Goal
|
||||
|
||||
Define the interfaces and request/response types later checkpoints will build
|
||||
against.
|
||||
Prove the extractor registry and runner compose without adding real extractors.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/contracts/contracts.go`
|
||||
- `internal/framework/contracts/contracts_test.go`
|
||||
|
||||
### Required Types And Interfaces
|
||||
|
||||
Create package `contracts`.
|
||||
|
||||
Define LLM-neutral message and completion types:
|
||||
|
||||
```go
|
||||
type LLMMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type StructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name"`
|
||||
Messages []LLMMessage `json:"messages"`
|
||||
Model string `json:"model,omitempty"`
|
||||
ResponseSchemaName string `json:"response_schema_name,omitempty"`
|
||||
ResponseSchema json.RawMessage `json:"response_schema,omitempty"`
|
||||
}
|
||||
|
||||
type StructuredCompletionResponse struct {
|
||||
Content json.RawMessage `json:"content"`
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
PromptTokens int `json:"prompt_tokens,omitempty"`
|
||||
CompletionTokens int `json:"completion_tokens,omitempty"`
|
||||
TotalTokens int `json:"total_tokens,omitempty"`
|
||||
}
|
||||
|
||||
type StructuredLLMClient interface {
|
||||
CompleteStructured(ctx context.Context, req StructuredCompletionRequest, out any) (StructuredCompletionResponse, error)
|
||||
}
|
||||
```
|
||||
|
||||
Define input adapter contract:
|
||||
|
||||
```go
|
||||
type ParseRequest struct {
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Path string `json:"path,omitempty"`
|
||||
Raw []byte `json:"-"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type InputAdapter interface {
|
||||
Key() string
|
||||
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
|
||||
}
|
||||
```
|
||||
|
||||
Define extractor contract:
|
||||
|
||||
```go
|
||||
type ExtractionRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ExtractionResult struct {
|
||||
Candidates []artifacts.Candidate `json:"candidates,omitempty"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type Extractor interface {
|
||||
Key() string
|
||||
ArtifactType() string
|
||||
SchemaVersion() string
|
||||
Validators() []Validator
|
||||
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
|
||||
}
|
||||
```
|
||||
|
||||
Define validator contract:
|
||||
|
||||
```go
|
||||
type ValidationRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
Candidates []artifacts.Candidate `json:"candidates"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ValidationDecision struct {
|
||||
CandidateIndex int `json:"candidate_index"`
|
||||
Approved bool `json:"approved"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
Message string `json:"message"`
|
||||
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
|
||||
}
|
||||
|
||||
type ValidationResult struct {
|
||||
ValidatorName string `json:"validator_name"`
|
||||
Decisions []ValidationDecision `json:"decisions"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type Validator interface {
|
||||
Name() string
|
||||
Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error)
|
||||
}
|
||||
```
|
||||
|
||||
Define shared warning type:
|
||||
|
||||
```go
|
||||
type Warning struct {
|
||||
Scope string `json:"scope,omitempty"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
```
|
||||
|
||||
### Required Tests
|
||||
|
||||
Use compile-time interface assertions with fake implementations:
|
||||
|
||||
```go
|
||||
var _ InputAdapter = fakeAdapter{}
|
||||
var _ Extractor = fakeExtractor{}
|
||||
var _ Validator = fakeValidator{}
|
||||
var _ StructuredLLMClient = fakeLLMClient{}
|
||||
```
|
||||
|
||||
Also test that a fake extractor can return a candidate and expose a validator
|
||||
without importing concrete adapter or extractor packages.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/contracts
|
||||
go test ./internal/framework/contracts
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 5: Cross-Package Contract Composition Tests
|
||||
|
||||
### Goal
|
||||
|
||||
Prove the checkpoint 1 contracts compose at a minimal end-to-end type level
|
||||
without implementing checkpoint 2 runner behavior.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/contracts/composition_test.go`
|
||||
|
||||
This can live in package `contracts_test` to verify public contracts from the
|
||||
outside.
|
||||
- `internal/framework/runner/registry_integration_test.go`
|
||||
|
||||
### Required Test Scenario
|
||||
|
||||
Create fakes in the test:
|
||||
Use `extractorregistry.New()` to register two fake extractor constructors.
|
||||
|
||||
1. fake adapter parses a `ParseRequest` into a valid `SourceDocument`;
|
||||
2. fake extractor receives that source and returns one `artifacts.Candidate`;
|
||||
3. fake validator approves that candidate;
|
||||
4. test manually calls those fakes in sequence.
|
||||
Run the runner with:
|
||||
|
||||
- a valid `SourceDocument`;
|
||||
- extractor keys in a deliberate order;
|
||||
- fake extractors that each return one candidate;
|
||||
- fake validators that approve or reject candidates.
|
||||
|
||||
Assertions:
|
||||
|
||||
- parsed source passes `source.ValidateDocument`;
|
||||
- candidate source refs pass `source.ValidateRef`;
|
||||
- validator returns one approved decision for the candidate index;
|
||||
- no test type or assertion depends on transcript or D&D concepts.
|
||||
|
||||
Do not add a registry or runner in this stage. Those belong to checkpoint 2.
|
||||
- registered fake extractors are built by key;
|
||||
- extractor execution follows configured key order;
|
||||
- approved artifacts are in deterministic order;
|
||||
- rejected artifacts are in deterministic order;
|
||||
- no concrete adapter or extractor packages are imported.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/contracts
|
||||
go test ./internal/framework/contracts
|
||||
gofmt -w internal/framework/runner
|
||||
go test ./internal/framework/runner
|
||||
go test ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
## Stage 6: Final Checkpoint 1 Review Pass
|
||||
## Stage 7: Final Checkpoint 2 Review Pass
|
||||
|
||||
### Goal
|
||||
|
||||
Clean up naming, formatting, and accidental scope creep before checkpoint 1 is
|
||||
Clean up naming, formatting, and accidental scope creep before checkpoint 2 is
|
||||
considered complete.
|
||||
|
||||
### Required Review
|
||||
@@ -504,8 +529,11 @@ Check:
|
||||
combat;
|
||||
- no concrete adapter or extractor package exists;
|
||||
- no LLM provider code exists;
|
||||
- no prompt, response schema, diagnostics, config, or source chunking package
|
||||
was added;
|
||||
- no third-party dependency was added;
|
||||
- `README.md` was not updated to describe unimplemented behavior;
|
||||
- runner operates on `SourceDocument`;
|
||||
- input adapter registry is not wired into runner yet;
|
||||
- roadmap docs remain the only place describing future behavior.
|
||||
|
||||
### Required Validation
|
||||
@@ -513,12 +541,15 @@ Check:
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w cmd internal
|
||||
gofmt -w internal/framework
|
||||
go test ./...
|
||||
go build ./cmd/notarius
|
||||
git status --short
|
||||
```
|
||||
|
||||
Remove the root `./notarius` binary produced by `go build ./cmd/notarius` before
|
||||
finishing the implementation turn.
|
||||
|
||||
The implementation response for this checkpoint should summarize:
|
||||
|
||||
- files added;
|
||||
@@ -527,8 +558,9 @@ The implementation response for this checkpoint should summarize:
|
||||
|
||||
## Open Questions
|
||||
|
||||
No blocking open questions remain for checkpoint 1.
|
||||
No blocking open questions remain for checkpoint 2.
|
||||
|
||||
The plan intentionally chooses a narrow generic model and fake-only composition
|
||||
tests. More complete orchestration, registries, real adapters, real extractors,
|
||||
and LLM runtime behavior are deferred to later checkpoint implementation plans.
|
||||
The plan intentionally keeps raw input parsing outside the runner. The input
|
||||
adapter registry is added and tested now because it is part of framework
|
||||
composition, but concrete parsing and adapter-runner wiring remain deferred
|
||||
until the Seriatim adapter checkpoint.
|
||||
|
||||
Reference in New Issue
Block a user