Clean up completed documentation roadmaps
This commit is contained in:
@@ -1,888 +0,0 @@
|
||||
# Implementation Plan: MVP
|
||||
|
||||
## Status
|
||||
|
||||
This is the staged implementation plan for the active MVP roadmap:
|
||||
[`mvp.md`](mvp.md).
|
||||
|
||||
The target audience is an LLM coding agent. Implement the stages in order.
|
||||
Each stage should leave the repository compiling and tested. Do not skip ahead
|
||||
to later stages unless the current stage's done criteria are satisfied.
|
||||
|
||||
## Policy Context
|
||||
|
||||
Follow:
|
||||
|
||||
- [`../policy/architecture.md`](../policy/architecture.md)
|
||||
- [`../policy/documentation.md`](../policy/documentation.md)
|
||||
- [`mvp.md`](mvp.md)
|
||||
- [`initial-architecture.md`](initial-architecture.md)
|
||||
|
||||
Required boundaries:
|
||||
|
||||
- framework packages must remain source-agnostic and domain-agnostic;
|
||||
- source-format behavior belongs in input modules;
|
||||
- D&D spell behavior, prompt assets, response schema assets, and stable
|
||||
prompt/schema identifiers belong in `internal/modules/extract/dnd/spells`;
|
||||
- stage business logic belongs under `internal/modules/<stage>/...` unless it
|
||||
is genuinely tiny shared framework plumbing;
|
||||
- structural pipeline selection must remain config-driven;
|
||||
- `--only` may select artifact lanes but must not alter pipeline structure;
|
||||
- output-stage warnings are out-of-band from artifact payloads and must be
|
||||
available to CLI/diagnostics;
|
||||
- keep planned documentation in `docs/roadmap/` until MVP behavior exists.
|
||||
|
||||
## Global Implementation Decisions
|
||||
|
||||
- Add no new third-party dependencies.
|
||||
- Keep YAML config version `1` unless a user-visible config syntax change is
|
||||
unavoidable. New module options can use existing binding `options`.
|
||||
- Use the existing six-stage workflow:
|
||||
`input -> chunk -> extract -> merge -> normalize -> output`.
|
||||
- Use production CLI wiring in `internal/cli` for the MVP instead of adding a
|
||||
new app package. The CLI may compose modules, but it must not own module
|
||||
business logic.
|
||||
- Keep `notarius run` serial over chunks for the MVP. The contracts and LLM
|
||||
scheduler should still permit later parallel execution.
|
||||
- Use one effective LLM profile per MVP run. The current runner accepts one
|
||||
`StructuredLLMClient`, so a selected pipeline with multiple distinct effective
|
||||
LLM profile IDs should fail clearly until multi-client runtime support is
|
||||
intentionally added.
|
||||
- Use the existing OpenAI-compatible client for real runs.
|
||||
- Add a scheduled LLM client wrapper so every structured completion passes
|
||||
through the configured scheduler.
|
||||
- Use the existing `seriatim` input module and `dnd/spells` extractor module.
|
||||
- Implement production default modules with these keys:
|
||||
- `generic` chunker;
|
||||
- `appendorder` merger;
|
||||
- `noop` normalizer;
|
||||
- `json` output encoder.
|
||||
- Put production default modules under:
|
||||
- `internal/modules/chunk/generic`;
|
||||
- `internal/modules/merge/appendorder`;
|
||||
- `internal/modules/normalize/noop`;
|
||||
- `internal/modules/output/json`.
|
||||
- The output encoder should return logical output files; the CLI/application
|
||||
layer should write those files to disk. Encoders should not own filesystem
|
||||
side effects.
|
||||
- Use an output directory per run. The MVP default output root should be
|
||||
`./notarius-output`, overrideable by `--output-dir`.
|
||||
- File writes for durable output should be atomic where practical: write to a
|
||||
temporary file in the target directory, then rename.
|
||||
- Use synthetic fixtures only. Do not add private campaign transcript content,
|
||||
real API keys, or private infrastructure values.
|
||||
|
||||
## Stage 1: Move D&D Prompt And Schema Assets Into The Spells Module
|
||||
|
||||
### Goal
|
||||
|
||||
Restore the intended framework/domain boundary before building additional MVP
|
||||
functionality.
|
||||
|
||||
`internal/framework/llm` and `internal/framework/prompt` should provide generic
|
||||
asset loading, metadata, and rendering primitives. They must not define
|
||||
D&D-specific prompt IDs, response schema keys, asset paths, or tests.
|
||||
|
||||
### Files To Update Or Move
|
||||
|
||||
Expected files:
|
||||
|
||||
- `internal/framework/llm/schema_registry.go`
|
||||
- `internal/framework/llm/schema_registry_test.go`
|
||||
- `internal/framework/llm/assets/schemas/dnd_spells.v1.json`
|
||||
- `internal/framework/prompt/registry.go`
|
||||
- `internal/framework/prompt/render.go`
|
||||
- `internal/framework/prompt/render_test.go`
|
||||
- `internal/framework/prompt/assets/dnd/spells/system.md`
|
||||
- `internal/framework/prompt/assets/dnd/spells/user.md`
|
||||
- `internal/modules/extract/dnd/spells/extractor.go`
|
||||
- `internal/modules/extract/dnd/spells/prompt.go`
|
||||
- `internal/modules/extract/dnd/spells/prompt_test.go`
|
||||
- `internal/modules/extract/dnd/spells/schema_test.go`
|
||||
- `docs/integrations/dnd-spells.md`, only if prompt/schema ownership text needs
|
||||
to be corrected.
|
||||
|
||||
### Required Design
|
||||
|
||||
Refactor `internal/framework/llm` so it can load schemas from caller-owned
|
||||
embedded files.
|
||||
|
||||
Add or expose a generic constructor similar to:
|
||||
|
||||
```go
|
||||
func LoadResponseSchema(fsys fs.FS, def ResponseSchemaDefinition) (ResponseSchema, error)
|
||||
```
|
||||
|
||||
where `ResponseSchemaDefinition` carries:
|
||||
|
||||
- key;
|
||||
- ID;
|
||||
- version;
|
||||
- name;
|
||||
- asset path.
|
||||
|
||||
The existing framework registry may keep test schemas, but it must not include
|
||||
`DNDSpellsSchemaKey` or `dnd_spells.v1.json`.
|
||||
|
||||
Refactor `internal/framework/prompt` so it can compile/render prompt pairs from
|
||||
caller-owned embedded files.
|
||||
|
||||
Add or expose a generic constructor/rendering type similar to:
|
||||
|
||||
```go
|
||||
type Bundle struct { ... }
|
||||
func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error)
|
||||
func (b *Bundle) RenderUserSystem(data any) (system string, user string, metadata Metadata, err error)
|
||||
```
|
||||
|
||||
The framework prompt package may continue to own shared hardening text if that
|
||||
is useful, but it must not include `DNDSpellsPromptID` or D&D prompt paths.
|
||||
|
||||
Move D&D prompt and schema assets under `internal/modules/extract/dnd/spells`.
|
||||
Recommended paths:
|
||||
|
||||
```text
|
||||
internal/modules/extract/dnd/spells/assets/prompts/system.md
|
||||
internal/modules/extract/dnd/spells/assets/prompts/user.md
|
||||
internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json
|
||||
```
|
||||
|
||||
The spells package should define its own stable identifiers:
|
||||
|
||||
```go
|
||||
const PromptID = "dnd.spells"
|
||||
const ResponseSchemaKey = "dnd_spells"
|
||||
const ResponseSchemaID = "notarius.dnd.spells"
|
||||
const ResponseSchemaName = "notarius_dnd_spells_v1"
|
||||
```
|
||||
|
||||
The spells extractor must call module-owned prompt/schema helpers and pass only
|
||||
generic framework values into the LLM client.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- Framework LLM schema tests prove test schemas still load, sort, clone, and
|
||||
omit raw schema content from diagnostics.
|
||||
- Framework LLM schema tests prove looking up `dnd_spells` in the framework
|
||||
registry fails.
|
||||
- Framework prompt tests prove test prompts still render and missing template
|
||||
data still errors.
|
||||
- Framework prompt tests contain no D&D prompt assertions.
|
||||
- Spells package schema tests load the module-owned D&D schema and verify:
|
||||
- key;
|
||||
- ID;
|
||||
- version;
|
||||
- response schema name;
|
||||
- valid JSON;
|
||||
- clone/mutation safety;
|
||||
- diagnostics omit raw schema content.
|
||||
- Spells package prompt tests render the module-owned prompt and verify
|
||||
hardening text and prompt metadata.
|
||||
- Existing spells extractor tests still pass without importing framework-owned
|
||||
D&D constants.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/llm internal/framework/prompt internal/modules/extract/dnd/spells
|
||||
go test ./internal/framework/llm ./internal/framework/prompt ./internal/modules/extract/dnd/spells
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 2: Add MVP Manifest And Logical Output File Contracts
|
||||
|
||||
### Goal
|
||||
|
||||
Make output and manifest contracts capable of representing the MVP's durable
|
||||
run output before implementing the production JSON encoder or CLI writing.
|
||||
|
||||
### Files To Update
|
||||
|
||||
Expected files:
|
||||
|
||||
- `internal/framework/contracts/contracts.go`
|
||||
- `internal/core/artifacts/*.go`
|
||||
- `internal/framework/pipeline/runner.go`
|
||||
- `internal/framework/pipeline/runner_test.go`
|
||||
- `internal/framework/contracts/contracts_test.go`
|
||||
- `docs/policy/architecture.md`, only if the implemented output contract
|
||||
requires clarifying policy text.
|
||||
|
||||
### Required Design
|
||||
|
||||
Extend the output contract to support logical files:
|
||||
|
||||
```go
|
||||
type OutputFile struct {
|
||||
Name string `json:"name"`
|
||||
ContentType string `json:"content_type,omitempty"`
|
||||
Bytes []byte `json:"-"`
|
||||
}
|
||||
|
||||
type OutputResult struct {
|
||||
Files []OutputFile `json:"files,omitempty"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
Remove or stop using the single `OutputResult.Bytes` / `ContentType` path once
|
||||
all tests are updated. If keeping those fields temporarily reduces churn, mark
|
||||
them as legacy in comments and make the runner prefer `Files`.
|
||||
|
||||
Add matching fields to `pipeline.RunOutput`:
|
||||
|
||||
```go
|
||||
OutputFiles []contracts.OutputFile `json:"-"`
|
||||
```
|
||||
|
||||
The runner should collect output-stage warnings exactly as it does now, after
|
||||
calling the output encoder.
|
||||
|
||||
Define safe logical file names:
|
||||
|
||||
- names are slash-separated relative paths;
|
||||
- names must not be empty, absolute, contain `..`, or contain `\`;
|
||||
- names are validated before the runner returns them;
|
||||
- file names are sorted deterministically by the encoder that creates them.
|
||||
|
||||
Extend manifest data enough for MVP provenance:
|
||||
|
||||
- add `RunManifest.LLMProfiles []LLMProfileManifest`;
|
||||
- add `ArtifactLaneManifest.Metadata map[string]any`;
|
||||
- add `RunManifest.StartedAt`, `CompletedAt`, and `RunID` population support
|
||||
in the runner input/output path.
|
||||
|
||||
Recommended structs:
|
||||
|
||||
```go
|
||||
type LLMProfileManifest struct {
|
||||
ID string `json:"id"`
|
||||
Provider string `json:"provider,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
Add an optional metadata interface for modules:
|
||||
|
||||
```go
|
||||
type ManifestMetadataProvider interface {
|
||||
ManifestMetadata() map[string]any
|
||||
}
|
||||
```
|
||||
|
||||
When a stage module implements the interface, the runner should include that
|
||||
metadata in the appropriate manifest area. For the MVP, the D&D spells extractor
|
||||
will use this to report prompt and response schema identifiers/hashes on its
|
||||
artifact lane.
|
||||
|
||||
Update `pipeline.RunInput` to accept:
|
||||
|
||||
- `RunID string`;
|
||||
- `StartedAt time.Time`;
|
||||
- `LLMProfiles []artifacts.LLMProfileManifest`.
|
||||
|
||||
The runner should set `CompletedAt` when the run finishes or fails after a
|
||||
manifest has been initialized.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- `contracts.OutputFile` JSON shape omits bytes and includes name/content type.
|
||||
- Runner rejects unsafe output file names returned by an encoder.
|
||||
- Runner preserves output warnings out-of-band.
|
||||
- Runner output contains logical files returned by the encoder.
|
||||
- Manifest includes run ID, started/completed timestamps when supplied or
|
||||
generated.
|
||||
- Manifest includes LLM profile metadata supplied in `RunInput`.
|
||||
- Manifest includes extractor-provided lane metadata when the extractor
|
||||
implements `ManifestMetadataProvider`.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/contracts internal/core/artifacts internal/framework/pipeline
|
||||
go test ./internal/framework/contracts ./internal/core/artifacts ./internal/framework/pipeline
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 3: Implement Production Default Stage Modules
|
||||
|
||||
### Goal
|
||||
|
||||
Make pipeline defaults real production modules instead of test-only fakes or
|
||||
framework-only helpers.
|
||||
|
||||
### Files To Add Or Update
|
||||
|
||||
Expected packages:
|
||||
|
||||
- `internal/modules/chunk/generic`
|
||||
- `internal/modules/merge/appendorder`
|
||||
- `internal/modules/normalize/noop`
|
||||
- `internal/modules/output/json`
|
||||
|
||||
Expected framework cleanup:
|
||||
|
||||
- `internal/framework/pipeline/generic_stages.go`
|
||||
- pipeline tests that currently instantiate framework `AppendOrderMerger` or
|
||||
`NoopNormalizer`.
|
||||
|
||||
### Required Design
|
||||
|
||||
#### `generic` chunker
|
||||
|
||||
Package: `internal/modules/chunk/generic`
|
||||
|
||||
Key: `generic`
|
||||
|
||||
Module spec:
|
||||
|
||||
- stage: `chunk`;
|
||||
- requires: `source.transcript` is **not** required;
|
||||
- provides: `chunks`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- accepts any valid `SourceDocument`;
|
||||
- preserves source-unit order;
|
||||
- returns stable chunk IDs: `chunk-000001`, `chunk-000002`, and so on;
|
||||
- copies source units defensively;
|
||||
- adds chunk metadata:
|
||||
- `start_unit_id`;
|
||||
- `end_unit_id`;
|
||||
- `unit_count`.
|
||||
|
||||
Options:
|
||||
|
||||
- `max_units`: positive integer, default `50`;
|
||||
- `overlap_units`: non-negative integer, default `0`, must be less than
|
||||
`max_units`.
|
||||
|
||||
If the source has no units, return a clear error. If options have the wrong type
|
||||
or invalid values, return a clear module-specific error.
|
||||
|
||||
#### `appendorder` merger
|
||||
|
||||
Package: `internal/modules/merge/appendorder`
|
||||
|
||||
Key: `appendorder`
|
||||
|
||||
Module spec:
|
||||
|
||||
- stage: `merge`;
|
||||
- requires: no artifact-type-specific capability;
|
||||
- provides: `merged`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- preserves chunk order as provided by the runner;
|
||||
- preserves candidate order within each chunk;
|
||||
- defensively copies candidates, payloads, source refs, and metadata;
|
||||
- does not merge, deduplicate, or rewrite source references.
|
||||
|
||||
#### `noop` normalizer
|
||||
|
||||
Package: `internal/modules/normalize/noop`
|
||||
|
||||
Key: `noop`
|
||||
|
||||
Module spec:
|
||||
|
||||
- stage: `normalize`;
|
||||
- requires: `merged`;
|
||||
- provides: `normalized`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- defensively copies candidates;
|
||||
- does not deduplicate, rewrite, or validate domain content.
|
||||
|
||||
#### `json` output encoder
|
||||
|
||||
Package: `internal/modules/output/json`
|
||||
|
||||
Key: `json`
|
||||
|
||||
Module spec:
|
||||
|
||||
- stage: `output`;
|
||||
- requires: `normalized`;
|
||||
- provides: `encoded`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- returns logical output files:
|
||||
- `index.json`;
|
||||
- `manifest.json`;
|
||||
- `artifacts/<artifact_type>.json` for each approved artifact type;
|
||||
- `rejected.json`;
|
||||
- `warnings.json`.
|
||||
- groups approved artifacts by `Artifact.ArtifactType`;
|
||||
- sorts artifact-type file names by artifact type;
|
||||
- preserves artifact order within each artifact type according to runner order;
|
||||
- pretty-prints JSON with two-space indentation and trailing newline;
|
||||
- uses content type `application/json`;
|
||||
- includes rejected artifacts and warnings even when the arrays are empty;
|
||||
- does not include output warnings inside artifact payloads.
|
||||
|
||||
File-name safety:
|
||||
|
||||
- artifact type may contain dots and hyphens;
|
||||
- replace any character outside `[A-Za-z0-9._-]` with `_` for artifact file
|
||||
names;
|
||||
- if sanitization produces an empty name, return an error.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- Generic chunker tests cover defaults, exact chunk boundaries, overlap,
|
||||
invalid options, empty source, defensive copies, and stable IDs.
|
||||
- Append-order merge tests cover ordering and defensive copies.
|
||||
- Noop normalizer tests cover pass-through behavior and defensive copies.
|
||||
- JSON output tests cover all logical files, grouping, sorted filenames,
|
||||
rejected/warnings presence, pretty JSON, unsafe artifact type sanitization,
|
||||
and no mutation of inputs.
|
||||
- Pipeline config tests using defaults resolve when these module specs are
|
||||
registered.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/modules/chunk/generic internal/modules/merge/appendorder internal/modules/normalize/noop internal/modules/output/json internal/framework/pipeline
|
||||
go test ./internal/modules/chunk/generic ./internal/modules/merge/appendorder ./internal/modules/normalize/noop ./internal/modules/output/json
|
||||
go test ./internal/framework/pipeline
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 4: Add Production CLI Catalog And Runtime Wiring
|
||||
|
||||
### Goal
|
||||
|
||||
Make implemented modules selectable by real CLI commands without test-injected
|
||||
catalogs.
|
||||
|
||||
### Files To Add Or Update
|
||||
|
||||
Expected files:
|
||||
|
||||
- `internal/cli/run.go`
|
||||
- new `internal/cli/catalog.go` or equivalent;
|
||||
- `internal/cli/run_test.go`;
|
||||
- module registry tests as needed.
|
||||
|
||||
### Required Design
|
||||
|
||||
Add production wiring in `internal/cli`:
|
||||
|
||||
```go
|
||||
func productionRegistries() (pipeline.Registries, error)
|
||||
func productionCatalog() (pipeline.ModuleCatalog, error)
|
||||
```
|
||||
|
||||
The production wiring must register:
|
||||
|
||||
- input: `seriatim`;
|
||||
- chunk: `generic`;
|
||||
- extract: `dnd/spells`;
|
||||
- merge: `appendorder`;
|
||||
- normalize: `noop`;
|
||||
- output: `json`.
|
||||
|
||||
Keep all business logic in module packages. `internal/cli` should only compose
|
||||
registries/catalogs and command behavior.
|
||||
|
||||
Update `cli.Options` so tests may inject registries/catalog/runtime without
|
||||
disabling production defaults unintentionally.
|
||||
|
||||
Recommended option fields:
|
||||
|
||||
```go
|
||||
type Options struct {
|
||||
Catalog pipeline.ModuleCatalog
|
||||
Registries pipeline.Registries
|
||||
LLMClientFactory LLMClientFactory
|
||||
LookupEnv func(string) (string, bool)
|
||||
Now func() time.Time
|
||||
}
|
||||
```
|
||||
|
||||
If `Catalog` or `Registries` is empty in normal `Run`, use production wiring.
|
||||
If tests provide either, use the provided value.
|
||||
|
||||
Define `LLMClientFactory` in `internal/cli` or a small local file:
|
||||
|
||||
```go
|
||||
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
|
||||
```
|
||||
|
||||
The production factory should:
|
||||
|
||||
- read the selected LLM profile from effective config;
|
||||
- construct `llm.OpenAICompatibleClient`;
|
||||
- construct `llm.Scheduler` using the most specific configured concurrency:
|
||||
profile `max_concurrency` if set, otherwise global `concurrency.total_llm`,
|
||||
otherwise `1`;
|
||||
- wrap the client in a scheduled client so every completion acquires/releases a
|
||||
scheduler permit;
|
||||
- return manifest-safe LLM metadata with profile ID, provider, and model.
|
||||
|
||||
Add a scheduled client wrapper in `internal/framework/llm` if it does not
|
||||
already exist:
|
||||
|
||||
```go
|
||||
func NewScheduledClient(client contracts.StructuredLLMClient, scheduler *Scheduler) contracts.StructuredLLMClient
|
||||
```
|
||||
|
||||
### Required CLI Behavior
|
||||
|
||||
- `notarius config validate --config <file> --pipeline <id>` uses the
|
||||
production catalog by default.
|
||||
- `notarius pipelines list --config <file>` still lists configured pipeline
|
||||
IDs and validates config shape.
|
||||
- `notarius pipelines list --config <file> --json` remains stable.
|
||||
|
||||
Do not implement `notarius run` in this stage.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- Production catalog includes the six MVP modules and their module specs.
|
||||
- `config validate --pipeline` succeeds for a real MVP config fixture using no
|
||||
injected catalog.
|
||||
- Unknown module keys still fail with stage/pipeline context.
|
||||
- Production LLM client factory rejects missing/invalid LLM profiles with clear
|
||||
errors.
|
||||
- Scheduled client wrapper enforces scheduler use and propagates errors.
|
||||
- Existing CLI tests using injected catalogs still pass.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/cli internal/framework/llm
|
||||
go test ./internal/cli ./internal/framework/llm
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
## Stage 5: Implement `notarius run` Without Durable File Writing
|
||||
|
||||
### Goal
|
||||
|
||||
Add the user-facing run command and prove it can drive the configured pipeline
|
||||
with injected fake runtime pieces. This stage should return/run data in memory
|
||||
or through test buffers, but durable file writing may be completed in Stage 6.
|
||||
|
||||
### Files To Update
|
||||
|
||||
Expected files:
|
||||
|
||||
- `internal/cli/run.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `cmd/notarius/main.go`, only if command wiring requires it.
|
||||
|
||||
### Required Command Shape
|
||||
|
||||
Support:
|
||||
|
||||
```sh
|
||||
notarius run <pipeline-id> --input path/to/source.json
|
||||
notarius run <pipeline-id> --input path/to/source.json --only spells
|
||||
```
|
||||
|
||||
Supported flags:
|
||||
|
||||
- `--config path`;
|
||||
- `--input path`, required;
|
||||
- `--only lane-a,lane-b`;
|
||||
- `--output-dir path`, parsed and passed through metadata for Stage 6;
|
||||
- `--diagnostics-dir path`, overrides config diagnostics work dir for this run;
|
||||
- `--llm-profile profile-id`, operational override for MVP runs.
|
||||
|
||||
Do not add flags for structural module selection, such as `--extractor`,
|
||||
`--chunker`, `--merge`, or `--output`.
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- Missing pipeline ID returns exit code `2`.
|
||||
- Missing `--input` returns exit code `2`.
|
||||
- Unknown flags return exit code `2`.
|
||||
- Config/load/resolve/runtime failures return exit code `1`.
|
||||
- Successful runs return exit code `0`.
|
||||
- `--only` uses existing lane selection behavior.
|
||||
- Extend `config.ResolveInput` with `LLMProfileOverride string` or an
|
||||
equivalent option. When `--llm-profile` is provided, apply it to every
|
||||
resolved module binding before the resolved pipeline digest is computed. This
|
||||
keeps the override operational rather than structural while still making the
|
||||
effective pipeline digest truthful.
|
||||
- After resolution, collect the distinct effective LLM profile IDs used by the
|
||||
selected pipeline. For the MVP, require exactly one distinct profile ID and
|
||||
fail clearly if more than one is present.
|
||||
- Input file is read as raw bytes and passed to the runner with path metadata.
|
||||
- Source ID defaults to the input path or basename only if the input adapter
|
||||
needs one; do not invent transcript-specific source IDs in the CLI.
|
||||
- The command resolves the selected pipeline with the production catalog.
|
||||
- The command constructs the LLM client through `LLMClientFactory`.
|
||||
- The command invokes `pipeline.New(registries).Run(...)`.
|
||||
- The command prints a concise success message that includes at least:
|
||||
- pipeline ID;
|
||||
- approved artifact count;
|
||||
- rejected artifact count.
|
||||
- If warnings exist, print a concise warning count to stderr.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Use fake LLM/runtime injection; do not call external services.
|
||||
|
||||
- Missing pipeline ID.
|
||||
- Missing input flag.
|
||||
- Unknown pipeline.
|
||||
- Unknown `--only` lane.
|
||||
- Invalid input file path.
|
||||
- Successful run invokes runner path through real registries and fake LLM.
|
||||
- `--only spells` runs only the selected lane.
|
||||
- LLM factory failure is reported clearly.
|
||||
- Validation rejection produces a failed/non-zero or successful-with-rejections
|
||||
behavior according to current runner semantics. For MVP, keep runner
|
||||
semantics: a run with rejected artifacts completes successfully with
|
||||
`ValidationStatus` set to `rejected`, unless an error occurs.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/cli
|
||||
go test ./internal/cli
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
## Stage 6: Write Durable Output And Diagnostics For `notarius run`
|
||||
|
||||
### Goal
|
||||
|
||||
Complete the MVP run workflow by writing output files and diagnostics.
|
||||
|
||||
### Files To Update
|
||||
|
||||
Expected files:
|
||||
|
||||
- `internal/cli/run.go`
|
||||
- `internal/cli/run_test.go`
|
||||
- `internal/core/diagnostics/*.go`, only if helper methods are needed.
|
||||
|
||||
### Required Design
|
||||
|
||||
Output directory behavior:
|
||||
|
||||
- default root: `./notarius-output`;
|
||||
- override: `--output-dir`;
|
||||
- each run writes to `<output-root>/<run-id>/`;
|
||||
- run ID comes from diagnostics run directory when available or from a
|
||||
generated UTC nanosecond timestamp using the same style as diagnostics;
|
||||
- create directories with `0755`;
|
||||
- write files with `0644`;
|
||||
- write each file atomically where practical.
|
||||
|
||||
Logical output files from `pipeline.RunOutput.OutputFiles` should be written
|
||||
under the run output directory. Reject unsafe logical file names before writing:
|
||||
|
||||
- empty;
|
||||
- absolute;
|
||||
- contains `..`;
|
||||
- contains backslash;
|
||||
- escapes the run output directory after path cleaning.
|
||||
|
||||
Diagnostics behavior:
|
||||
|
||||
- create a diagnostics run directory at command start unless retention is
|
||||
`never` and the implementation can still reliably capture failures; simplest
|
||||
MVP behavior is to create it and then apply retention at the end;
|
||||
- write invocation metadata;
|
||||
- write redacted effective config;
|
||||
- write resolved pipeline;
|
||||
- write run manifest;
|
||||
- write warnings;
|
||||
- write run report containing output path, counts, and validation status;
|
||||
- write error log on failure;
|
||||
- apply retention with existing diagnostics policy.
|
||||
|
||||
`--diagnostics-dir` should override `Config.Diagnostics.WorkDir` after file and
|
||||
environment config have been applied, without changing structural pipeline
|
||||
definition or pipeline digest.
|
||||
|
||||
Success output:
|
||||
|
||||
- stdout includes the durable output run directory path;
|
||||
- stderr includes warning count when warnings are present;
|
||||
- no raw prompt text, raw API keys, or large source payloads should be printed.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- Successful `notarius run` writes output files under a temp output directory.
|
||||
- Output write rejects unsafe logical file names from a fake encoder.
|
||||
- Writes are atomic enough that no temporary files remain after success.
|
||||
- Diagnostics artifacts are written on success.
|
||||
- Error log is written on failure after diagnostics directory creation.
|
||||
- Retention `never` removes successful warning-free diagnostics directories.
|
||||
- Warnings are present in diagnostics and are reported to stderr.
|
||||
- `--diagnostics-dir` overrides config diagnostics directory.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/cli internal/core/diagnostics
|
||||
go test ./internal/cli ./internal/core/diagnostics
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
## Stage 7: Add MVP Fixtures And End-To-End Acceptance Coverage
|
||||
|
||||
### Goal
|
||||
|
||||
Make the MVP path continuously testable without network access.
|
||||
|
||||
### Files To Add Or Update
|
||||
|
||||
Expected fixtures:
|
||||
|
||||
- `examples/seriatim-minimal-transcript.json`, if the example can be kept
|
||||
accurate before the deferred documentation pass;
|
||||
- `examples/dnd-spells.config.yml`, if config examples are tested in this
|
||||
stage;
|
||||
- or equivalent `internal/cli/testdata/...` fixtures if examples are deferred.
|
||||
|
||||
Expected tests:
|
||||
|
||||
- `internal/cli/run_test.go`
|
||||
- `internal/modules/extract/dnd/spells/runner_test.go`
|
||||
- config tests as needed.
|
||||
|
||||
### Required Design
|
||||
|
||||
Add a maintained MVP config fixture:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
llm_profiles:
|
||||
default:
|
||||
provider: openai-compatible
|
||||
base_url: http://127.0.0.1:1
|
||||
model: fake-model
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
chunk:
|
||||
module: generic
|
||||
options:
|
||||
max_units: 50
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
```
|
||||
|
||||
The fixture may use a fake base URL because tests should inject a fake LLM
|
||||
client factory. Do not require a real network call.
|
||||
|
||||
Acceptance tests should execute the public CLI entry path with:
|
||||
|
||||
```sh
|
||||
notarius run dnd-session --config <fixture> --input <fixture> --output-dir <tmp>
|
||||
notarius run dnd-session --config <fixture> --input <fixture> --only spells --output-dir <tmp>
|
||||
notarius config validate --config <fixture> --pipeline dnd-session
|
||||
notarius pipelines list --config <fixture>
|
||||
```
|
||||
|
||||
The fake LLM should return deterministic D&D spell output with valid source
|
||||
references. The resulting output files should be parsed as JSON and checked for:
|
||||
|
||||
- manifest pipeline ID and digest;
|
||||
- spell artifact payload;
|
||||
- source references;
|
||||
- prompt/schema metadata in manifest or artifact metadata;
|
||||
- validation status;
|
||||
- warning behavior.
|
||||
|
||||
### Required Failure Coverage
|
||||
|
||||
Add fixture-driven tests for:
|
||||
|
||||
- missing config;
|
||||
- unknown pipeline;
|
||||
- invalid Seriatim input;
|
||||
- invalid `--only` lane;
|
||||
- fake LLM failure;
|
||||
- malformed LLM response;
|
||||
- invalid source reference rejection.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/cli internal/modules/extract/dnd/spells
|
||||
go test ./internal/cli ./internal/modules/extract/dnd/spells
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
## Stage 8: MVP Final Review And Roadmap Cleanup
|
||||
|
||||
### Goal
|
||||
|
||||
Confirm the MVP is complete enough to trigger the deferred documentation pass.
|
||||
|
||||
### Required Review
|
||||
|
||||
Perform a code review against:
|
||||
|
||||
- [`mvp.md`](mvp.md);
|
||||
- [`../policy/architecture.md`](../policy/architecture.md);
|
||||
- [`../policy/documentation.md`](../policy/documentation.md).
|
||||
|
||||
Check specifically:
|
||||
|
||||
- no D&D prompt/schema assets or constants remain in framework packages;
|
||||
- production CLI commands use production wiring by default;
|
||||
- `config validate --pipeline` works with the MVP fixture;
|
||||
- `pipelines list` works with the MVP fixture;
|
||||
- `notarius run` writes durable output and diagnostics;
|
||||
- default modules resolve without test-only registration;
|
||||
- output warnings remain out-of-band from artifact payloads;
|
||||
- no private data or secrets appear in fixtures;
|
||||
- docs outside `docs/roadmap/` describe only implemented behavior.
|
||||
|
||||
### Required Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
### Required Roadmap Update
|
||||
|
||||
After the MVP is implemented and reviewed:
|
||||
|
||||
- update [`mvp.md`](mvp.md) to mark MVP functionality complete or reduce it to
|
||||
remaining release/documentation work;
|
||||
- keep the full documentation pass deferred until this review passes;
|
||||
- do not tag alpha `0.1.0` until the documentation pass is complete.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The plan above makes the required MVP implementation choices explicitly.
|
||||
Reference in New Issue
Block a user