Add a staged implementation plan for porting relevant code from audita
This commit is contained in:
@@ -31,73 +31,30 @@ Out of scope:
|
||||
- transcript mutation;
|
||||
- correction ledger terminology;
|
||||
- Audita module or validator behavior;
|
||||
- real D&D prompts or schemas unless needed as inert registry tests.
|
||||
- real D&D prompts or schemas;
|
||||
- domain-specific prompt or schema assets;
|
||||
- embedded built-in pipeline profiles.
|
||||
|
||||
## Proposed Stages
|
||||
## Target End State
|
||||
|
||||
### Stage 1: LLM Runtime
|
||||
The repository should contain reusable runtime infrastructure adapted from
|
||||
Audita where it directly supports Notarius contracts:
|
||||
|
||||
Port or adapt the OpenAI-compatible structured-output client and scheduler.
|
||||
- an OpenAI-compatible structured-output LLM client behind the existing
|
||||
`StructuredLLMClient` interface;
|
||||
- an LLM scheduler for bounded concurrency;
|
||||
- an embedded response-schema registry pattern;
|
||||
- an embedded prompt registry pattern;
|
||||
- a diagnostics run directory pattern using extraction-oriented artifact names;
|
||||
- config structs, loading, defaults, redaction, and validation for named
|
||||
pipeline profiles.
|
||||
|
||||
Keep the contract transport-neutral:
|
||||
Framework code should remain source-agnostic and domain-agnostic. Provider HTTP
|
||||
details should stay inside the LLM runtime package. Prompt and schema registries
|
||||
should use placeholder/test assets until real extractor prompts and schemas are
|
||||
introduced by later checkpoints.
|
||||
|
||||
- framework code should depend on a `StructuredLLMClient` interface;
|
||||
- provider-specific HTTP details should remain in the LLM runtime package;
|
||||
- errors must redact configured secrets.
|
||||
|
||||
### Stage 2: Response Schema Registry
|
||||
|
||||
Port or adapt the embedded JSON response-schema registry pattern.
|
||||
|
||||
The registry should track:
|
||||
|
||||
- schema key;
|
||||
- schema ID;
|
||||
- schema version;
|
||||
- schema name;
|
||||
- JSON schema content;
|
||||
- schema hash.
|
||||
|
||||
Use placeholder or test schemas if real extractor schemas are not ready.
|
||||
|
||||
### Stage 3: Prompt Registry
|
||||
|
||||
Port or adapt the embedded prompt registry pattern.
|
||||
|
||||
The registry should track:
|
||||
|
||||
- prompt ID;
|
||||
- prompt version;
|
||||
- prompt source;
|
||||
- embedded path;
|
||||
- prompt hash.
|
||||
|
||||
Do not add D&D prompt assets here unless the implementation naturally overlaps
|
||||
with checkpoint 6. Test prompts are acceptable for registry tests.
|
||||
|
||||
### Stage 4: Diagnostics Run Directory
|
||||
|
||||
Port or adapt the diagnostics run directory pattern.
|
||||
|
||||
Initial diagnostics should cover:
|
||||
|
||||
- invocation metadata;
|
||||
- redacted effective config;
|
||||
- selected pipeline ID;
|
||||
- resolved pipeline definition and digest;
|
||||
- source document artifact;
|
||||
- run report placeholder;
|
||||
- error log on failure.
|
||||
|
||||
Avoid Audita-specific artifact names such as correction ledger.
|
||||
|
||||
### Stage 5: Pipeline Profile Config
|
||||
|
||||
Add config structs, loading, defaults, and validation for named pipeline
|
||||
profiles.
|
||||
|
||||
Initial config areas:
|
||||
Config should support:
|
||||
|
||||
- `llm_profiles`;
|
||||
- `pipelines.<pipeline_id>.input`;
|
||||
@@ -123,17 +80,20 @@ Structural module selection should come from pipeline config. CLI flags may
|
||||
override operational settings and artifact lane selection, but should not offer
|
||||
ad hoc `--extractor` or `--chunker` wiring.
|
||||
|
||||
Add validation for unknown pipeline IDs, unknown module keys, missing required
|
||||
slots, missing capabilities, unknown LLM profiles, empty artifact-lane sets, and
|
||||
invalid lane selections.
|
||||
Config validation should fail fast for unknown pipeline IDs, unknown module
|
||||
keys, missing required slots, missing capabilities, unknown LLM profiles, empty
|
||||
artifact-lane sets, and invalid lane selections.
|
||||
|
||||
If the CLI shell is ready, add:
|
||||
If the CLI shell is ready, the checkpoint should expose discovery/validation
|
||||
commands for config and pipeline profiles:
|
||||
|
||||
```sh
|
||||
notarius config validate
|
||||
notarius pipelines list
|
||||
```
|
||||
|
||||
Implementation staging belongs in [`implementation.md`](implementation.md).
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- `go test ./...` passes.
|
||||
|
||||
915
docs/roadmap/implementation.md
Normal file
915
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,915 @@
|
||||
# Implementation Plan: Checkpoint 4 Portable Audita Infrastructure
|
||||
|
||||
## Status
|
||||
|
||||
This is a staged implementation plan for
|
||||
[`4-portable-audita-infrastructure.md`](4-portable-audita-infrastructure.md).
|
||||
It is intended for an LLM coding agent to follow stage by stage.
|
||||
|
||||
This plan implements only checkpoint 4. Do not add real Seriatim parsing, real
|
||||
D&D extraction, real domain prompts or schemas, production extraction modules,
|
||||
durable final artifact writing, or `notarius run` execution behavior in this
|
||||
checkpoint.
|
||||
|
||||
## Policy Context
|
||||
|
||||
Follow:
|
||||
|
||||
- [`docs/policy/architecture.md`](../policy/architecture.md)
|
||||
- [`docs/policy/documentation.md`](../policy/documentation.md)
|
||||
|
||||
Required boundaries:
|
||||
|
||||
- copy only reusable Audita infrastructure, not Audita's transcript-correction
|
||||
model;
|
||||
- keep framework code source-agnostic and domain-agnostic;
|
||||
- keep provider-specific HTTP request/response details inside
|
||||
`internal/framework/llm`;
|
||||
- keep prompt construction and prompt assets out of provider adapters;
|
||||
- keep structural pipeline wiring in named pipeline config, not ad hoc CLI
|
||||
flags;
|
||||
- preserve the fixed pipeline shape:
|
||||
`input -> chunk -> extract -> merge -> normalize -> output`;
|
||||
- keep capabilities as flat strings;
|
||||
- redact configured secrets from diagnostics, effective config, and surfaced
|
||||
provider errors.
|
||||
|
||||
Audita files useful for adaptation:
|
||||
|
||||
- `../audita/internal/framework/llm/*`
|
||||
- `../audita/internal/framework/responseschema/registry.go`
|
||||
- `../audita/internal/prompts/registry.go`
|
||||
- `../audita/internal/prompts/render.go`
|
||||
- `../audita/internal/core/diagnostics/*`
|
||||
- `../audita/internal/core/config/*`
|
||||
- `../audita/internal/cli/*`
|
||||
- `../audita/examples/*.yml`
|
||||
|
||||
## Global Implementation Decisions
|
||||
|
||||
- Add `gopkg.in/yaml.v3` as the only new third-party dependency in this
|
||||
checkpoint. Audita already uses it, and YAML is the architecture-policy
|
||||
preference for Notarius config.
|
||||
- Put LLM client, scheduler, structured-output decoding helpers, secret
|
||||
redaction, and response-schema registry code in `internal/framework/llm`.
|
||||
Do not create separate `responseschema`, `structuredoutput`, or `warnings`
|
||||
framework packages.
|
||||
- Put prompt registry and rendering code in `internal/framework/prompt`.
|
||||
- Put config structs, defaults, file/env/flag precedence helpers, redaction, and
|
||||
validation in `internal/core/config`.
|
||||
- Put diagnostics run-directory code in `internal/core/diagnostics`. This
|
||||
package is justified by real durable diagnostic state in checkpoint 4; keep it
|
||||
extraction-oriented and do not copy correction-ledger concepts.
|
||||
- Keep CLI support limited to discovery/validation commands:
|
||||
`notarius config validate` and `notarius pipelines list`.
|
||||
- Do not add embedded built-in pipeline profiles. Config files are the only
|
||||
source of user-defined pipeline profiles in this checkpoint.
|
||||
- Do not add real D&D prompt/schema assets. Use inert placeholder assets only
|
||||
for registry tests.
|
||||
- Use JSON for diagnostics artifacts even when the input config is YAML.
|
||||
- Use `sha256:<hex>` for content digests written to diagnostics or registries,
|
||||
unless an existing Notarius type already expects bare hex. Prefer the prefixed
|
||||
form for new Notarius code.
|
||||
- Run `gofmt` on touched Go files after every stage.
|
||||
|
||||
## Stage 1: LLM Runtime
|
||||
|
||||
### Goal
|
||||
|
||||
Adapt Audita's OpenAI-compatible structured-output client, secret redaction, and
|
||||
scheduler to Notarius's existing `contracts.StructuredLLMClient`.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/llm/client_common.go`
|
||||
- `internal/framework/llm/openai_compatible_client.go`
|
||||
- `internal/framework/llm/openai_compatible_client_test.go`
|
||||
- `internal/framework/llm/scheduler.go`
|
||||
- `internal/framework/llm/scheduler_test.go`
|
||||
- `internal/framework/llm/secrets.go`
|
||||
- `internal/framework/llm/secrets_test.go`
|
||||
|
||||
### Required API
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
type OpenAICompatibleClientConfig struct {
|
||||
BaseURL string
|
||||
Model string
|
||||
APIKey string
|
||||
MaxRetries int
|
||||
HTTPClient *http.Client
|
||||
RequestTimeout time.Duration
|
||||
}
|
||||
|
||||
type OpenAICompatibleClient struct { /* unexported fields */ }
|
||||
|
||||
func NewOpenAICompatibleClient(cfg OpenAICompatibleClientConfig) (*OpenAICompatibleClient, error)
|
||||
func (c *OpenAICompatibleClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error)
|
||||
```
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
type Scheduler struct { /* unexported fields */ }
|
||||
func NewScheduler(maxConcurrency int) (*Scheduler, error)
|
||||
func (s *Scheduler) Acquire(ctx context.Context) (func(), error)
|
||||
func (s *Scheduler) Run(ctx context.Context, fn func(context.Context) error) error
|
||||
```
|
||||
|
||||
Add secret helpers:
|
||||
|
||||
```go
|
||||
func RedactSecrets(message string, secrets []string) string
|
||||
func ErrorWithSecretsRedacted(err error, secrets []string) error
|
||||
```
|
||||
|
||||
### Notarius Adaptations
|
||||
|
||||
Audita's client expects a typed response schema object. Notarius currently uses:
|
||||
|
||||
```go
|
||||
type StructuredCompletionRequest struct {
|
||||
ResponseSchemaName string
|
||||
ResponseSchema json.RawMessage
|
||||
}
|
||||
```
|
||||
|
||||
Therefore the Notarius client must:
|
||||
|
||||
- require `ResponseSchemaName` to be non-empty;
|
||||
- require `ResponseSchema` to be non-empty valid JSON;
|
||||
- send OpenAI-compatible `response_format.type=json_schema`;
|
||||
- send `json_schema.name` from `ResponseSchemaName`;
|
||||
- send `json_schema.schema` from `ResponseSchema`;
|
||||
- decode the assistant message content into `out`;
|
||||
- return raw assistant content in `StructuredCompletionResponse.Content`;
|
||||
- populate provider/model/token metadata when available;
|
||||
- use `req.Model` when present, otherwise the configured default model.
|
||||
|
||||
Provider request/response structs must be unexported.
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- Validate `out` is a non-nil pointer.
|
||||
- Reject empty base URL, empty default model, negative retries, empty model at
|
||||
call time, missing schema name, and invalid schema JSON.
|
||||
- Trim message roles/content and reject empty roles or content.
|
||||
- Retry transport errors, malformed provider envelopes, malformed assistant JSON,
|
||||
HTTP 429, and HTTP 5xx up to `MaxRetries`.
|
||||
- Do not retry non-retryable 4xx provider errors.
|
||||
- Respect `context.Context` cancellation.
|
||||
- Redact API keys and bearer tokens from returned errors.
|
||||
- Scheduler must bound concurrent calls, release permits exactly once, handle
|
||||
cancellation while queued, and avoid leaking permits when cancellation races
|
||||
with grant.
|
||||
|
||||
### Required Tests
|
||||
|
||||
Port and adapt Audita LLM tests using local `httptest.Server` fakes:
|
||||
|
||||
- successful structured completion;
|
||||
- request body includes model, messages, schema name, and schema JSON;
|
||||
- default model fallback and request model override;
|
||||
- invalid output target;
|
||||
- missing/invalid schema;
|
||||
- provider non-2xx behavior;
|
||||
- retry behavior for 429/5xx and malformed retryable responses;
|
||||
- no retry for non-retryable 4xx;
|
||||
- provider error redacts configured API key;
|
||||
- scheduler max concurrency;
|
||||
- scheduler cancellation while queued;
|
||||
- scheduler release function is idempotent.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/llm
|
||||
go test ./internal/framework/llm
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 2: Embedded Response Schema Registry
|
||||
|
||||
### Goal
|
||||
|
||||
Adapt Audita's embedded response-schema registry pattern into
|
||||
`internal/framework/llm` without adding real extractor schemas.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/llm/schema_registry.go`
|
||||
- `internal/framework/llm/schema_registry_test.go`
|
||||
- `internal/framework/llm/assets/schemas/test_artifact.v1.json`
|
||||
- `internal/framework/llm/assets/schemas/test_validator_decision.v1.json`
|
||||
|
||||
### Required API
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
type ResponseSchemaKey string
|
||||
|
||||
const (
|
||||
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
|
||||
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
|
||||
)
|
||||
|
||||
type ResponseSchema struct {
|
||||
Key ResponseSchemaKey `json:"key"`
|
||||
ID string `json:"id"`
|
||||
Version string `json:"version"`
|
||||
Name string `json:"name"`
|
||||
JSONSchema json.RawMessage `json:"json_schema"`
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
func RegisteredResponseSchemas() []ResponseSchema
|
||||
func LookupResponseSchema(key ResponseSchemaKey) (ResponseSchema, bool)
|
||||
func MustLookupResponseSchema(key ResponseSchemaKey) ResponseSchema
|
||||
func (s ResponseSchema) DiagnosticsMap() map[string]any
|
||||
```
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- Embed schema assets using `embed.FS`.
|
||||
- Validate assets are valid JSON at package initialization.
|
||||
- Compute `SHA256` from exact asset bytes and format as `sha256:<hex>`.
|
||||
- Return defensive copies of `json.RawMessage`.
|
||||
- Return registered schemas sorted by key.
|
||||
- Use Notarius-neutral IDs and names, for example
|
||||
`notarius.test_artifact` and `notarius_test_artifact_v1`.
|
||||
- Keep test schemas inert and clearly non-domain-specific.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- lookup succeeds for both test schemas;
|
||||
- unknown lookup returns false;
|
||||
- `MustLookupResponseSchema` panics for unknown keys;
|
||||
- registered schemas are sorted;
|
||||
- JSON schema content is valid JSON;
|
||||
- returned schema JSON is mutation-safe;
|
||||
- diagnostics map omits raw schema content and includes ID, version, name, and
|
||||
hash.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/llm
|
||||
go test ./internal/framework/llm
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 3: Prompt Registry
|
||||
|
||||
### Goal
|
||||
|
||||
Adapt Audita's embedded prompt registry and rendering pattern into
|
||||
`internal/framework/prompt` using only generic test prompts.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/framework/prompt/registry.go`
|
||||
- `internal/framework/prompt/render.go`
|
||||
- `internal/framework/prompt/registry_test.go`
|
||||
- `internal/framework/prompt/render_test.go`
|
||||
- `internal/framework/prompt/assets/shared/prompt_hardening.md`
|
||||
- `internal/framework/prompt/assets/test/generic/system.md`
|
||||
- `internal/framework/prompt/assets/test/generic/user.md`
|
||||
|
||||
### Required API
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
const (
|
||||
SourceBuiltin = "builtin"
|
||||
VersionV1 = "v1"
|
||||
TestGenericPromptID = "test.generic"
|
||||
)
|
||||
|
||||
type Metadata struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
PromptVersion string `json:"prompt_version"`
|
||||
PromptSource string `json:"prompt_source"`
|
||||
EmbeddedPath string `json:"embedded_path"`
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
func LookupMetadata(promptID string) (Metadata, bool)
|
||||
func MustLookupMetadata(promptID string) Metadata
|
||||
func RegisteredMetadata() []Metadata
|
||||
func HardeningText() string
|
||||
func RenderUserSystem(promptID string, data any) (system string, user string, metadata Metadata, err error)
|
||||
func (m Metadata) DiagnosticsMap() map[string]any
|
||||
```
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- Compile embedded system/user prompts with `text/template`.
|
||||
- Use `Option("missingkey=error")`.
|
||||
- Provide a `{{ hardening }}` template function backed by the shared hardening
|
||||
asset.
|
||||
- Trim rendered system/user text.
|
||||
- Compute `SHA256` from combined system and user source text and format as
|
||||
`sha256:<hex>`.
|
||||
- Return prompt metadata sorted by prompt ID.
|
||||
- Keep assets generic and non-domain-specific.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- metadata lookup succeeds for `test.generic`;
|
||||
- unknown lookup returns false;
|
||||
- `MustLookupMetadata` panics for unknown prompt ID;
|
||||
- registered metadata is sorted;
|
||||
- render returns system/user text and metadata;
|
||||
- missing template data returns an error;
|
||||
- hardening text is available and appears when referenced;
|
||||
- diagnostics map includes metadata but no rendered prompt text.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/framework/prompt
|
||||
go test ./internal/framework/prompt
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 4: Diagnostics Run Directory
|
||||
|
||||
### Goal
|
||||
|
||||
Adapt Audita's diagnostics run-directory pattern using extraction-oriented
|
||||
artifact names and Notarius types.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/core/diagnostics/artifacts.go`
|
||||
- `internal/core/diagnostics/run_dir.go`
|
||||
- `internal/core/diagnostics/run_dir_test.go`
|
||||
- `internal/core/diagnostics/artifacts_test.go`
|
||||
|
||||
### Required API
|
||||
|
||||
Add artifact names:
|
||||
|
||||
```go
|
||||
const (
|
||||
ArtifactInvocationMetadata = "invocation.json"
|
||||
ArtifactEffectiveConfig = "effective-config.json"
|
||||
ArtifactResolvedPipeline = "resolved-pipeline.json"
|
||||
ArtifactSourceDocument = "source-document.json"
|
||||
ArtifactRunManifest = "run-manifest.json"
|
||||
ArtifactRunReport = "run-report.json"
|
||||
ArtifactWarnings = "warnings.json"
|
||||
ArtifactErrorLog = "error.log"
|
||||
)
|
||||
```
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
type RunDirectory struct { /* unexported fields */ }
|
||||
|
||||
type RetentionMode string
|
||||
const (
|
||||
RetentionAuto RetentionMode = "auto"
|
||||
RetentionAlways RetentionMode = "always"
|
||||
RetentionNever RetentionMode = "never"
|
||||
)
|
||||
|
||||
type RetentionDecisionInput struct {
|
||||
RetentionMode RetentionMode
|
||||
RunSucceeded bool
|
||||
HasWarnings bool
|
||||
}
|
||||
|
||||
func ShouldRetainRunDirectory(input RetentionDecisionInput) bool
|
||||
func NewRunDirectory(workDir string, retention RetentionMode) (*RunDirectory, error)
|
||||
func (r *RunDirectory) Path() string
|
||||
func (r *RunDirectory) RunID() string
|
||||
func (r *RunDirectory) WriteInvocationMetadata(metadata InvocationMetadata) error
|
||||
func (r *RunDirectory) WriteRedactedEffectiveConfig(payload any) error
|
||||
func (r *RunDirectory) WriteResolvedPipeline(payload any) error
|
||||
func (r *RunDirectory) WriteSourceDocument(payload any) error
|
||||
func (r *RunDirectory) WriteRunManifest(manifest artifacts.RunManifest) error
|
||||
func (r *RunDirectory) WriteRunReport(payload any) error
|
||||
func (r *RunDirectory) WriteWarnings(warnings []contracts.Warning) error
|
||||
func (r *RunDirectory) WriteErrorLog(errorMessage string) error
|
||||
func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error
|
||||
func (r *RunDirectory) ApplyRetention(input RetentionDecisionInput) error
|
||||
```
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
type InvocationMetadata struct {
|
||||
Operation string `json:"operation"`
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
||||
InputPath string `json:"input_path,omitempty"`
|
||||
ConfigPath string `json:"config_path,omitempty"`
|
||||
ConfigSource string `json:"config_source,omitempty"`
|
||||
OnlyLanes []string `json:"only_lanes,omitempty"`
|
||||
RunID string `json:"run_id"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
}
|
||||
```
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- Default work directory is `/tmp/notarius`.
|
||||
- Create work directory with `0o755`.
|
||||
- Create unique run directories named `run-<unix-nano>`.
|
||||
- Write JSON artifacts with `json.MarshalIndent` and a trailing newline.
|
||||
- Write error logs as plain text with a trailing newline.
|
||||
- Reject artifact names that are empty, absolute paths, contain path
|
||||
separators, or resolve outside the run directory.
|
||||
- Retention behavior:
|
||||
- failed runs are always retained;
|
||||
- `always` retains successful runs;
|
||||
- `never` removes successful runs;
|
||||
- `auto` retains successful runs only when warnings exist.
|
||||
- Do not copy Audita names such as source transcript, normalized transcript,
|
||||
correction ledger, proposal, or replacement.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- run directory creation and run ID shape;
|
||||
- default work directory behavior using a temporary working directory where
|
||||
possible;
|
||||
- JSON artifact writes are indented and newline-terminated;
|
||||
- invocation metadata fills missing run ID and start time;
|
||||
- error log write;
|
||||
- artifact path rejection for unsafe names;
|
||||
- retention decisions for failed/successful runs across modes;
|
||||
- `ApplyRetention` removes only the run directory for removable cases.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/core/diagnostics
|
||||
go test ./internal/core/diagnostics
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 5: Config Model, YAML Loading, Redaction, And Environment Overrides
|
||||
|
||||
### Goal
|
||||
|
||||
Add Notarius config structs and loading primitives for named pipeline profiles,
|
||||
LLM profiles, operational settings, YAML files, redaction, and environment
|
||||
overrides.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/core/config/config.go`
|
||||
- `internal/core/config/file_config.go`
|
||||
- `internal/core/config/env.go`
|
||||
- `internal/core/config/redaction.go`
|
||||
- `internal/core/config/config_test.go`
|
||||
- `internal/core/config/file_config_test.go`
|
||||
- `internal/core/config/env_test.go`
|
||||
- `internal/core/config/redaction_test.go`
|
||||
|
||||
### Dependency Change
|
||||
|
||||
Add:
|
||||
|
||||
```sh
|
||||
go get gopkg.in/yaml.v3@v3.0.1
|
||||
```
|
||||
|
||||
Commit the resulting `go.mod` and `go.sum` changes.
|
||||
|
||||
### Required API
|
||||
|
||||
Add core config types:
|
||||
|
||||
```go
|
||||
const SupportedFileConfigVersion = 1
|
||||
|
||||
type Config struct {
|
||||
LLMProfiles map[string]LLMProfile
|
||||
Pipelines map[string]pipeline.PipelineProfile
|
||||
Concurrency ConcurrencyConfig
|
||||
Diagnostics DiagnosticsConfig
|
||||
}
|
||||
|
||||
type LLMProfile struct {
|
||||
Provider string
|
||||
BaseURL string
|
||||
Model string
|
||||
APIKey string
|
||||
APIKeyEnv string
|
||||
TimeoutSeconds int
|
||||
MaxRetries int
|
||||
MaxConcurrency int
|
||||
}
|
||||
|
||||
type ConcurrencyConfig struct {
|
||||
TotalLLM int
|
||||
}
|
||||
|
||||
type DiagnosticsConfig struct {
|
||||
WorkDir string
|
||||
Retention diagnostics.RetentionMode
|
||||
}
|
||||
|
||||
func Default() Config
|
||||
func (c Config) Redacted() Config
|
||||
func (c *Config) ApplyEnvOverrides() error
|
||||
func LoadFromEnv() (Config, error)
|
||||
```
|
||||
|
||||
Add file config loading:
|
||||
|
||||
```go
|
||||
type FileConfig struct { /* YAML-facing fields */ }
|
||||
func LoadFileConfig(path string) (FileConfig, error)
|
||||
func ParseFileConfigYAML(data []byte) (FileConfig, error)
|
||||
func (c *Config) ApplyFileConfig(fileCfg FileConfig) error
|
||||
```
|
||||
|
||||
Add unexported lookup seams for tests:
|
||||
|
||||
```go
|
||||
func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(string) (string, bool)) error
|
||||
func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool)) error
|
||||
```
|
||||
|
||||
### YAML Shape
|
||||
|
||||
Support this shape:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
llm_profiles:
|
||||
default:
|
||||
provider: openai-compatible
|
||||
base_url: https://example.invalid/v1
|
||||
model: test-model
|
||||
api_key_env: NOTARIUS_TEST_API_KEY
|
||||
timeout: 600s
|
||||
max_retries: 3
|
||||
max_concurrency: 1
|
||||
pipelines:
|
||||
example:
|
||||
input: fake/input
|
||||
chunk: generic
|
||||
artifacts:
|
||||
events:
|
||||
extract:
|
||||
module: fake/extract
|
||||
llm_profile: default
|
||||
options:
|
||||
temperature: 0
|
||||
merge: appendorder
|
||||
normalize: noop
|
||||
validators:
|
||||
- fake/validator
|
||||
output: json
|
||||
concurrency:
|
||||
total_llm: 1
|
||||
diagnostics:
|
||||
work_dir: /tmp/notarius
|
||||
retention: auto
|
||||
```
|
||||
|
||||
Module bindings must support both string shorthand and object form:
|
||||
|
||||
```yaml
|
||||
extract: fake/extract
|
||||
extract:
|
||||
module: fake/extract
|
||||
llm_profile: fast
|
||||
options:
|
||||
key: value
|
||||
```
|
||||
|
||||
Validator bindings must support a sequence of string or object bindings.
|
||||
|
||||
### Defaults
|
||||
|
||||
`Default()` must provide:
|
||||
|
||||
- `LLMProfiles["default"]` with provider `openai-compatible`,
|
||||
`TimeoutSeconds: 600`, `MaxRetries: 3`, and `MaxConcurrency: 1`;
|
||||
- no built-in pipeline profiles;
|
||||
- `Concurrency.TotalLLM: 1`;
|
||||
- `Diagnostics.WorkDir: "/tmp/notarius"`;
|
||||
- `Diagnostics.Retention: diagnostics.RetentionAuto`.
|
||||
|
||||
Config validation will later decide whether a selected LLM profile is complete
|
||||
enough to build a real client. This stage should not require default `BaseURL`
|
||||
or `Model` values.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Support these environment overrides:
|
||||
|
||||
- `NOTARIUS_LLM_DEFAULT_API_KEY`
|
||||
- `NOTARIUS_LLM_DEFAULT_BASE_URL`
|
||||
- `NOTARIUS_LLM_DEFAULT_MODEL`
|
||||
- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS`
|
||||
- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES`
|
||||
- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY`
|
||||
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`
|
||||
- `NOTARIUS_WORK_DIR`
|
||||
- `NOTARIUS_DIAGNOSTICS_RETENTION`
|
||||
|
||||
Do not support environment variables that structurally change pipeline module
|
||||
selection.
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- YAML parsing must use `KnownFields(true)`.
|
||||
- Config version is required and must equal `SupportedFileConfigVersion`.
|
||||
- Duration fields accept integer seconds or Go duration strings resolving to
|
||||
whole seconds.
|
||||
- `api_key_env` must be a valid environment-variable name and must resolve via
|
||||
the lookup function used by tests.
|
||||
- Redaction must replace all API key values with `[REDACTED]` while preserving
|
||||
non-secret fields.
|
||||
- Applying a file config must merge into defaults, not replace unspecified
|
||||
sections with zero values.
|
||||
- Environment overrides apply after file config.
|
||||
- Object-form `options` must preserve YAML scalar/list/map values as
|
||||
`map[string]any` suitable for `pipeline.ModuleBinding.Options`.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- default values;
|
||||
- parse minimal valid config;
|
||||
- reject unknown YAML fields;
|
||||
- reject missing/unsupported version;
|
||||
- string and object module-binding forms;
|
||||
- validator binding list with mixed shorthand/object forms;
|
||||
- duration parsing;
|
||||
- API key env resolution and invalid env-name rejection;
|
||||
- file config merges with defaults;
|
||||
- environment overrides operational and LLM values;
|
||||
- environment variables cannot change pipeline module wiring;
|
||||
- redacted config removes API key values.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/core/config
|
||||
go test ./internal/core/config
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 6: Config Validation And Pipeline Resolution
|
||||
|
||||
### Goal
|
||||
|
||||
Validate named pipeline profiles against registered module metadata, LLM
|
||||
profiles, lane selections, and resolved pipeline digest behavior.
|
||||
|
||||
### Files To Add
|
||||
|
||||
- `internal/core/config/validation.go`
|
||||
- `internal/core/config/effective_config.go`
|
||||
- `internal/core/config/validation_test.go`
|
||||
- `internal/core/config/effective_config_test.go`
|
||||
|
||||
### Required API
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
type ResolveInput struct {
|
||||
PipelineID string
|
||||
Only []string
|
||||
Catalog pipeline.ModuleCatalog
|
||||
}
|
||||
|
||||
type EffectiveConfig struct {
|
||||
Config Config
|
||||
PipelineID string
|
||||
Only []string
|
||||
ResolvedPipeline pipeline.ResolvedPipeline
|
||||
}
|
||||
|
||||
func (c Config) Validate() error
|
||||
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error)
|
||||
func (c Config) LLMProfile(id string) (LLMProfile, bool)
|
||||
func (c Config) OpenAICompatibleClientConfig(profileID string) (llm.OpenAICompatibleClientConfig, error)
|
||||
```
|
||||
|
||||
### Required Behavior
|
||||
|
||||
`Validate()` must check config-internal rules that do not need a module catalog:
|
||||
|
||||
- LLM profile IDs trim to non-empty unique IDs;
|
||||
- pipeline IDs trim to non-empty unique IDs;
|
||||
- LLM profile provider must be empty or `openai-compatible`;
|
||||
- LLM profile numeric values must be positive where applicable;
|
||||
- diagnostics work dir must not be empty;
|
||||
- diagnostics retention must be one of `auto`, `always`, or `never`;
|
||||
- `TotalLLM` must be greater than zero;
|
||||
- every LLM profile referenced by a pipeline binding must exist.
|
||||
|
||||
`Resolve(input)` must:
|
||||
|
||||
- call `Validate()`;
|
||||
- trim `PipelineID`;
|
||||
- reject empty or unknown pipeline IDs;
|
||||
- pass `Only` through to `pipeline.ResolvePipeline`;
|
||||
- return the resolved pipeline and digest;
|
||||
- surface errors that name the pipeline and lane where possible.
|
||||
|
||||
`OpenAICompatibleClientConfig(profileID)` must:
|
||||
|
||||
- reject unknown profile IDs;
|
||||
- reject unsupported providers;
|
||||
- require base URL and model;
|
||||
- pass API key, timeout, and max retries to the LLM client config.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- `Validate` success for valid config;
|
||||
- unknown LLM profile referenced by binding;
|
||||
- invalid provider;
|
||||
- invalid numeric fields;
|
||||
- invalid diagnostics retention;
|
||||
- empty/unknown selected pipeline ID;
|
||||
- lane filtering success and failure;
|
||||
- unknown module key through module catalog;
|
||||
- missing capability through module catalog;
|
||||
- resolved pipeline digest changes when effective config changes;
|
||||
- `OpenAICompatibleClientConfig` rejects incomplete default profile and succeeds
|
||||
when base URL/model are set.
|
||||
|
||||
Use fake module registries and fake module specs. Do not add real modules.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/core/config
|
||||
go test ./internal/core/config
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Stage 7: CLI Config Validation And Pipeline Listing
|
||||
|
||||
### Goal
|
||||
|
||||
Add minimal CLI support for config discovery/validation and pipeline profile
|
||||
listing without adding run execution.
|
||||
|
||||
### Files To Update
|
||||
|
||||
- `internal/cli/run.go`
|
||||
- `internal/cli/run_test.go`
|
||||
|
||||
### Required CLI
|
||||
|
||||
Support:
|
||||
|
||||
```sh
|
||||
notarius config validate --config path/to/config.yml
|
||||
notarius config validate --config path/to/config.yml --pipeline pipeline-id
|
||||
notarius config validate --config path/to/config.yml --pipeline pipeline-id --only lane-a,lane-b
|
||||
notarius pipelines list --config path/to/config.yml
|
||||
notarius pipelines list --config path/to/config.yml --json
|
||||
```
|
||||
|
||||
Also support the default config search path from architecture policy:
|
||||
|
||||
1. `--config` path, when provided;
|
||||
2. `NOTARIUS_CONFIG`, when set;
|
||||
3. `/usr/local/etc/notarius/config.yml`, when present.
|
||||
|
||||
Do not add `notarius run` in this checkpoint.
|
||||
|
||||
### Required Implementation Shape
|
||||
|
||||
The CLI needs module metadata to validate structural pipeline config. Add an
|
||||
internal test seam rather than hard-coding fake modules:
|
||||
|
||||
```go
|
||||
type Options struct {
|
||||
Catalog pipeline.ModuleCatalog
|
||||
LookupEnv func(string) (string, bool)
|
||||
}
|
||||
|
||||
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int
|
||||
```
|
||||
|
||||
Keep the existing `Run(args, stdout, stderr)` as a wrapper that passes default
|
||||
options. Default production options may contain empty registries until real
|
||||
modules are registered by later checkpoints.
|
||||
|
||||
### Required Behavior
|
||||
|
||||
- `config validate` exits `0` and writes a concise success message on valid
|
||||
config.
|
||||
- `config validate` exits `1` on config load/validation errors and writes the
|
||||
error to stderr.
|
||||
- `config validate --pipeline` resolves that pipeline against the supplied
|
||||
catalog.
|
||||
- `--only` is valid only when `--pipeline` is present.
|
||||
- `pipelines list` exits `0` and lists pipeline IDs sorted alphabetically.
|
||||
- `pipelines list --json` emits stable JSON, for example:
|
||||
|
||||
```json
|
||||
{"pipelines":["a","b"]}
|
||||
```
|
||||
|
||||
- Unknown commands and invalid flags exit `2`.
|
||||
- Usage text must include the new commands.
|
||||
- CLI flags must not allow ad hoc structural module overrides such as
|
||||
`--extractor`, `--chunker`, `--input`, `--merge`, or `--normalize`.
|
||||
|
||||
### Required Tests
|
||||
|
||||
- existing help/unknown-command tests still pass;
|
||||
- `config validate` success with fake catalog;
|
||||
- `config validate` reports config parse errors;
|
||||
- `config validate --pipeline --only` success and invalid lane failure;
|
||||
- `--only` without `--pipeline` fails;
|
||||
- `pipelines list` sorted text output;
|
||||
- `pipelines list --json` stable JSON output;
|
||||
- `NOTARIUS_CONFIG` is used when `--config` is absent;
|
||||
- missing config path produces actionable error;
|
||||
- no ad hoc structural flags are accepted.
|
||||
|
||||
### Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/cli
|
||||
go test ./internal/cli
|
||||
go test ./...
|
||||
go build ./cmd/notarius
|
||||
rm -f ./notarius
|
||||
```
|
||||
|
||||
## Stage 8: Final Checkpoint 4 Review Pass
|
||||
|
||||
### Goal
|
||||
|
||||
Remove accidental Audita coupling, confirm scope boundaries, and verify all
|
||||
runtime pieces compose.
|
||||
|
||||
### Required Review
|
||||
|
||||
Check:
|
||||
|
||||
- no correction proposal, replacement policy, transcript mutation, correction
|
||||
ledger, or Audita module behavior was copied;
|
||||
- no package names or artifact names mention corrections, proposals,
|
||||
replacements, or transcripts unless they refer to Audita source files in
|
||||
comments inside tests;
|
||||
- no real D&D prompts or schemas were added;
|
||||
- no Seriatim input adapter was added;
|
||||
- no `notarius run` behavior was added;
|
||||
- provider-specific HTTP structs stay unexported inside `internal/framework/llm`;
|
||||
- response schemas live inside `internal/framework/llm`, not a new
|
||||
`responseschema` package;
|
||||
- prompt assets live inside `internal/framework/prompt`;
|
||||
- config is centered on named pipeline profiles;
|
||||
- module selection comes from config, not ad hoc CLI flags;
|
||||
- diagnostics artifact names are extraction-oriented;
|
||||
- secrets are redacted from effective config and surfaced provider errors.
|
||||
|
||||
### Required Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
rm -f ./notarius
|
||||
git status --short
|
||||
```
|
||||
|
||||
The implementation response should summarize:
|
||||
|
||||
- files/packages added;
|
||||
- tests run;
|
||||
- any deviations from this plan and why.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The plan intentionally defers real extractor prompts/schemas, concrete
|
||||
input modules, real built-in module catalogs, and `notarius run` execution to
|
||||
later checkpoints.
|
||||
Reference in New Issue
Block a user