916 lines
27 KiB
Markdown
916 lines
27 KiB
Markdown
# Implementation Plan: Checkpoint 4 Portable Audita Infrastructure
|
|
|
|
## Status
|
|
|
|
This is a staged implementation plan for
|
|
[`4-portable-audita-infrastructure.md`](4-portable-audita-infrastructure.md).
|
|
It is intended for an LLM coding agent to follow stage by stage.
|
|
|
|
This plan implements only checkpoint 4. Do not add real Seriatim parsing, real
|
|
D&D extraction, real domain prompts or schemas, production extraction modules,
|
|
durable final artifact writing, or `notarius run` execution behavior in this
|
|
checkpoint.
|
|
|
|
## Policy Context
|
|
|
|
Follow:
|
|
|
|
- [`docs/policy/architecture.md`](../policy/architecture.md)
|
|
- [`docs/policy/documentation.md`](../policy/documentation.md)
|
|
|
|
Required boundaries:
|
|
|
|
- copy only reusable Audita infrastructure, not Audita's transcript-correction
|
|
model;
|
|
- keep framework code source-agnostic and domain-agnostic;
|
|
- keep provider-specific HTTP request/response details inside
|
|
`internal/framework/llm`;
|
|
- keep prompt construction and prompt assets out of provider adapters;
|
|
- keep structural pipeline wiring in named pipeline config, not ad hoc CLI
|
|
flags;
|
|
- preserve the fixed pipeline shape:
|
|
`input -> chunk -> extract -> merge -> normalize -> output`;
|
|
- keep capabilities as flat strings;
|
|
- redact configured secrets from diagnostics, effective config, and surfaced
|
|
provider errors.
|
|
|
|
Audita files useful for adaptation:
|
|
|
|
- `../audita/internal/framework/llm/*`
|
|
- `../audita/internal/framework/responseschema/registry.go`
|
|
- `../audita/internal/prompts/registry.go`
|
|
- `../audita/internal/prompts/render.go`
|
|
- `../audita/internal/core/diagnostics/*`
|
|
- `../audita/internal/core/config/*`
|
|
- `../audita/internal/cli/*`
|
|
- `../audita/examples/*.yml`
|
|
|
|
## Global Implementation Decisions
|
|
|
|
- Add `gopkg.in/yaml.v3` as the only new third-party dependency in this
|
|
checkpoint. Audita already uses it, and YAML is the architecture-policy
|
|
preference for Notarius config.
|
|
- Put LLM client, scheduler, structured-output decoding helpers, secret
|
|
redaction, and response-schema registry code in `internal/framework/llm`.
|
|
Do not create separate `responseschema`, `structuredoutput`, or `warnings`
|
|
framework packages.
|
|
- Put prompt registry and rendering code in `internal/framework/prompt`.
|
|
- Put config structs, defaults, file/env/flag precedence helpers, redaction, and
|
|
validation in `internal/core/config`.
|
|
- Put diagnostics run-directory code in `internal/core/diagnostics`. This
|
|
package is justified by real durable diagnostic state in checkpoint 4; keep it
|
|
extraction-oriented and do not copy correction-ledger concepts.
|
|
- Keep CLI support limited to discovery/validation commands:
|
|
`notarius config validate` and `notarius pipelines list`.
|
|
- Do not add embedded built-in pipeline profiles. Config files are the only
|
|
source of user-defined pipeline profiles in this checkpoint.
|
|
- Do not add real D&D prompt/schema assets. Use inert placeholder assets only
|
|
for registry tests.
|
|
- Use JSON for diagnostics artifacts even when the input config is YAML.
|
|
- Use `sha256:<hex>` for content digests written to diagnostics or registries,
|
|
unless an existing Notarius type already expects bare hex. Prefer the prefixed
|
|
form for new Notarius code.
|
|
- Run `gofmt` on touched Go files after every stage.
|
|
|
|
## Stage 1: LLM Runtime
|
|
|
|
### Goal
|
|
|
|
Adapt Audita's OpenAI-compatible structured-output client, secret redaction, and
|
|
scheduler to Notarius's existing `contracts.StructuredLLMClient`.
|
|
|
|
### Files To Add
|
|
|
|
- `internal/framework/llm/client_common.go`
|
|
- `internal/framework/llm/openai_compatible_client.go`
|
|
- `internal/framework/llm/openai_compatible_client_test.go`
|
|
- `internal/framework/llm/scheduler.go`
|
|
- `internal/framework/llm/scheduler_test.go`
|
|
- `internal/framework/llm/secrets.go`
|
|
- `internal/framework/llm/secrets_test.go`
|
|
|
|
### Required API
|
|
|
|
Add:
|
|
|
|
```go
|
|
type OpenAICompatibleClientConfig struct {
|
|
BaseURL string
|
|
Model string
|
|
APIKey string
|
|
MaxRetries int
|
|
HTTPClient *http.Client
|
|
RequestTimeout time.Duration
|
|
}
|
|
|
|
type OpenAICompatibleClient struct { /* unexported fields */ }
|
|
|
|
func NewOpenAICompatibleClient(cfg OpenAICompatibleClientConfig) (*OpenAICompatibleClient, error)
|
|
func (c *OpenAICompatibleClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error)
|
|
```
|
|
|
|
Add:
|
|
|
|
```go
|
|
type Scheduler struct { /* unexported fields */ }
|
|
func NewScheduler(maxConcurrency int) (*Scheduler, error)
|
|
func (s *Scheduler) Acquire(ctx context.Context) (func(), error)
|
|
func (s *Scheduler) Run(ctx context.Context, fn func(context.Context) error) error
|
|
```
|
|
|
|
Add secret helpers:
|
|
|
|
```go
|
|
func RedactSecrets(message string, secrets []string) string
|
|
func ErrorWithSecretsRedacted(err error, secrets []string) error
|
|
```
|
|
|
|
### Notarius Adaptations
|
|
|
|
Audita's client expects a typed response schema object. Notarius currently uses:
|
|
|
|
```go
|
|
type StructuredCompletionRequest struct {
|
|
ResponseSchemaName string
|
|
ResponseSchema json.RawMessage
|
|
}
|
|
```
|
|
|
|
Therefore the Notarius client must:
|
|
|
|
- require `ResponseSchemaName` to be non-empty;
|
|
- require `ResponseSchema` to be non-empty valid JSON;
|
|
- send OpenAI-compatible `response_format.type=json_schema`;
|
|
- send `json_schema.name` from `ResponseSchemaName`;
|
|
- send `json_schema.schema` from `ResponseSchema`;
|
|
- decode the assistant message content into `out`;
|
|
- return raw assistant content in `StructuredCompletionResponse.Content`;
|
|
- populate provider/model/token metadata when available;
|
|
- use `req.Model` when present, otherwise the configured default model.
|
|
|
|
Provider request/response structs must be unexported.
|
|
|
|
### Required Behavior
|
|
|
|
- Validate `out` is a non-nil pointer.
|
|
- Reject empty base URL, empty default model, negative retries, empty model at
|
|
call time, missing schema name, and invalid schema JSON.
|
|
- Trim message roles/content and reject empty roles or content.
|
|
- Retry transport errors, malformed provider envelopes, malformed assistant JSON,
|
|
HTTP 429, and HTTP 5xx up to `MaxRetries`.
|
|
- Do not retry non-retryable 4xx provider errors.
|
|
- Respect `context.Context` cancellation.
|
|
- Redact API keys and bearer tokens from returned errors.
|
|
- Scheduler must bound concurrent calls, release permits exactly once, handle
|
|
cancellation while queued, and avoid leaking permits when cancellation races
|
|
with grant.
|
|
|
|
### Required Tests
|
|
|
|
Port and adapt Audita LLM tests using local `httptest.Server` fakes:
|
|
|
|
- successful structured completion;
|
|
- request body includes model, messages, schema name, and schema JSON;
|
|
- default model fallback and request model override;
|
|
- invalid output target;
|
|
- missing/invalid schema;
|
|
- provider non-2xx behavior;
|
|
- retry behavior for 429/5xx and malformed retryable responses;
|
|
- no retry for non-retryable 4xx;
|
|
- provider error redacts configured API key;
|
|
- scheduler max concurrency;
|
|
- scheduler cancellation while queued;
|
|
- scheduler release function is idempotent.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/framework/llm
|
|
go test ./internal/framework/llm
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 2: Embedded Response Schema Registry
|
|
|
|
### Goal
|
|
|
|
Adapt Audita's embedded response-schema registry pattern into
|
|
`internal/framework/llm` without adding real extractor schemas.
|
|
|
|
### Files To Add
|
|
|
|
- `internal/framework/llm/schema_registry.go`
|
|
- `internal/framework/llm/schema_registry_test.go`
|
|
- `internal/framework/llm/assets/schemas/test_artifact.v1.json`
|
|
- `internal/framework/llm/assets/schemas/test_validator_decision.v1.json`
|
|
|
|
### Required API
|
|
|
|
Add:
|
|
|
|
```go
|
|
type ResponseSchemaKey string
|
|
|
|
const (
|
|
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
|
|
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
|
|
)
|
|
|
|
type ResponseSchema struct {
|
|
Key ResponseSchemaKey `json:"key"`
|
|
ID string `json:"id"`
|
|
Version string `json:"version"`
|
|
Name string `json:"name"`
|
|
JSONSchema json.RawMessage `json:"json_schema"`
|
|
SHA256 string `json:"sha256"`
|
|
}
|
|
|
|
func RegisteredResponseSchemas() []ResponseSchema
|
|
func LookupResponseSchema(key ResponseSchemaKey) (ResponseSchema, bool)
|
|
func MustLookupResponseSchema(key ResponseSchemaKey) ResponseSchema
|
|
func (s ResponseSchema) DiagnosticsMap() map[string]any
|
|
```
|
|
|
|
### Required Behavior
|
|
|
|
- Embed schema assets using `embed.FS`.
|
|
- Validate assets are valid JSON at package initialization.
|
|
- Compute `SHA256` from exact asset bytes and format as `sha256:<hex>`.
|
|
- Return defensive copies of `json.RawMessage`.
|
|
- Return registered schemas sorted by key.
|
|
- Use Notarius-neutral IDs and names, for example
|
|
`notarius.test_artifact` and `notarius_test_artifact_v1`.
|
|
- Keep test schemas inert and clearly non-domain-specific.
|
|
|
|
### Required Tests
|
|
|
|
- lookup succeeds for both test schemas;
|
|
- unknown lookup returns false;
|
|
- `MustLookupResponseSchema` panics for unknown keys;
|
|
- registered schemas are sorted;
|
|
- JSON schema content is valid JSON;
|
|
- returned schema JSON is mutation-safe;
|
|
- diagnostics map omits raw schema content and includes ID, version, name, and
|
|
hash.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/framework/llm
|
|
go test ./internal/framework/llm
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 3: Prompt Registry
|
|
|
|
### Goal
|
|
|
|
Adapt Audita's embedded prompt registry and rendering pattern into
|
|
`internal/framework/prompt` using only generic test prompts.
|
|
|
|
### Files To Add
|
|
|
|
- `internal/framework/prompt/registry.go`
|
|
- `internal/framework/prompt/render.go`
|
|
- `internal/framework/prompt/registry_test.go`
|
|
- `internal/framework/prompt/render_test.go`
|
|
- `internal/framework/prompt/assets/shared/prompt_hardening.md`
|
|
- `internal/framework/prompt/assets/test/generic/system.md`
|
|
- `internal/framework/prompt/assets/test/generic/user.md`
|
|
|
|
### Required API
|
|
|
|
Add:
|
|
|
|
```go
|
|
const (
|
|
SourceBuiltin = "builtin"
|
|
VersionV1 = "v1"
|
|
TestGenericPromptID = "test.generic"
|
|
)
|
|
|
|
type Metadata struct {
|
|
PromptID string `json:"prompt_id"`
|
|
PromptVersion string `json:"prompt_version"`
|
|
PromptSource string `json:"prompt_source"`
|
|
EmbeddedPath string `json:"embedded_path"`
|
|
SHA256 string `json:"sha256"`
|
|
}
|
|
|
|
func LookupMetadata(promptID string) (Metadata, bool)
|
|
func MustLookupMetadata(promptID string) Metadata
|
|
func RegisteredMetadata() []Metadata
|
|
func HardeningText() string
|
|
func RenderUserSystem(promptID string, data any) (system string, user string, metadata Metadata, err error)
|
|
func (m Metadata) DiagnosticsMap() map[string]any
|
|
```
|
|
|
|
### Required Behavior
|
|
|
|
- Compile embedded system/user prompts with `text/template`.
|
|
- Use `Option("missingkey=error")`.
|
|
- Provide a `{{ hardening }}` template function backed by the shared hardening
|
|
asset.
|
|
- Trim rendered system/user text.
|
|
- Compute `SHA256` from combined system and user source text and format as
|
|
`sha256:<hex>`.
|
|
- Return prompt metadata sorted by prompt ID.
|
|
- Keep assets generic and non-domain-specific.
|
|
|
|
### Required Tests
|
|
|
|
- metadata lookup succeeds for `test.generic`;
|
|
- unknown lookup returns false;
|
|
- `MustLookupMetadata` panics for unknown prompt ID;
|
|
- registered metadata is sorted;
|
|
- render returns system/user text and metadata;
|
|
- missing template data returns an error;
|
|
- hardening text is available and appears when referenced;
|
|
- diagnostics map includes metadata but no rendered prompt text.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/framework/prompt
|
|
go test ./internal/framework/prompt
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 4: Diagnostics Run Directory
|
|
|
|
### Goal
|
|
|
|
Adapt Audita's diagnostics run-directory pattern using extraction-oriented
|
|
artifact names and Notarius types.
|
|
|
|
### Files To Add
|
|
|
|
- `internal/core/diagnostics/artifacts.go`
|
|
- `internal/core/diagnostics/run_dir.go`
|
|
- `internal/core/diagnostics/run_dir_test.go`
|
|
- `internal/core/diagnostics/artifacts_test.go`
|
|
|
|
### Required API
|
|
|
|
Add artifact names:
|
|
|
|
```go
|
|
const (
|
|
ArtifactInvocationMetadata = "invocation.json"
|
|
ArtifactEffectiveConfig = "effective-config.json"
|
|
ArtifactResolvedPipeline = "resolved-pipeline.json"
|
|
ArtifactSourceDocument = "source-document.json"
|
|
ArtifactRunManifest = "run-manifest.json"
|
|
ArtifactRunReport = "run-report.json"
|
|
ArtifactWarnings = "warnings.json"
|
|
ArtifactErrorLog = "error.log"
|
|
)
|
|
```
|
|
|
|
Add:
|
|
|
|
```go
|
|
type RunDirectory struct { /* unexported fields */ }
|
|
|
|
type RetentionMode string
|
|
const (
|
|
RetentionAuto RetentionMode = "auto"
|
|
RetentionAlways RetentionMode = "always"
|
|
RetentionNever RetentionMode = "never"
|
|
)
|
|
|
|
type RetentionDecisionInput struct {
|
|
RetentionMode RetentionMode
|
|
RunSucceeded bool
|
|
HasWarnings bool
|
|
}
|
|
|
|
func ShouldRetainRunDirectory(input RetentionDecisionInput) bool
|
|
func NewRunDirectory(workDir string, retention RetentionMode) (*RunDirectory, error)
|
|
func (r *RunDirectory) Path() string
|
|
func (r *RunDirectory) RunID() string
|
|
func (r *RunDirectory) WriteInvocationMetadata(metadata InvocationMetadata) error
|
|
func (r *RunDirectory) WriteRedactedEffectiveConfig(payload any) error
|
|
func (r *RunDirectory) WriteResolvedPipeline(payload any) error
|
|
func (r *RunDirectory) WriteSourceDocument(payload any) error
|
|
func (r *RunDirectory) WriteRunManifest(manifest artifacts.RunManifest) error
|
|
func (r *RunDirectory) WriteRunReport(payload any) error
|
|
func (r *RunDirectory) WriteWarnings(warnings []contracts.Warning) error
|
|
func (r *RunDirectory) WriteErrorLog(errorMessage string) error
|
|
func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error
|
|
func (r *RunDirectory) ApplyRetention(input RetentionDecisionInput) error
|
|
```
|
|
|
|
Add:
|
|
|
|
```go
|
|
type InvocationMetadata struct {
|
|
Operation string `json:"operation"`
|
|
PipelineID string `json:"pipeline_id,omitempty"`
|
|
PipelineDigest string `json:"pipeline_digest,omitempty"`
|
|
InputPath string `json:"input_path,omitempty"`
|
|
ConfigPath string `json:"config_path,omitempty"`
|
|
ConfigSource string `json:"config_source,omitempty"`
|
|
OnlyLanes []string `json:"only_lanes,omitempty"`
|
|
RunID string `json:"run_id"`
|
|
StartedAt time.Time `json:"started_at"`
|
|
}
|
|
```
|
|
|
|
### Required Behavior
|
|
|
|
- Default work directory is `/tmp/notarius`.
|
|
- Create work directory with `0o755`.
|
|
- Create unique run directories named `run-<unix-nano>`.
|
|
- Write JSON artifacts with `json.MarshalIndent` and a trailing newline.
|
|
- Write error logs as plain text with a trailing newline.
|
|
- Reject artifact names that are empty, absolute paths, contain path
|
|
separators, or resolve outside the run directory.
|
|
- Retention behavior:
|
|
- failed runs are always retained;
|
|
- `always` retains successful runs;
|
|
- `never` removes successful runs;
|
|
- `auto` retains successful runs only when warnings exist.
|
|
- Do not copy Audita names such as source transcript, normalized transcript,
|
|
correction ledger, proposal, or replacement.
|
|
|
|
### Required Tests
|
|
|
|
- run directory creation and run ID shape;
|
|
- default work directory behavior using a temporary working directory where
|
|
possible;
|
|
- JSON artifact writes are indented and newline-terminated;
|
|
- invocation metadata fills missing run ID and start time;
|
|
- error log write;
|
|
- artifact path rejection for unsafe names;
|
|
- retention decisions for failed/successful runs across modes;
|
|
- `ApplyRetention` removes only the run directory for removable cases.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/core/diagnostics
|
|
go test ./internal/core/diagnostics
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 5: Config Model, YAML Loading, Redaction, And Environment Overrides
|
|
|
|
### Goal
|
|
|
|
Add Notarius config structs and loading primitives for named pipeline profiles,
|
|
LLM profiles, operational settings, YAML files, redaction, and environment
|
|
overrides.
|
|
|
|
### Files To Add
|
|
|
|
- `internal/core/config/config.go`
|
|
- `internal/core/config/file_config.go`
|
|
- `internal/core/config/env.go`
|
|
- `internal/core/config/redaction.go`
|
|
- `internal/core/config/config_test.go`
|
|
- `internal/core/config/file_config_test.go`
|
|
- `internal/core/config/env_test.go`
|
|
- `internal/core/config/redaction_test.go`
|
|
|
|
### Dependency Change
|
|
|
|
Add:
|
|
|
|
```sh
|
|
go get gopkg.in/yaml.v3@v3.0.1
|
|
```
|
|
|
|
Commit the resulting `go.mod` and `go.sum` changes.
|
|
|
|
### Required API
|
|
|
|
Add core config types:
|
|
|
|
```go
|
|
const SupportedFileConfigVersion = 1
|
|
|
|
type Config struct {
|
|
LLMProfiles map[string]LLMProfile
|
|
Pipelines map[string]pipeline.PipelineProfile
|
|
Concurrency ConcurrencyConfig
|
|
Diagnostics DiagnosticsConfig
|
|
}
|
|
|
|
type LLMProfile struct {
|
|
Provider string
|
|
BaseURL string
|
|
Model string
|
|
APIKey string
|
|
APIKeyEnv string
|
|
TimeoutSeconds int
|
|
MaxRetries int
|
|
MaxConcurrency int
|
|
}
|
|
|
|
type ConcurrencyConfig struct {
|
|
TotalLLM int
|
|
}
|
|
|
|
type DiagnosticsConfig struct {
|
|
WorkDir string
|
|
Retention diagnostics.RetentionMode
|
|
}
|
|
|
|
func Default() Config
|
|
func (c Config) Redacted() Config
|
|
func (c *Config) ApplyEnvOverrides() error
|
|
func LoadFromEnv() (Config, error)
|
|
```
|
|
|
|
Add file config loading:
|
|
|
|
```go
|
|
type FileConfig struct { /* YAML-facing fields */ }
|
|
func LoadFileConfig(path string) (FileConfig, error)
|
|
func ParseFileConfigYAML(data []byte) (FileConfig, error)
|
|
func (c *Config) ApplyFileConfig(fileCfg FileConfig) error
|
|
```
|
|
|
|
Add unexported lookup seams for tests:
|
|
|
|
```go
|
|
func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(string) (string, bool)) error
|
|
func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool)) error
|
|
```
|
|
|
|
### YAML Shape
|
|
|
|
Support this shape:
|
|
|
|
```yaml
|
|
version: 1
|
|
llm_profiles:
|
|
default:
|
|
provider: openai-compatible
|
|
base_url: https://example.invalid/v1
|
|
model: test-model
|
|
api_key_env: NOTARIUS_TEST_API_KEY
|
|
timeout: 600s
|
|
max_retries: 3
|
|
max_concurrency: 1
|
|
pipelines:
|
|
example:
|
|
input: fake/input
|
|
chunk: generic
|
|
artifacts:
|
|
events:
|
|
extract:
|
|
module: fake/extract
|
|
llm_profile: default
|
|
options:
|
|
temperature: 0
|
|
merge: appendorder
|
|
normalize: noop
|
|
validators:
|
|
- fake/validator
|
|
output: json
|
|
concurrency:
|
|
total_llm: 1
|
|
diagnostics:
|
|
work_dir: /tmp/notarius
|
|
retention: auto
|
|
```
|
|
|
|
Module bindings must support both string shorthand and object form:
|
|
|
|
```yaml
|
|
extract: fake/extract
|
|
extract:
|
|
module: fake/extract
|
|
llm_profile: fast
|
|
options:
|
|
key: value
|
|
```
|
|
|
|
Validator bindings must support a sequence of string or object bindings.
|
|
|
|
### Defaults
|
|
|
|
`Default()` must provide:
|
|
|
|
- `LLMProfiles["default"]` with provider `openai-compatible`,
|
|
`TimeoutSeconds: 600`, `MaxRetries: 3`, and `MaxConcurrency: 1`;
|
|
- no built-in pipeline profiles;
|
|
- `Concurrency.TotalLLM: 1`;
|
|
- `Diagnostics.WorkDir: "/tmp/notarius"`;
|
|
- `Diagnostics.Retention: diagnostics.RetentionAuto`.
|
|
|
|
Config validation will later decide whether a selected LLM profile is complete
|
|
enough to build a real client. This stage should not require default `BaseURL`
|
|
or `Model` values.
|
|
|
|
### Environment Variables
|
|
|
|
Support these environment overrides:
|
|
|
|
- `NOTARIUS_LLM_DEFAULT_API_KEY`
|
|
- `NOTARIUS_LLM_DEFAULT_BASE_URL`
|
|
- `NOTARIUS_LLM_DEFAULT_MODEL`
|
|
- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS`
|
|
- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES`
|
|
- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY`
|
|
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`
|
|
- `NOTARIUS_WORK_DIR`
|
|
- `NOTARIUS_DIAGNOSTICS_RETENTION`
|
|
|
|
Do not support environment variables that structurally change pipeline module
|
|
selection.
|
|
|
|
### Required Behavior
|
|
|
|
- YAML parsing must use `KnownFields(true)`.
|
|
- Config version is required and must equal `SupportedFileConfigVersion`.
|
|
- Duration fields accept integer seconds or Go duration strings resolving to
|
|
whole seconds.
|
|
- `api_key_env` must be a valid environment-variable name and must resolve via
|
|
the lookup function used by tests.
|
|
- Redaction must replace all API key values with `[REDACTED]` while preserving
|
|
non-secret fields.
|
|
- Applying a file config must merge into defaults, not replace unspecified
|
|
sections with zero values.
|
|
- Environment overrides apply after file config.
|
|
- Object-form `options` must preserve YAML scalar/list/map values as
|
|
`map[string]any` suitable for `pipeline.ModuleBinding.Options`.
|
|
|
|
### Required Tests
|
|
|
|
- default values;
|
|
- parse minimal valid config;
|
|
- reject unknown YAML fields;
|
|
- reject missing/unsupported version;
|
|
- string and object module-binding forms;
|
|
- validator binding list with mixed shorthand/object forms;
|
|
- duration parsing;
|
|
- API key env resolution and invalid env-name rejection;
|
|
- file config merges with defaults;
|
|
- environment overrides operational and LLM values;
|
|
- environment variables cannot change pipeline module wiring;
|
|
- redacted config removes API key values.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/core/config
|
|
go test ./internal/core/config
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 6: Config Validation And Pipeline Resolution
|
|
|
|
### Goal
|
|
|
|
Validate named pipeline profiles against registered module metadata, LLM
|
|
profiles, lane selections, and resolved pipeline digest behavior.
|
|
|
|
### Files To Add
|
|
|
|
- `internal/core/config/validation.go`
|
|
- `internal/core/config/effective_config.go`
|
|
- `internal/core/config/validation_test.go`
|
|
- `internal/core/config/effective_config_test.go`
|
|
|
|
### Required API
|
|
|
|
Add:
|
|
|
|
```go
|
|
type ResolveInput struct {
|
|
PipelineID string
|
|
Only []string
|
|
Catalog pipeline.ModuleCatalog
|
|
}
|
|
|
|
type EffectiveConfig struct {
|
|
Config Config
|
|
PipelineID string
|
|
Only []string
|
|
ResolvedPipeline pipeline.ResolvedPipeline
|
|
}
|
|
|
|
func (c Config) Validate() error
|
|
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error)
|
|
func (c Config) LLMProfile(id string) (LLMProfile, bool)
|
|
func (c Config) OpenAICompatibleClientConfig(profileID string) (llm.OpenAICompatibleClientConfig, error)
|
|
```
|
|
|
|
### Required Behavior
|
|
|
|
`Validate()` must check config-internal rules that do not need a module catalog:
|
|
|
|
- LLM profile IDs trim to non-empty unique IDs;
|
|
- pipeline IDs trim to non-empty unique IDs;
|
|
- LLM profile provider must be empty or `openai-compatible`;
|
|
- LLM profile numeric values must be positive where applicable;
|
|
- diagnostics work dir must not be empty;
|
|
- diagnostics retention must be one of `auto`, `always`, or `never`;
|
|
- `TotalLLM` must be greater than zero;
|
|
- every LLM profile referenced by a pipeline binding must exist.
|
|
|
|
`Resolve(input)` must:
|
|
|
|
- call `Validate()`;
|
|
- trim `PipelineID`;
|
|
- reject empty or unknown pipeline IDs;
|
|
- pass `Only` through to `pipeline.ResolvePipeline`;
|
|
- return the resolved pipeline and digest;
|
|
- surface errors that name the pipeline and lane where possible.
|
|
|
|
`OpenAICompatibleClientConfig(profileID)` must:
|
|
|
|
- reject unknown profile IDs;
|
|
- reject unsupported providers;
|
|
- require base URL and model;
|
|
- pass API key, timeout, and max retries to the LLM client config.
|
|
|
|
### Required Tests
|
|
|
|
- `Validate` success for valid config;
|
|
- unknown LLM profile referenced by binding;
|
|
- invalid provider;
|
|
- invalid numeric fields;
|
|
- invalid diagnostics retention;
|
|
- empty/unknown selected pipeline ID;
|
|
- lane filtering success and failure;
|
|
- unknown module key through module catalog;
|
|
- missing capability through module catalog;
|
|
- resolved pipeline digest changes when effective config changes;
|
|
- `OpenAICompatibleClientConfig` rejects incomplete default profile and succeeds
|
|
when base URL/model are set.
|
|
|
|
Use fake module registries and fake module specs. Do not add real modules.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/core/config
|
|
go test ./internal/core/config
|
|
go test ./...
|
|
```
|
|
|
|
## Stage 7: CLI Config Validation And Pipeline Listing
|
|
|
|
### Goal
|
|
|
|
Add minimal CLI support for config discovery/validation and pipeline profile
|
|
listing without adding run execution.
|
|
|
|
### Files To Update
|
|
|
|
- `internal/cli/run.go`
|
|
- `internal/cli/run_test.go`
|
|
|
|
### Required CLI
|
|
|
|
Support:
|
|
|
|
```sh
|
|
notarius config validate --config path/to/config.yml
|
|
notarius config validate --config path/to/config.yml --pipeline pipeline-id
|
|
notarius config validate --config path/to/config.yml --pipeline pipeline-id --only lane-a,lane-b
|
|
notarius pipelines list --config path/to/config.yml
|
|
notarius pipelines list --config path/to/config.yml --json
|
|
```
|
|
|
|
Also support the default config search path from architecture policy:
|
|
|
|
1. `--config` path, when provided;
|
|
2. `NOTARIUS_CONFIG`, when set;
|
|
3. `/usr/local/etc/notarius/config.yml`, when present.
|
|
|
|
Do not add `notarius run` in this checkpoint.
|
|
|
|
### Required Implementation Shape
|
|
|
|
The CLI needs module metadata to validate structural pipeline config. Add an
|
|
internal test seam rather than hard-coding fake modules:
|
|
|
|
```go
|
|
type Options struct {
|
|
Catalog pipeline.ModuleCatalog
|
|
LookupEnv func(string) (string, bool)
|
|
}
|
|
|
|
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int
|
|
```
|
|
|
|
Keep the existing `Run(args, stdout, stderr)` as a wrapper that passes default
|
|
options. Default production options may contain empty registries until real
|
|
modules are registered by later checkpoints.
|
|
|
|
### Required Behavior
|
|
|
|
- `config validate` exits `0` and writes a concise success message on valid
|
|
config.
|
|
- `config validate` exits `1` on config load/validation errors and writes the
|
|
error to stderr.
|
|
- `config validate --pipeline` resolves that pipeline against the supplied
|
|
catalog.
|
|
- `--only` is valid only when `--pipeline` is present.
|
|
- `pipelines list` exits `0` and lists pipeline IDs sorted alphabetically.
|
|
- `pipelines list --json` emits stable JSON, for example:
|
|
|
|
```json
|
|
{"pipelines":["a","b"]}
|
|
```
|
|
|
|
- Unknown commands and invalid flags exit `2`.
|
|
- Usage text must include the new commands.
|
|
- CLI flags must not allow ad hoc structural module overrides such as
|
|
`--extractor`, `--chunker`, `--input`, `--merge`, or `--normalize`.
|
|
|
|
### Required Tests
|
|
|
|
- existing help/unknown-command tests still pass;
|
|
- `config validate` success with fake catalog;
|
|
- `config validate` reports config parse errors;
|
|
- `config validate --pipeline --only` success and invalid lane failure;
|
|
- `--only` without `--pipeline` fails;
|
|
- `pipelines list` sorted text output;
|
|
- `pipelines list --json` stable JSON output;
|
|
- `NOTARIUS_CONFIG` is used when `--config` is absent;
|
|
- missing config path produces actionable error;
|
|
- no ad hoc structural flags are accepted.
|
|
|
|
### Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal/cli
|
|
go test ./internal/cli
|
|
go test ./...
|
|
go build ./cmd/notarius
|
|
rm -f ./notarius
|
|
```
|
|
|
|
## Stage 8: Final Checkpoint 4 Review Pass
|
|
|
|
### Goal
|
|
|
|
Remove accidental Audita coupling, confirm scope boundaries, and verify all
|
|
runtime pieces compose.
|
|
|
|
### Required Review
|
|
|
|
Check:
|
|
|
|
- no correction proposal, replacement policy, transcript mutation, correction
|
|
ledger, or Audita module behavior was copied;
|
|
- no package names or artifact names mention corrections, proposals,
|
|
replacements, or transcripts unless they refer to Audita source files in
|
|
comments inside tests;
|
|
- no real D&D prompts or schemas were added;
|
|
- no Seriatim input adapter was added;
|
|
- no `notarius run` behavior was added;
|
|
- provider-specific HTTP structs stay unexported inside `internal/framework/llm`;
|
|
- response schemas live inside `internal/framework/llm`, not a new
|
|
`responseschema` package;
|
|
- prompt assets live inside `internal/framework/prompt`;
|
|
- config is centered on named pipeline profiles;
|
|
- module selection comes from config, not ad hoc CLI flags;
|
|
- diagnostics artifact names are extraction-oriented;
|
|
- secrets are redacted from effective config and surfaced provider errors.
|
|
|
|
### Required Validation
|
|
|
|
Run:
|
|
|
|
```sh
|
|
gofmt -w internal
|
|
go test ./...
|
|
go vet ./...
|
|
go build ./cmd/notarius
|
|
rm -f ./notarius
|
|
git status --short
|
|
```
|
|
|
|
The implementation response should summarize:
|
|
|
|
- files/packages added;
|
|
- tests run;
|
|
- any deviations from this plan and why.
|
|
|
|
## Open Questions
|
|
|
|
None. The plan intentionally defers real extractor prompts/schemas, concrete
|
|
input modules, real built-in module catalogs, and `notarius run` execution to
|
|
later checkpoints.
|