From 1d66934577f93317f12ceb9406a7a4bdc713c337 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 3 Jul 2026 11:32:43 -0500 Subject: [PATCH] Add a staged implementation plan for porting relevant code from audita --- .../4-portable-audita-infrastructure.md | 90 +- docs/roadmap/implementation.md | 915 ++++++++++++++++++ 2 files changed, 940 insertions(+), 65 deletions(-) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/4-portable-audita-infrastructure.md b/docs/roadmap/4-portable-audita-infrastructure.md index 155d050..595fa57 100644 --- a/docs/roadmap/4-portable-audita-infrastructure.md +++ b/docs/roadmap/4-portable-audita-infrastructure.md @@ -31,73 +31,30 @@ Out of scope: - transcript mutation; - correction ledger terminology; - Audita module or validator behavior; -- real D&D prompts or schemas unless needed as inert registry tests. +- real D&D prompts or schemas; +- domain-specific prompt or schema assets; - embedded built-in pipeline profiles. -## Proposed Stages +## Target End State -### Stage 1: LLM Runtime +The repository should contain reusable runtime infrastructure adapted from +Audita where it directly supports Notarius contracts: -Port or adapt the OpenAI-compatible structured-output client and scheduler. +- an OpenAI-compatible structured-output LLM client behind the existing + `StructuredLLMClient` interface; +- an LLM scheduler for bounded concurrency; +- an embedded response-schema registry pattern; +- an embedded prompt registry pattern; +- a diagnostics run directory pattern using extraction-oriented artifact names; +- config structs, loading, defaults, redaction, and validation for named + pipeline profiles. -Keep the contract transport-neutral: +Framework code should remain source-agnostic and domain-agnostic. Provider HTTP +details should stay inside the LLM runtime package. Prompt and schema registries +should use placeholder/test assets until real extractor prompts and schemas are +introduced by later checkpoints. -- framework code should depend on a `StructuredLLMClient` interface; -- provider-specific HTTP details should remain in the LLM runtime package; -- errors must redact configured secrets. - -### Stage 2: Response Schema Registry - -Port or adapt the embedded JSON response-schema registry pattern. - -The registry should track: - -- schema key; -- schema ID; -- schema version; -- schema name; -- JSON schema content; -- schema hash. - -Use placeholder or test schemas if real extractor schemas are not ready. - -### Stage 3: Prompt Registry - -Port or adapt the embedded prompt registry pattern. - -The registry should track: - -- prompt ID; -- prompt version; -- prompt source; -- embedded path; -- prompt hash. - -Do not add D&D prompt assets here unless the implementation naturally overlaps -with checkpoint 6. Test prompts are acceptable for registry tests. - -### Stage 4: Diagnostics Run Directory - -Port or adapt the diagnostics run directory pattern. - -Initial diagnostics should cover: - -- invocation metadata; -- redacted effective config; -- selected pipeline ID; -- resolved pipeline definition and digest; -- source document artifact; -- run report placeholder; -- error log on failure. - -Avoid Audita-specific artifact names such as correction ledger. - -### Stage 5: Pipeline Profile Config - -Add config structs, loading, defaults, and validation for named pipeline -profiles. - -Initial config areas: +Config should support: - `llm_profiles`; - `pipelines..input`; @@ -123,17 +80,20 @@ Structural module selection should come from pipeline config. CLI flags may override operational settings and artifact lane selection, but should not offer ad hoc `--extractor` or `--chunker` wiring. -Add validation for unknown pipeline IDs, unknown module keys, missing required -slots, missing capabilities, unknown LLM profiles, empty artifact-lane sets, and -invalid lane selections. +Config validation should fail fast for unknown pipeline IDs, unknown module +keys, missing required slots, missing capabilities, unknown LLM profiles, empty +artifact-lane sets, and invalid lane selections. -If the CLI shell is ready, add: +If the CLI shell is ready, the checkpoint should expose discovery/validation +commands for config and pipeline profiles: ```sh notarius config validate notarius pipelines list ``` +Implementation staging belongs in [`implementation.md`](implementation.md). + ## Done Criteria - `go test ./...` passes. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..ad23c69 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,915 @@ +# Implementation Plan: Checkpoint 4 Portable Audita Infrastructure + +## Status + +This is a staged implementation plan for +[`4-portable-audita-infrastructure.md`](4-portable-audita-infrastructure.md). +It is intended for an LLM coding agent to follow stage by stage. + +This plan implements only checkpoint 4. Do not add real Seriatim parsing, real +D&D extraction, real domain prompts or schemas, production extraction modules, +durable final artifact writing, or `notarius run` execution behavior in this +checkpoint. + +## Policy Context + +Follow: + +- [`docs/policy/architecture.md`](../policy/architecture.md) +- [`docs/policy/documentation.md`](../policy/documentation.md) + +Required boundaries: + +- copy only reusable Audita infrastructure, not Audita's transcript-correction + model; +- keep framework code source-agnostic and domain-agnostic; +- keep provider-specific HTTP request/response details inside + `internal/framework/llm`; +- keep prompt construction and prompt assets out of provider adapters; +- keep structural pipeline wiring in named pipeline config, not ad hoc CLI + flags; +- preserve the fixed pipeline shape: + `input -> chunk -> extract -> merge -> normalize -> output`; +- keep capabilities as flat strings; +- redact configured secrets from diagnostics, effective config, and surfaced + provider errors. + +Audita files useful for adaptation: + +- `../audita/internal/framework/llm/*` +- `../audita/internal/framework/responseschema/registry.go` +- `../audita/internal/prompts/registry.go` +- `../audita/internal/prompts/render.go` +- `../audita/internal/core/diagnostics/*` +- `../audita/internal/core/config/*` +- `../audita/internal/cli/*` +- `../audita/examples/*.yml` + +## Global Implementation Decisions + +- Add `gopkg.in/yaml.v3` as the only new third-party dependency in this + checkpoint. Audita already uses it, and YAML is the architecture-policy + preference for Notarius config. +- Put LLM client, scheduler, structured-output decoding helpers, secret + redaction, and response-schema registry code in `internal/framework/llm`. + Do not create separate `responseschema`, `structuredoutput`, or `warnings` + framework packages. +- Put prompt registry and rendering code in `internal/framework/prompt`. +- Put config structs, defaults, file/env/flag precedence helpers, redaction, and + validation in `internal/core/config`. +- Put diagnostics run-directory code in `internal/core/diagnostics`. This + package is justified by real durable diagnostic state in checkpoint 4; keep it + extraction-oriented and do not copy correction-ledger concepts. +- Keep CLI support limited to discovery/validation commands: + `notarius config validate` and `notarius pipelines list`. +- Do not add embedded built-in pipeline profiles. Config files are the only + source of user-defined pipeline profiles in this checkpoint. +- Do not add real D&D prompt/schema assets. Use inert placeholder assets only + for registry tests. +- Use JSON for diagnostics artifacts even when the input config is YAML. +- Use `sha256:` for content digests written to diagnostics or registries, + unless an existing Notarius type already expects bare hex. Prefer the prefixed + form for new Notarius code. +- Run `gofmt` on touched Go files after every stage. + +## Stage 1: LLM Runtime + +### Goal + +Adapt Audita's OpenAI-compatible structured-output client, secret redaction, and +scheduler to Notarius's existing `contracts.StructuredLLMClient`. + +### Files To Add + +- `internal/framework/llm/client_common.go` +- `internal/framework/llm/openai_compatible_client.go` +- `internal/framework/llm/openai_compatible_client_test.go` +- `internal/framework/llm/scheduler.go` +- `internal/framework/llm/scheduler_test.go` +- `internal/framework/llm/secrets.go` +- `internal/framework/llm/secrets_test.go` + +### Required API + +Add: + +```go +type OpenAICompatibleClientConfig struct { + BaseURL string + Model string + APIKey string + MaxRetries int + HTTPClient *http.Client + RequestTimeout time.Duration +} + +type OpenAICompatibleClient struct { /* unexported fields */ } + +func NewOpenAICompatibleClient(cfg OpenAICompatibleClientConfig) (*OpenAICompatibleClient, error) +func (c *OpenAICompatibleClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) +``` + +Add: + +```go +type Scheduler struct { /* unexported fields */ } +func NewScheduler(maxConcurrency int) (*Scheduler, error) +func (s *Scheduler) Acquire(ctx context.Context) (func(), error) +func (s *Scheduler) Run(ctx context.Context, fn func(context.Context) error) error +``` + +Add secret helpers: + +```go +func RedactSecrets(message string, secrets []string) string +func ErrorWithSecretsRedacted(err error, secrets []string) error +``` + +### Notarius Adaptations + +Audita's client expects a typed response schema object. Notarius currently uses: + +```go +type StructuredCompletionRequest struct { + ResponseSchemaName string + ResponseSchema json.RawMessage +} +``` + +Therefore the Notarius client must: + +- require `ResponseSchemaName` to be non-empty; +- require `ResponseSchema` to be non-empty valid JSON; +- send OpenAI-compatible `response_format.type=json_schema`; +- send `json_schema.name` from `ResponseSchemaName`; +- send `json_schema.schema` from `ResponseSchema`; +- decode the assistant message content into `out`; +- return raw assistant content in `StructuredCompletionResponse.Content`; +- populate provider/model/token metadata when available; +- use `req.Model` when present, otherwise the configured default model. + +Provider request/response structs must be unexported. + +### Required Behavior + +- Validate `out` is a non-nil pointer. +- Reject empty base URL, empty default model, negative retries, empty model at + call time, missing schema name, and invalid schema JSON. +- Trim message roles/content and reject empty roles or content. +- Retry transport errors, malformed provider envelopes, malformed assistant JSON, + HTTP 429, and HTTP 5xx up to `MaxRetries`. +- Do not retry non-retryable 4xx provider errors. +- Respect `context.Context` cancellation. +- Redact API keys and bearer tokens from returned errors. +- Scheduler must bound concurrent calls, release permits exactly once, handle + cancellation while queued, and avoid leaking permits when cancellation races + with grant. + +### Required Tests + +Port and adapt Audita LLM tests using local `httptest.Server` fakes: + +- successful structured completion; +- request body includes model, messages, schema name, and schema JSON; +- default model fallback and request model override; +- invalid output target; +- missing/invalid schema; +- provider non-2xx behavior; +- retry behavior for 429/5xx and malformed retryable responses; +- no retry for non-retryable 4xx; +- provider error redacts configured API key; +- scheduler max concurrency; +- scheduler cancellation while queued; +- scheduler release function is idempotent. + +### Validation + +Run: + +```sh +gofmt -w internal/framework/llm +go test ./internal/framework/llm +go test ./... +``` + +## Stage 2: Embedded Response Schema Registry + +### Goal + +Adapt Audita's embedded response-schema registry pattern into +`internal/framework/llm` without adding real extractor schemas. + +### Files To Add + +- `internal/framework/llm/schema_registry.go` +- `internal/framework/llm/schema_registry_test.go` +- `internal/framework/llm/assets/schemas/test_artifact.v1.json` +- `internal/framework/llm/assets/schemas/test_validator_decision.v1.json` + +### Required API + +Add: + +```go +type ResponseSchemaKey string + +const ( + TestArtifactSchemaKey ResponseSchemaKey = "test_artifact" + TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision" +) + +type ResponseSchema struct { + Key ResponseSchemaKey `json:"key"` + ID string `json:"id"` + Version string `json:"version"` + Name string `json:"name"` + JSONSchema json.RawMessage `json:"json_schema"` + SHA256 string `json:"sha256"` +} + +func RegisteredResponseSchemas() []ResponseSchema +func LookupResponseSchema(key ResponseSchemaKey) (ResponseSchema, bool) +func MustLookupResponseSchema(key ResponseSchemaKey) ResponseSchema +func (s ResponseSchema) DiagnosticsMap() map[string]any +``` + +### Required Behavior + +- Embed schema assets using `embed.FS`. +- Validate assets are valid JSON at package initialization. +- Compute `SHA256` from exact asset bytes and format as `sha256:`. +- Return defensive copies of `json.RawMessage`. +- Return registered schemas sorted by key. +- Use Notarius-neutral IDs and names, for example + `notarius.test_artifact` and `notarius_test_artifact_v1`. +- Keep test schemas inert and clearly non-domain-specific. + +### Required Tests + +- lookup succeeds for both test schemas; +- unknown lookup returns false; +- `MustLookupResponseSchema` panics for unknown keys; +- registered schemas are sorted; +- JSON schema content is valid JSON; +- returned schema JSON is mutation-safe; +- diagnostics map omits raw schema content and includes ID, version, name, and + hash. + +### Validation + +Run: + +```sh +gofmt -w internal/framework/llm +go test ./internal/framework/llm +go test ./... +``` + +## Stage 3: Prompt Registry + +### Goal + +Adapt Audita's embedded prompt registry and rendering pattern into +`internal/framework/prompt` using only generic test prompts. + +### Files To Add + +- `internal/framework/prompt/registry.go` +- `internal/framework/prompt/render.go` +- `internal/framework/prompt/registry_test.go` +- `internal/framework/prompt/render_test.go` +- `internal/framework/prompt/assets/shared/prompt_hardening.md` +- `internal/framework/prompt/assets/test/generic/system.md` +- `internal/framework/prompt/assets/test/generic/user.md` + +### Required API + +Add: + +```go +const ( + SourceBuiltin = "builtin" + VersionV1 = "v1" + TestGenericPromptID = "test.generic" +) + +type Metadata struct { + PromptID string `json:"prompt_id"` + PromptVersion string `json:"prompt_version"` + PromptSource string `json:"prompt_source"` + EmbeddedPath string `json:"embedded_path"` + SHA256 string `json:"sha256"` +} + +func LookupMetadata(promptID string) (Metadata, bool) +func MustLookupMetadata(promptID string) Metadata +func RegisteredMetadata() []Metadata +func HardeningText() string +func RenderUserSystem(promptID string, data any) (system string, user string, metadata Metadata, err error) +func (m Metadata) DiagnosticsMap() map[string]any +``` + +### Required Behavior + +- Compile embedded system/user prompts with `text/template`. +- Use `Option("missingkey=error")`. +- Provide a `{{ hardening }}` template function backed by the shared hardening + asset. +- Trim rendered system/user text. +- Compute `SHA256` from combined system and user source text and format as + `sha256:`. +- Return prompt metadata sorted by prompt ID. +- Keep assets generic and non-domain-specific. + +### Required Tests + +- metadata lookup succeeds for `test.generic`; +- unknown lookup returns false; +- `MustLookupMetadata` panics for unknown prompt ID; +- registered metadata is sorted; +- render returns system/user text and metadata; +- missing template data returns an error; +- hardening text is available and appears when referenced; +- diagnostics map includes metadata but no rendered prompt text. + +### Validation + +Run: + +```sh +gofmt -w internal/framework/prompt +go test ./internal/framework/prompt +go test ./... +``` + +## Stage 4: Diagnostics Run Directory + +### Goal + +Adapt Audita's diagnostics run-directory pattern using extraction-oriented +artifact names and Notarius types. + +### Files To Add + +- `internal/core/diagnostics/artifacts.go` +- `internal/core/diagnostics/run_dir.go` +- `internal/core/diagnostics/run_dir_test.go` +- `internal/core/diagnostics/artifacts_test.go` + +### Required API + +Add artifact names: + +```go +const ( + ArtifactInvocationMetadata = "invocation.json" + ArtifactEffectiveConfig = "effective-config.json" + ArtifactResolvedPipeline = "resolved-pipeline.json" + ArtifactSourceDocument = "source-document.json" + ArtifactRunManifest = "run-manifest.json" + ArtifactRunReport = "run-report.json" + ArtifactWarnings = "warnings.json" + ArtifactErrorLog = "error.log" +) +``` + +Add: + +```go +type RunDirectory struct { /* unexported fields */ } + +type RetentionMode string +const ( + RetentionAuto RetentionMode = "auto" + RetentionAlways RetentionMode = "always" + RetentionNever RetentionMode = "never" +) + +type RetentionDecisionInput struct { + RetentionMode RetentionMode + RunSucceeded bool + HasWarnings bool +} + +func ShouldRetainRunDirectory(input RetentionDecisionInput) bool +func NewRunDirectory(workDir string, retention RetentionMode) (*RunDirectory, error) +func (r *RunDirectory) Path() string +func (r *RunDirectory) RunID() string +func (r *RunDirectory) WriteInvocationMetadata(metadata InvocationMetadata) error +func (r *RunDirectory) WriteRedactedEffectiveConfig(payload any) error +func (r *RunDirectory) WriteResolvedPipeline(payload any) error +func (r *RunDirectory) WriteSourceDocument(payload any) error +func (r *RunDirectory) WriteRunManifest(manifest artifacts.RunManifest) error +func (r *RunDirectory) WriteRunReport(payload any) error +func (r *RunDirectory) WriteWarnings(warnings []contracts.Warning) error +func (r *RunDirectory) WriteErrorLog(errorMessage string) error +func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error +func (r *RunDirectory) ApplyRetention(input RetentionDecisionInput) error +``` + +Add: + +```go +type InvocationMetadata struct { + Operation string `json:"operation"` + PipelineID string `json:"pipeline_id,omitempty"` + PipelineDigest string `json:"pipeline_digest,omitempty"` + InputPath string `json:"input_path,omitempty"` + ConfigPath string `json:"config_path,omitempty"` + ConfigSource string `json:"config_source,omitempty"` + OnlyLanes []string `json:"only_lanes,omitempty"` + RunID string `json:"run_id"` + StartedAt time.Time `json:"started_at"` +} +``` + +### Required Behavior + +- Default work directory is `/tmp/notarius`. +- Create work directory with `0o755`. +- Create unique run directories named `run-`. +- Write JSON artifacts with `json.MarshalIndent` and a trailing newline. +- Write error logs as plain text with a trailing newline. +- Reject artifact names that are empty, absolute paths, contain path + separators, or resolve outside the run directory. +- Retention behavior: + - failed runs are always retained; + - `always` retains successful runs; + - `never` removes successful runs; + - `auto` retains successful runs only when warnings exist. +- Do not copy Audita names such as source transcript, normalized transcript, + correction ledger, proposal, or replacement. + +### Required Tests + +- run directory creation and run ID shape; +- default work directory behavior using a temporary working directory where + possible; +- JSON artifact writes are indented and newline-terminated; +- invocation metadata fills missing run ID and start time; +- error log write; +- artifact path rejection for unsafe names; +- retention decisions for failed/successful runs across modes; +- `ApplyRetention` removes only the run directory for removable cases. + +### Validation + +Run: + +```sh +gofmt -w internal/core/diagnostics +go test ./internal/core/diagnostics +go test ./... +``` + +## Stage 5: Config Model, YAML Loading, Redaction, And Environment Overrides + +### Goal + +Add Notarius config structs and loading primitives for named pipeline profiles, +LLM profiles, operational settings, YAML files, redaction, and environment +overrides. + +### Files To Add + +- `internal/core/config/config.go` +- `internal/core/config/file_config.go` +- `internal/core/config/env.go` +- `internal/core/config/redaction.go` +- `internal/core/config/config_test.go` +- `internal/core/config/file_config_test.go` +- `internal/core/config/env_test.go` +- `internal/core/config/redaction_test.go` + +### Dependency Change + +Add: + +```sh +go get gopkg.in/yaml.v3@v3.0.1 +``` + +Commit the resulting `go.mod` and `go.sum` changes. + +### Required API + +Add core config types: + +```go +const SupportedFileConfigVersion = 1 + +type Config struct { + LLMProfiles map[string]LLMProfile + Pipelines map[string]pipeline.PipelineProfile + Concurrency ConcurrencyConfig + Diagnostics DiagnosticsConfig +} + +type LLMProfile struct { + Provider string + BaseURL string + Model string + APIKey string + APIKeyEnv string + TimeoutSeconds int + MaxRetries int + MaxConcurrency int +} + +type ConcurrencyConfig struct { + TotalLLM int +} + +type DiagnosticsConfig struct { + WorkDir string + Retention diagnostics.RetentionMode +} + +func Default() Config +func (c Config) Redacted() Config +func (c *Config) ApplyEnvOverrides() error +func LoadFromEnv() (Config, error) +``` + +Add file config loading: + +```go +type FileConfig struct { /* YAML-facing fields */ } +func LoadFileConfig(path string) (FileConfig, error) +func ParseFileConfigYAML(data []byte) (FileConfig, error) +func (c *Config) ApplyFileConfig(fileCfg FileConfig) error +``` + +Add unexported lookup seams for tests: + +```go +func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(string) (string, bool)) error +func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool)) error +``` + +### YAML Shape + +Support this shape: + +```yaml +version: 1 +llm_profiles: + default: + provider: openai-compatible + base_url: https://example.invalid/v1 + model: test-model + api_key_env: NOTARIUS_TEST_API_KEY + timeout: 600s + max_retries: 3 + max_concurrency: 1 +pipelines: + example: + input: fake/input + chunk: generic + artifacts: + events: + extract: + module: fake/extract + llm_profile: default + options: + temperature: 0 + merge: appendorder + normalize: noop + validators: + - fake/validator + output: json +concurrency: + total_llm: 1 +diagnostics: + work_dir: /tmp/notarius + retention: auto +``` + +Module bindings must support both string shorthand and object form: + +```yaml +extract: fake/extract +extract: + module: fake/extract + llm_profile: fast + options: + key: value +``` + +Validator bindings must support a sequence of string or object bindings. + +### Defaults + +`Default()` must provide: + +- `LLMProfiles["default"]` with provider `openai-compatible`, + `TimeoutSeconds: 600`, `MaxRetries: 3`, and `MaxConcurrency: 1`; +- no built-in pipeline profiles; +- `Concurrency.TotalLLM: 1`; +- `Diagnostics.WorkDir: "/tmp/notarius"`; +- `Diagnostics.Retention: diagnostics.RetentionAuto`. + +Config validation will later decide whether a selected LLM profile is complete +enough to build a real client. This stage should not require default `BaseURL` +or `Model` values. + +### Environment Variables + +Support these environment overrides: + +- `NOTARIUS_LLM_DEFAULT_API_KEY` +- `NOTARIUS_LLM_DEFAULT_BASE_URL` +- `NOTARIUS_LLM_DEFAULT_MODEL` +- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS` +- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES` +- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY` +- `NOTARIUS_TOTAL_LLM_CONCURRENCY` +- `NOTARIUS_WORK_DIR` +- `NOTARIUS_DIAGNOSTICS_RETENTION` + +Do not support environment variables that structurally change pipeline module +selection. + +### Required Behavior + +- YAML parsing must use `KnownFields(true)`. +- Config version is required and must equal `SupportedFileConfigVersion`. +- Duration fields accept integer seconds or Go duration strings resolving to + whole seconds. +- `api_key_env` must be a valid environment-variable name and must resolve via + the lookup function used by tests. +- Redaction must replace all API key values with `[REDACTED]` while preserving + non-secret fields. +- Applying a file config must merge into defaults, not replace unspecified + sections with zero values. +- Environment overrides apply after file config. +- Object-form `options` must preserve YAML scalar/list/map values as + `map[string]any` suitable for `pipeline.ModuleBinding.Options`. + +### Required Tests + +- default values; +- parse minimal valid config; +- reject unknown YAML fields; +- reject missing/unsupported version; +- string and object module-binding forms; +- validator binding list with mixed shorthand/object forms; +- duration parsing; +- API key env resolution and invalid env-name rejection; +- file config merges with defaults; +- environment overrides operational and LLM values; +- environment variables cannot change pipeline module wiring; +- redacted config removes API key values. + +### Validation + +Run: + +```sh +gofmt -w internal/core/config +go test ./internal/core/config +go test ./... +``` + +## Stage 6: Config Validation And Pipeline Resolution + +### Goal + +Validate named pipeline profiles against registered module metadata, LLM +profiles, lane selections, and resolved pipeline digest behavior. + +### Files To Add + +- `internal/core/config/validation.go` +- `internal/core/config/effective_config.go` +- `internal/core/config/validation_test.go` +- `internal/core/config/effective_config_test.go` + +### Required API + +Add: + +```go +type ResolveInput struct { + PipelineID string + Only []string + Catalog pipeline.ModuleCatalog +} + +type EffectiveConfig struct { + Config Config + PipelineID string + Only []string + ResolvedPipeline pipeline.ResolvedPipeline +} + +func (c Config) Validate() error +func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) +func (c Config) LLMProfile(id string) (LLMProfile, bool) +func (c Config) OpenAICompatibleClientConfig(profileID string) (llm.OpenAICompatibleClientConfig, error) +``` + +### Required Behavior + +`Validate()` must check config-internal rules that do not need a module catalog: + +- LLM profile IDs trim to non-empty unique IDs; +- pipeline IDs trim to non-empty unique IDs; +- LLM profile provider must be empty or `openai-compatible`; +- LLM profile numeric values must be positive where applicable; +- diagnostics work dir must not be empty; +- diagnostics retention must be one of `auto`, `always`, or `never`; +- `TotalLLM` must be greater than zero; +- every LLM profile referenced by a pipeline binding must exist. + +`Resolve(input)` must: + +- call `Validate()`; +- trim `PipelineID`; +- reject empty or unknown pipeline IDs; +- pass `Only` through to `pipeline.ResolvePipeline`; +- return the resolved pipeline and digest; +- surface errors that name the pipeline and lane where possible. + +`OpenAICompatibleClientConfig(profileID)` must: + +- reject unknown profile IDs; +- reject unsupported providers; +- require base URL and model; +- pass API key, timeout, and max retries to the LLM client config. + +### Required Tests + +- `Validate` success for valid config; +- unknown LLM profile referenced by binding; +- invalid provider; +- invalid numeric fields; +- invalid diagnostics retention; +- empty/unknown selected pipeline ID; +- lane filtering success and failure; +- unknown module key through module catalog; +- missing capability through module catalog; +- resolved pipeline digest changes when effective config changes; +- `OpenAICompatibleClientConfig` rejects incomplete default profile and succeeds + when base URL/model are set. + +Use fake module registries and fake module specs. Do not add real modules. + +### Validation + +Run: + +```sh +gofmt -w internal/core/config +go test ./internal/core/config +go test ./... +``` + +## Stage 7: CLI Config Validation And Pipeline Listing + +### Goal + +Add minimal CLI support for config discovery/validation and pipeline profile +listing without adding run execution. + +### Files To Update + +- `internal/cli/run.go` +- `internal/cli/run_test.go` + +### Required CLI + +Support: + +```sh +notarius config validate --config path/to/config.yml +notarius config validate --config path/to/config.yml --pipeline pipeline-id +notarius config validate --config path/to/config.yml --pipeline pipeline-id --only lane-a,lane-b +notarius pipelines list --config path/to/config.yml +notarius pipelines list --config path/to/config.yml --json +``` + +Also support the default config search path from architecture policy: + +1. `--config` path, when provided; +2. `NOTARIUS_CONFIG`, when set; +3. `/usr/local/etc/notarius/config.yml`, when present. + +Do not add `notarius run` in this checkpoint. + +### Required Implementation Shape + +The CLI needs module metadata to validate structural pipeline config. Add an +internal test seam rather than hard-coding fake modules: + +```go +type Options struct { + Catalog pipeline.ModuleCatalog + LookupEnv func(string) (string, bool) +} + +func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int +``` + +Keep the existing `Run(args, stdout, stderr)` as a wrapper that passes default +options. Default production options may contain empty registries until real +modules are registered by later checkpoints. + +### Required Behavior + +- `config validate` exits `0` and writes a concise success message on valid + config. +- `config validate` exits `1` on config load/validation errors and writes the + error to stderr. +- `config validate --pipeline` resolves that pipeline against the supplied + catalog. +- `--only` is valid only when `--pipeline` is present. +- `pipelines list` exits `0` and lists pipeline IDs sorted alphabetically. +- `pipelines list --json` emits stable JSON, for example: + +```json +{"pipelines":["a","b"]} +``` + +- Unknown commands and invalid flags exit `2`. +- Usage text must include the new commands. +- CLI flags must not allow ad hoc structural module overrides such as + `--extractor`, `--chunker`, `--input`, `--merge`, or `--normalize`. + +### Required Tests + +- existing help/unknown-command tests still pass; +- `config validate` success with fake catalog; +- `config validate` reports config parse errors; +- `config validate --pipeline --only` success and invalid lane failure; +- `--only` without `--pipeline` fails; +- `pipelines list` sorted text output; +- `pipelines list --json` stable JSON output; +- `NOTARIUS_CONFIG` is used when `--config` is absent; +- missing config path produces actionable error; +- no ad hoc structural flags are accepted. + +### Validation + +Run: + +```sh +gofmt -w internal/cli +go test ./internal/cli +go test ./... +go build ./cmd/notarius +rm -f ./notarius +``` + +## Stage 8: Final Checkpoint 4 Review Pass + +### Goal + +Remove accidental Audita coupling, confirm scope boundaries, and verify all +runtime pieces compose. + +### Required Review + +Check: + +- no correction proposal, replacement policy, transcript mutation, correction + ledger, or Audita module behavior was copied; +- no package names or artifact names mention corrections, proposals, + replacements, or transcripts unless they refer to Audita source files in + comments inside tests; +- no real D&D prompts or schemas were added; +- no Seriatim input adapter was added; +- no `notarius run` behavior was added; +- provider-specific HTTP structs stay unexported inside `internal/framework/llm`; +- response schemas live inside `internal/framework/llm`, not a new + `responseschema` package; +- prompt assets live inside `internal/framework/prompt`; +- config is centered on named pipeline profiles; +- module selection comes from config, not ad hoc CLI flags; +- diagnostics artifact names are extraction-oriented; +- secrets are redacted from effective config and surfaced provider errors. + +### Required Validation + +Run: + +```sh +gofmt -w internal +go test ./... +go vet ./... +go build ./cmd/notarius +rm -f ./notarius +git status --short +``` + +The implementation response should summarize: + +- files/packages added; +- tests run; +- any deviations from this plan and why. + +## Open Questions + +None. The plan intentionally defers real extractor prompts/schemas, concrete +input modules, real built-in module catalogs, and `notarius run` execution to +later checkpoints.