27 KiB
Implementation Plan: Checkpoint 4 Portable Audita Infrastructure
Status
This is a staged implementation plan for
4-portable-audita-infrastructure.md.
It is intended for an LLM coding agent to follow stage by stage.
This plan implements only checkpoint 4. Do not add real Seriatim parsing, real
D&D extraction, real domain prompts or schemas, production extraction modules,
durable final artifact writing, or notarius run execution behavior in this
checkpoint.
Policy Context
Follow:
Required boundaries:
- copy only reusable Audita infrastructure, not Audita's transcript-correction model;
- keep framework code source-agnostic and domain-agnostic;
- keep provider-specific HTTP request/response details inside
internal/framework/llm; - keep prompt construction and prompt assets out of provider adapters;
- keep structural pipeline wiring in named pipeline config, not ad hoc CLI flags;
- preserve the fixed pipeline shape:
input -> chunk -> extract -> merge -> normalize -> output; - keep capabilities as flat strings;
- redact configured secrets from diagnostics, effective config, and surfaced provider errors.
Audita files useful for adaptation:
../audita/internal/framework/llm/*../audita/internal/framework/responseschema/registry.go../audita/internal/prompts/registry.go../audita/internal/prompts/render.go../audita/internal/core/diagnostics/*../audita/internal/core/config/*../audita/internal/cli/*../audita/examples/*.yml
Global Implementation Decisions
- Add
gopkg.in/yaml.v3as the only new third-party dependency in this checkpoint. Audita already uses it, and YAML is the architecture-policy preference for Notarius config. - Put LLM client, scheduler, structured-output decoding helpers, secret
redaction, and response-schema registry code in
internal/framework/llm. Do not create separateresponseschema,structuredoutput, orwarningsframework packages. - Put prompt registry and rendering code in
internal/framework/prompt. - Put config structs, defaults, file/env/flag precedence helpers, redaction, and
validation in
internal/core/config. - Put diagnostics run-directory code in
internal/core/diagnostics. This package is justified by real durable diagnostic state in checkpoint 4; keep it extraction-oriented and do not copy correction-ledger concepts. - Keep CLI support limited to discovery/validation commands:
notarius config validateandnotarius pipelines list. - Do not add embedded built-in pipeline profiles. Config files are the only source of user-defined pipeline profiles in this checkpoint.
- Do not add real D&D prompt/schema assets. Use inert placeholder assets only for registry tests.
- Use JSON for diagnostics artifacts even when the input config is YAML.
- Use
sha256:<hex>for content digests written to diagnostics or registries, unless an existing Notarius type already expects bare hex. Prefer the prefixed form for new Notarius code. - Run
gofmton touched Go files after every stage.
Stage 1: LLM Runtime
Goal
Adapt Audita's OpenAI-compatible structured-output client, secret redaction, and
scheduler to Notarius's existing contracts.StructuredLLMClient.
Files To Add
internal/framework/llm/client_common.gointernal/framework/llm/openai_compatible_client.gointernal/framework/llm/openai_compatible_client_test.gointernal/framework/llm/scheduler.gointernal/framework/llm/scheduler_test.gointernal/framework/llm/secrets.gointernal/framework/llm/secrets_test.go
Required API
Add:
type OpenAICompatibleClientConfig struct {
BaseURL string
Model string
APIKey string
MaxRetries int
HTTPClient *http.Client
RequestTimeout time.Duration
}
type OpenAICompatibleClient struct { /* unexported fields */ }
func NewOpenAICompatibleClient(cfg OpenAICompatibleClientConfig) (*OpenAICompatibleClient, error)
func (c *OpenAICompatibleClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error)
Add:
type Scheduler struct { /* unexported fields */ }
func NewScheduler(maxConcurrency int) (*Scheduler, error)
func (s *Scheduler) Acquire(ctx context.Context) (func(), error)
func (s *Scheduler) Run(ctx context.Context, fn func(context.Context) error) error
Add secret helpers:
func RedactSecrets(message string, secrets []string) string
func ErrorWithSecretsRedacted(err error, secrets []string) error
Notarius Adaptations
Audita's client expects a typed response schema object. Notarius currently uses:
type StructuredCompletionRequest struct {
ResponseSchemaName string
ResponseSchema json.RawMessage
}
Therefore the Notarius client must:
- require
ResponseSchemaNameto be non-empty; - require
ResponseSchemato be non-empty valid JSON; - send OpenAI-compatible
response_format.type=json_schema; - send
json_schema.namefromResponseSchemaName; - send
json_schema.schemafromResponseSchema; - decode the assistant message content into
out; - return raw assistant content in
StructuredCompletionResponse.Content; - populate provider/model/token metadata when available;
- use
req.Modelwhen present, otherwise the configured default model.
Provider request/response structs must be unexported.
Required Behavior
- Validate
outis a non-nil pointer. - Reject empty base URL, empty default model, negative retries, empty model at call time, missing schema name, and invalid schema JSON.
- Trim message roles/content and reject empty roles or content.
- Retry transport errors, malformed provider envelopes, malformed assistant JSON,
HTTP 429, and HTTP 5xx up to
MaxRetries. - Do not retry non-retryable 4xx provider errors.
- Respect
context.Contextcancellation. - Redact API keys and bearer tokens from returned errors.
- Scheduler must bound concurrent calls, release permits exactly once, handle cancellation while queued, and avoid leaking permits when cancellation races with grant.
Required Tests
Port and adapt Audita LLM tests using local httptest.Server fakes:
- successful structured completion;
- request body includes model, messages, schema name, and schema JSON;
- default model fallback and request model override;
- invalid output target;
- missing/invalid schema;
- provider non-2xx behavior;
- retry behavior for 429/5xx and malformed retryable responses;
- no retry for non-retryable 4xx;
- provider error redacts configured API key;
- scheduler max concurrency;
- scheduler cancellation while queued;
- scheduler release function is idempotent.
Validation
Run:
gofmt -w internal/framework/llm
go test ./internal/framework/llm
go test ./...
Stage 2: Embedded Response Schema Registry
Goal
Adapt Audita's embedded response-schema registry pattern into
internal/framework/llm without adding real extractor schemas.
Files To Add
internal/framework/llm/schema_registry.gointernal/framework/llm/schema_registry_test.gointernal/framework/llm/assets/schemas/test_artifact.v1.jsoninternal/framework/llm/assets/schemas/test_validator_decision.v1.json
Required API
Add:
type ResponseSchemaKey string
const (
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
)
type ResponseSchema struct {
Key ResponseSchemaKey `json:"key"`
ID string `json:"id"`
Version string `json:"version"`
Name string `json:"name"`
JSONSchema json.RawMessage `json:"json_schema"`
SHA256 string `json:"sha256"`
}
func RegisteredResponseSchemas() []ResponseSchema
func LookupResponseSchema(key ResponseSchemaKey) (ResponseSchema, bool)
func MustLookupResponseSchema(key ResponseSchemaKey) ResponseSchema
func (s ResponseSchema) DiagnosticsMap() map[string]any
Required Behavior
- Embed schema assets using
embed.FS. - Validate assets are valid JSON at package initialization.
- Compute
SHA256from exact asset bytes and format assha256:<hex>. - Return defensive copies of
json.RawMessage. - Return registered schemas sorted by key.
- Use Notarius-neutral IDs and names, for example
notarius.test_artifactandnotarius_test_artifact_v1. - Keep test schemas inert and clearly non-domain-specific.
Required Tests
- lookup succeeds for both test schemas;
- unknown lookup returns false;
MustLookupResponseSchemapanics for unknown keys;- registered schemas are sorted;
- JSON schema content is valid JSON;
- returned schema JSON is mutation-safe;
- diagnostics map omits raw schema content and includes ID, version, name, and hash.
Validation
Run:
gofmt -w internal/framework/llm
go test ./internal/framework/llm
go test ./...
Stage 3: Prompt Registry
Goal
Adapt Audita's embedded prompt registry and rendering pattern into
internal/framework/prompt using only generic test prompts.
Files To Add
internal/framework/prompt/registry.gointernal/framework/prompt/render.gointernal/framework/prompt/registry_test.gointernal/framework/prompt/render_test.gointernal/framework/prompt/assets/shared/prompt_hardening.mdinternal/framework/prompt/assets/test/generic/system.mdinternal/framework/prompt/assets/test/generic/user.md
Required API
Add:
const (
SourceBuiltin = "builtin"
VersionV1 = "v1"
TestGenericPromptID = "test.generic"
)
type Metadata struct {
PromptID string `json:"prompt_id"`
PromptVersion string `json:"prompt_version"`
PromptSource string `json:"prompt_source"`
EmbeddedPath string `json:"embedded_path"`
SHA256 string `json:"sha256"`
}
func LookupMetadata(promptID string) (Metadata, bool)
func MustLookupMetadata(promptID string) Metadata
func RegisteredMetadata() []Metadata
func HardeningText() string
func RenderUserSystem(promptID string, data any) (system string, user string, metadata Metadata, err error)
func (m Metadata) DiagnosticsMap() map[string]any
Required Behavior
- Compile embedded system/user prompts with
text/template. - Use
Option("missingkey=error"). - Provide a
{{ hardening }}template function backed by the shared hardening asset. - Trim rendered system/user text.
- Compute
SHA256from combined system and user source text and format assha256:<hex>. - Return prompt metadata sorted by prompt ID.
- Keep assets generic and non-domain-specific.
Required Tests
- metadata lookup succeeds for
test.generic; - unknown lookup returns false;
MustLookupMetadatapanics for unknown prompt ID;- registered metadata is sorted;
- render returns system/user text and metadata;
- missing template data returns an error;
- hardening text is available and appears when referenced;
- diagnostics map includes metadata but no rendered prompt text.
Validation
Run:
gofmt -w internal/framework/prompt
go test ./internal/framework/prompt
go test ./...
Stage 4: Diagnostics Run Directory
Goal
Adapt Audita's diagnostics run-directory pattern using extraction-oriented artifact names and Notarius types.
Files To Add
internal/core/diagnostics/artifacts.gointernal/core/diagnostics/run_dir.gointernal/core/diagnostics/run_dir_test.gointernal/core/diagnostics/artifacts_test.go
Required API
Add artifact names:
const (
ArtifactInvocationMetadata = "invocation.json"
ArtifactEffectiveConfig = "effective-config.json"
ArtifactResolvedPipeline = "resolved-pipeline.json"
ArtifactSourceDocument = "source-document.json"
ArtifactRunManifest = "run-manifest.json"
ArtifactRunReport = "run-report.json"
ArtifactWarnings = "warnings.json"
ArtifactErrorLog = "error.log"
)
Add:
type RunDirectory struct { /* unexported fields */ }
type RetentionMode string
const (
RetentionAuto RetentionMode = "auto"
RetentionAlways RetentionMode = "always"
RetentionNever RetentionMode = "never"
)
type RetentionDecisionInput struct {
RetentionMode RetentionMode
RunSucceeded bool
HasWarnings bool
}
func ShouldRetainRunDirectory(input RetentionDecisionInput) bool
func NewRunDirectory(workDir string, retention RetentionMode) (*RunDirectory, error)
func (r *RunDirectory) Path() string
func (r *RunDirectory) RunID() string
func (r *RunDirectory) WriteInvocationMetadata(metadata InvocationMetadata) error
func (r *RunDirectory) WriteRedactedEffectiveConfig(payload any) error
func (r *RunDirectory) WriteResolvedPipeline(payload any) error
func (r *RunDirectory) WriteSourceDocument(payload any) error
func (r *RunDirectory) WriteRunManifest(manifest artifacts.RunManifest) error
func (r *RunDirectory) WriteRunReport(payload any) error
func (r *RunDirectory) WriteWarnings(warnings []contracts.Warning) error
func (r *RunDirectory) WriteErrorLog(errorMessage string) error
func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error
func (r *RunDirectory) ApplyRetention(input RetentionDecisionInput) error
Add:
type InvocationMetadata struct {
Operation string `json:"operation"`
PipelineID string `json:"pipeline_id,omitempty"`
PipelineDigest string `json:"pipeline_digest,omitempty"`
InputPath string `json:"input_path,omitempty"`
ConfigPath string `json:"config_path,omitempty"`
ConfigSource string `json:"config_source,omitempty"`
OnlyLanes []string `json:"only_lanes,omitempty"`
RunID string `json:"run_id"`
StartedAt time.Time `json:"started_at"`
}
Required Behavior
- Default work directory is
/tmp/notarius. - Create work directory with
0o755. - Create unique run directories named
run-<unix-nano>. - Write JSON artifacts with
json.MarshalIndentand a trailing newline. - Write error logs as plain text with a trailing newline.
- Reject artifact names that are empty, absolute paths, contain path separators, or resolve outside the run directory.
- Retention behavior:
- failed runs are always retained;
alwaysretains successful runs;neverremoves successful runs;autoretains successful runs only when warnings exist.
- Do not copy Audita names such as source transcript, normalized transcript, correction ledger, proposal, or replacement.
Required Tests
- run directory creation and run ID shape;
- default work directory behavior using a temporary working directory where possible;
- JSON artifact writes are indented and newline-terminated;
- invocation metadata fills missing run ID and start time;
- error log write;
- artifact path rejection for unsafe names;
- retention decisions for failed/successful runs across modes;
ApplyRetentionremoves only the run directory for removable cases.
Validation
Run:
gofmt -w internal/core/diagnostics
go test ./internal/core/diagnostics
go test ./...
Stage 5: Config Model, YAML Loading, Redaction, And Environment Overrides
Goal
Add Notarius config structs and loading primitives for named pipeline profiles, LLM profiles, operational settings, YAML files, redaction, and environment overrides.
Files To Add
internal/core/config/config.gointernal/core/config/file_config.gointernal/core/config/env.gointernal/core/config/redaction.gointernal/core/config/config_test.gointernal/core/config/file_config_test.gointernal/core/config/env_test.gointernal/core/config/redaction_test.go
Dependency Change
Add:
go get gopkg.in/yaml.v3@v3.0.1
Commit the resulting go.mod and go.sum changes.
Required API
Add core config types:
const SupportedFileConfigVersion = 1
type Config struct {
LLMProfiles map[string]LLMProfile
Pipelines map[string]pipeline.PipelineProfile
Concurrency ConcurrencyConfig
Diagnostics DiagnosticsConfig
}
type LLMProfile struct {
Provider string
BaseURL string
Model string
APIKey string
APIKeyEnv string
TimeoutSeconds int
MaxRetries int
MaxConcurrency int
}
type ConcurrencyConfig struct {
TotalLLM int
}
type DiagnosticsConfig struct {
WorkDir string
Retention diagnostics.RetentionMode
}
func Default() Config
func (c Config) Redacted() Config
func (c *Config) ApplyEnvOverrides() error
func LoadFromEnv() (Config, error)
Add file config loading:
type FileConfig struct { /* YAML-facing fields */ }
func LoadFileConfig(path string) (FileConfig, error)
func ParseFileConfigYAML(data []byte) (FileConfig, error)
func (c *Config) ApplyFileConfig(fileCfg FileConfig) error
Add unexported lookup seams for tests:
func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(string) (string, bool)) error
func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool)) error
YAML Shape
Support this shape:
version: 1
llm_profiles:
default:
provider: openai-compatible
base_url: https://example.invalid/v1
model: test-model
api_key_env: NOTARIUS_TEST_API_KEY
timeout: 600s
max_retries: 3
max_concurrency: 1
pipelines:
example:
input: fake/input
chunk: generic
artifacts:
events:
extract:
module: fake/extract
llm_profile: default
options:
temperature: 0
merge: appendorder
normalize: noop
validators:
- fake/validator
output: json
concurrency:
total_llm: 1
diagnostics:
work_dir: /tmp/notarius
retention: auto
Module bindings must support both string shorthand and object form:
extract: fake/extract
extract:
module: fake/extract
llm_profile: fast
options:
key: value
Validator bindings must support a sequence of string or object bindings.
Defaults
Default() must provide:
LLMProfiles["default"]with provideropenai-compatible,TimeoutSeconds: 600,MaxRetries: 3, andMaxConcurrency: 1;- no built-in pipeline profiles;
Concurrency.TotalLLM: 1;Diagnostics.WorkDir: "/tmp/notarius";Diagnostics.Retention: diagnostics.RetentionAuto.
Config validation will later decide whether a selected LLM profile is complete
enough to build a real client. This stage should not require default BaseURL
or Model values.
Environment Variables
Support these environment overrides:
NOTARIUS_LLM_DEFAULT_API_KEYNOTARIUS_LLM_DEFAULT_BASE_URLNOTARIUS_LLM_DEFAULT_MODELNOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDSNOTARIUS_LLM_DEFAULT_MAX_RETRIESNOTARIUS_LLM_DEFAULT_MAX_CONCURRENCYNOTARIUS_TOTAL_LLM_CONCURRENCYNOTARIUS_WORK_DIRNOTARIUS_DIAGNOSTICS_RETENTION
Do not support environment variables that structurally change pipeline module selection.
Required Behavior
- YAML parsing must use
KnownFields(true). - Config version is required and must equal
SupportedFileConfigVersion. - Duration fields accept integer seconds or Go duration strings resolving to whole seconds.
api_key_envmust be a valid environment-variable name and must resolve via the lookup function used by tests.- Redaction must replace all API key values with
[REDACTED]while preserving non-secret fields. - Applying a file config must merge into defaults, not replace unspecified sections with zero values.
- Environment overrides apply after file config.
- Object-form
optionsmust preserve YAML scalar/list/map values asmap[string]anysuitable forpipeline.ModuleBinding.Options.
Required Tests
- default values;
- parse minimal valid config;
- reject unknown YAML fields;
- reject missing/unsupported version;
- string and object module-binding forms;
- validator binding list with mixed shorthand/object forms;
- duration parsing;
- API key env resolution and invalid env-name rejection;
- file config merges with defaults;
- environment overrides operational and LLM values;
- environment variables cannot change pipeline module wiring;
- redacted config removes API key values.
Validation
Run:
gofmt -w internal/core/config
go test ./internal/core/config
go test ./...
Stage 6: Config Validation And Pipeline Resolution
Goal
Validate named pipeline profiles against registered module metadata, LLM profiles, lane selections, and resolved pipeline digest behavior.
Files To Add
internal/core/config/validation.gointernal/core/config/effective_config.gointernal/core/config/validation_test.gointernal/core/config/effective_config_test.go
Required API
Add:
type ResolveInput struct {
PipelineID string
Only []string
Catalog pipeline.ModuleCatalog
}
type EffectiveConfig struct {
Config Config
PipelineID string
Only []string
ResolvedPipeline pipeline.ResolvedPipeline
}
func (c Config) Validate() error
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error)
func (c Config) LLMProfile(id string) (LLMProfile, bool)
func (c Config) OpenAICompatibleClientConfig(profileID string) (llm.OpenAICompatibleClientConfig, error)
Required Behavior
Validate() must check config-internal rules that do not need a module catalog:
- LLM profile IDs trim to non-empty unique IDs;
- pipeline IDs trim to non-empty unique IDs;
- LLM profile provider must be empty or
openai-compatible; - LLM profile numeric values must be positive where applicable;
- diagnostics work dir must not be empty;
- diagnostics retention must be one of
auto,always, ornever; TotalLLMmust be greater than zero;- every LLM profile referenced by a pipeline binding must exist.
Resolve(input) must:
- call
Validate(); - trim
PipelineID; - reject empty or unknown pipeline IDs;
- pass
Onlythrough topipeline.ResolvePipeline; - return the resolved pipeline and digest;
- surface errors that name the pipeline and lane where possible.
OpenAICompatibleClientConfig(profileID) must:
- reject unknown profile IDs;
- reject unsupported providers;
- require base URL and model;
- pass API key, timeout, and max retries to the LLM client config.
Required Tests
Validatesuccess for valid config;- unknown LLM profile referenced by binding;
- invalid provider;
- invalid numeric fields;
- invalid diagnostics retention;
- empty/unknown selected pipeline ID;
- lane filtering success and failure;
- unknown module key through module catalog;
- missing capability through module catalog;
- resolved pipeline digest changes when effective config changes;
OpenAICompatibleClientConfigrejects incomplete default profile and succeeds when base URL/model are set.
Use fake module registries and fake module specs. Do not add real modules.
Validation
Run:
gofmt -w internal/core/config
go test ./internal/core/config
go test ./...
Stage 7: CLI Config Validation And Pipeline Listing
Goal
Add minimal CLI support for config discovery/validation and pipeline profile listing without adding run execution.
Files To Update
internal/cli/run.gointernal/cli/run_test.go
Required CLI
Support:
notarius config validate --config path/to/config.yml
notarius config validate --config path/to/config.yml --pipeline pipeline-id
notarius config validate --config path/to/config.yml --pipeline pipeline-id --only lane-a,lane-b
notarius pipelines list --config path/to/config.yml
notarius pipelines list --config path/to/config.yml --json
Also support the default config search path from architecture policy:
--configpath, when provided;NOTARIUS_CONFIG, when set;/usr/local/etc/notarius/config.yml, when present.
Do not add notarius run in this checkpoint.
Required Implementation Shape
The CLI needs module metadata to validate structural pipeline config. Add an internal test seam rather than hard-coding fake modules:
type Options struct {
Catalog pipeline.ModuleCatalog
LookupEnv func(string) (string, bool)
}
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int
Keep the existing Run(args, stdout, stderr) as a wrapper that passes default
options. Default production options may contain empty registries until real
modules are registered by later checkpoints.
Required Behavior
config validateexits0and writes a concise success message on valid config.config validateexits1on config load/validation errors and writes the error to stderr.config validate --pipelineresolves that pipeline against the supplied catalog.--onlyis valid only when--pipelineis present.pipelines listexits0and lists pipeline IDs sorted alphabetically.pipelines list --jsonemits stable JSON, for example:
{"pipelines":["a","b"]}
- Unknown commands and invalid flags exit
2. - Usage text must include the new commands.
- CLI flags must not allow ad hoc structural module overrides such as
--extractor,--chunker,--input,--merge, or--normalize.
Required Tests
- existing help/unknown-command tests still pass;
config validatesuccess with fake catalog;config validatereports config parse errors;config validate --pipeline --onlysuccess and invalid lane failure;--onlywithout--pipelinefails;pipelines listsorted text output;pipelines list --jsonstable JSON output;NOTARIUS_CONFIGis used when--configis absent;- missing config path produces actionable error;
- no ad hoc structural flags are accepted.
Validation
Run:
gofmt -w internal/cli
go test ./internal/cli
go test ./...
go build ./cmd/notarius
rm -f ./notarius
Stage 8: Final Checkpoint 4 Review Pass
Goal
Remove accidental Audita coupling, confirm scope boundaries, and verify all runtime pieces compose.
Required Review
Check:
- no correction proposal, replacement policy, transcript mutation, correction ledger, or Audita module behavior was copied;
- no package names or artifact names mention corrections, proposals, replacements, or transcripts unless they refer to Audita source files in comments inside tests;
- no real D&D prompts or schemas were added;
- no Seriatim input adapter was added;
- no
notarius runbehavior was added; - provider-specific HTTP structs stay unexported inside
internal/framework/llm; - response schemas live inside
internal/framework/llm, not a newresponseschemapackage; - prompt assets live inside
internal/framework/prompt; - config is centered on named pipeline profiles;
- module selection comes from config, not ad hoc CLI flags;
- diagnostics artifact names are extraction-oriented;
- secrets are redacted from effective config and surfaced provider errors.
Required Validation
Run:
gofmt -w internal
go test ./...
go vet ./...
go build ./cmd/notarius
rm -f ./notarius
git status --short
The implementation response should summarize:
- files/packages added;
- tests run;
- any deviations from this plan and why.
Open Questions
None. The plan intentionally defers real extractor prompts/schemas, concrete
input modules, real built-in module catalogs, and notarius run execution to
later checkpoints.