Compare commits

...

8 Commits

52 changed files with 4376 additions and 667 deletions

View File

@@ -2,12 +2,12 @@
## Status
This is the active roadmap for reaching the first functional Notarius MVP.
The first functional Notarius MVP implementation is complete enough to start the
deferred documentation pass.
The previous numbered checkpoint roadmaps have been implemented and retired.
This document captures the remaining work needed to turn the implemented
architecture into a usable MVP, with the current architectural review findings
folded in as first-class work.
This document now records the implemented MVP scope and the remaining
release/documentation work before alpha `0.1.0`.
Implementation staging belongs in [`implementation.md`](implementation.md).
@@ -36,163 +36,71 @@ The MVP remains intentionally narrow:
- config-driven pipeline profiles;
- OpenAI-compatible structured LLM execution through the existing LLM client.
## MVP Work Areas
## Implemented MVP Behavior
### Framework/Domain Asset Boundaries
The MVP now includes:
This is the highest-priority remaining architecture correction.
- framework/domain asset boundaries: D&D spell prompt and response schema assets
are owned by `internal/modules/extract/dnd/spells`, while framework prompt and
LLM packages provide only generic primitives;
- production CLI wiring that registers `seriatim`, `dnd/spells`, and the
default `generic`, `appendorder`, `noop`, and `json` modules;
- a config-driven `notarius run` command that reads Seriatim input, resolves a
configured pipeline, invokes the runner, and writes durable output plus
diagnostics;
- fixture-driven CLI acceptance coverage using maintained example config and
transcript fixtures with a fake LLM path, so `go test ./...` exercises the MVP
without network access.
Framework packages must remain source-agnostic and domain-agnostic. D&D spell
prompt assets, response schema assets, prompt IDs, response schema keys, and
domain-specific prompt/schema tests should not live in `internal/framework/llm`
or `internal/framework/prompt`.
Target state:
- `internal/framework/llm` provides generic structured-output client,
scheduler, schema metadata, schema loading, and schema lookup/registration
primitives.
- `internal/framework/prompt` provides generic prompt metadata, prompt loading,
rendering, hardening, and lookup/registration primitives.
- `internal/modules/extract/dnd/spells` owns the D&D spell prompt assets,
response schema assets, stable prompt ID, stable response schema key, and
module-specific prompt/schema tests.
- Framework tests use placeholder/test assets only.
- The D&D spells extractor depends on generic framework APIs, not
framework-owned D&D constants.
This work should not change the external artifact shape or module key. It is an
ownership and package-boundary correction.
### Production Application Catalog Wiring
The implemented modules and registries are currently exercised mostly through
tests that inject catalogs. The MVP needs a production assembly point that
builds the catalog and stage registries used by real CLI commands.
Target state:
- a small app-level package or CLI wiring function constructs the production
`pipeline.ModuleCatalog`;
- production wiring registers `seriatim`;
- production wiring registers `dnd/spells`;
- production wiring registers the default `generic`, `appendorder`, `noop`, and
`json` modules;
- `notarius config validate --pipeline ...` validates real configured
pipelines without test-only catalog injection;
- `notarius pipelines list` reports production-registered modules where useful
for discoverability.
The production wiring should not move domain behavior into the CLI. The CLI may
compose modules, but module packages should continue to own their own behavior
and metadata.
### Default Production Stage Modules
Pipeline defaults are already part of the architecture:
The production pipeline defaults are:
- `chunk: generic`;
- `merge: appendorder`;
- `normalize: noop`;
- `output: json`.
The MVP should make those defaults real production modules rather than
test-only conveniences.
Target state:
- `generic` chunking creates ordered chunks over generic source units and is
configurable enough for transcript MVP use;
- `appendorder` merge serializes artifact candidates in deterministic source
and chunk order;
- `noop` normalize passes merged artifacts through unchanged while preserving
diagnostics;
- `json` output encodes approved artifacts, rejected artifacts, warnings,
manifest data, and relevant run metadata in a durable JSON shape;
- each default module declares module specs and capabilities compatible with
pipeline validation;
- default modules are registered by production app wiring.
If a default module remains implemented in `internal/framework/pipeline`, its
production registration still needs to be explicit and discoverable. If its
logic grows beyond a small generic helper, move it under `internal/modules`.
### `notarius run`
The MVP needs a functional run command that drives the already-implemented
pipeline runner.
Target state:
- command shape:
The implemented command shape is:
```sh
notarius run <pipeline-id> --input path/to/source.json
notarius run <pipeline-id> --input path/to/source.json --only spells
```
- required flags and arguments produce clear usage errors;
- `--config` selects the config file;
- `--only` filters artifact lanes without changing structural pipeline config;
- operational overrides may cover output directory, work directory,
concurrency, and LLM profile/model settings where already supported by config;
- structural stage selection remains config-driven;
- the command parses input through the configured input adapter;
- the command constructs the configured LLM client and scheduler;
- the command invokes the pipeline runner;
- the command writes durable output and diagnostics;
- failures return stable non-zero exit codes and useful error messages.
Current output and diagnostics behavior:
The command should be covered by fixture-driven CLI tests with fake LLM behavior
where network calls would otherwise be required.
- output is written under `<output-root>/<run-id>/`, defaulting to
`./notarius-output`;
- JSON output includes a manifest, grouped approved artifacts, rejected
artifacts, and warnings;
- artifact records include generic source references;
- output-stage warnings remain out-of-band from artifact payloads and are
captured for CLI reporting and diagnostics;
- run manifest data includes source digest, resolved pipeline digest, LLM
profile/model metadata, prompt/schema identifiers, and validation status;
- diagnostics redact secrets and include invocation metadata, redacted effective
config, resolved pipeline, manifest, warnings, run report, and error logs.
### MVP Output And Diagnostics Behavior
Maintained MVP fixtures:
The MVP should produce inspectable files that are stable enough for downstream
experiments, without pretending to be a final public artifact contract.
- `examples/dnd-spells.config.yml`;
- `examples/seriatim-minimal-transcript.json`;
- `internal/cli/testdata/invalid-seriatim-empty-segments.json`.
Target state:
- output path behavior is deterministic and documented in code/tests;
- JSON output includes approved artifacts grouped or ordered predictably;
- each artifact includes its generic source references;
- rejected artifacts and validation decisions remain inspectable;
- output-stage warnings remain out-of-band from the durable artifact payload but
are captured for CLI reporting and diagnostics;
- run manifest data includes source digest, resolved pipeline digest, relevant
model/profile information, prompt/schema identifiers, and validation status;
- diagnostics redact secrets and include the resolved effective configuration
needed to debug a run.
### MVP Fixtures And Acceptance Tests
The MVP should be continuously testable without external services.
Target state:
- maintained Seriatim transcript fixture for the D&D spells MVP;
- maintained minimal config fixture for the MVP pipeline;
- fake LLM path for deterministic CLI and runner tests;
- config validation tests using the production catalog;
- `notarius run` fixture test from input file to output JSON;
- failure tests for missing config, unknown pipeline, invalid input,
invalid lane selection, LLM failure, and validation rejection;
- `go test ./...` is sufficient to exercise the MVP path without network
access.
### Documentation Pass Preparation
## Remaining Release And Documentation Work
The full documentation pass is intentionally deferred until MVP functionality
exists. It should happen before tagging alpha `0.1.0`.
The MVP implementation should still leave clear hooks for the documentation
rewrite:
Remaining work before alpha `0.1.0`:
- command behavior should be stable enough to document in `docs/cli.md`;
- config behavior should be stable enough to document in `docs/config.md`;
- output behavior should be stable enough to document in integration docs;
- examples should be generated from or validated against maintained fixtures
where practical.
- move implemented CLI behavior into `docs/cli.md`;
- move implemented config behavior into `docs/config.md`;
- document run output and diagnostics behavior in canonical docs;
- update integration docs for Seriatim input, D&D spell artifacts, and JSON
output where needed;
- update `README.md` with a shortest useful command based on the maintained
examples;
- keep examples validated by tests.
## Out Of Scope For MVP
@@ -234,9 +142,9 @@ rewrite:
## Deferred Documentation Pass
After MVP behavior is implemented and before alpha `0.1.0`, complete a full
documentation pass/rewrite. That pass should move implemented behavior out of
roadmap documents and into canonical docs required by
Before alpha `0.1.0`, complete a full documentation pass/rewrite. That pass
should move implemented behavior out of roadmap documents and into canonical
docs required by
[`../policy/documentation.md`](../policy/documentation.md), including at least:
- `README.md`;

View File

@@ -0,0 +1,16 @@
version: 1
llm_profiles:
default:
provider: openai-compatible
base_url: http://127.0.0.1:1
model: fake-model
pipelines:
dnd-session:
input: seriatim
chunk:
module: generic
options:
max_units: 50
artifacts:
spells:
extract: dnd/spells

View File

@@ -0,0 +1,22 @@
{
"metadata": {
"id": "session-alpha",
"title": "Synthetic D&D spell session"
},
"segments": [
{
"id": "seg-001",
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria raises her holy symbol and casts Cure Wounds."
},
{
"id": "seg-002",
"start": 4,
"end": 8,
"speaker": "DM",
"text": "The bandit mage casts Shield as the blow lands."
}
]
}

172
internal/cli/catalog.go Normal file
View File

@@ -0,0 +1,172 @@
package cli
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
)
func productionRegistries() (pipeline.Registries, error) {
registries := pipeline.Registries{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
Validators: pipeline.NewValidatorRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
if err := seriatim.Register(registries.Inputs); err != nil {
return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
}
if err := generic.Register(registries.Chunkers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
}
if err := spells.Register(registries.Extractors); err != nil {
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
}
if err := appendorder.Register(registries.Mergers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
}
if err := noop.Register(registries.Normalizers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
}
if err := jsonoutput.Register(registries.Outputs); err != nil {
return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
}
return registries, nil
}
func productionCatalog() (pipeline.ModuleCatalog, error) {
registries, err := productionRegistries()
if err != nil {
return pipeline.ModuleCatalog{}, err
}
return catalogFromRegistries(registries), nil
}
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
if !isEmptyCatalog(opts.Catalog) {
return opts.Catalog, nil
}
if !isEmptyRegistries(opts.Registries) {
return catalogFromRegistries(opts.Registries), nil
}
return productionCatalog()
}
func effectiveRegistries(opts Options) (pipeline.Registries, error) {
if !isEmptyRegistries(opts.Registries) {
return opts.Registries, nil
}
if !isEmptyCatalog(opts.Catalog) {
return registriesFromCatalog(opts.Catalog), nil
}
return productionRegistries()
}
func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalog {
return pipeline.ModuleCatalog{
Inputs: registries.Inputs,
Chunkers: registries.Chunkers,
Extractors: registries.Extractors,
Mergers: registries.Mergers,
Normalizers: registries.Normalizers,
Validators: registries.Validators,
Outputs: registries.Outputs,
}
}
func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
return pipeline.Registries{
Inputs: catalog.Inputs,
Chunkers: catalog.Chunkers,
Extractors: catalog.Extractors,
Mergers: catalog.Mergers,
Normalizers: catalog.Normalizers,
Validators: catalog.Validators,
Outputs: catalog.Outputs,
}
}
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
return catalog.Inputs == nil &&
catalog.Chunkers == nil &&
catalog.Extractors == nil &&
catalog.Mergers == nil &&
catalog.Normalizers == nil &&
catalog.Validators == nil &&
catalog.Outputs == nil
}
func isEmptyRegistries(registries pipeline.Registries) bool {
return registries.Inputs == nil &&
registries.Chunkers == nil &&
registries.Extractors == nil &&
registries.Mergers == nil &&
registries.Normalizers == nil &&
registries.Validators == nil &&
registries.Outputs == nil
}
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
trimmedID := strings.TrimSpace(profileID)
if trimmedID == "" {
trimmedID = pipeline.DefaultLLMProfile
}
profile, ok := cfg.LLMProfile(trimmedID)
if !ok {
return nil, nil, fmt.Errorf("LLM profile %q is not configured", trimmedID)
}
clientCfg, err := cfg.OpenAICompatibleClientConfig(trimmedID)
if err != nil {
return nil, nil, err
}
client, err := llm.NewOpenAICompatibleClient(clientCfg)
if err != nil {
return nil, nil, fmt.Errorf("create LLM client for profile %q: %w", trimmedID, err)
}
scheduler, err := llm.NewScheduler(effectiveLLMConcurrency(cfg, profile))
if err != nil {
return nil, nil, fmt.Errorf("create LLM scheduler for profile %q: %w", trimmedID, err)
}
provider := strings.TrimSpace(profile.Provider)
if provider == "" {
provider = "openai-compatible"
}
metadata := []artifacts.LLMProfileManifest{
{
ID: trimmedID,
Provider: provider,
Model: strings.TrimSpace(profile.Model),
},
}
return llm.NewScheduledClient(client, scheduler), metadata, nil
}
func effectiveLLMConcurrency(cfg config.Config, profile config.LLMProfile) int {
if profile.MaxConcurrency > 0 {
return profile.MaxConcurrency
}
if cfg.Concurrency.TotalLLM > 0 {
return cfg.Concurrency.TotalLLM
}
return 1
}

View File

@@ -1,31 +1,45 @@
package cli
import (
"context"
"encoding/json"
"flag"
"fmt"
"io"
"os"
"path"
"path/filepath"
"sort"
"strings"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
const defaultOutputRoot = "./notarius-output"
const usage = `Usage:
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
`
type Options struct {
Catalog pipeline.ModuleCatalog
LookupEnv func(string) (string, bool)
Catalog pipeline.ModuleCatalog
Registries pipeline.Registries
LLMClientFactory LLMClientFactory
LookupEnv func(string) (string, bool)
Now func() time.Time
}
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
// Run executes the command-line interface and returns a process exit code.
func Run(args []string, stdout, stderr io.Writer) int {
return RunWithOptions(args, stdout, stderr, Options{})
@@ -46,6 +60,8 @@ func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
return runConfig(args[1:], stdout, stderr, opts)
case "pipelines":
return runPipelines(args[1:], stdout, stderr, opts)
case "run":
return runPipelineCommand(args[1:], stdout, stderr, opts)
default:
fmt.Fprintf(stderr, "notarius: unknown command %q\n", args[0])
writeUsage(stderr)
@@ -61,9 +77,389 @@ func normalizeOptions(opts Options) Options {
if opts.LookupEnv == nil {
opts.LookupEnv = os.LookupEnv
}
if opts.Now == nil {
opts.Now = time.Now
}
if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactory
}
return opts
}
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
fs := flag.NewFlagSet("run", flag.ContinueOnError)
fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path")
inputPath := fs.String("input", "", "source input file path")
onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
outputDir := fs.String("output-dir", "", "output directory")
diagnosticsDir := fs.String("diagnostics-dir", "", "diagnostics directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override")
if err := fs.Parse(reorderRunArgs(args)); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
if fs.NArg() == 0 {
fmt.Fprintln(stderr, "notarius: run requires a pipeline ID")
return 2
}
if fs.NArg() > 1 {
fmt.Fprintf(stderr, "notarius: unexpected argument %q\n", fs.Arg(1))
return 2
}
pipelineID := strings.TrimSpace(fs.Arg(0))
if pipelineID == "" {
fmt.Fprintln(stderr, "notarius: run requires a pipeline ID")
return 2
}
if strings.TrimSpace(*inputPath) == "" {
fmt.Fprintln(stderr, "notarius: run requires --input")
return 2
}
only, err := parseOnly(*onlyRaw)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
cfg, loadedConfigPath, err := loadConfig(*configPath, opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if dir := strings.TrimSpace(*diagnosticsDir); dir != "" {
cfg.Diagnostics.WorkDir = dir
}
startedAt := opts.Now().UTC()
runDir, err := diagnostics.NewRunDirectory(cfg.Diagnostics.WorkDir, cfg.Diagnostics.Retention)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
invocation := diagnostics.InvocationMetadata{
Operation: "run",
PipelineID: pipelineID,
InputPath: strings.TrimSpace(*inputPath),
ConfigPath: loadedConfigPath,
ConfigSource: configSource(*configPath),
OnlyLanes: append([]string(nil), only...),
RunID: runDir.RunID(),
StartedAt: startedAt,
}
if err := runDir.WriteInvocationMetadata(invocation); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics invocation metadata: %w", err))
}
catalog, err := effectiveCatalog(opts)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
effective, err := cfg.Resolve(config.ResolveInput{
PipelineID: pipelineID,
Only: only,
Catalog: catalog,
LLMProfileOverride: *llmProfile,
})
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
invocation.PipelineDigest = effective.ResolvedPipeline.Digest
if err := runDir.WriteInvocationMetadata(invocation); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics invocation metadata: %w", err))
}
if err := runDir.WriteRedactedEffectiveConfig(effective); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics effective config: %w", err))
}
if err := runDir.WriteResolvedPipeline(effective.ResolvedPipeline); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved pipeline: %w", err))
}
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
if len(profileIDs) != 1 {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("pipeline %q uses %d distinct LLM profiles; current runs require exactly one: %s", pipelineID, len(profileIDs), strings.Join(profileIDs, ", ")))
}
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
}
registries, err := effectiveRegistries(opts)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
ctx := context.Background()
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, profileIDs[0])
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create LLM client for profile %q: %w", profileIDs[0], err))
}
output, err := pipeline.New(registries).Run(ctx, pipeline.RunInput{
Pipeline: effective.ResolvedPipeline,
Path: strings.TrimSpace(*inputPath),
RawInput: rawInput,
LLMClient: llmClient,
RunID: runDir.RunID(),
StartedAt: startedAt,
LLMProfiles: llmProfiles,
Metadata: runMetadata(*outputDir, *diagnosticsDir),
})
if err != nil {
if output.Manifest.PipelineID != "" {
_ = runDir.WriteRunManifest(output.Manifest)
}
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("run pipeline %q: %w", pipelineID, err))
}
runOutputDir := filepath.Join(outputRoot(*outputDir), runDir.RunID())
if err := writeOutputFiles(runOutputDir, output.OutputFiles); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
if err := runDir.WriteRunManifest(output.Manifest); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics run manifest: %w", err))
}
if err := runDir.WriteWarnings(output.Warnings); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics warnings: %w", err))
}
if err := runDir.WriteRunReport(runReport{
RunID: runDir.RunID(),
PipelineID: effective.PipelineID,
OutputPath: runOutputDir,
DiagnosticsPath: runDir.Path(),
ApprovedCount: len(output.Approved),
RejectedCount: len(output.Rejected),
WarningCount: len(output.Warnings),
ValidationStatus: output.Manifest.ValidationStatus,
}); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics run report: %w", err))
}
if err := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RetentionMode: cfg.Diagnostics.Retention,
RunSucceeded: true,
HasWarnings: len(output.Warnings) > 0,
}); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("apply diagnostics retention: %w", err))
}
fmt.Fprintf(stdout, "pipeline %q complete: approved=%d rejected=%d output=%s\n", effective.PipelineID, len(output.Approved), len(output.Rejected), runOutputDir)
if len(output.Warnings) > 0 {
fmt.Fprintf(stderr, "notarius: run completed with %d warning(s)\n", len(output.Warnings))
}
return 0
}
type runReport struct {
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputPath string `json:"output_path"`
DiagnosticsPath string `json:"diagnostics_path,omitempty"`
ApprovedCount int `json:"approved_count"`
RejectedCount int `json:"rejected_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status,omitempty"`
}
func failPipelineCommand(stderr io.Writer, runDir *diagnostics.RunDirectory, retention diagnostics.RetentionMode, err error) int {
fmt.Fprintf(stderr, "notarius: %v\n", err)
if runDir != nil {
if logErr := runDir.WriteErrorLog(err.Error()); logErr != nil {
fmt.Fprintf(stderr, "notarius: write diagnostics error log: %v\n", logErr)
}
if retentionErr := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RetentionMode: retention,
RunSucceeded: false,
}); retentionErr != nil {
fmt.Fprintf(stderr, "notarius: apply diagnostics retention: %v\n", retentionErr)
}
}
return 1
}
func configSource(configPath string) string {
if strings.TrimSpace(configPath) != "" {
return "flag"
}
return "discovered"
}
func outputRoot(outputDir string) string {
if dir := strings.TrimSpace(outputDir); dir != "" {
return dir
}
return defaultOutputRoot
}
func writeOutputFiles(runOutputDir string, files []contracts.OutputFile) error {
type outputTarget struct {
path string
file contracts.OutputFile
}
targets := make([]outputTarget, 0, len(files))
for _, file := range files {
targetPath, err := outputFilePath(runOutputDir, file.Name)
if err != nil {
return err
}
targets = append(targets, outputTarget{path: targetPath, file: file})
}
if err := os.MkdirAll(runOutputDir, 0o755); err != nil {
return fmt.Errorf("create output directory %q: %w", runOutputDir, err)
}
for _, target := range targets {
if err := os.MkdirAll(filepath.Dir(target.path), 0o755); err != nil {
return fmt.Errorf("create output directory %q: %w", filepath.Dir(target.path), err)
}
if err := writeFileAtomic(target.path, target.file.Bytes, 0o644); err != nil {
return fmt.Errorf("write output file %q: %w", target.file.Name, err)
}
}
return nil
}
func outputFilePath(runOutputDir, logicalName string) (string, error) {
name := strings.TrimSpace(logicalName)
if name == "" {
return "", fmt.Errorf("output file name must not be empty")
}
if strings.Contains(name, `\`) {
return "", fmt.Errorf("output file name %q must use slash-separated relative paths", name)
}
if path.IsAbs(name) || filepath.IsAbs(name) {
return "", fmt.Errorf("output file name %q must be relative", name)
}
if strings.Contains(name, "..") {
return "", fmt.Errorf("output file name %q must not contain ..", name)
}
cleaned := path.Clean(name)
if cleaned == "." || cleaned != name {
return "", fmt.Errorf("output file name %q must be clean", name)
}
root, err := filepath.Abs(runOutputDir)
if err != nil {
return "", fmt.Errorf("resolve output directory %q: %w", runOutputDir, err)
}
target, err := filepath.Abs(filepath.Join(root, filepath.FromSlash(cleaned)))
if err != nil {
return "", fmt.Errorf("resolve output file %q: %w", name, err)
}
rel, err := filepath.Rel(root, target)
if err != nil {
return "", fmt.Errorf("resolve output file %q: %w", name, err)
}
if rel == "." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || rel == ".." {
return "", fmt.Errorf("output file name %q resolves outside output directory", name)
}
return target, nil
}
func writeFileAtomic(path string, data []byte, perm os.FileMode) error {
dir := filepath.Dir(path)
temp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".tmp-*")
if err != nil {
return err
}
tempPath := temp.Name()
removeTemp := true
defer func() {
if removeTemp {
_ = os.Remove(tempPath)
}
}()
if _, err := temp.Write(data); err != nil {
_ = temp.Close()
return err
}
if err := temp.Chmod(perm); err != nil {
_ = temp.Close()
return err
}
if err := temp.Close(); err != nil {
return err
}
if err := os.Rename(tempPath, path); err != nil {
return err
}
removeTemp = false
return nil
}
func reorderRunArgs(args []string) []string {
var flags []string
var positionals []string
for i := 0; i < len(args); i++ {
arg := args[i]
if arg == "--" {
positionals = append(positionals, args[i+1:]...)
break
}
if strings.HasPrefix(arg, "-") {
flags = append(flags, arg)
if runFlagTakesValue(arg) && !strings.Contains(arg, "=") && i+1 < len(args) {
i++
flags = append(flags, args[i])
}
continue
}
positionals = append(positionals, arg)
}
return append(flags, positionals...)
}
func runFlagTakesValue(arg string) bool {
switch arg {
case "--config", "--input", "--only", "--output-dir", "--diagnostics-dir", "--llm-profile":
return true
default:
return false
}
}
func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
seen := make(map[string]struct{})
add := func(binding pipeline.ModuleBinding) {
id := strings.TrimSpace(binding.LLMProfile)
if id != "" {
seen[id] = struct{}{}
}
}
add(resolved.Input)
add(resolved.Chunk)
add(resolved.Output)
for _, lane := range resolved.ArtifactLanes {
add(lane.Extract)
add(lane.Merge)
add(lane.Normalize)
for _, validator := range lane.Validators {
add(validator)
}
}
ids := make([]string, 0, len(seen))
for id := range seen {
ids = append(ids, id)
}
sort.Strings(ids)
return ids
}
func runMetadata(outputDir, diagnosticsDir string) map[string]any {
metadata := make(map[string]any)
if dir := strings.TrimSpace(outputDir); dir != "" {
metadata["output_dir"] = dir
}
if dir := strings.TrimSpace(diagnosticsDir); dir != "" {
metadata["diagnostics_dir"] = dir
}
if len(metadata) == 0 {
return nil
}
return metadata
}
func runConfig(args []string, stdout, stderr io.Writer, opts Options) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "notarius: config requires a subcommand")
@@ -111,10 +507,15 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
}
if strings.TrimSpace(*pipelineID) != "" {
catalog, err := effectiveCatalog(opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if _, err := cfg.Resolve(config.ResolveInput{
PipelineID: *pipelineID,
Only: only,
Catalog: opts.Catalog,
Catalog: catalog,
}); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,6 @@
{
"metadata": {
"id": "session-alpha"
},
"segments": []
}

View File

@@ -34,11 +34,18 @@ type RejectedArtifact struct {
}
type ArtifactLaneManifest struct {
ID string `json:"id"`
Extractor string `json:"extractor"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
Validators []string `json:"validators,omitempty"`
ID string `json:"id"`
Extractor string `json:"extractor"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
Validators []string `json:"validators,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type LLMProfileManifest struct {
ID string `json:"id"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
}
type RunManifest struct {
@@ -53,6 +60,7 @@ type RunManifest struct {
Normalizer string `json:"normalizer,omitempty"`
OutputEncoder string `json:"output_encoder,omitempty"`
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
ValidationStatus string `json:"validation_status,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"`

View File

@@ -127,6 +127,9 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
manifest := RunManifest{
PipelineID: "pipeline-1",
PipelineDigest: "sha256:abc123",
LLMProfiles: []LLMProfileManifest{
{ID: "default", Provider: "openai-compatible", Model: "model-a"},
},
ArtifactLanes: []ArtifactLaneManifest{
{
ID: "events",
@@ -134,6 +137,9 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
Merger: "appendorder",
Normalizer: "noop",
Validators: []string{"grounded"},
Metadata: map[string]any{
"extractor": map[string]any{"prompt_id": "test.prompt"},
},
},
},
}
@@ -148,7 +154,20 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
t.Fatalf("json.Unmarshal() error = %v", err)
}
assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes")
assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes", "llm_profiles")
profiles, ok := got["llm_profiles"].([]any)
if !ok {
t.Fatalf("llm_profiles = %#v, want array", got["llm_profiles"])
}
if len(profiles) != 1 {
t.Fatalf("len(llm_profiles) = %d, want 1", len(profiles))
}
profile, ok := profiles[0].(map[string]any)
if !ok {
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
}
assertHasKeys(t, profile, "id", "provider", "model")
lanes, ok := got["artifact_lanes"].([]any)
if !ok {
@@ -161,7 +180,7 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
if !ok {
t.Fatalf("artifact_lanes[0] = %#v, want object", lanes[0])
}
assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "validators")
assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "validators", "metadata")
}
func assertHasKeys(t *testing.T, values map[string]any, keys ...string) {

View File

@@ -10,9 +10,10 @@ import (
)
type ResolveInput struct {
PipelineID string
Only []string
Catalog pipeline.ModuleCatalog
PipelineID string
Only []string
Catalog pipeline.ModuleCatalog
LLMProfileOverride string
}
type EffectiveConfig struct {
@@ -38,6 +39,12 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}
profile = clonePipelineProfile(profile)
profile.ID = pipelineID
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
if !hasLLMProfile(c.LLMProfiles, override) {
return EffectiveConfig{}, fmt.Errorf("LLM profile override %q is not configured", override)
}
applyLLMProfileOverride(&profile, override)
}
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{Only: input.Only}, input.Catalog)
if err != nil {
@@ -52,6 +59,21 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}, nil
}
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
profile.Input.LLMProfile = profileID
profile.Chunk.LLMProfile = profileID
profile.Output.LLMProfile = profileID
for laneID, lane := range profile.Artifacts {
lane.Extract.LLMProfile = profileID
lane.Merge.LLMProfile = profileID
lane.Normalize.LLMProfile = profileID
for i := range lane.Validators {
lane.Validators[i].LLMProfile = profileID
}
profile.Artifacts[laneID] = lane
}
}
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
pipelineID = strings.TrimSpace(pipelineID)
for rawID, profile := range profiles {

View File

@@ -118,6 +118,51 @@ func TestResolveDigestChangesWhenEffectiveConfigChanges(t *testing.T) {
}
}
func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
cfg := validConfig()
cfg.LLMProfiles["runtime"] = LLMProfile{Provider: "openai-compatible"}
base, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
if err != nil {
t.Fatalf("Resolve base: %v", err)
}
effective, err := cfg.Resolve(ResolveInput{
PipelineID: "example",
Catalog: fakeCatalog(t),
LLMProfileOverride: "runtime",
})
if err != nil {
t.Fatalf("Resolve override: %v", err)
}
if base.ResolvedPipeline.Digest == effective.ResolvedPipeline.Digest {
t.Fatalf("expected digest to change after LLM profile override")
}
for _, binding := range resolvedBindings(effective.ResolvedPipeline) {
if binding.LLMProfile != "runtime" {
t.Fatalf("binding profile = %q, want runtime", binding.LLMProfile)
}
}
_, err = cfg.Resolve(ResolveInput{
PipelineID: "example",
Catalog: fakeCatalog(t),
LLMProfileOverride: "missing",
})
if err == nil || !strings.Contains(err.Error(), "LLM profile override") {
t.Fatalf("expected override profile error, got %v", err)
}
}
func resolvedBindings(resolved pipeline.ResolvedPipeline) []pipeline.ModuleBinding {
bindings := []pipeline.ModuleBinding{resolved.Input, resolved.Chunk, resolved.Output}
for _, lane := range resolved.ArtifactLanes {
bindings = append(bindings, lane.Extract, lane.Merge, lane.Normalize)
bindings = append(bindings, lane.Validators...)
}
return bindings
}
func TestOpenAICompatibleClientConfigRejectsIncompleteDefaultProfile(t *testing.T) {
cfg := Default()

View File

@@ -173,7 +173,7 @@ func (r *RunDirectory) WriteErrorLog(errorMessage string) error {
if err != nil {
return err
}
if err := os.WriteFile(path, []byte(errorMessage+"\n"), 0o644); err != nil {
if err := writeFileAtomic(path, []byte(errorMessage+"\n"), 0o644); err != nil {
return fmt.Errorf("write diagnostics artifact %q: %w", ArtifactErrorLog, err)
}
return nil
@@ -193,7 +193,7 @@ func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error {
return fmt.Errorf("marshal diagnostics artifact %q: %w", name, err)
}
data = append(data, '\n')
if err := os.WriteFile(path, data, 0o644); err != nil {
if err := writeFileAtomic(path, data, 0o644); err != nil {
return fmt.Errorf("write diagnostics artifact %q: %w", name, err)
}
return nil
@@ -241,3 +241,39 @@ func (r *RunDirectory) artifactPath(name string) (string, error) {
}
return artifactPath, nil
}
func writeFileAtomic(path string, data []byte, perm os.FileMode) error {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o755); err != nil {
return err
}
temp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".tmp-*")
if err != nil {
return err
}
tempPath := temp.Name()
removeTemp := true
defer func() {
if removeTemp {
_ = os.Remove(tempPath)
}
}()
if _, err := temp.Write(data); err != nil {
_ = temp.Close()
return err
}
if err := temp.Chmod(perm); err != nil {
_ = temp.Close()
return err
}
if err := temp.Close(); err != nil {
return err
}
if err := os.Rename(tempPath, path); err != nil {
return err
}
removeTemp = false
return nil
}

View File

@@ -118,6 +118,24 @@ func TestWriteJSONArtifactWritesIndentedNewlineTerminatedJSON(t *testing.T) {
}
}
func TestWriteJSONArtifactLeavesNoTemporaryFiles(t *testing.T) {
runDir := newTestRunDirectory(t)
if err := runDir.WriteJSONArtifact("artifact.json", map[string]any{"value": "ok"}); err != nil {
t.Fatalf("WriteJSONArtifact: %v", err)
}
entries, err := os.ReadDir(runDir.Path())
if err != nil {
t.Fatalf("read run directory: %v", err)
}
for _, entry := range entries {
if strings.Contains(entry.Name(), ".tmp-") {
t.Fatalf("temporary diagnostics file remains after success: %s", entry.Name())
}
}
}
func TestWriteInvocationMetadataFillsMissingRunIDAndStartTime(t *testing.T) {
runDir := newTestRunDirectory(t)

View File

@@ -124,10 +124,13 @@ func TestContractsComposeAcrossPackages(t *testing.T) {
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
if len(output.Files) != 1 {
t.Fatalf("len(Files) = %d, want 1", len(output.Files))
}
if len(output.Bytes) == 0 {
if output.Files[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.Files[0].ContentType)
}
if len(output.Files[0].Bytes) == 0 {
t.Fatal("len(Bytes) = 0, want encoded bytes")
}
}
@@ -293,7 +296,12 @@ func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contract
}
return contracts.OutputResult{
Bytes: encoded,
ContentType: "application/json",
Files: []contracts.OutputFile{
{
Name: "artifacts/generic.json",
ContentType: "application/json",
Bytes: encoded,
},
},
}, nil
}

View File

@@ -182,8 +182,17 @@ type OutputRequest struct {
Metadata map[string]any `json:"metadata,omitempty"`
}
type OutputFile struct {
Name string `json:"name"`
ContentType string `json:"content_type,omitempty"`
Bytes []byte `json:"-"`
}
type OutputResult struct {
Bytes []byte `json:"-"`
Files []OutputFile `json:"files,omitempty"`
// Bytes is the legacy single-output payload. New encoders should return Files.
Bytes []byte `json:"-"`
// ContentType is the legacy single-output content type. New encoders should return Files.
ContentType string `json:"content_type,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
@@ -192,3 +201,7 @@ type OutputEncoder interface {
Key() string
Encode(ctx context.Context, req OutputRequest) (OutputResult, error)
}
type ManifestMetadataProvider interface {
ManifestMetadata() map[string]any
}

View File

@@ -230,11 +230,44 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if encoder.Key() != "generic-output" {
t.Fatalf("OutputEncoder.Key() = %q, want generic-output", encoder.Key())
}
if encoded.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", encoded.ContentType)
if len(encoded.Files) != 1 {
t.Fatalf("len(Files) = %d, want 1", len(encoded.Files))
}
if string(encoded.Bytes) != `{"run_id":"run-1","approved_count":1}` {
t.Fatalf("Bytes = %s, want encoded output", encoded.Bytes)
if encoded.Files[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", encoded.Files[0].ContentType)
}
if string(encoded.Files[0].Bytes) != `{"run_id":"run-1","approved_count":1}` {
t.Fatalf("Bytes = %s, want encoded output", encoded.Files[0].Bytes)
}
}
func TestOutputFileJSONShapeOmitsBytes(t *testing.T) {
file := OutputFile{
Name: "artifacts/events.json",
ContentType: "application/json",
Bytes: []byte(`{"ignored":true}`),
}
encoded, err := json.Marshal(file)
if err != nil {
t.Fatalf("json.Marshal() error = %v, want nil", err)
}
var got map[string]any
if err := json.Unmarshal(encoded, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v, want nil", err)
}
if got["name"] != "artifacts/events.json" {
t.Fatalf("name = %#v, want logical file name", got["name"])
}
if got["content_type"] != "application/json" {
t.Fatalf("content_type = %#v, want application/json", got["content_type"])
}
if _, ok := got["Bytes"]; ok {
t.Fatalf("encoded output file leaked Bytes: %s", encoded)
}
if _, ok := got["bytes"]; ok {
t.Fatalf("encoded output file leaked bytes: %s", encoded)
}
}
@@ -397,7 +430,12 @@ func (encoder fakeOutputEncoder) Key() string {
func (encoder fakeOutputEncoder) Encode(ctx context.Context, req OutputRequest) (OutputResult, error) {
return OutputResult{
Bytes: []byte(`{"run_id":"` + req.Manifest.RunID + `","approved_count":1}`),
ContentType: "application/json",
Files: []OutputFile{
{
Name: "artifacts/generic.json",
ContentType: "application/json",
Bytes: []byte(`{"run_id":"` + req.Manifest.RunID + `","approved_count":1}`),
},
},
}, nil
}

View File

@@ -0,0 +1,43 @@
package llm
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type scheduledClient struct {
client contracts.StructuredLLMClient
scheduler *Scheduler
}
func NewScheduledClient(client contracts.StructuredLLMClient, scheduler *Scheduler) contracts.StructuredLLMClient {
return &scheduledClient{
client: client,
scheduler: scheduler,
}
}
func (c *scheduledClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
if c == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scheduled LLM client must not be nil")
}
if c.client == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scheduled LLM client inner client must not be nil")
}
if c.scheduler == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scheduled LLM client scheduler must not be nil")
}
var response contracts.StructuredCompletionResponse
err := c.scheduler.Run(ctx, func(ctx context.Context) error {
var callErr error
response, callErr = c.client.CompleteStructured(ctx, req, out)
return callErr
})
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return response, nil
}

View File

@@ -0,0 +1,115 @@
package llm
import (
"context"
"encoding/json"
"errors"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestScheduledClientEnforcesSchedulerLimit(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
inner := &blockingStructuredClient{
release: make(chan struct{}),
}
client := NewScheduledClient(inner, scheduler)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
var out map[string]any
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err != nil {
t.Errorf("CompleteStructured() error = %v, want nil", err)
}
}()
}
waitForAtomicAtLeast(t, &inner.calls, 1)
time.Sleep(20 * time.Millisecond)
if got := atomic.LoadInt32(&inner.maxInFlight); got > 1 {
t.Fatalf("max in-flight calls = %d, want <= 1", got)
}
close(inner.release)
wg.Wait()
if got := atomic.LoadInt32(&inner.calls); got != 3 {
t.Fatalf("calls = %d, want 3", got)
}
}
func TestScheduledClientPropagatesClientError(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
expected := errors.New("provider unavailable")
client := NewScheduledClient(&errorStructuredClient{err: expected}, scheduler)
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &struct{}{})
if !errors.Is(err, expected) {
t.Fatalf("CompleteStructured() error = %v, want %v", err, expected)
}
}
func TestScheduledClientPropagatesSchedulerError(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
client := NewScheduledClient(&errorStructuredClient{}, scheduler)
_, err = client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{}, &struct{}{})
if !errors.Is(err, context.Canceled) {
t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
}
}
type blockingStructuredClient struct {
release chan struct{}
inFlight int32
maxInFlight int32
calls int32
}
func (c *blockingStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
atomic.AddInt32(&c.calls, 1)
current := atomic.AddInt32(&c.inFlight, 1)
for {
seen := atomic.LoadInt32(&c.maxInFlight)
if current <= seen || atomic.CompareAndSwapInt32(&c.maxInFlight, seen, current) {
break
}
}
defer atomic.AddInt32(&c.inFlight, -1)
select {
case <-c.release:
case <-ctx.Done():
return contracts.StructuredCompletionResponse{}, ctx.Err()
}
if target, ok := out.(*map[string]any); ok {
*target = map[string]any{"ok": true}
}
return contracts.StructuredCompletionResponse{
Content: json.RawMessage(`{"ok":true}`),
}, nil
}
type errorStructuredClient struct {
err error
}
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
return contracts.StructuredCompletionResponse{}, c.err
}

View File

@@ -6,6 +6,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"io/fs"
"sort"
"strings"
)
@@ -17,13 +18,21 @@ var schemaAssets embed.FS
type ResponseSchemaKey string
const (
DNDSpellsSchemaKey ResponseSchemaKey = "dnd_spells"
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
schemaVersionV1 = "v1"
)
// ResponseSchemaDefinition identifies a caller-owned structured response schema asset.
type ResponseSchemaDefinition struct {
Key ResponseSchemaKey
ID string
Version string
Name string
AssetPath string
}
// ResponseSchema describes one registered structured response schema.
type ResponseSchema struct {
Key ResponseSchemaKey `json:"key"`
@@ -35,27 +44,20 @@ type ResponseSchema struct {
}
var responseSchemaRegistry = map[ResponseSchemaKey]ResponseSchema{
DNDSpellsSchemaKey: mustLoadResponseSchema(
DNDSpellsSchemaKey,
"notarius.dnd.spells",
schemaVersionV1,
"notarius_dnd_spells_v1",
"assets/schemas/dnd_spells.v1.json",
),
TestArtifactSchemaKey: mustLoadResponseSchema(
TestArtifactSchemaKey,
"notarius.test_artifact",
schemaVersionV1,
"notarius_test_artifact_v1",
"assets/schemas/test_artifact.v1.json",
),
TestValidatorDecisionSchemaKey: mustLoadResponseSchema(
TestValidatorDecisionSchemaKey,
"notarius.test_validator_decision",
schemaVersionV1,
"notarius_test_validator_decision_v1",
"assets/schemas/test_validator_decision.v1.json",
),
TestArtifactSchemaKey: mustLoadResponseSchema(schemaAssets, ResponseSchemaDefinition{
Key: TestArtifactSchemaKey,
ID: "notarius.test_artifact",
Version: schemaVersionV1,
Name: "notarius_test_artifact_v1",
AssetPath: "assets/schemas/test_artifact.v1.json",
}),
TestValidatorDecisionSchemaKey: mustLoadResponseSchema(schemaAssets, ResponseSchemaDefinition{
Key: TestValidatorDecisionSchemaKey,
ID: "notarius.test_validator_decision",
Version: schemaVersionV1,
Name: "notarius_test_validator_decision_v1",
AssetPath: "assets/schemas/test_validator_decision.v1.json",
}),
}
// RegisteredResponseSchemas returns all registered response schemas sorted by key.
@@ -102,40 +104,35 @@ func (s ResponseSchema) DiagnosticsMap() map[string]any {
}
}
func mustLoadResponseSchema(
key ResponseSchemaKey,
id string,
version string,
name string,
path string,
) ResponseSchema {
key = ResponseSchemaKey(strings.TrimSpace(string(key)))
id = strings.TrimSpace(id)
version = strings.TrimSpace(version)
name = strings.TrimSpace(name)
path = strings.TrimSpace(path)
// LoadResponseSchema loads a structured response schema from a caller-owned filesystem.
func LoadResponseSchema(fsys fs.FS, def ResponseSchemaDefinition) (ResponseSchema, error) {
key := ResponseSchemaKey(strings.TrimSpace(string(def.Key)))
id := strings.TrimSpace(def.ID)
version := strings.TrimSpace(def.Version)
name := strings.TrimSpace(def.Name)
path := strings.TrimSpace(def.AssetPath)
if key == "" {
panic("response schema key must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema key must not be empty")
}
if id == "" {
panic("response schema id must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema id must not be empty")
}
if version == "" {
panic("response schema version must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema version must not be empty")
}
if name == "" {
panic("response schema name must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema name must not be empty")
}
if path == "" {
panic("response schema asset path must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema asset path must not be empty")
}
rawSchema, err := schemaAssets.ReadFile(path)
rawSchema, err := fs.ReadFile(fsys, path)
if err != nil {
panic(fmt.Sprintf("read response schema %s: %v", path, err))
return ResponseSchema{}, fmt.Errorf("read response schema %s: %w", path, err)
}
if !json.Valid(rawSchema) {
panic(fmt.Sprintf("response schema %s is not valid JSON", path))
return ResponseSchema{}, fmt.Errorf("response schema %s is not valid JSON", path)
}
hash := sha256.Sum256(rawSchema)
@@ -146,7 +143,15 @@ func mustLoadResponseSchema(
Name: name,
JSONSchema: append(json.RawMessage(nil), rawSchema...),
SHA256: "sha256:" + hex.EncodeToString(hash[:]),
}, nil
}
func mustLoadResponseSchema(fsys fs.FS, def ResponseSchemaDefinition) ResponseSchema {
schema, err := LoadResponseSchema(fsys, def)
if err != nil {
panic(err)
}
return schema
}
func cloneResponseSchema(in ResponseSchema) ResponseSchema {

View File

@@ -9,7 +9,6 @@ import (
func TestLookupResponseSchemaSucceedsForRegisteredSchemas(t *testing.T) {
tests := []ResponseSchemaKey{
DNDSpellsSchemaKey,
TestArtifactSchemaKey,
TestValidatorDecisionSchemaKey,
}
@@ -39,6 +38,12 @@ func TestLookupResponseSchemaUnknownReturnsFalse(t *testing.T) {
}
}
func TestLookupResponseSchemaDNDSpellsIsNotFrameworkRegistered(t *testing.T) {
if schema, ok := LookupResponseSchema("dnd_spells"); ok {
t.Fatalf("expected D&D spells schema lookup to fail in framework registry, got %+v", schema)
}
}
func TestMustLookupResponseSchemaPanicsForUnknownKey(t *testing.T) {
defer func() {
if recover() == nil {
@@ -51,8 +56,8 @@ func TestMustLookupResponseSchemaPanicsForUnknownKey(t *testing.T) {
func TestRegisteredResponseSchemasSortedByKey(t *testing.T) {
schemas := RegisteredResponseSchemas()
if len(schemas) != 3 {
t.Fatalf("expected three schemas, got %d", len(schemas))
if len(schemas) != 2 {
t.Fatalf("expected two schemas, got %d", len(schemas))
}
keys := make([]string, len(schemas))
@@ -64,8 +69,8 @@ func TestRegisteredResponseSchemasSortedByKey(t *testing.T) {
if !sort.StringsAreSorted(keys) {
t.Fatalf("expected sorted keys, got %v", keys)
}
if !seen[DNDSpellsSchemaKey] {
t.Fatalf("registered schemas = %v, want %q", keys, DNDSpellsSchemaKey)
if !seen[TestArtifactSchemaKey] || !seen[TestValidatorDecisionSchemaKey] {
t.Fatalf("registered schemas = %v, want test schemas", keys)
}
}
@@ -78,7 +83,7 @@ func TestResponseSchemaContentIsValidJSON(t *testing.T) {
}
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
for _, key := range []ResponseSchemaKey{DNDSpellsSchemaKey, TestArtifactSchemaKey} {
for _, key := range []ResponseSchemaKey{TestArtifactSchemaKey, TestValidatorDecisionSchemaKey} {
t.Run(string(key), func(t *testing.T) {
first := MustLookupResponseSchema(key)
first.JSONSchema[0] = '['

View File

@@ -0,0 +1,124 @@
package pipeline_test
import (
"context"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
)
func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
cfg := config.Default()
cfg.Pipelines = map[string]pipeline.PipelineProfile{
"defaults": {
Input: pipeline.Binding("input"),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"events": {Extract: pipeline.Binding("extract")},
},
},
}
resolved, err := cfg.Resolve(config.ResolveInput{
PipelineID: "defaults",
Catalog: defaultModuleCatalog(t),
})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
pipeline := resolved.ResolvedPipeline
if pipeline.Chunk.Module != generic.Key {
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, generic.Key)
}
if pipeline.Output.Module != jsonoutput.Key {
t.Fatalf("Output.Module = %q, want %q", pipeline.Output.Module, jsonoutput.Key)
}
lane := pipeline.ArtifactLanes[0]
if lane.Merge.Module != appendorder.Key {
t.Fatalf("Merge.Module = %q, want %q", lane.Merge.Module, appendorder.Key)
}
if lane.Normalize.Module != noop.Key {
t.Fatalf("Normalize.Module = %q, want %q", lane.Normalize.Module, noop.Key)
}
}
func defaultModuleCatalog(t *testing.T) pipeline.ModuleCatalog {
t.Helper()
inputs := pipeline.NewInputAdapterRegistry()
chunkers := pipeline.NewChunkerRegistry()
extractors := pipeline.NewExtractorRegistry()
mergers := pipeline.NewMergerRegistry()
normalizers := pipeline.NewNormalizerRegistry()
outputs := pipeline.NewOutputEncoderRegistry()
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{
Key: "input",
Stage: pipeline.StageInput,
Provides: []string{"source"},
}, func() (contracts.InputAdapter, error) {
return defaultInput{}, nil
}); err != nil {
t.Fatalf("register input: %v", err)
}
if err := generic.Register(chunkers); err != nil {
t.Fatalf("register generic chunker: %v", err)
}
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
Key: "extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"records"},
}, func() (contracts.Extractor, error) {
return defaultExtractor{}, nil
}); err != nil {
t.Fatalf("register extractor: %v", err)
}
if err := appendorder.Register(mergers); err != nil {
t.Fatalf("register appendorder merger: %v", err)
}
if err := noop.Register(normalizers); err != nil {
t.Fatalf("register noop normalizer: %v", err)
}
if err := jsonoutput.Register(outputs); err != nil {
t.Fatalf("register json output: %v", err)
}
return pipeline.ModuleCatalog{
Inputs: inputs,
Chunkers: chunkers,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
Outputs: outputs,
}
}
type defaultInput struct{}
func (defaultInput) Key() string { return "input" }
func (defaultInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
return nil, nil
}
type defaultExtractor struct{}
func (defaultExtractor) Key() string { return "extract" }
func (defaultExtractor) ArtifactType() string { return "record" }
func (defaultExtractor) SchemaVersion() string { return "v1" }
func (defaultExtractor) Validators() []contracts.Validator { return nil }
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
}

View File

@@ -1,70 +0,0 @@
package pipeline
import (
"context"
"encoding/json"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type AppendOrderMerger struct{}
func (m AppendOrderMerger) Key() string {
return DefaultMergeModule
}
func (m AppendOrderMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, copyArtifactCandidates(chunkArtifacts.Candidates)...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
type NoopNormalizer struct{}
func (n NoopNormalizer) Key() string {
return DefaultNormalizeModule
}
func (n NoopNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: copyArtifactCandidates(req.Candidates)}, nil
}
func copyArtifactCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
}
copied := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
copied = append(copied, copyArtifactCandidate(candidate))
}
return copied
}
func copyArtifactCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: copyArtifactMetadata(candidate.Metadata),
}
}
func copyArtifactMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
copied := make(map[string]any, len(metadata))
for key, value := range metadata {
copied[key] = value
}
return copied
}

View File

@@ -1,221 +0,0 @@
package pipeline
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestGenericMergeAndNormalizeKeys(t *testing.T) {
merger := AppendOrderMerger{}
normalizer := NoopNormalizer{}
if merger.Key() != DefaultMergeModule {
t.Fatalf("AppendOrderMerger.Key() = %q, want %q", merger.Key(), DefaultMergeModule)
}
if normalizer.Key() != DefaultNormalizeModule {
t.Fatalf("NoopNormalizer.Key() = %q, want %q", normalizer.Key(), DefaultNormalizeModule)
}
}
func TestAppendOrderMergerConcatenatesByChunkAndCandidateOrder(t *testing.T) {
merger := AppendOrderMerger{}
chunks := []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{
candidate(2, "first-b"),
candidate(1, "first-a"),
},
},
{
Chunk: sourceChunk(1),
Candidates: []artifacts.ArtifactCandidate{
candidate(4, "second-b"),
candidate(3, "second-a"),
},
},
}
result, err := merger.Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: chunks})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"first-b", "first-a", "second-b", "second-a"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
}
func TestAppendOrderMergerReturnsMutationSafeCandidates(t *testing.T) {
merger := AppendOrderMerger{}
input := []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{
candidate(1, "original"),
},
},
}
result, err := merger.Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: input})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Candidates[0].Index = 99
input[0].Candidates[0].Payload[0] = '['
input[0].Candidates[0].SourceRefs[0].StartUnitID = "changed"
input[0].Candidates[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].StartUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestNoopNormalizerPreservesOrderAndValues(t *testing.T) {
normalizer := NoopNormalizer{}
input := []artifacts.ArtifactCandidate{
candidate(3, "third"),
candidate(1, "first"),
candidate(2, "second"),
}
result, err := normalizer.Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"third", "first", "second"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
if !reflect.DeepEqual(result.Candidates[0].SourceRefs, input[0].SourceRefs) {
t.Fatalf("SourceRefs = %#v, want %#v", result.Candidates[0].SourceRefs, input[0].SourceRefs)
}
if !reflect.DeepEqual(result.Candidates[0].Metadata, input[0].Metadata) {
t.Fatalf("Metadata = %#v, want %#v", result.Candidates[0].Metadata, input[0].Metadata)
}
}
func TestNoopNormalizerReturnsMutationSafeCandidates(t *testing.T) {
normalizer := NoopNormalizer{}
input := []artifacts.ArtifactCandidate{candidate(1, "original")}
result, err := normalizer.Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Index = 99
input[0].Payload[0] = '['
input[0].SourceRefs[0].EndUnitID = "changed"
input[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].EndUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestGenericMergeAndNormalizeHandleEmptyInput(t *testing.T) {
merger := AppendOrderMerger{}
normalizer := NoopNormalizer{}
mergeResult, err := merger.Merge(context.Background(), contracts.MergeRequest{})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(mergeResult.Candidates) != 0 {
t.Fatalf("len(mergeResult.Candidates) = %d, want 0", len(mergeResult.Candidates))
}
if len(mergeResult.Warnings) != 0 {
t.Fatalf("merge warnings = %#v, want none", mergeResult.Warnings)
}
normalizeResult, err := normalizer.Normalize(context.Background(), contracts.NormalizeRequest{})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(normalizeResult.Candidates) != 0 {
t.Fatalf("len(normalizeResult.Candidates) = %d, want 0", len(normalizeResult.Candidates))
}
if len(normalizeResult.Warnings) != 0 {
t.Fatalf("normalize warnings = %#v, want none", normalizeResult.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}
func sourceChunk(index int) contracts.SourceChunk {
return contracts.SourceChunk{
ID: "chunk",
SourceID: "source-1",
Index: index,
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
},
}
}

View File

@@ -3,6 +3,9 @@ package pipeline
import (
"context"
"fmt"
"path"
"strings"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
@@ -29,21 +32,27 @@ func New(registries Registries) *Runner {
}
type RunInput struct {
Pipeline ResolvedPipeline
SourceID string
Path string
RawInput []byte
LLMClient contracts.StructuredLLMClient
Metadata map[string]any
Pipeline ResolvedPipeline
SourceID string
Path string
RawInput []byte
LLMClient contracts.StructuredLLMClient
RunID string
StartedAt time.Time
LLMProfiles []artifacts.LLMProfileManifest
Metadata map[string]any
}
type RunOutput struct {
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved,omitempty"`
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
EncodedOutput []byte `json:"-"`
ContentType string `json:"content_type,omitempty"`
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved,omitempty"`
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
OutputFiles []contracts.OutputFile `json:"-"`
// EncodedOutput is the legacy single-output payload. New callers should use OutputFiles.
EncodedOutput []byte `json:"-"`
// ContentType is the legacy single-output content type. New callers should use OutputFiles.
ContentType string `json:"content_type,omitempty"`
}
func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
@@ -58,7 +67,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
return output, err
}
output.Manifest = manifestFromPipeline(input.Pipeline)
output.Manifest = manifestFromPipeline(input)
adapter, err := r.registries.Inputs.Build(input.Pipeline.Input.Module)
if err != nil {
@@ -110,6 +119,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
} else {
output.Manifest.ValidationStatus = "approved"
}
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
encoder, err := r.registries.Outputs.Build(input.Pipeline.Output.Module)
if err != nil {
@@ -128,8 +138,15 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
if err != nil {
return failOutput(output), fmt.Errorf("encode output with encoder %q: %w", encoder.Key(), err)
}
output.EncodedOutput = encoded.Bytes
output.ContentType = encoded.ContentType
files, err := outputFilesFromResult(encoded)
if err != nil {
return failOutput(output), fmt.Errorf("validate output files from encoder %q: %w", encoder.Key(), err)
}
output.OutputFiles = files
if len(encoded.Files) == 0 {
output.EncodedOutput = append([]byte(nil), encoded.Bytes...)
output.ContentType = encoded.ContentType
}
return output, nil
}
@@ -147,6 +164,7 @@ func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.Source
if err != nil {
return fmt.Errorf("build normalizer %q for lane %q: %w", lane.Normalize.Module, lane.ID, err)
}
setLaneManifestMetadata(output, lane.ID, extractor, merger, normalizer)
var validators []validatorExecution
if len(lane.Validators) > 0 {
@@ -315,7 +333,17 @@ func validateRunInput(input RunInput) error {
return nil
}
func manifestFromPipeline(pipeline ResolvedPipeline) artifacts.RunManifest {
func manifestFromPipeline(input RunInput) artifacts.RunManifest {
startedAt := input.StartedAt
if startedAt.IsZero() {
startedAt = time.Now().UTC()
}
runID := strings.TrimSpace(input.RunID)
if runID == "" {
runID = fmt.Sprintf("run-%d", startedAt.UnixNano())
}
pipeline := input.Pipeline
manifest := artifacts.RunManifest{
PipelineID: pipeline.ID,
PipelineDigest: pipeline.Digest,
@@ -323,6 +351,9 @@ func manifestFromPipeline(pipeline ResolvedPipeline) artifacts.RunManifest {
Chunker: pipeline.Chunk.Module,
OutputEncoder: pipeline.Output.Module,
ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.ArtifactLanes)),
RunID: runID,
StartedAt: timePtr(startedAt),
LLMProfiles: cloneLLMProfiles(input.LLMProfiles),
}
for _, lane := range pipeline.ArtifactLanes {
@@ -343,10 +374,124 @@ func manifestFromPipeline(pipeline ResolvedPipeline) artifacts.RunManifest {
func failOutput(output RunOutput) RunOutput {
if output.Manifest.PipelineID != "" {
output.Manifest.ValidationStatus = "failed"
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
}
return output
}
func setLaneManifestMetadata(output *RunOutput, laneID string, modules ...any) {
if output == nil {
return
}
for i := range output.Manifest.ArtifactLanes {
if output.Manifest.ArtifactLanes[i].ID != laneID {
continue
}
metadata := make(map[string]any)
for _, module := range modules {
provider, ok := module.(contracts.ManifestMetadataProvider)
if !ok {
continue
}
moduleMetadata := cloneMetadata(provider.ManifestMetadata())
if len(moduleMetadata) == 0 {
continue
}
key := manifestMetadataKey(module)
if key == "" {
continue
}
metadata[key] = moduleMetadata
}
if len(metadata) > 0 {
output.Manifest.ArtifactLanes[i].Metadata = metadata
}
return
}
}
func manifestMetadataKey(module any) string {
switch module.(type) {
case contracts.Extractor:
return "extractor"
case contracts.Merger:
return "merger"
case contracts.Normalizer:
return "normalizer"
default:
return ""
}
}
func outputFilesFromResult(result contracts.OutputResult) ([]contracts.OutputFile, error) {
files := result.Files
if len(files) == 0 && len(result.Bytes) > 0 {
files = []contracts.OutputFile{
{
Name: "output",
ContentType: result.ContentType,
Bytes: result.Bytes,
},
}
}
out := make([]contracts.OutputFile, 0, len(files))
for _, file := range files {
if err := validateOutputFileName(file.Name); err != nil {
return nil, err
}
out = append(out, contracts.OutputFile{
Name: file.Name,
ContentType: file.ContentType,
Bytes: append([]byte(nil), file.Bytes...),
})
}
return out, nil
}
func validateOutputFileName(name string) error {
if strings.TrimSpace(name) == "" {
return fmt.Errorf("output file name must not be empty")
}
if strings.Contains(name, "\\") {
return fmt.Errorf("output file name %q must use slash-separated relative paths", name)
}
if path.IsAbs(name) {
return fmt.Errorf("output file name %q must be relative", name)
}
if strings.Contains(name, "..") {
return fmt.Errorf("output file name %q must not contain ..", name)
}
cleaned := path.Clean(name)
if cleaned == "." || cleaned != name {
return fmt.Errorf("output file name %q must be clean", name)
}
return nil
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func cloneLLMProfiles(profiles []artifacts.LLMProfileManifest) []artifacts.LLMProfileManifest {
if len(profiles) == 0 {
return nil
}
return append([]artifacts.LLMProfileManifest(nil), profiles...)
}
func timePtr(t time.Time) *time.Time {
return &t
}
func pipelineUsesConfiguredValidators(pipeline ResolvedPipeline) bool {
for _, lane := range pipeline.ArtifactLanes {
if len(lane.Validators) > 0 {

View File

@@ -6,6 +6,7 @@ import (
"reflect"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
@@ -32,6 +33,7 @@ func TestNewAndDataTypes(t *testing.T) {
Approved: []artifacts.Artifact{{ExtractorKey: "extract-alpha"}},
Rejected: []artifacts.RejectedArtifact{{ValidatorName: "validator"}},
Warnings: []contracts.Warning{{ReasonCode: "note", Message: "message"}},
OutputFiles: []contracts.OutputFile{{Name: "artifacts/generic.json", ContentType: "application/json", Bytes: []byte(`{}`)}},
EncodedOutput: []byte(`{}`),
ContentType: "application/json",
}
@@ -39,7 +41,7 @@ func TestNewAndDataTypes(t *testing.T) {
if input.Pipeline.ID != "pipeline-1" || input.SourceID != "source-1" {
t.Fatalf("RunInput = %#v, want constructed fields", input)
}
if output.Manifest.PipelineID != "pipeline-1" || len(output.Approved) != 1 || len(output.Rejected) != 1 || len(output.Warnings) != 1 {
if output.Manifest.PipelineID != "pipeline-1" || len(output.Approved) != 1 || len(output.Rejected) != 1 || len(output.Warnings) != 1 || len(output.OutputFiles) != 1 {
t.Fatalf("RunOutput = %#v, want constructed fields", output)
}
}
@@ -604,11 +606,18 @@ func TestRunOutputEncoderReceivesManifestAndArtifacts(t *testing.T) {
t.Fatalf("Run() error = %v, want nil", err)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
if len(output.OutputFiles) != 1 {
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
}
if string(output.EncodedOutput) != `{"encoded":true}` {
t.Fatalf("EncodedOutput = %s, want encoded payload", output.EncodedOutput)
file := output.OutputFiles[0]
if file.Name != "artifacts/generic.json" {
t.Fatalf("OutputFiles[0].Name = %q, want artifacts/generic.json", file.Name)
}
if file.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", file.ContentType)
}
if string(file.Bytes) != `{"encoded":true}` {
t.Fatalf("OutputFiles[0].Bytes = %s, want encoded payload", file.Bytes)
}
if len(modules.output.requests) != 1 {
t.Fatalf("len(output requests) = %d, want 1", len(modules.output.requests))
@@ -622,6 +631,35 @@ func TestRunOutputEncoderReceivesManifestAndArtifacts(t *testing.T) {
}
}
func TestRunRejectsUnsafeOutputFileNames(t *testing.T) {
tests := []struct {
name string
fileName string
}{
{name: "empty", fileName: ""},
{name: "absolute", fileName: "/tmp/output.json"},
{name: "parent", fileName: "artifacts/../manifest.json"},
{name: "backslash", fileName: `artifacts\manifest.json`},
{name: "unclean", fileName: "artifacts//manifest.json"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
modules := defaultRunnerModules()
modules.output.files = []contracts.OutputFile{
{Name: test.fileName, ContentType: "application/json", Bytes: []byte(`{}`)},
}
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
assertRunError(t, err, "output file name")
if output.Manifest.ValidationStatus != "failed" {
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
}
})
}
}
func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
modules := defaultRunnerModules()
modules.output.err = errors.New("encode failed")
@@ -632,6 +670,9 @@ func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
if output.Manifest.ValidationStatus != "failed" {
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
}
if output.Manifest.CompletedAt == nil {
t.Fatal("CompletedAt = nil, want failed run completion timestamp")
}
if len(output.Approved) != 2 {
t.Fatalf("len(Approved) = %d, want partial approved output", len(output.Approved))
}
@@ -668,6 +709,76 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
}
}
func TestRunManifestIncludesRunTimingAndLLMProfiles(t *testing.T) {
startedAt := time.Now().Add(-time.Minute).UTC()
profiles := []artifacts.LLMProfileManifest{
{ID: "default", Provider: "openai-compatible", Model: "model-a"},
}
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{
Pipeline: resolvedPipeline(),
RunID: "run-test",
StartedAt: startedAt,
LLMProfiles: profiles,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
manifest := output.Manifest
if manifest.RunID != "run-test" {
t.Fatalf("RunID = %q, want run-test", manifest.RunID)
}
if manifest.StartedAt == nil || !manifest.StartedAt.Equal(startedAt) {
t.Fatalf("StartedAt = %v, want %s", manifest.StartedAt, startedAt)
}
if manifest.CompletedAt == nil || manifest.CompletedAt.Before(startedAt) {
t.Fatalf("CompletedAt = %v, want timestamp after start", manifest.CompletedAt)
}
if !reflect.DeepEqual(manifest.LLMProfiles, profiles) {
t.Fatalf("LLMProfiles = %#v, want %#v", manifest.LLMProfiles, profiles)
}
}
func TestRunManifestGeneratesRunIDAndTimestamps(t *testing.T) {
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if !strings.HasPrefix(output.Manifest.RunID, "run-") {
t.Fatalf("RunID = %q, want generated run ID", output.Manifest.RunID)
}
if output.Manifest.StartedAt == nil {
t.Fatal("StartedAt = nil, want generated timestamp")
}
if output.Manifest.CompletedAt == nil {
t.Fatal("CompletedAt = nil, want generated timestamp")
}
}
func TestRunManifestIncludesExtractorMetadata(t *testing.T) {
modules := defaultRunnerModules()
modules.extractors["extract-alpha"].manifestMetadata = map[string]any{
"prompt_id": "test.prompt",
"response_schema_name": "test_schema",
}
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
lane := output.Manifest.ArtifactLanes[0]
extractorMetadata, ok := lane.Metadata["extractor"].(map[string]any)
if !ok {
t.Fatalf("lane metadata = %#v, want extractor metadata", lane.Metadata)
}
if extractorMetadata["prompt_id"] != "test.prompt" || extractorMetadata["response_schema_name"] != "test_schema" {
t.Fatalf("extractor metadata = %#v, want prompt and schema metadata", extractorMetadata)
}
}
func TestRunReturnsPartialOutputWhenLaterLaneFails(t *testing.T) {
modules := defaultRunnerModules()
modules.extractors["extract-beta"] = &runnerExtractor{key: "extract-beta", artifactType: "artifact", schemaVersion: "v1", err: errors.New("extract failed")}
@@ -783,7 +894,12 @@ func defaultRunnerModules() *runnerModules {
"configured": {name: "configured", decisions: approveAll},
"second-validator": {name: "second-validator", decisions: approveAll},
},
output: &runnerOutputEncoder{key: "output", bytes: []byte(`{"encoded":true}`), contentType: "application/json"},
output: &runnerOutputEncoder{
key: "output",
files: []contracts.OutputFile{
{Name: "artifacts/generic.json", ContentType: "application/json", Bytes: []byte(`{"encoded":true}`)},
},
},
}
}
@@ -885,17 +1001,18 @@ func (chunker *runnerChunker) Chunk(ctx context.Context, req contracts.ChunkRequ
}
type runnerExtractor struct {
key string
artifactType string
schemaVersion string
candidates []artifacts.ArtifactCandidate
validators []contracts.Validator
warnings []contracts.Warning
err error
requests []contracts.ExtractionRequest
seenChunkIDs []string
seenLLMClients []contracts.StructuredLLMClient
seenMetadata []map[string]any
key string
artifactType string
schemaVersion string
manifestMetadata map[string]any
candidates []artifacts.ArtifactCandidate
validators []contracts.Validator
warnings []contracts.Warning
err error
requests []contracts.ExtractionRequest
seenChunkIDs []string
seenLLMClients []contracts.StructuredLLMClient
seenMetadata []map[string]any
}
func (extractor *runnerExtractor) Key() string {
@@ -910,6 +1027,10 @@ func (extractor *runnerExtractor) SchemaVersion() string {
return extractor.schemaVersion
}
func (extractor *runnerExtractor) ManifestMetadata() map[string]any {
return extractor.manifestMetadata
}
func (extractor *runnerExtractor) Validators() []contracts.Validator {
return extractor.validators
}
@@ -1020,6 +1141,7 @@ func (validator *runnerValidator) Validate(ctx context.Context, req contracts.Va
type runnerOutputEncoder struct {
key string
files []contracts.OutputFile
bytes []byte
contentType string
warnings []contracts.Warning
@@ -1034,6 +1156,7 @@ func (encoder *runnerOutputEncoder) Key() string {
func (encoder *runnerOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
encoder.requests = append(encoder.requests, req)
return contracts.OutputResult{
Files: encoder.files,
Bytes: encoder.bytes,
ContentType: encoder.contentType,
Warnings: encoder.warnings,

View File

@@ -135,7 +135,7 @@ func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
Stage: StageMerge,
Requires: []string{"fake_artifacts"},
}, func() (contracts.Merger, error) {
return AppendOrderMerger{}, nil
return walkingSkeletonMerger{}, nil
}); err != nil {
t.Fatalf("register append-order merger: %v", err)
}
@@ -143,7 +143,7 @@ func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
Key: DefaultNormalizeModule,
Stage: StageNormalize,
}, func() (contracts.Normalizer, error) {
return NoopNormalizer{}, nil
return walkingSkeletonNormalizer{}, nil
}); err != nil {
t.Fatalf("register no-op normalizer: %v", err)
}
@@ -309,6 +309,30 @@ func (client *walkingSkeletonLLMClient) CompleteStructured(ctx context.Context,
}, nil
}
type walkingSkeletonMerger struct{}
func (merger walkingSkeletonMerger) Key() string {
return DefaultMergeModule
}
func (merger walkingSkeletonMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
type walkingSkeletonNormalizer struct{}
func (normalizer walkingSkeletonNormalizer) Key() string {
return DefaultNormalizeModule
}
func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: req.Candidates}, nil
}
type walkingSkeletonOutput struct{}
func (output walkingSkeletonOutput) Key() string {

View File

@@ -5,6 +5,7 @@ import (
"embed"
"encoding/hex"
"fmt"
"io/fs"
"path"
"sort"
"strings"
@@ -17,7 +18,6 @@ var embeddedAssets embed.FS
const (
SourceBuiltin = "builtin"
VersionV1 = "v1"
DNDSpellsPromptID = "dnd.spells"
TestGenericPromptID = "test.generic"
)
@@ -41,21 +41,31 @@ func (m Metadata) DiagnosticsMap() map[string]any {
}
}
type definition struct {
id string
version string
embeddedDir string
systemPath string
userPath string
// Definition identifies a caller-owned system/user prompt bundle.
type Definition struct {
PromptID string
Version string
EmbeddedPath string
SystemPath string
UserPath string
}
type compiledPrompt struct {
// Bundle is a compiled system/user prompt pair.
type Bundle struct {
systemTmpl *template.Template
userTmpl *template.Template
metadata Metadata
}
var promptRegistry map[string]compiledPrompt
// Metadata returns metadata for the compiled prompt bundle.
func (b *Bundle) Metadata() Metadata {
if b == nil {
return Metadata{}
}
return b.metadata
}
var promptRegistry map[string]*Bundle
var sharedHardening string
func init() {
@@ -65,30 +75,23 @@ func init() {
panic(err)
}
defs := []definition{
defs := []Definition{
{
id: DNDSpellsPromptID,
version: VersionV1,
embeddedDir: "assets/dnd/spells",
systemPath: "assets/dnd/spells/system.md",
userPath: "assets/dnd/spells/user.md",
},
{
id: TestGenericPromptID,
version: VersionV1,
embeddedDir: "assets/test/generic",
systemPath: "assets/test/generic/system.md",
userPath: "assets/test/generic/user.md",
PromptID: TestGenericPromptID,
Version: VersionV1,
EmbeddedPath: "assets/test/generic",
SystemPath: "assets/test/generic/system.md",
UserPath: "assets/test/generic/user.md",
},
}
promptRegistry = make(map[string]compiledPrompt, len(defs))
promptRegistry = make(map[string]*Bundle, len(defs))
for _, def := range defs {
compiled, compileErr := compilePrompt(def)
compiled, compileErr := LoadBundle(embeddedAssets, def)
if compileErr != nil {
panic(compileErr)
}
promptRegistry[def.id] = compiled
promptRegistry[compiled.metadata.PromptID] = compiled
}
}
@@ -138,51 +141,68 @@ func readAsset(assetPath string) (string, error) {
return string(content), nil
}
func compilePrompt(def definition) (compiledPrompt, error) {
if strings.TrimSpace(def.id) == "" {
return compiledPrompt{}, fmt.Errorf("prompt id must not be empty")
// LoadBundle compiles a system/user prompt bundle from a caller-owned filesystem.
func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error) {
promptID := strings.TrimSpace(def.PromptID)
version := strings.TrimSpace(def.Version)
embeddedPath := strings.TrimSpace(def.EmbeddedPath)
systemPath := strings.TrimSpace(def.SystemPath)
userPath := strings.TrimSpace(def.UserPath)
if promptID == "" {
return nil, fmt.Errorf("prompt id must not be empty")
}
if strings.TrimSpace(def.version) == "" {
return compiledPrompt{}, fmt.Errorf("prompt version must not be empty")
if version == "" {
return nil, fmt.Errorf("prompt version must not be empty")
}
if strings.TrimSpace(def.embeddedDir) == "" {
return compiledPrompt{}, fmt.Errorf("prompt embedded path must not be empty")
if embeddedPath == "" {
return nil, fmt.Errorf("prompt embedded path must not be empty")
}
systemSource, err := readAsset(def.systemPath)
systemSource, err := readPromptAsset(fsys, systemPath)
if err != nil {
return compiledPrompt{}, err
return nil, err
}
userSource, err := readAsset(def.userPath)
userSource, err := readPromptAsset(fsys, userPath)
if err != nil {
return compiledPrompt{}, err
return nil, err
}
funcs := template.FuncMap{
"hardening": func() string { return sharedHardening },
}
systemTmpl, err := template.New(path.Base(def.systemPath)).Option("missingkey=error").Funcs(funcs).Parse(systemSource)
systemTmpl, err := template.New(path.Base(systemPath)).Option("missingkey=error").Funcs(funcs).Parse(systemSource)
if err != nil {
return compiledPrompt{}, fmt.Errorf("parse embedded system prompt %q: %w", def.systemPath, err)
return nil, fmt.Errorf("parse embedded system prompt %q: %w", systemPath, err)
}
userTmpl, err := template.New(path.Base(def.userPath)).Option("missingkey=error").Funcs(funcs).Parse(userSource)
userTmpl, err := template.New(path.Base(userPath)).Option("missingkey=error").Funcs(funcs).Parse(userSource)
if err != nil {
return compiledPrompt{}, fmt.Errorf("parse embedded user prompt %q: %w", def.userPath, err)
return nil, fmt.Errorf("parse embedded user prompt %q: %w", userPath, err)
}
hashInput := systemSource + "\n\n" + userSource
hash := sha256.Sum256([]byte(hashInput))
metadata := Metadata{
PromptID: strings.TrimSpace(def.id),
PromptVersion: strings.TrimSpace(def.version),
PromptID: promptID,
PromptVersion: version,
PromptSource: SourceBuiltin,
EmbeddedPath: strings.TrimSpace(def.embeddedDir),
EmbeddedPath: embeddedPath,
SHA256: "sha256:" + hex.EncodeToString(hash[:]),
}
return compiledPrompt{
return &Bundle{
systemTmpl: systemTmpl,
userTmpl: userTmpl,
metadata: metadata,
}, nil
}
func readPromptAsset(fsys fs.FS, assetPath string) (string, error) {
if strings.TrimSpace(assetPath) == "" {
return "", fmt.Errorf("prompt asset path must not be empty")
}
content, err := fs.ReadFile(fsys, assetPath)
if err != nil {
return "", fmt.Errorf("read embedded prompt asset %q: %w", assetPath, err)
}
return string(content), nil
}

View File

@@ -11,7 +11,6 @@ func TestLookupMetadataSucceedsForRegisteredPrompts(t *testing.T) {
promptID string
embeddedPath string
}{
{promptID: DNDSpellsPromptID, embeddedPath: "assets/dnd/spells"},
{promptID: TestGenericPromptID, embeddedPath: "assets/test/generic"},
}
@@ -59,8 +58,8 @@ func TestMustLookupMetadataPanicsForUnknownPromptID(t *testing.T) {
func TestRegisteredMetadataSortedByPromptID(t *testing.T) {
registered := RegisteredMetadata()
if len(registered) != 2 {
t.Fatalf("expected two registered prompts, got %d", len(registered))
if len(registered) != 1 {
t.Fatalf("expected one registered prompt, got %d", len(registered))
}
ids := make([]string, len(registered))
@@ -72,8 +71,8 @@ func TestRegisteredMetadataSortedByPromptID(t *testing.T) {
if !sort.StringsAreSorted(ids) {
t.Fatalf("expected sorted prompt IDs, got %v", ids)
}
if !seen[DNDSpellsPromptID] {
t.Fatalf("registered prompt IDs = %v, want %q", ids, DNDSpellsPromptID)
if !seen[TestGenericPromptID] {
t.Fatalf("registered prompt IDs = %v, want %q", ids, TestGenericPromptID)
}
}

View File

@@ -13,16 +13,23 @@ func RenderUserSystem(promptID string, data any) (system string, user string, me
if !ok {
return "", "", Metadata{}, fmt.Errorf("unknown prompt id %q", promptID)
}
return compiled.RenderUserSystem(data)
}
// RenderUserSystem renders the bundle's system and user prompts.
func (b *Bundle) RenderUserSystem(data any) (system string, user string, metadata Metadata, err error) {
if b == nil {
return "", "", Metadata{}, fmt.Errorf("prompt bundle must not be nil")
}
var systemBuf bytes.Buffer
if err := compiled.systemTmpl.Execute(&systemBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render system prompt %q: %w", trimmedID, err)
if err := b.systemTmpl.Execute(&systemBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render system prompt %q: %w", b.metadata.PromptID, err)
}
var userBuf bytes.Buffer
if err := compiled.userTmpl.Execute(&userBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render user prompt %q: %w", trimmedID, err)
if err := b.userTmpl.Execute(&userBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render user prompt %q: %w", b.metadata.PromptID, err)
}
return strings.TrimSpace(systemBuf.String()), strings.TrimSpace(userBuf.String()), compiled.metadata, nil
return strings.TrimSpace(systemBuf.String()), strings.TrimSpace(userBuf.String()), b.metadata, nil
}

View File

@@ -64,47 +64,3 @@ func TestRenderUserSystemIncludesHardeningText(t *testing.T) {
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
}
}
func TestRenderDNDSpellsPromptIncludesHardeningText(t *testing.T) {
system, user, metadata, err := RenderUserSystem(DNDSpellsPromptID, map[string]any{
"SourceID": "session-alpha",
"HasChunk": true,
"ChunkID": "session-alpha:chunk:0",
"ChunkIndex": 0,
"Units": []map[string]any{
{
"ID": "seg-001",
"Text": "Aria casts Cure Wounds.",
"Metadata": []map[string]string{
{"Key": "speaker", "Value": "Alice"},
},
},
},
})
if err != nil {
t.Fatalf("RenderUserSystem: %v", err)
}
hardening := strings.TrimSpace(HardeningText())
if hardening == "" {
t.Fatalf("expected hardening text")
}
if !strings.Contains(system, hardening) {
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
}
for _, want := range []string{"session-alpha", "session-alpha:chunk:0", "seg-001", "Aria casts Cure Wounds.", "speaker: Alice"} {
if !strings.Contains(user, want) {
t.Fatalf("rendered user prompt = %q, want substring %q", user, want)
}
}
if metadata.PromptID != DNDSpellsPromptID {
t.Fatalf("unexpected metadata: %+v", metadata)
}
}
func TestRenderDNDSpellsPromptMissingTemplateDataReturnsError(t *testing.T) {
_, _, _, err := RenderUserSystem(DNDSpellsPromptID, map[string]any{})
if err == nil || !strings.Contains(err.Error(), "SourceID") {
t.Fatalf("expected missing SourceID error, got %v", err)
}
}

View File

@@ -0,0 +1,241 @@
package generic
import (
"context"
"encoding/json"
"fmt"
"math"
"strconv"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "generic"
const (
defaultMaxUnits = 50
defaultOverlapUnits = 0
)
var _ contracts.Chunker = (*Chunker)(nil)
type Chunker struct{}
func New() *Chunker {
return &Chunker{}
}
func (c *Chunker) Key() string {
return Key
}
func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
if c == nil {
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
}
if ctx == nil {
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.ChunkResult{}, chunkerErrorf("context error before chunking: %w", err)
}
if req.Source == nil {
return contracts.ChunkResult{}, chunkerErrorf("source must not be nil")
}
if len(req.Source.Units) == 0 {
return contracts.ChunkResult{}, chunkerErrorf("source units must not be empty")
}
if err := source.ValidateDocument(req.Source); err != nil {
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
}
opts, err := chunkOptionsFrom(req.Options)
if err != nil {
return contracts.ChunkResult{}, err
}
step := opts.maxUnits - opts.overlapUnits
chunks := make([]contracts.SourceChunk, 0, (len(req.Source.Units)+step-1)/step)
for start := 0; start < len(req.Source.Units); start += step {
end := start + opts.maxUnits
if end > len(req.Source.Units) {
end = len(req.Source.Units)
}
units := cloneUnits(req.Source.Units[start:end])
chunks = append(chunks, contracts.SourceChunk{
ID: fmt.Sprintf("chunk-%06d", len(chunks)+1),
SourceID: req.Source.ID,
Index: len(chunks),
Units: units,
Metadata: map[string]any{
"start_unit_id": units[0].ID,
"end_unit_id": units[len(units)-1].ID,
"unit_count": len(units),
},
})
if end == len(req.Source.Units) {
break
}
}
return contracts.ChunkResult{Chunks: chunks}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
Provides: []string{"chunks"},
}
}
func Register(registry *pipeline.ChunkerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Chunker, error) {
return New(), nil
})
}
type chunkOptions struct {
maxUnits int
overlapUnits int
}
func chunkOptionsFrom(options map[string]any) (chunkOptions, error) {
opts := chunkOptions{
maxUnits: defaultMaxUnits,
overlapUnits: defaultOverlapUnits,
}
var err error
if value, ok := options["max_units"]; ok {
opts.maxUnits, err = positiveIntOption("max_units", value)
if err != nil {
return chunkOptions{}, err
}
}
if value, ok := options["overlap_units"]; ok {
opts.overlapUnits, err = nonNegativeIntOption("overlap_units", value)
if err != nil {
return chunkOptions{}, err
}
}
if opts.overlapUnits >= opts.maxUnits {
return chunkOptions{}, chunkerErrorf("overlap_units must be less than max_units")
}
return opts, nil
}
func positiveIntOption(name string, value any) (int, error) {
got, err := intOption(name, value)
if err != nil {
return 0, err
}
if got <= 0 {
return 0, chunkerErrorf("%s must be positive", name)
}
return got, nil
}
func nonNegativeIntOption(name string, value any) (int, error) {
got, err := intOption(name, value)
if err != nil {
return 0, err
}
if got < 0 {
return 0, chunkerErrorf("%s must be non-negative", name)
}
return got, nil
}
func intOption(name string, value any) (int, error) {
switch typed := value.(type) {
case int:
return typed, nil
case int8:
return int(typed), nil
case int16:
return int(typed), nil
case int32:
return int(typed), nil
case int64:
if typed > maxInt() || typed < minInt() {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case uint:
if uint64(typed) > uint64(maxInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case uint8:
return int(typed), nil
case uint16:
return int(typed), nil
case uint32:
if uint64(typed) > uint64(maxInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case uint64:
if typed > uint64(maxInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case float64:
if typed != math.Trunc(typed) {
return 0, chunkerErrorf("%s must be an integer", name)
}
if typed > float64(maxInt()) || typed < float64(minInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case json.Number:
parsed, err := typed.Int64()
if err != nil {
return 0, chunkerErrorf("%s must be an integer", name)
}
if parsed > maxInt() || parsed < minInt() {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(parsed), nil
default:
return 0, chunkerErrorf("%s must be an integer", name)
}
}
func maxInt() int64 {
return int64(1<<(strconv.IntSize-1) - 1)
}
func minInt() int64 {
return -maxInt() - 1
}
func cloneUnits(units []source.SourceUnit) []source.SourceUnit {
out := make([]source.SourceUnit, 0, len(units))
for _, unit := range units {
out = append(out, source.SourceUnit{
ID: unit.ID,
Kind: unit.Kind,
Text: unit.Text,
Metadata: cloneMetadata(unit.Metadata),
})
}
return out
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func chunkerErrorf(format string, args ...any) error {
return fmt.Errorf("generic chunker: "+format, args...)
}

View File

@@ -0,0 +1,213 @@
package generic
import (
"context"
"encoding/json"
"fmt"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
Provides: []string{"chunks"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewChunkerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
chunker, err := registry.Build(Key)
if err != nil {
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
}
if chunker.Key() != Key {
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
}
}
func TestChunkUsesDefaultsForSingleChunk(t *testing.T) {
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{Source: testSource(3), Options: nil})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"chunk-000001"}) {
t.Fatalf("chunk IDs = %#v, want one stable ID", got)
}
chunk := result.Chunks[0]
if chunk.Index != 0 || chunk.SourceID != "source-1" {
t.Fatalf("chunk = %#v, want source and index fields", chunk)
}
if got := unitIDs(chunk.Units); !reflect.DeepEqual(got, []string{"u001", "u002", "u003"}) {
t.Fatalf("unit IDs = %#v, want all units", got)
}
if chunk.Metadata["start_unit_id"] != "u001" || chunk.Metadata["end_unit_id"] != "u003" || chunk.Metadata["unit_count"] != 3 {
t.Fatalf("metadata = %#v, want chunk bounds", chunk.Metadata)
}
}
func TestChunkExactBoundaries(t *testing.T) {
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: testSource(6),
Options: map[string]any{"max_units": 2},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"chunk-000001", "chunk-000002", "chunk-000003"}) {
t.Fatalf("chunk IDs = %#v, want stable IDs", got)
}
gotUnits := [][]string{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units), unitIDs(result.Chunks[2].Units)}
wantUnits := [][]string{{"u001", "u002"}, {"u003", "u004"}, {"u005", "u006"}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
}
func TestChunkOverlap(t *testing.T) {
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: testSource(7),
Options: map[string]any{"max_units": 3, "overlap_units": 1},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
gotUnits := make([][]string, 0, len(result.Chunks))
for _, chunk := range result.Chunks {
gotUnits = append(gotUnits, unitIDs(chunk.Units))
}
wantUnits := [][]string{{"u001", "u002", "u003"}, {"u003", "u004", "u005"}, {"u005", "u006", "u007"}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
}
func TestChunkRejectsInvalidOptions(t *testing.T) {
tests := []struct {
name string
options map[string]any
want string
}{
{name: "max wrong type", options: map[string]any{"max_units": "2"}, want: "max_units"},
{name: "max fractional", options: map[string]any{"max_units": 1.5}, want: "integer"},
{name: "max zero", options: map[string]any{"max_units": 0}, want: "positive"},
{name: "overlap negative", options: map[string]any{"overlap_units": -1}, want: "non-negative"},
{name: "overlap too large", options: map[string]any{"max_units": 2, "overlap_units": 2}, want: "less than"},
{name: "json number", options: map[string]any{"max_units": json.Number("bad")}, want: "integer"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: testSource(3),
Options: test.options,
})
if err == nil {
t.Fatal("Chunk() error = nil, want error")
}
if !strings.Contains(err.Error(), "generic chunker") || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Chunk() error = %q, want module context and %q", err.Error(), test.want)
}
})
}
}
func TestChunkRejectsEmptySource(t *testing.T) {
doc := testSource(1)
doc.Units = nil
_, err := New().Chunk(context.Background(), contracts.ChunkRequest{Source: doc})
if err == nil {
t.Fatal("Chunk() error = nil, want empty source error")
}
if !strings.Contains(err.Error(), "generic chunker") || !strings.Contains(err.Error(), "units") {
t.Fatalf("Chunk() error = %q, want empty source context", err.Error())
}
}
func TestChunkDefensivelyCopiesUnits(t *testing.T) {
doc := testSource(2)
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: doc,
Options: map[string]any{"max_units": 1},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if len(result.Chunks) != 2 {
t.Fatalf("len(Chunks) = %d, want 2", len(result.Chunks))
}
doc.Units[0].ID = "changed"
doc.Units[0].Metadata["speaker"] = "changed"
if result.Chunks[0].Units[0].ID != "u001" {
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
}
if result.Chunks[0].Units[0].Metadata["speaker"] != "speaker-001" {
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
}
}
func testSource(count int) *source.SourceDocument {
units := make([]source.SourceUnit, 0, count)
for i := 1; i <= count; i++ {
id := "u" + zeroPad3(i)
units = append(units, source.SourceUnit{
ID: id,
Kind: "unit",
Text: "Text for " + id,
Metadata: map[string]any{
"speaker": "speaker-" + zeroPad3(i),
},
})
}
return &source.SourceDocument{
ID: "source-1",
Kind: "document",
Format: "text/plain",
Digest: "sha256:source",
Units: units,
}
}
func zeroPad3(value int) string {
return fmt.Sprintf("%03d", value)
}
func chunkIDs(chunks []contracts.SourceChunk) []string {
ids := make([]string, 0, len(chunks))
for _, chunk := range chunks {
ids = append(ids, chunk.ID)
}
return ids
}
func unitIDs(units []source.SourceUnit) []string {
ids := make([]string, 0, len(units))
for _, unit := range units {
ids = append(ids, unit.ID)
}
return ids
}

View File

@@ -0,0 +1,6 @@
package spells
import "embed"
//go:embed assets/prompts/*.md assets/schemas/*.json
var embeddedAssets embed.FS

View File

@@ -11,6 +11,8 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
)
func TestPipelineConfigLoadsAndResolvesWithDNDSpellsExtractor(t *testing.T) {
@@ -188,7 +190,7 @@ func dndSpellsTestCatalog(t *testing.T, specs dndSpellsCatalogSpecs) pipeline.Mo
Stage: pipeline.StageMerge,
Requires: []string{"dnd.spell_casts"},
}, func() (contracts.Merger, error) {
return pipeline.AppendOrderMerger{}, nil
return appendorder.New(), nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
@@ -196,7 +198,7 @@ func dndSpellsTestCatalog(t *testing.T, specs dndSpellsCatalogSpecs) pipeline.Mo
Key: pipeline.DefaultNormalizeModule,
Stage: pipeline.StageNormalize,
}, func() (contracts.Normalizer, error) {
return pipeline.NoopNormalizer{}, nil
return noop.New(), nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}

View File

@@ -9,7 +9,6 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -46,6 +45,23 @@ func (e *Extractor) SchemaVersion() string {
return SchemaVersion
}
func (e *Extractor) ManifestMetadata() map[string]any {
promptMetadata := spellsPromptBundle.Metadata()
metadata := map[string]any{
"prompt_id": PromptID,
"prompt_version": promptMetadata.PromptVersion,
"prompt_sha256": promptMetadata.SHA256,
"response_schema_key": string(ResponseSchemaKey),
"response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName,
}
if schema, err := loadResponseSchema(); err == nil {
metadata["response_schema_version"] = schema.Version
metadata["response_schema_sha256"] = schema.SHA256
}
return metadata
}
func (e *Extractor) Validators() []contracts.Validator {
return []contracts.Validator{
ShapeValidator{},
@@ -80,9 +96,9 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest
if err != nil {
return contracts.ExtractionResult{}, extractorErrorf("render prompt: %w", err)
}
schema, ok := llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)
if !ok {
return contracts.ExtractionResult{}, extractorErrorf("lookup response schema %q", llm.DNDSpellsSchemaKey)
schema, err := loadResponseSchema()
if err != nil {
return contracts.ExtractionResult{}, extractorErrorf("load response schema %q: %w", ResponseSchemaKey, err)
}
var response extractionResponse

View File

@@ -10,7 +10,6 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
@@ -42,7 +41,10 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
if req.StageName != Key {
t.Fatalf("StageName = %q, want %q", req.StageName, Key)
}
schema := llm.MustLookupResponseSchema(llm.DNDSpellsSchemaKey)
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if req.ResponseSchemaName != schema.Name {
t.Fatalf("ResponseSchemaName = %q, want %q", req.ResponseSchemaName, schema.Name)
}
@@ -90,6 +92,30 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
}
}
func TestExtractorManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
metadata := New().ManifestMetadata()
tests := map[string]string{
"prompt_id": PromptID,
"prompt_version": SchemaVersion,
"response_schema_key": string(ResponseSchemaKey),
"response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName,
"response_schema_version": SchemaVersion,
}
for key, want := range tests {
if metadata[key] != want {
t.Fatalf("metadata[%q] = %#v, want %q", key, metadata[key], want)
}
}
for _, key := range []string{"prompt_sha256", "response_schema_sha256"} {
value, ok := metadata[key].(string)
if !ok || !strings.HasPrefix(value, "sha256:") {
t.Fatalf("metadata[%q] = %#v, want sha256 value", key, metadata[key])
}
}
}
func TestExtractReturnsNoCandidatesForEmptyResponse(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}

View File

@@ -28,6 +28,22 @@ type promptMetadata struct {
Value string
}
var spellsPromptBundle = mustLoadPromptBundle()
func mustLoadPromptBundle() *prompt.Bundle {
bundle, err := prompt.LoadBundle(embeddedAssets, prompt.Definition{
PromptID: PromptID,
Version: SchemaVersion,
EmbeddedPath: "assets/prompts",
SystemPath: "assets/prompts/system.md",
UserPath: "assets/prompts/user.md",
})
if err != nil {
panic(err)
}
return bundle
}
func buildPromptData(req contracts.ExtractionRequest) (promptData, error) {
if req.Source == nil {
return promptData{}, fmt.Errorf("dnd spells prompt: source must not be nil")
@@ -58,7 +74,7 @@ func renderPrompt(req contracts.ExtractionRequest) (system string, user string,
if err != nil {
return "", "", prompt.Metadata{}, err
}
system, user, metadata, err = prompt.RenderUserSystem(prompt.DNDSpellsPromptID, data)
system, user, metadata, err = spellsPromptBundle.RenderUserSystem(data)
if err != nil {
return "", "", prompt.Metadata{}, fmt.Errorf("dnd spells prompt: %w", err)
}

View File

@@ -92,8 +92,14 @@ func TestRenderPromptIncludesSourceContext(t *testing.T) {
t.Fatalf("user prompt = %q, want substring %q", user, want)
}
}
if metadata.PromptID != prompt.DNDSpellsPromptID {
t.Fatalf("metadata.PromptID = %q, want %q", metadata.PromptID, prompt.DNDSpellsPromptID)
if metadata.PromptID != PromptID {
t.Fatalf("metadata.PromptID = %q, want %q", metadata.PromptID, PromptID)
}
if metadata.PromptVersion != SchemaVersion {
t.Fatalf("metadata.PromptVersion = %q, want %q", metadata.PromptVersion, SchemaVersion)
}
if metadata.EmbeddedPath != "assets/prompts" {
t.Fatalf("metadata.EmbeddedPath = %q, want assets/prompts", metadata.EmbeddedPath)
}
}

View File

@@ -93,6 +93,15 @@ func TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor(t *testing.T) {
if lane.ID != "spells" || lane.Extractor != Key {
t.Fatalf("manifest lane = %#v, want spells lane with dnd/spells extractor", lane)
}
extractorMetadata, ok := lane.Metadata["extractor"].(map[string]any)
if !ok {
t.Fatalf("manifest lane metadata = %#v, want extractor metadata", lane.Metadata)
}
if extractorMetadata["prompt_id"] != PromptID ||
extractorMetadata["response_schema_key"] != string(ResponseSchemaKey) ||
extractorMetadata["response_schema_name"] != ResponseSchemaName {
t.Fatalf("extractor metadata = %#v, want prompt/schema identifiers", extractorMetadata)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
}

View File

@@ -0,0 +1,20 @@
package spells
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
const (
PromptID = "dnd.spells"
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_spells")
ResponseSchemaID = "notarius.dnd.spells"
ResponseSchemaName = "notarius_dnd_spells_v1"
)
func loadResponseSchema() (llm.ResponseSchema, error) {
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
Key: ResponseSchemaKey,
ID: ResponseSchemaID,
Version: SchemaVersion,
Name: ResponseSchemaName,
AssetPath: "assets/schemas/dnd_spells.v1.json",
})
}

View File

@@ -4,23 +4,24 @@ import (
"encoding/json"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestLookupResponseSchemaForSpells(t *testing.T) {
schema, ok := llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)
if !ok {
t.Fatalf("LookupResponseSchema(%q) ok = false, want true", llm.DNDSpellsSchemaKey)
func TestLoadResponseSchemaForSpells(t *testing.T) {
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if schema.ID != "notarius.dnd.spells" {
t.Fatalf("schema.ID = %q, want notarius.dnd.spells", schema.ID)
if schema.Key != ResponseSchemaKey {
t.Fatalf("schema.Key = %q, want %q", schema.Key, ResponseSchemaKey)
}
if schema.ID != ResponseSchemaID {
t.Fatalf("schema.ID = %q, want %q", schema.ID, ResponseSchemaID)
}
if schema.Version != SchemaVersion {
t.Fatalf("schema.Version = %q, want %q", schema.Version, SchemaVersion)
}
if schema.Name != "notarius_dnd_spells_v1" {
t.Fatalf("schema.Name = %q, want notarius_dnd_spells_v1", schema.Name)
if schema.Name != ResponseSchemaName {
t.Fatalf("schema.Name = %q, want %q", schema.Name, ResponseSchemaName)
}
if !strings.HasPrefix(schema.SHA256, "sha256:") {
t.Fatalf("schema.SHA256 = %q, want sha256 prefix", schema.SHA256)
@@ -30,12 +31,34 @@ func TestLookupResponseSchemaForSpells(t *testing.T) {
}
}
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
first, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
first.JSONSchema[0] = '['
second, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if !json.Valid(second.JSONSchema) {
t.Fatalf("schema JSON was mutated: %s", second.JSONSchema)
}
if len(second.JSONSchema) > 0 && second.JSONSchema[0] == '[' {
t.Fatalf("schema JSON did not use defensive copy")
}
}
func TestResponseSchemaDiagnosticsOmitRawSchema(t *testing.T) {
schema := llm.MustLookupResponseSchema(llm.DNDSpellsSchemaKey)
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
diagnostics := schema.DiagnosticsMap()
if diagnostics["key"] != llm.DNDSpellsSchemaKey {
t.Fatalf("diagnostics[key] = %#v, want %q", diagnostics["key"], llm.DNDSpellsSchemaKey)
if diagnostics["key"] != ResponseSchemaKey {
t.Fatalf("diagnostics[key] = %#v, want %q", diagnostics["key"], ResponseSchemaKey)
}
for _, key := range []string{"id", "version", "name", "sha256"} {
if diagnostics[key] == "" {
@@ -45,4 +68,7 @@ func TestResponseSchemaDiagnosticsOmitRawSchema(t *testing.T) {
if _, ok := diagnostics["json_schema"]; ok {
t.Fatalf("diagnostics should omit raw schema content: %#v", diagnostics)
}
if _, ok := diagnostics["JSONSchema"]; ok {
t.Fatalf("diagnostics should omit raw schema content: %#v", diagnostics)
}
}

View File

@@ -10,6 +10,8 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
)
func TestPipelineConfigLoadsAndResolvesWithSeriatimInput(t *testing.T) {
@@ -179,7 +181,7 @@ func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, s
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) {
return pipeline.AppendOrderMerger{}, nil
return appendorder.New(), nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
@@ -188,7 +190,7 @@ func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pi
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) {
return pipeline.NoopNormalizer{}, nil
return noop.New(), nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}

View File

@@ -12,6 +12,8 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
)
func TestRunnerProcessesSeriatimInputWithFakeModules(t *testing.T) {
@@ -121,12 +123,12 @@ func seriatimRunnerRegistries(t *testing.T, extractor contracts.Extractor) pipel
t.Fatalf("register extractor: %v", err)
}
if err := mergers.Register(pipeline.DefaultMergeModule, func() (contracts.Merger, error) {
return pipeline.AppendOrderMerger{}, nil
return appendorder.New(), nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
if err := normalizers.Register(pipeline.DefaultNormalizeModule, func() (contracts.Normalizer, error) {
return pipeline.NoopNormalizer{}, nil
return noop.New(), nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}

View File

@@ -0,0 +1,97 @@
package appendorder
import (
"context"
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "appendorder"
var _ contracts.Merger = (*Merger)(nil)
type Merger struct{}
func New() *Merger {
return &Merger{}
}
func (m *Merger) Key() string {
return Key
}
func (m *Merger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
if m == nil {
return contracts.MergeResult{}, mergerErrorf("merger must not be nil")
}
if ctx == nil {
return contracts.MergeResult{}, mergerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.MergeResult{}, mergerErrorf("context error before merge: %w", err)
}
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, cloneCandidates(chunkArtifacts.Candidates)...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageMerge,
Provides: []string{"merged"},
}
}
func Register(registry *pipeline.MergerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Merger, error) {
return New(), nil
})
}
func cloneCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
}
out := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
out = append(out, cloneCandidate(candidate))
}
return out
}
func cloneCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
}
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func mergerErrorf(format string, args ...any) error {
return fmt.Errorf("appendorder merger: "+format, args...)
}

View File

@@ -0,0 +1,147 @@
package appendorder
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageMerge,
Provides: []string{"merged"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewMergerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
}
func TestMergePreservesChunkAndCandidateOrder(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{
ChunkArtifacts: []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{candidate(2, "first-b"), candidate(1, "first-a")},
},
{
Chunk: sourceChunk(1),
Candidates: []artifacts.ArtifactCandidate{candidate(4, "second-b"), candidate(3, "second-a")},
},
},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
got := candidateNames(result.Candidates)
want := []string{"first-b", "first-a", "second-b", "second-a"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func TestMergeDefensivelyCopiesCandidates(t *testing.T) {
input := []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{candidate(1, "original")},
},
}
result, err := New().Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: input})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Candidates[0].Index = 99
input[0].Candidates[0].Payload[0] = '['
input[0].Candidates[0].SourceRefs[0].StartUnitID = "changed"
input[0].Candidates[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].StartUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestMergeHandlesEmptyInput(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("len(Candidates) = %d, want 0", len(result.Candidates))
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}
func sourceChunk(index int) contracts.SourceChunk {
return contracts.SourceChunk{
ID: "chunk",
SourceID: "source-1",
Index: index,
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
},
}
}

View File

@@ -0,0 +1,89 @@
package noop
import (
"context"
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "noop"
var _ contracts.Normalizer = (*Normalizer)(nil)
type Normalizer struct{}
func New() *Normalizer {
return &Normalizer{}
}
func (n *Normalizer) Key() string {
return Key
}
func (n *Normalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
if n == nil {
return contracts.NormalizeResult{}, normalizerErrorf("normalizer must not be nil")
}
if ctx == nil {
return contracts.NormalizeResult{}, normalizerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.NormalizeResult{}, normalizerErrorf("context error before normalize: %w", err)
}
return contracts.NormalizeResult{Candidates: cloneCandidates(req.Candidates)}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
}
}
func Register(registry *pipeline.NormalizerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Normalizer, error) {
return New(), nil
})
}
func cloneCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
}
out := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
out = append(out, artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
})
}
return out
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func normalizerErrorf(format string, args ...any) error {
return fmt.Errorf("noop normalizer: "+format, args...)
}

View File

@@ -0,0 +1,133 @@
package noop
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewNormalizerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
}
func TestNormalizePassesThroughOrderAndValues(t *testing.T) {
input := []artifacts.ArtifactCandidate{
candidate(3, "third"),
candidate(1, "first"),
candidate(2, "second"),
}
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"third", "first", "second"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
if !reflect.DeepEqual(result.Candidates[0].SourceRefs, input[0].SourceRefs) {
t.Fatalf("SourceRefs = %#v, want %#v", result.Candidates[0].SourceRefs, input[0].SourceRefs)
}
if !reflect.DeepEqual(result.Candidates[0].Metadata, input[0].Metadata) {
t.Fatalf("Metadata = %#v, want %#v", result.Candidates[0].Metadata, input[0].Metadata)
}
}
func TestNormalizeDefensivelyCopiesCandidates(t *testing.T) {
input := []artifacts.ArtifactCandidate{candidate(1, "original")}
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Index = 99
input[0].Payload[0] = '['
input[0].SourceRefs[0].EndUnitID = "changed"
input[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].EndUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestNormalizeHandlesEmptyInput(t *testing.T) {
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("len(Candidates) = %d, want 0", len(result.Candidates))
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}

View File

@@ -0,0 +1,253 @@
package json
import (
"context"
stdjson "encoding/json"
"fmt"
"regexp"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "json"
const contentTypeJSON = "application/json"
var safeArtifactFileChar = regexp.MustCompile(`[^A-Za-z0-9._-]`)
var _ contracts.OutputEncoder = (*Encoder)(nil)
type Encoder struct{}
func New() *Encoder {
return &Encoder{}
}
func (e *Encoder) Key() string {
return Key
}
func (e *Encoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
if e == nil {
return contracts.OutputResult{}, encoderErrorf("encoder must not be nil")
}
if ctx == nil {
return contracts.OutputResult{}, encoderErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.OutputResult{}, encoderErrorf("context error before encoding: %w", err)
}
files, err := logicalFiles(req)
if err != nil {
return contracts.OutputResult{}, err
}
return contracts.OutputResult{Files: files}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageOutput,
Requires: []string{"normalized"},
Provides: []string{"encoded"},
}
}
func Register(registry *pipeline.OutputEncoderRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.OutputEncoder, error) {
return New(), nil
})
}
type indexFile struct {
ManifestFile string `json:"manifest_file"`
ArtifactFiles []artifactFileIndex `json:"artifact_files"`
RejectedFile string `json:"rejected_file"`
WarningsFile string `json:"warnings_file"`
}
type artifactFileIndex struct {
ArtifactType string `json:"artifact_type"`
File string `json:"file"`
}
type artifactFile struct {
ArtifactType string `json:"artifact_type"`
Artifacts []artifacts.Artifact `json:"artifacts"`
}
type rejectedFile struct {
Rejected []artifacts.RejectedArtifact `json:"rejected"`
}
type warningsFile struct {
Warnings []contracts.Warning `json:"warnings"`
}
func logicalFiles(req contracts.OutputRequest) ([]contracts.OutputFile, error) {
artifactsByType := make(map[string][]artifacts.Artifact)
for _, artifact := range req.Approved {
artifactsByType[artifact.ArtifactType] = append(artifactsByType[artifact.ArtifactType], cloneArtifact(artifact))
}
artifactTypes := make([]string, 0, len(artifactsByType))
for artifactType := range artifactsByType {
artifactTypes = append(artifactTypes, artifactType)
}
sort.Strings(artifactTypes)
artifactIndexes := make([]artifactFileIndex, 0, len(artifactTypes))
files := make([]contracts.OutputFile, 0, len(artifactTypes)+4)
manifestFile, err := jsonFile("manifest.json", req.Manifest)
if err != nil {
return nil, err
}
files = append(files, manifestFile)
usedArtifactFiles := make(map[string]string, len(artifactTypes))
for _, artifactType := range artifactTypes {
name, err := artifactFileName(artifactType)
if err != nil {
return nil, err
}
if existingType, ok := usedArtifactFiles[name]; ok {
return nil, encoderErrorf("artifact types %q and %q produce duplicate output file %q", existingType, artifactType, name)
}
usedArtifactFiles[name] = artifactType
artifactIndexes = append(artifactIndexes, artifactFileIndex{
ArtifactType: artifactType,
File: name,
})
file, err := jsonFile(name, artifactFile{
ArtifactType: artifactType,
Artifacts: artifactsByType[artifactType],
})
if err != nil {
return nil, err
}
files = append(files, file)
}
index := indexFile{
ManifestFile: "manifest.json",
ArtifactFiles: artifactIndexes,
RejectedFile: "rejected.json",
WarningsFile: "warnings.json",
}
indexOutput, err := jsonFile("index.json", index)
if err != nil {
return nil, err
}
rejectedOutput, err := jsonFile("rejected.json", rejectedFile{Rejected: cloneRejected(req.Rejected)})
if err != nil {
return nil, err
}
warningsOutput, err := jsonFile("warnings.json", warningsFile{Warnings: cloneWarnings(req.Warnings)})
if err != nil {
return nil, err
}
files = append(files, indexOutput, rejectedOutput, warningsOutput)
sort.Slice(files, func(i, j int) bool {
return files[i].Name < files[j].Name
})
return files, nil
}
func jsonFile(name string, value any) (contracts.OutputFile, error) {
data, err := marshalPretty(value)
if err != nil {
return contracts.OutputFile{}, encoderErrorf("encode %s: %w", name, err)
}
return contracts.OutputFile{
Name: name,
ContentType: contentTypeJSON,
Bytes: data,
}, nil
}
func marshalPretty(value any) ([]byte, error) {
data, err := stdjson.MarshalIndent(value, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
}
func artifactFileName(artifactType string) (string, error) {
sanitized := safeArtifactFileChar.ReplaceAllString(strings.TrimSpace(artifactType), "_")
for strings.Contains(sanitized, "..") {
sanitized = strings.ReplaceAll(sanitized, "..", "__")
}
sanitized = strings.Trim(sanitized, "._")
if sanitized == "" {
return "", encoderErrorf("artifact type %q cannot produce a safe file name", artifactType)
}
return "artifacts/" + sanitized + ".json", nil
}
func cloneArtifact(artifact artifacts.Artifact) artifacts.Artifact {
return artifacts.Artifact{
ExtractorKey: artifact.ExtractorKey,
ArtifactType: artifact.ArtifactType,
SchemaVersion: artifact.SchemaVersion,
Payload: append(stdjson.RawMessage(nil), artifact.Payload...),
SourceRefs: append([]source.SourceRef(nil), artifact.SourceRefs...),
Metadata: cloneMetadata(artifact.Metadata),
}
}
func cloneRejected(rejected []artifacts.RejectedArtifact) []artifacts.RejectedArtifact {
if len(rejected) == 0 {
return []artifacts.RejectedArtifact{}
}
out := make([]artifacts.RejectedArtifact, 0, len(rejected))
for _, item := range rejected {
out = append(out, artifacts.RejectedArtifact{
Candidate: cloneCandidate(item.Candidate),
ValidatorName: item.ValidatorName,
ReasonCode: item.ReasonCode,
Message: item.Message,
})
}
return out
}
func cloneCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(stdjson.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
}
}
func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
if len(warnings) == 0 {
return []contracts.Warning{}
}
return append([]contracts.Warning(nil), warnings...)
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func encoderErrorf(format string, args ...any) error {
return fmt.Errorf("json output encoder: "+format, args...)
}

View File

@@ -0,0 +1,316 @@
package json
import (
"context"
stdjson "encoding/json"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageOutput,
Requires: []string{"normalized"},
Provides: []string{"encoded"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewOutputEncoderRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
}
func TestEncodeReturnsLogicalFilesGroupedByArtifactType(t *testing.T) {
req := contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1", PipelineID: "pipeline-1"},
Approved: []artifacts.Artifact{
artifact("dnd.spell-cast", "first"),
artifact("notes/item", "item"),
artifact("dnd.spell-cast", "second"),
},
Rejected: []artifacts.RejectedArtifact{
{
Candidate: candidate("bad type", "bad"),
ValidatorName: "validator",
ReasonCode: "invalid",
Message: "not accepted",
},
},
Warnings: []contracts.Warning{{ReasonCode: "warning", Message: "check source"}},
}
result, err := New().Encode(context.Background(), req)
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
wantNames := []string{
"artifacts/dnd.spell-cast.json",
"artifacts/notes_item.json",
"index.json",
"manifest.json",
"rejected.json",
"warnings.json",
}
if got := outputFileNames(result.Files); !reflect.DeepEqual(got, wantNames) {
t.Fatalf("file names = %#v, want %#v", got, wantNames)
}
for _, file := range result.Files {
if file.ContentType != contentTypeJSON {
t.Fatalf("%s ContentType = %q, want %q", file.Name, file.ContentType, contentTypeJSON)
}
if !strings.HasSuffix(string(file.Bytes), "\n") {
t.Fatalf("%s does not end with newline: %q", file.Name, string(file.Bytes))
}
if !stdjson.Valid(file.Bytes) {
t.Fatalf("%s has invalid JSON: %s", file.Name, file.Bytes)
}
}
spellFile := decodeObject(t, fileBytes(t, result.Files, "artifacts/dnd.spell-cast.json"))
if spellFile["artifact_type"] != "dnd.spell-cast" {
t.Fatalf("artifact_type = %#v, want dnd.spell-cast", spellFile["artifact_type"])
}
spells := spellFile["artifacts"].([]any)
if len(spells) != 2 {
t.Fatalf("len(spells) = %d, want 2", len(spells))
}
firstPayload := spells[0].(map[string]any)["payload"].(map[string]any)
secondPayload := spells[1].(map[string]any)["payload"].(map[string]any)
if firstPayload["name"] != "first" || secondPayload["name"] != "second" {
t.Fatalf("spell order payloads = %#v then %#v, want runner order", firstPayload, secondPayload)
}
index := decodeObject(t, fileBytes(t, result.Files, "index.json"))
artifactFiles := index["artifact_files"].([]any)
if len(artifactFiles) != 2 {
t.Fatalf("len(index artifact_files) = %d, want 2", len(artifactFiles))
}
firstIndex := artifactFiles[0].(map[string]any)
secondIndex := artifactFiles[1].(map[string]any)
if firstIndex["artifact_type"] != "dnd.spell-cast" || secondIndex["artifact_type"] != "notes/item" {
t.Fatalf("artifact_files = %#v, want sorted by artifact type", artifactFiles)
}
}
func TestEncodeIncludesRejectedAndWarningsWhenEmpty(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
rejected := decodeObject(t, fileBytes(t, result.Files, "rejected.json"))
if got := rejected["rejected"].([]any); len(got) != 0 {
t.Fatalf("rejected = %#v, want empty array", got)
}
warnings := decodeObject(t, fileBytes(t, result.Files, "warnings.json"))
if got := warnings["warnings"].([]any); len(got) != 0 {
t.Fatalf("warnings = %#v, want empty array", got)
}
}
func TestEncodePrettyPrintsJSON(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
manifest := string(fileBytes(t, result.Files, "manifest.json"))
if !strings.Contains(manifest, "\n \"run_id\": \"run-1\"\n") {
t.Fatalf("manifest JSON = %q, want two-space indentation", manifest)
}
}
func TestEncodeRejectsArtifactTypeWithoutSafeFileName(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("///", "unsafe")},
})
if err == nil {
t.Fatal("Encode() error = nil, want unsafe artifact type error")
}
if !strings.Contains(err.Error(), "json output encoder") || !strings.Contains(err.Error(), "safe file name") {
t.Fatalf("Encode() error = %q, want safe file name context", err.Error())
}
}
func TestEncodeSanitizesParentPathSequences(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("dnd..spell.", "spell")},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
if got := outputFileNames(result.Files); !containsString(got, "artifacts/dnd__spell.json") {
t.Fatalf("file names = %#v, want sanitized artifact filename", got)
}
}
func TestEncodeRejectsSanitizedFilenameCollisions(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{
artifact("a/b", "slash"),
artifact("a?b", "question"),
},
})
if err == nil {
t.Fatal("Encode() error = nil, want duplicate file error")
}
if !strings.Contains(err.Error(), "duplicate output file") {
t.Fatalf("Encode() error = %q, want duplicate file context", err.Error())
}
}
func TestEncodeDoesNotMutateInputs(t *testing.T) {
req := contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
Approved: []artifacts.Artifact{
artifact("dnd.spell", "original"),
},
Rejected: []artifacts.RejectedArtifact{
{
Candidate: candidate("bad", "rejected"),
ValidatorName: "validator",
ReasonCode: "invalid",
Message: "not accepted",
},
},
Warnings: []contracts.Warning{{ReasonCode: "warning", Message: "message"}},
}
before := mustMarshal(t, req)
result, err := New().Encode(context.Background(), req)
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
after := mustMarshal(t, req)
if before != after {
t.Fatalf("request mutated:\nbefore: %s\nafter: %s", before, after)
}
req.Approved[0].Payload[0] = '['
req.Approved[0].SourceRefs[0].StartUnitID = "changed"
req.Approved[0].Metadata["name"] = "changed"
req.Rejected[0].Candidate.Payload[0] = '['
req.Warnings[0].Message = "changed"
if !stdjson.Valid(fileBytes(t, result.Files, "artifacts/dnd.spell.json")) {
t.Fatal("artifact output changed after request mutation")
}
warnings := decodeObject(t, fileBytes(t, result.Files, "warnings.json"))
gotWarnings := warnings["warnings"].([]any)
if gotWarnings[0].(map[string]any)["message"] != "message" {
t.Fatalf("warnings output changed after request mutation: %#v", gotWarnings)
}
}
func TestArtifactFilesDoNotContainWarnings(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("dnd.spell", "spell")},
Warnings: []contracts.Warning{
{ReasonCode: "pipeline-warning", Message: "warning"},
},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
artifactFile := decodeObject(t, fileBytes(t, result.Files, "artifacts/dnd.spell.json"))
if _, ok := artifactFile["warnings"]; ok {
t.Fatalf("artifact file contains warnings: %#v", artifactFile)
}
}
func artifact(artifactType, name string) artifacts.Artifact {
return artifacts.Artifact{
ExtractorKey: "extractor",
ArtifactType: artifactType,
SchemaVersion: "v1",
Payload: stdjson.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{"name": name},
}
}
func candidate(artifactType, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: 1,
ExtractorKey: "extractor",
ArtifactType: artifactType,
SchemaVersion: "v1",
Payload: stdjson.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{"name": name},
}
}
func outputFileNames(files []contracts.OutputFile) []string {
names := make([]string, 0, len(files))
for _, file := range files {
names = append(names, file.Name)
}
return names
}
func containsString(values []string, want string) bool {
for _, value := range values {
if value == want {
return true
}
}
return false
}
func fileBytes(t *testing.T, files []contracts.OutputFile, name string) []byte {
t.Helper()
for _, file := range files {
if file.Name == name {
return file.Bytes
}
}
t.Fatalf("file %q not found in %#v", name, outputFileNames(files))
return nil
}
func decodeObject(t *testing.T, data []byte) map[string]any {
t.Helper()
var got map[string]any
if err := stdjson.Unmarshal(data, &got); err != nil {
t.Fatalf("Unmarshal() error = %v, want nil\n%s", err, data)
}
return got
}
func mustMarshal(t *testing.T, value any) string {
t.Helper()
data, err := stdjson.Marshal(value)
if err != nil {
t.Fatalf("Marshal() error = %v, want nil", err)
}
return string(data)
}