Compare commits

...

4 Commits

Author SHA1 Message Date
1da29e6788 Document subprocess run results 2026-07-26 17:22:27 +00:00
0d947549fb Test JSON run result behavior 2026-07-26 17:17:35 +00:00
950fba17ce Add JSON run result mode 2026-07-26 17:14:16 +00:00
678d2c6099 Add run result wire model 2026-07-26 17:12:57 +00:00
13 changed files with 703 additions and 13 deletions

View File

@@ -36,6 +36,8 @@ demonstrates all implemented D&D lanes and the supporting campaign references.
handling. handling.
- [Integration contracts](docs/integrations/) — Seriatim input and published - [Integration contracts](docs/integrations/) — Seriatim input and published
artifact formats. artifact formats.
- [Subprocess consumer guide](docs/consumers/subprocess.md) — invoke Notarius
from an orchestrator and consume a published result.
- [Internal overview](docs/internal/overview.md) — implemented component map - [Internal overview](docs/internal/overview.md) — implemented component map
for maintainers. for maintainers.
- [Developer guide](docs/development.md) — contributor orientation and - [Developer guide](docs/development.md) — contributor orientation and

View File

@@ -10,7 +10,7 @@ defined in [Operations](operations.md).
~~~ ~~~
notarius help notarius help
notarius run <pipeline-id> --input path/to/source.json [flags] notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list [--config path/to/config.yml] [--json] notarius pipelines list [--config path/to/config.yml] [--json]
~~~ ~~~
@@ -21,7 +21,7 @@ writes the command summary to standard output and exits with status 0.
## run ## run
~~~ ~~~
notarius run <pipeline-id> --input path/to/source.json [flags] notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
~~~ ~~~
The **run** command executes the named pipeline for one input file. The The **run** command executes the named pipeline for one input file. The
@@ -32,6 +32,7 @@ pipeline ID and **--input** are required.
| **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). | | **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
| **--input path** | Source input file to process. Required. | | **--input path** | Source input file to process. Required. |
| **--output-dir path** | Override the configured output root for this run. | | **--output-dir path** | Override the configured output root for this run. |
| **--json** | Write the successful run-result receipt as JSON to standard output. |
| **--chunk_cache auto\|bypass\|refresh** | Override chunk-plan cache handling for this run. | | **--chunk_cache auto\|bypass\|refresh** | Override chunk-plan cache handling for this run. |
| **--resume** | Reuse compatible recorded checkpoints when checkpoint recording is enabled. | | **--resume** | Reuse compatible recorded checkpoints when checkpoint recording is enabled. |
| **--recompute-step step-id** | With **--resume**, recompute the selected ordered step and its dependent lanes. It cannot be combined with **--only**. | | **--recompute-step step-id** | With **--resume**, recompute the selected ordered step and its dependent lanes. It cannot be combined with **--only**. |
@@ -70,11 +71,18 @@ names, requiredness, and configured bindings are part of the
### Run output ### Run output
On success, standard output contains the completed pipeline ID, counts of Without **--json**, standard output contains the completed pipeline ID, counts
normalized and rejected outputs, and the output directory. A debug-enabled run of normalized and rejected outputs, and the output directory. A debug-enabled
also prints its debug-bundle path to standard output. A successful run with run also prints its debug-bundle path to standard output. A successful run with
warnings reports the warning count to standard error. The published JSON warnings reports the warning count to standard error. The published JSON bundle
envelope is defined by the [JSON output contract](integrations/json-output.md). is defined by the [JSON output contract](integrations/json-output.md).
With **--json**, successful standard output is exactly one
`notarius.run-result.v1` JSON document followed by a newline, with no
human-oriented status or debug-path line. Its fields and compatibility policy
are defined by the [run-result contract](integrations/run-result.md). A caller
must check for exit status 0 before decoding this output; a failed write can
leave incomplete standard-output bytes that are not a result document.
Example: Example:
@@ -134,6 +142,10 @@ go run ./cmd/notarius pipelines list \
Successful commands write their primary result to standard output. Warnings and Successful commands write their primary result to standard output. Warnings and
errors are written to standard error. errors are written to standard error.
For **run --json**, warnings remain on standard error and standard output is a
machine-readable success result only. Syntax and runtime diagnostics remain on
standard error. Parse the result only after the process exits with status 0.
| Status | Meaning | | Status | Meaning |
| --- | --- | | --- | --- |
| 0 | The command completed successfully, including root help. | | 0 | The command completed successfully, including root help. |

View File

@@ -0,0 +1,64 @@
# Using Notarius As A Subprocess
Use this workflow when an orchestrator runs Notarius and consumes its published
artifacts. The [CLI reference](../cli.md) owns invocation syntax and exit
statuses, while the [run-result receipt](../integrations/run-result.md) and
[Published JSON Output contract](../integrations/json-output.md) own the
durable result formats.
## Run And Check The Process
Optionally preflight a selected configuration and pipeline before work starts:
```sh
notarius config validate --config /path/to/notarius.yml --pipeline pipeline-id
```
Invoke the run with explicit paths and machine-readable output. Capture
standard output and standard error separately; do not combine them before
processing the result.
```sh
notarius run pipeline-id \
--config /path/to/notarius.yml \
--input /path/to/source.json \
--output-dir /path/to/output-root \
--json
```
Use absolute paths for supplied input, configuration, output-root, and
reference files. When a stable prompt session identifier or references are
needed, pass the supported CLI flags. Supply credentials through Notarius's
documented configuration and environment mechanisms, never as command-line
arguments or generated secret-bearing configuration.
Wait for the process before interpreting standard output. Only an exit status
of 0 permits decoding the receipt. On a nonzero exit, retain standard error for
diagnosis and ignore all standard-output bytes: a failed receipt write may have
left a partial document.
## Discover Required Artifacts
Decode the successful receipt and accept the schema versions supported by the
caller. Use its `output_directory` as the bundle root. For the production JSON
output, resolve `index_file` under that root with a confinement check and reject
an absolute path or a result that escapes the root.
Read the resulting `index.json` and locate each artifact by `lane_id`, not by a
guessed filename. Before decoding a selected payload, verify its descriptor's
media type and schema identity against the relevant published artifact
contract. The JSON bundle contract links to the available lane contracts.
A zero exit status may still report rejected outputs, warnings, or absent
lanes. The caller decides which lane IDs are required for its own work and
which are optional; it should make that decision explicitly rather than infer
failure from the receipt counts alone.
## Preserve Provenance And Handle Data Carefully
Keep the receipt with the published `manifest.json`, and retain
`rejected.json` and `warnings.json` when review or later provenance requires
them. Treat the input, output bundle, cache, debug bundle, and captured process
logs as potentially sensitive data. Apply the caller's access controls and
retention policy, and avoid copying secrets into arguments, logs, or
provenance records.

View File

@@ -18,6 +18,7 @@ implemented component map.
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. | | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
| Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. | | Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
| CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. | | CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. |
| Building a subprocess caller or changing its result protocol | [Subprocess Consumer Guide](consumers/subprocess.md), [Run Result Receipt](integrations/run-result.md), and [CLI Internals](internal/cli.md) | These separate caller workflow, durable receipt contract, and CLI implementation behavior. |
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. | | Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. | | Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |

View File

@@ -12,6 +12,10 @@ All paths below are logical, relative, slash-separated bundle paths. The
encoder always emits the first four JSON files below and adds lane or chunk-map encoder always emits the first four JSON files below and adds lane or chunk-map
files when their corresponding artifacts are available: files when their corresponding artifacts are available:
A subprocess caller first obtains the physical bundle root from the
[run-result receipt](run-result.md), then resolves `index.json` beneath that
root for the logical discovery described here.
| Path | Purpose | | Path | Purpose |
| --- | --- | | --- | --- |
| `index.json` | Entry point that names the other published files and lane payloads. | | `index.json` | Entry point that names the other published files and lane payloads. |

View File

@@ -0,0 +1,68 @@
# Run Result Receipt
`notarius run --json` writes this receipt to standard output when a run
completes successfully. It lets a subprocess caller discover the physical root
of the published output bundle without parsing interactive command output.
Command syntax, streams, and exit statuses are defined in the
[CLI reference](../cli.md); logical files within the bundle are defined in the
[Published JSON Output contract](json-output.md).
## Schema
The current schema version is `notarius.run-result.v1`.
| Field | Required | Meaning |
| --- | --- | --- |
| `schema_version` | Yes | Exactly `notarius.run-result.v1`. |
| `run_id` | Yes | The finalized Notarius run identifier. |
| `pipeline_id` | Yes | The effective pipeline identifier. |
| `output_directory` | Yes | Absolute path to the published, run-specific output bundle. |
| `index_file` | For the production JSON output | Logical path `index.json`; omitted for other output modules. |
| `normalized_output_count` | Yes | Number of final normalized outputs. |
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
| `warning_count` | Yes | Number of final run warnings. |
| `validation_status` | Yes | The final run manifest validation status. |
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
For the production `json` output module, `index_file` is present only when the
completed run returned exactly one logical output file named `index.json`.
For another output module, its absence does not indicate a failed run.
```json
{
"schema_version": "notarius.run-result.v1",
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
"pipeline_id": "dnd-session",
"output_directory": "/work/results/run-1770000000000000000-0123456789abcdef0123456789abcdef",
"index_file": "index.json",
"normalized_output_count": 6,
"rejected_output_count": 2,
"warning_count": 1,
"validation_status": "approved"
}
```
## Paths And Bundle Discovery
`output_directory` and `debug_directory`, when present, are lexical absolute
paths. They identify the paths used by Notarius and do not resolve symlinks.
`output_directory` is the run-specific bundle, not the configured output root.
The receipt is a summary and discovery document. It does not contain lane
descriptors, payloads, manifest data, rejections, warnings, or file contents.
For the production JSON output, resolve `index_file` beneath
`output_directory`, reject path escapes, and use the
[Published JSON Output contract](json-output.md) to discover logical files and
lane payloads.
## Delivery And Compatibility
Notarius writes the receipt only after the output bundle has been published and
any requested debug terminal reporting has completed. Standard output is not
transactional: a result-write failure returns a nonzero status and can leave
partial bytes. Consumers must ignore standard output unless the process exits
with status 0.
Future versions may add optional fields to this schema. Consumers must tolerate
unknown fields. An incompatible field or semantic change requires a new
`schema_version` value.

View File

@@ -84,6 +84,16 @@ final command result. Detailed state lifecycle, resume handling, and physical
path confinement are maintained in [Run State Internals](state.md) and path confinement are maintained in [Run State Internals](state.md) and
[Operations](../operations.md). [Operations](../operations.md).
For `run --json`, the CLI constructs and encodes its private run-result receipt
after a successful runner result is available, before it publishes logical
output files. It writes the prepared receipt to standard output only after
output publication and requested debug terminalization succeed. A receipt-write
failure is a runtime command failure; output may be partial, the CLI reports a
bounded command-owned error, and it does not repeat terminal reporting. The
receipt remains a CLI reporting concern rather than a framework or output-module
responsibility; its public contract is the
[run-result receipt](../integrations/run-result.md).
## Failure Mapping And Terminal Reporting ## Failure Mapping And Terminal Reporting
Argument, flag, and invocation-combination failures are reported to standard Argument, flag, and invocation-combination failures are reported to standard

View File

@@ -2,7 +2,7 @@
## Status ## Status
Ready for implementation. Completed.
## Objective ## Objective

View File

@@ -2,7 +2,7 @@
## Status ## Status
Accepted for implementation. Implemented.
## Purpose ## Purpose

View File

@@ -5,6 +5,7 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"errors"
"flag" "flag"
"fmt" "fmt"
"io" "io"
@@ -28,7 +29,7 @@ import (
const defaultConfigPath = "/usr/local/etc/notarius/config.yml" const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
const usage = `Usage: const usage = `Usage:
notarius help notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector] notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--output-dir path] [--json] [--chunk_cache auto|bypass|refresh] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--only lane-a,lane-b] [--session-id id] [--reference selector=path] [--without-reference selector]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json] notarius pipelines list --config path/to/config.yml [--json]
` `
@@ -132,6 +133,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
inputPath := fs.String("input", "", "source input file path") inputPath := fs.String("input", "", "source input file path")
onlyRaw := fs.String("only", "", "comma-separated artifact lanes") onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
outputDir := fs.String("output-dir", "", "output directory") outputDir := fs.String("output-dir", "", "output directory")
machineOutput := fs.Bool("json", false, "write the successful run result as JSON")
debug := fs.Bool("debug", false, "write a debug bundle") debug := fs.Bool("debug", false, "write a debug bundle")
debugDir := fs.String("debug-dir", "", "debug bundle directory") debugDir := fs.String("debug-dir", "", "debug bundle directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override") llmProfile := fs.String("llm-profile", "", "LLM profile override")
@@ -415,6 +417,17 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err := writePartialSummary(summary, output); err != nil { if err := writePartialSummary(summary, output); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug summary: %w", err)) return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug summary: %w", err))
} }
var encodedResult []byte
if *machineOutput {
result, err := newRunResult(effective.ResolvedPipeline, output, runOutputDir, debugPath)
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
encodedResult, err = encodeRunResult(result)
if err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err)
}
}
if err := writeOutputFiles(runOutputDir, output.OutputFiles); err != nil { if err := writeOutputFiles(runOutputDir, output.OutputFiles); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, err) return failPipelineCommand(stderr, commandState, terminalWriter, err)
} }
@@ -422,9 +435,15 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return writePipelineCommandFailure(stderr, commandState, primaryErr, persistenceErr) return writePipelineCommandFailure(stderr, commandState, primaryErr, persistenceErr)
} }
fmt.Fprintf(stdout, "pipeline %q complete: outputs=%d rejected=%d output=%s\n", effective.PipelineID, len(output.NormalizeOutputs), len(output.Rejected), runOutputDir) if *machineOutput {
if debugPath != "" { if err := writeRunResult(stdout, encodedResult); err != nil {
fmt.Fprintf(stdout, "debug=%s\n", debugPath) return writePipelineCommandFailure(stderr, commandState, errors.New("write run result"), nil)
}
} else {
fmt.Fprintf(stdout, "pipeline %q complete: outputs=%d rejected=%d output=%s\n", effective.PipelineID, len(output.NormalizeOutputs), len(output.Rejected), runOutputDir)
if debugPath != "" {
fmt.Fprintf(stdout, "debug=%s\n", debugPath)
}
} }
if len(output.Warnings) > 0 { if len(output.Warnings) > 0 {
fmt.Fprintf(stderr, "notarius: run completed with %d warning(s)\n", len(output.Warnings)) fmt.Fprintf(stderr, "notarius: run completed with %d warning(s)\n", len(output.Warnings))

111
internal/cli/run_result.go Normal file
View File

@@ -0,0 +1,111 @@
package cli
import (
"encoding/json"
"fmt"
"io"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const runResultSchemaVersion = "notarius.run-result.v1"
type runResult struct {
SchemaVersion string `json:"schema_version"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputDirectory string `json:"output_directory"`
IndexFile string `json:"index_file,omitempty"`
NormalizedOutputCount int `json:"normalized_output_count"`
RejectedOutputCount int `json:"rejected_output_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status"`
DebugDirectory string `json:"debug_directory,omitempty"`
}
func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string) (runResult, error) {
if strings.TrimSpace(output.Manifest.RunID) == "" {
return runResult{}, fmt.Errorf("run result requires a run ID")
}
if strings.TrimSpace(resolved.ID) == "" {
return runResult{}, fmt.Errorf("run result requires a resolved pipeline ID")
}
if strings.TrimSpace(output.Manifest.PipelineID) == "" {
return runResult{}, fmt.Errorf("run result requires a manifest pipeline ID")
}
if output.Manifest.PipelineID != resolved.ID {
return runResult{}, fmt.Errorf("run result pipeline ID does not match resolved pipeline")
}
if strings.TrimSpace(output.Manifest.ValidationStatus) == "" {
return runResult{}, fmt.Errorf("run result requires a validation status")
}
if strings.TrimSpace(outputDirectory) == "" {
return runResult{}, fmt.Errorf("run result requires an output directory")
}
absOutputDirectory, err := filepath.Abs(outputDirectory)
if err != nil {
return runResult{}, fmt.Errorf("make output directory absolute: %w", err)
}
result := runResult{
SchemaVersion: runResultSchemaVersion,
RunID: output.Manifest.RunID,
PipelineID: resolved.ID,
OutputDirectory: absOutputDirectory,
NormalizedOutputCount: len(output.NormalizeOutputs),
RejectedOutputCount: len(output.Rejected),
WarningCount: len(output.Warnings),
ValidationStatus: output.Manifest.ValidationStatus,
}
if strings.TrimSpace(debugDirectory) != "" {
absDebugDirectory, err := filepath.Abs(debugDirectory)
if err != nil {
return runResult{}, fmt.Errorf("make debug directory absolute: %w", err)
}
result.DebugDirectory = absDebugDirectory
}
if resolved.Output.Module == pipeline.DefaultOutputModule {
indexCount := 0
for _, file := range output.OutputFiles {
if file.Name == "index.json" {
indexCount++
}
}
if indexCount != 1 {
return runResult{}, fmt.Errorf("production JSON output must contain exactly one index.json file")
}
result.IndexFile = "index.json"
}
return result, nil
}
func encodeRunResult(result runResult) ([]byte, error) {
encoded, err := json.Marshal(result)
if err != nil {
return nil, fmt.Errorf("encode run result: %w", err)
}
return append(encoded, '\n'), nil
}
func writeRunResult(writer io.Writer, content []byte) error {
for len(content) > 0 {
written, err := writer.Write(content)
if written < 0 || written > len(content) {
return io.ErrShortWrite
}
content = content[written:]
if err != nil {
return err
}
if written == 0 {
return io.ErrShortWrite
}
}
return nil
}

View File

@@ -0,0 +1,199 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_reject"
)
func TestMaintainedMinimalInvocationEmitsRunResult(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"run", "dnd-session",
"--config", repositoryPath("examples", "dnd-minimal.config.yml"),
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--only", "spells", "--chunk_cache", "bypass", "--output-dir", outputRoot, "--json",
}, &stdout, &stderr, productionRunOptions(t, &productionFakeLLMClient{}))
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
receipt := decodeRunResultDocument(t, stdout.String())
if got := receipt["schema_version"]; got != "notarius.run-result.v1" {
t.Fatalf("schema_version = %q", got)
}
if got := receipt["run_id"]; got != productionRunID {
t.Fatalf("run_id = %q", got)
}
if got := receipt["pipeline_id"]; got != "dnd-session" {
t.Fatalf("pipeline_id = %q", got)
}
if got := receipt["index_file"]; got != "index.json" {
t.Fatalf("index_file = %q", got)
}
if got := receipt["normalized_output_count"]; got != float64(1) {
t.Fatalf("normalized_output_count = %v", got)
}
if got := receipt["rejected_output_count"]; got != float64(0) {
t.Fatalf("rejected_output_count = %v", got)
}
if got := receipt["warning_count"]; got != float64(0) {
t.Fatalf("warning_count = %v", got)
}
if got := receipt["validation_status"]; got != "approved" {
t.Fatalf("validation_status = %q", got)
}
outputDirectory, ok := receipt["output_directory"].(string)
if !ok || !filepath.IsAbs(outputDirectory) || outputDirectory != filepath.Join(outputRoot, productionRunID) {
t.Fatalf("output_directory = %q", receipt["output_directory"])
}
indexFile := receipt["index_file"].(string)
assertFile(t, filepath.Join(outputDirectory, indexFile))
}
func TestRunResultReportsWarningsAndDebugBundle(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "contract-warning", Message: "warning retained"}}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--debug", "--json",
}, &stdout, &stderr, harness.options())
if code != 0 || !strings.Contains(stderr.String(), "1 warning(s)") {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
receipt := decodeRunResultDocument(t, stdout.String())
if got := receipt["warning_count"]; got != float64(1) {
t.Fatalf("warning_count = %v", got)
}
debugDirectory, ok := receipt["debug_directory"].(string)
if !ok || !filepath.IsAbs(debugDirectory) || debugDirectory != onlyChildDir(t, roots.debug) {
t.Fatalf("debug_directory = %q", receipt["debug_directory"])
}
if strings.Contains(stdout.String(), "complete:") || strings.Contains(stdout.String(), "debug=") {
t.Fatalf("machine stdout contains human reporting: %q", stdout.String())
}
}
func TestRunResultReportsSuccessfulRejection(t *testing.T) {
roots := newStateTestRoots(t)
configBytes, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n"))
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
t.Fatal(err)
}
harness := newStateTestHarness()
opts := harness.options()
if err := alwaysreject.RegisterTyped[stateTestArtifact](opts.Registries.Validators, stateTestArtifactKind); err != nil {
t.Fatal(err)
}
opts.Catalog = catalogFromRegistries(opts.Registries)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--json",
}, &stdout, &stderr, opts)
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
receipt := decodeRunResultDocument(t, stdout.String())
if got := receipt["normalized_output_count"]; got != float64(0) {
t.Fatalf("normalized_output_count = %v", got)
}
if got := receipt["rejected_output_count"]; got != float64(1) {
t.Fatalf("rejected_output_count = %v", got)
}
if got := receipt["validation_status"]; got != "rejected" {
t.Fatalf("validation_status = %q", got)
}
}
func TestRunResultIsAbsentForSyntaxAndRuntimeFailures(t *testing.T) {
t.Run("syntax", func(t *testing.T) {
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--json"}, &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
t.Run("runtime", func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.extractErr = errors.New("injected extraction failure")
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--json",
}, &stdout, &stderr, harness.options())
if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
func TestRunResultDeliveryFailureRetainsPublishedBundles(t *testing.T) {
roots := newStateTestRoots(t)
writerErr := errors.New("result writer sentinel")
stdout := &resultDeliveryWriter{err: writerErr}
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--debug", "--json",
}, stdout, &stderr, newStateTestHarness().options())
if code != 1 || !strings.Contains(stderr.String(), "write run result") || strings.Contains(stderr.String(), writerErr.Error()) {
t.Fatalf("code=%d stderr=%q", code, stderr.String())
}
if stdout.accepted.Len() != 0 {
t.Fatalf("accepted stdout = %q", stdout.accepted.String())
}
assertStateTestOutput(t, roots.output)
debugBundle := onlyChildDir(t, roots.debug)
report := readStateTestRunReport(t, debugBundle)
if !report.Succeeded {
t.Fatalf("debug report = %#v, want successful persisted run", report)
}
if strings.Contains(readAllFiles(t, debugBundle), writerErr.Error()) {
t.Fatalf("debug bundle contains result writer error")
}
}
func decodeRunResultDocument(t *testing.T, stdout string) map[string]any {
t.Helper()
if strings.Count(stdout, "\n") != 1 {
t.Fatalf("stdout = %q, want one JSON document", stdout)
}
var receipt map[string]any
if err := json.Unmarshal([]byte(stdout), &receipt); err != nil {
t.Fatalf("decode run result: %v; stdout=%q", err, stdout)
}
return receipt
}
type resultDeliveryWriter struct {
err error
accepted bytes.Buffer
}
func (w *resultDeliveryWriter) Write(content []byte) (int, error) {
if w.err != nil {
return 0, w.err
}
return w.accepted.Write(content)
}

View File

@@ -0,0 +1,200 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"io"
"path/filepath"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRunResultEncodesRequiredFieldsAndCounts(t *testing.T) {
result, err := newRunResult(testResolvedPipeline(pipeline.DefaultOutputModule), testRunOutput(), "relative-output", "relative-debug")
if err != nil {
t.Fatal(err)
}
encoded, err := encodeRunResult(result)
if err != nil {
t.Fatal(err)
}
if encoded[len(encoded)-1] != '\n' || bytes.Count(encoded, []byte{'\n'}) != 1 {
t.Fatalf("encoded result is not one newline-terminated object: %q", encoded)
}
var decoded map[string]any
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatal(err)
}
if got := decoded["schema_version"]; got != runResultSchemaVersion {
t.Fatalf("schema_version = %q", got)
}
if got := decoded["run_id"]; got != "run-123" {
t.Fatalf("run_id = %q", got)
}
if got := decoded["pipeline_id"]; got != "sample" {
t.Fatalf("pipeline_id = %q", got)
}
if got := decoded["validation_status"]; got != "rejected" {
t.Fatalf("validation_status = %q", got)
}
if got := decoded["index_file"]; got != "index.json" {
t.Fatalf("index_file = %q", got)
}
if got := decoded["normalized_output_count"]; got != float64(2) {
t.Fatalf("normalized_output_count = %v", got)
}
if got := decoded["rejected_output_count"]; got != float64(1) {
t.Fatalf("rejected_output_count = %v", got)
}
if got := decoded["warning_count"]; got != float64(1) {
t.Fatalf("warning_count = %v", got)
}
if got := decoded["output_directory"]; got != filepath.Join(mustWorkingDirectory(t), "relative-output") {
t.Fatalf("output_directory = %q", got)
}
if got := decoded["debug_directory"]; got != filepath.Join(mustWorkingDirectory(t), "relative-debug") {
t.Fatalf("debug_directory = %q", got)
}
}
func TestRunResultRejectsInvalidRequiredValues(t *testing.T) {
tests := []struct {
name string
resolved pipeline.ResolvedPipeline
output pipeline.RunOutput
directory string
}{
{name: "blank run ID", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.RunID = " " }), directory: "output"},
{name: "blank resolved pipeline ID", resolved: pipeline.ResolvedPipeline{Output: pipeline.ModuleBinding{Module: pipeline.DefaultOutputModule}}, output: testRunOutput(), directory: "output"},
{name: "blank manifest pipeline ID", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.PipelineID = "" }), directory: "output"},
{name: "mismatched pipeline IDs", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.PipelineID = "other" }), directory: "output"},
{name: "blank validation status", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.ValidationStatus = " " }), directory: "output"},
{name: "blank output directory", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutput(), directory: " "},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if _, err := newRunResult(tt.resolved, tt.output, tt.directory, ""); err == nil {
t.Fatal("newRunResult() succeeded")
}
})
}
}
func TestRunResultOmitsIndexFileForOtherOutputModules(t *testing.T) {
result, err := newRunResult(testResolvedPipeline("test/output"), testRunOutputWithout(func(output *pipeline.RunOutput) {
output.OutputFiles = nil
}), "output", "")
if err != nil {
t.Fatal(err)
}
if result.IndexFile != "" {
t.Fatalf("index_file = %q", result.IndexFile)
}
encoded, err := encodeRunResult(result)
if err != nil {
t.Fatal(err)
}
var decoded map[string]any
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatal(err)
}
if _, ok := decoded["index_file"]; ok {
t.Fatalf("encoded non-JSON result contains index_file: %s", encoded)
}
if _, ok := decoded["debug_directory"]; ok {
t.Fatalf("encoded result without debug capture contains debug_directory: %s", encoded)
}
}
func TestRunResultRequiresOneProductionIndexFile(t *testing.T) {
tests := []struct {
name string
files []contracts.OutputFile
}{
{name: "missing", files: nil},
{name: "duplicate", files: []contracts.OutputFile{{Name: "index.json"}, {Name: "index.json"}}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
output := testRunOutput()
output.OutputFiles = tt.files
if _, err := newRunResult(testResolvedPipeline(pipeline.DefaultOutputModule), output, "output", ""); err == nil {
t.Fatal("newRunResult() succeeded")
}
})
}
}
func TestWriteRunResultCompletesAndReportsWriterFailure(t *testing.T) {
content := []byte("result\n")
var target bytes.Buffer
if err := writeRunResult(partialResultWriter{writer: &target, limit: 2}, content); err != nil {
t.Fatal(err)
}
if got := target.String(); got != string(content) {
t.Fatalf("written result = %q", got)
}
writerErr := errors.New("result writer failed")
if err := writeRunResult(failingResultWriter{err: writerErr}, content); !errors.Is(err, writerErr) {
t.Fatalf("writeRunResult() error = %v", err)
}
if err := writeRunResult(zeroResultWriter{}, content); !errors.Is(err, io.ErrShortWrite) {
t.Fatalf("zero-progress error = %v", err)
}
}
func testResolvedPipeline(outputModule string) pipeline.ResolvedPipeline {
return pipeline.ResolvedPipeline{ID: "sample", Output: pipeline.ModuleBinding{Module: outputModule}}
}
func testRunOutput() pipeline.RunOutput {
return pipeline.RunOutput{
Manifest: artifacts.RunManifest{RunID: "run-123", PipelineID: "sample", ValidationStatus: "rejected"},
NormalizeOutputs: []contracts.SerializedOutput{{}, {}},
Rejected: []contracts.RejectedOutput{{}},
Warnings: []contracts.Warning{{}},
OutputFiles: []contracts.OutputFile{{Name: "index.json"}},
}
}
func testRunOutputWithout(change func(*pipeline.RunOutput)) pipeline.RunOutput {
output := testRunOutput()
change(&output)
return output
}
func mustWorkingDirectory(t *testing.T) string {
t.Helper()
workingDirectory, err := filepath.Abs(".")
if err != nil {
t.Fatal(err)
}
return workingDirectory
}
type partialResultWriter struct {
writer io.Writer
limit int
}
func (w partialResultWriter) Write(content []byte) (int, error) {
if len(content) > w.limit {
content = content[:w.limit]
}
return w.writer.Write(content)
}
type failingResultWriter struct{ err error }
func (w failingResultWriter) Write([]byte) (int, error) { return 0, w.err }
type zeroResultWriter struct{}
func (zeroResultWriter) Write([]byte) (int, error) { return 0, nil }