Switch commands to ConfigKit configuration

This commit is contained in:
2026-09-13 17:12:56 +00:00
parent d809085406
commit 92a6a4985a
23 changed files with 393 additions and 331 deletions

View File

@@ -12,8 +12,8 @@ defined in [Operations](operations.md).
notarius help notarius help
notarius --version notarius --version
notarius run <pipeline-id> --input path/to/source.json [--json] [flags] notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate [--config path/to/config.yml] [--config-profile name] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list [--config path/to/config.yml] [--json] notarius pipelines list [--config path/to/config.yml] [--config-profile name] [--json]
~~~ ~~~
Running Notarius without arguments, or with **help**, **--help**, or **-h**, Running Notarius without arguments, or with **help**, **--help**, or **-h**,
@@ -42,6 +42,7 @@ pipeline ID and **--input** are required.
| Flag | Meaning | | Flag | Meaning |
| --- | --- | | --- | --- |
| **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). | | **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
| **--config-profile name** | Select a ConfigKit configuration overlay. Empty and **default** select the composed base configuration. |
| **--input path** | Source input file to process. Required. | | **--input path** | Source input file to process. Required. |
| **--output-dir path** | Override the configured output root for this run. | | **--output-dir path** | Override the configured output root for this run. |
| **--json** | Write the successful run-result receipt as JSON to standard output. | | **--json** | Write the successful run-result receipt as JSON to standard output. |
@@ -154,10 +155,10 @@ OPENROUTER_API_KEY=your-api-key \
## config validate ## config validate
~~~ ~~~
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate [--config path/to/config.yml] [--config-profile name] [--pipeline pipeline-id] [--only lane-a,lane-b]
~~~ ~~~
This command loads and validates a configuration. With **--pipeline**, it also This command loads and validates the selected composed configuration. With **--pipeline**, it also
resolves that pipeline against the production module catalog. **--only** selects resolves that pipeline against the production module catalog. **--only** selects
lanes during that resolution and requires **--pipeline**. lanes during that resolution and requires **--pipeline**.
@@ -183,7 +184,7 @@ validation; it cannot run a provider-backed pipeline.
## pipelines list ## pipelines list
~~~ ~~~
notarius pipelines list [--config path/to/config.yml] [--json] notarius pipelines list [--config path/to/config.yml] [--config-profile name] [--json]
~~~ ~~~
This command lists configured pipeline IDs in sorted order. By default, it This command lists configured pipeline IDs in sorted order. By default, it

View File

@@ -1,7 +1,7 @@
# Configuration # Configuration
This is the canonical reference for Notarius configuration. Configuration files This is the canonical reference for Notarius configuration. Configuration files
are YAML and must declare version 4. They select pipelines and their modules; are YAML and must declare version 5. They select pipelines and their modules;
the [CLI reference](cli.md) owns invocation syntax, and the [CLI reference](cli.md) owns invocation syntax, and
[Operations](operations.md) owns run-state procedures. [Operations](operations.md) owns run-state procedures.
@@ -19,12 +19,15 @@ The command fails if none of these paths provides a configuration file.
For configuration values, precedence is: For configuration values, precedence is:
1. built-in defaults; 1. built-in defaults;
2. the selected YAML file; 2. composed YAML sources, including imports;
3. supported operational environment variables; and 3. the selected ConfigKit profile; and
4. the CLI run overrides that apply to a command. 4. explicitly supplied configuration-valued run flags.
Environment variables do not provide a second configuration schema. They only `--config-profile` selects a ConfigKit overlay from `profiles/<name>.yml`
override the fields listed below. beneath the selected configuration root. Empty and `default` select no overlay.
Configuration profiles are distinct from PromptKit LLM profiles selected by
`llm_profile` or `--llm-profile`. The retired operational environment variables
have no effect; `NOTARIUS_CONFIG` remains a source-discovery input.
## Maintained Examples ## Maintained Examples
@@ -46,7 +49,7 @@ other than **version** is optional.
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| **version** | integer | none | Required; must be 4. | | **version** | integer or quoted string | none | Required; must be 5. Maintained files use `version: 5`. |
| **promptkit** | object | none | Profile source and optional local-backend configuration. | | **promptkit** | object | none | Profile source and optional local-backend configuration. |
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. | | **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
| **concurrency** | object | see below | Global LLM and extraction limits. | | **concurrency** | object | see below | Global LLM and extraction limits. |
@@ -73,13 +76,26 @@ Selecting a chunk-plan or checkpoint directory does not configure filesystem
access policy. Cache permissions come from deployment provisioning, process access policy. Cache permissions come from deployment provisioning, process
umask, setgid inheritance, and default ACLs; see [Operations](operations.md#cache-provisioning). umask, setgid inheritance, and default ACLs; see [Operations](operations.md#cache-provisioning).
Every nonempty relative configured destination or reference path resolves from
the canonical directory of the file that supplies its surviving declaration.
This includes imported fragments and selected configuration profiles. Built-in
relative destinations use the selected main configuration file's directory;
explicit `--output-dir` and `--debug-dir` values use the invocation working
directory. Absolute values remain unchanged.
The root file can use ConfigKit imports to compose disjoint definitions. Imports
and selected profiles remain under the configuration root; they do not merge or
append a pipeline's `steps` sequence. A profile replaces sequences as defined
by ConfigKit, so changing an individual ordered binding requires supplying the
whole replacement sequence.
## PromptKit Profiles ## PromptKit Profiles
The optional **promptkit** object selects one source of profile definitions and The optional **promptkit** object selects one source of profile definitions and
may register one conventional local OpenAI-compatible backend: may register one conventional local OpenAI-compatible backend:
~~~yaml ~~~yaml
version: 4 version: 5
promptkit: promptkit:
profile_dir: ./profiles profile_dir: ./profiles
@@ -97,11 +113,12 @@ promptkit:
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. | | **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. | | **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
Set at most one of **profile_dir** and **profile_file**. Relative values use Set at most one of **profile_dir** and **profile_file**. Every nonempty relative
the process working directory, not the configuration file's directory. The configured filesystem path is resolved from the canonical directory of the
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore file that supplies it. The complete example's
valid when Notarius is launched from the repository root; use an absolute path `./profiles/dnd-extraction.yml` value therefore resolves beside that example.
for services and containers. Use absolute paths for services and containers when their deployment layout is
not configuration-relative.
An operator source is optional. For a requested ID, PromptKit checks the An operator source is optional. For a requested ID, PromptKit checks the
configured operator source first, then Notarius's embedded fallback profiles, configured operator source first, then Notarius's embedded fallback profiles,
@@ -150,30 +167,15 @@ requiring credential values. It rejects absent, malformed, or incompatible
profiles before a run prepares modules. Credential availability is checked only profiles before a run prepares modules. Credential availability is checked only
when a generation is prepared. when a generation is prepared.
## Migrating Version 3 Configuration ## Version 5 Migration
Version 3 files are not decoded or rewritten. Change **version: 3** to Version 3 and version 4 files are rejected and are not rewritten at load time.
**version: 4** and rename the top-level **scriptorium:** section to Use `version: 5`, rename the former `scriptorium` section to `promptkit`, and
**promptkit:**. Version 4 decoding is strict, so a remaining **scriptorium** convert every module binding to an object containing `module`. Convert every
field is rejected as unknown. external reference to `{path: ...}` and move any lane-level references to the
corresponding extraction binding. Put durable operational values in YAML or
## Operational Environment Variables use the documented explicit run flags; the former field-level environment
variables are not read.
These variables are applied after YAML values:
| Variable | Overrides | Rules |
| --- | --- | --- |
| **NOTARIUS_TOTAL_LLM_CONCURRENCY** | **concurrency.total_llm** | Integer. |
| **NOTARIUS_STAGE_WORKERS_EXTRACT** | **concurrency.stage_workers.extract** | Integer. |
| **NOTARIUS_OUTPUT_DIR** | **output.directory** | Non-empty path. |
| **NOTARIUS_CACHE_CHUNK_PLANS_MODE** | **cache.chunk_plans.mode** | **auto**, **bypass**, or **refresh**. |
| **NOTARIUS_CACHE_CHUNK_PLANS_DIR** | **cache.chunk_plans.directory** | Non-empty path. |
| **NOTARIUS_CACHE_CHECKPOINTS_DIR** | **cache.checkpoints.directory** | Non-empty path. |
| **NOTARIUS_DEBUG_DIR** | **debug.directory** | Non-empty path. |
Integer values are trimmed then parsed as base-10 integers. Directory and
output values reject NUL characters. **NOTARIUS_CONFIG** participates only in
configuration discovery.
## Concurrency, Output, Cache, And Debug ## Concurrency, Output, Cache, And Debug
@@ -198,13 +200,14 @@ debug:
**concurrency.total_llm** must be greater than zero. The only supported **concurrency.total_llm** must be greater than zero. The only supported
**concurrency.stage_workers** key is **extract**; its value must be from 1 **concurrency.stage_workers** key is **extract**; its value must be from 1
through **total_llm**. When omitted, it is recalculated from the effective through **total_llm**. When omitted, it is recalculated from the effective
**total_llm** after YAML and environment precedence. **total_llm** after imports, profile selection, and explicit ConfigKit
overrides.
**cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**. **cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**.
**cache.checkpoints.enabled** is a boolean. The CLI can override the output **cache.checkpoints.enabled** is a boolean. The CLI can override the output
directory and chunk-plan mode for one run; see [CLI reference](cli.md#run). directory and chunk-plan mode for one run; see [CLI reference](cli.md#run).
Cache directory fields select only locations; Notarius has no cache permission Cache directory fields select only locations; Notarius has no cache permission
field or environment override. field or operational environment override.
## Pipelines ## Pipelines
@@ -214,14 +217,20 @@ Each **pipelines** entry has a unique, non-empty ID and the following shape:
pipelines: pipelines:
dnd-session: dnd-session:
llm_profile: dnd-extraction llm_profile: dnd-extraction
input: seriatim input:
chunk: generic module: seriatim
output: json chunk:
module: generic
output:
module: json
artifacts: artifacts:
spells: spells:
extract: dnd/spells extract:
merge: appendorder module: dnd/spells
normalize: dnd/spells merge:
module: appendorder
normalize:
module: dnd/spells
~~~ ~~~
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
@@ -291,19 +300,20 @@ A lane has these fields:
| **extract** | module binding | none | Required. | | **extract** | module binding | none | Required. |
| **merge** | module binding | **appendorder** | Optional. | | **merge** | module binding | **appendorder** | Optional. |
| **normalize** | module binding | **noop** | Optional. | | **normalize** | module binding | **noop** | Optional. |
| **references** | map | none | Supported compatibility alias for **extract.references**. | | **references** | map | none | Not supported at lane scope; place references on the eligible binding. |
| **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. | | **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. |
The lane-level **references** alias remains accepted. When the alias and Lane-level **references** are not supported. Put a lane extraction reference
**extract.references** bind the same slot, **extract.references** wins. Use under **extract.references**.
the binding-local form in new configurations.
## Module Bindings And Validators ## Module Bindings And Validators
Use a module key directly when no other binding fields are needed: Every module and validator binding uses an object, even when it has no fields
beyond its module key:
~~~yaml ~~~yaml
input: seriatim input:
module: seriatim
~~~ ~~~
Use an object for fields: Use an object for fields:
@@ -314,7 +324,8 @@ extract:
llm_profile: dnd-extraction llm_profile: dnd-extraction
retries: 2 retries: 2
references: references:
spell_catalog: ./dnd-spell-catalog.json spell_catalog:
path: ./dnd-spell-catalog.json
~~~ ~~~
| Binding field | Type | Default | Rules | | Binding field | Type | Default | Rules |
@@ -386,18 +397,20 @@ When enabled, it publishes the selected source-unit excerpt defined by the
## References And Ordered Handoffs ## References And Ordered Handoffs
Reference maps bind named slots that the selected target declares. A scalar is Reference maps bind named slots that the selected target declares. An external
an external path. Pipeline-level maps accept only external paths; step-local reference is an object with `path`. Pipeline-level maps accept only external
and binding-local maps may also select a normalized artifact from an earlier paths; step-local and binding-local maps may also select a normalized artifact
step: from an earlier step:
~~~yaml ~~~yaml
steps: steps:
- id: describe-session - id: describe-session
artifacts: artifacts:
npc-registry: npc-registry:
extract: dnd/npc-registry extract:
normalize: dnd/npc-registry module: dnd/npc-registry
normalize:
module: dnd/npc-registry
- id: extract-events - id: extract-events
references: references:
npc_registry: npc_registry:
@@ -406,8 +419,10 @@ steps:
lane: npc-registry lane: npc-registry
artifacts: artifacts:
spells: spells:
extract: dnd/spells extract:
normalize: dnd/spells module: dnd/spells
normalize:
module: dnd/spells
~~~ ~~~
An artifact selector contains only **step** and **lane**. The producer must be An artifact selector contains only **step** and **lane**. The producer must be
@@ -422,9 +437,10 @@ bindings are then operational overrides of configured external paths; their
pipeline, lane, and binding scopes and precedence are defined by the pipeline, lane, and binding scopes and precedence are defined by the
[CLI reference](cli.md#reference-selectors). A CLI file reference cannot [CLI reference](cli.md#reference-selectors). A CLI file reference cannot
replace a configured generated artifact handoff. Required slots must be bound replace a configured generated artifact handoff. Required slots must be bound
after configuration and CLI reference actions are applied. Reference paths in after configuration and CLI reference actions are applied. A YAML reference
YAML are resolved relative to the configuration file; CLI reference paths are path resolves from the file that supplies its surviving declaration, including
resolved relative to the process working directory. an imported fragment or selected profile. CLI reference paths resolve relative
to the process working directory.
### D&D Reference Slots ### D&D Reference Slots
@@ -573,7 +589,7 @@ go run ./cmd/notarius config validate \
~~~ ~~~
Configuration validation rejects invalid YAML, unsupported fields, invalid Configuration validation rejects invalid YAML, unsupported fields, invalid
defaults or environment overrides, incompatible module keys, unknown options, defaults or explicit CLI overrides, incompatible module keys, unknown options,
invalid reference bindings, missing required reference slots, invalid validator invalid reference bindings, missing required reference slots, invalid validator
overrides, and incompatible generated artifact handoffs. Use overrides, and incompatible generated artifact handoffs. Use
[pipelines list](cli.md#pipelines-list) to inspect configured IDs. [pipelines list](cli.md#pipelines-list) to inspect configured IDs.

View File

@@ -23,16 +23,14 @@ A deployment must provide its own PromptKit profile and campaign reference
files. Use absolute paths for service and subprocess deployments. In files. Use absolute paths for service and subprocess deployments. In
particular, observe these different resolution rules: particular, observe these different resolution rules:
- reference paths in YAML are resolved relative to the Notarius configuration - reference and PromptKit profile paths in YAML are resolved relative to the
file; file that supplies their surviving declaration;
- reference paths passed with `--reference` are resolved relative to the - reference paths passed with `--reference` are resolved relative to the
Notarius process working directory; and Notarius process working directory.
- `promptkit.profile_file` is resolved relative to the Notarius process working
directory.
Do not copy the repository example's relative profile path into a deployment Do not copy the repository example's relative profile path into a deployment
without also controlling that working directory. The complete path and profile without preserving its configuration-relative layout. The complete path and
rules are defined in [Configuration](../config.md). profile rules are defined in [Configuration](../config.md).
Preflight the deployed configuration before processing sessions and whenever Preflight the deployed configuration before processing sessions and whenever
it changes: it changes:

View File

@@ -15,9 +15,12 @@ configured lane inventory, and downstream acceptance checklist, see
Optionally preflight a selected configuration and pipeline before work starts: Optionally preflight a selected configuration and pipeline before work starts:
```sh ```sh
notarius config validate --config /path/to/notarius.yml --pipeline pipeline-id notarius config validate --config /path/to/notarius.yml --config-profile production --pipeline pipeline-id
``` ```
Omit `--config-profile` when the composed base configuration is the intended
selection; it is independent from any `--llm-profile` run override.
Invoke the run with explicit paths and machine-readable output. Capture Invoke the run with explicit paths and machine-readable output. Capture
standard output and standard error separately; do not combine them before standard output and standard error separately; do not combine them before
processing the result. processing the result.
@@ -25,6 +28,7 @@ processing the result.
```sh ```sh
notarius run pipeline-id \ notarius run pipeline-id \
--config /path/to/notarius.yml \ --config /path/to/notarius.yml \
--config-profile production \
--input /path/to/source.json \ --input /path/to/source.json \
--output-dir /path/to/output-root \ --output-dir /path/to/output-root \
--json --json

View File

@@ -126,11 +126,11 @@ notarius config validate --config /etc/notarius/config.yml --pipeline dnd-sessio
An unset optional `api_key_env` reaches the provider without authorization and An unset optional `api_key_env` reaches the provider without authorization and
may receive a 401 or 403 response. may receive a 401 or 403 response.
Profile paths are currently resolved from the process working directory, not Configured PromptKit profile paths resolve from the file that supplies their
from the configuration file. The complete example's surviving declaration. The complete example's
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root `./profiles/dnd-extraction.yml` path is therefore relative to that example.
invocation only. Use absolute paths such as Use absolute paths such as `/etc/notarius/profiles/dnd-extraction.yml` for
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers. services and containers when that is clearer for the deployment.
## Run Lifecycle ## Run Lifecycle
@@ -308,7 +308,8 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
trace/ trace/
~~~ ~~~
The summary contains redacted invocation and resolution information plus run, The summary contains redacted invocation and resolution information, including
`effective-config.yml` and the separately redacted resolved pipeline, plus run,
final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
terminal records contain bounded attempt kinds, validator outcomes, policy, terminal records contain bounded attempt kinds, validator outcomes, policy,
decision, PromptKit repair count, and usage; they do not contain assistant decision, PromptKit repair count, and usage; they do not contain assistant
@@ -318,7 +319,8 @@ content. Neither surface is a cache input. Do not treat a debug bundle as safe
to share merely because its configuration summary is redacted. Invocation to share merely because its configuration summary is redacted. Invocation
metadata omits reasoning effort when it is inherited, records the replacement metadata omits reasoning effort when it is inherited, records the replacement
value when one is supplied, and records an empty value when inherited reasoning value when one is supplied, and records an empty value when inherited reasoning
was explicitly cleared. was explicitly cleared. It records `config_profile` only when a non-base
configuration overlay was selected.
Notarius never creates debug state without an explicit request and never Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command automatically deletes a requested bundle. If allocation succeeds, the command

View File

@@ -484,6 +484,8 @@ incompatible generated references.
## Stage 10 — Switch Run And The Maintained V5 Configurations ## Stage 10 — Switch Run And The Maintained V5 Configurations
**Status:** Completed 2026-09-13.
**Goal:** Complete production cutover and make all runs use one ConfigKit load. **Goal:** Complete production cutover and make all runs use one ConfigKit load.
**Work:** **Work:**

View File

@@ -1,6 +1,6 @@
version: 4 version: 5
promptkit: promptkit:
profile_file: ./examples/profiles/dnd-extraction.yml profile_file: ./profiles/dnd-extraction.yml
concurrency: concurrency:
total_llm: 2 total_llm: 2
stage_workers: stage_workers:
@@ -19,11 +19,14 @@ debug:
pipelines: pipelines:
dnd-session: dnd-session:
llm_profile: dnd-extraction llm_profile: dnd-extraction
input: seriatim input:
module: seriatim
# Stable campaign context is shared by every module that accepts these slots. # Stable campaign context is shared by every module that accepts these slots.
references: references:
party: ./dnd-party.txt party:
glossary: ./dnd-glossary.txt path: ./dnd-party.txt
glossary:
path: ./dnd-glossary.txt
chunk: chunk:
module: dnd/scenes module: dnd/scenes
retries: 2 retries: 2
@@ -52,13 +55,16 @@ pipelines:
extract: extract:
module: dnd/item-registry module: dnd/item-registry
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/item-registry module: appendorder
normalize:
module: dnd/item-registry
npc-registry: npc-registry:
extract: extract:
module: dnd/npc-registry module: dnd/npc-registry
retries: 2 retries: 2
merge: appendorder merge:
module: appendorder
normalize: normalize:
module: dnd/npc-registry module: dnd/npc-registry
retries: 2 retries: 2
@@ -66,7 +72,8 @@ pipelines:
extract: extract:
module: dnd/location-registry module: dnd/location-registry
retries: 2 retries: 2
merge: appendorder merge:
module: appendorder
normalize: normalize:
module: dnd/location-registry module: dnd/location-registry
retries: 2 retries: 2
@@ -74,8 +81,10 @@ pipelines:
extract: extract:
module: dnd/scene-descriptions module: dnd/scene-descriptions
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/scene-descriptions module: appendorder
normalize:
module: dnd/scene-descriptions
- id: extract-events - id: extract-events
# Accepted registry artifacts and scene-description eligibility artifacts # Accepted registry artifacts and scene-description eligibility artifacts
# are supplied in memory to their compatible consumers in this step. # are supplied in memory to their compatible consumers in this step.
@@ -101,38 +110,49 @@ pipelines:
extract: extract:
module: dnd/item-occurrences module: dnd/item-occurrences
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/item-occurrences module: appendorder
normalize:
module: dnd/item-occurrences
spells: spells:
extract: extract:
module: dnd/spells module: dnd/spells
retries: 2 retries: 2
references: references:
spell_catalog: ./dnd-spell-catalog.json spell_catalog:
merge: appendorder path: ./dnd-spell-catalog.json
merge:
module: appendorder
# Stage-local file references are intentionally bound at each stage. # Stage-local file references are intentionally bound at each stage.
normalize: normalize:
module: dnd/spells module: dnd/spells
references: references:
spell_catalog: ./dnd-spell-catalog.json spell_catalog:
path: ./dnd-spell-catalog.json
combat-turns: combat-turns:
extract: extract:
module: dnd/combat-turns module: dnd/combat-turns
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/combat-turns module: appendorder
normalize:
module: dnd/combat-turns
npc-occurrences: npc-occurrences:
extract: extract:
module: dnd/npc-occurrences module: dnd/npc-occurrences
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/npc-occurrences module: appendorder
normalize:
module: dnd/npc-occurrences
location-occurrences: location-occurrences:
extract: extract:
module: dnd/location-occurrences module: dnd/location-occurrences
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/location-occurrences module: appendorder
normalize:
module: dnd/location-occurrences
- id: track-enemies - id: track-enemies
references: references:
npc_registry: npc_registry:
@@ -156,5 +176,7 @@ pipelines:
extract: extract:
module: dnd/enemy-events module: dnd/enemy-events
retries: 2 retries: 2
merge: appendorder merge:
normalize: dnd/enemy-events module: appendorder
normalize:
module: dnd/enemy-events

View File

@@ -1,9 +1,12 @@
version: 4 version: 5
pipelines: pipelines:
dnd-session: dnd-session:
llm_profile: dnd-extraction llm_profile: dnd-extraction
input: seriatim input:
module: seriatim
artifacts: artifacts:
spells: spells:
extract: dnd/spells extract:
normalize: dnd/spells module: dnd/spells
normalize:
module: dnd/spells

View File

@@ -23,7 +23,7 @@ func TestRunChunkPlanModePrecedenceAndValidation(t *testing.T) {
}{ }{
{name: "default", wantStores: 1}, {name: "default", wantStores: 1},
{name: "file", fileMode: "bypass"}, {name: "file", fileMode: "bypass"},
{name: "environment", envMode: "bypass"}, {name: "retired environment has no effect", envMode: "bypass", wantStores: 1},
{name: "cli", envMode: "refresh", cliMode: "bypass"}, {name: "cli", envMode: "refresh", cliMode: "bypass"},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -81,24 +81,13 @@ func TestRunChunkPlanModePrecedenceAndValidation(t *testing.T) {
for _, tt := range []struct { for _, tt := range []struct {
name string name string
fileConfig bool
}{ }{
{name: "invalid environment mode"}, {name: "invalid file mode"},
{name: "invalid file mode", fileConfig: true},
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t) roots := newStateTestRoots(t)
opts := newStateTestHarness().options() opts := newStateTestHarness().options()
if tt.fileConfig {
replaceStateTestConfigLine(t, roots.config, " mode: auto\n", " mode: invalid\n") replaceStateTestConfigLine(t, roots.config, " mode: auto\n", " mode: invalid\n")
} else {
opts.LookupEnv = func(name string) (string, bool) {
if name == "NOTARIUS_CACHE_CHUNK_PLANS_MODE" {
return "invalid", true
}
return "", false
}
}
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts) code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 { if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 {
@@ -207,16 +196,20 @@ func TestRunAutoReusesPlanWhenRunInputsChange(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
configText := replaceRequiredOnce(t, string(data), " chunk: test/chunk\n", ` chunk: configText := replaceRequiredOnce(t, string(data), " chunk:\n module: test/chunk\n", ` chunk:
module: test/chunk module: test/chunk
options: options:
strategy: first strategy: first
`) `)
configText = replaceRequiredOnce(t, configText, " output: test/output\n", ` other: configText = replaceRequiredOnce(t, configText, " output:\n module: test/output\n", ` other:
extract: test/extract extract:
merge: test/merge module: test/extract
normalize: test/normalize merge:
output: test/output module: test/merge
normalize:
module: test/normalize
output:
module: test/output
`) `)
if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil { if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)

View File

@@ -140,7 +140,7 @@ func TestConfigLoadingFailuresReturnOneWithPathContext(t *testing.T) {
stdout.Reset() stdout.Reset()
stderr.Reset() stderr.Reset()
code = RunWithOptions([]string{"config", "validate", "--config", malformed}, &stdout, &stderr, commandContractOptions(t)) code = RunWithOptions([]string{"config", "validate", "--config", malformed}, &stdout, &stderr, commandContractOptions(t))
if code != 1 || !strings.Contains(stderr.String(), malformed) || !strings.Contains(stderr.String(), "parse config file") || stdout.Len() != 0 { if code != 1 || !strings.Contains(stderr.String(), "configuration contains invalid YAML") || stdout.Len() != 0 {
t.Fatalf("malformed config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String()) t.Fatalf("malformed config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
} }
} }
@@ -185,14 +185,16 @@ func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
} }
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) { func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
configPath := writeCommandConfigContent(t, `version: 4 configPath := writeCommandConfigContent(t, `version: 5
pipelines: pipelines:
demo: demo:
llm_profile: dnd-extraction llm_profile: dnd-extraction
input: seriatim input:
module: seriatim
artifacts: artifacts:
spells: spells:
extract: dnd/spells extract:
module: dnd/spells
`) `)
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{}) code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
@@ -261,19 +263,21 @@ func commandContractOptionsWithLookup(t *testing.T, lookup func(string) (string,
func writeCommandConfig(t *testing.T, firstID, secondID string) string { func writeCommandConfig(t *testing.T, firstID, secondID string) string {
t.Helper() t.Helper()
content := fmt.Sprintf("version: 4\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID) content := fmt.Sprintf("version: 5\npipelines:\n %q:\n input:\n module: seriatim\n %q:\n input:\n module: seriatim\n", firstID, secondID)
return writeCommandConfigContent(t, content) return writeCommandConfigContent(t, content)
} }
func writeResolvableCommandConfig(t *testing.T) string { func writeResolvableCommandConfig(t *testing.T) string {
t.Helper() t.Helper()
return writeCommandConfigContent(t, `version: 4 return writeCommandConfigContent(t, `version: 5
pipelines: pipelines:
demo: demo:
input: seriatim input:
module: seriatim
artifacts: artifacts:
spells: spells:
extract: dnd/spells extract:
module: dnd/spells
`) `)
} }

View File

@@ -11,7 +11,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
) )
func runConfigKitValidate(args []string, stdout, stderr io.Writer, opts Options) int { func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) int {
fs := flag.NewFlagSet("config validate", flag.ContinueOnError) fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
fs.SetOutput(io.Discard) fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path") configPath := fs.String("config", "", "config file path")
@@ -62,7 +62,7 @@ func runConfigKitValidate(args []string, stdout, stderr io.Writer, opts Options)
return 0 return 0
} }
func runConfigKitPipelinesList(args []string, stdout, stderr io.Writer, opts Options) int { func runPipelinesList(args []string, stdout, stderr io.Writer, opts Options) int {
fs := flag.NewFlagSet("pipelines list", flag.ContinueOnError) fs := flag.NewFlagSet("pipelines list", flag.ContinueOnError)
fs.SetOutput(io.Discard) fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path") configPath := fs.String("config", "", "config file path")

View File

@@ -12,14 +12,15 @@ func TestConfigKitReadOnlyCommandsLoadImportsAndProfiles(t *testing.T) {
root := t.TempDir() root := t.TempDir()
path := filepath.Join(root, "notarius.yml") path := filepath.Join(root, "notarius.yml")
writeConfigKitFile(t, filepath.Join(root, "fragment.yml"), "pipelines:\n imported:\n input: {module: input}\n") writeConfigKitFile(t, filepath.Join(root, "fragment.yml"), "pipelines:\n imported:\n input: {module: input}\n")
writeConfigKitFile(t, filepath.Join(root, "profiles", "development.yml"), "pipelines:\n profiled:\n input: {module: input}\n")
writeConfigKitFile(t, path, "version: 5\nconfigkit:\n imports:\n - path: fragment.yml\npipelines:\n base:\n input: {module: input}\n") writeConfigKitFile(t, path, "version: 5\nconfigkit:\n imports:\n - path: fragment.yml\npipelines:\n base:\n input: {module: input}\n")
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
if code := runConfigKitPipelinesList([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 0 || stdout.String() != "base\nimported\n" { if code := RunWithOptions([]string{"pipelines", "list", "--config", path, "--config-profile", "development"}, &stdout, &stderr, commandContractOptions(t)); code != 0 || stdout.String() != "base\nimported\nprofiled\n" {
t.Fatalf("list = %d, %q, %q", code, stdout.String(), stderr.String()) t.Fatalf("list = %d, %q, %q", code, stdout.String(), stderr.String())
} }
stdout.Reset() stdout.Reset()
stderr.Reset() stderr.Reset()
if code := runConfigKitValidate([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 0 || !strings.Contains(stdout.String(), "is valid") { if code := RunWithOptions([]string{"config", "validate", "--config", path, "--config-profile", "development"}, &stdout, &stderr, commandContractOptions(t)); code != 0 || !strings.Contains(stdout.String(), "is valid") {
t.Fatalf("validate = %d, %q, %q", code, stdout.String(), stderr.String()) t.Fatalf("validate = %d, %q, %q", code, stdout.String(), stderr.String())
} }
} }
@@ -29,7 +30,7 @@ func TestConfigKitReadOnlyCommandsRejectVersionFour(t *testing.T) {
path := filepath.Join(t.TempDir(), "notarius.yml") path := filepath.Join(t.TempDir(), "notarius.yml")
writeConfigKitFile(t, path, "version: 4\n") writeConfigKitFile(t, path, "version: 4\n")
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
if code := runConfigKitPipelinesList([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 1 || !strings.Contains(stderr.String(), "version") { if code := runPipelinesList([]string{"--config", path}, &stdout, &stderr, commandContractOptions(t)); code != 1 || !strings.Contains(stderr.String(), "version") {
t.Fatalf("list = %d, %q", code, stderr.String()) t.Fatalf("list = %d, %q", code, stderr.String())
} }
} }

View File

@@ -267,6 +267,7 @@ func completeExampleConfigWithTemporaryCache(t *testing.T) string {
"./dnd-party.txt": repositoryPath("examples", "dnd-party.txt"), "./dnd-party.txt": repositoryPath("examples", "dnd-party.txt"),
"./dnd-glossary.txt": repositoryPath("examples", "dnd-glossary.txt"), "./dnd-glossary.txt": repositoryPath("examples", "dnd-glossary.txt"),
"./dnd-spell-catalog.json": repositoryPath("examples", "dnd-spell-catalog.json"), "./dnd-spell-catalog.json": repositoryPath("examples", "dnd-spell-catalog.json"),
"./profiles/dnd-extraction.yml": repositoryPath("examples", "profiles", "dnd-extraction.yml"),
} { } {
updated = strings.ReplaceAll(updated, relative, fmt.Sprintf("%q", absolute)) updated = strings.ReplaceAll(updated, relative, fmt.Sprintf("%q", absolute))
} }

View File

@@ -23,7 +23,7 @@ func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *test
t.Fatal(err) t.Fatal(err)
} }
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints") checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
content := fmt.Sprintf(`version: 4 content := fmt.Sprintf(`version: 5
cache: cache:
chunk_plans: chunk_plans:
mode: bypass mode: bypass
@@ -32,14 +32,17 @@ cache:
directory: %q directory: %q
pipelines: pipelines:
dnd-session: dnd-session:
input: seriatim input:
module: seriatim
artifacts: artifacts:
spells: spells:
extract: extract:
module: dnd/spells module: dnd/spells
references: references:
npc_registry: %q npc_registry:
normalize: dnd/spells path: %q
normalize:
module: dnd/spells
`, checkpointRoot, npcPath) `, checkpointRoot, npcPath)
configPath := filepath.Join(t.TempDir(), "config.yml") configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil { if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {

View File

@@ -737,13 +737,13 @@ func TestProductionConfigValidationCoversModuleAndVariantFailures(t *testing.T)
}{ }{
{ {
name: "unknown module", name: "unknown module",
content: replaceRequiredOnce(t, base, " input: seriatim\n", " input: missing/input\n"), content: replaceRequiredOnce(t, base, " input:\n module: seriatim\n", " input:\n module: missing/input\n"),
options: productionCLIOptions(t), options: productionCLIOptions(t),
fragments: []string{"pipeline \"dnd-session\"", "input", "missing/input"}, fragments: []string{"pipeline \"dnd-session\"", "input", "missing/input"},
}, },
{ {
name: "unknown validator", name: "unknown validator",
content: replaceRequiredOnce(t, base, " extract: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: missing/validator\n"), content: replaceRequiredOnce(t, base, " extract:\n module: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: missing/validator\n"),
options: productionCLIOptions(t), options: productionCLIOptions(t),
fragments: []string{"validator", "missing/validator"}, fragments: []string{"validator", "missing/validator"},
}, },
@@ -755,7 +755,7 @@ func TestProductionConfigValidationCoversModuleAndVariantFailures(t *testing.T)
}, },
{ {
name: "deterministic validator with profile", name: "deterministic validator with profile",
content: replaceRequiredOnce(t, base, " extract: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: generic/valid_json\n llm_profile: forbidden-profile\n"), content: replaceRequiredOnce(t, base, " extract:\n module: dnd/spells\n", " extract:\n module: dnd/spells\n validators:\n - module: generic/valid_json\n llm_profile: forbidden-profile\n"),
options: productionCLIOptions(t), options: productionCLIOptions(t),
fragments: []string{"deterministic validator", "llm_profile"}, fragments: []string{"deterministic validator", "llm_profile"},
}, },
@@ -779,7 +779,7 @@ func TestProductionConfigValidationCoversModuleAndVariantFailures(t *testing.T)
func TestProductionNormalizeValidatorOverrideRemainsAuthoritative(t *testing.T) { func TestProductionNormalizeValidatorOverrideRemainsAuthoritative(t *testing.T) {
base := string(readRepositoryFile(t, "examples", "dnd-minimal.config.yml")) base := string(readRepositoryFile(t, "examples", "dnd-minimal.config.yml"))
content := replaceRequiredOnce(t, base, " normalize: dnd/spells\n", " normalize:\n module: dnd/spells\n validators:\n - module: generic/always_accept\n - module: generic/valid_json\n") content := replaceRequiredOnce(t, base, " normalize:\n module: dnd/spells\n", " normalize:\n module: dnd/spells\n validators:\n - module: generic/always_accept\n - module: generic/valid_json\n")
path := writeProductionContractConfig(t, content) path := writeProductionContractConfig(t, content)
components := productionTestComponents(t) components := productionTestComponents(t)
effective, err := loadMaintainedExample(t, path).Resolve(resolveInputForMaintainedExample(components, "dnd-session")) effective, err := loadMaintainedExample(t, path).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
@@ -879,7 +879,7 @@ func maintainedExampleFiles(t *testing.T) []maintainedExample {
func productionSpellCatalogContractConfig(t *testing.T) string { func productionSpellCatalogContractConfig(t *testing.T) string {
t.Helper() t.Helper()
return fmt.Sprintf(`version: 4 return fmt.Sprintf(`version: 5
cache: cache:
chunk_plans: chunk_plans:
mode: bypass mode: bypass
@@ -888,21 +888,26 @@ cache:
directory: "" directory: ""
pipelines: pipelines:
dnd-session: dnd-session:
input: seriatim input:
module: seriatim
references: references:
party: %q party:
glossary: %q path: %q
glossary:
path: %q
artifacts: artifacts:
spells: spells:
extract: extract:
module: dnd/spells module: dnd/spells
retries: 2 retries: 2
references: references:
spell_catalog: %q spell_catalog:
path: %q
normalize: normalize:
module: dnd/spells module: dnd/spells
references: references:
spell_catalog: %q spell_catalog:
path: %q
`, repositoryPath("examples", "dnd-party.txt"), repositoryPath("examples", "dnd-glossary.txt"), repositoryPath("examples", "dnd-spell-catalog.json"), repositoryPath("examples", "dnd-spell-catalog.json")) `, repositoryPath("examples", "dnd-party.txt"), repositoryPath("examples", "dnd-glossary.txt"), repositoryPath("examples", "dnd-spell-catalog.json"), repositoryPath("examples", "dnd-spell-catalog.json"))
} }
@@ -913,14 +918,11 @@ func writeProductionSpellCatalogContractConfig(t *testing.T) string {
func loadMaintainedExample(t *testing.T, path string) config.Config { func loadMaintainedExample(t *testing.T, path string) config.Config {
t.Helper() t.Helper()
fileConfig, err := config.LoadFileConfig(path) loaded, err := config.Load(path, config.LoadOptions{})
if err != nil { if err != nil {
t.Fatalf("load maintained config %q: %v", path, err) t.Fatalf("load maintained config %q: %v", path, err)
} }
cfg := config.Default() cfg := loaded.RuntimeConfig()
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("apply maintained config %q: %v", path, err)
}
if err := cfg.Validate(); err != nil { if err := cfg.Validate(); err != nil {
t.Fatalf("validate maintained config %q: %v", path, err) t.Fatalf("validate maintained config %q: %v", path, err)
} }
@@ -977,7 +979,7 @@ func productionRunOptions(t *testing.T, fake *productionFakeLLMClient) Options {
} }
func productionRunConfig(outputRoot, chunkModule string) string { func productionRunConfig(outputRoot, chunkModule string) string {
return fmt.Sprintf(`version: 4 return fmt.Sprintf(`version: 5
output: output:
directory: %q directory: %q
cache: cache:
@@ -988,8 +990,10 @@ debug:
directory: %q directory: %q
pipelines: pipelines:
dnd-session: dnd-session:
input: seriatim input:
chunk: %s module: seriatim
chunk:
module: %s
output: output:
module: json module: json
options: options:
@@ -998,8 +1002,10 @@ pipelines:
- id: identify-items - id: identify-items
artifacts: artifacts:
item-registry: item-registry:
extract: dnd/item-registry extract:
normalize: dnd/item-registry module: dnd/item-registry
normalize:
module: dnd/item-registry
- id: extract-occurrences - id: extract-occurrences
references: references:
item_registry: item_registry:
@@ -1008,9 +1014,11 @@ pipelines:
lane: item-registry lane: item-registry
artifacts: artifacts:
spells: spells:
extract: dnd/spells extract:
module: dnd/spells
item-occurrences: item-occurrences:
extract: dnd/item-occurrences extract:
module: dnd/item-occurrences
`, outputRoot, filepath.Join(filepath.Dir(outputRoot), "debug"), chunkModule) `, outputRoot, filepath.Join(filepath.Dir(outputRoot), "debug"), chunkModule)
} }

View File

@@ -194,7 +194,7 @@ func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest
func newRecomputeTestRoots(t *testing.T) stateTestRoots { func newRecomputeTestRoots(t *testing.T) stateTestRoots {
t.Helper() t.Helper()
roots := newStateTestRoots(t) roots := newStateTestRoots(t)
config := fmt.Sprintf(`version: 4 config := fmt.Sprintf(`version: 5
output: output:
directory: %q directory: %q
cache: cache:
@@ -208,19 +208,27 @@ debug:
directory: %q directory: %q
pipelines: pipelines:
sample: sample:
input: test/input input:
chunk: test/chunk module: test/input
chunk:
module: test/chunk
steps: steps:
- id: first - id: first
artifacts: artifacts:
producer: producer:
extract: test/extract/producer extract:
merge: test/merge module: test/extract/producer
normalize: test/normalize merge:
module: test/merge
normalize:
module: test/normalize
unrelated: unrelated:
extract: test/extract/unrelated extract:
merge: test/merge module: test/extract/unrelated
normalize: test/normalize merge:
module: test/merge
normalize:
module: test/normalize
- id: second - id: second
references: references:
upstream: upstream:
@@ -229,9 +237,12 @@ pipelines:
lane: producer lane: producer
artifacts: artifacts:
middle: middle:
extract: test/extract/middle extract:
merge: test/merge module: test/extract/middle
normalize: test/normalize merge:
module: test/merge
normalize:
module: test/normalize
- id: third - id: third
references: references:
upstream: upstream:
@@ -240,10 +251,14 @@ pipelines:
lane: middle lane: middle
artifacts: artifacts:
dependent: dependent:
extract: test/extract/dependent extract:
merge: test/merge module: test/extract/dependent
normalize: test/normalize merge:
output: test/recompute-output module: test/merge
normalize:
module: test/normalize
output:
module: test/recompute-output
`, roots.output, roots.plans, roots.checkpoints, roots.debug) `, roots.output, roots.plans, roots.checkpoints, roots.debug)
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil { if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)

View File

@@ -289,7 +289,7 @@ func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T)
workingDir := t.TempDir() workingDir := t.TempDir()
cfg := referenceContractConfig() cfg := referenceContractConfig()
configPath := filepath.Join(configDir, "config.yml") configPath := filepath.Join(configDir, "config.yml")
if err := os.WriteFile(configPath, []byte("version: 4\n"), 0o600); err != nil { if err := os.WriteFile(configPath, []byte("version: 5\n"), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil { if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {

View File

@@ -4,7 +4,6 @@ import (
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json"
"errors" "errors"
"flag" "flag"
"fmt" "fmt"
@@ -16,6 +15,7 @@ import (
"strings" "strings"
"time" "time"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/notarius/internal/buildinfo" "gitea.maximumdirect.net/eric/notarius/internal/buildinfo"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts" "gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
@@ -34,8 +34,8 @@ const usage = `Usage:
notarius help notarius help
notarius --version notarius --version
notarius run <pipeline-id> --input path/to/source.json [--json] [flags] notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate --config path/to/config.yml [--config-profile name] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json] notarius pipelines list --config path/to/config.yml [--config-profile name] [--json]
` `
type Options struct { type Options struct {
@@ -158,6 +158,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
fs := flag.NewFlagSet("run", flag.ContinueOnError) fs := flag.NewFlagSet("run", flag.ContinueOnError)
fs.SetOutput(io.Discard) fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path") configPath := fs.String("config", "", "config file path")
configProfile := fs.String("config-profile", "", "configuration profile")
inputPath := fs.String("input", "", "source input file path") inputPath := fs.String("input", "", "source input file path")
onlyRaw := fs.String("only", "", "comma-separated artifact lanes") onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
outputDir := fs.String("output-dir", "", "output directory") outputDir := fs.String("output-dir", "", "output directory")
@@ -265,20 +266,12 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return 2 return 2
} }
cfg, loadedConfigPath, err := loadConfig(*configPath, opts) loaded, loadedConfigPath, err := loadConfigKit(*configPath, *configProfile, runConfigurationOverrides(chunkCache, *outputDir, *debugDir), opts)
if err != nil { if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err) writeConfigLoadError(stderr, err)
return 1 return 1
} }
if chunkCache.set { cfg := loaded.RuntimeConfig()
cfg.Cache.ChunkPlans.Mode = chunkCache.value
}
if dir := strings.TrimSpace(*outputDir); dir != "" {
cfg.Output.Directory = dir
}
if dir := strings.TrimSpace(*debugDir); dir != "" {
cfg.Debug.Directory = dir
}
if err := cfg.Validate(); err != nil { if err := cfg.Validate(); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err) fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1 return 1
@@ -336,6 +329,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
InputPath: strings.TrimSpace(*inputPath), InputPath: strings.TrimSpace(*inputPath),
ConfigPath: loadedConfigPath, ConfigPath: loadedConfigPath,
ConfigSource: configSource(*configPath), ConfigSource: configSource(*configPath),
ConfigProfile: effectiveConfigProfile(*configProfile),
OnlyLanes: append([]string(nil), only...), OnlyLanes: append([]string(nil), only...),
ChunkCacheOverride: chunkCache.explicitValue(), ChunkCacheOverride: chunkCache.explicitValue(),
ReasoningEffortOverride: runtimeOverrides.ReasoningEffort, ReasoningEffortOverride: runtimeOverrides.ReasoningEffort,
@@ -394,7 +388,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil { if err := writeSummary(summary, func() error { return summary.WriteInvocation(invocation) }); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err)) return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug invocation metadata: %w", err))
} }
if err := writeSummary(summary, func() error { return summary.WriteRedactedEffectiveConfig(effective) }); err != nil { if err := writeSummary(summary, func() error { return summary.WriteEffectiveConfig(loaded) }); err != nil {
return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug effective config: %w", err)) return failPipelineCommand(stderr, commandState, terminalWriter, fmt.Errorf("write debug effective config: %w", err))
} }
if err := writeSummary(summary, func() error { return summary.WriteResolvedPipeline(effective) }); err != nil { if err := writeSummary(summary, func() error { return summary.WriteResolvedPipeline(effective) }); err != nil {
@@ -881,13 +875,39 @@ func reorderRunArgs(args []string) []string {
func runFlagTakesValue(arg string) bool { func runFlagTakesValue(arg string) bool {
switch arg { switch arg {
case "--config", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step": case "--config", "--config-profile", "--input", "--only", "--output-dir", "--debug-dir", "--llm-profile", "--session-id", "--reasoning-effort", "--chunk_cache", "--reference", "--without-reference", "--recompute-step":
return true return true
default: default:
return false return false
} }
} }
func runConfigurationOverrides(chunkCache chunkCacheFlag, outputDir, debugDir string) []configkit.Override {
overrides := make([]configkit.Override, 0, 3)
if chunkCache.set {
overrides = append(overrides, configkit.Override{
Path: configkit.Keys("cache", "chunk_plans", "mode"),
Value: string(chunkCache.value),
Flag: "--chunk_cache",
})
}
if outputDir = strings.TrimSpace(outputDir); outputDir != "" {
overrides = append(overrides, configkit.Override{
Path: configkit.Keys("output", "directory"),
Value: outputDir,
Flag: "--output-dir",
})
}
if debugDir = strings.TrimSpace(debugDir); debugDir != "" {
overrides = append(overrides, configkit.Override{
Path: configkit.Keys("debug", "directory"),
Value: debugDir,
Flag: "--debug-dir",
})
}
return overrides
}
type chunkCacheFlag struct { type chunkCacheFlag struct {
value pipeline.ChunkCacheMode value pipeline.ChunkCacheMode
set bool set bool
@@ -941,7 +961,7 @@ func chunkPlanStoreForRun(cfg config.ChunkPlanCacheConfig, opts Options) (pipeli
func validateRunFlagValues(args []string) error { func validateRunFlagValues(args []string) error {
for i, arg := range args { for i, arg := range args {
if arg != "--session-id" && arg != "--reasoning-effort" { if arg != "--config-profile" && arg != "--session-id" && arg != "--reasoning-effort" {
continue continue
} }
if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") { if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") {
@@ -951,6 +971,13 @@ func validateRunFlagValues(args []string) error {
return nil return nil
} }
func effectiveConfigProfile(profile string) string {
if profile == "" || profile == "default" {
return ""
}
return profile
}
func flagWasProvided(args []string, name string) bool { func flagWasProvided(args []string, name string) bool {
for _, arg := range args { for _, arg := range args {
if arg == name || strings.HasPrefix(arg, name+"=") { if arg == name || strings.HasPrefix(arg, name+"=") {
@@ -1033,67 +1060,6 @@ func runConfig(args []string, stdout, stderr io.Writer, opts Options) int {
} }
} }
func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) int {
fs := flag.NewFlagSet("config validate", flag.ContinueOnError)
fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path")
pipelineID := fs.String("pipeline", "", "pipeline ID")
onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
if err := fs.Parse(args); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
if fs.NArg() != 0 {
fmt.Fprintf(stderr, "notarius: unexpected argument %q\n", fs.Arg(0))
return 2
}
if strings.TrimSpace(*onlyRaw) != "" && strings.TrimSpace(*pipelineID) == "" {
fmt.Fprintln(stderr, "notarius: --only requires --pipeline")
return 2
}
only, err := parseOnly(*onlyRaw)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
cfg, path, err := loadConfig(*configPath, opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if strings.TrimSpace(*pipelineID) != "" {
catalog, err := effectiveCatalog(opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
effective, err := cfg.Resolve(config.ResolveInput{
PipelineID: *pipelineID,
Only: only,
Catalog: catalog,
})
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if err := validateExplicitPromptKitProfiles(context.Background(), effective.Config, effectiveLLMProfileIDs(effective.ResolvedPipeline), opts.promptKitAssets); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
fmt.Fprintf(stdout, "config %q is valid for pipeline %q\n", path, strings.TrimSpace(*pipelineID))
return 0
}
if err := cfg.Validate(); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
fmt.Fprintf(stdout, "config %q is valid\n", path)
return 0
}
func runPipelines(args []string, stdout, stderr io.Writer, opts Options) int { func runPipelines(args []string, stdout, stderr io.Writer, opts Options) int {
if len(args) == 0 { if len(args) == 0 {
fmt.Fprintln(stderr, "notarius: pipelines requires a subcommand") fmt.Fprintln(stderr, "notarius: pipelines requires a subcommand")
@@ -1110,50 +1076,6 @@ func runPipelines(args []string, stdout, stderr io.Writer, opts Options) int {
} }
} }
func runPipelinesList(args []string, stdout, stderr io.Writer, opts Options) int {
fs := flag.NewFlagSet("pipelines list", flag.ContinueOnError)
fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path")
jsonOutput := fs.Bool("json", false, "write JSON output")
if err := fs.Parse(args); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
if fs.NArg() != 0 {
fmt.Fprintf(stderr, "notarius: unexpected argument %q\n", fs.Arg(0))
return 2
}
cfg, _, err := loadConfig(*configPath, opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if err := cfg.Validate(); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
ids := sortedPipelineIDs(cfg)
if *jsonOutput {
payload := struct {
Pipelines []string `json:"pipelines"`
}{Pipelines: ids}
encoded, err := json.Marshal(payload)
if err != nil {
fmt.Fprintf(stderr, "notarius: marshal pipeline list: %v\n", err)
return 1
}
fmt.Fprintf(stdout, "%s\n", encoded)
return 0
}
for _, id := range ids {
fmt.Fprintln(stdout, id)
}
return 0
}
func loadConfig(configPath string, opts Options) (config.Config, string, error) { func loadConfig(configPath string, opts Options) (config.Config, string, error) {
path, err := discoverConfigPath(configPath, opts) path, err := discoverConfigPath(configPath, opts)
if err != nil { if err != nil {

View File

@@ -12,6 +12,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts" "gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint" "gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline" "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
@@ -37,6 +38,9 @@ func TestRunControlsRejectSyntaxWithoutAllocatingState(t *testing.T) {
{name: "blank debug directory", args: func(roots stateTestRoots) []string { {name: "blank debug directory", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", ""} return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", ""}
}}, }},
{name: "missing configuration profile", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--config-profile"}
}},
{name: "debug directory without debug", args: func(roots stateTestRoots) []string { {name: "debug directory without debug", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", filepath.Join(filepath.Dir(roots.debug), "requested-debug")} return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", filepath.Join(filepath.Dir(roots.debug), "requested-debug")}
}}, }},
@@ -79,7 +83,7 @@ func TestRecomputeStepCLIContract(t *testing.T) {
{ {
name: "explicit step", name: "explicit step",
configure: func(t *testing.T, roots stateTestRoots) { configure: func(t *testing.T, roots stateTestRoots) {
replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n") replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract:\n module: test/extract\n merge:\n module: test/merge\n normalize:\n module: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract:\n module: test/extract\n merge:\n module: test/merge\n normalize:\n module: test/normalize\n")
}, },
flags: []string{"--resume", "--recompute-step", "chosen"}, flags: []string{"--resume", "--recompute-step", "chosen"},
wantCode: 0, wantCode: 0,
@@ -161,7 +165,7 @@ func TestRunOnlyExecutesSelectedLanes(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
data = []byte(replaceRequiredOnce(t, string(data), " output: test/output\n", " other:\n extract: test/extract\n output: test/output\n")) data = []byte(replaceRequiredOnce(t, string(data), " output:\n module: test/output\n", " other:\n extract:\n module: test/extract\n output:\n module: test/output\n"))
if err := os.WriteFile(roots.config, data, 0o600); err != nil { if err := os.WriteFile(roots.config, data, 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -180,7 +184,7 @@ func TestRunOnlyExecutesSelectedLanes(t *testing.T) {
} }
func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) { func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
t.Run("environment roots", func(t *testing.T) { t.Run("retired environment roots have no effect", func(t *testing.T) {
roots := newStateTestRoots(t) roots := newStateTestRoots(t)
environmentOutput := filepath.Join(t.TempDir(), "environment-output") environmentOutput := filepath.Join(t.TempDir(), "environment-output")
environmentDebug := filepath.Join(t.TempDir(), "environment-debug") environmentDebug := filepath.Join(t.TempDir(), "environment-debug")
@@ -193,10 +197,10 @@ func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
if result.code != 0 { if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr) t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
} }
assertFile(t, filepath.Join(environmentOutput, filepath.Base(onlyChildDir(t, environmentOutput)), "result.json")) assertFile(t, filepath.Join(roots.output, filepath.Base(onlyChildDir(t, roots.output)), "result.json"))
onlyChildDir(t, environmentDebug) onlyChildDir(t, roots.debug)
assertAbsent(t, roots.output) assertAbsent(t, environmentOutput)
assertAbsent(t, roots.debug) assertAbsent(t, environmentDebug)
}) })
t.Run("command flags override environment", func(t *testing.T) { t.Run("command flags override environment", func(t *testing.T) {
@@ -227,23 +231,57 @@ func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
text := string(data) text := string(data)
text = replaceRequiredOnce(t, text, fmt.Sprintf(" directory: %q\n", roots.output), "") text = replaceRequiredOnce(t, text, fmt.Sprintf("output:\n directory: %q\n", roots.output), "")
text = replaceRequiredOnce(t, text, fmt.Sprintf(" directory: %q\n", roots.debug), "") text = replaceRequiredOnce(t, text, fmt.Sprintf("debug:\n directory: %q\n", roots.debug), "")
if err := os.WriteFile(roots.config, []byte(text), 0o600); err != nil { if err := os.WriteFile(roots.config, []byte(text), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
workDir := t.TempDir()
t.Chdir(workDir)
opts := newStateTestHarness().options() opts := newStateTestHarness().options()
result := runWithStateRoots(t, roots, opts, nil) result := runWithStateRoots(t, roots, opts, nil)
if result.code != 0 { if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr) t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
} }
assertFile(t, filepath.Join(workDir, "notarius-output", filepath.Base(onlyChildDir(t, filepath.Join(workDir, "notarius-output"))), "result.json")) configDirectory := filepath.Dir(roots.config)
onlyChildDir(t, filepath.Join(workDir, "notarius-debug")) assertFile(t, filepath.Join(configDirectory, "notarius-output", filepath.Base(onlyChildDir(t, filepath.Join(configDirectory, "notarius-output"))), "result.json"))
onlyChildDir(t, filepath.Join(configDirectory, "notarius-debug"))
}) })
} }
func TestRunAppliesConfigurationProfileAndRecordsSelection(t *testing.T) {
roots := newStateTestRoots(t)
profileDirectory := filepath.Join(filepath.Dir(roots.config), "profiles")
profileOutput := filepath.Join(profileDirectory, "profile-output")
profileDebug := filepath.Join(profileDirectory, "profile-debug")
writeConfigKitFile(t, filepath.Join(profileDirectory, "development.yml"), "output:\n directory: ./profile-output\ndebug:\n directory: ./profile-debug\n")
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--config-profile", "development",
"--input", roots.input, "--chunk_cache", "bypass", "--debug",
}, &stdout, &stderr, newStateTestHarness().options())
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertFile(t, filepath.Join(profileOutput, filepath.Base(onlyChildDir(t, profileOutput)), "result.json"))
bundle := onlyChildDir(t, profileDebug)
var invocation debugbundle.Invocation
readStateTestSummaryJSON(t, bundle, "invocation.json", &invocation)
if invocation.ConfigProfile != "development" {
t.Fatalf("config profile = %q, want development", invocation.ConfigProfile)
}
}
func TestEffectiveConfigProfileOmitsBaseSelection(t *testing.T) {
for _, profile := range []string{"", "default"} {
if got := effectiveConfigProfile(profile); got != "" {
t.Fatalf("effectiveConfigProfile(%q) = %q, want empty", profile, got)
}
}
if got := effectiveConfigProfile("development"); got != "development" {
t.Fatalf("effectiveConfigProfile(development) = %q", got)
}
}
func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) { func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) { t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
roots := newStateTestRoots(t) roots := newStateTestRoots(t)
@@ -567,7 +605,7 @@ func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
data = []byte(replaceRequiredOnce(t, string(data), "extract: test/extract", "extract: test/failing-extract")) data = []byte(replaceRequiredOnce(t, string(data), "module: test/extract", "module: test/failing-extract"))
if err := os.WriteFile(roots.config, data, 0o600); err != nil { if err := os.WriteFile(roots.config, data, 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -92,7 +92,7 @@ func TestRunResultReportsSuccessfulRejection(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n validation_policy:\n semantic_rejection: reject_output\n")) configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize:\n module: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - module: generic/always_reject\n validation_policy:\n semantic_rejection: reject_output\n"))
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil { if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -258,7 +258,7 @@ func TestRunRedactsSensitiveModuleOptionsFromConfigAndPipelineSummaries(t *testi
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
configText := replaceRequiredOnce(t, string(data), " input: test/input\n", ` input: configText := replaceRequiredOnce(t, string(data), " input:\n module: test/input\n", ` input:
module: test/input module: test/input
options: options:
api_key: CONFIG_SUMMARY_SECRET_SENTINEL api_key: CONFIG_SUMMARY_SECRET_SENTINEL
@@ -276,7 +276,7 @@ func TestRunRedactsSensitiveModuleOptionsFromConfigAndPipelineSummaries(t *testi
t.Fatalf("code=%d stderr=%q", result.code, result.stderr) t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
} }
summaryRoot := filepath.Join(onlyChildDir(t, roots.debug), "summary") summaryRoot := filepath.Join(onlyChildDir(t, roots.debug), "summary")
for _, name := range []string{"effective-config.json", "resolved-pipeline.json"} { for _, name := range []string{"effective-config.yml", "resolved-pipeline.json"} {
contents, err := os.ReadFile(filepath.Join(summaryRoot, name)) contents, err := os.ReadFile(filepath.Join(summaryRoot, name))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -687,7 +687,7 @@ func newStateTestRoots(t *testing.T) stateTestRoots {
t.Fatal(err) t.Fatal(err)
} }
roots.config = filepath.Join(base, "config.yml") roots.config = filepath.Join(base, "config.yml")
config := fmt.Sprintf("version: 4\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug) config := fmt.Sprintf("version: 5\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input:\n module: test/input\n chunk:\n module: test/chunk\n artifacts:\n items:\n extract:\n module: test/extract\n merge:\n module: test/merge\n normalize:\n module: test/normalize\n output:\n module: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil { if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -40,6 +40,9 @@ func normalizeDocumentPaths(document *Document, resolution *configkit.Resolution
if profile.Output != nil { if profile.Output != nil {
normalizeDocumentBindingReferencePaths(profile.Output, path.Key("output"), resolution, state) normalizeDocumentBindingReferencePaths(profile.Output, path.Key("output"), resolution, state)
} }
if documentStepsContainRelativeReferencePaths(profile.Steps) {
resolution.Record(path.Key("steps"), configkit.ChangeNormalization, "resolved configured paths in ordered steps")
}
for laneID, lane := range profile.Artifacts { for laneID, lane := range profile.Artifacts {
normalizeDocumentLaneReferencePaths(&lane, path.Key("artifacts").Key(laneID), resolution, state) normalizeDocumentLaneReferencePaths(&lane, path.Key("artifacts").Key(laneID), resolution, state)
profile.Artifacts[laneID] = lane profile.Artifacts[laneID] = lane
@@ -57,6 +60,31 @@ func normalizeDocumentPaths(document *Document, resolution *configkit.Resolution
} }
} }
func documentStepsContainRelativeReferencePaths(steps []PipelineStepDocument) bool {
for _, step := range steps {
if documentReferencesContainRelativePath(step.References) {
return true
}
for _, lane := range step.Artifacts {
if documentReferencesContainRelativePath(lane.Extract.References) ||
(lane.Merge != nil && documentReferencesContainRelativePath(lane.Merge.References)) ||
(lane.Normalize != nil && documentReferencesContainRelativePath(lane.Normalize.References)) {
return true
}
}
}
return false
}
func documentReferencesContainRelativePath(references map[string]ReferenceDocument) bool {
for _, source := range references {
if source.Path != nil && *source.Path != "" && !filepath.IsAbs(*source.Path) {
return true
}
}
return false
}
func initializeDocumentMainDirectory(resolution *configkit.Resolution, state *documentLoadState) { func initializeDocumentMainDirectory(resolution *configkit.Resolution, state *documentLoadState) {
if state == nil || state.mainDirectory != "" { if state == nil || state.mainDirectory != "" {
return return

View File

@@ -41,6 +41,7 @@ type Invocation struct {
InputPath string `json:"input_path,omitempty"` InputPath string `json:"input_path,omitempty"`
ConfigPath string `json:"config_path,omitempty"` ConfigPath string `json:"config_path,omitempty"`
ConfigSource string `json:"config_source,omitempty"` ConfigSource string `json:"config_source,omitempty"`
ConfigProfile string `json:"config_profile,omitempty"`
OnlyLanes []string `json:"only_lanes,omitempty"` OnlyLanes []string `json:"only_lanes,omitempty"`
ChunkCacheOverride string `json:"chunk_cache_override,omitempty"` ChunkCacheOverride string `json:"chunk_cache_override,omitempty"`
ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"` ReasoningEffortOverride *string `json:"reasoning_effort_override,omitempty"`